Single Pass Cel Shading

Cel shading – aka “toon shading” – is an effect used to make 3D rendering look like cartoon. It was used in many flavours in a few games such as XIII or Zelda Wind Wakers.

TL;DR

Click on the picture above to launch the demonstration.

The final rendering owns a lot to the orginal diffuse textures used for rendering. But this cartoon style is also achieved using two distinct features on the GPU:

  1. The light is discretized using a level function.
  2. A black outline on the edges.

This is usually done using two passes (or even more). One pass renders the scene in the backbuffer with the lighting modified by the level function. Another pass renders the normals only and then pass is done as a post-process to perform an edge detection algorithm (a Sobel filter) to add the black outline.

Another technique uses two passes: one to render the object with the lighting, and a second one with front-face culling and a scale offset to render the outline with a solid black color.

But I thought it might be done more efficiently in one pass using a few tricks. The most difficult part here is how to get the black outline. Indeed, we are working on a per-vertex (vertex shader) or per-pixel (fragment shader) basis. Thus, it’s pretty hard to work on edges. It’s pretty much the same problem we encountered when working on wireframe, except here we want to outline the object and not the triangles. But this little difference actually makes it a lot easier for us.

Why? Because detecting the edges of the 3D shape is much easier.

The Outline

The trick is to get the angle between the eye-to-vertex vector and the vertex normal. If that very angle is close to PI/2 (=90°), then the vertex is on an edge. If the vertex is on an edge, then we will displace it a bit toward its normal. The vertices deplaced this way will form an outline around around shape:

public class CelShadingShader
{
  private var _isEdge	: SValue	= null;

  override protected function getOutputPosition() : SValue
  {
	var eyeToVertex : SValue = normalize(subtract(vertexPosition, cameraLocalPosition));

	_isEdge = lessThan(
		-0.05,
		dotProduct3(vertexNormal, eyeToVertex)
	);

	var thickness : SValue	= getNamedParameter("thickness", .04);
	var delta : SValue = multiply(_isEdge, vertexNormal.xyz, thickness);

	return multiply4x4(
		add(vertexPosition, float4(delta, 0)),
		localToScreenMatrix
	);
  }

  override protected function getOutputColor() : SValue
  {
	var outline : SValue = lessThan(interpolate(_isEdge), 0.1);

	return float4(float3(outline), 1);
  }
}

Our fragment shader is pretty simple here: _isEdge is supposed to contain 1 if the vertex is on an edge, 0 otherwise. Therefore, as we want our outline to be black, we simply use the “lessThan” operation. If the vertex is on an edge, the outline value will be 0 and 1 otherwise. We just have to multiply “outline” with whatever color we want to use.

You’ll get the following result:

The Light Level Function

There are many ways to transform a continuous per-pixel lighting equation into another one that will use levels. The best way to make it possible for the artists to customize it is to use a light texture. The Lambert factor is then used as the UVs to sample that texture.

But here I wanted this effect to rely on no textures (except a diffuse one eventually). So I implemented a very simple level function using an euclidian division:

private static const NUM_LEVELS			: uint		= 6;

private static const DEFAULT_THICKNESS		: Number	= 0.04;
private static const DEFAULT_AMBIENT		: Number	= .2;
private static const DEFAULT_LIGHT_DIRECTION	: ConstVector4	= new ConstVector4(1., -1., 0.);
private static const DEFAULT_DIFFUSE_COLOR	: ConstVector4	= new ConstVector4(1., 1., 1., 1.);

override protected function getOutputColor() : SValue
{
  var vertexNormal : SValue = normalize(interpolate(vertexNormal));
  var lightDirection : SValue = getNamedParameter("light direction", DEFAULT_LIGHT_DIRECTION);

  lightDirection.normalize();

  // diffuse lighting
  var lambertFactor : SValue = saturate(
    negate(dotProduct3(lightDirection, vertexNormal))
  );

  // cel shading
  lambertFactor = multiply(lambertFactor, NUM_LEVELS);
  lambertFactor = subtract(lambertFactor, fractional(lambertFactor));
  lambertFactor = divide(lambertFactor, NUM_LEVELS);

  // ambient lighting
  var ambient	: SValue = getNamedParameter("ambient", DEFAULT_AMBIENT);

  lambertFactor.incrementBy(ambient);

  var diffuseColor : SValue = getNamedParameter("diffuse color", DEFAULT_DIFFUSE_COLOR);

  return float4(
    multiply(diffuseColor.rgb, lambertFactor),
    diffuseColor.a
  );
}

Here is what you get:

The Final Shader

You can combine both effects by simply multiply the Lambert factor by the outline value we used to output.

And voilà!

public class CelShadingShader
{
  private static const NUM_LEVELS		: uint		= 6;

  private static const DEFAULT_THICKNESS	: Number	= 0.04;
  private static const DEFAULT_AMBIENT		: Number	= .2;
  private static const DEFAULT_LIGHT_DIRECTION	: ConstVector4	= new ConstVector4(1., -1., 0.);
  private static const DEFAULT_DIFFUSE_COLOR	: ConstVector4	= new ConstVector4(1., 1., 1., 1.);

  private var _isEdge	: SValue	= null;

  override protected function getOutputPosition() : SValue
  {
	var eyeToVertex : SValue = normalize(subtract(vertexPosition, cameraLocalPosition));

	_isEdge = lessThan(
		-0.05,
		dotProduct3(vertexNormal, eyeToVertex)
	);

	var thickness : SValue	= getNamedParameter("thickness", .04);
	var delta : SValue = multiply(_isEdge, vertexNormal.xyz, thickness);

	return multiply4x4(
		add(vertexPosition, float4(delta, 0)),
		localToScreenMatrix
	);
  }

  override protected function getOutputColor() : SValue
  {
    var vertexNormal : SValue = normalize(interpolate(vertexNormal));
    var lightDirection : SValue = getNamedParameter(
      "light direction",
      DEFAULT_LIGHT_DIRECTION
    );

    lightDirection.normalize();

    // diffuse lighting
    var lambertFactor : SValue = saturate(
      negate(dotProduct3(lightDirection, vertexNormal))
    );

    // cel shading
    lambertFactor = multiply(lambertFactor, NUM_LEVELS);
    lambertFactor = subtract(lambertFactor, fractional(lambertFactor));
    lambertFactor = divide(lambertFactor, NUM_LEVELS);

    // ambient lighting
    var ambient	: SValue = getNamedParameter("ambient", DEFAULT_AMBIENT);

    lambertFactor.incrementBy(ambient);

    var diffuseColor : SValue = getNamedParameter(
      "diffuse color",
      DEFAULT_DIFFUSE_COLOR
    );

    // outline
    var outline : SValue = lessThan(interpolate(_isEdge), 0.1);

    lambertFactor.scaleBy(outline);

    return float4(
      multiply(diffuseColor.rgb, lambertFactor),
      diffuseColor.a
    );
  }
}

27 thoughts on “Single Pass Cel Shading

  1. Pingback: Single Pass Cel Shading | Jean-Marc Le Roux | Everything about Flash | Scoop.it

  2. Daniel Freeman

    That’s clever. Playing with the demo first, I was trying to figure out how you could possibly draw the outline. Then I read the explanation, about using the normal angle, and it made me feel stupid for figuring it out.

    Reply
    1. Promethe Post author

      Thanks. Unfortunately it works only on a geometry with specific constraints.
      But when it works, it works really well and its painless.

      I’ll make another article about other cel shading methods.

      Reply
  3. Pierre Chamberlain

    Very nice shader!

    Just letting you know there’s a typo in your first paragraph following “The Outline” header:

    “If the vertex is on an edge, then we will displace it a bit **toard** its normal”
    (I believe you meant **toward**?)

    Reply
  4. Pingback: ShaderLab: Embedding Shaders In A Web Page | Jean-Marc Le Roux

  5. David

    I’ve been trying to recreate this in as3 AGAL code based on your explanation but i’m not getting very far. Do you have any example of the AGAL code that this uses?

    Reply
    1. Promethe Post author

      Why would you use AGAL when you can work with AS3? The full code for the single-pass CelShading shader is in the minko-examples repository.

      If you want to have a look at a sample of the AGAL code generated by the Minko’s JIT compiler you can use:

      Minko.debugLevel = DebugLevel.SHADER_AGAL;
      

      But don’t think that’s it:
      - The AGAL code is optimized by the compiler, so it will not likely be as naïve as what we – poor humans – would write: some operations are automatically deferred to the CPU, the constants are compressed, etc…
      - The AS3 shader is parametric, so it can output many different AGAL code depending on the settings of the scene.

      In the end, the AS3 shader is much easier to read/write, much faster but also much much more flexible. It will always outperform AGAL even for the simplest things.

      Take your “3d earth” snippet for example: your rendering two spheres when you could easily create a shader that blends the earth diffuse texture with the clouds. And you could also easily add normal mapping and lighting thanks to minko-lighting‘s shader parts. All of this with nothing more than pure plain AS3… Don’t believe me? Here you go:

      public class EarthShader extends BasicShader
      {
        override protected function getPixelColor() : SFloat
        {
          var uv : SFloat = interpolate(vertexUV);
      
          uv.incrementBy(float2(0, divide(time, 1000)));
      
          var diffuse : SFloat = sampleTexture(meshBindings.getTexture('diffuseMap'), uv);
          var clouds : SFloat = sampleTexture(meshBindings.getTexture('cloudsMap'), uv);
      
          return float4(add(diffuse.rgb, clouds.rgb), 1);
        }
      }
      

      And here to create the scene:

      var earth : Mesh = new Mesh(
        new SphereGeometry(30),
        {
          diffuseMap : TextureLoader.load(new URLRequest('diffuse.jpg')),
          cloudsMap : TextureLoader.load(new URLRequest('clouds.jpg'))
        },
        new Effect(new EarthShader())
      );    
      

      And voilà: earth with spinning clouds using hardware accelerated blending.

      You should stop using AGAL: it’s conter-productive and it was never made for you – 3D applications developers – to build shaders with. Never. If the engine you’re using offer no other solution, then it’s time to switch engines don’t you think? :)

      Reply
  6. David

    I see, well I appreciate the advice and while I can see why AGAL may be counter productive in a serious project, I am not using this for any serious projects.

    It is just something I started to play around with for learning purposes and for fun. The satisfaction I get is from learning how something works and then putting this understanding to some use. Usually in doing so i pickup other things along the way. I’m sure you can understand that as a programmer.

    If I were looking for a 3d library to create a game or something more substantial, there is no doubt i would investigate my options and pick something that met my needs and something that wouldn’t mean I was staring at AGAL code for hours on end!

    The example you have shown certainly looks a lot simpler than mine, perhaps in the future i will do some experiments using Minko too and see what i learn ;)

    As for the cellshader I managed to create something similar but i am still tweaking it, thank you again for the blog post it has helped my understanding of how this works.

    Reply
  7. Promethe Post author

    I think that by “learning purposes and for fun” you mean “learning shader/GPU programming”. But learning AGAL and learning GPU programming are two very different things. Actually, learning AGAL will keep you away from the real deal behind GPU programming.

    Why? For the exact same reason you’re struggling right now to try to mimic with AGAL something that took me 1 hour to create from scratch with AS3: you’re not using the right technology. If you read any interesting papers about GPU programming it’s unlikely you’ll find any assembly code. Even for the most “basic” things such as cel-shading or shadow mapping.

    Pretending learning/mastering AGAL has anything to do with what actual GPU programming is about is like pretending that learning AS3 bytecode will make you a better OOP programmer.

    The satisfaction I get is from learning how something works and then putting this understanding to some use. I’m sure you can understand that as a programmer.

    As a programmer, I know that the language I use and the concept I learn are two completely different things. And if I can use a better language that will make me capable of doing and learning a lot more I won’t hesitate. In this case, it’s even better because AS3 is a language you already know. So it’s even cooler to discover how you can do completely new things with a language you thought you knew.

    As GPU programmer, I also know that AGAL did not learn me anything GLSL, HLSL or AS3 shaders wouldn’t have. Actually, learning actual high-level shaders programming languages makes me an even better AGAL programmer because using the right tech. made it possible for me to trully experience GPU programming.

    Most of the AGAL code I read is really completely messed up and uses 1/3 of the available features. So the “learning from the low level” is not even real: I’ve never seen any AGAL shader with more than 10 lines that was properly implemented and fully understood by it’s programmer.

    If you were working on a native app, would you use ARB assembly code or GLSL? Of course you would use GLSL. So why using AGAL when you have better alternatives that will make you capable of learning even more?

    Do you think there is more to learn using MS Paint, or Photoshop?

    Reply
  8. David

    wow ok I can see you are quite passionate about this, I agree with everything you are saying but I think you are missing my point or rather my goal.

    I understand that its much easier to not write AGAL but to use something else just like your analogy of ARM assembly code or GLSL. Believe me there is no doubt in my mind if i wanted to make something in 3d thats worthwhile i would use the “right technology” to get it done. Just as if i want to make a good image i would use photoshop over mspaint as it has the tools to get the job done.

    My point is that i’m not doing this to become a shader expert or to continue to write AGAL code, or to master something, its purely a dive into the unknown. Is it something i can do? Lets try and along the way i might learn something new. That is what i mean when i say learning and fun. My goal isn’t particularly to learn shaders or gpu programming, its really just to learn new things.

    Do i need to do this? no. The 3d library I already use has shaders already written for me that i can make use of and have done for some time. I could even use minko or unity or something else if i wanted.

    Have you ever wondered how something worked? That is the driving force behind most of the things i’m doing. Will learning this prove useful in the future maybe, maybe not, will i be able to apply anything i learn here to other things i do? possibly.

    I’ve learnt many things from experimenting with 3d over the last year or so that I knew nothing of before. Just like with AGAL its spured me to learn what different shaders are what matrix multiplication is and am learning other things because of this time spent on it. Ok it could have been learnt in many other ways but its purely a conduit for teaching myself.

    Do i think there is more to learn with MS Paint over Photoshop? No. but if someone said i’ll give you details on how phtoshop works so you can write your own paint bucket tool I’d say yes.

    Just like I want to know how AGAL works behind the shaders i’m using. Would i continue to write every shader i need when they already exist? no. Just as i wouldn’t write every photoshop tool when they already exist.

    would I write my own game engine to make a game? No, i would use an existing one. However i still might want to understand how some of it works.

    I hope that explains where I am coming from a little better. I completely agree that using AGAL is counter productive.

    Reply
    1. Promethe Post author

      Lets try and along the way i might learn something new.

      How can you say you are trying to learn “something new”?

      You’re doing exactly the opposite! :o
      You stick to AGAL because you refuse to learn something more than just what’s inside Flash.
      You stick to AGAL so you write tiny naive programs that won’t require to learn anything but an assembly language – and a bad one with that.
      You even asked me to copy/paste the AGAL code to study it instead of writing it yourself!
      I told you how to get that AGAL code and I’m pretty sure you did not even try…

      You want to learn new things?
      Learn the maths behind GPU programming.
      Learn the techniques behind color manipulations.
      Learn how skeletons can animate a geometry on the GPU.
      Learn how one can simply build an hardware accelerated physics engine with simple maths equations.
      Learn how you can use multiple passes to create stunning rendering effects that are in fact a lot simpler than you ever thought.

      So really… please stop it with the “learning new things” stuff: you’re over thinking this little useless cel-shading trick and make it look like rocket science. But it’s the 101 of shaders.

      We’ve created a JIT compiler so you guys could learn about this new world of GPU programming with a badass programming language: ActionScript 3. That’s a huge amount of work, and all of it for free and open source. And now you’re asking me to copy/paste some AGAL code pretending you want to learn?

      Just like I want to know how AGAL works behind the shaders i’m using.

      If :

      - the 3D engine you’re working with uses shaders written in assembly code
      - or you can understand how its shaders work despite the fact it likely took you more than 1 hour to port this cel-shading stuff with AGAL

      maybe it’s time to use one that was written less than 10 years ago and does some real stuff :)

      Reply
  9. David

    Ok well I can see you didn’t get my point and are being quite rude about it so lets just agree to disagree as this isn’t going anywhere fun.

    Reply
    1. Promethe Post author

      So are you here to learn or make a point?
      They are pretty much rather mutually exclusive when you consider you come here to read about things you don’t know…

      Reply
  10. DevAS3

    Bonjour,
    J’ai une petite question, d’ou viennent lessThan et getNamedParameter qui sont dans le shader de Cell Shading? svp :) J’ai essayé de voir si les noms avaient changé mais non.
    Merci par avance pour votre réponse.

    Reply
    1. Promethe Post author

      lessThan() is a protected method of ShaderPart so it should be available in any class extending Shader.
      getNamedParameter() is now meshBindings.getParameter().

      Reply
  11. DevAS3

    Thanks for your fast answer. I’ve another problem ^^ FlashBuilder don’t recognize meshBindings.getParameter(). What’s the problem? an import?

    Reply
      1. DevAS3

        ooh thanks^^ i worked with 1.0 :) but now i must replace some function like Max3DSParser or SinglePassRenderingEffect ^^ Thanks a lot for your answer. :)

        Reply
        1. Promethe Post author

          SinglePassRenderingEffect does not exist anymore: it has been replaced with the generic Effect class than can handle both single and multi-pass.

          The 3DS parser is not available for 2.0 right now. But you can work with the Collada loader.

          We might add the cel shading in minko-effects. But we will more likely implement the actual algorithm rather than this little geometric trick :)

          If you have more questions you can post on Aerys Answers, Minko’s official support forum.

          Reply
  12. Stimpy

    Hi,

    i found this articel by google searching about drawing outlines. It’s very intressting but i’m to stupid to translate it to agal – Because i don’t use minko. Have you an AGAL Version to draw the outline of a mesh? That were be great!

    Reply
      1. Stimpy

        Hmmmmm, that dosn’t work for me. When i understand the math right, then i have must a 3d edge?! So, i have a plane 2d Mesh – Or what i mean, all my normals goes up. That can’t be work then, or i’m wrong?

        Have you some idea or solution to silhouete / outline a planar mesh?!

        Reply
        1. Promethe Post author

          This implementation uses a geometric trick. It won’t work unless the model is properly tesselated. So a plane is not a good target for this shader.

          You should consider using:
          - two passes cel-shadding, with a lighting pass and a contour pass
          - three passes cel-shadding, with a normal pass, lighting pass and sobel filter contour detection based post processing pass

          Those two techniques are described on wikipedia: http://en.wikipedia.org/wiki/Cel_shading

          Reply
  13. Pingback: Sobel filter using AGAL | Blog about augmented reality application development

  14. Pingback: Cel shading with AGAL - Stage3d - AGAL experiments

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>