Single Pass Cel Shading

23Jan/124

Cel shading - aka "toon shading" - is an effect used to make 3D rendering look like cartoon. It was used in many flavours in a few games such as XIII or Zelda Wind Wakers.

TL;DR

Click on the picture above to launch the demonstration.

The final rendering owns a lot to the orginal diffuse textures used for rendering. But this cartoon style is also achieved using two distinct features on the GPU:

The light is discretized using a level function.
A black outline on the edges.

This is usually done using two passes (or even more). One pass renders the scene in the backbuffer with the lighting modified by the level function. Another pass renders the normals only and then pass is done as a post-process to perform an edge detection algorithm (a Sobel filter) to add the black outline.

Another technique uses two passes: one to render the object with the lighting, and a second one with front-face culling and a scale offset to render the outline with a solid black color.

But I thought it might be done more efficiently in one pass using a few tricks. The most difficult part here is how to get the black outline. Indeed, we are working on a per-vertex (vertex shader) or per-pixel (fragment shader) basis. Thus, it's pretty hard to work on edges. It's pretty much the same problem we encountered when working on wireframe, except here we want to outline the object and not the triangles. But this little difference actually makes it a lot easier for us.

Why? Because detecting the edges of the 3D shape is much easier.

The Outline

The trick is to get the angle between the eye-to-vertex vector and the vertex normal. If that very angle is close to PI/2 (=90°), then the vertex is on an edge. If the vertex is on an edge, then we will displace it a bit toward its normal. The vertices deplaced this way will form an outline around around shape:

public class CelShadingShader
{
  private var _isEdge	: SValue	= null;
 
  override protected function getOutputPosition() : SValue
  {
	var eyeToVertex : SValue = normalize(subtract(vertexPosition, cameraLocalPosition));
 
	_isEdge = lessThan(
		-0.05,
		dotProduct3(vertexNormal, eyeToVertex)
	);
 
	var thickness : SValue	= getNamedParameter("thickness", .04);
	var delta : SValue = multiply(_isEdge, vertexNormal.xyz, thickness);
 
	return multiply4x4(
		add(vertexPosition, float4(delta, 0)),
		localToScreenMatrix
	);
  }
 
  override protected function getOutputColor() : SValue
  {
	var outline : SValue = lessThan(interpolate(_isEdge), 0.1);
 
	return float4(float3(outline), 1);
  }
}

Our fragment shader is pretty simple here: _isEdge is supposed to contain 1 if the vertex is on an edge, 0 otherwise. Therefore, as we want our outline to be black, we simply use the "lessThan" operation. If the vertex is on an edge, the outline value will be 0 and 1 otherwise. We just have to multiply "outline" with whatever color we want to use.

You'll get the following result:

The Light Level Function

There are many ways to transform a continuous per-pixel lighting equation into another one that will use levels. The best way to make it possible for the artists to customize it is to use a light texture. The Lambert factor is then used as the UVs to sample that texture.

But here I wanted this effect to rely on no textures (except a diffuse one eventually). So I implemented a very simple level function using an euclidian division:

private static const NUM_LEVELS			: uint		= 6;
 
private static const DEFAULT_THICKNESS		: Number	= 0.04;
private static const DEFAULT_AMBIENT		: Number	= .2;
private static const DEFAULT_LIGHT_DIRECTION	: ConstVector4	= new ConstVector4(1., -1., 0.);
private static const DEFAULT_DIFFUSE_COLOR	: ConstVector4	= new ConstVector4(1., 1., 1., 1.);
 
override protected function getOutputColor() : SValue
{
  var vertexNormal : SValue = normalize(interpolate(vertexNormal));
  var lightDirection : SValue = getNamedParameter("light direction", DEFAULT_LIGHT_DIRECTION);
 
  lightDirection.normalize();
 
  // diffuse lighting
  var lambertFactor : SValue = saturate(
    negate(dotProduct3(lightDirection, vertexNormal))
  );
 
  // cel shading
  lambertFactor = multiply(lambertFactor, NUM_LEVELS);
  lambertFactor = subtract(lambertFactor, fractional(lambertFactor));
  lambertFactor = divide(lambertFactor, NUM_LEVELS);
 
  // ambient lighting
  var ambient	: SValue = getNamedParameter("ambient", DEFAULT_AMBIENT);
 
  lambertFactor.incrementBy(ambient);
 
  var diffuseColor : SValue = getNamedParameter("diffuse color", DEFAULT_DIFFUSE_COLOR);
 
  return float4(
    multiply(diffuseColor.rgb, lambertFactor),
    diffuseColor.a
  );
}

Here is what you get:

The Final Shader

You can combine both effects by simply multiply the Lambert factor by the outline value we used to output.

And voilà!

public class CelShadingShader
{
  private static const NUM_LEVELS		: uint		= 6;
 
  private static const DEFAULT_THICKNESS	: Number	= 0.04;
  private static const DEFAULT_AMBIENT		: Number	= .2;
  private static const DEFAULT_LIGHT_DIRECTION	: ConstVector4	= new ConstVector4(1., -1., 0.);
  private static const DEFAULT_DIFFUSE_COLOR	: ConstVector4	= new ConstVector4(1., 1., 1., 1.);
 
  private var _isEdge	: SValue	= null;
 
  override protected function getOutputPosition() : SValue
  {
	var eyeToVertex : SValue = normalize(subtract(vertexPosition, cameraLocalPosition));
 
	_isEdge = lessThan(
		-0.05,
		dotProduct3(vertexNormal, eyeToVertex)
	);
 
	var thickness : SValue	= getNamedParameter("thickness", .04);
	var delta : SValue = multiply(_isEdge, vertexNormal.xyz, thickness);
 
	return multiply4x4(
		add(vertexPosition, float4(delta, 0)),
		localToScreenMatrix
	);
  }
 
  override protected function getOutputColor() : SValue
  {
    var vertexNormal : SValue = normalize(interpolate(vertexNormal));
    var lightDirection : SValue = getNamedParameter(
      "light direction",
      DEFAULT_LIGHT_DIRECTION
    );
 
    lightDirection.normalize();
 
    // diffuse lighting
    var lambertFactor : SValue = saturate(
      negate(dotProduct3(lightDirection, vertexNormal))
    );
 
    // cel shading
    lambertFactor = multiply(lambertFactor, NUM_LEVELS);
    lambertFactor = subtract(lambertFactor, fractional(lambertFactor));
    lambertFactor = divide(lambertFactor, NUM_LEVELS);
 
    // ambient lighting
    var ambient	: SValue = getNamedParameter("ambient", DEFAULT_AMBIENT);
 
    lambertFactor.incrementBy(ambient);
 
    var diffuseColor : SValue = getNamedParameter(
      "diffuse color",
      DEFAULT_DIFFUSE_COLOR
    );
 
    // outline
    var outline : SValue = lessThan(interpolate(_isEdge), 0.1);
 
    lambertFactor.scaleBy(outline);
 
    return float4(
      multiply(diffuseColor.rgb, lambertFactor),
      diffuseColor.a
    );
  }
}

Tagged as: 3D, ActionScript 3.0, Flash 11, hardware acceleration, Minko, shader, Stage3D Leave a comment

Comments (4) Trackbacks (2) ( subscribe to comments on this post )

Daniel Freeman
January 26th, 2012 - 17:57

That’s clever. Playing with the demo first, I was trying to figure out how you could possibly draw the outline. Then I read the explanation, about using the normal angle, and it made me feel stupid for figuring it out.

( REPLY )
- Promethe
  January 26th, 2012 - 18:08
  
  Thanks. Unfortunately it works only on a geometry with specific constraints.
  But when it works, it works really well and its painless.
  
  I’ll make another article about other cel shading methods.
  
  ( REPLY )
Pierre Chamberlain
January 30th, 2012 - 12:56

Very nice shader!

Just letting you know there’s a typo in your first paragraph following “The Outline” header:

“If the vertex is on an edge, then we will displace it a bit **toard** its normal”
(I believe you meant **toward**?)

( REPLY )
- Promethe
  January 30th, 2012 - 13:07
  
  Thanks! I’ll fix the typo right now
  
  ( REPLY )

Aerys

Twitter: promethe42

@makc3d @inspirit @joa @pleclech we created AS3 shaders, much more powerful than GLSL/FLSL, why don't you use it? 11:37:54 PM May 14, 2012 from TweetDeck in reply to makc3d Reply Retweet Favorite
@makc3d yesterday I implemetend a radial blur in 10 minutes. Too bad you still refuse to work with tools for grown ups :) 11:34:00 PM May 14, 2012 from TweetDeck in reply to makc3d Reply Retweet Favorite
just implemented BF3-like screen space lens flare with #minko #as3 #shader http://t.co/kXUZtC0a 11:22:20 PM May 14, 2012 from TweetDeck Reply Retweet Favorite
Someone implemented AS3Mod for Away3D 4 on the CPU... SERIOUSLY?! You REALLY don't want to learn anything new Flashers?! man shaders! 11:03:19 PM May 11, 2012 from TweetDeck Reply Retweet Favorite
RT @DisneyPixar: One thing you’ll never regret is giving your best effort. 02:07:10 PM May 06, 2012 from Twitter for Android Reply Retweet Favorite

@promethe42

Jean-Marc Le Roux Web, RIAs and chocolate spaghettis…