Jean-Marc Le Roux Web, RIAs and chocolate spaghettis…

Pixel shader demo using Flash 10, Pixel Bender and Minko

I'm really excited to announce Minko (which is, by the way, the final name for my 3D library) has reached a new level: pixel shader integration! Pixel shaders are little programs that run on each pixel and can modify their final color. They are often written in C-like languages and in this precise case we use Pixel Bender, the shader language introduced with Flash 10.

In this post I will:

• Explain how any 3D scene is built when using Minko
• Explain how pixel shaders are integrated in the 3D scene
• Explain how pixel shaders are built using Pixel Bender
• Show you a very simple demo of the kind of effects pixel shaders will provide
• Explain how the demo was built

Here are a two screenshots to show the results:

Phong shading + spheric environment mapping on a 2700+ polygons Lamborghini

Phong shading + spheric environment mapping demo

The Scene Graph API

This API is used to build and manage the 3D scene. Scene graph is the most common data-structure used to build 3D scenes. It is simple to understand and both memory wise and CPU efficient. To explain what the scene graph is, start thinking about Flash's display list. In ActionScript 3.0, the display list is actually a tree. The root of the tree is the Stage and you can add multiple children to it. Each child can be a leaf, such as a Bitmap object, or a container such as a Sprite. One of the limitations of a tree is that every node can have multiple children but each child can have only one parent. This property is what makes it look like a tree: branches (and leaves for that matter...) have only one parent branch.

A direct consequence of that paradigm is that if you want 20 red circles, you will have to create 20 red circles and add them to the display tree. Despite the fact that those 20 red circles will be exactly the same you will still have to create and add all of them to the display tree. Thus, you will use 19 times more memory than what is actually needed. That's where the graph kicks in...

A graph is just like a tree except every node can have multiple parents. You can see graphs as a generalization of trees. But you can also say that trees are particular graphs where each node can have only one parent. Anyway, we can now create one red circle and use it as many times as we need! To understand how this apply to a 3D scene, lets consider the following rendering:

And here is the corresponding scene graph:

There are five things to notice:

1. The figure doesn't show the whole graph (I removed the camera and materials).
2. The Scene Graph API is all about interfaces. You can add new features to the API and still extend the classes you need/want (ie. EventDispatcher).
3. The CubeMesh.cubeMesh, SquareMesh.squareMesh and SphereMesh.sphereMesh nodes, which contain the primitives geometry, exist only once each but are used by many IDisplayObject3D. Therefore there is no useless memory consumption! The same can be done with any node in the graph: containers, display objects, materials, etc...
4. The orange circled nodes apply a (3D) transformation to their children, just like Flash containers apply (2D) transformations to their own children.
5. Z-Sorting is here enabled by two different technics: the drawing order (which is pretty much the same as in a classic-2D-Flash display tree) and the DepthSortContainer3D node. This node takes care of z-sorting by sorting its children using their average depth. This way, z-sorting occurs only when needed and only on relevant geometry.

There a lot to explain about the Scene Graph API! But the main point is to understand how it compares to Flash's display tree and what benefits it provides: lower memory consumption and powerful scene building. You are likely to need to find a way to fit in the Scene Graph API anytime you want to extend Minko to add a new feature. In this precise case I will talk about how pixel shaders fits in.

Pixel shader integration

The first step is to find "where" in the graph your new feature will be used. Close to the scene "root" you will more likely find containers. And the "leaves" are often more tangible objects such as a materials or meshes. A pixel shader is some kind of a material so that's where we should start looking!

In the Scene Graph API, there is a special way to handle materials: the "material stack". It's a part of the graph - a subgraph really - that looks very much like a stack because every node uses an "underlaying" material node. Lets consider the orange box from the previous example:

This figure shows the orange cube subgraph. On this graph I added the material nodes to demonstrate how the material stack works. In this precise case, the stack is made of a WireframeMaterial node and a SolidMaterial node. The first one provides a wireframe rendering and the second one is a solid fill. When rendering, the resulting graphics operations will be the list of all the graphics instructions outputed by each node of the stack. Thus, we will here obtain an (orange) solid fill with a (purple) wireframe. So where does the pixel shader fits in the material stack? Well you might have noticed there are two different kinds of nodes in the material stack:

1. the bottom of the stack is more likely to be a "base material", like a solid fill or a texture
2. and the rest of the stack will be mainly "modifiers". In the case of the orange cube, the WireframeMaterial node acts as a modifier for the SolidMaterial node

Our pixel shader material will be a modifier because it will work on an underlaying texture. Now we know our pixel shader material should be on the top of the material stack we have to know what data it needs and how to provide it!

Pixel shader creation using Pixel Bender

A pixel shader works at the pixel level (true story). Therefore, we need per-pixel data. But because we use Flash's drawing API we don't have a direct access to the actual pixels (I'm oversimplifying but that's the idea). The trick is to work on the material BitmapData itself rather than the screen pixels. Thus, I will now talk about "texels" (texture pixels) rather than pixels (so I guess it should be called a "texel shader" then...). Anyway, our shader will then most certainly always need three inputs:

1. The color of each texel, also called a "diffuse map" or... a texture. Any material will do the trick (we can get a BitmapData out of any material).
2. The position of each texel, also called a "position map".
3. The normal of each texel, also called a "normal map".

The position map will be generated at runtime. It is mainly a BitmapData where each pixel stores the (x, y, z) values of the position of the corresponding texel instead of an (r, g, b) color value. The normal map can be generated at runtime. It can also be any material as long as it provides a relevant BitmapData. This BitmapData stores the (nx, ny, nz) values of the normal of the corresponding texel. Here is an example:

In this example both the normal and the position maps have been generated using Minko's NormalMap and PositionMap classes. This picture is actually a screenshot of a demo Flash application. I wont talk too much about this generation step for now, but its done using Pixel Bender and its lighting fast!

In order to make this data available to any pixel shader, the diffuse, normal and position maps are stored in a one-and-only scene graph node: the PixelMaterial. This node will then provide per-texel data access to its modifiers. We can now implement a real pixel shader on top of any PixelMaterial! Everything we need now is a Pixel Bender kernel that takes our 3 maps (diffuse, position and normal) in input.

The following Pixel Bender filter performs Phong shading:

void
evaluatePixel()
{
        float2 o = outCoord();
        float3 position = sampleNearest(positionMap, o);
        float3 normal = sampleNearest(normalMap, o) * 2. - ONE;
        float3 light = normalize(light - position);
 
        // ambient
        float l = ambient * intensity;
 
        // diffuse
        l += max(0., dot(light, normal)) * diffuse * intensity;
 
        dst = sampleNearest(diffuseMap, o) * float4(l, l, l, 1.);
}

As you can see its really straight forward. This is due to the Scene Graph API which makes it possible to do every computation in local space. Therefore we don't have to care about where those computations happen in the 3D scene.

Demo

Phong shading or reflection mapping are great examples of pixel shaders. The following demo uses both (the white diamond stands for the light source):

Phong shading + spheric environment mapping demo

Phong shading + spheric environment mapping scene graph

And to put it in a nutshell, a few performances considerations:

• Everything is done in real time.
• Thanks to the Scene Graph API, every computation is done in local space. It avoids matrix multiplications by the thousand and gives a significant performance boost.
• The application takes less than 20Mb of memory and weights less than 100Kb.
• There are a total of 6 pixel shaders run at each frame (only the visible faces materials are computed and there are 2 shaders per face).
• Those shaders work on 128*128 pixels diffuse/normal/position maps
• That's a total of 98,304 pixels per frame
• Each frame is "processed" in an average of 10ms. (Core2 Duo U9400 @ 1.4Ghz, Flash 10.1 RC7, Chrome 5)
• That's 9,830,400 pixels per second!
• There is a ~30% performance boost with Flash 10.1 RC7
• The number of triangles does not impact the performances
• Do not hesitate to post your own results (with your configuration of course)!