Jean-Marc Le Roux Web, RIAs and chocolate spaghettis…

Tutorial: Add pixel-perfect 3D mouse interactivity

In this tutorial we're going to see how you can add pixel-perfect 3D mouse interactivity. I've already introduced a technique called "ray casting" in another article. But it works only with very basic static shapes. And sometimes, testing very complex shapes can be very painful performance wise. It's even more expensive when you want it to be very precise.

In this article, we will see a technique called "pixel picking". This technique uses hardware acceleration to provide pixel perfect mouse interactivity. It works very well for both static and animated models. The concept is very simple: we render the scene with one color per mesh. Then, we just have to get the pixel under the mouse cursor to know what mesh is "interactive". Of course, things are much more complicated in the real life: this kind of stunts are pretty hard to push properly in a general purpose rendering pipeline.

But Minko provides everything required out of the box! Even better, the minko-picking extension features a simple controller - the PickingController - that provides all the mouse signals we might need! This tutorial will explain how to setup the PickingController and listen for the mouse signals.

Pixel picking test application (sources)

Create and setup the PickingController

The first step is to instanciate a new PickingController:

var picking : PickingController = new PickingController();

The constructor takes only one argument: the "picking rate" of the controller. This value will determine how many times per second the controller will try to execute the picking pass and the relevant mouse signals. The lower the picking rate, the better the performances. A picking rate of 30 should be more than enough for 99% of the applications. You can also set that value at any time using the PickingController.pickingRate property:

picking.pickingRate = stage.frameRate / 2.;

Setting the picking rate to the half of the frame rate will work just fine for most applications and should be completely painless performance wise. By default, the picking rate is fixed to 15.

Set the mouse events source

The job of the PickingController is to listen for the mouse events on one (or more) specific dispatcher(s) and re-dispatch them as mouse signals. The difference between the original events and the signals executed by the PickingController is that the signals are aware of the 3D scene. To setup the dispatcher to listen, you just have to call the PickingController.bindDefaultInputs() and provide the IDispatcher object to listen:

picking.bindDefaultInputs(viewport);

Setup the PickingController on the 3D scene

In most cases, you don't want the whole 3D scene to be mouse interactive. Sometimes it's just a Mesh or a Group. The PickingController can be added to any Mesh/Group so it's easy to target precisely what is interactive and what is not. The basic use case is to add mouse interactivity on a single Mesh:

mesh.addController(picking);

BUt you also might want to listen for the mouse signals trigerred by a whole sub-scene instead of a single mesh. For example, some skinned 3D assets have multiple meshes animated by a single skeleton. To do this, we can add the PickingController on Group:

group.addController(picking);

In the code snippet above, the PickingController will execute mouse signals for all the Mesh descendants of the target group. You don't have to worry about the descendants of the groups targeted by a PickingController: it will listen for the Group.descendantsAdded and Group.descendantsRemoved to start/stop tracking any descendant Mesh added to this part of the scene.

Thus, if your whole 3D scene is interactive, you can add the PickingController directly on the Scene node:

scene.addController(picking);

Listen for the mouse signals

To catch 3D mouse events, you just have to add callback(s) to any of the PickingController.mouse* signals. The available signals are:

• mouseClick, mouseDown, mouseUp: executed when the left button is clicked, down or up
• mouseRightClick, mouseRightDown, mouseRightUp: executed when the right button is clicked, down or up
• mouseMiddleClick, mouseMiddleDown, mouseMiddleUp: executed when the right button is clicked, down or up
• mouseDoubleClick: executed when the user makes a double click
• mouseMove: executed when the mouse moves
• mouseWheel: executed when the mouse wheel turns
• mouseRollOver, mouseRollOut: executed when the mouse roll over/out a mesh

The following code sample will catch the left and the right click signals:

picking.mouseClick.add(
	function(ctrl : PickingController, mesh : Mesh, mouseX : Number, mouseY : Number) : void
	{
		trace('click: ' + (mesh ? mesh.name : null));
	}
);
picking.mouseRightClick.add(
	function(ctrl : PickingController, mesh : Mesh, mouseX : Number, mouseY : Number) : void
	{
		trace('right click: ' + (mesh ? mesh.name : null));
	}
);

It would be too difficult to use the PickingController if the mouse signals where triggered only when an actual 3D object is under the cursor. For example, it would be pretty hard to select/unselect objects without listening to some actual 2D mouse events. The code would then quickly become very complicated to mix both 2D mouse events and 3D mouse signals.

Therefore, the mouse signals are triggered whenever the corresponding mouse event is dispatched (and when the picking rate allows it of course). As a direct consequence, the mesh : Mesh argument is null when there is no actual interactive 3D object under the mouse cursor.

Conclusion

You can find the complete source code of the picking example demo in the minko-examples repository on github. If you have questions/suggestions regarding this comment, you can ask them in the comments or on Aerys Answers, the official support forum for Minko.