On Wednesday, Microsoft released additional information about their upcoming SDK for the Kinect. However, if you don't want to wait for that release, there are some great alternatives available already.

To better understand the potential Kinect holds for retail and other installation work at Odopod, I've been exploring different ways to integrate Kinect into Adobe Air applications. We're using Air because it allows us to quickly build prototypes and explore this exciting new technology.

From the moment it shipped, the Kinect has been popular among hackers and digital artists wanting to use the remarkably inexpensive 3d camera within their own creations. Thanks in part to a cash bounty offered by Adafruit Industries, drivers for the Kinect were publicly available six days after its launch.

Since then, several open-source projects have sprung up and the core capabilities of Kinect are now available in a variety of programming languages on Windows, Mac and Linux systems. A Google search for Kinect hacks will give you a sense of the variety of things being done with these projects.

A quick note about the technology

Before I go into the different options I have found for connecting Kinect and Flash, it is good to understand what the Kinect hardware can do. The open-source Kinect drivers provide access to the device's motor, LED, standard camera feed and depth camera feed. The depth feed uses gradients to encode the depth of thousands of points within the scene and is what most distinguishes the Kinect from standard cameras.

Kinect's depth camera feed (left) and standard camera feed (right)

I have not found any support for the Kinect's array of microphones within the drivers or sample code I have looked at. Furthermore, neither the Kinect hardware nor its drivers provide the facial recognition, motion tracking or gesture recognition that it is famous for. These features come from additional software that use the depth video feed to get a better picture (literally) of a scene and recognize objects within it.

Kinect and Flash

Now, back to the task at hand. I've found three promising options for integrating the Kinect into Air applications: OpenKinect, TuioKinect, and OpenNI/PrimeSense NITE.

OpenKinect

Of the three projects above, I've spent the least amount of time with OpenKinect. The project is currently focused on making components of the hardware (cameras, LED, and motor) available to a very wide variety of programming languages. In addition to basic access to the hardware, the ActionScript 3 wrapper for OpenKinect provides some blob detection within the scene.

OpenKinect is a good place to start if you want to work with the raw depth information or if you want to track objects within the scene as they move around. The depth camera provides more reliable object and motion detection than a standard web camera so using the OpenKinect for projects requiring computer sight will provide more flexibility when it comes to scene lighting and background noise.

An example project using the OpenKinect AS3 wrapper

TuioKinect

TuioKinect builds upon the OpenKinect project. It includes a c++ application that processes the camera feeds, identifies objects within a specific depth range and converts those objects into Tuio events (Tuio is a standard protocol developed as a common API for multitouch surfaces). A proxy application is available to make the events generated by TuioKinect available to Flash applications and there is an ActionScript 3 library for managing the events within your Flash application.

While the Tuio spec includes support for 3d tracking, most servers and clients (including the current version of TuioKinect) focus on 2d implementations with events similar to those you get on a touch device like the iPhone: touchstart, touchmove and touchend.

If you want to do standard blob detection and tracking, TuioKinect will probably perform better than if you are processing the raw video feeds within Flash; however, it provides less flexibility in terms of what your application can see. The TuioKinect roadmap includes the ability to distinguish between open, closed and pointing hands, which would be great addition to the standard touch events and a unique offering among the options I have looked at.

An example project using TuioKinect and Tuio AS3. Full depth camera (upper left), objects detected within the configured depth range (lower left) and a Flash application's rendering of touch points (right). The camera feeds on the left are part of the TuioKinect application window sitting behind the Flash application.

OpenNI and PrimeSense NITE

The OpenNI project and and its NITE module focus more on natural interactions than the Kinect sensor hardware itself. With OpenNI, developers have access the video and depth information, but the project is really a framework for recognizing people in the scene and gestures they are making. In short, OpenNI and NITE provide more of the magic that the Kinect is known for.

The framework is extensible, allowing c++ developers to create custom object and gesture detection but the core functionality is quite sophisticated by itself and includes:

- Human body recognition within the scene
- Full human skeleton tracking (once a specific pose is performed by user)
- Hand tracking and gesture detection (once a wave or push gesture is performed by user). These include:

Skeleton tracking example using OpenNI/NITE. Note the pose required at very beginning.

as3OpenNI

As a c++ project, OpenNI is not well suited to the the rapid prototypes we want to use to explore the potential for Kinect in our work. Fortunately, there is as3openNI to bridge the OpenNI/NITE and Flash worlds. This project is currently in aplha release but already offers a great deal of functionality and has a very low barrier of entry for Flash developers.

New features are added to the project daily, so I encourage you to check the as3OpenNI site for more details. At the time of this writing it includes support for all the OpenNI features I list above as well as support for gestures similar to those used in the FAAST project (a real-world build of the Gmail's April Fool's prank that also uses OpenNI).

Are people ready for focus events?

Early prototypes here indicate that the more robust functionality offered by OpenNI require a bit of effort from users. Specific focus gestures and poses create a barrier to interaction and in general the gestures require a good deal of patience and a fairly steady hand. Pulling an audience in with more passive interactions that can be tracked without focus events will likely lead to more engagement. Either way, as we learn more we'll share our prototypes.

For now, you can download the code for a carousel that uses both the trackpad and gesture features powered by as3OpenNI. The examples in the as3OpenNI repository use Flash Builder and this example shows how to create a project in Flash Professional for those preferring that environment.

Screen capture of the carousel example

Lastly, if you want to dig deeper into this yourself, here's more on getting up and running with OpenNI and NITE on a Mac.