Loading...
Logo
Processing Forum

Hey everyone!

My latest tutorial for CreativeApplications.net has just gone live. It's about using a Kinect to interact with geometry on your screen. Several Processing libraries are used (SimpleOpenNI, v3ga, Toxiclibs & PBox2D) to achieve this effect. It's a hands on tutorial so the main content is made up of three fully commented code examples. These exampes will also show you how you can turn a silhouette blob into a polygon, which is useful for many things, even besides 2D physics interaction.

Blog post & source code

Vimeo video

Check it out!

Replies(37)

I just wanted to start working on this a little bit. This gives me now a real kick-start !
Thnx a lot for sharing.

Dear Amnon,

Sorry for contacting you this way, You have helped me before and i would consider you one of the kings of processing, no messing. You always share and help out and thats what open source is all about.So I would like to ask your help one more time. Best I just share the link 


thanks. 
brendan flynn
pretty cool tutorial! thanks for sharing!
Some amazing work there amnon.owed, I'm really impressed. Really thank you for sharing, learning some great ideas on using kinect with animation classes.
Amazing! Thanks for sharing, I'll have to get a kinect now...
Thanks guys, I appreciate the positive feedback.
Thank you SO MUCH for this!  I can finally move forward after a month of messing around with Processing.

One thing I've run into on the Silhouette tutorial is that when I ran it in Processing 2.0 I got an "Out of memory" error.  I tried raising memory in system (I'm using windows 7 on a 64bit laptop 2.26GHz 4GB RAM, GeForce 9800M GS) but that didn't help.  When I ran it in Processing 1.5 it worked just fine.  Is this a problem with my computer processor power or Processing 2.0?

On that note, would you be able to recommend what I would look for if I were to purchase a new laptop to use something like your examples with a projector and dancers on a stage?

Thanks again.
Good to hear my tutorial was useful.

The tutorial was written in/for Processing 1.5.1. I've tested it in 2.0b3 (and it worked), but when enhancing the sketch I've run into similar issues as well, so I am still mainly working in 1.5.1. I assume these are some remaining issues with the beta's and it's not so much related to your computer specs.

In general you always want as good a laptop as your budget allows. These examples should generally run fine on any modern computer. Depending on the available hardware and the resulting fps, you can always (and probably will usually have to) further optimize the code. For example the flow example can be much further optimized. Also it usually depends on the amount of shapes, lines etc. on your screen what your fps will be. Then there is the option of moving things into the GPU / shaders. The thing with the kinect is that it physically limits the framerate to 30fps anyway, so usually your sketch will be faster than that. My guess is that the relaying of imagery via HDMI or to a projector, does not take a lot of processing power, but I have only experience with the former.

Beautiful! Did it hurt when all those shapes fell on your head?
Thanks!

Luckily I was wearing polygon-resistant clothing that day, otherwise it would have been a bloody affair.
Awesome tutorial! :D Was so glad to stumble across this! Thanks!

I have a question..
Has anyone had any issues combining SimpleOpenNI's enableDepth with enableScene ? I'm using depthMap somewhere else in my code but that seems to mess with the Polygon as thats using sceneImage! For some reason this:

void setup() {
 
  size(1280, 720, OPENGL);
  
  context = new SimpleOpenNI(this);
  context.enableDepth();
 
  if (!context.enableScene()) { 
    println("Kinect not connected!"); 
    exit();
  } 
  else {
    smooth();
       context.setMirror(true);
       context.enableScene();
       context.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
....
causes the polygon to jump all over the place! Like this:


But when I simply move enableDepth() below enableScene() it works just fine!

However, for some strange reason the function context.convertRealWorldToProjective() returns zeros if I don't call it right after
             context = new SimpleOpenNI(this);

Has anyone else stumbled across this issue? :/ Or know of a way to draw this polygon shape with using something other than SceneImage? Maybe using DepthImage?

Cheers :)
Indeed, the order in which you enable stuff matters with the SimpleOpenNI library. The reason your polygon is distorted is because sceneimage returns a quasi-perfect silhouette of a person, while depthimage tends to include all kinds of stuff, also from the background. For an example using depthimage to create a blob, see this thread.
Thanks for the super quick response! That was exactly what I needed! :D
Hi again.. Has anyone tried to create a polygon as in  amnon.owed's  CreativeApplications tutorial, from the depthImage? Although it looks a lot better than what I had before, it still jumps around quite a bit. I tried to limit the blobdetection to only part of the Image but the polygon still seems to jump around. Any idea how to solve this? :S
As said above, one option returns a silhouette of a person, the other a (filtered) depth map. So the former solution is preferred because it will probably always give better results (not to mention the fact it does so, out-of-the-box).
Your  context.convertRealWorldToProjective()  problem is probably this bug
Is fixed in 0.27 as far as I know.
Did you installed the newest version of SimpleOpenNI ?
Wow thanks for the quick response guys! :D Unfortunately installing the latest version didn't solve it but thanks anyways. :)
Version 0.27 didn't fix it for me too.

The workaround I'm using is enabling only the depth map (enableDepth()) and not the scene image. Then you can recreate the scene image using the user map ( getUsersPixels(). 
What is the hardware needed to use Kinect? Only the Kinect? I don't need an XBox, right? I just found it selling bundled with "Kinect adventures" game, are they sold alone? Do I need to buy any thing else? Like cables or adapters? How does it connect with my computer? (a macbook pro)?
Thanks
You only need a kinect, not an xbox. The kinect 360 package which comes with a game is the stand-alone product. You can just throw the game in the trash. Everything you need is in the package namely the kinect, a cable that connects the kinect via usb to a pc and a cable for power.
Thanks Amnon. Just to be sure, it will work on a mac won't it?
Yes. Both the kinect and this tutorial will work on a mac.
Thanks a lot for your tutorials amnon!

I found that opencv's blobs gives you an ordered, polygon friendly, list of points for each blob that works perfectly for the same purpose. Is there any performance reason why you didn't use it?

Thanks
The reason is simply that I didn't know. Which opencv library are you using?
The old one at http://ubaa.net/shared/processing/opencv/ which used OpenCV 1.0, I haven' t tried yet the new one, JavacvPro, but it is next on the to do list along with diewald_CV-kit.
When I run the Kinect Physics sample, the shapes stop falling a little after the user is detected. If I leave the FOV and user is lost/exit, they start falling again (I come back, it stops). I'm not sure if this happened when I tried before, maybe it's a version conflict? I'm trying with the 2.0b6 version of Processing.

I'm looking into this, but any idea is welcome.
The original example adds shapes based on the frameRate. Apparantly something is causing the frameRate to drop significantly when you're on-screen.
Thanks for the tutorial. Using the Kinect with Processing is something I am looking to get into.

FYI, I've posted a link to the tutorial over in the Linkedin Processing Programming Language Group at http://www.linkedin.com/groups?home=&gid=4549593

Best Regards, Jim
Indeed the code is optimized for the situations as shown in the video, that someone is fully in front of it. I've walked back as well, about 4 meters or so. And for me this worked without problems. It's a less detailed silhouette, but I guess this is logical because the person is just smaller/less detailed when far away. Of course those settings in the code were tweaked to precisely prevent the issues you are seeing. So the fact that you are seeing them because you changed the settings. Well... There are also other choices in this code, for example with regard to the polygon detail in the lower half. Those are just specific choices for this. You want something different, so you can see this code as a starting point, knowing that it wasn't written for your specific goals, so changes will be required.

Anyway, here are some suggestions that you could try.

Try increasing the resolution. Currently the blob detection (and thus the polygon creation after that) is running on a third of the resolution of the kinect input. So it goes from the kinect's 640x480 to 213x160 which is not a lot of detail. When you move backwards there won't be many pixels (aka detail) left, that the polygon creation code can salvage. Try to run at full resolution by changing this line...
Copy code
  1. blobs=createImage(kinectWidth/3,kinectHeight/3,RGB);
Try pre-filtering the image more. I used the kinect image because it could return a very clear silhouette, but I'm not really seeing this anymore in your image. Seems like there are many other blob in your image?

Try runninng the code on an image segment aka the image section where the person is standing aka the person blob. Blob detection can give you this rectangle.

I doubt tweaking a few specific settings in my code example will improve your results. You'll have to think about your specific situation and look+test at a more fundamental level which code you would need.
You could check out the tip given above by kosowski. Apparantly opencv gives a polygon-correct list of points for a blob. That sounds like a very good alternative that will shorten the code and improve the quality of the outcome. Haven't tried it myself though.




why am I getting such an OUTPUT for the code ??
please help 
i even ran the code on a stable version ?
what has gone wrong ?
As said on vimeo, the most likely cause is that your are using a more recent beta version of Processing. From your screenshot this seems to be confirmed as I'm seeing 2.0b7. Instead try it with Processing 1.5.1.
The output above is a snapshot after i ran the code in Processing 1.5.1 

Please help
what part is gone wrong ?
All right. Well in that case I don't really know what's going wrong.

I tried running the second sketch (the one with the pixels) but I keep getting this error in the particle class

expecting 'EOF' found '2'

I can get the third sketch (the one with the falling polygons) working just fine

I copied the code exactly from the creative applications website so im confused as to what could be the issue

nm, i solved the prob
Why sketches are stop working?

Windows 7 Ultimate 64 bit
processing-1.5.1
SimpleOpenNI Version 0.27
xbox kinect
drivers downloaded from ZIGFU "http://zigfu.com/en/downloads/browserplugin/" - ZigJSOpenNI_v1.0.1.exe

After 10 seconds for example 2 and example 3, I get this message: