Another OSX Kinect library

in Library and Tool Development • 2 years ago

Hi,

Just wanted to share a library I've been putting together over the past few days. I'd been using Shiffman's library (kudos!), but found I needed something faster. The source is on github, with a link to a downloadable binary in the README:

https://github.com/nrocy/processing-openkinect

It runs the Kinect event handler on a separate thread so the video/depthmap feeds are pretty much full speed.

There's LED support (woo) - and access to the newish IR camera feed, working on motor/accelerometer.

Constructive feedback/contribution welcome, it's my first Processing library / JNI endeavour.

Cheers,

- Paul

Replies(31)

andreas.g..

Re: Another OSX Kinect library

2 years ago

Nice work! Thanks for sharing it…

tabbed

Re: Another OSX Kinect library

2 years ago

Hey Paul, nice work with the lib. I'm having trouble getting it to run on osx leopard with processing 1.2, i'm getting the unsatisfiedlinkerror message for libkinect.jnilib. Libs are all in the right place. Do I need to recompile the xcode source?

Cheers.

paulking1

Re: Another OSX Kinect library

2 years ago

Hiya tabbed,

It's built on Snow Leopard and tested against Processing 1.2.1 - unfortunately I've no access to a Leopard install to test.

Could you please run the following from the terminal and send over the output?

otool -L libkinect.jnilib

Here's mine:

~/Documents/Processing/libraries/kinect/library]$ otool -L libkinect.jnilib

libkinect.jnilib:

libkinect.jnilib (compatibility version 1.0.0, current version 1.0.0)

/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1038.0.0)

/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation (compatibility version 1.0.0, current version 1.0.0)

/System/Library/Frameworks/JavaVM.framework/Versions/A/JavaVM (compatibility version 1.0.0, current version 1.0.0)

/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)

/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 123.0.0)

/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.0.0)

Thanks,

- Paul

tabbed

Re: Another OSX Kinect library

2 years ago

Cool - thanks for getting back to me.

Here's the output I'm getting, looks the same?
webdesign1:library webdesign1$ otool -L libkinect.jnilib
libkinect.jnilib:
    libkinect.jnilib (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1038.0.0)
    /System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/JavaVM.framework/Versions/A/JavaVM (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 123.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.0.0)
webdesign1:library webdesign1$

paulking1

Re: Another OSX Kinect library

2 years ago

Hi tabbed,

I've added a new Leopard specific build to the github homepage, please test it and let me know if this works?

Thanks,

- Paul

tabbed

Re: Another OSX Kinect library

2 years ago

Awesome, thanks Paul. Just tested on Leopard - and all works perfect!

jeff_g

Re: Another OSX Kinect library

2 years ago

Nice!

Jesse Sco..

Re: Another OSX Kinect library

2 years ago

Paul, am I correct in assuming that your method NativeKinect.start() replaces Daniel's NativeKinect.update() ??

(( and that it should be put in setup() where his is put in draw() ?? ))

paulking1

Re: Another OSX Kinect library

2 years ago

Hi Jesse,

Yeah, you've got it. The library spawns a thread to service the kinect updates so you don't need to manually update like in Daniel's library. The start() method kicks off the updates.

Does the sample at the bottom of this page work okay for you?

https://github.com/nrocy/processing-openkinect

- Paul

Jesse Sco..

Re: Another OSX Kinect library

2 years ago

Yes, it does.

I've also gone through some examples by Henry Palonen and Greg Borenstein with the updated methods, and they are all working great (much faster than Daniels!)

mbuchetics

Re: Another OSX Kinect library

2 years ago

Hi, I was also trying to get Greg's example to run with this library. But whenever I setup an OpenGL windows (size(640,480, OPENGL);) I receive an error when grabbing the depth image in the draw method. The color image works fine.

This is the error: Invalid memory access of location 0x0 eip=0x29026c86

And it's thrown in the kinect library.

Can I maybe see your code how you updated the point cloud example? Thanks!

mbuchetics

Re: Another OSX Kinect library

2 years ago

After some more testing I think there is a general problem when using OpenGL together with this library, can anyone confirm this? Paul?

Thanks for the great work though!

paulking1

Re: Another OSX Kinect library

2 years ago

Hi,

Thanks for the update!

I'll look into it, if you get a chance please post your processing script so that I can test?

Cheers,

- Paul

mbuchetics

Re: Another OSX Kinect library

2 years ago

You can use your example:

import processing.opengl.*;
import king.kinect.*;
PImage img, depth;
void setup()
{
size(1280, 480, OPENGL);
NativeKinect.init();
NativeKinect.start();
img = createImage(640,480,RGB);
depth = createImage(640,480,RGB);
}
void draw()
{
img.pixels = NativeKinect.getVideo();
img.updatePixels();
image(img,0,0,640,480);
//depth.pixels = NativeKinect.getDepthMap();
depth.updatePixels();
image(depth,640,0,640,480);
}

If I uncomment the //depth.pixels ... line, I get the error I mentioned above. Otherwise it runs fine.

Thanks,

Matthias

paulking1

Re: Another OSX Kinect library

2 years ago

Hiya Matthias,

Thanks for that - I'm not at home at the moment so I can't test this against the device, but could you try this version of the library?

http://nrocy.github.com/files/kinect20101219.zip

This should fix the crash, I hope!

- Paul

mbuchetics

Re: Another OSX Kinect library

2 years ago

Works fine now! Thanks!

shiffman

Re: Another OSX Kinect library

2 years ago

I'm not sure how we can best consolidate all these efforts, but I updated my library to use the OpenKinect JNI wrapper by Michael Nischt. The source is on github and linked from below. Am now getting 30 FPS with only the depth or only the RGB image, but asking for both images results in about 20 FPS.

http://www.shiffman.net/2010/12/18/updated-kinect-library-for-processing/

paulking1

Re: Another OSX Kinect library

2 years ago

Cool, nice one!

I'm not sure of the best way forward either, I can't remember the frame-rate I was getting but will check when I get home.

Couple of the things you've listed on your TODO are already catered for by my library (IR image/LED). Currently I'm working mostly on the cross-platform thing - have a buggy version of it building against/working on Windows.

Please feel free to message me on Github/twitter/etc if you want to discuss anything :)

- Paul

mbuchetics

Re: Another OSX Kinect library

2 years ago

Daniel, I wanted to try out your library as well but I always get the following error when trying to run the example:

You are missing the kinectEvent() method. java.lang.NoSuchMethodException: RGBDepthTest.kinectEvent(org.openkinect.processing.Kinect)

Also, I wanted to write this as a comment on your blog but the posts are always gone after a reload. Tried a couple of times but they never showed up. Just to let you know.

Jesse Sco..

Re: Another OSX Kinect library

2 years ago

@shiffman

much faster Daniel!

Im getting approx 30fps on both the RGb and Depth images, and a general sketch fps of 46-50 (10.6.5, MBP Core i7).

@shiffman @mbuchetics

I am getting the same error, but the sketch runs.

However, I am getting an error message upon closing the applet ("RGBDepthTest quit unexpectedly while using the libKinect.jnilib plugin")

Exception Type: EXC_BAD_ACCESS (SIGBUS)

Exception Codes: KERN_PROTECTION_FAILURE at 0x00000000000000e4

Crashed Thread: 25 Java: Thread-2

Application Specific Information:

Java information:

Exception type: Bus Error (0xa) at pc=0000000093cada74

Java VM: Java HotSpot(TM) Client VM (17.1-b03-307 mixed mode macosx-x86)

I can post or pm full details if you like.

@shiffman @paulking1

merge!!!! (or at least, one of you rename your library!)

shiffman

Re: Another OSX Kinect library

2 years ago

1) comments on my site are fixed, thanks for pointing that out!

2) the kinectEvent() method error can be completely ignored. I originally imagined the library having a kinectEvent() method (much like captureEvent()), but at the moment, it's not implemented. So that's just a placeholder.

3) "RGBDepthTest quit unexpectedly while using the libKinect.jnilib plugin" -- this is a known bug. i am going to add it to the github issues list. i think i'm not properly closing /disposing the kinect device or something.

mbuchetics

Re: Another OSX Kinect library

2 years ago

@shiffman Yes, it works anyway, that was my bad.

It is interesting that both libraries produce different depth image results. Using Paul's library, points far away are black. Using Daniel's OpenKinect library, the same points are gray.

paulking1

Re: Another OSX Kinect library

2 years ago

The depth map image is actually generated by the library from the depth data :)

Those differences will be based on the different algorithms that we use, would be worth discussing those Daniel? Is there a standard?

Jesse: Indeed we should, just need to work out how best to :)

- Paul

shiffman

Re: Another OSX Kinect library

2 years ago

"RGBDepthTest quit unexpectedly while using the libKinect.jnilib plugin"

This error is now fixed if you downloaded the updated library and examples. Had to add something to stop().

void stop() {

kinect.quit();

super.stop();

}

@paulking -- I will PM you, let's start an e-mail discussion.

shiffman

Re: Another OSX Kinect library

2 years ago

Also, re: grayscale mapping, the algorithm I'm using is incredibly simple and just maps the depth data (a short, 0-2047 to a 0-255).

int brightness = Math.round((1 - (depth / 2047f)) * 255f);

I don't see pure black unless there's something the kinect can't see, i.e. a shadow or objects too close?. But I'm in a small room.

ventoline

Re: Another OSX Kinect library

2 years ago

I got it to work without problems, Thanks for the Lib!
I am looking forward to save the 3D data, has anyone implemented this?
What format shall i save it to? ( My want to create an animation in processing, C4D, Ae...)
If you know any link to point me in the direction please go ahead, i cant find much information (quite surprisingly).

Thanks

Jesse Sco..

Re: Another OSX Kinect library

2 years ago

Can't remember where I found this anymore, but here is an example by Henry Palonen to save .obj's ... not that in this example I had already switched the syntax to use Paul's library instead of Daniel's...

import peasy.*;
import king.kinect.*;
import superCAD.*;
boolean record = false;
/*
Simple Kinect point-cloud demo v. 0.2
Henry Palonen <h@yty.net>
Using Daniel Shiffman's great processing-library for Kinect:
http://www.google.com/url?sa=D&q=http://www.shiffman.net/2010/11/14/kinect-and-processing/&usg=AFQjCNH8kZWDMhFueeNBn5x97XoDR3v9oQ
Based on Kyle McDonalds Structure Light scanner:
http://www.openprocessing.org/visuals/?visualID=1014
Using also SuperCAD for outputting the .obj - files: http://labelle.spacekit.ca/supercad/
History
-------
17.11.2010 - 0.1 - First version, simple point-cloud working
18.11.2010 - 0.2 - Output to .obj for importing to Blender, gray-color for distance and small lines as output
*/
float zscale = 1.5; // 1-5
float zskew = 5; // 5-10
int inputWidth = 640;
int inputHeight = 480;
PeasyCam cam;
float[][] gray = new float[inputHeight][inputWidth];
PImage depth;
static final int gray(color value) {
return max((value >> 16) & 0xff, (value >> 8 ) & 0xff, value & 0xff);
}
void setup() {
size(inputWidth, inputHeight, P3D);
cam = new PeasyCam(this, width);
NativeKinect.init();
NativeKinect.start();
depth = createImage(640,480,RGB);
stroke(255);
}
void draw () {
background(0);
if (record == true) {
beginRaw("superCAD.ObjFile", "kinec_out.obj"); // Start recording to the file
}
depth.pixels = NativeKinect.getDepthMap();
depth.updatePixels();
//NativeKinect.update();
for (int y = 0; y < inputHeight; y++) {
for (int x = 0; x < inputWidth; x++) {
// FIXME: this loses Z-resolution about tenfold ...
// -> should grab the real distance instead...
color argb = depth.pixels[y*width+x];
gray[y][x] = gray(argb);
}
}
// Kyle McDonald's original source used here
translate(-inputWidth / 2, -inputHeight / 2);
int step = 2;
for (int y = step; y < inputHeight; y += step) {
float planephase = 0.5 - (y - (inputHeight / 2)) / zskew;
//println(gray);
for (int x = step; x < inputWidth; x += step)
{
stroke(gray[y][x]);
point(x, y, (gray[y][x] - planephase) * zscale);
//line(x, y, (gray[y][x] - planephase) * zscale, x+1, y, (gray[y][x] - planephase) * zscale);
}
}
if (record == true) {
endRaw();
record = false; // Stop recording to the file
}
}
void keyPressed() {
if (key == 'R' || key == 'r') { // Press R to save the file
record = true;
}
}

Hope that helps...

Jesse Sco..

Re: Another OSX Kinect library

2 years ago

??? It's a Jar, I've had no issues with it... though I did move back to Shiffman's library, as it started to get faster (and had better example's to get me started on different applications)... though now I'm using Max's SimpleOpenNI...

ventoline

Re: Another OSX Kinect library

2 years ago

I get it to run, thnks for your help and patience, this is my first processing project.

i was wondering if it was possible to "record" the 3d, like output an object every 100ms or register the animation?
what library as well would allow me to record the data as video too? does processing have a built one for this? or peasycam?

shiffman

Re: Another OSX Kinect library

1 year ago

@smebs. i think the grayscale values in mine are actually a bit off. this is on my list of things to fix when i have time to return to working on the library and updating it.

paulking1

Re: Another OSX Kinect library

1 year ago

@smebs: yeah, the depth image isn't output from the kinect. you get a feed of depth information that we convert over to something visual :)

As such there's not really a right nor wrong way of doing it, as you're compressing a large range of depth values (0..2048, I think) to 256 grayscales while trying to retain best fidelity.

@shiffman: my algo is here if it helps, but it's not perfect either!
https://github.com/nrocy/processing-openkinect/blob/master/src/native/Kinect.mm (line 143)

- Paul

Top Reply

Response title

Attachments

New to this Portal?