Kinect and OpenCV Blob Tracking

elamda

Re: Kinect and OpenCV Blob Tracking

2 years ago

I do this exact same thing and I found out how to do it by following Shiffman's examples included with his Processing library(hint, hint):

PImage display;

// Get the raw depth as array of integers

   depth = kinect.getRawDepth();

   // Being overly cautious here

   if (depth == null) {

   return;

   }

   display.loadPixels();

   for(int x = 0; x < kw; x++) {



   for(int y = 0; y < kh; y++) {



   // mirroring image

   int offset = kw-x-1+y*kw;



   // Raw depth

   int rawDepth = depth[offset];

   int pix = x + y * display.width;

   if (rawDepth < threshold) {



   // A red color instead

   display.pixels[pix] = color(255,0,0);

   }

   else {

   display.pixels[pix] = color(0,0,0);

   }

   }



   }

   display.updatePixels();

This will get you an image derived from the Kinect depth map that you can feed into OpenCv. It has only two colors so as to make tracking with OpenCV a trivial exercise. Once the image is updated, you can track it with something similar to the following:

void calculateBlobs() {

   OpenCV opencv;

   opencv.copy(display);

   opencv.threshold( CVThreshold );

   currentBlobs = opencv.blobs( minCVDetectArea, maxCVDetectArea, maxNumBlobs , false, OpenCV.MAX_VERTICES*4 );



   updateBlobVectors();



  }

I've cut-n-pasted this code from a current project, but it's fairly self explanatory and derived from the OpenCV examples included with the examples from the OpenCV Processing distribution. In short, turn the Kinect depth map into a simple two tone image, use OpenCV to track blobs, do something interesting with said blobs.

shiffman

Re: Kinect and OpenCV Blob Tracking

2 years ago

Great adaptation of the code! One thing to be aware of is openCV can be a bit slow so you may want to sample down the Kinect 640x480 image to half size to improve speed. Let me know how else I can help!

elamda

Re: Kinect and OpenCV Blob Tracking

2 years ago

Hi Daniel! Thanks for the Kinect wrapper. It's a lot of fun.

I am running into a problem of sorts. I read your PVector tutorial and implemented a ipad/iphone-ish rotation motion for objects based on the velocity of the blobs, like here:

http://www.youtube.com/watch?v=Suqlf7kibVw

Rotation is simply based on the velocity of a blob, so the faster you move your hand the faster it rotates, then settles. The code is pretty much adapted from your PVector tutorial code:

void updateBlobVectors() {



   // One blob means lets move/rotate 2d/3d object.

   if( currentBlobs.length == 1 ) {



   diffX = currentBlobs[0].centroid.x - previousBlobs[0].centroid.x;

   diffY = currentBlobs[0].centroid.y - previousBlobs[0].centroid.y ;

   objectVelocityVector.add( new PVector(diffX, diffY) );

   }

   objectLocationVector.add( objectVelocityVector );

   // Let's degrade the object velocity so it

   // doesn't fly around all crazy like.

   objectVelocityVector.mult(.9);



   previousBlobs = currentBlobs;

}

// called from the draw() method.

// rotates the model.

void drawObjectOfAttention() {

translate( width/2, height/2,0);

rotateX( radians( objectLocationVector.y ) * .05 );

rotateY( radians( objectLocationVector.x ) * .05 );

objectOfAttention.draw();

}

I'd like to do the same thing for scaling, but I can't quite get a good algorithm. Currently, I've settled on simply taking the magnitude of the distance between the blobs that represent my hands. The problem is that the position of the blobs are absolute so scaling tends to jump around a lot and it doesn't ease in and out like rotation does because I'm not accounting for velocity at all.

Thanks for the OpenCV advice. I'll give it a shot. I'm currently geting about 15 fps on my macbook pro for a 20,000 polygon model. But that may be due to other things rather than OpenCV.

Cheers,

- Elam

shiffman

Re: Re: Kinect and OpenCV Blob Tracking

2 years ago

Cool project!

A quick solution would be to use an easing equation, Processing's lerp() function will take care of this for you:

http://www.processing.org/reference/lerp_.html

(It's like taking a running average).

If you want to get fancy, take a look at the Penner easing library.

http://jesusgollonet.com/processing/pennerEasing/

cinnamon_..

Re: Re: Kinect and OpenCV Blob Tracking

2 years ago

Hi All,

I realize this is already a dated topic...but I am struggling with it! I am doing an exam project in Processing, and would like to be able to capture a silhouette of a person. So I was advised to use a combo of Kinect and openCV. When I found this post, I was overjoyed.

I attempted to use the code, but I keep getting the line depth = kinect.getRawDepth(); (as in the code at the top) highlighted with an error "cannot convert from int[] to boolean. So it is referring to the depth ...but how do I fix this error this?

I am also unsure of the following int offset = kw-x-1+y*kw; what does the variable kw refer to? I would need to define at, but not sure as what .

Help would really, really be appreciated!

Thanks

cinnamon_..

Re: Re: Kinect and OpenCV Blob Tracking

2 years ago

Hi!

I started using the AveragePointTracker kinect code, which is brilliant. When I call the openCV code to copy the kinect image into blob detection mode, the image freezes and I get a NullPointerExeception error. Currently it should on key press t start the blob detection - but like I said, it just freezes.

Here is my code, as derived from the D. Shiffman kinect code, and then OpenCV Blob Detection code:

/*

// Daniel Shiffman
// Tracking the average location beyond a given depth threshold
// Thanks to Dan O'Sullivan
// http://www.shiffman.net
// https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing
import org.openkinect.*;
import org.openkinect.processing.*;
import hypermedia.video.*;
import processing.video.*;
import java.awt.*;

OpenCV opencv;

// Showing how we can farm all the kinect stuff out to a separate class
KinectTracker tracker;
// Kinect Library object
Kinect kinect;

void setup() {
size(1280, 480);
kinect = new Kinect(this);
tracker = new KinectTracker();
}

void draw() {
background(255);

// Run the tracking analysis
tracker.track();
// Show the image
tracker.display();

// Let's draw the raw location
// PVector v1 = tracker.getPos();
// fill(50,100,250,200);
// noStroke();
// ellipse(v1.x,v1.y,20,20);
//
// // Let's draw the "lerped" location
// PVector v2 = tracker.getLerpedPos();
// fill(100,250,50,200);
// noStroke();
// ellipse(v2.x,v2.y,20,20);

// Display some info
int t = tracker.getThreshold();
fill(0);
text("threshold: " + t + "    " + "framerate: " + (int)frameRate + "    " + "UP increase threshold, DOWN decrease threshold", 10, 500);
}

void keyPressed() {

if (key == 't') {
    tracker.calculateBlobs();
}
}

void stop() {
tracker.quit();
super.stop();
}

*/

And the class...

/*

class KinectTracker {

// Size of kinect image
int kw = 640;
int kh = 480;
int threshold = 745;

// Raw location
PVector loc;

// Interpolated location
PVector lerpedLoc;

// Depth data
int[] depth;

PImage display;

KinectTracker() {
    kinect.start();
    kinect.enableDepth(true);

    // We could skip processing the grayscale image for efficiency
    // but this example is just demonstrating everything
    kinect.processDepthImage(true);

    display = createImage(kw, kh, PConstants.RGB);

    loc = new PVector(0, 0);
    lerpedLoc = new PVector(0, 0);
}

void track() {

    // Get the raw depth as array of integers
    depth = kinect.getRawDepth();

    // Being overly cautious here
    if (depth == null) return;

    float sumX = 0;
    float sumY = 0;
    float count = 0;

    for (int x = 0; x < kw; x++) {
      for (int y = 0; y < kh; y++) {
        // Mirroring the image
        int offset = kw-x-1+y*kw;
        // Grabbing the raw depth
        int rawDepth = depth[offset];

        // Testing against threshold
        if (rawDepth < threshold) {
          sumX += x;
          sumY += y;
          count++;
        }
      }
    }
    // As long as we found something
    if (count != 0) {
      loc = new PVector(sumX/count, sumY/count);
    }

    // Interpolating the location, doing it arbitrarily for now
    lerpedLoc.x = PApplet.lerp(lerpedLoc.x, loc.x, 0.3f);
    lerpedLoc.y = PApplet.lerp(lerpedLoc.y, loc.y, 0.3f);
}

PVector getLerpedPos() {
    return lerpedLoc;
}

PVector getPos() {
    return loc;
}

void display() {
    PImage img = kinect.getDepthImage();

    // Being overly cautious here
    if (depth == null || img == null) return;

    // Going to rewrite the depth image to show which pixels are in threshold
    // A lot of this is redundant, but this is just for demonstration purposes
    display.loadPixels();
    for (int x = 0; x < kw; x++) {
      for (int y = 0; y < kh; y++) {
        // mirroring image
        int offset = kw-x-1+y*kw;
        // Raw depth
        int rawDepth = depth[offset];

        int pix = x+y*display.width;
        if (rawDepth < threshold) {
          // A red color instead
          display.pixels[pix] = color(150, 50, 50);
        }
        else {
          display.pixels[pix] = img.pixels[offset];
        }
      }
    }
    display.updatePixels();

    // Draw the image
    image(display, 0, 0);
}

void calculateBlobs() {
    println(display);
    println("test");

    opencv.copy(display);

    image( opencv.image(), 640, 0 );
    opencv.threshold( 80 );
    Blob[] blobs = opencv.blobs( 10, width*height/2, 100, true, OpenCV.MAX_VERTICES*4 );

    // draw blob results
    //    for( int i=0; i<blobs.length; i++ ) {
    //     beginShape();
    //       for( int j=0; j<blobs[i].points.length; j++ ) {
    //          vertex( blobs[i].points[j].x, blobs[i].points[j].y );
    //       }
    //      endShape(CLOSE);
    //   }
}

void quit() {
    kinect.quit();
}

int getThreshold() {
    return threshold;
}

void setThreshold(int t) {
    threshold = t;
}
}

*/

I am thinking it is probably something very stupid, but that I am just not experienced enough to see!

Any advice?

Thanks!!!!

cinnamon_..

Re: Re: Kinect and OpenCV Blob Tracking

2 years ago

Hi!

I started using the AveragePointTracker kinect code, which is brilliant. When I call the openCV code to copy the kinect image into blob detection mode, the image freezes and I get a NullPointerExeception error. Currently it should on key press t start the blob detection - but like I said, it just freezes.

Here is my code, as derived from the D. Shiffman kinect code, and then OpenCV Blob Detection code:

/*

// Daniel Shiffman
// Tracking the average location beyond a given depth threshold
// Thanks to Dan O'Sullivan
// http://www.shiffman.net
// https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing
import org.openkinect.*;
import org.openkinect.processing.*;
import hypermedia.video.*;
import processing.video.*;
import java.awt.*;

OpenCV opencv;

// Showing how we can farm all the kinect stuff out to a separate class
KinectTracker tracker;
// Kinect Library object
Kinect kinect;

void setup() {
size(1280, 480);
kinect = new Kinect(this);
tracker = new KinectTracker();
}

void draw() {
background(255);

// Run the tracking analysis
tracker.track();
// Show the image
tracker.display();

// Let's draw the raw location
// PVector v1 = tracker.getPos();
// fill(50,100,250,200);
// noStroke();
// ellipse(v1.x,v1.y,20,20);
//
// // Let's draw the "lerped" location
// PVector v2 = tracker.getLerpedPos();
// fill(100,250,50,200);
// noStroke();
// ellipse(v2.x,v2.y,20,20);

// Display some info
int t = tracker.getThreshold();
fill(0);
text("threshold: " + t + "    " + "framerate: " + (int)frameRate + "    " + "UP increase threshold, DOWN decrease threshold", 10, 500);
}

void keyPressed() {

if (key == 't') {
    tracker.calculateBlobs();
}
}

void stop() {
tracker.quit();
super.stop();
}

*/

And the class...

/*

class KinectTracker {

// Size of kinect image
int kw = 640;
int kh = 480;
int threshold = 745;

// Raw location
PVector loc;

// Interpolated location
PVector lerpedLoc;

// Depth data
int[] depth;

PImage display;

KinectTracker() {
    kinect.start();
    kinect.enableDepth(true);

    // We could skip processing the grayscale image for efficiency
    // but this example is just demonstrating everything
    kinect.processDepthImage(true);

    display = createImage(kw, kh, PConstants.RGB);

    loc = new PVector(0, 0);
    lerpedLoc = new PVector(0, 0);
}

void track() {

    // Get the raw depth as array of integers
    depth = kinect.getRawDepth();

    // Being overly cautious here
    if (depth == null) return;

    float sumX = 0;
    float sumY = 0;
    float count = 0;

    for (int x = 0; x < kw; x++) {
      for (int y = 0; y < kh; y++) {
        // Mirroring the image
        int offset = kw-x-1+y*kw;
        // Grabbing the raw depth
        int rawDepth = depth[offset];

        // Testing against threshold
        if (rawDepth < threshold) {
          sumX += x;
          sumY += y;
          count++;
        }
      }
    }
    // As long as we found something
    if (count != 0) {
      loc = new PVector(sumX/count, sumY/count);
    }

    // Interpolating the location, doing it arbitrarily for now
    lerpedLoc.x = PApplet.lerp(lerpedLoc.x, loc.x, 0.3f);
    lerpedLoc.y = PApplet.lerp(lerpedLoc.y, loc.y, 0.3f);
}

PVector getLerpedPos() {
    return lerpedLoc;
}

PVector getPos() {
    return loc;
}

void display() {
    PImage img = kinect.getDepthImage();

    // Being overly cautious here
    if (depth == null || img == null) return;

    // Going to rewrite the depth image to show which pixels are in threshold
    // A lot of this is redundant, but this is just for demonstration purposes
    display.loadPixels();
    for (int x = 0; x < kw; x++) {
      for (int y = 0; y < kh; y++) {
        // mirroring image
        int offset = kw-x-1+y*kw;
        // Raw depth
        int rawDepth = depth[offset];

        int pix = x+y*display.width;
        if (rawDepth < threshold) {
          // A red color instead
          display.pixels[pix] = color(150, 50, 50);
        }
        else {
          display.pixels[pix] = img.pixels[offset];
        }
      }
    }
    display.updatePixels();

    // Draw the image
    image(display, 0, 0);
}

void calculateBlobs() {
    println(display);
    println("test");

    opencv.copy(display);

    image( opencv.image(), 640, 0 );
    opencv.threshold( 80 );
    Blob[] blobs = opencv.blobs( 10, width*height/2, 100, true, OpenCV.MAX_VERTICES*4 );

    // draw blob results
    //    for( int i=0; i<blobs.length; i++ ) {
    //     beginShape();
    //       for( int j=0; j<blobs[i].points.length; j++ ) {
    //          vertex( blobs[i].points[j].x, blobs[i].points[j].y );
    //       }
    //      endShape(CLOSE);
    //   }
}

void quit() {
    kinect.quit();
}

int getThreshold() {
    return threshold;
}

void setThreshold(int t) {
    threshold = t;
}
}

*/

I am thinking it is probably something very stupid, but that I am just not experienced enough to see!

Any advice?

Thanks!!!!

elamda

Re: Kinect and OpenCV Blob Tracking

2 years ago

The error is telling you that you have depth defined as boolean, when it should be defined as a integer array.

So declare something like this in your code before that getRawDepth() method is called:

int[] depth;

kw is short for Kinect Width, which is the width of the depth map that is sent from the Kinect. The depth map has a width of 640 and height of 480. i have something like this in my code:

int kw = 640;

or you could simply replace kw with 640 in that line of code.

nawka3030

Re: Kinect and OpenCV Blob Tracking

1 year ago

hey,
does anyone have a final code for this example (reading kinect with opencv and tracking blobs), I keep trying all these and it just won't work, thanks for all help!!!

buzzlight..

Re: Kinect and OpenCV Blob Tracking

1 year ago

Hi there,
I had to write this once...not all this code will apply, but try to extract what matters to you:

// import Daniel Shiffman library
import hypermedia.video.*;
import java.awt.*;
import org.openkinect.*;
import org.openkinect.processing.*;

import netP5.*;

OpenCV opencv;

int w = 640;
int h = 480;
int oCVthreshold = 5;
int numBox = 4;

// showing how we can farm all the kinect stuff out to a separate class
KinectTracker tracker;

// kinect library ovject
Kinect kinect;

// Depth data
int[] depth;

void setup()
{
size(w, h);

// initializes the kinect
kinect = new Kinect (this);

opencv = new OpenCV( this );
opencv.capture(w,h);

// initializes tracker
tracker = new KinectTracker();
}

void draw()
{
background(255);

//run the tracking analysis
tracker.track();
// show the image
tracker.display();

// display some info
int t = tracker.getThreshold();
//int d = tracker.getRawDepth();
fill(255);
text("threshold: "+t+"   "+"framerate:" + (int)frameRate + "   "+"UP increase threshold, DOWN decrease threshold", 10, h -10);
//text( "depth: "+d, 10, height-30);
PImage cp = get();

//opencv.allocate(width,height);                        // create the bufer
opencv.copy(cp);

opencv.absDiff();
opencv.threshold(oCVthreshold);
//image( opencv.image(OpenCV.MEMORY), 0, 0 ); // absolute difference image

// working with blobs
Blob[] blobs = opencv.blobs( 300, w*h/3, 16, false );

pushMatrix();
int cNumBox= blobs.length;
if(cNumBox>numBox)cNumBox=numBox;
for( int i=0; i<cNumBox; i++ )
{
    Rectangle bounding_rect    = blobs[i].rectangle;
    float area = blobs[i].area;
    float circumference = blobs[i].length;
    Point centroid = blobs[i].centroid;
    Point[] points = blobs[i].points;

    // rectangle
    noFill();
    stroke( 255 );
    rect( bounding_rect.x, bounding_rect.y, bounding_rect.width, bounding_rect.height );

    // centroid
    stroke(255);
    line( centroid.x-10, centroid.y, centroid.x+10, centroid.y );
    line( centroid.x, centroid.y-10, centroid.x, centroid.y+10 );
    noStroke();

    // data
    //fill(255,200);
    PVector cornerPt = new PVector();

    if ( points.length>0 )
    {
      if(blobs[i].area < 3000.000)
      {
        float maxDist = 0;

        beginShape();
        for( int j=0; j<points.length; j++ )
        {
          vertex( points[j].x, points[j].y );
          float cDist = dist(blobs[i].centroid.x,blobs[i].centroid.y,points[j].x,points[j].y);
          if (cDist> maxDist)
          {
            cornerPt.x = points[j].x;
            cornerPt.y = points[j].y;
          }
        }
        endShape(CLOSE);
      }
    }
    fill(0,255,0);
    rect(cornerPt.x,cornerPt

    stroke(255);
    fill(255);
    text( centroid.x, centroid.x+5, centroid.y+15 );
    text( centroid.y, centroid.x+5, centroid.y+30 );
    text( area, centroid.x+5, centroid.y+45 );

}
popMatrix();
println(oCVthreshold);

//SEND OSC
}

void keyPressed()
{
int t = tracker.getThreshold();
if (key == CODED)
{
    if(keyCode == UP)
    {
      t += 5;
      tracker.setThreshold(t);
    }
    else if(keyCode == DOWN)
    {
      t -= 5;
      tracker.setThreshold(t);
    }
}
}
void mouseDragged()
{
oCVthreshold = int( map(mouseX,0,width,0,255) );
}

void stop()
{
opencv.stop();
kinect.quit();
super.stop();
}

BUZZ

cinnamon_..

Re: Kinect and OpenCV Blob Tracking

1 year ago

I did get it figured out in the end. You also need to know that working on Mac vs PC there is a difference in getting it to work. PC needs some work arounds.

nawka3030

Re: Kinect and OpenCV Blob Tracking

1 year ago

cool!
this example is definitely seeing some blobs but I'm getting this crazy green color floating all over the image. Is it also possible to track blobs over a particular color, like red and have it create attractor points ?:) I know its asking for to much but I have been really struggling with this..