Silhouette depth image using Kinect

RyanAchten

Hey guys, I'm experiencing a bit of trouble getting the depth image to produce a silhouette. At the moment it just produces green streaks instead of an outline of the user. Any help would be much appreciated! Many thanks in advance, Ryan.

    import SimpleOpenNI.*;
    PFont font;

    //DECLARE 
    ArrayList ballCollection; 
    Table stateData;
    int rowCount;
    SimpleOpenNI kinect;
    PVector convertedRightHand = new PVector();

    //Network Display   Soma | Light Blue | Teal | Red | Lavender
    color[] SomaScheme = {
      #FDE7DC, #74C8CC, #219399, #FF5C74
    };
    color[] palette = SomaScheme;
    int SomaColour = palette[0];
    int ellipseColour = palette[1];
    //Circle/Table
    float age;
    float degree;
    float circleSize = 20;
    //Silhouette
    PImage userImage;
    int[] userMap;
    PImage rgbImage;
    color pixelColor;


    void setup() {
      size(600, 600);
      smooth();
      font = loadFont("font18.vlw");
      textFont(font);

      //INITIALISE
      //Initialise Table
      stateData = new Table("stateData.tsv"); 
      rowCount = stateData.getRowCount();

      //Initialise kinect
      kinect = new SimpleOpenNI(this);
      kinect.enableDepth();
      // turn on user tracking
      kinect.enableUser();
      //Silhouette
      kinect.enableRGB();
      userImage=createImage(width, height, RGB);

      //Initialise ArrayList
      ballCollection = new ArrayList();

      //Provide values for x & y
      for (int row = 0; row < rowCount; row++) {
        float degree = stateData.getFloat(row, 3);
        float age = stateData.getFloat(row, 4);
        float ellipseX = map(age, 30, 41, 0, width); 
        float ellipseY = map(degree, 14, 47, 0, height);
        String stateName = stateData.getString(row, 2);
        Ball myBall = new Ball (ellipseX, ellipseY, circleSize, circleSize, stateName); 
        ballCollection.add(myBall);
      }
    }

    void draw() {
      background(0);

      runKinect(); //handPosition

      //CALL FUNCTIONALITY
      for (int i = 0; i < ballCollection.size (); i++) { 
        Ball myBall = (Ball) ballCollection.get(i); 
        myBall.over();
        myBall.display();
      }
    }

    void onNewUser(SimpleOpenNI curContext, int userId)
    {
      println("onNewUser - userId: " + userId);
      println("\tstart tracking skeleton");

      kinect.startTrackingSkeleton(userId);
    }

    void onLostUser(SimpleOpenNI curContext, int userId)
    {
      println("onLostUser - userId: " + userId);
    }

    void onVisibleUser(SimpleOpenNI curContext, int userId)
    {
      println("onVisibleUser - userId: " + userId);
    }

    void runKinect() {
      kinect.update();
      rgbImage=kinect.rgbImage(); //!!!Silhouette - asinging rgbImage() to rgbImage
      PImage depth = kinect.depthImage();
      image(depth, 0, 0);
      tint(255, 126); // THIS MAY NOT BE NEEDED SOON
      IntVector userList = new IntVector(); // make a vector of ints to store the list of users
      kinect.getUsers(userList); // write the list of detected users into our vector

      ///!!!Silhouette\\\\
      userMap=kinect.userMap();
      for(int y=0;y<kinect.depthHeight();y++){
        for(int x=0;x<kinect.depthWidth();x++){
          int index=x+y*640;
          if(userMap[index]!=0){
              pixelColor=rgbImage.pixels[index];  //writes pixel colour to pixel array
            userImage.pixels[index]=color(0,255,0); 
          }else{
            userImage.pixels[index]=color(0);
          }
        }
      }
      //Circle Pointer statement
      if (userList.size() > 0) { // if we found any users
        int userId = userList.get(0); // get the first user
        if ( kinect.isTrackingSkeleton(userId)) {  // if we’re successfully calibrated
          PVector rightHand = new PVector(); // make a vector to store the left hand
          float confidence = kinect.getJointPositionSkeleton(userId, // put the position of the left hand into that vector
          SimpleOpenNI.SKEL_LEFT_HAND, 
          rightHand);
          convertedRightHand = new PVector(); // convert the detected hand position to "projective" coordinates that will match the depth image
          kinect.convertRealWorldToProjective(rightHand, convertedRightHand);
          fill(255, 0, 0); // and display it
          ellipse(convertedRightHand.x, convertedRightHand.y, 10, 10); //pointer for the hovering over ellipses
        }
      }
      ///!!!Silhouette\\\\
      userImage.updatePixels();
      image(userImage, 0, 0);
    }

Ball class: class Ball {

  // GLOBAL VARIABLES
  float x = 0; //declares these variables
  float y = 0;
  //float speedX = random(-1, 1); //speed not being used
  //float speedY = random(-1, 1); //speed not being used
  float circleSizeX;
  float circleSizeY;
  boolean over = false;
  boolean drag = false;
  String name;


  //CONSTRUCTOR
  Ball(float tempX, float tempY, float tempcircleSizeX, float tempcircleSizeY, String n) {  //Necessary to build the object, provides function arguments and can give 'setup' values that happens just once when called
    x = tempX; //intialises these variables so that they can be accessed anywhere
    y = tempY;
    circleSizeX = tempcircleSizeX;
    circleSizeY = tempcircleSizeY;
    name = n;
  }

  //FUNCTIONS
  void run() { //user function but standard?. used to describe behaviour in this case anyway
    //display   - here Jose is listing all of the behaviours he wants the sketch to include
    display(); //this is already written below so refer to it as a methodr
  }  

  void display() {   //this is a user function but seems to be pretty standard. 
    fill(palette[1]);
    if (over) {
      fill(palette[3]);
      //println(name);
      text(name, x, y+3);
    }
    ellipse(x, y, circleSizeX, circleSizeY);
  }

  void over() {
    //if (dist (x, y, mouseX, mouseY) < circleSize/2){
    if (dist (x, y, convertedRightHand.x, convertedRightHand.y) < circleSize/2){
    over = true;
    }
    else{
    over = false;
    }
  }
}

Hunter

I didn't go through all of your code, but I noticed one thing that might help. The length of the userMap array should be the same as the userImage pixel array. So, you don't need x & y for loops when creating the silhouette. Something like this would work better:

userMap=kinect.userMap();
for (int i=0; i<userMap.length; i++) {
  if (userMap[i]!=0) {
    pixelColor=rgbImage.pixels[i];  //writes pixel colour to pixel array
    userImage.pixels[i]=color(0, 255, 0);
  } else {
    userImage.pixels[i]=color(0);
  }
}

Hunter

Also, if you're trying to align the rgb and depth images, make sure you add the point to depth function in your setup:

// align depth data to image data
kinect.alternativeViewPointDepthToImage();

RyanAchten

Hey @Hunter, thanks for you reply man - that makes a lot of sense really. I've made the changes you suggested and keep getting returned an EOF error. Any chance you know why that would be? I've checked the {} etc but can't seem to work it out. Thanks for your help! Ryan

Here's the updated code: import SimpleOpenNI.*; PFont font;

//DECLARE 
ArrayList ballCollection; 
Table stateData;
int rowCount;
SimpleOpenNI kinect;
PVector convertedRightHand = new PVector();

//Network Display   Soma | Light Blue | Teal | Red | Lavender
color[] SomaScheme = {
  #FDE7DC, #74C8CC, #219399, #FF5C74
};
color[] palette = SomaScheme;
int SomaColour = palette[0];
int ellipseColour = palette[1];
//Circle/Table
float age;
float degree;
float circleSize = 20;
//Silhouette
PImage userImage;
int[] userMap;
PImage rgbImage;
color pixelColor;


void setup() {
  size(600, 600);
  smooth();
  font = loadFont("font18.vlw");
  textFont(font);

  //INITIALISE
  //Initialise Table
  stateData = new Table("stateData.tsv"); 
  rowCount = stateData.getRowCount();

  //Initialise kinect
  kinect = new SimpleOpenNI(this);
  kinect.enableDepth();
  // turn on user tracking
  kinect.enableUser();
  //Silhouette
  kinect.enableRGB();
  // align depth data to image data
  kinect.alternativeViewPointDepthToImage();

  userImage=createImage(width, height, RGB);

  //Initialise ArrayList
  ballCollection = new ArrayList();

  //Provide values for x & y
  for (int row = 0; row < rowCount; row++) {
    float degree = stateData.getFloat(row, 3);
    float age = stateData.getFloat(row, 4);
    float ellipseX = map(age, 30, 41, 0, width); 
    float ellipseY = map(degree, 14, 47, 0, height);
    String stateName = stateData.getString(row, 2);
    Ball myBall = new Ball (ellipseX, ellipseY, circleSize, circleSize, stateName); 
    ballCollection.add(myBall);
  }
}

void draw() {
  background(0);

  runKinect(); //handPosition

  //CALL FUNCTIONALITY
  for (int i = 0; i < ballCollection.size (); i++) { 
    Ball myBall = (Ball) ballCollection.get(i); 
    myBall.over();
    myBall.display();
  }
}

void onNewUser(SimpleOpenNI curContext, int userId)
{
  println("onNewUser - userId: " + userId);
  println("\tstart tracking skeleton");

  kinect.startTrackingSkeleton(userId);
}

void onLostUser(SimpleOpenNI curContext, int userId)
{
  println("onLostUser - userId: " + userId);
}

void onVisibleUser(SimpleOpenNI curContext, int userId)
{
  println("onVisibleUser - userId: " + userId);
}

void runKinect() {
  kinect.update();
  rgbImage=kinect.rgbImage(); //!!!Silhouette - asinging rgbImage() to rgbImage
  PImage depth = kinect.depthImage();
  image(depth, 0, 0);
  tint(255, 126); // THIS MAY NOT BE NEEDED SOON
  IntVector userList = new IntVector(); // make a vector of ints to store the list of users
  kinect.getUsers(userList); // write the list of detected users into our vector

  ///!!!Silhouette\\\\
  userMap=kinect.userMap();

  for (int i = 0; i<userMap.length; i++) {
    //int index=x+y*640;
    if (userMap[i]!=0) {
      pixelColor=rgbImage.pixels[i];  //writes pixel colour to pixel array
      userImage.pixels[i]=color(0, 255, 0);
    } else {
      userImage.pixels[i]=color(0);
    }
  }
}
//Circle Pointer statement
if (userList.size() > 0) { // if we found any users
  int userId = userList.get(0); // get the first user
  if ( kinect.isTrackingSkeleton(userId)) {  // if we’re successfully calibrated
    PVector rightHand = new PVector(); // make a vector to store the left hand
    float confidence = kinect.getJointPositionSkeleton(userId, // put the position of the left hand into that vector
    SimpleOpenNI.SKEL_LEFT_HAND, 
    rightHand);
    convertedRightHand = new PVector(); // convert the detected hand position to "projective" coordinates that will match the depth image
    kinect.convertRealWorldToProjective(rightHand, convertedRightHand);
    fill(255, 0, 0); // and display it
    ellipse(convertedRightHand.x, convertedRightHand.y, 10, 10); //pointer for the hovering over ellipses
  }
}
///!!!Silhouette\\\\
userImage.updatePixels();
image(userImage, 0, 0);
}

Hunter

you got an extra closing bracket right above the //Circle Pointer statement comment

Hunter

Also a few more tips for you. Use lots of TABS, makes things easier to find and read. You also might want to try adding the PDE X programming mode. It adds in some cool debugging features that help find mistakes like the one above. You can add contributed modes by selecting "Add Mode..." from the Mode menu in the upper-right corner of the PDE.

RyanAchten

You're a legend man, thank you so much! I found an extra reference for anyone else experiencing this issue which basically seems reiterate what Hunter was saying: http://blog.3dsense.org/programming/programming-for-kinect-4-kinect-app-with-skeleton-tracking-openni-2-0/ Cheers, Ryan

Pablovitamin

Hello Ryan,

I am experiencing a similar problem. Did you finally solved the problem? I´ll appreciate some help!

Thanks

kosalacool

If anyone know how to draw a bonding box around the extracted human silhouette? and out put the width and height of BB?

tlecoz

Hello !

You should use a "marching square" algorithm to extract the silhouette from the background.

When you will get the positions of each pixel on the border of the silhouette (from Marching Square), you would have to loop on them to define minimal x-position, minimal y-position, maximal x-position, maximal y-position, and then you'll have your bounding box.

Good luck !

tlecoz

If you do not need a very accurate result, there is an easier way to do it :

• Create a PImage with small dimension (64x48 for example)
• Draw the "kinect view" inside that PImage
• load the pixels of the PImage
• loop on them, check the color of each pixel to know if it's part of the silhouette or not
• it the pixel is a part of the silhouette, then check for each if it's position correspond to a minimal X/Y or a maximal X/Y
• when you have your data, you just have to rescale them to fit in the size of your screen

But marching-square method is better and probably faster, and not so complex to understand, you definitly should look at it

kosalacool

Thanks tlecoz !! I want to have exact measurements

kosalacool

where can I find the marching square implementation for kinect depth ? I searched on google. i`m unable to find it still