Silhouette depth image using Kinect

edited August 2014 in Kinect

Hey guys, I'm experiencing a bit of trouble getting the depth image to produce a silhouette. At the moment it just produces green streaks instead of an outline of the user. Any help would be much appreciated! Many thanks in advance, Ryan.

    import SimpleOpenNI.*;
    PFont font;

    //DECLARE 
    ArrayList ballCollection; 
    Table stateData;
    int rowCount;
    SimpleOpenNI kinect;
    PVector convertedRightHand = new PVector();

    //Network Display   Soma | Light Blue | Teal | Red | Lavender
    color[] SomaScheme = {
      #FDE7DC, #74C8CC, #219399, #FF5C74
    };
    color[] palette = SomaScheme;
    int SomaColour = palette[0];
    int ellipseColour = palette[1];
    //Circle/Table
    float age;
    float degree;
    float circleSize = 20;
    //Silhouette
    PImage userImage;
    int[] userMap;
    PImage rgbImage;
    color pixelColor;


    void setup() {
      size(600, 600);
      smooth();
      font = loadFont("font18.vlw");
      textFont(font);

      //INITIALISE
      //Initialise Table
      stateData = new Table("stateData.tsv"); 
      rowCount = stateData.getRowCount();

      //Initialise kinect
      kinect = new SimpleOpenNI(this);
      kinect.enableDepth();
      // turn on user tracking
      kinect.enableUser();
      //Silhouette
      kinect.enableRGB();
      userImage=createImage(width, height, RGB);

      //Initialise ArrayList
      ballCollection = new ArrayList();

      //Provide values for x & y
      for (int row = 0; row < rowCount; row++) {
        float degree = stateData.getFloat(row, 3);
        float age = stateData.getFloat(row, 4);
        float ellipseX = map(age, 30, 41, 0, width); 
        float ellipseY = map(degree, 14, 47, 0, height);
        String stateName = stateData.getString(row, 2);
        Ball myBall = new Ball (ellipseX, ellipseY, circleSize, circleSize, stateName); 
        ballCollection.add(myBall);
      }
    }

    void draw() {
      background(0);

      runKinect(); //handPosition

      //CALL FUNCTIONALITY
      for (int i = 0; i < ballCollection.size (); i++) { 
        Ball myBall = (Ball) ballCollection.get(i); 
        myBall.over();
        myBall.display();
      }
    }

    void onNewUser(SimpleOpenNI curContext, int userId)
    {
      println("onNewUser - userId: " + userId);
      println("\tstart tracking skeleton");

      kinect.startTrackingSkeleton(userId);
    }

    void onLostUser(SimpleOpenNI curContext, int userId)
    {
      println("onLostUser - userId: " + userId);
    }

    void onVisibleUser(SimpleOpenNI curContext, int userId)
    {
      println("onVisibleUser - userId: " + userId);
    }

    void runKinect() {
      kinect.update();
      rgbImage=kinect.rgbImage(); //!!!Silhouette - asinging rgbImage() to rgbImage
      PImage depth = kinect.depthImage();
      image(depth, 0, 0);
      tint(255, 126); // THIS MAY NOT BE NEEDED SOON
      IntVector userList = new IntVector(); // make a vector of ints to store the list of users
      kinect.getUsers(userList); // write the list of detected users into our vector

      ///!!!Silhouette\\\\
      userMap=kinect.userMap();
      for(int y=0;y<kinect.depthHeight();y++){
        for(int x=0;x<kinect.depthWidth();x++){
          int index=x+y*640;
          if(userMap[index]!=0){
              pixelColor=rgbImage.pixels[index];  //writes pixel colour to pixel array
            userImage.pixels[index]=color(0,255,0); 
          }else{
            userImage.pixels[index]=color(0);
          }
        }
      }
      //Circle Pointer statement
      if (userList.size() > 0) { // if we found any users
        int userId = userList.get(0); // get the first user
        if ( kinect.isTrackingSkeleton(userId)) {  // if we’re successfully calibrated
          PVector rightHand = new PVector(); // make a vector to store the left hand
          float confidence = kinect.getJointPositionSkeleton(userId, // put the position of the left hand into that vector
          SimpleOpenNI.SKEL_LEFT_HAND, 
          rightHand);
          convertedRightHand = new PVector(); // convert the detected hand position to "projective" coordinates that will match the depth image
          kinect.convertRealWorldToProjective(rightHand, convertedRightHand);
          fill(255, 0, 0); // and display it
          ellipse(convertedRightHand.x, convertedRightHand.y, 10, 10); //pointer for the hovering over ellipses
        }
      }
      ///!!!Silhouette\\\\
      userImage.updatePixels();
      image(userImage, 0, 0);
    }

Ball class: class Ball {

  // GLOBAL VARIABLES
  float x = 0; //declares these variables
  float y = 0;
  //float speedX = random(-1, 1); //speed not being used
  //float speedY = random(-1, 1); //speed not being used
  float circleSizeX;
  float circleSizeY;
  boolean over = false;
  boolean drag = false;
  String name;


  //CONSTRUCTOR
  Ball(float tempX, float tempY, float tempcircleSizeX, float tempcircleSizeY, String n) {  //Necessary to build the object, provides function arguments and can give 'setup' values that happens just once when called
    x = tempX; //intialises these variables so that they can be accessed anywhere
    y = tempY;
    circleSizeX = tempcircleSizeX;
    circleSizeY = tempcircleSizeY;
    name = n;
  }

  //FUNCTIONS
  void run() { //user function but standard?. used to describe behaviour in this case anyway
    //display   - here Jose is listing all of the behaviours he wants the sketch to include
    display(); //this is already written below so refer to it as a methodr
  }  

  void display() {   //this is a user function but seems to be pretty standard. 
    fill(palette[1]);
    if (over) {
      fill(palette[3]);
      //println(name);
      text(name, x, y+3);
    }
    ellipse(x, y, circleSizeX, circleSizeY);
  }

  void over() {
    //if (dist (x, y, mouseX, mouseY) < circleSize/2){
    if (dist (x, y, convertedRightHand.x, convertedRightHand.y) < circleSize/2){
    over = true;
    }
    else{
    over = false;
    }
  }
}
Tagged:

Answers

  • edited August 2014

    I didn't go through all of your code, but I noticed one thing that might help. The length of the userMap array should be the same as the userImage pixel array. So, you don't need x & y for loops when creating the silhouette. Something like this would work better:

    userMap=kinect.userMap();
    for (int i=0; i<userMap.length; i++) {
      if (userMap[i]!=0) {
        pixelColor=rgbImage.pixels[i];  //writes pixel colour to pixel array
        userImage.pixels[i]=color(0, 255, 0);
      } else {
        userImage.pixels[i]=color(0);
      }
    }
    
  • Also, if you're trying to align the rgb and depth images, make sure you add the point to depth function in your setup:

    // align depth data to image data
    kinect.alternativeViewPointDepthToImage();
    
  • Hey @Hunter, thanks for you reply man - that makes a lot of sense really. I've made the changes you suggested and keep getting returned an EOF error. Any chance you know why that would be? I've checked the {} etc but can't seem to work it out. Thanks for your help! Ryan

    Here's the updated code: import SimpleOpenNI.*; PFont font;

    //DECLARE 
    ArrayList ballCollection; 
    Table stateData;
    int rowCount;
    SimpleOpenNI kinect;
    PVector convertedRightHand = new PVector();
    
    //Network Display   Soma | Light Blue | Teal | Red | Lavender
    color[] SomaScheme = {
      #FDE7DC, #74C8CC, #219399, #FF5C74
    };
    color[] palette = SomaScheme;
    int SomaColour = palette[0];
    int ellipseColour = palette[1];
    //Circle/Table
    float age;
    float degree;
    float circleSize = 20;
    //Silhouette
    PImage userImage;
    int[] userMap;
    PImage rgbImage;
    color pixelColor;
    
    
    void setup() {
      size(600, 600);
      smooth();
      font = loadFont("font18.vlw");
      textFont(font);
    
      //INITIALISE
      //Initialise Table
      stateData = new Table("stateData.tsv"); 
      rowCount = stateData.getRowCount();
    
      //Initialise kinect
      kinect = new SimpleOpenNI(this);
      kinect.enableDepth();
      // turn on user tracking
      kinect.enableUser();
      //Silhouette
      kinect.enableRGB();
      // align depth data to image data
      kinect.alternativeViewPointDepthToImage();
    
      userImage=createImage(width, height, RGB);
    
      //Initialise ArrayList
      ballCollection = new ArrayList();
    
      //Provide values for x & y
      for (int row = 0; row < rowCount; row++) {
        float degree = stateData.getFloat(row, 3);
        float age = stateData.getFloat(row, 4);
        float ellipseX = map(age, 30, 41, 0, width); 
        float ellipseY = map(degree, 14, 47, 0, height);
        String stateName = stateData.getString(row, 2);
        Ball myBall = new Ball (ellipseX, ellipseY, circleSize, circleSize, stateName); 
        ballCollection.add(myBall);
      }
    }
    
    void draw() {
      background(0);
    
      runKinect(); //handPosition
    
      //CALL FUNCTIONALITY
      for (int i = 0; i < ballCollection.size (); i++) { 
        Ball myBall = (Ball) ballCollection.get(i); 
        myBall.over();
        myBall.display();
      }
    }
    
    void onNewUser(SimpleOpenNI curContext, int userId)
    {
      println("onNewUser - userId: " + userId);
      println("\tstart tracking skeleton");
    
      kinect.startTrackingSkeleton(userId);
    }
    
    void onLostUser(SimpleOpenNI curContext, int userId)
    {
      println("onLostUser - userId: " + userId);
    }
    
    void onVisibleUser(SimpleOpenNI curContext, int userId)
    {
      println("onVisibleUser - userId: " + userId);
    }
    
    void runKinect() {
      kinect.update();
      rgbImage=kinect.rgbImage(); //!!!Silhouette - asinging rgbImage() to rgbImage
      PImage depth = kinect.depthImage();
      image(depth, 0, 0);
      tint(255, 126); // THIS MAY NOT BE NEEDED SOON
      IntVector userList = new IntVector(); // make a vector of ints to store the list of users
      kinect.getUsers(userList); // write the list of detected users into our vector
    
      ///!!!Silhouette\\\\
      userMap=kinect.userMap();
    
      for (int i = 0; i<userMap.length; i++) {
        //int index=x+y*640;
        if (userMap[i]!=0) {
          pixelColor=rgbImage.pixels[i];  //writes pixel colour to pixel array
          userImage.pixels[i]=color(0, 255, 0);
        } else {
          userImage.pixels[i]=color(0);
        }
      }
    }
    //Circle Pointer statement
    if (userList.size() > 0) { // if we found any users
      int userId = userList.get(0); // get the first user
      if ( kinect.isTrackingSkeleton(userId)) {  // if we’re successfully calibrated
        PVector rightHand = new PVector(); // make a vector to store the left hand
        float confidence = kinect.getJointPositionSkeleton(userId, // put the position of the left hand into that vector
        SimpleOpenNI.SKEL_LEFT_HAND, 
        rightHand);
        convertedRightHand = new PVector(); // convert the detected hand position to "projective" coordinates that will match the depth image
        kinect.convertRealWorldToProjective(rightHand, convertedRightHand);
        fill(255, 0, 0); // and display it
        ellipse(convertedRightHand.x, convertedRightHand.y, 10, 10); //pointer for the hovering over ellipses
      }
    }
    ///!!!Silhouette\\\\
    userImage.updatePixels();
    image(userImage, 0, 0);
    }
    
  • Answer ✓

    you got an extra closing bracket right above the //Circle Pointer statement comment

  • Also a few more tips for you. Use lots of TABS, makes things easier to find and read. You also might want to try adding the PDE X programming mode. It adds in some cool debugging features that help find mistakes like the one above. You can add contributed modes by selecting "Add Mode..." from the Mode menu in the upper-right corner of the PDE.

  • You're a legend man, thank you so much! I found an extra reference for anyone else experiencing this issue which basically seems reiterate what Hunter was saying: http://blog.3dsense.org/programming/programming-for-kinect-4-kinect-app-with-skeleton-tracking-openni-2-0/ Cheers, Ryan

  • Hello Ryan,

    I am experiencing a similar problem. Did you finally solved the problem? I´ll appreciate some help!

    Thanks

  • If anyone know how to draw a bonding box around the extracted human silhouette? and out put the width and height of BB?

  • Hello !

    You should use a "marching square" algorithm to extract the silhouette from the background.

    When you will get the positions of each pixel on the border of the silhouette (from Marching Square), you would have to loop on them to define minimal x-position, minimal y-position, maximal x-position, maximal y-position, and then you'll have your bounding box.

    Good luck !

  • If you do not need a very accurate result, there is an easier way to do it :

    • Create a PImage with small dimension (64x48 for example)
    • Draw the "kinect view" inside that PImage
    • load the pixels of the PImage
    • loop on them, check the color of each pixel to know if it's part of the silhouette or not
    • it the pixel is a part of the silhouette, then check for each if it's position correspond to a minimal X/Y or a maximal X/Y
    • when you have your data, you just have to rescale them to fit in the size of your screen

    But marching-square method is better and probably faster, and not so complex to understand, you definitly should look at it

  • Thanks tlecoz !! I want to have exact measurements

  • where can I find the marching square implementation for kinect depth ? I searched on google. i`m unable to find it still

Sign In or Register to comment.