Can one use he_mesh to triangulate a kinect point cloud? if so, how?

gersart

Primarily for @wblut

I have a series of points culled from a kinect point cloud (basically only those points relevant to the person, any past a certain depth are ignored)

I am having trouble choosing the appropriate tool in he_mesh to generate a mesh of these point clouds. I have been going through the examples and the references but I cannot seem to find the most appropriate method for achieving this.

If I set things up using the triangulate2d, I lose any depth information, resulting in a flat plane in the shape of the silhouette

I tried triangulate3d, but I get tetrahedra, which don't really feel like they are the right tools for this. I am hoping for a mesh I can use to finagle some awesome things out of. The example with the tetrahedra is... not that.

So, in short, what would be the best way to generate a mesh whose surface is composed of specific points from a Kinect point cloud? I'm not interested in the back of the mesh right now.

This is the code I have so far, minus the he_mesh stuff that would let me explore further.

import org.openkinect.freenect.*;
import org.openkinect.processing.*;
import java.util.List;

//for the hemesh stuff
import wblut.hemesh.*;
import wblut.core.*;
import wblut.geom.*;
import wblut.processing.*;
import wblut.math.*;

///for the gui control
import controlP5.*;

// Kinect Library object
Kinect kinect;

//for the hemesh stuff

HE_Mesh firstMesh;
WB_Render3D renderMesh;

int[] tetrahedra;

boolean makeMesh;
boolean newMesh;

///for gui controls
ControlP5 sliderStuff;

///for control values
int dCutoff = 848;


// Angle for rotation
float a = 0;

//to store points from each frame of kinect
//WB_Point[] points;
//List <WB_Coords> points
List <WB_Point> points;


// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[2048];

void setup() {
  // Rendering in P3D
  size(800, 600, P3D);
  kinect = new Kinect(this);
  kinect.initDepth();

  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }

  //slider to remove points past a certain depth from consideration.
  sliderStuff = new ControlP5(this);
  sliderStuff.addSlider("dCutoff").setPosition(10,10).setRange(0,2048);

   ///only make the mesh once
  makeMesh = false;
  newMesh = false;
}

void draw(){
  background(0);
  text("depth cutoff: " + dCutoff,60,60);

  pushMatrix();
  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepth();

  translate(width/2, height/2,0);
  int skip = 8;
  points = new ArrayList <WB_Point>();
  for (int x = 0; x < kinect.width; x += skip) {
    for (int y = 0; y < kinect.height; y += skip) {
      int offset = x + y*kinect.width;

      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      if (rawDepth<dCutoff){
        PVector v = depthToWorld(x, y, rawDepth);

        stroke(255);
        pushMatrix();
        // Scale up by 200
        float factor = 200;
        translate(v.x*factor, v.y*factor, factor-v.z*factor);
        /* used for when points was an array
            points[x+y*(kinect.width/skip)][0] = v.x;
            points[x+y*(kinect.width/skip)][1] = v.y;
            points[x+y*(kinect.width/skip)][2] = v.z;
         */
            points.add (new WB_Point(v.x, v.y, v.z));
        // Draw a point
          point(0, 0);

        popMatrix();
      }
    }
  }
  if (makeMesh & newMesh) {
    //tried this, have no idea what to do with tetrahedra
    //WB_Triangulation3D triangulation = WB_Triangulate.triangulate3D(points);
    //tetrahedra=triangulation.getTetrahedra();

    //confirms that there are points being generated
    println ("there are "+points.size()+" points"); 


    //HE_Mesh thisMesh = new HEC_FromTriangulation().setPoints(points);

   //this next line results in a Null Pointer Exception, but the previous line works so: ??
    firstMesh=new HE_Mesh(new HEC_FromTriangulation().setPoints(points));
    firstMesh.smooth();
    newMesh = false; //only make the mesh once
  }

  if (makeMesh){
    pushMatrix();
    directionalLight(255, 255, 255, 1, 1, -1);
    directionalLight(127, 127, 127, -1, -1, 1);
    /*
      //translate(-objectRadius,-objectRadius);
      renderMesh.drawFaces(firstMesh);
      */
      rotateY(mouseX*1.0f/width*TWO_PI);
      rotateX(mouseY*1.0f/height*TWO_PI);
      for(int i=0;i<tetrahedra.length;i+=4){

      WB_Point center;
      pushMatrix();
        /*draw the mesh stuff al'a your suggestion here
      popMatrix();
      }
    popMatrix();
  }
  popMatrix();

}

void keyPressed(){
  if (key == 'r') {
    if (makeMesh){
      makeMesh = false;
      newMesh = false;
    } else {
      makeMesh  = true;
      newMesh = true;
    }
  }
}

// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
  if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  depthLookUp[depthValue];//rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

solub

Hi @gersart,

I realise you posted this question more than a year ago and that you've probably moved on but I feel like it'd be better to provide an answer anyway, hoping it will be useful for future forum users.

I had the same problem you're describing (trying to compute 3D triangulation with Hemesh) and with the precious help of this this forum I could come up with a rather simple solution: using the Triangulate library by Nicolas Clavaud instead.

The library is available here

You can find an example sketch here

For a 3D example sketch you can have a look at the script I posted on this thread (post from may the 3rd, it's in Python but really easy to understand).

Hope that helps.

koogs

I realise you posted this question more than a year ago

didn't stop you bumping those other threads that were from 2015

solub

@koogs I'm confused, is it a problem ? I did it because I thought it could be useful for future forum users. I've personally spent a lot of time reading all these questions regarding 3D Delaunay triangulation hoping to find a proper solution. Sharing this workaround is a way for me:

• to provide help to people who are in a similar situation (It would have saved me a substantial amount of time to find a solution reading one of those threads)
• to give something back to the Processing community

Please tell me what is your opinion/guidelines on "bumping" old threads.