How align textures using GLSL for depth and RGB of kinect

Aurelian24

Im working on a proyect that needs to cut the background and and show only the figure with RGB colors for this im using a kinect and use a video as background. This has to work on the raspberry PI. The only kinect library that works on the raspberry is open Kinect. Open kinect doesn´t have the mask function nor the skeleton function.

I´ve accoplish to create a mask that works pretty well on the raspberry using shaders thought the problem that im having is that the RGB and Depth textures are not aligned. I´ve heard that this is something that happends with the kinect 1414 and not with the kinect 1417, thought i´ve only got the 1414 model , so i have to make it work in here.

This is a reference pic of how the mask is working:

The thing in the background is the video, look how wrong the mask is.

Im testing it on windows, so im using kinect4win library, and i know kinect4win has the kinect.getMask() function that works perfectly but i need to run on the raspberry so that function is useless and so i have to make my own. Here is my code using kinect4win than then im going to translated openKinect :

PROCESSING :

import processing.video.*;
import kinect4WinSDK.Kinect;

import processing.video.*;

Movie movie;

Kinect kinect;


PGraphics canvas;
PShader shader;

// brightness threshold
float minBrightness = 0.8;


//Values tested on  kinect4windows
float scale = 0.9099997;
float xoffset = 0.08226191;
float yoffset = 0.08226191;

//values tested on openKinect for raspberry pi
/*float scale = 1.0799996;
float xoffset = -0.007174231;
float yoffset = -0.07838542;*/

void setup()
{
  fullScreen(P2D);
  //size(640, 480, P2D);
  frameRate(60);

  kinect = new Kinect(this);
  // load shader and set threshold
  shader = loadShader("mask.glsl");

  // Load and play the video in a loop
 /* movie = new Movie(this, "transit.mov");
  movie.loop();*/

  shader.set("xoffset", xoffset);
  shader.set("yoffset", yoffset);
  shader.set("scale", scale);
}

void draw()
{
  // reset screen
  background(0, 0);
  // visualise result
  shader.set("umbral", minBrightness);

  //0.05 PUEDE IR EH
  float xoffset = map(mouseX, 0, width, -0.1, 0.1);
  float yoffset = map(mouseY, 0, height, -0.1, 0.1);



  println("X Offset : ", xoffset);
  println("Y Offset : ", yoffset);


  fill(255, 0, 0);
  ellipse(mouseX, mouseY, 20, 20);

  /*fill(0);
  image(movie, 0, 0, width, height);
  fill(255);
  */

  shader.set("texture", kinect.GetDepth());
  shader.set("texture2", kinect.GetImage());

  shader(shader);
  image(kinect.GetDepth(), 0, 0, width, height);
  resetShader();

  fill(0, 255, 0);

  float sep = 20;
  text("FPS: " + frameRate, 10, height-sep);
  text("xoffset :" + xoffset, 10, height-sep*2);
  text("yoffset :" + xoffset, 10, height-sep*3);
  text("scale :" + scale, 10, height-sep*4);
}

void movieEvent(Movie m) {
  m.read();
}

void keyPressed() {

  if (key == '+') {

    minBrightness +=0.1;
  }
  if (key == '-') {

    minBrightness -=0.1;
  }

  if(key == 'a'){

   scale +=0.02; 
  }
  if(key == 's'){
   scale -= 0.02; 
  }
}

This is my GLSL CODE :

     /*#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif
*/
uniform sampler2D texture; //LA DEPTH

uniform sampler2D texture2; // LA RGB

varying vec4 vertColor;
varying vec4 vertTexCoord;

uniform float umbral;
uniform float xoffset;
uniform float yoffset;
uniform float scale;

void main() {
    vec3 luminanceVector = vec3(0.2125, 0.7154, 0.0721);
    vec4 c = texture2D(texture, vertTexCoord.st) * vertColor;


    //CORRECT DEPTH RGB ALIGMENT : 

    vec4 c2 =  texture2D(texture2, (vertTexCoord.st+vec2(xoffset,yoffset))*scale) * vertColor;

    float luminance = dot(luminanceVector, c.xyz);
    luminance = max(0.0, luminance - umbral);


    c.r = c2.r;
    c.g = c2.g;
    c.b = c2.b;
    c.a = sign(luminance);

    gl_FragColor = vec4(c.rgb,c.a);
}

Im posting this on shaders category cause all that i need to do is translate the depth mask to be align to the RGB, but im not very good using shaders. so my question is . How to translate depth mask to align to rgb texture?

UPDATE : i´ve managed to set a Xoffset and Yoffset for aligning the textures, yet it doesn´t seem to be only a problem of aligment but also a problem of scale, like the depth image is a bit larger. Maybe some OpenCV algorithm for improving it?

UPDATE2: i´ve managed to set a SCALE, better, but still inacurate, i don´t know how to solve this now.

UPDATE3: SOLVE IT ! the scale was not really add it, now the code is fully functional