how to place rgb camera pixels onto depth camera array V1 Kinect

harveyshaw9864

Processsing 3 Kinect v1

Basically, i've stripped down my code so it's just this, atm it only shows the user when you're in the correct threshold. You go pink as I have made it that color. I want to know how to only show the RGB pixels for the bits that are pink.

Could I also just lay over a black shape where the threshold is not?

Many thanks for reading, any feedback is much appreciated

            import org.openkinect.freenect.*;
            import org.openkinect.freenect2.*;
            import org.openkinect.processing.*;
            import org.openkinect.tests.*;


            Kinect kinect;


            int kinectWidth = 640;
            int kinectHeight = 480;
            PImage cam = createImage(640, 480, RGB);
            int minThresh = 300;
            int maxThresh = 700;
            float reScale;


            void setup() {

              size(640, 480, P3D);
              kinect = new Kinect(this);
              kinect.enableMirror(true);
              kinect.initDepth();
              reScale = (float) width / kinectWidth;
            }


            void draw() {

              cam.loadPixels();

              int[] depth = kinect.getRawDepth();

              for (int x = 0; x < kinect.width; x++) {
                for (int y = 0; y < kinect.height; y++) {
                  int offset = x + y * kinect.width;
                  int d = depth[offset];


                  if (d > minThresh && d < maxThresh) {        
                    cam.pixels[offset] = color(255,0,200);

                  } else{
                    cam.pixels[offset] = color(0);
                  }
                }
              }

              cam.updatePixels();

              background(255);
              image(cam,0,0);

            }

kfrajer

Does the current code work? Can you map depth to cam.loadpixels array? in other words, are they the same size and are they representing the same field of view?

What about you do this small change from:

  if (d > minThresh && d < maxThresh) {        
    cam.pixels[offset] = color(255,0,200); 
  } else{
    cam.pixels[offset] = color(0);
  }

to

  if ( ! (d > minThresh && d < maxThresh) ) 
    cam.pixels[offset] = color(0);
  }

This resets all those pixels outside the threshold. The pixels within the Threshold are not modified and they will contain the original RGB color.

Another suggestion is to explore previous posts if you haven't done so: https://forum.processing.org/two/search?Search=getRawDepth and in your case, make sure you are working with post relevant to Kinect V1.

Kf

harveyshaw9864

Firstly thank you so much for replying!!!

Unfortunately, I have tried similar edits like that before.

Selecting everything outside the threshold just returns a solid color on the screen, and if I take out the else in the original code, it just draws the threshold image over itself (much like if background() is in setup)

I even tried adding transparency, for example

'cam.pixels[offset] = color(0,0,0,0);'

but to no avail, it didn't change a thing.

the background is never being shown either. very strange.

harveyshaw9864

I think I've cracked it. And the solution involved your suggestion @kfrajer so thankyou

import org.openkinect.freenect.*;
import org.openkinect.freenect2.*;
import org.openkinect.processing.*;
import org.openkinect.tests.*;

Kinect kinect;

int kinectWidth = 640;
int kinectHeight = 480;

PImage cam;
int minThresh = 300;
int maxThresh = 700;
float reScale;

void setup() {

  size(640, 480, P2D);
  kinect = new Kinect(this);
  kinect.enableMirror(true);
  kinect.initDepth();
  kinect.initVideo();
  reScale = (float) width / kinectWidth;
  background(255);
}


void draw() {

  image(kinect.getVideoImage(),0,0);

  PImage cam = kinect.getVideoImage();

  int[] depth = kinect.getRawDepth();

  for (int x = 0; x < 640; x++) {
    for (int y = 0; y < 480; y++) {
      int offset = x + y * kinect.width;
      int d = depth[offset];

      if (!(d > minThresh && d < maxThresh)) {        
        cam.pixels[offset] = color(0);
      } 
    }
  }
}

Basically I think I just had to draw the image in draw() every time and im not 100% sure why but just thank god it works. It was kinda of a happy accident to be honest!!

However there is still the problem of solving the offset between the rgb cam and the depth cam.

I've tried moving the rgb left a bit but the depth seems to go with it???

kfrajer

I've tried moving the rgb left a bit but the depth seems to go with it???

What do you mean with this?

And why do you want to do this? Aren't they coupled? Please provide a bit more of background to make the question more clear of what you want to do next.

In your prvious code, I don't see where you are drawing your image in the sketch. Is the code complete?

also please avoid using hard core values, if possible, in draw. Use width and height "available" keywords to refer to your screen parameters. It is a good practice to do this.

Kf

harveyshaw9864

here are some images of what I mean.

I know kinect v2 has the ability to map the depth to the rgb camera but Im wondering if there is some kind of easy work around to just move the under lying rgb camera image to the left slightly?

i tried changing this

image(kinect.getVideoImage(),0,0);

to this

image(kinect.getVideoImage(),-40,0);

but the depth image seems to move with it?

Many thanks

kfrajer

Unfortunately I can't help as I don't have a kinect unit to do testing. I suggest you check previous approaches in the forum while you get feddback from other forum goers.

Another suggestion is to check the Shiffman's series on kinect. YOu can google it and I believe he did his video series using kinect V1.

Kf

harveyshaw9864

Okay well thankyou so much for all your help! it is much appreciated !!!

kfrajer

By the way, your images didn't displayed properly in your post. Try posting them again.

Kf

Aurelian24

So i´ve managed to make some kind of mask using shaders wich runs like a charm. It´s using ofkinect4windows, only change the name functions to make it work :

i´ve attached here in this google drive link :

https://drive.google.com/open?id=1M37lKRr69vONIZLYiCNqBzE7zgVr-oMh