Intel Realsense R200 Obtain Depth Image in Canvas

matt82au

Hey everyone. I am certainly a bit over my head on this one, but chipping away bit by bit. I hope someone might have some excellent insight to put my head in the right direction :)

I am trying to obtain depth data from an Intel Realsense r200 camera. Below is my code (with some errors to show where I am falling down..)

import intel.rssdk.*;
//import java.lang.System.*;
//import java.util.*;
//import javax.swing.*;
//import java.awt.event.*;
//import java.awt.image.*;
//import java.awt.*;


PXCMSenseManager sm = null;
int nframes = 0; 
PImage imgDepth = null; 

void setup() {
  size(640, 480);


// Create a SenseManager instance
PXCMSenseManager sm=PXCMSenseManager.CreateInstance();

// Select the color and depth streams
sm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_COLOR,640,480,30);
sm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_DEPTH,320,240,30);



// Initialize and Stream Samples
sm.Init();
for (;;) {
   // This function blocks until both samples are ready
   if (sm.AcquireFrame(true).isError()) break;

   // retrieve the samples
   PXCMCapture.Sample sample=sm.QuerySample();

   // work on the samples: sample.color & sample.depth
 // ...
   if (sample.depth != null) {
       PXCMImage.ImageData dData = new PXCMImage.ImageData();
        sample.depth.AcquireAccess(PXCMImage.Access.ACCESS_READ,PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32, dData);
        //imgDepth = dData.ToPImage(0, imgDepth); 
        imgDepth = dData.ToPImage(0, imgDepth); 
        sample.depth.ReleaseAccess(dData);
        image(imgDepth, 0,0);
 }



println(sample.depth);
  //hoping to convert the PXCMImage data to something drawable in canvas. String fail
  //String depth=sample.depth;
  //loadImage("depth");


   // go fetching the next samples
   sm.ReleaseFrame();}
}





 void draw() {
      //need to get data from sample.depth here.   
      //image("sample.depth", 0, 0);

}

// Close down
void dispose(){
  sm.close();

}

I am able to initialize the camera, and use println to view the data coming from the camera (sample.depth)

The readouts from the console comes in the form of..

intel.rssdk.PXCMImage@7E89278e intel.rssdk.PXCMImage@70a5104a intel.rssdk.PXCMImage@7548d086

etc etc etc.

The error I am getting at this stage is "The function ToPImage(int,PImage) does not exist". (line 43 I hope..)

This is code I pulled from one of the examples (which I am unable to successfully get running, I have had no luck getting any depth images into Processing at all). It seems to be a fundamental aspect of converting the cameras image format to PImage. Intels reference can be found here..

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?topimage_imagedata_pxcmimage.html

I believe the examples may have been designed for Processing 2.x so this could be an issue. Any insights would be amazing. Thanks!

jeremydouglass

@matt92au -- regarding your error:

"The function ToPImage(int,PImage) does not exist"

..are you sure the method shouldn't be "toPImage()" -- not "ToPImage()"? It is very unusual for a method to begin uppercase in Java.

You should be able to search the library intel.rssdk.* and find that method if it exists. It is also possible that you are passing the wrong arguments or in the wrong order -- you are passing an int and a PImage, but should be passing a PImage and an int (or something else).

Your debugging that prints "intel.rssdk.PXCMImage@7E89278e" is telling you that you are printing a depth object ("PXCMImage") and the memory address of that object ("7E89278e").

matt82au

Thank you for the response Jeremy. I have tried that approach also without any luck.

There actually is another reference in the RSSDK called PXCMImage.ImageData.ToPImage

which has
PImage ToPImage(int index, PImage dest); as its syntaxes.

Though I cant seem to get this to link to a PImage class in Processing.

Any other thoughts? Thanks :)

jeremydouglass

Sorry, no other ideas -- I'm not familiar with rssdk and don't have a Realsense on hand. Perhaps check past discussions of Processing+rssdk or of that error message if you have not already:

• https://forum.processing.org/two/search?Search=rssdk
• https://www.google.com/search?q="ToPImage"+"does+not+exist"

matt82au

Thanks. I checked on the Intel side and it seems that there may be issues with ToPImage since Processing 3. I still think its user error on my side.

To take a different approach, I can specify what type of format I am streaming my vision. The SDK has 2 other functions that might be useable. Essentially, taking the data to a string, or a float array.

If I am able to do this, would there be a way to convert this to a PImage and draw on canvas?

kfrajer

It is better if you provide more details about the nature of the output of your other two functions. Float is preferable. If the values represent color, then you can converted to the right color format used in Processing.

Kf

jeremydouglass

@matt82au --

If you have a float array that represents your camera's pixel grid then you can load this into pixels[] on the main sketch canvas or into a PImage.pixels[] or PGraphics.pixels[].

• https://processing.org/reference/pixels.html
• https://processing.org/reference/PImage.html
• https://processing.org/reference/PImage_pixels.html

Please do provide a short example sketch of what you are trying to do.

matt82au

Thank you so much for sticking with me on this one guys. I have tidied up the code below.

Line 82 shows the commented out potentially obsolete method of converting PXCImage data to PImage (unless anyone can see me doing something incorrect? The SDK has a different method at this link)

Lines 58 to 78 show the next approach I think may work if the PImage approach does not. This is converting the PXCImage to a float array. Original SDK doc is this...

The fact I can see my Image Data streaming to my "dData" Print Out surely confirms there is a way to get this data as an image on the canvas.

Again, thank you for your your thoughts. I am going through a whole bunch of trial and error trying to sort this out, especially trying to sort the Java syntax from the SDK and integrate it into the Processing sketch :)

import intel.rssdk.*;
import java.lang.System.*;
import java.util.*;
import javax.swing.*;
import java.awt.event.*;
import java.awt.image.*;
import java.awt.*;

PXCMSenseManager sm = null;
pxcmStatus sts = pxcmStatus.PXCM_STATUS_NO_ERROR;
PImage imgDepth;

void setup() {

     size(640, 480, P2D);

    //Create PXCMsession for Realsense Camera
  PXCMSession session = PXCMSession.CreateInstance();
  if (session == null) {
    System.out.print("Failed to create a session instance\n");
    System.exit(3);
  }

  // Create SenseManager for Realsense Camera
  sm = session.CreateSenseManager();
  if (sm == null) {
    print("Failed to create a SenseManager instance\n");
    return;
  }
  // Specify the camera stream. Depth 320*240 30fps
  sm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_DEPTH,320,240,30);

  PXCMCaptureManager captureMgr = sm.QueryCaptureManager();
  captureMgr.FilterByDeviceInfo("RealSense", null, 0);

  sts = sm.Init();
  if (sts.compareTo(pxcmStatus.PXCM_STATUS_NO_ERROR)>=0) {

  } else {
    print("Failed to initial a RSSDK pipeline instance.\n");
    return;

  }
  imgDepth = createImage(640, 480, RGB); 
}

  void draw()  {

    sm.AcquireFrame(true);

      PXCMCapture.Sample sample=sm.QuerySample();

        //create PXCM Image Data, as dData      
        PXCMImage.ImageData dData = new PXCMImage.ImageData(); {
            sample.depth.AcquireAccess(PXCMImage.Access.ACCESS_READ,PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32, dData);
            println(dData);

//Namespace Hierarchy             
//PXCMImage.ImageData.ToFloatArray

//Syntax#Java

//float[] ToFloatArray(int plane, int size);
//float[] ToFloatArray(int plane, float[] dest);

//Parameters

//plane - The data plane index


//size - The data array size


//dest - The application-allocated destination array.

//Description - The ToFloatArray function retrieves a copy of the image plane as a float array.

//Return Status - The float array instance, or null if there is any error.



            //imgDepth = dData.ToPImage(0, imgDepth);  - This code from existing example seems obsolote. need to get dData to PImage, by float array?
            sample.depth.ReleaseAccess(dData);
            image(imgDepth, 0,0);
    }       
  }
  // Close the Sensemanager is no more data
  void dispose() { 
    if (sm==null) return;
    sm.close();
    sm=null;
}

jeremydouglass

@matt82au -- Am I right in understanding that you are currently loading RGB32 pixel values into dData (what output does println(dData) give?) -- and that you now need to copy these values into a PImage.pixels[]?

If so, do something like this (untested):

// create image the size of the depth camera
PImage img = createImage(320, 240, RGB);

// copy image into the PImage
img.loadPixels();
for (int i = 0; i < img.pixels.length; i++) {
  img.pixels[i] = dData.pitches[i];
}
img.updatePixels();

// display image
image(img, 0, 0);    

For details on this kind of pixel array copy, see:

• https://processing.org/reference/createImage_.html
• https://processing.org/tutorials/pixels/
• https://processing.org/examples/pixelarray.html

matt82au

Thanks for your assistance. Sorry for the slow reply. I was informed by developers on the Realsense forum that Processing 3 was not supported, which is frustrating given how close seems to be. The above code caused an array out of bounds exception error. I havent pursued that much further, but Id like to think there is something else in it.

Overall, i feel as though I am hitting a bit of a dead end with the Realsense, and it seems that Intel dont plan to support it much further for Java / Processing languages :(

jeremydouglass

That is too bad -- it sounds like you should either use Processing 2.x (whatever they support) or else use different hardware.

vld

anyone found an upgrade to 3?

jeremydouglass

@vld -- are you asking if Realsense now supports Processing 3? If so you should probably ask on their developer forum....