How can I stream my Kinect feed from one computer to another with OSC?

KMat

Hi all, this is my first post here

I am trying to feed my slightly tweaked PointCloud example from Open Kinect v1 library from one computer to another (for example sake, from one processing file to be displayed in another) through the OSC library.

I have attempted myself, but was unable to send the depth data properly (at all) and I am just lost as to where to start and how to even send the data to be displayed.

Here is my Kinect code (without any OSC), can anyone help or guide me through what to send/receive? (I'm on Mac OS 10.13 and Processing 3.0.1)

import org.openkinect.freenect.*;
import org.openkinect.processing.*;

// Kinect Library object
Kinect kinect;

// Angle for rotation
float a = 0;

// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[750];

void setup() {
  // Rendering in P3D
  size(800, 600, P3D);
  kinect = new Kinect(this);
  kinect.initDepth();

  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }
}

void draw() {

  background(0);

  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepth();

  // We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
  int skip = 4; //

  // Translate and rotate
  translate(width/2, height/2, 300); //dot distance
  //rotateY(a);

  for (int x = 0; x < kinect.width; x += skip) {
    for (int y = 0; y < kinect.height; y += skip) {
      int offset = x + y*kinect.width;

      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      PVector v = depthToWorld(x, y, rawDepth);

      stroke(255, 0, 0);
      pushMatrix();
      float factor = 400; //overall Scale
      translate(v.x*factor, v.y*factor, factor-v.z*factor);
      // Draw a point
      point(0, 0);
      //line(0,0,2,2);
      popMatrix();
    }
  }

  // Rotate
  //a += 0.015f;
}

// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
  if (depthValue < 750) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

kfrajer

Have you run your oscP5 provided examples to verify you can send data between computers? Examples at: http://www.sojamo.de/libraries/oscP5/#examples

I have attempted myself, but was unable to send the depth data properly

Can you show your code, what did you try? Did you get any error at all?

Kf

KMat

@kfrajer I can send data between two computers (I have a PONG game osc example where theres a controller on one computer and the game on another- also what the setup of my current code is based off of).

My apologies-by not being able to send depth data, I meant I'm not entirely sure how to send and declare the data. It is in an int[], which I'm able to send, but not sure how to declare the data on the receiver.

Below is my attempt of doing so

Im so sorry if this is messy or not properly formatted If this is too much to interpret, its understandable.

Here is the source code, "hello" is the test variable I'm trying to send

    import netP5.*;
    import oscP5.*;
    import org.openkinect.freenect.*;
    import org.openkinect.processing.*;

    OscP5 osc;

    int myListeningPort = 12001;

    // Someone to talk to
    NetAddress destination;
    String destinationIP = "127.0.0.1"; // pong IP
    int destinationPort =   12000;        // pong port

    String playerAddressPattern = "/";

    // Kinect Library object
    Kinect kinect;

    float[] depthLookUp = new float[750];

    int[] hello; //Declaring the variable to send

    void setup() {
      // Rendering in P3D
      size(200, 200, P3D);

      kinect = new Kinect(this);
      kinect.initDepth();

      // Lookup table for all possible depth values (0 - 2047)
      osc = new OscP5(this, myListeningPort);

      destination = new NetAddress(destinationIP, destinationPort);
    }

    void draw() {
      int[] depth = kinect.getRawDepth();

      hello = depth;
    }

    void OscEvent(OscMessage theOscMessage) {
      //create a message with a unique address pattern
      OscMessage myOutGoingMessage = new OscMessage( playerAddressPattern );  
      //myOutGoingMessage.add("hi");
      myOutGoingMessage.add(hello); //send the depth data (as an int string)

      osc.send( myOutGoingMessage, destination );  // actually do the sending
    }

This is the Receiver, receiving the depth data. The second last line of code is what I am assuming the issue is - in not properly declaring the incoming int[] as an int[]. I know that I am currently trying to declare it as a regular int.

import oscP5.*;
import netP5.*;
import org.openkinect.freenect.*;
import org.openkinect.processing.*;

// Kinect Library object
Kinect kinect;

OscP5 oscP5;  // osc object
OscMessage currentMessage;

int iNet_myListeningPort  = 12000;  // port I am listening on 

float a = 0;

// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[750];

int[] hello;//Declare the depth that is coming in from the kinect. 


void setup() {

  size(800, 600, P3D);
  background(0);

  // This needs to be sent from pointcloud
  //kinect = new Kinect(this);
  kinect.initDepth();

  //kinect.update();

  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }

  oscP5 = new OscP5(this, iNet_myListeningPort);
}

void draw() {
  background(0);
  int[] depth = hello; // the actual depth values from the remote kinect. 

  // We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
  int skip = 4; //


  translate(width/2, height/2, 300); //dot distance

  for (int x = 0; x < kinect.width; x += skip) {
    for (int y = 0; y < kinect.height; y += skip) {
      int offset = x + y*kinect.width;

      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      PVector v = depthToWorld(x, y, rawDepth);

      stroke(255, 0, 0);
      pushMatrix();
      float factor = 400; //overall Scale
      translate(v.x*factor, v.y*factor, factor-v.z*factor);
      // Draw a point
      point(0, 0);
      popMatrix();
    }
  }
}

float rawDepthToMeters(int depthValue) {
  if (depthValue < 750) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

void OscEvent (OscMessage theOscMessage) {
  hello=theOscMessage.get(0).intvalue(); //the value being recieved is an int[], not an int- how do i declare this?
}

This is my first time using Kinect and only my second time using oscP5 (with my first oscP5 work only requiring me to change the remote) so I really put myself in an awkward position here lol.

kfrajer

Continue here (?) https://forum.processing.org/two/discussion/25184/how-do-i-send-kinect-v1-depth-data-over-oscp5-from-one-computer-to-another#latest

Kf