How can I stream my Kinect feed from one computer to another with OSC?

edited November 2017 in Kinect

Hi all, this is my first post here

I am trying to feed my slightly tweaked PointCloud example from Open Kinect v1 library from one computer to another (for example sake, from one processing file to be displayed in another) through the OSC library.

I have attempted myself, but was unable to send the depth data properly (at all) and I am just lost as to where to start and how to even send the data to be displayed.

Here is my Kinect code (without any OSC), can anyone help or guide me through what to send/receive? (I'm on Mac OS 10.13 and Processing 3.0.1)

import org.openkinect.freenect.*;
import org.openkinect.processing.*;

// Kinect Library object
Kinect kinect;

// Angle for rotation
float a = 0;

// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[750];

void setup() {
  // Rendering in P3D
  size(800, 600, P3D);
  kinect = new Kinect(this);
  kinect.initDepth();

  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }
}

void draw() {

  background(0);

  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepth();

  // We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
  int skip = 4; //

  // Translate and rotate
  translate(width/2, height/2, 300); //dot distance
  //rotateY(a);

  for (int x = 0; x < kinect.width; x += skip) {
    for (int y = 0; y < kinect.height; y += skip) {
      int offset = x + y*kinect.width;

      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      PVector v = depthToWorld(x, y, rawDepth);

      stroke(255, 0, 0);
      pushMatrix();
      float factor = 400; //overall Scale
      translate(v.x*factor, v.y*factor, factor-v.z*factor);
      // Draw a point
      point(0, 0);
      //line(0,0,2,2);
      popMatrix();
    }
  }

  // Rotate
  //a += 0.015f;
}

// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
  if (depthValue < 750) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

Answers

  • Have you run your oscP5 provided examples to verify you can send data between computers? Examples at: http://www.sojamo.de/libraries/oscP5/#examples

    I have attempted myself, but was unable to send the depth data properly

    Can you show your code, what did you try? Did you get any error at all?

    Kf

  • edited November 2017

    @kfrajer I can send data between two computers (I have a PONG game osc example where theres a controller on one computer and the game on another- also what the setup of my current code is based off of).

    My apologies-by not being able to send depth data, I meant I'm not entirely sure how to send and declare the data. It is in an int[], which I'm able to send, but not sure how to declare the data on the receiver.

    Below is my attempt of doing so

    Im so sorry if this is messy or not properly formatted If this is too much to interpret, its understandable.

    Here is the source code, "hello" is the test variable I'm trying to send

        import netP5.*;
        import oscP5.*;
        import org.openkinect.freenect.*;
        import org.openkinect.processing.*;
    
        OscP5 osc;
    
        int myListeningPort = 12001;
    
        // Someone to talk to
        NetAddress destination;
        String destinationIP = "127.0.0.1"; // pong IP
        int destinationPort =   12000;        // pong port
    
        String playerAddressPattern = "/";
    
        // Kinect Library object
        Kinect kinect;
    
        float[] depthLookUp = new float[750];
    
        int[] hello; //Declaring the variable to send
    
        void setup() {
          // Rendering in P3D
          size(200, 200, P3D);
    
          kinect = new Kinect(this);
          kinect.initDepth();
    
          // Lookup table for all possible depth values (0 - 2047)
          osc = new OscP5(this, myListeningPort);
    
          destination = new NetAddress(destinationIP, destinationPort);
        }
    
        void draw() {
          int[] depth = kinect.getRawDepth();
    
          hello = depth;
        }
    
        void OscEvent(OscMessage theOscMessage) {
          //create a message with a unique address pattern
          OscMessage myOutGoingMessage = new OscMessage( playerAddressPattern );  
          //myOutGoingMessage.add("hi");
          myOutGoingMessage.add(hello); //send the depth data (as an int string)
    
          osc.send( myOutGoingMessage, destination );  // actually do the sending
        }
    

    This is the Receiver, receiving the depth data. The second last line of code is what I am assuming the issue is - in not properly declaring the incoming int[] as an int[]. I know that I am currently trying to declare it as a regular int.

    import oscP5.*;
    import netP5.*;
    import org.openkinect.freenect.*;
    import org.openkinect.processing.*;
    
    // Kinect Library object
    Kinect kinect;
    
    OscP5 oscP5;  // osc object
    OscMessage currentMessage;
    
    int iNet_myListeningPort  = 12000;  // port I am listening on 
    
    float a = 0;
    
    // We'll use a lookup table so that we don't have to repeat the math over and over
    float[] depthLookUp = new float[750];
    
    int[] hello;//Declare the depth that is coming in from the kinect. 
    
    
    void setup() {
    
      size(800, 600, P3D);
      background(0);
    
      // This needs to be sent from pointcloud
      //kinect = new Kinect(this);
      kinect.initDepth();
    
      //kinect.update();
    
      for (int i = 0; i < depthLookUp.length; i++) {
        depthLookUp[i] = rawDepthToMeters(i);
      }
    
      oscP5 = new OscP5(this, iNet_myListeningPort);
    }
    
    void draw() {
      background(0);
      int[] depth = hello; // the actual depth values from the remote kinect. 
    
      // We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
      int skip = 4; //
    
    
      translate(width/2, height/2, 300); //dot distance
    
      for (int x = 0; x < kinect.width; x += skip) {
        for (int y = 0; y < kinect.height; y += skip) {
          int offset = x + y*kinect.width;
    
          // Convert kinect data to world xyz coordinate
          int rawDepth = depth[offset];
          PVector v = depthToWorld(x, y, rawDepth);
    
          stroke(255, 0, 0);
          pushMatrix();
          float factor = 400; //overall Scale
          translate(v.x*factor, v.y*factor, factor-v.z*factor);
          // Draw a point
          point(0, 0);
          popMatrix();
        }
      }
    }
    
    float rawDepthToMeters(int depthValue) {
      if (depthValue < 750) {
        return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
      }
      return 0.0f;
    }
    
    PVector depthToWorld(int x, int y, int depthValue) {
    
      final double fx_d = 1.0 / 5.9421434211923247e+02;
      final double fy_d = 1.0 / 5.9104053696870778e+02;
      final double cx_d = 3.3930780975300314e+02;
      final double cy_d = 2.4273913761751615e+02;
    
      PVector result = new PVector();
      double depth =  rawDepthToMeters(depthValue);
      result.x = (float)((x - cx_d) * depth * fx_d);
      result.y = (float)((y - cy_d) * depth * fy_d);
      result.z = (float)(depth);
      return result;
    }
    
    void OscEvent (OscMessage theOscMessage) {
      hello=theOscMessage.get(0).intvalue(); //the value being recieved is an int[], not an int- how do i declare this?
    }
    

    This is my first time using Kinect and only my second time using oscP5 (with my first oscP5 work only requiring me to change the remote) so I really put myself in an awkward position here lol.

This discussion has been closed.