Update

Now Processing 3 is compatible with Kinect V2, link here

Hi,

I try to create something like this : https://www.youtube.com/watch?v=YBKkVpRxUfs, using a kinect. At the moment I made some test with something like this :

class User {
  PGraphics pgBodyBg; 
  PImage bodyImg; 
  float colorIndexCounter; 
  int id; 
  ArrayList<DelayImg> bodyImageList = new ArrayList<DelayImg>();
  PGraphics pgTotal; 
  DelayImg[] buffer;

  User(int i) {
    colorIndexCounter=random(10); 
    pgTotal=createGraphics(512, 424);  
    pgTotal.beginDraw(); 
    pgTotal.background(0, 0); 
    pgTotal.endDraw(); 
    buffer=new DelayImg[50];
  }


  void delayEffect() {
    fillBuffer(); 
    changeColor(); 
    display();
  }

  void fillBuffer() {
    PImage bi= bodyImg.copy(); 
    DelayImg di=new DelayImg(bi); 
    for (int i = buffer.length-1; i > 0; i--) {
      buffer[i] = buffer[i-1];
    }
    buffer[0]=di;
  }



  void changeColor() {
    pgTotal.beginDraw(); 
    pgTotal.clear(); 
    pgTotal.pushStyle(); 
    for (int j=buffer.length-1; j>=0; j--) { 
      if (buffer[j]!=null) buffer[j].changeColor(pgTotal);
    }
    pgTotal.popStyle();
    pgTotal.endDraw();
  }

  void display() {
    image(pgTotal, 0, 0);
  }
}



class DelayImg {
  PImage img; 
  float alf=255; 
  PShape shape; 
  PGraphics pgBg; 
  int partSize=200; 
  DelayImg(PImage _img) {
    img=_img; 
    img.filter(THRESHOLD); 
    pgBg=createGraphics(512, 424);
  }

  void display(PGraphics pg) {
    pg.tint(255, alf); 
    pg.image(img, 0, 0); 
    if (alf>0)alf-=0.5;
    else alf=0;
  }

  void changeColor(PGraphics pg) {
    pgBg.beginDraw(); 
    pgBg.background(random(255), random(255), random(255));  
     pgBg.mask(img);
    pgBg.endDraw();
    pg.image(pgBg, 0, 0);
  }
}

bodyImg is update each frame, coming from kinect.

I want to be able to change the color everytime, that why pgTotal is clear every frame and display fresh new images, but the way I do it is very slow (10 fps). Do you have any suggestion, different kind of algorithm to improve frameRate? I try to use some vertex and textures aswell, but no real improvement...

Thanks for your help!

A.

im using the latest processing .. and im a mac user im facing a few problems - my 'add library ' tool does not for (not for open kinect not for other libs) so wen i downloaded open kinect from github and copied it in the lib folder. - https://github.com/shiffman/OpenKinect-for-Processing/releases I receive a wide range of errors in different examples.

mostly kinect/kinect2 class does not exist .

im assuming the name of the downloaded lib could have some naming issue . or the updated library may have diff class names .. the existing names are like those mentioned in the documentation. can anyone suggest what should i do ?

p.s. the kinect lib were working fine before .. recently i deleted all the libraries to investigate why the 'add library' tool would not work .

What a wonderful resource, @totovr76 -- your SimpleOpenni repository should be extremely helpful for people trying to put Kinect skeleton tracking into production.

Instructions

I set up the instructions in this repository

Hardware

• Kinect V1
• Kinect V2

Software

• Processing 2.2.1
• Processing 3.3.6
• Processing 3.3.7

kinect pv2 works on processing 3 on my windows machine

You save those values yourself.

You know that setup runs once and then draw runs every frame, at a rate of 30 fps.

Then:

float prevTime;
PVector prevPos;

void setup(){
  size(400,600);
  prevPosition=new PVector();
}

void draw(){

  //....Get current postion and time
  PVector currPos=new PVector(...x,...y);  //From kinect
  float currTime=millis();  //Current time

  //Process current and prev position: Calculate velocity for instance
  //...


  //Update previous position
  prevPosition=...;
  prevTime=currTime;

}

Kf

Velocity requires two differentials:

Differential w.r.t. space aka delta x, or dx for short

Differential w.r.t. time aka delta t, or dt for short

For each frame you capture the position and the time. In your code, you are getting position. For time do something like:

float timePrev;  //GLOBAL scope

timePrev=millis();  //In your draw(): Update previous time when you are done with your prev frame

so now to calculate velocity between two contiguos frames:

float dx=[pos current frame[ - [position prev frame];  //Up to you how you calculate position using your kinect 
float dt = millis() - timePrev;  
float velocity=dx/(dt/1000.0); 
println("Current velocity is "+velocity+" [your units]/seconds");

Notice I calculate velocity is in seconds. This is a pseudo-code which you can easily integrate in your code.

Kf

/** * Patterns. * * Move the cursor over the image to draw with a software tool * which responds to the speed of the mouse. */ void setup() { size(640, 360); background(102); }

void draw() { // Call the variableEllipse() method and send it the // parameters for the current mouse position // and the previous mouse position variableEllipse(mouseX, mouseY, pmouseX, pmouseY); }

// The simple method variableEllipse() was created specifically // for this program. It calculates the speed of the mouse // and draws a small ellipse if the mouse is moving slowly // and draws a large ellipse if the mouse is moving quickly

void variableEllipse(int x, int y, int px, int py) { float speed = abs(x-px) + abs(y-py); stroke(speed); ellipse(x, y, speed, speed); println(speed); }

Hello, I want to calculate the speed of a moving object as they do in the example above. However, pmouseX calculates the parameters of the previous position automatically and I am not sure how to do it manually for another object. What I want to do is to compare the x,y positions from the Kinect in order to calculate the speed of the user. I saw several examples and I think that I need to use Vectors(https://processing.org/examples/accelerationwithvectors.html) and calculate the velocity by subtracting the current point from the previous. However, as the Kinect captures 30frames per second I think that I need to create an ArrayList that gets the frames each second(with the position) and compare them with the previous. I am confused and not sure how I could implement that though. Any help would be appreciated. Thanks

Processing 3.3.7, Kinect-V1 or V2, one Mac and SimpleOpenni library

This library is working on Mac, and now it works with Kinect-V2

• Youtube

In Processing, press Ctrl+t to format your code in the PDE. Then copy and paste the code and format it in the forum (exactly as you have done above).

This could be relevant: https://forum.processing.org/two/discussion/25094/set-distance-value-with-ultrasonic-sensor#latest

Right now, what does your code do? Can you get any fading at all? Please notice we don't have the videos nor your setup (not kinect here). Can you comment on what type of kinect unit are you using?

For volume, check these:

https://forum.processing.org/two/discussion/13521/minim-help-with-volume-control
https://forum.processing.org/two/discussion/7237/how-to-set-volume-in-minim

Kf

Hello, I was able to make my sound play and stop with Kinect but it doesn't loop or fade out. I want my sound to continue playing with the video and just fade in if the interaction is activated and fade out when the interaction is no longer happening. Also, I want my sound to loop.

     import processing.sound.*;
     import org.openkinect.processing.*;
     import processing.video.*;


    Movie vid;
    Movie vid1;
    SoundFile sound1;
    SoundFile sound2;
    Kinect2 kinect2;

    //PImage depthImg;
    //PImage img1;

    //pixel
    int minDepth=0;
    int maxDepth=4500; //4.5m

    boolean off = false;

    void setup() {
      size(1920,1080);
      //fullScreen();
      vid = new Movie(this, "test_1.1.mp4");
      vid1 = new Movie(this, "test_1.1.mp4");
      sound1 = new SoundFile(this, "cosmos.mp3");
      sound2 = new SoundFile(this, "NosajThing_Distance.mp3");

      //MOVIE FILES
          //01.MOV
          //03.MOV
          //02.mov (File's too big)
          //Urban Streams.mp4
          //HiddenNumbers_KarinaLopez.mov
          //test_w-sound.mp4
          //test_1.1.mp4
          //test005.mov
      //SOUND FILES      
          //cosmos.mp3
          //NosajThing_Distance.mp3

      vid.loop();
      vid1.loop();
      kinect2 = new Kinect2(this);
      kinect2.initDepth();
      kinect2.initDevice();
    //depthImg = new PImage(kinect2.depthWidth, kinect2.depthHeight);
    //img1 = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
    }

    void movieEvent(Movie vid){
      vid.read();
      vid1.read();
    }


    void draw() { 
      vid.loadPixels();
      vid1.loadPixels();

      //image(kinect2.getDepthImage(), 0, 0);

        int[] depth = kinect2.getRawDepth();

      float sumX=0;
      float sumY=0;
      float totalPixels=0;

        for (int x = 0; x < kinect2.depthWidth; x++){
          for (int y = 0; y < kinect2.depthHeight; y++){
            int offset = x + y * kinect2.depthWidth;
            int d = depth[offset];

            if ( d > 0 && d < 1000){
          //    //video.pixels[offset] = color(255, 100, 15);
          sumX +=x;
          sumY+=y;
          totalPixels++;
             brightness(0);
            } else {
          //    //video.pixels[offset] = color(150, 250, 180);
              brightness(255);
            }      }
        }
    vid.updatePixels();
    vid1.updatePixels();

    float avgX = sumX/totalPixels;
    float avgY=sumY/totalPixels;


    //VID 01 - Screen 01
    if (avgX>300 && avgX<500){
    tint(255, (avgX)/2);
    image(vid1, 1920/2, 0);
    if(sound2.isPlaying()==0){
    sound2.play(0.5);
    sound2.amp(0.5);
    }
    }else{
    tint(0, (avgX)/2);
    image(vid1, 1920/2, 0);
    if(sound2.isPlaying()==1){
     delay(1);
    //IT DIMS THE VOLUME TO 0 BUT IT DOESN'T GO BACK TO VOLUME 0.5 [sound2.amp(0.5);]
     sound2.amp(0);
     }
    }
     //VID 02 - Screen 01
     if (avgX>50 && avgX<200){
    tint(255, (avgX)/3);
    image(vid, 0-(1920/2), 0);
    }else{
       tint(0, (avgX)/3);
       image(vid, 0-(1920/2), 0);
     }
    }

I am not very skilled in Processing, but I have been trying to convert my sketch from mouse clicks to using Kinect hand states. How do I use variables from function1 in function2, when the variables were declared using function1's arguments?

void shapes(int posx, int posy)
{
 pushMatrix();
translate(width/3,200);
 for (int i=0; i<cols; i++) {
    for (int j=0; j<rows; j++) {

      int shapeIndex = shapeNumbers[i][j];
      PShape currentShape = shapes[shapeIndex];
     currentShape.setFill(colors[i][j]);
     currentShape.setStroke(false);
     shape(currentShape,i*posx,j*posy,shapesize,shapesize);

    if (mousePressed ==true){
    int  x = i*posx+width/3;
     int y = j*posy+200;
     if (mouseX > x && mouseX < (x + shapesize) && mouseY > y && mouseY < (y + shapesize)){
      colors[i][j]=color(255);
    }
    }
    }
  }
    popMatrix();
}

void shapes(int posx, int posy){
 pushMatrix();
  translate(width/3,200);
 for (int i=0; i<cols; i++) {
    for (int j=0; j<rows; j++) {

      int shapeIndex = shapeNumbers[i][j];
PShape currentShape = shapes[shapeIndex];
     currentShape.setFill(colors[i][j]);
     currentShape.setStroke(false);
    shape(currentShape,i*posx,j*posy,shapesize,shapesize);

    if (mousePressed ==true){
    int  x = i*posx+width/3;
     int y = j*posy+200;
     if (mouseX > x && mouseX < (x + shapesize) && mouseY > y && mouseY < (y + shapesize)){
      colors[i][j]=color(255);
    }
    }
    }
  }
    popMatrix();
}

We were able to figure it out, thank you for the help! The usermap can be indexed to let us know the userid and from there we can just mess around with the pixels

  rgbImage = kinect.rgbImage();
   userMap = kinect.userMap();
   for (int i = 0; i < userMap.length; i++)

   {
     if (userMap[i] == 1)
     {
    rgbImage.pixels[i]=color(random(255) );
     }
   else if (userMap[i] == 2)
   {
     rgbImage.pixels[i] = color(255,0,0);
   }
   }

     rgbImage.updatePixels();   
     image(rgbImage, 0, 0);

Not an expert in kinect but I have few questions:

Do you detect when a new user enter or leaves the scene? Do you get your messages from onNewUser()/onLostUser()?

What documentation are you using for kinect? The question is to relate kinect.userMap() to each user as stored in kinect.getUsers(). I could suggest a solution by implementing your own userID assignation directly on your userMap pixel data (aka. detect your own users).... but no need to re-invent the wheel if it is already done for you by the library.

Also what kinect device are you using? I could help me if you provide the specs.

Kf

Hi everyone,

I'm trying to make a processing program where each person (silhouette) will be assigned a different glitch. Essentially something like this

Right now we can only assign the same glitch for each person.

This is our code:

    class User
    {
      int userid; 
      int glitchid;

      User(SimpleOpenNI curContext, int id)
      {
        glitchid = (int)random(1,2);
        userid = id;
      }

      User()
      {
      }
    }

    import processing.opengl.*;
    import SimpleOpenNI.*;

    SimpleOpenNI  kinect;
    int userID;
    int[] userMap;
    int randomNum; 
    float red;
    float green;
    float blue;
    PImage rgbImage;

    int count;
    ArrayList <User> users = new ArrayList <User> ();                              

    void setup() {
      size(640, 480, P3D);
      kinect = new SimpleOpenNI(this);
      kinect.setMirror(true);
      kinect.enableDepth();
      kinect.enableUser();
      kinect.enableRGB();
    }

    void draw() {  
      kinect.update();  
      rgbImage = kinect.rgbImage();
      image(rgbImage, 0, 0);

     int[] userList = kinect.getUsers(); 
      for(int i=0;i<userList.length;i++)
      {
       int glitch =  users.get(i).glitchid;
       if (glitch == 1)
       {
         crt();
       }
       else
      {
        crtcolor();
      }
      }

      stroke(100); 
      smooth();
    }

    void onNewUser(SimpleOpenNI curContext, int userId)
    {
      println("onNewUser - userId: " + userId);
      User u = new User(curContext, userId); 
      users.add(u); 
      println("glitchid " + u.glitchid); 
      curContext.startTrackingSkeleton(userId);
    }

    void onLostUser(SimpleOpenNI curContext, int userId)
    {
      println("onLostUser - userId: " + userId);
    }


    void crtcolor()
    {
      color randomColor = color(random(255), random(255), random(255), 255);
      boolean flag = false;
      if (kinect.getNumberOfUsers() > 0) {  
        userMap = kinect.userMap();   
        loadPixels();
        for (int x = 0; x < width; x++) {
          for (int y = 0; y < height; y++) {
            int loc = x + y * width;
            if (userMap[loc] !=0) { 
              if (random(100) < 50 || (flag == true && random(100) < 80))
              {
                flag = true;
                color pixelColor = pixels[loc];
                float mixPercentage = .5 + random(50)/100;
                pixels[loc] =  lerpColor(pixelColor, randomColor, mixPercentage);
              } else
              {
                flag = false;
                randomColor = color(random(255), random(255), random(255), 255);
              }
            }
          }
        }   
        updatePixels();
      }
    }

    void crt()
    {
      if (kinect.getNumberOfUsers() > 0) {  
        userMap = kinect.userMap();   
        loadPixels();
        for (int x = 0; x < width; x++) {
          for (int y = 0; y < height; y++) {
            int loc = x + y * width;
            if (userMap[loc] !=0) { 
              pixels[loc] = color(random(255) );
            }
          }
        }
        updatePixels();
      }
    }

I know that usermap will give me all the pixels and in my crt/crtcolor code i am changing all the pixels which is why it is not working. Is there a way/function that will give me the pixels associated with each user instead?

I answer to myself : after a very long day trying a lot of very old stuff I mangaged to get the skeleton using Kinect4winSDK, with a kinect model 1414 and Microsoft Kinect SDK 1.8 on Win 10. I'm pretty sure I tried this before but for some reason it didn't work until that time.

Years ago I managed to use Processing (1.5 ? 2 ? I can't say) + one of the kinect librairies (maybe SimpleOpenNi, that doesn't seem to ba available anymore) in order to get the joints of a skeleton. It worked fine at that time. This year, nothing works at all, or quite : I'm able to get a cloud of points with openKinect, but not on any operating system, just on Ubuntu (I also use Windows 7 and Windows 10). I'm quite lost, I tried many old drivers/sdk installations etc.

So my question is : can somebody who actually (and recently) managed to get a skeletton tell me : - which operating system was used - which version of Processing - which processing library (SimpleOpenNi, Kinect4WinSDK, OpenKinect, Kinect V2) - which driver or sdk was used

It would be a huge help.

You need to install the blob detector and run the provided examples.

Second: Can you get image from your kinect unit? You can check previous posts to get started: https://forum.processing.org/two/search?Search=kinect

Get the different pieces working by themselves. When you have a good understanding how each pieces works and performs, then you can think in bring them together.

Kf