making a particle emitter from kinect2 depth data

sirbrad

Hey there.

I'm trying to make a particle emitter out of kinect2 depth data, for a dance piece. Essentially what I want to do is extract the body shape, and make that the particle emitter.

What I currently have coded is, crude, but working, but also killing my machine (a mid 2012 Macbook pro, 2.9GHz processor, 8 GB 1600 MHz DDR3 memory) and I am yet to add in extra detail.

My code essentially does this - gets the kinect depth raw data, loops through the data if the pixel is in the depth range specified, its counted as an emitter and added to an array list.

These pixel coordinates are then checked against an arraylist of particles. If there is no particle at that point, then add one to the arrayList.

THEN, i want to start the particle decaying if it is not in the emitter area, so I loop again through my array of particles,

if particle is not in my array of emiter locations - then do things.

THEN loop through and update and draw the particles

Can some one offer advice as to how this can be optimised? Code is below in all its seriously messy glory.

import org.openkinect.processing.*; 


Kinect2 kinect2;
PImage img;

float RANGE_MIN = 1000;
float RANGE_MAX = 2000;
int KINECT_WIDTH= 515;
int KINECT_HEIGHT = 424;

// define all the variables
ParticleEmitter particleEmitter;
//constants - colors
color BACKGROUND = color(0, 0, 0, 100);

// set size.
void setup() {
  size(512,424);
  kinect2 = new Kinect2(this);
  //initialise the depth feed
  kinect2.initDepth();
  // initialise the whole device.
  kinect2.initDevice();

  particleEmitter = new ParticleEmitter(kinect2);

}

void draw(){
  background(BACKGROUND);

  //particleEmitter.display();
  //fill(255);
  //ellipse(10, 10, 1, 1);
  particleEmitter.refresh();
  particleEmitter.emit();
}

// -----------------
// ParticleEmitter class  
// - will eventually read in data from the kinect and extract the performers body shape 
// - using this as the emitter.
// -----------------
import java.util.Iterator;

class ParticleEmitter {
  PImage img;

  //constants
  int PARTICLE_DISTANCE = 5; // distance between particles in the emitter.

  //particles
  ArrayList<Particle> particles = new ArrayList<Particle>() ;
  ArrayList<PVector> emitter = new ArrayList<PVector>() ;

  //kinect
  Kinect2 kinect;

  // ----
  //constructor 
  // ----
  ParticleEmitter(Kinect2 _kinect){
    kinect = _kinect;
    //create a blank image to put all the depth info onto.
     img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
  }


  // ----
  // Refresh
  //
  // clears the arraylist of locations where particles can spawn from
  // creates new array list of locations where particles can spawn from
  // ----
  void refresh(){
    emitter.clear();
     img.loadPixels(); //need to operate on the pixels of the image. 
                    //goes hand in hand with image.updatePixels();

    //get depth data from the kinect
    int [] depth = kinect2.getRawDepth();  //this is an array of ints relating to pixels in the depth image.
                                         // 0 - 4500 --> 0 relating to distance in mm;
    //first - establish the locations where particles can grow
    for (int x = 0; x < kinect2.depthWidth; x++){
      for(int y = 0; y < kinect2.depthHeight; y++){
        int offset  = x + (y * kinect2.depthWidth);
        int d = depth[offset];
        if (d > RANGE_MIN && d < RANGE_MAX){
          PVector location = new PVector(x, y);
          emitter.add(location);  
          //does it exist already?
          if (!has(location)){
            float temp = random(0,90000);
            if (temp > 89990){
              Particle newParticle = new Particle(location);
              particles.add(newParticle);
            }
          }
        } 
      } 
    } 
  }

  // ----
  // emit
  //
  // updates the particles. 
  // for each particle, if it is not in the emitter area, then begin its decomposition
  // if it is in the emitter area - do nothing
  // 
  void emit(){
    Iterator <Particle> it = particles.iterator();
    while (it.hasNext()) {
      Particle p = it.next();
      //loop through emitter area - is it in or out?
      boolean kill = true;
      for (PVector emitLocation: emitter){
        //if theres a match make it false
        if (p.location.dist(emitLocation) < PARTICLE_DISTANCE){
          kill = false;
        }
      }
      p.kill(kill);
      p.run();
      if (p.isDead()) {
        it.remove();
      }

    }
  }
  // ---
  // Has
  //
  // checks to see if a particle exists within this location
  // ---
  boolean has(PVector location){
    boolean has = false;
    for (Particle particle : particles){
        if(location.dist(particle.location) < PARTICLE_DISTANCE){
          has = true;
        }
    }
    return has;
  }
}  

// ----------
// Particle class 
//
// defines the individual particles.
// methods for behaviour.
// 
// ----------
class Particle {
  //constants
  float MAX_LIFESPAN = 255;
  color PARTICLE_COLOR = color(255,255,255);

  // force and direction 
  PVector location;
  PVector velocity;
  PVector acceleration;

  //lifespan and activity
  boolean dying;
  float lifespan;
  float mass; //may or may not be used...




  // --- 
  // constructor
  // ---
  Particle(PVector _location){
    location = _location;

    acceleration = new PVector(0,0.1);//downward
    velocity = new PVector(random(-1,1),random(-1,2));

    dying = false;
    //lifespan = RESET_LIFESPAN;
    lifespan = 10.0;
  }

  void run(){
    update();
    display();
  }

  void update(){
    println(lifespan);
    if (dying){
      velocity.add(acceleration);
      location.add(velocity);
      lifespan -= 2.0;
    }
    else{
      if (lifespan < MAX_LIFESPAN){
        lifespan += 4.0;
      }
    }
  }

  void display(){
    noStroke();
    fill(PARTICLE_COLOR, lifespan);
    ellipse(location.x, location.y, 3, 3);
  }

  boolean isDead() {
    if (lifespan < 0) {
      return true;
    } else {
      return false;
    }
  }

  void kill(boolean _dying){
    dying = _dying;    
  }

}  

jeremydouglass

@sirBrad --

Brainstorm:

You could drop your max particle count. Have you tried simply lowering the emitter resolution and seeing what the performance impact is? E.g.

for (int x = 0; x < kinect2.depthWidth; x=x+3){
  for(int y = 0; y < kinect2.depthHeight; y=y+3){

...also, have you tried specifying a lower target frameRate() in setup -- e.g. 30?

You could also separate the create, update, and remove cycles for particles into different draw frames -- e.g. update existing particle locations every frame, but only check for new particles on frames 0, 4, 8 and only check to remove existing ones on frames 2, 6, 10 ...

OR just run things on stronger hardware! Get a loaner, load the sketch, and see if a processor from the past 2-3 years or a desktop machine solves all your problems without refactoring. If it does, great -- if not, at least you know that refactoring is simply required.

sirbrad

Great ideas thanks! I'll give it all a try and post back here.- and yeah, I am strongly considering upgrading the hardware :)