I'm working on a sketch that takes a movie (in this example an image sequence), extrudes each frame, and stacks them. Each pixel becomes a cube of the same color; 'background' pixels are keyed out. Right now, it's static, but eventually I would love to have it be dynamic, loading new frames on top of the stack. This sketch is a test run, using 50 frames of a 100x100px video. (Note the weird moire pattern is due to my laziness in keying out the 'background' pixels)

Performance is dead slow (even on my Mac Pro), which I kind of expected: I'm rendering between 200 000 and 500 000 cubes! I've read up on some possible methods to speed things up, and I would love some feedback on which might be most effective.

1) Implement Vertex Buffer Objects
2) Create a custom cube class based on rects rather than use Processing's box() class
3) Avoid rendering unnecessary cubes (how?)
4) Implement a 'preview' view using something like points, then show the cubes when the camera comes to rest
5) Instead of rending individual cubes for each frame 'layer', render an object that aggregates their geometry and texture it with the pixel array. (Note: I would need to keep the cubes array around in memory for processing, but this might be a solution for rendering)

Separate question: what's the best tool for profiling OpenGL use in a processing app?

I would really appreciate any advice on which of these (if any) would be the best place to start. If anybody has other ideas, please post them! Thanks....

Applet here (http://www.ingenuepictures.com/pixelextrusion/index.html), code below. The applet's ~3megs and may take a minute to load.


__________________________

import processing.video.*;

import peasy.org.apache.commons.math.*;
import peasy.*;
import peasy.org.apache.commons.math.geometry.*;

import processing.opengl.*;

PeasyCam cam;
PImage img;
int imgHeight;
int imgWidth;
Cube[][][] cubes = new Cube[140][287][51];
int globalCubeRadius;
color black;
color white;
PImage[] imgs=new PImage[51];

void setup() {
 size(400,400, OPENGL);
 background(0);
 lights();
 noFill();
 noStroke();
 //frameRate(30);
 for(int i=1;i<52;i++){
   imgs[i-1]=loadImage("testmov"+i+".png");
 }
 img = imgs[0];
 imgHeight=img.height;
 imgWidth=img.width;
 globalCubeRadius=5;
 cam = new PeasyCam(this,imgWidth*5/2,imgHeight*5/2, 0,700);
 cam.setMinimumDistance(60);
 cam.setMaximumDistance(1000);
 black=#000000;
 white=#d9d9d9;
 
 for(int h=0; h<51;h++){
   img=imgs[h];
   for(int i=0;i<imgWidth;i++){
     for(int j=0;j<imgHeight;j++){
       if(img.pixels[j*imgWidth+i]>=white){
         cubes[i][j][h]=new Cube(0,0,0,globalCubeRadius);
       }else{
         cubes[i][j][h]=new Cube(0,0,0,globalCubeRadius,img.pixels[j*imgWidth+i]);
       }
     }
   }
 }
   
}

void draw() {
 background(0);
 for(int h=0;h<51;h++){
   pushMatrix();
   translate(0,0,h*globalCubeRadius);
   for(int j=0;j<imgHeight;j++){
     pushMatrix();
     translate(-globalCubeRadius,j*globalCubeRadius,0);
     for(int i=0;i<imgWidth;i++){  
       translate(globalCubeRadius,0,0);
       cubes[i][j][h].drawObj();
     }
     popMatrix();
   }
 popMatrix();
 }
}


class Cube {
 float centerX, centerY, centerZ;
 int radius;
 color col;
 Boolean noColor;
 
 Cube(float cubeCenterX, float cubeCenterY, float cubeCenterZ, int cubeRadius, color cubeCol){
   centerX=cubeCenterX;
   centerY= cubeCenterY;
   centerZ=cubeCenterZ;
   radius=cubeRadius;
   col=cubeCol;
   noColor=false;
 }
 
 Cube(float cubeCenterX, float cubeCenterY, float cubeCenterZ, int cubeRadius){
   centerX=cubeCenterX;
   centerY= cubeCenterY;
   centerZ=cubeCenterZ;
   radius=cubeRadius;
   noColor=true;
 }
 
 void drawObj() {
   if(cubeExists()==true){
     fill(col);
     box(radius);
   }  
 }
 
 Boolean cubeExists(){
   if(noColor==false){
     return true;
   }else{
     return false;
   }
 }
}

Thanks, that's helpful. I do want to update the geometry every frame eventually so it sounds like a VBO in STREAM_DRAW argument would be most appropriate, but I might work my way up from Display lists for the exercise.

How would you profile an OpenGL-based applet?

Thanks again...

Hi Josn,

Thanks so much for all the detail. I'm going to summarize just to make sure I understand your suggestions. I have a few questions as well...

1) Precalculate the absolute positions of the cubes instead of using translate() and matrix stack calls. What data structure would you suggest for the lookup table?

2) When you say use one-dimensional arrays, do you mean to store the entire voxel-grid? For example: for a 10x10x10 space, I have one 1D array with 1000 elements, linked to a lookup table which has 4 values for each element (float x/y/z, colour)?

3) OK, so your while() loop is to iterate through a 1D array; what's the idea of the for() loop? I don't quite understand...

3) I probably won't implement visibility detection as I've found I also like the representation in points or as a 3d grid (ie just drawing the strokes of the cubes).

4) I'll actually be using quads + points to create an isometric view as well.

If anybody else is doing this sort of volumetric stuff, there are some techniques from medical imaging that are proving helpful. Subsampling the volumetric data especially; also (with points only) drawing the points as textures on layered quads.

Thanks again Josn for taking the time!

Wow, that's pretty stunning. Thanks Kaeltyk.