Looks great!

Thanks, it's coded using processing-py which is simply a python implementation of the processing environment (A good blog post on how to set it up here: http://bit.ly/KBNxx2 ) Here's the python code for now anyway, I have commented most of the lines of code, so even if you don't know python it shouldn't be that hard to understand/recreate in java if you wanted:

1. #image input = 480x270
2. #cropped image (without border) = 480x192
3. count = 0
4. def setup():
5.         size(1280, 720, P3D)
6.         background(255)
7.         sphereDetail(3)
8.         lights()
9.        
10. def draw():
11.         #centres the drawing
12.         rotateX(0.12)
13.         translate(width/3 , (height-192)/2, -100)
14.    
15.         background(255)
16.        
17.         frame = imageUpdate() #read in the frame to be used
18.        
19.         pointMap(frame)
20.        
21.         screenshot() #saves an image of the current canvas
22. 
    
23. #Draws the output/3d spheres based on the frame given
24. def pointMap(frame):
25.         frame.loadPixels() #load the pixels of the frame
26.        
27.         for x in range(frame.width):
28.                 for y in range(39, 231): #altered slightly to get rid of the black top and bottom
29.                         pixelColour = frame.get(x, y) #get pixel colour
30.                         fill(pixelColour, 50) #change the fill and stroke to the pixel colour
31.                         stroke(pixelColour)
32.                         if (x%4)==0 and (y%4)==0: #only use every 4th pixel on x and y axis
33.                                 pushMatrix()
34.                                 #move sphere according to x and y coordindates
    
35.                                 #and the brightness of the colour
36.                                 translate(x, y, 2*(int(brightness(pixelColour))))
    
37.                                 sphere(3)
38.                                 popMatrix()
39. 
    
40. #returns the next frame to be drawn
41. def imageUpdate():
42.         fileName = 'Y/Y ' #Location and beginning of the file name
43.         fileNumber = str(frameCount%5817) #Frame number       
44.         fileNumber= fileNumber.zfill(4) #Padding the digits to 4 digits
45.         fileName = fileName + fileNumber + '.jpg'
46.         return loadImage(fileName)
47. 
    
48. #Saves an image of each frame
49. def screenshot():
50.     global count
51.     filename="720p\screen "+ str(count) +".png"
52.     save(filename)
53.     count+=1

Note: The original videos frames were stored in a folder Y in the same directory as the sketch and named Y0000, Y0001, Y0002


I used the P3D renderer.

The basic algorithm is simply read in the image/frame, load the pixels of the frame into a pixelarray, every 4th pixel get the colour and draw a sphere with the same colour, it's position depends on the x and y coordinate of the original pixel it's based on and it's z coordinate depends on the brightness of the colour, then save the canvas into an image.

So like you said it's not in real time as I found I wasn't able to calculate everything fast enough, definitely one of the limitations of python, so it's based on the saved frames and I have rerendered all of the output frames together using a movie editing software. As a result it also meant that I didn't have to use a video library and simply used the PImage class that python provides.