I am trying to figure out the best way to map projected images to a screen placed at slight (~5-10 degree) angle to the projector. I hope to be able to project onto multiple flat but angled screens from the same sketch (using a Mac Pro with two attached projectors and one long, multi-screen sketch the graphics will be blobs pulled from camera streams using OpenCV).

The basic effect I'm looking for is something like this: http://www.vimeo.com/806183 - I even took a look at OpenFrameworks to take advantage of that code, but I don't have long to get moving and I'm comfortable in Processing.

For speed using OpenGL seems like a good bet. I have tried two approaches with some success, but neither is quite right yet:

1) Apply the video data as a texture to a single OpenGL quad in a flat Z plane (ie using it as a 2D shape) with the vertices placed to match the corners of the screen in the projected image.

This seems like it would be a great and simple to calibrate solution, if only I could get the texture to map ok. I am experiencing the problem described here https://developer.sonyericsson.com/site/global/techsupport/tipstrickscode/java/p
_java_0801.jsp

My next step is to try mapping to multiple triangles (or one triangle strip). However I haven't figured out how to calculate the U, V texture coordinates on the fly as the corners of the screen surface are adjusted.

2) use a quad with parallel sides to avoid the texture mapping flaws and then manually shift/rotate the quad in 3D space to map projection to screen.

This seems to work ok, at least the texture looks good. The problem is that it's extremely counter-intuitive to try and get the right position to map to the screen, tonnes of trial and error and I'm still not sure if all screen shapes/positions in my setup can be accommodated this way.

Hopefully this is clear, I'm brand-new to OpenGL and still getting a grip on it. Any suggestions would be much appreciated, OpenGL based or not. I'm surprised that I haven't been able to find examples in Processing, this seems like a common VJ trick so I'm sure someone is doing it.

Thank you, Patrick

Hi Memo
Cool, I came across your video as I was researching this (http://vimeo.com/2076022?pg=embed&sec=2076022). If I could recreate that in Processing it would be awesome.

Any pointers on how to to convert the code to Processing? Looking at your code I'm rather lost and I can't open the Quartz patch (I assume because I'm on Tiger).

Thank you, Patrick

Thank you for your help Memo. It's appreciated and I am much closer that I was.

Using your GLSL approach I can apply my texture image to a quad and then warp it much more cleanly. However the geometry of the warping is not working as expected. There's a lot of code here, though I tried to cut it down to the minimum to show the problem I'm having.

I've been bashing my head against this for hours. I suspect that there may just be a simple problem that I'm missing as an OpenGL novice. Any suggestions would be very welcome.



Complete sketch is online at http://hogtownconsulting.com/image_hosting/shadowd_dev.zip

I'm using JohnG's GLSL class (first response in the thread at http://processing.org/discourse/yabb_beta/YaBB.cgi?board=OpenGL;action=display;num=1159494801;) with one method of my own:

Code:

void setUniformValue2f(int uniformLocation, float v0, float v1) {


gl.glUniform2f(uniformLocation, v0, v1);

}
 

My quadwarp.vert file contains this:

Code:

uniform vec2 BL, BR, TL, TR;
uniform vec2 renderSize;

void main() {

// transform from Processing coords to 0...1
vec2 p = vec2(gl_Vertex.x/renderSize.x, gl_Vertex.y/renderSize.y);
 
// interpolate bottom edge x coordinate
vec2 x1 = mix(BL, BR, p.x);
 
// interpolate top edge x coordinate
vec2 x2 = mix(TL, TR, p.x);

// interpolate y position
p = mix(x1, x2, p.y);

// transform from 0...1 to Processing screen coords
p = vec2(p.x*renderSize.x, p.y*renderSize.y);
 
gl_Position 
= gl_ModelViewProjectionMatrix * vec4(p, 0, 1);

gl_FrontColor
= gl_Color;

gl_TexCoord[0]
= gl_TextureMatrix[0] * gl_MultiTexCoord0;
}
 

And the sketch itself looks like this:

Code:

import javax.media.opengl.GL;
import javax.media.opengl.glu.GLU;
import com.sun.opengl.util.BufferUtil;
import java.nio.*;
import processing.opengl.PGraphicsOpenGL;
import processing.core.*;

GLSL glsl;
boolean shaderOn;
PGraphicsOpenGL pgl;
GL gl;
GLU glu = new GLU();

int texture;

PImage img;

void setup() {

size(800,600,OPENGL);

glsl = new GLSL();
glsl.loadVertexShader("data/quadwarp.vert");
glsl.useShaders(); 

img = loadImage("data/grid_256x256.jpg");

pgl = (PGraphicsOpenGL) g;
gl = pgl.gl;

gl.glShadeModel(GL.GL_SMOOTH); // Enable Smooth Shading
gl.glClearColor(0.0f, 0.0f, 0.0f, 0.5f); // Black Background
gl.glClearDepth(1.0f); // Depth Buffer Setup
gl.glEnable(GL.GL_DEPTH_TEST); // Enables Depth Testing

gl.glDepthFunc(GL.GL_LEQUAL);// The Type Of Depth Testing To Do

gl.glHint(GL.GL_PERSPECTIVE_CORRECTION_HINT, GL.GL_NICEST);
// Really Nice Perspective Calculations

gl.glEnable(GL.GL_TEXTURE_2D);
texture = genTexture(gl);
gl.glBindTexture(GL.GL_TEXTURE_2D, texture);
gl.glTexParameteri(GL.GL_TEXTURE_2D, GL.GL_TEXTURE_MIN_FILTER, GL.GL_LINEAR);
gl.glTexParameteri(GL.GL_TEXTURE_2D, GL.GL_TEXTURE_MAG_FILTER, GL.GL_LINEAR);
}

void draw() {
pgl.beginGL();

glsl.startShader();
glsl.setUniformValue2f(glsl.getUniformLocation("BL"), 0.0, 0.0);
glsl.setUniformValue2f(glsl.getUniformLocation("BR"), 1.0, 0.0);
glsl.setUniformValue2f(glsl.getUniformLocation("TL"), 0.25, 1.0);
glsl.setUniformValue2f(glsl.getUniformLocation("TR"), 1.0, 1.0);
glsl.setUniformValue2f(glsl.getUniformLocation("renderSize"), (float)width/2, (float)height);
gl.glTexImage2D(GL.GL_TEXTURE_2D, 0, GL.GL_RGB, img.width, img.height, 0, GL.GL_RGB, GL.GL_UNSIGNED_BYTE, getTextureByteBuffer( false ) );
gl.glClear(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT);
gl.glLoadIdentity(); // Reset The View
gl.glScalef( 100, 100, 100);
gl.glTranslatef(0.0, 0.0, -5.0);
gl.glBindTexture(GL.GL_TEXTURE_2D, texture);

gl.glBegin(GL.GL_QUADS);
gl.glTexCoord2f(0,1);
gl.glVertex3f(0.0f, 1.0f, 1.0f); // Top Left Of The Texture and Quad

gl.glTexCoord2f(0,0);
gl.glVertex3f(0.0f, 0.0f, 1.0f); // Bottom Left Of The Texture and Quad

gl.glTexCoord2f(1,0);
gl.glVertex3f( 1.0f, 0.0f, 1.0f); // Bottom Right Of The Texture and Quad

gl.glTexCoord2f(1,1);
gl.glVertex3f( 1.0f, 1.0f, 1.0f); // Top Right Of The Texture and Quad

gl.glEnd();
glsl.endShader();
pgl.endGL();
}

int genTexture(GL gl) {
final int[] tmp = new int[1];
gl.glGenTextures(1, tmp, 0);
return tmp[0];
}

ByteBuffer getTextureByteBuffer ( boolean useAlphaChannel ) {
int bytesPerPixel = useAlphaChannel ? 4 : 3;

ByteBuffer unpackedPixels = BufferUtil.newByteBuffer( img.pixels.length * bytesPerPixel );

for (int row = img.height - 1; row >= 0; row--) {
for (int col = 0; col < img.width; col++) {
int packedPixel = img.pixels[row * img.width + col];
unpackedPixels.put((byte) ((packedPixel >> 16) & 0xFF));
unpackedPixels.put((byte) ((packedPixel >> 8) & 0xFF));
unpackedPixels.put((byte) ((packedPixel >> 0) & 0xFF));
if ( useAlphaChannel ) {
unpackedPixels.put((byte) ((packedPixel >> 24) & 0xFF));
}
}
}
unpackedPixels.flip();

return unpackedPixels;
}
 

Not sure how I managed to miss the bit about grids in your instructions Memo. Now I'm applying the texture to a grid of quads (40x40 of them) rather than a single quad.

I am still getting exactly the same problem though. When I create a grid that forms a square the texture gets applied as expected. But once I warp the grid things stop working as expected. e.g. if I push the top-left corner of the textured grid to the right then the top-right corner also pushes to the right by the same amount (as shown in the image in my last post).

This is the code to create a grid of quads in place of the single quad I was using above:

Code:

int gridCols = 40;
int gridRows = 40;
float quadWidth = 1.0;
float quadHeight = 1.0;
float gridSquareWidth = quadWidth / (float)gridCols;
float gridSquareHeight = quadHeight / (float)gridRows;
float texXPerSquare = 1.0/gridCols;
float texYPerSquare = 1.0/gridRows;

gl.glBegin(GL.GL_QUADS);

for ( int row = 0; row < gridRows; row++ ) {
  for ( int col = 0; col < gridCols; col++ ) {
    gl.glTexCoord2f( 0.0+(float)col*texXPerSquare, 0.0+(float)(row+1)*texYPerSquare );
    gl.glVertex3f((float)gridSquareWidth*col, (float)gridSquareHeight*(row+1), 1.0f);
// Top Left Of The Texture and Quad

    gl.glTexCoord2f( 0.0+(float)col*texXPerSquare, 0.0+(float)row*texYPerSquare );
    gl.glVertex3f((float)gridSquareWidth*col, (float)gridSquareHeight*row, 1.0f);
// Bottom Left Of The Texture and Quad

    gl.glTexCoord2f( 0.0+(float)(col+1)*texXPerSquare, 0.0+(float)row*texYPerSquare );
    gl.glVertex3f((float)gridSquareWidth*(col+1), (float)gridSquareHeight*row, 1.0f);
// Bottom Right Of The Texture and Quad
  
    gl.glTexCoord2f( 0.0+(float)(col+1)*texXPerSquare, 0.0+(float)(row+1)*texYPerSquare );
    gl.glVertex3f((float)gridSquareWidth*(col+1), (float)gridSquareHeight*(row+1), 1.0f);
// Top Right Of The Texture and Quad
  }
}

gl.glEnd();
 

I have two ideas about what might be happening here:
1) I'm just not getting the concept properly, hence my code isn't working.
2) A possible issue with the GMA950 integrated video card in my Macbook, as Googling suggests it may not handle some parts of the GLSL spec properly. I don't have another machine to test on right now though so I can't say for sure.

I'm a processing newcomer (so shoot me down if this doesn't apply!), but shouldn't it be possible to map a texture to the grid 'square on' (i.e a single quad) and then simply use rotateY(angle) to produce the projection distortion?

In my small set of experiments, I seem to be getting a result which is close to your Photoshopped requirement...

pdinnen, did you find a solution to your problem? I'm after something much similar to what you were trying to achieve. Would be great to get some pointers/help in warping/perspective correcting an image.

looks promosing, especially the planned updates for 3d model mapping etc. i believe i read apost from marius watz where he said that he hacked something similar. but im not sure... anyway this would be a great feature for processing. Thank you!