Compensating for keystone distortion, or creating some kind of homography routine?

Ben Purdy

Compensating for keystone distortion, or creating some kind of homography routine?

in Contributed Library Questions • 2 years ago

I've been playing with a projector, trying to project from processing onto various surfaces (a cardboard box, for example). After looking around the forums I found the keystone library, but it has a big problem.. After adjusting the keystone rectangle to fit the cardboard box, the grid squares are not consistently sized. They are wider on one side of the projection than they are on the other. The result is that if you move a circle (or any other shape) across the keystoned region, it will slowly stretch horizontally. Not only does this look bad, but it makes any attempt to match the rendering with any real world features virtually impossible. For example, if I wanted to project onto a wall that had a grid painted on it, the projected grid wouldn't line up due to the horizontal stretching. Has anybody else noticed this problem? Is there an easy solution?

Additionally:

After lots more googling, I found a technique called homography which is a way to create a projection matrix which solves the same problem, but seems to work much better. The tool vvvv has a node that does this, and it works perfectly in my test setup. The problem is, my brain just doesn't work with patch-based tools so I'd like to stick with processing if at all possible. I looked around on the forum and it seems that the projection matrices for the rendering system aren't accessible (not that I know enough math to implement the homography solution anyway.. :| ).

So.. thoughs? ideas? While I'm obviously interested in a solution I'm also just hoping to spur some discussion on this topic since I couldn't find much about it on the forums so far.

Replies(7)

shaheeb

Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

Hello!

I would be interested in working toward a solution on this issue. I'm very curious about automatic projector calibration, and particularly interested in projection mapping techniques. I am not terribly strong in math, but am somewhat competent at writing code. I'll take a closer look at the keystone library. I've played around with the MatLab procam lib ( http://code.google.com/p/procamcalib/) that's kind of like the "square one" for most CV applications. Perhaps there is some scope to apply those principles to the keystone lib and extend it to do what we need.

What do you think?

Shaheeb Roshan
www.shaheeb.com

Ben Purdy

Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

Shaheeb,

Good news, I've found a solution to this problem!

I've used the java advanced imaging library to create a quad-to-quad transformation based on a source rectangle and a set of four arbitrary destination points. I'm building what I have into a class that will function in a similar fashion to the keystone library, but will include proper perspective (instead of simply making equally spaced grids within a given quad).

Once I have a working sketch I'll post it here. In the mean time, check out the java advanced imaging stuff if you're curious. I'm using PerspectiveTransform.getQuadToQuad() to set up the transform, then I just use the transform() method with each set of grid coordinates to create the warped, textured quad.

I'll post more soon, with code samples..

Ben Purdy

Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

I thought I'd take some screenshots to better illustrate what I'm talking about in this thread.

Here's the keystone library, with the problem I've described. Notice that the vertical grid lines are spaced evenly from left to right. But if you actually look at a rectangular grid along an angle, you don't see this, you'll see that the vertical grid lines get closer together as they approach the vanishing point. So, using the keystone library will result in distorted images when projecting onto a surface that isn't perpendicular to the projector.

Here's my sketch so far, using the PerspectiveTransform class in the Java Advanced Imaging library. You can see that the vertical lines are spaced correctly according to the perspective of the rectangle.

I'm going to keep working on this and attach the sketch later, or maybe even make a library. Stay tuned..

Ben Purdy

Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

Another interesting bit..

I was looking for a way to find the mouse cursor position translated onto the grid, and the PerspectiveTransform class provides a handy method to compute the inverse of the transform, so it's really easy to go from screen coordinates back to the transformed quad coordinates. See the docs here..

Also, texturing the quad using an offscreen PGraphics instance is now working..

maujabur

Re: Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

Hi!

I'm looking for exactily the same thing!

Would you mind posting the sample sketch? I'm new to Java and processing, so I'm a bit lost about how to use JAI inside processing.

Thanks!

Ben Purdy

Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

Ok, here's the sketch so far... It's kind of getting huge, I might have to switch to eclipse soon and split this out into a bunch of different files so I don't loose my mind.

The JAI files you need are the two that are used for the PerspectiveTransform class. They're jai_core.jar and jai_codec.jar, you should be able to find them if you install JAI and search for those file names.

In the sketch, pressing the '1' key will toggle in and out of "configuration" mode. Left-click-drag corners or center point, right-click-drag will move in 1/10th pixel increments for fine adjustment. If you drag a corner far enough to make the shape convex it really freaks out, just so you know...

My plan is still to get this into a library once I get the ProjectionManager class fleshed out, so this is a work-in-progress, but you asked for it! :)

import processing.opengl.*;
import javax.media.jai.*;
import codeanticode.glgraphics.*;
import processing.core.*;
ProjectedSurface q;
GLGraphicsOffScreen offscreen, offscreen2;
int offset;
void setup()
{
size(1024, 768, GLConstants.GLGRAPHICS);
offscreen = new GLGraphicsOffScreen(this, 720, 540);
q = new ProjectedSurface( this, 720, 540 );
q.setTextureRect( 0, 0, 720, 540 );
}
// Render some random blocks of color..
void drawOffscreen()
{
offset++;
offscreen.beginDraw();
offscreen.background(0);
offscreen.fill(255);
offscreen.noStroke();
int stepX = offscreen.width / 12;
int stepY = offscreen.height / 9;
for(int x = 0; x < 12; x++)
{
for(int y = 0; y < 9; y++)
{
float v = noise(x*0.1, y*0.1, offset*0.01);
if(v > 0.5)
{
offscreen.fill(v * 255);
offscreen.rect( x * stepX, y * stepY, stepX, stepY);
}
else
{
offscreen.fill(0, 255-(v * 255), 0);
offscreen.rect( x * stepX, y * stepY, stepX, stepY);
}
}
}
offscreen.endDraw();
}
void draw()
{
drawOffscreen();
GLTexture t = offscreen.getTexture();
background(0);
q.draw( t );
q.update();
}
void keyPressed()
{
switch(key)
{
case '1':
q.toggleMode();
break;
}
}
// Class for managing lots of projected surfaces at once.
class ProjectionMapper
{
ArrayList<ProjectedSurface> surfaces;
PApplet parentApplet;
ProjectionMapper(PApplet parent)
{
this.parentApplet = parent;
this.parentApplet.registerDispose(this);
}
public ProjectedSurface createSurface(String name, int width, int height)
{
ProjectedSurface surf = new ProjectedSurface(this.parentApplet, width, height);
surf.name = name;
this.surfaces.add(surf);
return surf;
}
public ProjectedSurface getSurface( String name )
{
for(int i = 0; i < this.surfaces.size(); i++)
{
if(this.surfaces.get(i).name.equals(name))
{
return this.surfaces.get(i);
}
}
return null;
}
void update()
{
for(int i = 0; i < this.surfaces.size(); i++)
{
this.surfaces.get(i).update();
}
}
public void dispose()
{
}
}
class ProjectedSurface
{
final private int MODE_RENDER = 0;
final private int MODE_CALIBRATE = 1;
PApplet parentApplet;
public String name = "Projected Quad";
private int MODE = MODE_RENDER;
private int activePoint = -1; // Which corner point is selected?
private final float SELECTION_DISTANCE = 15; // Maximum distance from cursor to corner point before considering it "selected"
private final int GRID_LINE_COLOR = color(128,128,128);
private final int CORNER_MARKER_COLOR = color(255,255,255);
private final int SELECTED_CORNER_MARKER_COLOR = color(255,0,0);
int GRID_RESOLUTION = 30; // Number of grid lines.. Increasing this will mean better looking results but will decrease performance.
// Dimensions of the source rectangle.
float sourceWidth = 100;
float sourceHeight = 100;
// Metrics for the projected texture..
float textureX = 0;
float textureY = 0;
float textureWidth = 0;
float textureHeight = 0;
// The four corners of the transformed quad (in 2d screen space)
Point2D[] destinationPoints;
// The entire list of transformed grid points are stored in this array (from left to right, top to bottom, like pixels..).
// This list is updated whenever the updateTransform() method is invoked.
Point2D[] gridPoints;
// The raw list of verticies to be pumped out each frame. This array
// holds the pre-computed list, including duplicates, to save on computation during rendering.
Point2D[] vertexPoints;
// A list of draggable control points (corners and center point). This list is only used
// for calibration mode to make thing easier when checking the mouse cursor position.
// Ordered clock-wise starting with top-left.
Point2D[] controlPoints;
// The transform! Thank you Java Advanced Imaging, now I don't have to learn a bunch of math..
// Docs: http://download.oracle.com/docs/cd/E17802_01/products/products/java-media/jai/forDevelopers/jai-apidocs/javax/media/jai/PerspectiveTransform.html
PerspectiveTransform transform;
// Constructor, initialize the varous arrays of points.
ProjectedSurface(PApplet parent, int sourceWidth, int sourceHeight)
{
this.parentApplet = parent;
this.sourceWidth = sourceWidth;
this.sourceHeight = sourceHeight;
println(g.width + ", " + g.height);
this.destinationPoints = new Point2D[4];
this.controlPoints = new Point2D[5];
this.gridPoints = new Point2D[ this.GRID_RESOLUTION * this.GRID_RESOLUTION ];
this.vertexPoints = new Point2D[this.gridPoints.length * 4];
for(int i = 0; i < this.gridPoints.length; i++)
{
this.gridPoints[i] = new Point2D();
}
for(int i = 0; i < this.vertexPoints.length; i++)
{
this.vertexPoints[i] = new Point2D();
}
for(int i = 0; i < this.controlPoints.length; i++)
{
this.controlPoints[i] = new Point2D();
}
for(int i = 0; i < this.destinationPoints.length; i++)
{
this.destinationPoints[i] = new Point2D();
}
this.destinationPoints[0].x = 0;
this.destinationPoints[0].y = 0;
this.destinationPoints[1].x = this.sourceWidth;
this.destinationPoints[1].y = 0;
this.destinationPoints[2].x = this.sourceWidth;
this.destinationPoints[2].y = this.sourceHeight;
this.destinationPoints[3].x = 0;
this.destinationPoints[3].y = this.sourceHeight;
this.textureWidth = sourceWidth;
this.textureHeight = sourceHeight;
this.updateTransform();
}
void setParentApplet(PApplet applet)
{
this.parentApplet = applet;
}
// Sets the coordinates of the texture. This allows for a single texture to span
// more than one surface rather than have to draw on multiple offscreen buffers.
void setTextureRect(float x, float y, float w, float h)
{
this.textureX = x;
this.textureY = y;
this.textureWidth = w;
this.textureHeight = h;
this.updateTransform();
}
// Set the coordinates of one of the target corner points.
void setCornerPoint(int pointIndex, float x, float y)
{
this.destinationPoints[pointIndex].x = x;
this.destinationPoints[pointIndex].y = y;
this.updateTransform();
}
void setModeCalibrate()
{
this.MODE = this.MODE_CALIBRATE;
}
void setModeRender()
{
this.MODE = this.MODE_RENDER;
}
void toggleMode()
{
if(this.MODE == this.MODE_RENDER)
{
this.MODE = this.MODE_CALIBRATE;
}
else
{
this.MODE = this.MODE_RENDER;
}
}
// Update the UI (only needed in calibration mode..)
void update()
{
if(this.MODE == this.MODE_CALIBRATE)
{
if( mousePressed && (this.activePoint != -1) )
{
float deltaX = mouseX - pmouseX;
float deltaY = mouseY - pmouseY;
// Right mouse button drags very slowly.
if(mouseButton == RIGHT)
{
deltaX *= 0.1;
deltaY *= 0.1;
}
// special case.
// index 4 is the center point so move all four corners.
if(this.activePoint == 4)
{
for(int i = 0; i < 4; i++)
{
this.setCornerPoint( i,
this.destinationPoints[i].x + deltaX,
this.destinationPoints[i].y + deltaY );
}
}
else
{
// Move a single corner point.
this.setCornerPoint( this.activePoint,
this.destinationPoints[this.activePoint].x + deltaX,
this.destinationPoints[this.activePoint].y + deltaY );
}
}
else
{
this.activePoint = this.getActiveControlPointIndex();
}
}
}
void draw(PImage tex)
{
this.drawQuad( tex );
if(this.MODE == this.MODE_CALIBRATE)
{
this.drawGrid();
}
}
// Get the index of the control point that is close to the mouse cursor.
// Only used in calibration mode.
int getActiveControlPointIndex()
{
for(int i = 0; i < this.controlPoints.length; i++)
{
if( dist( mouseX, mouseY, this.controlPoints[i].x, this.controlPoints[i].y ) < this.SELECTION_DISTANCE)
{
return i;
}
}
return -1;
}
// Set all four corners of the destination quad.
void setDestionationPoints(float x0, float y0, float x1, float y1, float x2, float y2, float x3, float y3)
{
this.destinationPoints[0].x = x0;
this.destinationPoints[0].y = y0;
this.destinationPoints[1].x = x1;
this.destinationPoints[1].y = y1;
this.destinationPoints[2].x = x2;
this.destinationPoints[2].y = y2;
this.destinationPoints[3].x = x3;
this.destinationPoints[3].y = y3;
this.updateTransform();
}
// Update the various coordinates using the perspective transform.
// This must be called whenever the source dimensions or destination coordinates are changed.
void updateTransform()
{
// Update the PerspectiveTransform with the current width, height, and destination coordinates.
this.transform = PerspectiveTransform.getQuadToQuad(
0, 0,
this.sourceWidth, 0,
this.sourceWidth, this.sourceHeight,
0, this.sourceHeight,
this.destinationPoints[0].x, this.destinationPoints[0].y,
this.destinationPoints[1].x, this.destinationPoints[1].y,
this.destinationPoints[2].x, this.destinationPoints[2].y,
this.destinationPoints[3].x, this.destinationPoints[3].y
);
// calculate the x and y interval to subdivide the source rectangle into the desired resolution.
float stepX = this.sourceWidth / float(this.GRID_RESOLUTION-1);
float stepY = this.sourceHeight / float(this.GRID_RESOLUTION-1);
// figure out the number of points in the whole grid.
int numPoints = this.GRID_RESOLUTION * this.GRID_RESOLUTION;
// create the array of floats (used for input into the transform method, it requires a single array of floats)
float[] srcPoints = new float[numPoints * 2];
// calculate the source coordinates of the grid points, as well as the texture coordinates of the destination points.
int i = 0;
for(int y = 0; y < this.GRID_RESOLUTION; y++)
{
for(int x = 0; x < this.GRID_RESOLUTION; x++)
{
float percentX = (x * stepX) / this.sourceWidth;
float percentY = (y * stepY) / this.sourceHeight;
this.gridPoints[x + y * this.GRID_RESOLUTION].u = this.textureWidth * percentX + this.textureX;
this.gridPoints[x + y * this.GRID_RESOLUTION].v = this.textureHeight * percentY + this.textureY; // y * stepY;
srcPoints[i++] = x * stepX;
srcPoints[i++] = y * stepY;
}
}
// create an array for the transformed points (populated by PerspectiveTransform.transform())
float[] transformed = new float[srcPoints.length];
// perform the transformation.
this.transform.transform( srcPoints, 0, transformed, 0, numPoints);
// convert the array of float values back into x/y pairs in the Point2D class for ease of use later.
for(int p = 0; p < numPoints; p++)
{
this.gridPoints[p].x = transformed[p*2];
this.gridPoints[p].y = transformed[p*2+1];
}
// Precompute the verticies for use in rendering.
int offset = 0;
int vertextIndex = 0;
for(int y = 0; y < this.GRID_RESOLUTION - 1; y++)
{
for(int x = 0; x < this.GRID_RESOLUTION - 1; x++)
{
offset = x + y * this.GRID_RESOLUTION;
this.vertexPoints[vertextIndex++].copyPoint( this.gridPoints[offset] );
this.vertexPoints[vertextIndex++].copyPoint( this.gridPoints[offset+1] );
offset = x + (y + 1) * this.GRID_RESOLUTION;
this.vertexPoints[vertextIndex++].copyPoint( this.gridPoints[offset+1] );
this.vertexPoints[vertextIndex++].copyPoint( this.gridPoints[offset] );
}
}
// keep track of the four transformed corner points for use in calibration mode.
this.controlPoints[0].x = this.gridPoints[0].x;
this.controlPoints[0].y = this.gridPoints[0].y;
this.controlPoints[1].x = this.gridPoints[this.GRID_RESOLUTION - 1].x;
this.controlPoints[1].y = this.gridPoints[this.GRID_RESOLUTION - 1].y;
this.controlPoints[2].x = this.gridPoints[this.gridPoints.length - 1].x;
this.controlPoints[2].y = this.gridPoints[this.gridPoints.length - 1].y;
this.controlPoints[3].x = this.gridPoints[this.gridPoints.length - this.GRID_RESOLUTION].x;
this.controlPoints[3].y = this.gridPoints[this.gridPoints.length - this.GRID_RESOLUTION].y;
// Find the average position of all the control points, use that as the center point.
float avgX = 0;
float avgY = 0;
for(int c = 0; c < 4; c++)
{
avgX += this.controlPoints[c].x;
avgY += this.controlPoints[c].y;
}
avgX /= 4;
avgY /= 4;
this.controlPoints[4].x = avgX;
this.controlPoints[4].y = avgY;
}
// Translate a point on the screen into a point on the quad.
public PVector screenCoordinatesToQuad(float x, float y)
{
double[] srcPts = new double[2];
srcPts[0] = x;
srcPts[1] = y;
double[] dstPts = new double[2];
try
{
this.transform.inverseTransform(srcPts, 0, dstPts, 0, 1);
}
catch(Exception e)
{
return new PVector(0,0);
}
return new PVector((float)dstPts[0], (float)dstPts[1]);
}
// Draw the textured transformed quad.
public void drawQuad(PImage tex)
{
this.parentApplet.noStroke();
this.parentApplet.beginShape( QUADS );
this.parentApplet.texture( tex );
// Blast through the precomputed list of verticies (see updateTransform)
int c = this.vertexPoints.length;
for(int i = 0; i < c; i++)
{
this.parentApplet.vertex(this.vertexPoints[i].x, this.vertexPoints[i].y,this.vertexPoints[i].u,this.vertexPoints[i].v);
}
this.parentApplet.endShape(CLOSE);
}
public void drawGrid()
{
this.parentApplet.noFill();
this.parentApplet.stroke(this.GRID_LINE_COLOR);
// Draw vertial grid lines
int rowOffset = (this.GRID_RESOLUTION * this.GRID_RESOLUTION) - this.GRID_RESOLUTION;
for(int i = 0; i < this.GRID_RESOLUTION; i++)
{
this.parentApplet.line( this.gridPoints[i].x, this.gridPoints[i].y,
this.gridPoints[i+rowOffset].x, this.gridPoints[i+rowOffset].y );
}
// Draw horezontal grid lines
for(int y = 0; y < this.GRID_RESOLUTION; y++)
{
int row = this.GRID_RESOLUTION * y;
this.parentApplet.line( this.gridPoints[row].x, this.gridPoints[row].y,
this.gridPoints[row + this.GRID_RESOLUTION - 1].x, this.gridPoints[row + this.GRID_RESOLUTION - 1].y );
}
// Draw the control points.
for(int i = 0; i < this.controlPoints.length; i++)
{
this.drawControlPoint(this.controlPoints[i].x, this.controlPoints[i].y, (this.activePoint == i));
}
}
void drawControlPoint(float x, float y, boolean selected )
{
if(selected)
{
this.parentApplet.stroke( this.SELECTED_CORNER_MARKER_COLOR );
}
else
{
this.parentApplet.stroke(this.CORNER_MARKER_COLOR);
}
this.parentApplet.ellipse( x, y, 16, 16);
this.parentApplet.line( x, y - 8, x, y + 8);
this.parentApplet.line( x - 8, y, x + 8, y);
}
}
class Point2D
{
float x = 0;
float y = 0;
float u = 0;
float v = 0;
void copyPoint( Point2D other )
{
this.x = other.x;
this.y = other.y;
this.v = other.v;
this.u = other.u;
}
}

david.bou..

Re: Compensating for keystone distortion, or creating some kind of homography routine?

2 years ago

Hey everyone,

Just letting you know that the problem discussed above has been fixed in the Keystone library, using the JAI perspective transform method. The cursor position code works as well..

Better late than never,
d.

Top Reply