Processing 1.0 - Processing Discourse - Easy mouse-to-world function

We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.

‹ Previous Topic | Next Topic ›

Pages: 1

Easy mouse-to-world function (Read 3198 times)

JohnG

Easy mouse-to-world function
Apr 13^th, 2007, 6:54pm

After a good coding run this afternoon, I managed to find enough hints to cobble together an OpenGL way of finding the 3D world co-ordinates of the mouse position.

Code:

import javax.media.opengl.*;
import javax.media.opengl.glu.*;

GL gl;
GLU glu;

void setup()
{
  ...
  gl=((PGraphicsOpenGL)g).gl;
  glu=((PGraphicsOpenGL)g).glu;
}
  

void draw()
{
  // DRAW ALL 3D stuff
  float[] mousePos=getMouse3D();
//returns x,y,z values hopefully.
}

float[] getMouse3D()
{
  ((PGraphicsOpenGL)g).beginGL();
  int viewport[] = new int[4];
  double[] proj=new double[16];
  double[] model=new double[16];
  gl.glGetIntegerv(GL.GL_VIEWPORT, viewport, 0);
  gl.glGetDoublev(GL.GL_PROJECTION_MATRIX,proj,0);
  gl.glGetDoublev(GL.GL_MODELVIEW_MATRIX,model,0);
  FloatBuffer fb=ByteBuffer.allocateDirect(4).order(ByteOrder.nativeOrder()).asFloatBuffer();
  gl.glReadPixels(mouseX, height-mouseY, 1, 1, GL.GL_DEPTH_COMPONENT, GL.GL_FLOAT, fb);
  fb.rewind();
  double[] mousePosArr=new double[4];
  glu.gluUnProject((double)mouseX,height-(double)mouseY,(double)fb.get(0),model,0,proj,0,viewport,0,mousePosArr,0);
  ((PGraphicsOpenGL)g).endGL();
  return new float[]{(float)mousePosArr[0],(float)mousePosArr[1],(float)mousePosArr[2]};
}

prinds

Re: Easy mouse-to-world function
Reply #1 - Apr 20^th, 2007, 1:26am

Hi John,

Very happy to see your post. Just what I was looking for.

Unfortunately it doesn't work for me. I get the following error message :

Semantic Error: Type "FloatBuffer" was not found.

.. and I can't figure out why. I tried adding imports like 'import java.nio.FloatBuffer.*;', but it doesn't help me..

If anyone can help me, it would sure make my day (and night) better..

_prinds

prinds

Re: Easy mouse-to-world function
Reply #2 - Apr 20^th, 2007, 1:48am

... sometimes the next thing you try is the solution..

this was the case here. Instead of this line :
Code:


FloatBuffer fb = ByteBuffer.allocateDirect(4).order(ByteOrder.nativeOrder()).asFloatBuffer();

i wrote this line :
Code:


java.nio.FloatBuffer fb = java.nio.ByteBuffer.allocateDirect(4).order(java.nio.ByteOrder.nativeOrder()).asFloatBuffer();

and it works.. thanks to JohnG for a nice piece of work..

JohnG Ex Member	Re: Easy mouse-to-world function Reply #3 - Apr 20^th, 2007, 9:42am Ah yes, Iforgot about that, the line you need is "import java.nio.*"

mflux Senior Member Offline Posts: 388	Re: Easy mouse-to-world function Reply #4 - Apr 21^st, 2007, 5:09am Hi John We are always amazed at your openGL skills. Thanks for posting this. Would you care to run us through, line by line in your mouse function, of what each line does? I would love to understand exactly what you're doing here.
	Portfolio

JohnG

Re: Easy mouse-to-world function
Reply #5 - Apr 21^st, 2007, 2:44pm

I can have a go at explaining it, it's fairly straight forwards, just uses one magic glu function to do the real clever stuff.

Code:

float[] getMouse3D()
{
  ((PGraphicsOpenGL)g).beginGL();
//have to get processing to dump all it's matricies into GL, so the functions work.

  int viewport[] = new int[4];
//For the viewport matrix... not sure what all the values are, I think the first two are width and height, and all Matricies in GL seem to be 4 or 16...

  double[] proj=new double[16];
//For the Projection Matrix, 4x4

  double[] model=new double[16];
//For the Modelview Matrix, 4x4

  gl.glGetIntegerv(GL.GL_VIEWPORT, viewport, 0);
//fill the viewport matrix

  gl.glGetDoublev(GL.GL_PROJECTION_MATRIX,proj,0);
//projection matrix

  gl.glGetDoublev(GL.GL_MODELVIEW_MATRIX,model,0);
//modelview matrix

  FloatBuffer fb=ByteBuffer.allocateDirect(4).order(ByteOrder.nativeOrder()).asFloatBuffer();
//set up a floatbuffer to get the depth buffer value of the mouse position

  gl.glReadPixels(mouseX, height-mouseY, 1, 1, GL.GL_DEPTH_COMPONENT, GL.GL_FLOAT, fb);
//Get the depth buffer value at the mouse position. have to do height-mouseY, as GL puts 0,0 in the bottom left, not top left.

  fb.rewind(); //finish setting up this.

  double[] mousePosArr=new double[4];
//the result x,y,z will be put in this.. 4th value will be 1, but I think it's "scale" in GL terms, but I think it'll always be 1.

 glu.gluUnProject((double)mouseX,height-(double)mouseY,(double)fb.get(0),model,0,proj,0,viewport,0,mousePosArr,0);
//the magic function. You put all the values in, and magically the x,y,z values come out :)

  ((PGraphicsOpenGL)g).endGL();
  return new float[]{(float)mousePosArr[0],(float)mousePosArr[1],(float)mousePosArr[2]};
//The values are all doubles, so throw them into floats to make life easier.
}

mflux Senior Member Offline Posts: 388	Re: Easy mouse-to-world function Reply #6 - Apr 25^th, 2007, 12:42am wow that is.. pretty freaking amazing... great work!! Thanks for the explanation! Although I still have no idea how it works (don't know enough about projection/modelview matrices).
	Portfolio

sgsrules

Re: Easy mouse-to-world function
Reply #7 - Jun 4^th, 2008, 10:26pm

Works perfectly, except i'm trying to have the mouse pointer hover a certain z distance from the viewport instead of using the Z buffer information. pretty easy to do just replace the (double)fb.get(0) with a value, and get rid of the float buffer. I understand that the value needs to be between 0-1. My question is how do i figure out how far this point is in correlation to world coordinates? 0 is obviously at the viewport, but how far back does a value of 1 go? is 1 the end of the clipping volume?

glu.gluUnProject((double)mouseX,height-(double)mouseY,(double)fb.get(0), model,0,proj,0,viewport,0,mousePosArr,0);

Stephen Shepard (sgsrules)

JohnG

Re: Easy mouse-to-world function
Reply #8 - Jun 4^th, 2008, 10:41pm

Unfortunately as far as I can tell the values aren't directly translatable to world coordinates, as it depends on the clip planes and probably many other things.

If you know the camera position, and get the mouse-in-world position, you could then fairly easily with some vector maths get the position X units away from the camera, or from the mouse-in-world position.

e.g.
Code:

float[] cam=new float[]{20,30,40};
float[] mouse=new float{120,30,250}; //return values from mouseToWorld();
float[] dirVec=new float[]{mouse[0]-cam[0],mouse[1]-cam[1],mouse[2]-cam[2]};
float length=dist(mouse[0],mouse[1],mouse[2],cam[0],cam[1],cam[2])-20; //20 is distance in front of world.
float[] newMouse=new float[]{dirVec[0]*dist,dirVec[1]*dist,dirVec[2]*dist};

evan.r

Re: Easy mouse-to-world function
Reply #9 - Jul 8^th, 2008, 12:53pm

I'm getting very inconsistent results with this. I checked the viewport and it's returning the right size, and tried manually sticking values into gluUnproject for the z coordinate, and the results are consistent but wrong:

glu.gluUnProject((double)mouseX,height-(double)mouseY, 0.0,
model,0,
proj,0,
viewport,0,
mousePosArr,0);

For a 512x512 sketch, I get min/max x and y values of about 230/280, which leads me to believe that the modelview or projection matrix is the issue. Oddly, in a another sketch, after I draw a few figures this mouse function will (inconsistently) return the correct values! Any ideas? I'm on OS X 10.4.11 and tried both version 135 and 142.

Pages: 1

‹ Previous Topic | Next Topic ›