I'm trying to use the mouse to select a point on a plane that has been rotated along the x-axis relative to the camera. I've figured out how to find the vertical position of the mouse (on the plane) and I tried to use that to find it's horizontal position but there are some errors:
In the link above, if you move to the mouse up and down in the center of the window, you'll see that the mouse stays centered in the cursor (the red box). But, if you move it up and down along either side of the window, you'll see that the cursor seems to move in an arc rather than a straight line beneath the mouse. The vertical position remains accurate, but the horizontal position gets screwy.
I believe the most relevant code can be found in map3d.pde within the function calcMouseAngles(). The variable cursorPosition should be the 3D coordinates of the mouse on the plane. cursorPosition.x is it's horizontal position on that plane.
I've seen other mouse picking algorithms that involves painting different shapes in different colors, and that seems unreasonable for selecting an arbitrary point on a visible surface. It seems like I'm close to a solution so I would prefer to stick to this approach. Any help would be appreciated.