Finding camera up with yaw, pitch and roll?

szemy2

Hey!

I am trying to do object picking with rays in Processing. Everything works fine, except I cannot find the right 'up' vector of the camera. I am using Peasy library.

I could get the cameraUp vector by:

right = PVector.cross(view,new PVector(0,0,1)); then

cameraUp = PVector.cross(view, right);

However the problem is that when I am orbiting with the camera the up vector (0,0,1) gets rotated. I would like to find the up vector of the rotated coordinate system. Peasy lets you access the pitch, yaw and roll.

I tried doing it with:

    up.x = cos(cam.getRotations()[0])*sin(cam.getRotations()[1]);
    up.y = -cos(cam.getRotations()[1]);
    up.z = sin(cam.getRotations()[0])*sin(cam.getRotations()[1]);

and a few similar versions, but I just can't get it to work. it might be due to peasy defining the axis differently.

Any hint on how I could solve this?

Any answer would be much appreciated:)

quark

The up vector is normally defined as [0,1,0] in other words the y axis

szemy2

szemy2

The white rectangle represents the viewport when clicked in world space. After clicking I zoomed out so its visible. My problem is that as you can see if I only define an upvector according to the original coordinate system, then the rotation distorts what is considered 'up'

koogs

what information do you have?

camera position? look at position? right?

runnable examples will get you the best results.

szemy2

I have cameraPosition, cameraLookAt. That part of my code works. What I have is I can already find my mouse position where I clicked in 3D world (and the viewport in 3D coordinates), when the cameras 'roll' is 0.

However when I rotate around the lookAt these vectors won't correspond.

PVector lookAt = new PVector(cam.getLookAt()[0], cam.getLookAt()[1], cam.getLookAt()[2]);
PVector camPosition = new PVector (cam.getPosition()[0], cam.getPosition()[1],  cam.getPosition()[2]);

view=PVector.sub(lookAt, camPosition);
view.normalize();

PVector up = new PVector(0f, 0f, 1f);
PVector right = view.cross(up);

PVector cameraUp = view.cross(right);

I can also access: cam.getDirection[0] = rotation on the X-Axis cam.getDirection[1] = rotation on the Y-Axis cam.getDirection[2] = rotation on the Z-Axis

szemy2

szemy2

Bump up please

koogs

A runnable example will help.

I did initially think that your code snippet was a bit suspect - defining right in terms of view and 0, 0, 1 and then defining up in terms of view and right - but I guess that will work

Wait... Do you mean 0, 0, 1 or 0, 1, 0? Y is traditionally up.

koogs

(Ha, quark said the same thing 4 days ago)

szemy2

:) I understand this already. The way my code is written (and how Peasy defines Z) is by (0,0,1).

However I would like to understand what the new 'up' is when the camera is rotated. In the end I would like to get the cameraUP, which is a vector perpendicular to both the view vector of the camera (the vector between position of the cam and where it looks) and the world up.

The camera gets however rotated, which means the camera also 'rolls' on the axis of the view vector. I would like to calculate what this roll is. This is not straight forwards as the library offers the rotations around the world axis but not on the view vector's axis.

If I understand correctly:)

koogs

A runnable example will still help 8)

szemy2

import peasy.*;
import peasy.org.apache.commons.math.*;
import peasy.org.apache.commons.math.geometry.*;
//import peasy.test.*;

Box [] boxes = new Box[20];
PeasyCam cam;

PVector view;
float fov = 100;
float clippingPlaneNear = 100;
PVector screenHorizontally = new PVector(); 
PVector screenVertically = new PVector(); 
float boxSize = 20;
CameraState pState;

Ghost ghost;

PVector xz, yz, xy;
//---> For drawing vectors
PVector up;
PVector lookAt;
PVector camPosition;
float [] rotationsAtClick;
PVector interSectXY= new PVector();


void setup()
{

  size(1600, 1000, P3D); 
  cam = new PeasyCam(this, fov);
  cam.setMinimumDistance(clippingPlaneNear);
  pState = cam.getState();
  //cam.setPitchRotationMode();


  colorMode(HSB);
  rectMode(CENTER);

  for (int i=0; i<boxes.length; i++)
  {
    boxes[i]= new Box(random(-100, 100), random(-100, 100), random(-100, 100), boxSize);
  }

  setupPickingRay();
}

void draw()
{
  background(0);
  drawGrid();

  for (int i=0; i<boxes.length; i++)
  {
    //boxes[i].display();
  }

  if (ghost != null)
  {
    ghost.display();
  }

  /**
  if (cam.getState() == pState)
   {
   setupPickingRay();
   pState = cam.getState();
   println("Camera Moved");
   }
   */
}


void mouseClicked()
{
  if (mouseButton == LEFT)
  {
    PickingRay picking = new PickingRay();
    setupPickingRay();
    picking(mouseX, mouseY, picking);



    interSectXY = picking.intersectionWithXyPlane();
    //println("interSectXY: " + interSectXY);

    for (int i=0; i<boxes.length; i++)
    {
      if (interSectXY.x> boxes[i].position.x - boxSize/2
        && interSectXY.x< boxes[i].position.x + boxSize/2 
        && interSectXY.y> boxes[i].position.y - boxSize/2
        && interSectXY.y< boxes[i].position.y + boxSize/2)
      {
        boxes[i].colour = random(360);
      }
    }
  }
}


void setupPickingRay()
{
  view = new PVector(0f, 0f, 0f);
  // look direction

  lookAt = new PVector(cam.getLookAt()[0], cam.getLookAt()[1], cam.getLookAt()[2]);

  println("___________");
  println("1. LookAt: " + lookAt);
  camPosition = new PVector (cam.getPosition()[0], cam.getPosition()[1], cam.getPosition()[2]);
  println("2. camPosition: " + camPosition);

  view=PVector.sub(lookAt, camPosition);

  view.normalize();
  println("3. view: " + view);


  //Calculate camera UP Vector
  up = new PVector(0f, 0f, 1f);

  println("x: " + cam.getRotations()[0] + " | y: " + cam.getRotations()[1] + " | z: " + cam.getRotations()[2]);


  up = calcNewUp(0);


  rotationsAtClick=cam.getRotations();




  println("UPVECTOR: " + up);

  up.normalize();

  PVector right = view.cross(up);
  //PVector right = up;

  PVector cameraUp = view.cross(right);
  println("4. cameraUp: " + cameraUp);

  // screenX
  screenHorizontally = view.cross(cameraUp);
  screenHorizontally.normalize();
  println("5. screenHorizontally: " + screenHorizontally);

  // screenY
  screenVertically = screenHorizontally.cross(view);
  screenVertically.normalize();
  println("6. screenVertically: " + screenVertically);
  //println("5. screenHorizontally: " + screenHorizontally);

  final float radians = (float) (60*Math.PI / 180f); //fov*Math.PI / 180f);
  println("RADIANS: " + radians);
  float halfHeight = (float) (tan(radians/2f)*clippingPlaneNear);
  println("6.A. halfHeight: " + halfHeight);
  float halfScaledAspectRatio = halfHeight*((float)width/((float)height));//getViewportAspectRatio();


  println("6.B. halfScaledAspectRatio: " + halfScaledAspectRatio);

  screenVertically.mult(halfHeight);
  screenHorizontally.mult(halfScaledAspectRatio);

  println("7. screenVertically: " + screenVertically + "|  screenHorizontally: " + screenHorizontally);
}

public void picking(float screenX, float screenY, PickingRay pickingRay)
{
  PVector position= new PVector(cam.getPosition()[0], cam.getPosition()[1], cam.getPosition()[2]);
  println("8. position of cam: " + position);
  pickingRay.clickPosInWorld.set(position);
  println("view * clippingPlaneNear: " + PVector.mult(view, -clippingPlaneNear));
  pickingRay.clickPosInWorld.add(PVector.mult(view, clippingPlaneNear));
  println("9B.  pickingRay.clickPosInWorld: " +  pickingRay.clickPosInWorld);

  PVector centrePointOfScreen = new PVector();
  centrePointOfScreen.set(pickingRay.clickPosInWorld);

  screenX -= (float)width/2f;
  screenY -= (float)height/2f;

  // normalize to 1
  screenX /= ((float)width/2f);
  screenY /= ((float)height/2f);

  pickingRay.getClickPosInWorld().x += screenHorizontally.x*screenX + screenVertically.x*screenY;
  pickingRay.getClickPosInWorld().y += screenHorizontally.y*screenX + screenVertically.y*screenY;
  pickingRay.getClickPosInWorld().z += screenHorizontally.z*screenX + screenVertically.z*screenY;

  pickingRay.getDirection().set(pickingRay.getClickPosInWorld());
  pickingRay.getDirection().sub(position);

  //createGhost(pickingRay, centrePointOfScreen);
  float [] u = new float[3];
  u[0]=screenHorizontally.x;
  u[1]=screenHorizontally.y;
  u[2]=screenHorizontally.z;
  float [] v = new float[3];
  v[0]=screenVertically.x;
  v[1]=screenVertically.y;
  v[2]=screenVertically.z;

  ghost = new Ghost(pickingRay.getClickPosInWorld(), centrePointOfScreen, u, v);
}


void keyPressed()
{
  if (keyCode==LEFT)
  {
    cam.rotateZ(QUARTER_PI);
  }

  //if (key == ' ')
  //{
  //  cam.setRotations(0, QUARTER_PI, 0);
  //}
}

PVector calcNewUp(int num)
{
  PVector tUp = new PVector();

  float x=cam.getRotations()[0]; 
  float y=cam.getRotations()[1];
  float z=cam.getRotations()[2];

  x*=-1;
  z*=-1;
  switch(num)
  {
    //Z*X*Y // x- y-z   x-z-y   z-x-y   z-y-x  y-x-z  y-z-x
  case 0:
    tUp.x=(sin(y)*cos(z))+cos(y)*sin(z)*sin(x);
    tUp.y=(sin(y)*sin(z))-cos(y)*cos(z)*sin(x);
    tUp.z=cos(y)*cos(x);
    break;

    //X*Y*Z
  case 1:
    tUp.x=sin(y);
    tUp.y=-cos(y)*sin(x);
    tUp.z= cos(y)*cos(x);
    break;

    //Y*X*Z
  case 2: // x- y-z    x-z-y   z-x-y  z-y-x    y-x-z y-z-x
    tUp.y= sin(y)*cos(x);
    tUp.z=-sin(x);
    tUp.x=cos(y)*cos(x);
    break;
    //X*Z*Y
  case 3:
    tUp.x=sin(y)*cos(z);
    tUp.y=sin(y)*sin(z)*cos(x)-cos(y)*sin(x);
    tUp.z=cos(y)*cos(x)+sin(y)*sin(z)*sin(x);
    break;

    //Y*Z*X
  case 4:
    tUp.x= sin(y)*cos(x)+cos(y)*sin(z)*sin(x);
    tUp.y=-cos(z)*sin(x);
    tUp.z=cos(y)*cos(x)-sin(y)*sin(z)*sin(x);
    break;

    //Z*X*Y
  case 5:
    tUp.x=sin(y)*cos(z)*cos(x)+sin(z)*sin(x);
    tUp.y=sin(y)*sin(z)*cos(x)-cos(z)*sin(x);
    tUp.z=cos(y)*cos(x);

    break;
  }

  return tUp;
}

class PickingRay
{  
  PVector clickPosInWorld = new PVector();
  PVector direction = new PVector();

  PickingRay()
  {
  }

  /**
   * Computes the intersection of this ray with the X-Y Plane (where Z = 0)
   * and writes it back to the provided vector.
   */
  public PVector intersectionWithXyPlane()
  {
    PVector mouseInWorld = new PVector();
    float s = -clickPosInWorld.z/(direction.z)  ;

    mouseInWorld.x = (float)(clickPosInWorld.x+(direction.x*s));
    mouseInWorld.y = (float)(clickPosInWorld.y+(direction.y*s));
    mouseInWorld.z = 0;

    println("s: " + s);
    println("clickPosInWorld: " + clickPosInWorld);
    println("direction: " + direction);
    println("mouseInWorld: " + mouseInWorld);

    return mouseInWorld;
  }

  public PVector getClickPosInWorld() {
    return clickPosInWorld;
  }
  public PVector getDirection() {
    return direction;
  }
}

class Ghost
{
  PVector clickPos = new PVector();
  PVector centrePt = new PVector();
  PVector v = new PVector();
  PVector u = new PVector();

  Ghost(PVector _clickPos, PVector _centrePt, float [] _v, float [] _u)
  {
    clickPos=_clickPos;
    centrePt=_centrePt;
    v.set(_v);
    u.set(_u);
    //v=new PVector(_v[0], _v[1], _v[2]);
    //u=new PVector(_u[0], _u[1], _u[2]);

    //v.normalize();
    //u.normalize();
  }

  void display()
  {
    pushMatrix();
    {
      translate(clickPos.x, clickPos.y, clickPos.z);
      stroke(255);
      strokeWeight(2);

float makeSmaller = 10f;

      line(-v.x*0.5f/makeSmaller, -v.y*0.5f/makeSmaller, -v.z*0.5f/makeSmaller, v.x*0.5f/makeSmaller, v.y*0.5f/makeSmaller, v.z*0.5f/makeSmaller);
      line(-u.x*0.5f/makeSmaller, -u.y*0.5f/makeSmaller, -u.z*0.5f/makeSmaller, u.x*0.5f/makeSmaller, u.y*0.5f/makeSmaller, u.z*0.5f/makeSmaller);
      point(0, 0, 0);
    }
    popMatrix();

    noFill();
    strokeWeight(3);
    pushMatrix();
    {
      //translate(centrePt.x, centrePt.y, centrePt.z);
      beginShape();
      vertex(centrePt.x - v.x + u.x, centrePt.y - v.y + u.y, centrePt.z - v.z + u.z);
      vertex(centrePt.x + v.x + u.x, centrePt.y + v.y + u.y, centrePt.z + v.z + u.z);
      vertex(centrePt.x + v.x - u.x, centrePt.y + v.y - u.y, centrePt.z + v.z - u.z);
      vertex(centrePt.x - v.x - u.x, centrePt.y - v.y - u.y, centrePt.z - v.z - u.z);
      endShape(CLOSE);
    }
    popMatrix();
  }
}

szemy2

class Box
{
  PVector position;
  float size;
  float colour;

  Box(float x, float y, float z, float _size)
  {
    position = new PVector(x, y, 0);
    size=_size;
    colour =360;
  }

  void display()
  {
    noStroke();
    pushMatrix();
    translate(position.x, position.y, 0f);
    fill(colour, 360, 360);

    //box(size, size, size);
    rect(0, 0, size, size);
    textSize(7);
    fill(255);
    textMode(SHAPE);
    String s = str(position.x) + " | " + str(position.y) + " | " + str(position.z);
    text(s, 0, 0);
    textMode(MODEL);
    stroke(255);
    strokeWeight(15);
    point(0, 0, 0);
    popMatrix();
  }
}

szemy2

What you can see at calcNewUp() is that I would like to get the new vector by calculating the rotation matrix from the roll, pitch and yaw rotations that peasy gives.

There are six cases, as I don't know which order peasy applies the rotations: X * Y then XY*Z or which order. Then I multiply the resulting matrix with the upvector (0,0,1) to get the new upvector. This means basically selecting the third column of the rotation matrix.

Unfortunately it doesn't work with any order.

szemy2

void drawGrid()
{
  //x-Axis
  strokeWeight(5);
  stroke(0, 360, 360);
  line(0, 0, 0, width/2, 0, 0);
  //line(0, 0, 0, xz.x, xz.y, xz.z);

  //y-Axis
  stroke(100, 360, 360);
  line(0, 0, 0, 0, height/2, 0);
  //line(0, 0, 0, yz.x, yz.y, yz.z);

  //z-Axis
  stroke(180, 360, 360);
  line(0, 0, 0, 0, 0, height/2);
  //line(0, 0, 0, xy.x, xy.y, xy.z);


  stroke(180,360,360);

  //line(lookAt.x,lookAt.y,lookAt.z,view.x,view.y,view.z);
  stroke(250,360,100);
  line(interSectXY.x,interSectXY.y,interSectXY.z, camPosition.x,camPosition.y,camPosition.z);
  stroke(360,360,100);
  line(0,0,0,up.x*100f,up.y*100f,up.z*100f);


  pushMatrix();
  {
    strokeWeight(2);
    noFill();
    translate(camPosition.x,camPosition.y,camPosition.z);
   //translate(view.x*10,view.y*10,view.z*10);
    rotateX(rotationsAtClick[0]);
    rotateY(rotationsAtClick[1]);
    rotateZ(rotationsAtClick[2]);
    box(10,10,10);
  }
  popMatrix();
}

koogs

"The function drawGrid() does not exist."

szemy2

Sorry, forgot it, just posted it!

jeremydouglass

@szemy2 --

This is just brainstorming without deeply engaging your problem, but have you looked at using QueasyCam instead? I thought maybe it returned the frame-of-reference vectors you need.

szemy2

I have not used QueasyCam before, but I downloaded it now that you suggested it. Maybe it is easier to get it working with Quesy, however I find Peasy more intuitive so I wouldn't want to switch. Also I used Peasy in nearly all my projects. Anyway: thank you for the suggestion.

szemy2

I promise I will make a tutorial on this object picking thing or a library (that doesn't use off-screen coloured buffer like the picking library), once I figure this out :D

jeremydouglass

Great! Looking forward to what you share.

Understood about wanting to stick with the camera library you know.

In case it is helpful, here are the relevant parts of the QueasyCam README:

• getForward() = returns the forward-pointing PVector of the camera

• getUp() = returns the upward-pointing PVector of the camera

• getRight() = returns the rightward-pointing PVector of the camera

QueasyCam also makes its variables "pan" and "tilt" public.

koogs

ok, thanks for the code. but you might need to explain what i'm seeing and what you want to be seeing instead 8)

if i rotate a bit and click i get a dark red ray into the screen (which looks correct in that it's going directly away from me). and a white rectangle which i think is meant to be straight up and down in relation to the screen, but is often rotated around the dark red ray. i also get another dark red ray, attached to the origin, which is, i think, the up vector.

szemy2

What you are seeing: the white rectangle is the canvas calculated in world space. The white cross is the position of the click on the canvas in world space.

The dark red is the X-Axis, the green the Y-Axis and the blue the Z-Axis. Firstly, you were right the Y-Axis is the 'up' in this case (the rotations are zero when we are looking straight down.) There is also a ray (I think this is what you asked about) that is simply a line from the camera position to where the pickingRay would intersect with the xy plane. The white rectangle gets rotated around the middle of the canvas not around this ray.

The canvas doesn't get calculated correctly if the camera is rotated (if it is only rotated around the X axis, then it works). If it was working correctly the white rectangle would touch the edges of the canvas and the white cross would be exactly below where you clicked with the mouse.

Also I started extracting the rotation according to the world matrix:

szemy2

  PMatrix pm = getMatrix();
  float [] myMatrix = new float[16];
  pm.get(myMatrix);

  up.x=-myMatrix[1];
  up.y=-myMatrix[5];
  up.z=-myMatrix[9];

myMatrix contains 16 floats that represent the world matrix. I multiply this matrix by (0,-1,0,0), which means simply picking the second column of the rotation matrix and multiplying x, y and z values with -1.

Still doesn't work, but seems to behave a little better!

szemy2

bump up please!

Lord_of_the_Galaxy

This is a very difficult question. You may need to bump many more times.

jeremydouglass

This is not straight forwards as the library offers the rotations around the world axis but not on the view vector's axis. If I understand correctly

@szemy2 -- when you looked at the example of getUp() from QuesyCam that I already shared with you that accomplishes what you are trying to do, did the method work for you, or no? When you looked at the QuesyCam source code, was the source code that implements the method clear, or no?

I'm trying to understand the specific nature of your continuing problem given that this seemed to answer your question.