How to move a 3D object according to the screen's XY axis, instead of the world's X,Y,Z(PeasyCam)

mossmossmoss

hi! Im writing a 3D sketch in which the user rotates the camera with peasyCam while left clicking and moving the mouse. The thing is that I want to move the objects while right click is pressed so that the user can drag the object across the screen's X and Y axis. Of course I know how to use mouseX and mouseY inputs to modify the translation but only across the 3D space coordinates as it shows on the GIF below:

imgur.com/nWbpGhH

example code of whats happening in the image:

import peasy.*;
import peasy.org.apache.commons.math.*;
import peasy.org.apache.commons.math.geometry.*;

PeasyCam cam;

float x=15;float y=15; float z=15;
float e;

void setup(){
  size (700,700,P3D);
  cam = new PeasyCam(this, 200);
  cam.setRightDragHandler(null);


}
void draw(){
  background(0);
  pushMatrix();
  translate(5, 5, 0);
  fill(255);
  stroke(255);
  sphere(5);
  popMatrix();
  pushMatrix();
  fill(255,0,0);
  stroke(255,0,0);
  translate(x, y, z);
  sphere(5);
  popMatrix();
  stroke(0,0,255);
  line(5,5,0,x,y,z);
  //obvoiusly not working method
  if(mousePressed && (mouseButton == RIGHT)){
    x= x+(mouseX-pmouseX);
    y= y+(mouseY-pmouseY);
  }
}
void mouseWheel(MouseEvent event) {
  e = event.getCount();
  z=z+e;
  println(e);
}
void mousePressed(){
  if (mouseButton==RIGHT){
    cam.setActive(false);
  }
}
void mouseReleased(){
 cam.setActive(true);
}

What I would need is to be able to drag the sphere only on the screens X/Y axis, at a fixed Z just like image below shows(simple simulation I made of the behaviour im looking for).

imgur.com/jc8wVWy

PeasyCam is for exploring the 3D space. The question might be difficult to undesrtand. The problem is about moving the object on the 3D world, using the screen/canvas 2D coordinates to make the object follow the cursor's movement. If the mouse goes to the left (x axis decreases), the object should move to the left on the screen, and not just on the worlds X axis. This is how the second example image behaves, but achieved by just simulating the 3D space with no actual rotations to the x,y,z axis.

I've been scratching my head with this thing and I cant seem to figure it out. I wouldn't have asked here otherwise. Thanks in advance guys.

kfrajer

Here is my attempt. First thing is to have a reference frame to be able to make comparisons and see things more clear. The program begin with an axis drawn,RGB for x,y,z. In this mode the cam is active. Then if you hit the 'g' key (for grid) you get a grid. The cam is disable. Notice that only in the grid state you can unlock the mouse. The mouse always starts lock in the grid state. If you hit m, you can toggle the mouse's state. The mouseX/mouseY dynamics changes the position of your sphere of interest and the sphere's position is mapped to the width and height of your sketch. Remember, you need to lock/unlock the mouse to be able to adjust x and y. Remark: Locking/unlocking the mouse's state in non-grid mode just does nothing.

You might consider checking the getRotations/getPosition from the peasycam's reference page. You could also use that information to map the 2D plane of your sketch into the 3D world set by the peasy cam. However, I think this solution below could be good but probably not sufficient.

Kf

import peasy.*;
import peasy.org.apache.commons.math.*;
import peasy.org.apache.commons.math.geometry.*;  

//axisgrid gridaxis scene rotation

PeasyCam cam;

float x=15;
float y=15; 
float z=15;
float e;

boolean showGrid=false;
boolean showAxis=true;  //false;
boolean lockMouse=true;

void setup() {
  size (700, 700, P3D);
  cam = new PeasyCam(this, 700);
  cam.setRightDragHandler(null);
}


void draw() {
  background(0);

  if (showGrid) drawGrid(200, 200, 100, 2);
  if (showAxis) drawAxis(200);

  pushMatrix();
  //translate(5, 5, 0);
  fill(255);
  stroke(255);
  sphere(5);
  popMatrix();
  pushMatrix();
  fill(255, 0, 0);
  stroke(255, 0, 0);
  translate(x, y, z);
  sphere(5);


  //obvoiusly not working method
  //if (mousePressed && (mouseButton == RIGHT)) {
  if (showGrid==true && lockMouse==false) {
    //x= x+(mouseX-pmouseX);
    //y= y+(mouseY-pmouseY);
    x=mouseX;
    y=mouseY;
  }
  popMatrix();

  stroke(0, 0, 255);
  line(0, 0, 0, x, y, z);
}
void mouseWheel(MouseEvent event) {
  e = event.getCount();
  z=z+e;
  println(e);
}
void mousePressed() {
  if (mouseButton==RIGHT) {
    //cam.setActive(false);
  }
}
void mouseReleased() {
  //cam.setActive(true);
}

void keyReleased() {


  //if (key=='a')
  //showAxis=!showAxis;


  if (key=='g') {
    showGrid=!showGrid;
    lockMouse=true;   //Mouse always lock when changing states
    cam.setActive(!showGrid);
  }

  if (key=='m' && showGrid==true) {
    lockMouse=!lockMouse;
  }
}



// -------------------------------------------------------
void drawAxis(float len) {
  drawAxis(len, len, len);
}


// -------------------------------------------------------
void drawAxis(float len1, float len2, float len3) {

  pushStyle();
  strokeWeight(3);

  stroke(255, 0, 0);  //RED
  line(0, 0, 0, len1, 0, 0);

  stroke(0, 255, 0);  //GREEN
  line(0, 0, 0, 0, len2, 0);

  stroke(0, 0, 255);  //BLUE
  line(0, 0, 0, 0, 0, len3);

  popStyle();
}

// -------------------------------------------------------
// @Args:  Plane: 0=YZ  1=XZ  2=XY 
void drawGrid(int size, int w, int h, int plane) {
  pushStyle();
  noFill();
  if (plane == 0) stroke(255, 0, 0);
  if (plane == 1) stroke(0, 255, 0);
  if (plane == 2) stroke(0, 0, 255);
  int total = w * h;
  int tw = w * size;
  int th = h * size;
  beginShape(LINES);
  for (int i = 0; i < total; i++) {
    int x = (i % w) * size;
    int y = (i / w) * size;
    if (plane == 0) {
      vertex(0, x, 0);
      vertex(0, x, th);
      vertex(0, 0, y);
      vertex(0, tw, y);
    }
    if (plane == 1) {
      vertex(x, 0, 0);
      vertex(x, 0, th);
      vertex(0, 0, y);
      vertex(tw, 0, y);
    }
    if (plane == 2) {
      vertex(x, 0, 0);
      vertex(x, th, 0);
      vertex(0, y, 0);
      vertex(tw, y, 0);
    }
  }
  endShape();
  popStyle();
}

mossmossmoss

Thanks kf! your code makes it pretty easy to understand the problem in question . I've been trying to grasp the concept of rotations to use peasycam's getRotations() to implement a solution, but I haven't figured it out yet (I probably need to know a little bit more about rotation matricies). Thanks a lot for spending the time man, I'm going to use your code to experiment with getRotations(). The grid and the axis are really helpfull.

nabr

@kfrajer nice.

@mossmossmoss As far as i understand it, you need a Picking Library .

T_D

Some short, updated, example i did for myself a while ago to handle interactive 3D mouse actions. It handle picking and object transformations.

The code is (at some points) probably a bit complicated, some matrix stuff, lots of PShapes, etc. But it is kind of an universal solution and also quite efficient ... I tried to add some comments.

import peasy.PeasyCam;

//
// Demo: Picking, Coordinate Transform (screen<->world), Object Transform (screen-aligned)
//
// author: Thomas Diewald
//
// controls:
//
// 'w' + mouse: Move objects (screen-aligned)
//
// LMB: orbit
// MMB: pan
// RMB: zoom
//



PeasyCam cam;

PGraphics3D pg_pick;

CoordinateTransform transform = new CoordinateTransform();

SceneObject[] scene_objects;

PShape shp_gizmo;
PShape shp_grid;
PShape group_render;
PShape group_pick;

public void settings(){
  size(1280, 720, P3D); 
  smooth(8);
}


public void setup() {
  
  cam = new PeasyCam(this, 1000);
  
  perspective(60 * PI/180f, width/(float)height, 1, 200000);
  
  // picking buffer
  pg_pick = (PGraphics3D) createGraphics(width, height, P3D);
  pg_pick.smooth(0);
  
  createScene();
}



public void draw(){
  
  // disable peasycam when objects are moved
  cam.setActive(!MOVE_OBJECT);
  
  // primary graphics
  PGraphics3D pg_canvas = (PGraphics3D) this.g;

  // render scene to the picking buffer
  pg_pick.beginDraw();
  pg_pick.modelview .set(pg_canvas.modelview);   // copy current modelview
  pg_pick.projection.set(pg_canvas.projection);  // copy current projection
  pg_pick.updateProjmodelview();
  displayScene(pg_pick);
  pg_pick.endDraw();
  pg_pick.loadPixels();
  
  // update moving objects, mouse over, etc...
  updateMouseAction();
  
  // display final scene
  displayScene(pg_canvas);
  

  // picking buffer
  // cam.beginHUD();
  // noLights();
  // background(0);
  // image(pg_pick, 0,0);
  // cam.endHUD();
  
  String txt_fps = String.format(getClass().getName()+ "   [objects %d]  [fps %6.2f]", scene_objects.length, frameRate);
  surface.setTitle(txt_fps);
}


// create scene objects, gizmo, grid, etc...
public void createScene(){
  
  createGizmo(600);
  createGrid(15, 1200);

  group_render = createShape(GROUP);
  group_pick   = createShape(GROUP);
  
  scene_objects = new SceneObject[5000];
  
  colorMode(HSB, 360, 100, 100);
  
  for(int i = 0; i < scene_objects.length; i++){
    float pr = 300;
    float posx = random(-pr,pr);
    float posy = random(-pr,pr);
    float posz = random(-pr,pr);
    
    float dmin = 20;
    float dmax = 40;
    float dimx = random(dmin, dmax);
    float dimy = random(dmin, dmax);
    float dimz = random(dmin, dmax);
    
    float colr = random(40, 240);
    float colg = random(30,70);
    float colb = 100;
    
    SceneObject obj = new SceneObject(i, scene_objects);
    obj.fill       = color(colr, colg, colb);
    obj.shp_render = createShape(BOX, dimx, dimy, dimz);
    obj.shp_pick   = createShape(BOX, dimx, dimy, dimz);
      
    // render shape style
    obj.shp_render.setStroke(true);
    obj.shp_render.setStroke(color(0));
    obj.shp_render.setStrokeWeight(1f);
    obj.shp_render.setFill(true);
    obj.shp_render.setFill(obj.fill);

    // picking shape style
    obj.shp_pick.setStroke(false);
    obj.shp_pick.setFill(true);
    obj.shp_pick.setFill(obj.pick);

    // apply some local transformations
    obj.mat.reset();
    obj.mat.translate(posx, posy, posz);
    obj.mat.rotateZ(random(PI));
    obj.mat.rotateX(random(PI));
    obj.mat.rotateY(random(PI));
    obj.udpateShapesTransform();
    
    // add shapes to global group (for fast rendering)
    group_render.addChild(obj.shp_render);
    group_pick  .addChild(obj.shp_pick);
  }
  
  colorMode(RGB, 255, 255, 255);
}


public void createGizmo(float s){
  shp_gizmo = createShape();
  shp_gizmo.beginShape(LINES);
  shp_gizmo.strokeWeight(1.5f);
  shp_gizmo.stroke(255,0,0); shp_gizmo.vertex(0, 0, 0); shp_gizmo.vertex(s, 0, 0);
  shp_gizmo.stroke(0,255,0); shp_gizmo.vertex(0, 0, 0); shp_gizmo.vertex(0, s, 0);
  shp_gizmo.stroke(0,0,255); shp_gizmo.vertex(0, 0, 0); shp_gizmo.vertex(0, 0, s);
  shp_gizmo.endShape();
}


public void createGrid(int n, float s){
  shp_grid = createShape();
  shp_grid.beginShape(LINES);
  shp_grid.strokeWeight(0.5f);
  shp_grid.stroke(128);
  shp_grid.translate(-s/2, -s/2);
  float grid_d = s/(n-1);
  for(int y = 0; y < n; y++){
    for(int x = 0; x < n; x++){
      float sx = x * grid_d;
      float sy = y * grid_d;
      shp_grid.vertex(sx, 0, 0); shp_grid.vertex(sx, s, 0);
      shp_grid.vertex(0, sy, 0); shp_grid.vertex(s, sy, 0);    
    }
  }
  shp_grid.endShape();
}



public void displayScene(PGraphics3D canvas){
  if(canvas == pg_pick){
    canvas.blendMode(REPLACE);
    canvas.clear();
    canvas.noLights();
    canvas.shape(group_pick);
  } else {
    canvas.blendMode(BLEND);
    canvas.background(255);
    canvas.lights();
    canvas.shape(shp_gizmo);
    canvas.shape(shp_grid);
    canvas.shape(group_render);
  }
}



SceneObject obj_move = null;
PMatrix3D obj_mat_copy = new PMatrix3D();
int pick_id_prev = 0;

float[] sel_off = null;

public void updateMouseAction(){
  if(obj_move == null){
    if(mouseX >= 0 && mouseX <  pg_pick.width && mouseY >= 0 && mouseY <  pg_pick.height) {
      int pick_id_curr = pg_pick.pixels[mouseY * pg_pick.width + mouseX];
      
      // no mouse-over, just return
      if(pick_id_curr == 0){
        scene_objects[pick_id_prev].shp_render.setFill(scene_objects[pick_id_prev].fill);
        return;
      }

      // mouse-over, change fill color
      pick_id_curr &= 0x00FFFFFF;
      scene_objects[pick_id_prev].shp_render.setFill(scene_objects[pick_id_prev].fill);
      scene_objects[pick_id_curr].shp_render.setFill(color(255, 64, 0));
      pick_id_prev = pick_id_curr;
      
      // mouse-over AND selection mode is active -> keep the object, and
      // keep a backup of its transformation matrix
      if(SELECT_OBJECT){
        obj_move = scene_objects[pick_id_curr];
        obj_mat_copy.set(obj_move.mat);
        sel_off = null;
      }
    } 
  }
  
  // object selected and ready to be moved
  if(obj_move != null){
    PGraphics3D pg_canvas = (PGraphics3D) this.g;
    
    // build object matrix
    PMatrix3D mvp = pg_canvas.projmodelview.get();
    mvp.apply(obj_mat_copy);
    
    transform.set(pg_canvas.width, pg_canvas.height, mvp);

    // transform object to screen-coords
    float[] screen = new float[4];
    float[] world  = new float[4];
    transform.worldToScreen(world, screen);
    
    // respect mouse-offset (to object center)
    if(sel_off == null){
      sel_off = new float[2];
      sel_off[0] = mouseX - screen[0];
      sel_off[1] = mouseY - screen[1];
    }
    
    // transform object (new screen-coords!) back to world-coords
    screen[0] = mouseX - sel_off[0];
    screen[1] = mouseY - sel_off[1];
    transform.screenToWorld(screen, world);

    // modify object matrix
    obj_move.mat.set(obj_mat_copy);
    obj_move.mat.translate(world[0], world[1], world[2]);
    obj_move.udpateShapesTransform();
  }
}




boolean MOVE_OBJECT = false;
boolean SELECT_OBJECT = false;

public void keyPressed(){
  if(key == 'w'){
    MOVE_OBJECT = true;
  }
}

public void keyReleased(){
  MOVE_OBJECT = false;
}

public void mousePressed(){
  SELECT_OBJECT = MOVE_OBJECT;
}

public void mouseReleased(){
  SELECT_OBJECT = false;
  obj_move = null;
}





static class SceneObject{
  final int pick;
  int fill;
  PShape shp_render;
  PShape shp_pick;
  PMatrix3D mat = new PMatrix3D();

  public SceneObject(int ID, SceneObject[] list){
    this.pick = 0xFF000000 | ID;
    list[ID] = this;
  }
  
  public void udpateShapesTransform(){
    shp_render.resetMatrix();
    shp_render.applyMatrix(mat);
    
    shp_pick.resetMatrix();
    shp_pick.applyMatrix(mat);
  }
}



/**
 * class for transforming coordinates: screen <-> world
 * 
 * source: github.com/diwi/PixelFlow/blob/master/src/com/thomasdiewald/pixelflow/java/utils/DwCoordinateTransform.java
 * 
 * author: Thomas Diewald
 *
 */
static public class CoordinateTransform{
  public float canvas_w, canvas_h;

  public float[] world  = new float[4];
  public float[] screen = new float[4];
  
  public PMatrix3D mat_projmodelview     = new PMatrix3D();
  public PMatrix3D mat_projmodelview_inv = new PMatrix3D();

  public void set(float canvas_w, float canvas_h, PMatrix3D mvp){
    this.canvas_w = canvas_w;
    this.canvas_h = canvas_h;
    this.mat_projmodelview    .set(mvp);
    this.mat_projmodelview_inv.set(mvp);
    this.mat_projmodelview_inv.invert();
  }

  // this transforms a coordinate (vec4) from world-space to screen-space
  public void worldToScreen(float[] src_world, float[] dst_screen){
    src_world[3] = 1;
    mat_projmodelview.mult(src_world, dst_screen);
    float w_inv = 1f/dst_screen[3];
    dst_screen[0] = ((dst_screen[0] * w_inv) * +0.5f + 0.5f) * canvas_w;
    dst_screen[1] = ((dst_screen[1] * w_inv) * -0.5f + 0.5f) * canvas_h;
    dst_screen[2] = ((dst_screen[2] * w_inv) * +0.5f + 0.5f);
  }
  
  // this transforms a coordinate (vec4) from screen-space to world-space
  public void screenToWorld(float[] src_screen, float[] dst_world){
    src_screen[0] = ((src_screen[0]/(float) canvas_w) * 2 - 1) * +1;
    src_screen[1] = ((src_screen[1]/(float) canvas_h) * 2 - 1) * -1;
    src_screen[2] = ((src_screen[2]                 ) * 2 - 1) * +1;
    src_screen[3] = 1;
    mat_projmodelview_inv.mult(src_screen, dst_world);
    float w_inv = 1f/dst_world[3];
    dst_world[0] *= w_inv;
    dst_world[1] *= w_inv;
    dst_world[2] *= w_inv;
  }
  
}

nabr

Further Reading / Advanced Techniques / Inspiration

https://en.wikipedia.org/wiki/Gimbal_lock

jeremydouglass

@T_D how does your picking example above compare to the Picking Library -- are they both essentially the same approach?