How to move a 3D object according to the screen's XY axis, instead of the world's X,Y,Z(PeasyCam)

hi! Im writing a 3D sketch in which the user rotates the camera with peasyCam while left clicking and moving the mouse. The thing is that I want to move the objects while right click is pressed so that the user can drag the object across the screen's X and Y axis. Of course I know how to use mouseX and mouseY inputs to modify the translation but only across the 3D space coordinates as it shows on the GIF below:

imgur.com/nWbpGhH

example code of whats happening in the image:

import peasy.*;
import peasy.org.apache.commons.math.*;
import peasy.org.apache.commons.math.geometry.*;

PeasyCam cam;

float x=15;float y=15; float z=15;
float e;

void setup(){
  size (700,700,P3D);
  cam = new PeasyCam(this, 200);
  cam.setRightDragHandler(null);


}
void draw(){
  background(0);
  pushMatrix();
  translate(5, 5, 0);
  fill(255);
  stroke(255);
  sphere(5);
  popMatrix();
  pushMatrix();
  fill(255,0,0);
  stroke(255,0,0);
  translate(x, y, z);
  sphere(5);
  popMatrix();
  stroke(0,0,255);
  line(5,5,0,x,y,z);
  //obvoiusly not working method
  if(mousePressed && (mouseButton == RIGHT)){
    x= x+(mouseX-pmouseX);
    y= y+(mouseY-pmouseY);
  }
}
void mouseWheel(MouseEvent event) {
  e = event.getCount();
  z=z+e;
  println(e);
}
void mousePressed(){
  if (mouseButton==RIGHT){
    cam.setActive(false);
  }
}
void mouseReleased(){
 cam.setActive(true);
}

What I would need is to be able to drag the sphere only on the screens X/Y axis, at a fixed Z just like image below shows(simple simulation I made of the behaviour im looking for).

imgur.com/jc8wVWy

PeasyCam is for exploring the 3D space. The question might be difficult to undesrtand. The problem is about moving the object on the 3D world, using the screen/canvas 2D coordinates to make the object follow the cursor's movement. If the mouse goes to the left (x axis decreases), the object should move to the left on the screen, and not just on the worlds X axis. This is how the second example image behaves, but achieved by just simulating the 3D space with no actual rotations to the x,y,z axis.

I've been scratching my head with this thing and I cant seem to figure it out. I wouldn't have asked here otherwise. Thanks in advance guys.

Tagged:

Answers

  • edited May 2017

    Here is my attempt. First thing is to have a reference frame to be able to make comparisons and see things more clear. The program begin with an axis drawn,RGB for x,y,z. In this mode the cam is active. Then if you hit the 'g' key (for grid) you get a grid. The cam is disable. Notice that only in the grid state you can unlock the mouse. The mouse always starts lock in the grid state. If you hit m, you can toggle the mouse's state. The mouseX/mouseY dynamics changes the position of your sphere of interest and the sphere's position is mapped to the width and height of your sketch. Remember, you need to lock/unlock the mouse to be able to adjust x and y. Remark: Locking/unlocking the mouse's state in non-grid mode just does nothing.

    You might consider checking the getRotations/getPosition from the peasycam's reference page. You could also use that information to map the 2D plane of your sketch into the 3D world set by the peasy cam. However, I think this solution below could be good but probably not sufficient.

    Kf

    import peasy.*;
    import peasy.org.apache.commons.math.*;
    import peasy.org.apache.commons.math.geometry.*;  
    

    //axisgrid gridaxis scene rotation

    PeasyCam cam;
    
    float x=15;
    float y=15; 
    float z=15;
    float e;
    
    boolean showGrid=false;
    boolean showAxis=true;  //false;
    boolean lockMouse=true;
    
    void setup() {
      size (700, 700, P3D);
      cam = new PeasyCam(this, 700);
      cam.setRightDragHandler(null);
    }
    
    
    void draw() {
      background(0);
    
      if (showGrid) drawGrid(200, 200, 100, 2);
      if (showAxis) drawAxis(200);
    
      pushMatrix();
      //translate(5, 5, 0);
      fill(255);
      stroke(255);
      sphere(5);
      popMatrix();
      pushMatrix();
      fill(255, 0, 0);
      stroke(255, 0, 0);
      translate(x, y, z);
      sphere(5);
    
    
      //obvoiusly not working method
      //if (mousePressed && (mouseButton == RIGHT)) {
      if (showGrid==true && lockMouse==false) {
        //x= x+(mouseX-pmouseX);
        //y= y+(mouseY-pmouseY);
        x=mouseX;
        y=mouseY;
      }
      popMatrix();
    
      stroke(0, 0, 255);
      line(0, 0, 0, x, y, z);
    }
    void mouseWheel(MouseEvent event) {
      e = event.getCount();
      z=z+e;
      println(e);
    }
    void mousePressed() {
      if (mouseButton==RIGHT) {
        //cam.setActive(false);
      }
    }
    void mouseReleased() {
      //cam.setActive(true);
    }
    
    void keyReleased() {
    
    
      //if (key=='a')
      //showAxis=!showAxis;
    
    
      if (key=='g') {
        showGrid=!showGrid;
        lockMouse=true;   //Mouse always lock when changing states
        cam.setActive(!showGrid);
      }
    
      if (key=='m' && showGrid==true) {
        lockMouse=!lockMouse;
      }
    }
    
    
    
    // -------------------------------------------------------
    void drawAxis(float len) {
      drawAxis(len, len, len);
    }
    
    
    // -------------------------------------------------------
    void drawAxis(float len1, float len2, float len3) {
    
      pushStyle();
      strokeWeight(3);
    
      stroke(255, 0, 0);  //RED
      line(0, 0, 0, len1, 0, 0);
    
      stroke(0, 255, 0);  //GREEN
      line(0, 0, 0, 0, len2, 0);
    
      stroke(0, 0, 255);  //BLUE
      line(0, 0, 0, 0, 0, len3);
    
      popStyle();
    }
    
    // -------------------------------------------------------
    // @Args:  Plane: 0=YZ  1=XZ  2=XY 
    void drawGrid(int size, int w, int h, int plane) {
      pushStyle();
      noFill();
      if (plane == 0) stroke(255, 0, 0);
      if (plane == 1) stroke(0, 255, 0);
      if (plane == 2) stroke(0, 0, 255);
      int total = w * h;
      int tw = w * size;
      int th = h * size;
      beginShape(LINES);
      for (int i = 0; i < total; i++) {
        int x = (i % w) * size;
        int y = (i / w) * size;
        if (plane == 0) {
          vertex(0, x, 0);
          vertex(0, x, th);
          vertex(0, 0, y);
          vertex(0, tw, y);
        }
        if (plane == 1) {
          vertex(x, 0, 0);
          vertex(x, 0, th);
          vertex(0, 0, y);
          vertex(tw, 0, y);
        }
        if (plane == 2) {
          vertex(x, 0, 0);
          vertex(x, th, 0);
          vertex(0, y, 0);
          vertex(tw, y, 0);
        }
      }
      endShape();
      popStyle();
    }
    
  • Thanks kf! your code makes it pretty easy to understand the problem in question . I've been trying to grasp the concept of rotations to use peasycam's getRotations() to implement a solution, but I haven't figured it out yet (I probably need to know a little bit more about rotation matricies). Thanks a lot for spending the time man, I'm going to use your code to experiment with getRotations(). The grid and the axis are really helpfull.

  • @kfrajer nice.

    @mossmossmoss As far as i understand it, you need a Picking Library .

  • T_DT_D
    edited May 2017

    Some short, updated, example i did for myself a while ago to handle interactive 3D mouse actions. It handle picking and object transformations.

    The code is (at some points) probably a bit complicated, some matrix stuff, lots of PShapes, etc. But it is kind of an universal solution and also quite efficient ... I tried to add some comments.

    
    import peasy.PeasyCam;
    
    //
    // Demo: Picking, Coordinate Transform (screen<->world), Object Transform (screen-aligned)
    //
    // author: Thomas Diewald
    //
    // controls:
    //
    // 'w' + mouse: Move objects (screen-aligned)
    //
    // LMB: orbit
    // MMB: pan
    // RMB: zoom
    //
    
    
    
    PeasyCam cam;
    
    PGraphics3D pg_pick;
    
    CoordinateTransform transform = new CoordinateTransform();
    
    SceneObject[] scene_objects;
    
    PShape shp_gizmo;
    PShape shp_grid;
    PShape group_render;
    PShape group_pick;
    
    public void settings(){
      size(1280, 720, P3D); 
      smooth(8);
    }
    
    
    public void setup() {
      
      cam = new PeasyCam(this, 1000);
      
      perspective(60 * PI/180f, width/(float)height, 1, 200000);
      
      // picking buffer
      pg_pick = (PGraphics3D) createGraphics(width, height, P3D);
      pg_pick.smooth(0);
      
      createScene();
    }
    
    
    
    public void draw(){
      
      // disable peasycam when objects are moved
      cam.setActive(!MOVE_OBJECT);
      
      // primary graphics
      PGraphics3D pg_canvas = (PGraphics3D) this.g;
    
      // render scene to the picking buffer
      pg_pick.beginDraw();
      pg_pick.modelview .set(pg_canvas.modelview);   // copy current modelview
      pg_pick.projection.set(pg_canvas.projection);  // copy current projection
      pg_pick.updateProjmodelview();
      displayScene(pg_pick);
      pg_pick.endDraw();
      pg_pick.loadPixels();
      
      // update moving objects, mouse over, etc...
      updateMouseAction();
      
      // display final scene
      displayScene(pg_canvas);
      
    
      // picking buffer
      // cam.beginHUD();
      // noLights();
      // background(0);
      // image(pg_pick, 0,0);
      // cam.endHUD();
      
      String txt_fps = String.format(getClass().getName()+ "   [objects %d]  [fps %6.2f]", scene_objects.length, frameRate);
      surface.setTitle(txt_fps);
    }
    
    
    // create scene objects, gizmo, grid, etc...
    public void createScene(){
      
      createGizmo(600);
      createGrid(15, 1200);
    
      group_render = createShape(GROUP);
      group_pick   = createShape(GROUP);
      
      scene_objects = new SceneObject[5000];
      
      colorMode(HSB, 360, 100, 100);
      
      for(int i = 0; i < scene_objects.length; i++){
        float pr = 300;
        float posx = random(-pr,pr);
        float posy = random(-pr,pr);
        float posz = random(-pr,pr);
        
        float dmin = 20;
        float dmax = 40;
        float dimx = random(dmin, dmax);
        float dimy = random(dmin, dmax);
        float dimz = random(dmin, dmax);
        
        float colr = random(40, 240);
        float colg = random(30,70);
        float colb = 100;
        
        SceneObject obj = new SceneObject(i, scene_objects);
        obj.fill       = color(colr, colg, colb);
        obj.shp_render = createShape(BOX, dimx, dimy, dimz);
        obj.shp_pick   = createShape(BOX, dimx, dimy, dimz);
          
        // render shape style
        obj.shp_render.setStroke(true);
        obj.shp_render.setStroke(color(0));
        obj.shp_render.setStrokeWeight(1f);
        obj.shp_render.setFill(true);
        obj.shp_render.setFill(obj.fill);
    
        // picking shape style
        obj.shp_pick.setStroke(false);
        obj.shp_pick.setFill(true);
        obj.shp_pick.setFill(obj.pick);
    
        // apply some local transformations
        obj.mat.reset();
        obj.mat.translate(posx, posy, posz);
        obj.mat.rotateZ(random(PI));
        obj.mat.rotateX(random(PI));
        obj.mat.rotateY(random(PI));
        obj.udpateShapesTransform();
        
        // add shapes to global group (for fast rendering)
        group_render.addChild(obj.shp_render);
        group_pick  .addChild(obj.shp_pick);
      }
      
      colorMode(RGB, 255, 255, 255);
    }
    
    
    public void createGizmo(float s){
      shp_gizmo = createShape();
      shp_gizmo.beginShape(LINES);
      shp_gizmo.strokeWeight(1.5f);
      shp_gizmo.stroke(255,0,0); shp_gizmo.vertex(0, 0, 0); shp_gizmo.vertex(s, 0, 0);
      shp_gizmo.stroke(0,255,0); shp_gizmo.vertex(0, 0, 0); shp_gizmo.vertex(0, s, 0);
      shp_gizmo.stroke(0,0,255); shp_gizmo.vertex(0, 0, 0); shp_gizmo.vertex(0, 0, s);
      shp_gizmo.endShape();
    }
    
    
    public void createGrid(int n, float s){
      shp_grid = createShape();
      shp_grid.beginShape(LINES);
      shp_grid.strokeWeight(0.5f);
      shp_grid.stroke(128);
      shp_grid.translate(-s/2, -s/2);
      float grid_d = s/(n-1);
      for(int y = 0; y < n; y++){
        for(int x = 0; x < n; x++){
          float sx = x * grid_d;
          float sy = y * grid_d;
          shp_grid.vertex(sx, 0, 0); shp_grid.vertex(sx, s, 0);
          shp_grid.vertex(0, sy, 0); shp_grid.vertex(s, sy, 0);    
        }
      }
      shp_grid.endShape();
    }
    
    
    
    public void displayScene(PGraphics3D canvas){
      if(canvas == pg_pick){
        canvas.blendMode(REPLACE);
        canvas.clear();
        canvas.noLights();
        canvas.shape(group_pick);
      } else {
        canvas.blendMode(BLEND);
        canvas.background(255);
        canvas.lights();
        canvas.shape(shp_gizmo);
        canvas.shape(shp_grid);
        canvas.shape(group_render);
      }
    }
    
    
    
    SceneObject obj_move = null;
    PMatrix3D obj_mat_copy = new PMatrix3D();
    int pick_id_prev = 0;
    
    float[] sel_off = null;
    
    public void updateMouseAction(){
      if(obj_move == null){
        if(mouseX >= 0 && mouseX <  pg_pick.width && mouseY >= 0 && mouseY <  pg_pick.height) {
          int pick_id_curr = pg_pick.pixels[mouseY * pg_pick.width + mouseX];
          
          // no mouse-over, just return
          if(pick_id_curr == 0){
            scene_objects[pick_id_prev].shp_render.setFill(scene_objects[pick_id_prev].fill);
            return;
          }
    
          // mouse-over, change fill color
          pick_id_curr &= 0x00FFFFFF;
          scene_objects[pick_id_prev].shp_render.setFill(scene_objects[pick_id_prev].fill);
          scene_objects[pick_id_curr].shp_render.setFill(color(255, 64, 0));
          pick_id_prev = pick_id_curr;
          
          // mouse-over AND selection mode is active -> keep the object, and
          // keep a backup of its transformation matrix
          if(SELECT_OBJECT){
            obj_move = scene_objects[pick_id_curr];
            obj_mat_copy.set(obj_move.mat);
            sel_off = null;
          }
        } 
      }
      
      // object selected and ready to be moved
      if(obj_move != null){
        PGraphics3D pg_canvas = (PGraphics3D) this.g;
        
        // build object matrix
        PMatrix3D mvp = pg_canvas.projmodelview.get();
        mvp.apply(obj_mat_copy);
        
        transform.set(pg_canvas.width, pg_canvas.height, mvp);
    
        // transform object to screen-coords
        float[] screen = new float[4];
        float[] world  = new float[4];
        transform.worldToScreen(world, screen);
        
        // respect mouse-offset (to object center)
        if(sel_off == null){
          sel_off = new float[2];
          sel_off[0] = mouseX - screen[0];
          sel_off[1] = mouseY - screen[1];
        }
        
        // transform object (new screen-coords!) back to world-coords
        screen[0] = mouseX - sel_off[0];
        screen[1] = mouseY - sel_off[1];
        transform.screenToWorld(screen, world);
    
        // modify object matrix
        obj_move.mat.set(obj_mat_copy);
        obj_move.mat.translate(world[0], world[1], world[2]);
        obj_move.udpateShapesTransform();
      }
    }
    
    
    
    
    boolean MOVE_OBJECT = false;
    boolean SELECT_OBJECT = false;
    
    public void keyPressed(){
      if(key == 'w'){
        MOVE_OBJECT = true;
      }
    }
    
    public void keyReleased(){
      MOVE_OBJECT = false;
    }
    
    public void mousePressed(){
      SELECT_OBJECT = MOVE_OBJECT;
    }
    
    public void mouseReleased(){
      SELECT_OBJECT = false;
      obj_move = null;
    }
    
    
    
    
    
    static class SceneObject{
      final int pick;
      int fill;
      PShape shp_render;
      PShape shp_pick;
      PMatrix3D mat = new PMatrix3D();
    
      public SceneObject(int ID, SceneObject[] list){
        this.pick = 0xFF000000 | ID;
        list[ID] = this;
      }
      
      public void udpateShapesTransform(){
        shp_render.resetMatrix();
        shp_render.applyMatrix(mat);
        
        shp_pick.resetMatrix();
        shp_pick.applyMatrix(mat);
      }
    }
    
    
    
    /**
     * class for transforming coordinates: screen <-> world
     * 
     * source: github.com/diwi/PixelFlow/blob/master/src/com/thomasdiewald/pixelflow/java/utils/DwCoordinateTransform.java
     * 
     * author: Thomas Diewald
     *
     */
    static public class CoordinateTransform{
      public float canvas_w, canvas_h;
    
      public float[] world  = new float[4];
      public float[] screen = new float[4];
      
      public PMatrix3D mat_projmodelview     = new PMatrix3D();
      public PMatrix3D mat_projmodelview_inv = new PMatrix3D();
    
      public void set(float canvas_w, float canvas_h, PMatrix3D mvp){
        this.canvas_w = canvas_w;
        this.canvas_h = canvas_h;
        this.mat_projmodelview    .set(mvp);
        this.mat_projmodelview_inv.set(mvp);
        this.mat_projmodelview_inv.invert();
      }
    
      // this transforms a coordinate (vec4) from world-space to screen-space
      public void worldToScreen(float[] src_world, float[] dst_screen){
        src_world[3] = 1;
        mat_projmodelview.mult(src_world, dst_screen);
        float w_inv = 1f/dst_screen[3];
        dst_screen[0] = ((dst_screen[0] * w_inv) * +0.5f + 0.5f) * canvas_w;
        dst_screen[1] = ((dst_screen[1] * w_inv) * -0.5f + 0.5f) * canvas_h;
        dst_screen[2] = ((dst_screen[2] * w_inv) * +0.5f + 0.5f);
      }
      
      // this transforms a coordinate (vec4) from screen-space to world-space
      public void screenToWorld(float[] src_screen, float[] dst_world){
        src_screen[0] = ((src_screen[0]/(float) canvas_w) * 2 - 1) * +1;
        src_screen[1] = ((src_screen[1]/(float) canvas_h) * 2 - 1) * -1;
        src_screen[2] = ((src_screen[2]                 ) * 2 - 1) * +1;
        src_screen[3] = 1;
        mat_projmodelview_inv.mult(src_screen, dst_world);
        float w_inv = 1f/dst_world[3];
        dst_world[0] *= w_inv;
        dst_world[1] *= w_inv;
        dst_world[2] *= w_inv;
      }
      
    }
    
    
  • edited May 2017
    Further Reading / Advanced Techniques / Inspiration

    https://en.wikipedia.org/wiki/Gimbal_lock

  • @T_D how does your picking example above compare to the Picking Library -- are they both essentially the same approach?

Sign In or Register to comment.