Working with GoPro Hero5 sensors

Kajuna

I've been working on a way to use the sensors built in the latest GoPro to show the forces applied to the camera and use that info to also stabilise the image without using the built-in digital stabiliser. I finally found a way to extract the data, that I explain here: https://community.gopro.com/t5/Hero5-Metadata-Visualisation/Extracting-the-metadata-in-a-useful-format/gpm-p/40293 (It's tedious, I know). Edit: Now the data can be extracted easily. Just Uncompress this folder and drop your files on GPMD2CSV.bat (for Windows): http://tailorandwayne.com/GPMD2CSV.zip

So I want to share the code I'm using because I'm sure it can be improved and I want more people to use these data, so that GoPro cares about us and provides appropriate tools. I worked with my Hero5 Session, but if you own a H5 Black you will also have GPS data to work with. For the sketch to run, you need to extract the data in csv format as I explain in the previous link, then rename the video file as input.mp4. It is important to set the variable for the vertical field of view and the size() of the sketch, which should match the original file ratio. The sketch will stabilise the image based on the gyro and display info for the gyroscope (in RPM), accelerometer (G force) and temperature (Celsius).

Some issues I've found: - There is drift. Some of the functions compensate for it, but are quite imperfect. - Related to that. There is no compass sensor, so we can't have an absolute orientation to force a flat horizon or fully compensate the gyro drift. - For some reason very quick spins are not properly compensated. I would guess the sensor is not accurate at high angular speeds, but there might also be something wrong in my code.

Things to note: - I find the best way to use the stabilisation code to first optically compensate the image and generate extended margins. This is a good tutorial on how to do that (but this slows the workflow even more, of course): http://abekislevitz.com/gopro-tutorial-fisheye-removal/ - I think most customisable variables are explained, but let me know if they're not clear.

And here's the code. Please feel free to suggest improvements or alternatives.

Kajuna

//To export GoPro metadata: https://community.gopro.com/t5/Cameras/Hero5-Session-Telemetry/td-p/22962/highlight/false/page/3#M20188
//credit for the data extraction goes to https://github.com/stilldavid/gopro-utils

//TODO
//update if firmware v2.00 adds useful info (shutter spped, iso?) Problematic for now as firmware appears to have many bugs

import processing.video.*;

Table table;  //will contain the input gyro file
Table acclTable;  //will contain the input accelerometer file
Table tempTable;  //will contain the input temperature file
PVector rotation = new PVector (0,0,0);
float pixelsPerRad = 0;
float lastMilliseconds = 0;  //time of last info applied
float acclLastMilliseconds = 0;  //time of last info applied
float tempLastMilliseconds = 0;  //time of last info applied
int currentFrame = 0;  
int lastRow = 0;  
float realWidth;  //deduced from conditional realheight and sketch ratio
PImage currentImage = new PImage();
int savedFrames = 0;
Movie myMovie;
int currentFrameLoad = 0;
boolean areFramesSaved = false;
int validFrame;
boolean setupFinished = false;
boolean finishingSetup = false;
boolean skipMovie = false;
int acclLastRow = 0;  //to read accl csv
int tempLastRow = 0;  //to read accl csv
float tempDisplay = 0; 

//Customisable
//IMPORTANT
float vFov = 69.7;
//vertical field of view in degrees. H5 Session 16:9 = 69.7 |||||| 4:3 = 94.5  https://gopro.com/help/articles/Question_Answer/HERO5-Session-Field-of-View-FOV-Information
// H5 Black: https://gopro.com/help/articles/Question_Answer/HERO5-Black-Field-of-View-FOV-Information

//OPTIONAL
float augment = 1; //increase size to hide borders (there are better ways to hide / regenerate borders, with video editing software). 1 = no resize.
boolean embedData = false;  //whether or not metadata is printed on the output images (you see it anyway)
float goproRate = 24f;  //framerate of video will be overwritten if a video file with a different framerate is present. Important when working with frames only
int firstDataFrame = 0;  //first frame to analyse from original GoPro file
boolean dataVideoSync = true; // False to start frames at 0 when stills have been extracted a section of the original GoPro file
float rotationZoffset = 0.0;  //if horizon is generally tilted, compensate here. Radianss
float realHeight = 0;//zero for auto (no optically corrected clips) //real use in pixels of the fov in the input file (useful when optically compensated) - 4k optically compensated: approx 2378 (check in AE)
int offset = -30;//in milliseconds, can be float for better adjustment
String filename = "input";//file to search for
String fileType = ".jpg";//extension
String videoFileType = ".mp4";//if loading from a video
int digits = 5;//number of zeros. Will be overwritten if frames are extracted from the video file
float smooth = .96;//reduce x and y rotation progressively, 1 for not smoothing
float smoothZ= .996;//reduce z rotation progressively, 1 
float limitY = .1;//limit movement per axis to percentage of image/angle. -1 = no limit
float limitX = .2;//if optically compensated, X can be wider
float limitZ = -1;//not really a percentage, -1 FOR FLAT HORIZON (MotoGP gyrocam style.
//Note: the H5 does not have enough sensors to guess the absolute orientation, and there appears to be some drift caused by small inaccuracies that add up so an always flat horizon is not possible, IMO)
boolean rescale = true;  //scale to sketch size

//NOTE: remember to set the proper sketch size for your video in setup()

void setup() {
  size(1280, 720);  //needs the same ratio as the input usable area, 16:9 4:3 etc
  smooth();
  imageMode(CENTER);  //draw from centre
  if (realHeight == 0) {
    realHeight = height;
  }
  realWidth = realHeight*(width/height);//for limiting movement
  pixelsPerRad = realHeight/radians(vFov);
  File movieFile = new File(dataPath(filename+videoFileType));  //look for movie file
  if (movieFile.exists()) {  //check if a video file exists to launch it and deduce its info (duration, framerate)
    myMovie = new Movie(this, filename+videoFileType);  //start movie
    myMovie.play();
  } else {  //if no movie file we can continu
    finishSetup();
  }
}

void finishSetup() {  //we need to receive a movie frame to use its info (duration, framerate...) before we finish loading

  File csvFile = new File(dataPath("gyro.csv"));  //try to load CSV file
  if (csvFile.exists()) {
    table = loadTable("gyro.csv", "header");
    //find first row to analyse
    float currentMilliseconds = ((float(firstDataFrame))*(1000f/goproRate))+offset;//time we start at
    int checkRow = 0;
    while (float(table.getRow(checkRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
      checkRow++;
      lastMilliseconds = float(table.getRow(checkRow).getString("Milliseconds"));  //save current time
    }
    lastRow = checkRow;
    currentFrame = firstDataFrame;
    //create frames if they don't exist
    validFrame = currentFrame;  //frame count taking offsets into account
    if (!dataVideoSync) {
      validFrame -= firstDataFrame;
    }
    File stillsFile = new File(dataPath(filename+nf(validFrame,digits)+fileType));  //try to load necessary video files
    File movieFile = new File(dataPath(filename+videoFileType));

    if (stillsFile.exists()) {
      areFramesSaved = true;  //we can start
    } else if ( !movieFile.exists()) {  //otherwise
      println("No video file "+filename+videoFileType+" found in /data");
    }
  } else {
    println("No gyro.csv file found in /data");
  }

  File acclFile = new File(dataPath("accl.csv"));  //try to load CSV file
  if (acclFile.exists()) {
    acclTable = loadTable("accl.csv", "header");
    //load accl data
    float currentMilliseconds = ((float(firstDataFrame))*(1000f/goproRate))+offset;//time we start at
    int checkRow = 0;
    while (float(acclTable.getRow(checkRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
      checkRow++;
      acclLastMilliseconds = float(acclTable.getRow(checkRow).getString("Milliseconds"));  //save current time
    }
    acclLastRow = checkRow;
  }

  File tempFile = new File(dataPath("temp.csv"));  //try to load CSV file
  if (tempFile.exists()) {
    tempTable = loadTable("temp.csv", "header");
    //load temp data
    float currentMilliseconds = ((float(firstDataFrame))*(1000f/goproRate))+offset;//time we start at
    int checkRow = 0;
    while (float(tempTable.getRow(checkRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
      checkRow++;
      tempLastMilliseconds = float(tempTable.getRow(checkRow).getString("Milliseconds"));  //save current time
    }
    tempLastRow = checkRow;
  }

  setupFinished = true;
}

void draw() {
  if (setupFinished && areFramesSaved) {
    println("currentFrame: "+currentFrame+" lastRow: "+lastRow);
    int frameToLoad = currentFrame;
    if (!dataVideoSync) {  //copensate offsets
      frameToLoad -= firstDataFrame;
    }
    File stillsFile = new File(dataPath(filename+nf(frameToLoad,digits)+fileType));  //try to load frame

    if (stillsFile.exists()) {  //if we still have images
      currentImage = loadImage(filename+nf(frameToLoad,digits)+fileType);
      float currentMilliseconds = ((float(currentFrame+1))*(1000f/goproRate))+offset;//time we are at
      PVector gyroDisplay = new PVector(0,0,0);
      if (lastRow < table.getRowCount()) {
        while (float(table.getRow(lastRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
          float millisecondsDifference = (float(table.getRow(lastRow).getString("Milliseconds"))-lastMilliseconds)/1000f;  //how much time since last gyro?
          lastMilliseconds = float(table.getRow(lastRow).getString("Milliseconds"));  //save current time
          PVector preRotation = new PVector(float(table.getRow(lastRow).getString("GyroX")),float(table.getRow(lastRow).getString("GyroY")),float(table.getRow(lastRow).getString("GyroZ"))); //calculate perfect compensation of movements, (radians/time)*time passed
          preRotation.mult(millisecondsDifference);
          gyroDisplay.x += millisecondsDifference*float(table.getRow(lastRow).getString("GyroX"));  //update for displaying data, divide by time after loop
          gyroDisplay.y += millisecondsDifference*float(table.getRow(lastRow).getString("GyroY"));
          gyroDisplay.z += millisecondsDifference*float(table.getRow(lastRow).getString("GyroZ"));
          PVector smoother = new PVector(1,1,1);  //if all goes well, apply fully
          if (limitX >= 0) {  //if we have set limitations to X
            if ((rotation.x > 0 && preRotation.x > 0) || (rotation.x < 0 && preRotation.x < 0)) {  //check if rotation is going to increase
              smoother.x = constrain(1-((abs(rotation.x)*pixelsPerRad)/(limitX*realWidth)),0,1);  //smooth more the closest we are to the limit
            }
          }
          if (limitY >= 0) {  //if we have set limitations to Y
            if ((rotation.y > 0 && preRotation.y > 0) || (rotation.y < 0 && preRotation.y < 0)) {//the same for Y
              smoother.y = constrain(1-((abs(rotation.y)*pixelsPerRad)/(limitY*realHeight)),0,1);
            }
          }
          if (limitZ >= 0) {  //if we have set limitations to X
            if ((rotation.z > 0 && preRotation.z > 0) || (rotation.z < 0 && preRotation.z < 0)) {
              smoother.z =constrain(1-(abs(rotation.z)/(limitZ*QUARTER_PI)),0,1);    //this is a bit subjective, quarter pi is not a full circle but otherwise the limit is clearly wider than in X and Y, might need a separate setting.
            }
          }
          rotation.x += smoother.x*preRotation.x;   //apply rotations
          rotation.y += smoother.y*preRotation.y;
          rotation.z += smoother.z*preRotation.z; 
          lastRow++;  //next row
          if (lastRow >= table.getRowCount()) {
            println("CSV finished before video");
            break;
          } 
        }
          gyroDisplay.div(1000f/goproRate);  //multiply by seconds per frame, then convert to revs per minute
          gyroDisplay.div(TWO_PI);  //divide by one rev to get revs/second
          gyroDisplay.mult(60);  //multiply by 60 to get rpm
      } else {
        println("Skipping CSV");
      }

      //accelerometer calculations
      PVector acclDisplay = new PVector(0,0,0); //for displaying data
      if (acclLastRow < acclTable.getRowCount()) {
        int iterations = 0;
        while (float(acclTable.getRow(acclLastRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
          iterations++;
          acclLastMilliseconds = float(acclTable.getRow(acclLastRow).getString("Milliseconds"));  //save current time
          acclDisplay.x += float(acclTable.getRow(acclLastRow).getString("AcclX"));  //update for displaying data, divide by time after loop
          acclDisplay.y += float(acclTable.getRow(acclLastRow).getString("AcclY"));
          acclDisplay.z += float(acclTable.getRow(acclLastRow).getString("AcclZ"));
          acclLastRow++;  //next row
          if (acclLastRow >= acclTable.getRowCount()) {
            println("Accel CSV finished before video");
            break;
          } 
        }
        acclDisplay.div(iterations);  //divide by iterations to get the average in m/s2
        acclDisplay.div(9.80665);  //divide by gravity to express in Gs
      }
      //temperature calculations
      if (tempLastRow < tempTable.getRowCount()) {
        if (float(tempTable.getRow(tempLastRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
          tempLastMilliseconds = float(tempTable.getRow(tempLastRow).getString("Milliseconds"));  //save current time
          tempDisplay = float(tempTable.getRow(tempLastRow).getString("Temp"));  //update for displaying data,
          tempLastRow++;  //next row
          if (tempLastRow >= tempTable.getRowCount()) {
            println("Temp CSV finished before video");
          } 
        }
      }
    //let's draw  
      background(0);
      pushMatrix();
        translate(width/2, height/2);  //draw from centre
        while (rotation.y > PI) {  //simplify to +-360 degrees
          rotation.y -= TWO_PI;
        }
        while (rotation.y < -PI) {  //simplify to +-180 degrees
          rotation.y += TWO_PI;
        }
        while (rotation.x > PI) {
          rotation.x -= TWO_PI;
        }
        while (rotation.x < -PI) {
          rotation.x += TWO_PI;
        }
        while (rotation.z > PI) {
          rotation.z -= TWO_PI;
        }
        while (rotation.z < -PI) {
          rotation.z += TWO_PI;
        }
        rotation.x *= smooth;  //slowly go to the centre
        rotation.y *= smooth;
        rotation.z *= smoothZ;  //and straighten if we want so

        if (limitZ >= 0) {  //if we have set limitations to X
          rotate(constrain(-rotation.z,-limitZ*QUARTER_PI,limitZ*QUARTER_PI)+rotationZoffset);   //rotate first
        } else {
          rotate(-rotation.z+rotationZoffset);   //rotate first
        }
        if (limitX >= 0) {  //if we have set limitations to X
          translate(constrain(-rotation.x*pixelsPerRad,-limitX*realWidth,limitX*realWidth),0);    //then translate X
        } else {
          translate(-rotation.x*pixelsPerRad,0);    //then translate X
        }
        if (limitY >= 0) {  //if we have set limitations to Y
          translate(0,constrain(rotation.y*pixelsPerRad,-limitY*realHeight,limitX*realHeight));    //then translate Y
        } else {
          translate(0,rotation.y*pixelsPerRad);    //then translate Y
        }

        if (rescale) {
          image(currentImage, 0,0, width*augment, height*augment);     //print image
        } else {
          image(currentImage, 0,0, currentImage.width*augment, currentImage.height*augment);     //print image
        }
      popMatrix();
      if (embedData) {  //print data on screen if not printed on image
        displayData(gyroDisplay,acclDisplay,tempDisplay);  //display metadata after saving the image
      }
      saveFrame("gyrocam-"+nf(currentFrame,digits)+".jpg");    //save it
      if (!embedData) {  //print data on screen if not printed on image
        displayData(gyroDisplay,acclDisplay, tempDisplay);  //display metadata after saving the image
      }
      currentFrame++;  //next frame
    } else {  //end of video
      println("Video finished");
      noLoop();
    }
  }
}

static final String nfj(float n, int l, int r) {  //correct format problem with negative zeros
  String s = nfs(n, l, r);
  if (s.equals("-0.00")) s = " 0.00";
  if (s.equals("-0.0")) s = " 0.0";
  if (s.equals("-0.000")) s = " 0.000";
  return s;
}

void displayData(PVector g,PVector a, float t) {  //display metadata
  textSize(16);
  int shadow = 1;//distance to shadow
  String text = "GyroX:"+nfj(g.x,0,2)+"RPM\nGyroY:"+nfj(g.y,0,2)+"RPM\n"+"GyroZ:"+nfj(g.z,0,2)+"RPM\nAcclX:"+nfj(a.x,0,3)+"G\nAcclY:"+nfj(a.y,0,3)+"G\nAcclZ:"+nfj(a.z,0,3)+"G\nAccl:"+nfj(a.mag(),0,3)+"G\nTemp:"+nfj(t,0,1)+"ºC";
  fill(0);
  text(text, 10+shadow, 30+shadow); //draw shadow first
  fill(255);
  text(text, 10, 30); //then text
}

void movieEvent(Movie m) { //when a frame is ready
  m.read();
  if (!finishingSetup) {//finish loading everything according to the movie info
    finishingSetup = true;
    goproRate = myMovie.frameRate;  //we can now set the framerate based on the video file
    File stillsFile = new File(dataPath(filename+nf(validFrame,digits)+fileType));  //try to load necessary video files
    if (stillsFile.exists()) {  //if still frames are there
      skipMovie = true;  //don't extract movie frames
    }
    savedFrames = validFrame;    //count from the right one
    //check how many digits we need to save the frames
    float durationInFrames = myMovie.duration()*goproRate;
    digits = 1;
    while (durationInFrames > 1) {  //calculate digits needed to save the number of files and overwrite the preset
      durationInFrames /= 10;
      digits++;
    }

    stillsFile = new File(dataPath(filename+nf(validFrame,digits)+fileType));  //try to load necessary video files with this new value, which would be right if the movie has been extracted and the preset was wrong
    if (stillsFile.exists()) {  //if still frames are there
      skipMovie = true;//don't extract movie frames
    }
    finishSetup();
  }

  if (!skipMovie) {//only save frames if necessary
    m.save("/data/"+filename+nf(savedFrames,digits)+".jpg");    //save it
    println("Extracting frame "+savedFrames);
    savedFrames++;
    currentFrameLoad++; 
    if (m.time() >= m.duration()-(2*(1f/goproRate))) {  //if all loaded (actually a little earlier, since this is not precise and we don't want to risk it not firing)
      areFramesSaved = true;
    }
  }
}

Kajuna

Some changes I've made to minise the effects of sensor drift manually:

Declare this variable in the "customisable" section

float driftZ = 0.00309; //rads per second that we should compensate for in the Z axis. Might be different for each camera and situation. You can modify it on the go with the right and left keys to find the best value

Take it into account in calculations. After this line

preRotation.mult(millisecondsDifference);

add

          preRotation.z -= driftZ*millisecondsDifference;  //compensate for drift

Change

gyroDisplay.z += millisecondsDifference*float(table.getRow(lastRow).getString("GyroZ"));

for

gyroDisplay.z += millisecondsDifference*float(table.getRow(lastRow).getString("GyroZ"))-(driftZ*millisecondsDifference); //compensate for drift

After

if (!embedData) {  //print data on screen if not printed on image
        displayData(gyroDisplay,acclDisplay, tempDisplay);  //display metadata after saving the image
      }

add

displayData2();

The custom function nfj becomes

static final String nfj(float n, int l, int r) {  //correct format problem with negative zeros
  String s = nfs(n, l, r);
  if (s.equals("-0.00")) s = " 0.00";
  if (s.equals("-0.0")) s = " 0.0";
  if (s.equals("-0.000")) s = " 0.000";
  if (s.equals("-0.0000")) s = " 0.0000";
  if (s.equals("-0.00000")) s = " 0.00000";
  if (s.equals("-0.000000")) s = " 0.000000";
  if (s.equals("-0.0000000")) s = " 0.0000000";
  return s;
}

And we add these two functions to display and control drift on the go

void displayData2() {  //info for user, not for printing
  textSize(16);
  textAlign(RIGHT,BOTTOM);  
  int shadow = 1;//distance to shadow
  String text = "Drift: "+nfj(driftZ,0,5);
  fill(0);
  text(text, width-10+shadow, height-30+shadow); //draw shadow first
  fill(255);
  text(text, width-10, height-30); //then text
  textAlign(LEFT,TOP); 
}

and

void keyPressed() {  //
  if (key == CODED) {
    if (keyCode == RIGHT) {//increase or decrease driftZ compensation
      driftZ += 0.00001;
    } else if (keyCode == LEFT) {
      driftZ -= 0.00001;
    } 
  }
}

Not sure if this is the cleanest way to explain changes. I'll try to keep an updated version of the sketch here: http://tailorandwayne.com/gyrocam.pde

GoToLoop

// forum.Processing.org/two/discussion/21924/
// working-with-gopro-hero5-sensors#Item_3

// GoToLoop (2017-Apr-11)

static final String nfj(final float n, final int l, final int r) {
  final String s = nfs(n, l, r);
  return s.charAt(1) != '-'? s : s.replaceFirst("-", "");
}

void setup() {
  println("0123456789");

  float f = -0.0;
  println(nfs(f, 2, 3), " \tnfs():", f);
  println(nfj(f, 2, 3), " \tnfj():", f, ENTER);

  f = 0.0;
  println(nfs(f, 2, 3), " \tnfs():", f);
  println(nfj(f, 2, 3), " \tnfj():", f, ENTER);

  f = -1e-2;
  println(nfs(f, 2, 3), " \tnfs():", f);
  println(nfj(f, 2, 3), " \tnfj():", f);

  exit();
}

Kajuna

Yes, but

float f = -1.1369292E-4;
  println(nfs(f, 2, 3), " \tnfs():", f);
  println(nfj(f, 2, 3), " \tnfj():", f, ENTER);

returns

-00.000 nfs(): -1.1369292E-4

-00.000 nfj(): -1.1369292E-4

So we still get negative zeros

Kajuna

This works for me

static final String nfj(final float n, final int l, final int r) {
  String s = nfs(n, l, r);  
  if (float(s) == 0.0) {
    s = s.replaceFirst("-", " ");//blank space to match nfs() behaviour, not nf()
  }
  return s;
}

GoToLoop

Your current nfj() fails for when float f = -0.0;, while my version works! 8-X

For float f = -1.1369292E-4;, it outputs -0.00011369292042218149 if we allow 20 decimal digits.

W/ only 3 decimal digits, it outputs -0.000, and 4 decimal digits: -0.0001.

B/c -1.1369292E-4 isn't actually 0, the - shows up even when using your updated fix! :-&

In short, your latest posted fix isn't fixing anything for me here! :-@
At least my own alt() works for when it's-0.0. Check out the results below: :P

// forum.Processing.org/two/discussion/21924/
// working-with-gopro-hero5-sensors#Item_6

// GoToLoop (2017-Apr-11)

static final String nfj(final float n, final int l, final int r) {
  String s = nfs(n, l, r);
  if (float(s) == 0.0) {
    s = s.replaceFirst("-", " ");
  }
  return s;
}

static final String alt(final float n, final int l, final int r) {
  final String s = nfs(n, l, r);
  return s.charAt(1) != '-'? s : s.replaceFirst("-", "");
}

void setup() {
  println("0123456789");

  float f = -0.0;
  println(nfj(f, 0, 2), " \tnfj():", f);         //  -0.00
  println(alt(f, 0, 2), " \talt():", f, ENTER);  //  0.00

  f = -1.1369292E-4;
  println(nfj(f, 0, 3), " \tnfj():", f);         // -0.000
  println(alt(f, 0, 3), " \talt():", f, ENTER);  // -0.000

  println(nfj(f, 0, 4), " \tnfj():", f);         // -0.0001
  println(alt(f, 0, 4), " \talt():", f, ENTER);  // -0.0001

  println(nfj(f, 0, 20), " \tnfj():", f);        // -0.00011369292042218149
  println(alt(f, 0, 20), " \talt():", f);        // -0.00011369292042218149

  exit();
}

GoToLoop

Oops! Now I've found out why my side isn't working as yours! :-SS
My decimal separator in my OS is a comma , instead of a dot .! b-(
So your float(s) evals as NaN here!!! :-&
I need to change it to float(s.replaceFirst(",", ".")) in order to work here. :-<
Anyways, here's the fixed version for both of our versions: O:-)

// forum.Processing.org/two/discussion/21924/
// working-with-gopro-hero5-sensors#Item_7

// GoToLoop (2017-Apr-11)

static final String nfj(final float n, final int l, final int r) {
  String s = nfs(n, l, r);
  if (float(s.replaceFirst(",", ".")) == 0.0)
    s = s.replaceFirst("-", " ");
  return s;
}

static final String alt(final float n, final int l, final int r) {
  final String s = nfs(n, l, r);
  final boolean is2ndMinus = s.charAt(1) == '-';

  if (!is2ndMinus && s.charAt(0) == ' ')  return s;
  if (is2ndMinus)  return s.replaceFirst("-", "");

  return float(s.replaceFirst(",", ".")) == 0.0?
    s.replaceFirst("-", " ") : s;
}

void setup() {
  println("0123456789");

  float f = -0.0;
  println(nfj(f, 0, 2), " \tnfj():", f);         //   0.00
  println(alt(f, 0, 2), " \talt():", f, ENTER);  //  0.00

  f = -1.1369292e-4;
  println(nfj(f, 0, 3), " \tnfj():", f);         //  0.000
  println(alt(f, 0, 3), " \talt():", f, ENTER);  //  0.000

  println(nfj(f, 0, 4), " \tnfj():", f);         // -0.0001
  println(alt(f, 0, 4), " \talt():", f, ENTER);  // -0.0001

  println(nfj(f, 0, 20), " \tnfj():", f);        // -0.00011369292042218149
  println(alt(f, 0, 20), " \talt():", f);        // -0.00011369292042218149

  exit();
}

Kajuna

So if I understood you, now both functions work equally well. Correct? They do on my end

Kajuna

Updated the Accelerometer calculations bit to take into account changes in accelerometre sampling pace. My accelerometer is quite inaccurate anyway, it reports 1.035 Gs while sitting flat.

//accelerometer calculations
      PVector acclDisplay = new PVector(0,0,0); //for displaying data
      if (acclLastRow < acclTable.getRowCount()) {
        float acclFrameTime = 0;//will remember total time of frame
        ArrayList<float[]> acclVectors = new ArrayList<float[]>();  //store the data to apply it proportionally after looping through all valid rows
        while (float(acclTable.getRow(acclLastRow).getString("Milliseconds")) < currentMilliseconds) {  //get rows under our time
          float millisecondsDifference = (float(acclTable.getRow(acclLastRow).getString("Milliseconds"))-acclLastMilliseconds)/1000f;  //how much time since last gyro?
          acclLastMilliseconds = float(acclTable.getRow(acclLastRow).getString("Milliseconds"));  //save current time
          float[] acclRow = { float(acclTable.getRow(acclLastRow).getString("AcclX")),float(acclTable.getRow(acclLastRow).getString("AcclY")),float(acclTable.getRow(acclLastRow).getString("AcclZ")),millisecondsDifference};  //store all the relevant data of the row
          acclVectors.add( acclRow );  //save in the arraylist
          acclLastRow++;  //next row
          acclFrameTime += millisecondsDifference;
          if (acclLastRow >= acclTable.getRowCount()) {
            println("Accel CSV finished before video");
            break;
          } 
        }
        for (int i = 0; i < acclVectors.size() ; i++) {
          acclDisplay.x += (acclVectors.get(i)[0]*(acclVectors.get(i)[3]/acclFrameTime));  //multiply each row's acceleration by its proportional weight in the frame time, probably more accurate if the accl pace is not constant
          acclDisplay.y += (acclVectors.get(i)[1]*(acclVectors.get(i)[3]/acclFrameTime));
          acclDisplay.z += (acclVectors.get(i)[2]*(acclVectors.get(i)[3]/acclFrameTime));
        }
        acclDisplay.div(9.80665);  //divide by gravity to express in Gs

GoToLoop

... now both functions work equally well. Correct?

Almost! Only diff. is that your version outputs 2 indent spaces instead of 1 when we pass -0.0 to it. 8-|

Kajuna

Cool, thanks

Kajuna

Now the data can be extracted easily. Just uncompress this folder and drop your files on GPMD2CSV.bat (for Windows): http://tailorandwayne.com/GPMD2CSV.zip

I have changed my sketch to work with this new system and to correct for several other things (gyroscope data shown was not correct, way to compensate accel inaccuracies...)

I keep an updated version here: http://tailorandwayne.com/gyrocam.pde Might think about setting up a Github instead.

jeremydouglass

Glad you resolved it, and thanks for sharing!

Consider posting the GitHub in the forum under Share Your Work. If others are interested in working with the GoPro and you are feeling generous you might also consider wrapping it up in a contributed library.

Kajuna

@jeremydouglass Thanks! I will try to do that once I have some spare time, and hopefully I'll also be able to extract more data that the v2.00 firmware has (iso, shutter speed...). I've never created a library before, but that could be an interesting challenge.

I see some problems though. The data would still need to be extracted with the external tools. I don't think you can include that within the library functionality.

Also, what could the structure of the library be? You call for the data of a frame, or does it include the functions of processing.video.* but with the syncronised data?

Kajuna

So here's what I first used this workflow for:

I decided to organise the data in Processing and do the image processing in After Effects, in order to keep image quality as high as possible considering the huge amount of transformations I wanted to make. So the sketch now has an option to export tables that can be copypasted to After Effects layers (the video itself for position and rotation or other types of layers for gauges with other info).

The sketch is pretty messy at the moment and I doubt I will find the time to put a library together, for a while, but might be useful if someone is trying to achieve something similar: http://tailorandwayne.com/gyrocam.pde

notrompi

Fantastic! Thanks for sharing your work. How did you figure out how many milliseconds to offset the gyro data in time to line up with the video for stabilization? I'm finding if you put the gyro start time at 0 it is not lining up with the movie. It always needs to be offset by a frame or more. Is the correct offset stated in the metadata somewhere? Thanks in advance!

Kajuna

By trial and error. I don't think it's stated anywhere. I also find that the offset changes slightly depending on the format you are shooting

notrompi

Thanks for the reply! Were you ever able to find an offset through trial and error that works every time, or do you need to use a different offset each time?

Kajuna

I would say the offset is constant for a specific format. I can't check, as I am travellig

notrompi

Thanks again. Very interesting.

Kajuna

I published this in a Github to maximise the possibilities of anyone improving/reusing it: https://github.com/JuanIrache/gyrocam/

I don't have time to create a library (should learn to create libraries first!), and the demand does not seem very high. The GPMD2CSV tool should be published too in an organised way... I'll think about it.