Problems with saving every frame from videostream with dropping framerate

Per

I got a sketch like below. I want to save every frame from the video but the framerate is dropping and processing only process every fourth frame from the videostream and saves it with saveFrame.

How can I process and save every frame? How can I halt the next frame from the videostream until the frame is processed and saved?

Below is a print of the videoFrame from the videostream and the frame that is saved. As you can see only about every fourth frame is processed and saved

VideoFrame: 0 FRAMERATE: 18.936382
savedFrame: 1
VideoFrame: 1 FRAMERATE: 22.489614
VideoFrame: 2 FRAMERATE: 22.489614
VideoFrame: 3 FRAMERATE: 22.489614
VideoFrame: 4 FRAMERATE: 22.489614
VideoFrame: 5 FRAMERATE: 22.489614
VideoFrame: 6 FRAMERATE: 22.489614
savedFrame: 7
VideoFrame: 7 FRAMERATE: 20.717686
VideoFrame: 8 FRAMERATE: 20.717686
VideoFrame: 9 FRAMERATE: 20.717686
VideoFrame: 10 FRAMERATE: 20.717686
savedFrame: 11
VideoFrame: 11 FRAMERATE: 19.62102
VideoFrame: 12 FRAMERATE: 19.62102
savedFrame: 13
VideoFrame: 13 FRAMERATE: 18.655874
VideoFrame: 14 FRAMERATE: 18.655874
VideoFrame: 15 FRAMERATE: 18.655874
savedFrame: 16
VideoFrame: 16 FRAMERATE: 17.817186
VideoFrame: 17 FRAMERATE: 17.817186
VideoFrame: 18 FRAMERATE: 17.817186
VideoFrame: 19 FRAMERATE: 17.817186
savedFrame: 20
VideoFrame: 20 FRAMERATE: 17.098238
VideoFrame: 21 FRAMERATE: 17.098238
savedFrame: 22
VideoFrame: 22 FRAMERATE: 16.579494
VideoFrame: 23 FRAMERATE: 16.579494

Heres my sketch:

import processing.video.*;

Movie m;

int res = 5;
PVector loc;
int frameCounter = 0;
boolean export = false;
int frame = 0;

void settings() {
  size(960, 540);
}

void setup() {
  m = new Movie(this, "video.mp4");
  loc = new PVector(0, 0);
  m.play();
  m.loop();

  background(255);
  ellipseMode(LEFT);
}

void movieEvent(Movie m) {
  if (m.available()) {
    println("VideoFrame: " + frame + "  FRAMERATE: " + frameRate);
    frame++;
    m.read();
  }
}

void draw() {
  println("savedFrame: " + frame);
  run();
}

void run() {
  for (int i = 0; i <  m.width/res; i++) {
    for (int j = 0; j < m.height/res; j++) {
      loc.x = i*res;
      loc.y = j*res;

      color col = m.pixels[(int)loc.x  + (int)loc.y * m.width];
      noStroke();
      fill(col);
      ellipse(loc.x, loc.y, res, res);
    }
  }

  if (export && frameCounter < 3600) {
    saveFrame("frames/####.png");
    frameCounter++;
  } 

  if (frameCounter > 3598) {
    println("EXPORT KLAR");
  }
}

void keyReleased() {
  println("export"+ export + "  frameRate: " + frameRate + " frameCounter: " + frameCounter);
  if (key == 'e') {
    frameCounter = 0;
    println("export"+ export + "  frameRate: " + frameRate);
    export = !export;
  }
}

Per

I solved it like this:

m.speed(frameRate/30); // 30 is the frameRate of the movie

still dropping a frame every now and then though but works until 95%

koogs

just don't call run() until you have a movieEvent. you just need to sync the writing with the reading, breaking the usual async nature of the movie stuff.

Per

Hi!

Yes I tried to put the run() function in the movieEvent but this didnt display the processed image

void movieEvent(Movie m) {
  if (m.available()) {
    println("VideoFrame: " + frame + "  FRAMERATE: " + frameRate);
    frame++;
    m.read();
    run();
  }
}

Also tried before with a boolean in the drawmethod that became true when there was a movie event but still the same issue with that it doesnt save every videoframe.

void movieEvent(Movie m) {
  if (m.available()) {
    frame++;
    m.read();
    run();
    theEvent = true;
  }
}

void draw() {
  if (theEvent) {
    run();
  }
}

void run() {
  theEvent = false;

...

}

Have you got another suggestion?

koogs

i think that last one has a chance of working except for line 5, which i don't think should be there.

but i've not tested this.

kfrajer

I used the following post to get the number of frames: https://superuser.com/questions/84631/how-do-i-get-the-number-of-frames-in-a-video-on-the-linux-command-line

This is what I get:

C:\Users\Downloads\downloaded_installers\ffmpeg-20160115-git-b58cfa6-win64-static\ffmpeg-20160115-git-b58cfa6-win64-static\bin>ffmpeg.exe -i transit.mov -vcodec copy -f rawvideo  -y info.txt

ffmpeg version N-77870-gb58cfa6 Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 5.2.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
libavutil 55. 13.100 / 55. 13.100
libavcodec 57. 22.100 / 57. 22.100
libavformat 57. 21.101 / 57. 21.101
libavdevice 57. 0.100 / 57. 0.100
libavfilter 6. 24.100 / 6. 24.100
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.101 / 2. 0.101
libpostproc 54. 0.100 / 54. 0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'transit.mov':
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2012-08-31 20:17:39
Duration: 00:00:12.38, start: 0.000000, bitrate: 731 kb/s
Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709), 640x360, 727 kb/s, 29.97 fps, 29.97 tbr, 600 tbn, 1200 tbc (default)
Metadata:
creation_time : 2012-08-31 20:17:44
handler_name : Apple Alias Data Handler
encoder : H.264
Output #0, rawvideo, to 'info.txt':
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
encoder : Lavf57.21.101
Stream #0:0(eng): Video: h264 (avc1 / 0x31637661), yuv420p, 640x360, q=2-31, 727 kb/s, 29.97 fps, 29.97 tbr, 600 tbn, 600 tbc (default)
Metadata:
creation_time : 2012-08-31 20:17:44
handler_name : Apple Alias Data Handler
encoder : H.264
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame= 371 fps=0.0 q=-1.0 Lsize= 1099kB time=00:00:12.31 bitrate= 731.5kbits/s speed=3.52e+003x
video:1099kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

I run the following and I got all the 371 frames:

import processing.video.*;

Movie m;
int frame = 0;

void settings() {
  size(640, 360);
}

void setup() {

  m = new Movie(this, "transit.mov");
  m.play();
  m.speed(0.1);
  noLoop();
}

void draw() {
  image(m, 0, 0);
  saveFrame("frames/####.png");  
}

void movieEvent(Movie m) {
    println("Frames ctr vs. f-ctr: " + frameCount + " " +frame );
    frame++;
    m.read();
    redraw();
}

Notice that if I choose the speed even to be 0.2, I recorded 370 frames so it missed one.

Kf

GoToLoop

This is an eternal common case in Processing when dealing w/ libraries which create their own Thread:

Function run() MUTATES the sketch's canvas. :-O
Only 1 Thread is "authorized" for mutating the canvas: The "Animation Thread"! :>

If we need to mutate the canvas under a foreign Thread, there are 3 options I know about:

1. Declare a boolean variable, Set it true under the foreign Thread. Then under the "Animation Thread", set it to false & finally mutate the canvas.
2. Create a PGraphics for exclusive use for the foreign Thread. Have the "Animation Thread" image() it on the main canvas.
3. Dunno much about this 1. It's based on registerMethod() some hook function. For example the pre() callback.

Per

@koogs whoops, my mistake. forgot to take the run() away. :/ now its working! If Im setting the speed to very slow; m.speed(0.1);

@kfrajer thanks. redraw is doing the same as the boolean. thanks!

@GoToLoop yes, of course. thanks. the video library starts its own thread. I guess number 1., you mean like this:

void movieEvent(Movie m) {
  if (m.available()) {
    frame++;
    m.read();
    theEvent = true;
  }
}

void draw() {
  if (theEvent) {
    run();
  }
}

void run() {
  theEvent = false;

...

}

Dont really get your third option?