GL video 1.2.1

RaulF

Hi there!

I have installed the last GL video version, to see if I can solve my "jump" problems (https://forum.processing.org/two/discussion/13673/jump-in-videolibrary-causes-java-error#latest).

But, with version 1.2.1 playing the videos the image freezes constantly.

I'm testing it in 5 different Raspberry at the same time, with the same results (253% CPU!). All software updated.

PID USER      PR  NI    VIRT    RES       SHR S    %CPU  %MEM     TIME+ COMMAND                 
1033 pi        20   0  457288  91032  13084 S 253.2 23.2 351:44.38 java 

I'm also playing the same video file in 8 Raspberry, same hardware and software, but with GL video 1.2. Better performance:

PID USER      PR  NI    VIRT    RES        SHR S      %CPU   %MEM   TIME+ COMMAND                 
1069 pi         0 -20  371956  58540  12232 S  48.3 14.9 136:07.68 java   

Video file:

Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709), 800x480, 9462 kb/s, SAR 1:1 DAR 5:3, 60 fps, 60 tbr, 60 tbn, 120 tbc (default)
Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, 1 channels, s16, 768 kb/s (default)

Any help is welcome!

Best.

gohai

@RaulF Can you clarify what you mean by "image freezes"? Does the application hang? Or do you just mean that some frames of the video are dropped, causing stuttery playback?

Does the "image freezes" problem only occur with 1.2.1 or also with 1.2?

Can you upload a sketch and a video file for me to test somewhere?

RaulF

Hi gohai,

With 1.2 it works perfect. The video is a 60fps video, and some frames are dropped, but it works quite well.

With 1.2.1 (same Raspberry, software, etc.) the application never hangs, but a lot of frames are dropped. Sometimes you can see the same frame in the screen for 1 or 2 seconds. The CPU is > 200%.

I used this video in my last test:

https://www.dropbox.com/s/9xmjzf30za3sif3/test.mov?dl=0

It could be tested with

import gohai.glvideo.*;
GLMovie myMovie;

void setup() {
  size(800, 480, P2D);
  noCursor();
  background(0);

  // GLMovie
  myMovie = new GLMovie(this, "test.mov");
  myMovie.loop();
}

void draw() {
  if (myMovie.available()) {
    myMovie.read();
  }

  image(myMovie, 0, 0, width, height);
} 

Best.

GoToLoop

How about frameRate(30); in setup()?
And prematurely quit draw() when not available()?

void draw() {
  if (!myMovie.available())  return;
  myMovie.read();
  image(myMovie, 0, 0);
}

RaulF

How about frameRate(30); in setup()?

It's a 60fps movie.

And prematurely quit draw() when not available()?

I didn't try this.

Anyway, I think it's not a code problem. The same code works fine with GL 1.2.

GoToLoop

// forum.Processing.org/two/discussion/21587/gl-video-1-2-1#Item_5
// GoToLoop (2017-Mar-24)

import gohai.glvideo.GLMovie;
GLMovie myMovie;

static final String RENDERER  = JAVA2D; // JAVA2D, FX2D, P2D, P3D
static final String VIDEONAME = "test.mov";

boolean isRaster;

void settings() {
  myMovie = new GLMovie(this, VIDEONAME);
  myMovie.loop();

  while (!myMovie.available())  delay(10);

  size(myMovie.width, myMovie.height, RENDERER);
  noSmooth();
}

void setup() {
  frameRate(60);
  blendMode(REPLACE);

  isRaster = !getGraphics().isGL();
}

void draw() {
  if (!myMovie.available())  return;
  myMovie.read();

  if (isRaster)  background(myMovie);
  else           image(myMovie, 0, 0);
}

GoToLoop

// forum.Processing.org/two/discussion/21587/gl-video-1-2-1#Item_6
// GoToLoop (2017-Mar-24)

import gohai.glvideo.GLMovie;
GLMovie myMovie;

static final String RENDERER  = JAVA2D; // JAVA2D, FX2D, P2D, P3D
static final String VIDEONAME = "test.mov";

boolean isRaster;

void settings() {
  myMovie = new GLMovie(this, VIDEONAME);
  myMovie.loop();

  thread("grabFrameLoop");
  while (myMovie.height < 2)  delay(10);

  size(myMovie.width, myMovie.height, RENDERER);
  noSmooth();
}

void setup() {
  noLoop();
  frameRate(60);
  blendMode(REPLACE);

  isRaster = !getGraphics().isGL();
}

void draw() {
  if (isRaster)  background(myMovie);
  else           image(myMovie, 0, 0);
}

void grabFrameLoop() {
  final GLMovie vid = myMovie;

  for (;; delay(3))  if (vid.available()) {
    vid.read();
    redraw = true;
  }
}

RaulF

Hi GoToLoop,

Thanks for your code. I can see a couple of interesting ideas. I have adapted your code - it didn't work for me in my Rasp 2. For example GL Video needs P2D, and I got errors in settings() too - but the problem is still there.

Anyway, thanks for you code. It will be very useful.

Best.

gohai

@RaulF Thanks for your report. I'll try to look into this in a couple of days. The 1.2.1 version switches from one major GStreamer version to the next one - this might be the culprit, but it could also be an unrelated change. I'll try to get to the bottom of this..

GoToLoop

Hey @gohai! It'd be über nice if you would add glMovieEvent() & glCaptureEvent() to your library like Processing's video library. *-:)

Even much better: Add some endOfStreamEvent() as well, so we know exactly when the video finishes. \m/

RaulF

Hi @gohai! Did you have time to take a look into this?

Best.

gohai

@RaulF No, didn't get to it..

gohai

@RaulF I just tested your file with the upcoming release of the GL Video library I am working on, which will be based on GStreamer 1.12.

To my eyes the video looked fluid at all times, and when I added a global frames variable, and the following to the if-case when there is a new frame waiting to be read

frames++;
println(frames/(millis()/1000.0f));

the effective framerate was around 50 fps after a minute or so (still climbing)...

Please re-test with the upcoming release, but I believe this is working as it should

RaulF

Ok, I'll take a look!