Video 1.0.1 - Read a frame from a paused video?

Krystman

I'm using Processing to make a small app. I want to display and edit text overlays on top of a video file - similar to subtitles. For that, I want to be able to jump a paused video to a specific timecode and display the video frame at that time. Unfortunately, this doesn't seem to work.

So if I make something like this:

movie.pause(); movie.jump(30); movie.read();

The video will NOT show the frame at second 30, but whatever frame was on the screen before the jump. If I jump around in the video the same frame keeps being displayed until I hit play.

Is there a workaround for this?

kfrajer

Can you post your code showing your approach? It will also allow to reproduce your issue.

Kf

jeremydouglass

@Krystman -- Keep in mind that:

movie.pause();
movie.jump(30);
movie.read();

...doesn't do anything on-screen. This line has to run before the screen updates:

image(movie, 0, 0);

See for example the basic movie example sketch:

• https://www.processing.org/reference/libraries/video/Movie.html

Depending on where you are triggering the pause, you may need to either move it in draw() before the image() call or (if you are doing it in keyPressed() etc.) you may need to call redraw().

Krystman

I'm sorry, I presumed it was a known issue. Here is sample code illustrating the problem:

import processing.video.*;
Movie myMovie;

void setup() {
  size(640, 480);
  myMovie = new Movie(this, "test.mp4");
  myMovie.loop();
}

void draw() {
  image(myMovie, 0, 0);
}

// Called every time a new frame is available to read
void movieEvent(Movie m) {
  m.read();
}

void mousePressed() {
  float newTime;
  newTime = random(myMovie.duration());
  myMovie.pause();
  myMovie.jump(newTime);
  myMovie.read();
}

Clicking with the mouse will NOT show random frames from the video.

jeremydouglass

@Krystman --

I'm not sure, but I believe that you need to be playing in order for jump to take effect on the next frame -- otherwise you have specified a new time position, but the movie is paused so it hasn't loaded the frame yet.

Here is an example, adding a revised version of your mousePressed function to the bottom of the standard Processing Example for a video loop. Note that play is called before and pause is called after the new random frame has been loaded.

import processing.video.*;
Movie movie;

void setup() {
  size(640, 360);
  background(0);
  movie = new Movie(this, "transit.mov");
  movie.loop();
}

void movieEvent(Movie m) {
  m.read();
}

void draw() {
  image(movie, 0, 0, width, height);
}

void mousePressed() {
  float newTime;
  newTime = random(movie.duration());
  println(newTime);
  movie.play();
  movie.jump(newTime);
  if (movie.available()) {
    movie.read();
  }
  movie.pause();
}

neilcsmith_net

@jeremydouglass sounds like a bug in the video library if that's the case.

jeremydouglass

@neilcsmith_net -- I honestly can't tell from the reference page or example.

• https://processing.org/reference/libraries/video/Movie_jump_.html

However, code examples seem to indicate the need to play/jump/pause:

• https://github.com/processing/processing-video/search?utf8=✓&q=jump&type=Code

Whether a bug or feature request, it certainly seems reasonable to expect that jump() would update the current image, even while a movie is paused. Perhaps someone working with this wants to submit an issue to Processing-Video?

• https://github.com/processing/processing-video/issues

GoToLoop

... it certainly seems reasonable to expect that jump() would update the current image, even while a movie is paused.

• AFAIK, images should stop coming up when playback is paused.
• Method jump() changes current frame within the playback.
• However, it doesn't resume the playback at all.
• Maybe there should be some new method to grab a target frame from the playback?

neilcsmith_net

Why should images stop coming when playback is paused?! It would make more sense to follow the underlying GStreamer behaviour and emit all frame update events, so that jump() works as expected. If you don't do that, pause() is just another form of stop().

GoToLoop

make more sense to follow the underlying GStreamer behaviour and emit all frame update events,

I'm not familiar w/ GStreamer's features at all.
Video library is merely a wrapper for GStreamer.
And its pause() method temporarily halts the playback to spare CPU.

pause() is just another form of stop().

AFAIK, after stop(), resume isn't possible anymore. At least for library Minim. :P

kfrajer

A quick inspection on the source file, the difference between pause and stop is that stop calls jump(0) and resets a boolean field differently: https://github.com/processing/processing-video/blob/master/src/processing/video/Movie.java

From the source, I can't tell much about the behavior of jump() in regards to returning data. Maybe @codeanticode could comment on this?

Kf

neilcsmith_net

And its pause() method temporarily halts the playback to spare CPU.

GStreamer is already doing this though - it will only trigger an update if the image changes, eg. through jump(). The video library seems to be filtering events when paused, which won't actually save any CPU time.

GoToLoop

That's good to hear! So video library demands a refactoring in its logic. :P

neilcsmith_net

Well, video library requires lots of refactoring! GStreamer 1.x would be good for a start :-)

GoToLoop

I also have a feature request: a new callback event for when video reaches its playback end. $-)

kfrajer

I also have a feature request: a new callback event for when video reaches its playback end

I second that feature as well.

Kf

jeremydouglass

I also have a feature request

I second that feature as well

• https://github.com/processing/processing-video/issues

:-bd

GoToLoop

Regardless, I've already have a hack for the eosEvent() internal method: :ar!
https://forum.Processing.org/two/discussion/14990/movie-begin-and-end-events#Item_1

neilcsmith_net

@jeremydouglass - aren't all these in the issue queue already?! They're just not being addressed. If anyone wants to take a lead on updating the library then I'm happy to help out (I'm the GStreamer 1.x bindings maintainer).

@GoToLoop - nice hack! If that's called directly from the GStreamer thread (probably is), be careful about concurrency issues using that approach though.

jeremydouglass

@neilcsmith_net -- re:

aren't all these in the issue queue already?!

Really?

When I searched open issues for "jump" and "pause" I also didn't see any bug or enhancement request describing jump not updating the current frame while paused.

When I searched open issues for "callback" "end" "stop" and "done" I didn't see any open issue requesting a new callback event for when video reaches the end.

However, I didn't read through the complete list of open issues. Maybe they are in there somewhere but described using different terms.

neilcsmith_net

The EOS and GStreamer 1.x things are there. You're right that the OP's jump / pause issue isn't listed. Still doesn't change the fact the library has not seen any development for ~18 months. :-S

GoToLoop

If that's called directly from the GStreamer thread (probably is), ...

AFAIK, that's the same Thread which calls back movieEvent(). 8-X

GoToLoop

On 2nd look within internal initSink():
https://GitHub.com/processing/processing-video/blob/master/src/processing/video/Movie.java#L785-#L790

Bus bus = playbin.getBus();
bus.connect(new Bus.EOS() {
  public void endOfStream(GstObject element) {
    eosEvent();
  }
});

The Thread responsible for eosEvent() callback is whatever Bus.EOS's is. b-(

For movieEvent()'s Thread, it's either BufferDataAppSink.Listener or RGBDataAppSink.Listener I guess:

https://GitHub.com/processing/processing-video/blob/master/src/processing/video/Movie.java#L754-#L773

GoToLoop

Method invokeEvent() which is called back by 1 of those 2 Listener classes above is synchronized.
While eosEvent() by Bus.EOS isn't: 8-|

https://GitHub.com/processing/processing-video/blob/master/src/processing/video/Movie.java#L854-#L871

GoToLoop

• But even though eosEvent() isn't synchronized, it shouldn't cause any concurrency problems.
• The Bus.EOS event is passed as argument to Bus's connect() method.
• We should assume that Bus knows how to manage anything connect() to it. \m/

neilcsmith_net

@GoToLoop - er, no! Threads don't work like that. However, my point was not about what the library is doing. It's that the eos method in your code may be called at the same time as the animation thread is calling draw(). If you're reading or writing any of the same variables within both draw() and your eos method code you have a potential issue.

GoToLoop

You mean these statements?

void myEoS() {
  ended = true;
  frameRate(1);
}

Field ended got no problem. I guess you mean frameRate(1);

It doesn't mutate sketch's canvas in any significant way.
Therefore it's safe to call outside the "Animation Thread".

As an answerer of this forum, I've seen lotsa folks doing drawing commands outside the "Animation Thread". I guess I see 1 at least once for each 2 weeks. @-)

But I'm confident frameRate() is safe when called outside the "Animation Thread".
Most of what may happen is some innocuous race condition if more than 1 Thread invokes it at the same time. ;;)

GoToLoop

Threads don't work like that.

Well, as I had stated, I dunno the specifics of GStreamer.
And thus I dunno how many threads are created by it. 8-|

The only thing I know is that if we place println( Thread.currentThread() ); inside movieEvent(), the answer isn't "Animation Thread". [-(

neilcsmith_net

I don't mean any particular statements, just that the approach needs to be thought through. I'm also talking about threads in general - nothing specific to GStreamer.

In your example, you can't guarantee when the change to ended will be "seen" by the animation thread. In theory it could be never. And you have no idea whether any renderers implementation of frameRate() is thread safe, although the JOGL ones might be.

Lots of none drawing code isn't safe to blindly use, and race conditions are almost never innocuous. They cause weird crashes and bugs. The sort that don't show up when testing, just at the important times! ;-)

GoToLoop

... you can't guarantee when the change to ended will be "seen" by the animation thread.

This is mostly irrelevant! In Processing by default draw() is called back at about 60 FPS.

If I change some field outside the "Animation Thread", if it takes at most 2 FPS to be "felt" for the latter, it won't make much visually difference. ~:>

In theory it could be never.

In my Processing experience w/ concurrency, Java has never failed to update the actual field when it was changed by some foreign Thread. >:/

And for some exotic reason I really need it to be updated ASAP, I can always declare the target field w/ volatile. \m/

And you have no idea whether any renderers implementation of frameRate() is thread safe,

By the looks of it it should merely change some bunch of internal fields.
And it ended up that way when I just looked that up right now:

https://GitHub.com/processing/processing/blob/master/core/src/processing/core/PSurfaceNone.java#L245-L249

public void setFrameRate(float fps) {
  frameRateTarget = fps;
  frameRatePeriod = (long) (1000000000.0 / frameRateTarget);
}

I fail to see how changing those 2 fields could crash the renderer. :-@
Although I agree there's a chance for the values stored in those 2 fields ending up not matching if a race condition happens.

But still, frameRate() is mostly invoked within setup().
No sketch keeps mutating its FPS all the time after all, much less in multiple threads. :P

... and race conditions are almost never innocuous.

In Processing is just the opposite! Most of them won't be visually realized by the user watching an animation. 1 or 2 FPS w/ slightly wrong, not fully updated state aren't that serious. :ar!

Having said that, there are race conditions which crash a sketch code, especially now in version 3! :-&

Any drawing commands which mutate the pixels of the PGraphics main canvas crash P3 and can corrupt it under P2 & P1 if they are run outside the "Animation Thread"!!! :-SS

In addition to those, if more than 1 Thread relies on any color-related function, like color(), fill(), stroke(), background(), for the same PGraphics object, the results are gonna get corrupted if they happen at the same time!

Those 2 situations above are the only dangerous race conditions that I know for sure are problematic in Processing.

I know what you're warning us about are general Java protocols for concurrency safety.
But I don't believe they apply much for Processing API in general. >-)

neilcsmith_net

... and some people still believe the world is flat! ;-) In the past I've had stuff run for hours or days before it blew up. Trying to help you, but arguing with a brick wall is no fun. So, good luck with it.

On a more relevant note, if someone is going to take the initiative on the video library, PM me.