processing 2.0 video new API

Core Library Questions

eggnot

in Core Library Questions • 1 years ago

as far as i know, current video with gstream is far from to be frame-accurate. actually it's turns to total hell to process every frame of video exactly once.
probably, as i can guess, it's goes from underlayng ffmpeg which also can't jump to exact frame. probably, accurate jump it's not so necessary, but there's should be the way to get needed frame!

i think video should have another api. with more low-level methods exposed. something like this:

videoStream video = new videoStream();
video.open("bla.mov"); // returns True if all ok
//simple iteration (maybe it's better to have iterator througth frames?)
PImage frame0 = video.decodeFrameAndStep();
//and
video.stepFrame();
video.stepFrame();
PImage frame3 = video.decodeFrameAndStep();
// as i can assume such internal logic should know last I-frame and do decode from it.
// maybe, even frametype can be public
println(video.frame + ": " + video.frameType);
// and for jump, if it's not accurate there sould be a way to get needed frame
video.jump(toSomeTime);
while(video.frame>frameWeNeed)
video.stepToPreviosIntra();
while(video.frame<frameWeNeed)
video.stepFrame();
Pimage frameWeNeedImg = video.decodeFrameAndStep(); // or decodeFrame();

ok, maybe some intellegent jump() should do something like this internally,

but anyway, will be great to have more low-level control over video:
such as stepToNextIntra(), stepToPreviosIntra() for fast video rewind
some iteration thougth frames for fast every frame processing
and so on...

LET'S DISCUSS WHAT WE NEED, BUGGY JUMP() IS NOT ENOUGHT!

Replies(10)

andres

Re: processing 2.0 video new API

1 years ago

Hi, I'm trying to figure out the correct way to jump-to-frame with gstreamer (which is what the new video library in Processing is using for video decoding). Until now, I haven't found an accurate way of doing it (not sure if it is possible at all, I will check directly with the gstreamer developers on IRC).

For the time being, take a look at the Frames example included in the video category, it shows how to estimate frame count by using the framerate of the video, and move frame-by-frame through the entire clip.

andres

Re: processing 2.0 video new API

1 years ago

After looking at some discussion threads on the subject:

GStreamer is too general purpose to be organized around the concept of a frame. The best you can do is seek forward or backward by some multiple of (1/FPS). Sometimes you'll get a new frame when you do this, sometimes you won't. It depends to a large degree on the particular codecs with which you are working. Some codecs support seeking backward better than others. Most compressed formats do not really update the screen with complete frames, but rather changes to the previous frames.

eggnot

Re: processing 2.0 video new API

1 years ago

i wondering, is all other happy with Video module?

i spend a lot of time to get it working right(iterating and jumping) with no any success. sometimes it's returning same frame, sometimes it's skips, in some occasions it's ever return previous frame and so on... worse, worse...

so i invest some time and wrote simplest ffmpeg wrapper with jna. with most code copypasted and altered from ffmpeg-java. gona test and improve it, but for now it's already works well(accurate and fast) for me.
so later i'll release a library or at least share the code. good thing it's can be extended to do pretty everything with video\audio as ffmpeg(avformat, avcodec, etc.) do.

ps.
ok, maybe just for playing video in sketch and using webcamera current Video module is ok. in such cases you don't need frame accuracy, iteration over frames and low-level features.

andres

Re: processing 2.0 video new API

1 years ago

Well, sounds very cool that you are able to get more accurate jump-to-frame functionality using ffmpeg. What is the specific API from libav you currently using?

I also considered for a while using ffmpeg/libav, particularly after seeing the progress the javacv guys are making (btw, there is a new opencv library for Processing available based on javacv, is pretty cool). However, I found that most of the problems that I already solved with gstreamer/gstreamer-java (mainly packaging the binaries with the library itself so they properly work on each platform without doing additional installs, including in exported apps) needed to be tackled again with ffmpeg/libav.

Of course there are some trade-offs. As you have seen, the jump-to-frame functionality is not perfect in gstreamer, however, I was recently chatting with the gstreamer devels, and apparently there are ways to make it better, either by using the frame-stepping API, or setting a GstIndex on the demuxer/parser and letting it populate that with what I think are the time points along the stream that correspond to each individual frame.

Another issue with gstreamer-java is that is based on JNA, which tends to be slower than other bindings like JNI. I think that JavaCPP, which is the one used in JavaCV, internally creates the glue code using JNI. This problem is not that serious though, since even though calling a native function from JNA is much slower than, say, JNI, most of the CPU time is spent on decoding and rendering frames anyways.

Right now, and for the next couple of weeks, I will taking a break from the video stuff and focusing on the OpenGL renderer, so I don't think I will be able to fix the frame jump issues for a while. But we can keep discussing these issues on the forum. And of course, any code patches are always welcome :-)

eggnot

Re: processing 2.0 video new API

1 years ago

oh, JAVACV wraps ffmpeg is much better! it's full av* api and JNI. i'll make a tests, but as i found the api is pretty the same to my wrapper as well as classical use of ffmpeg ( http://dranger.com/ffmpeg/tutorial01.html )
javacv have it here - http://code.google.com/p/javacv/source/browse/trunk/javacv/src/com/googlecode/javacv/FFmpegFrameGrabber.java

moreover ffmpeg related stuff fully independent from rest of javacv and author is ok if one use only these classes.
http://groups.google.com/group/javacv/browse_thread/thread/12512bfe2923ad7b
so, i gona switch from my wrapper to javacv-av* ;anyway writing mine i teach how to deal with ffmpeg.

in short for frame accuracy:

first i used dumb iteration to needed frame with av_read_frame(param, packet)
if it is - avcodec_decode_video2(params..., packet)
and convert to rgb with sws_scale(params...)
great thing sws_scale() is super fast. also you can collect some info this way - I(key-)frames and so on. JNI at step of av_read_frame() can mean a speedup.

second i use av_seek_frame(params). it jumps to nearest(?) keyframe at given time. so i doing jumps until keyframe before needed frame. then doing first method (only not from beginning, but from keyframe).

as i'm very pleased and glad this such solution, for now i gona return to my main sketch routines. for ease of development i using jpeg sequence instead video. later however (at the end of January, i guess), i'll merge video code.

ps.
for processing i guess can be good to ask Samuel Audet (javacv developer) ho split ffmpeg to separate package and use it as working horse at least for video back-end.
pps.
with exposing some low-level functions close to ffmpeg native, as i mention early.

zumbaree

Re: processing 2.0 video new API

1 year ago

I havent got any solutions but just wanted to say frame accuracy is a very important feature. I have a few ideas for sketches using video, and I was waiting for processing 2.0 to come and solve various issues I had with current video class.

I understand why work on the new opengl renderer is more urgent, but just wanted say:
Having a STABLE frame by frame scan forward through the video and frame accuracy for seek is much more important for me than being able to play the video smoothly.

In many situations one may want to analyse content of a video file on a frame by frame basis, and even occasional duplicate or skipped frames would spoil it. At the moment splitting into jpg sequence is a workaround.

It would be great to have it as a packaged library rather than an ffmpeg hack, but this is not urgent and I can wait as long as it takes to get it right (any time before 2013 would be cool :).

Thanks for your great work andres and keep at it!

andres

Re: processing 2.0 video new API

1 year ago

Hello, you point about frame accuracy is well taken.

Right now, we are quite committed to gstreamer given all the work we have done already, so the video library in the final release of Processing 2.0 will very likely be what we have at this point in the alpha 4 release (plus some API improvements, bug fixes, updated versions of the gstreamer native libs etc)

I will explore some of the options to have better frame-by-frame scan in gstreamer ( frame-stepping API , GstIndex) after we get a couple of new alphas out showcasing the new opengl stuff.

However, if down the road some other video library, based on ffmpeg or other multimedia toolkit, proves to be more performant and accurate than gstreamer, then we would consider it as a potential replacement. The work eggnot is doing with javacv looks promising, but I think there are some issues that need to be solved with it as well (video capture on all platforms, native libs installation).

eggnot

Re: processing 2.0 video new API

1 year ago

100% agree with andews.

i will try to release the ffmpeg library in ~feb2012 but the problem i am not a developer(not even a programmer with skills) so it wouldn't be perfect.

for now, for my purposes it's works well. i can guess for other types of video there can be problems, eg. for mpeg-streams and so on. capturing is also the area i have no knowlege at all.

so the library would be more like "solution that warks for me with sources to extend for your needs" or so.

zumbaree

Re: processing 2.0 video new API

1 year ago

I know its not feb2012 quite yet, (few days left) but I really could use the 'frame-by-frame' stepping through a quicktime file for a project I am working on now.

the basis of what I want to do now, is reading a frame from file and writing the same (almost) frames back.

I am using 1920x1080 h264 files so realtime operation is still out of question in 2012. it HAS to be offline. its fine as long as there is a reliable way to do it in non-realtime.

Even if accurate jump() is impossible, a stable way of stepping a frame forward seems like something essencial, something like .stepFrame mentioned earlier.

Workarounds like checking if the video happened to be at the next frame already don't seem like a very good method of proceeding.

eggnot

Re: Re: processing 2.0 video new API

1 year ago

hi, zumbaree!

there is javacv update, so i have to make some changes to code for make it work. bad thing is i'm a bit busy now this another coding. i'll will try to release something working in next week or so. also, i can't gurantee it will work this your video as it only tested on canon dslr video.

Top Reply