Frustrated by Quicktime for Java, I've finally bitten the bullet and am developing a Java-FFMPEG library (I know there's another one out there, but I couldn't get it to work very well - and mine is faster. GStreamer is a bit complex and unwieldy for my purposes). At some point I will probably develop this into a Processing library (for Windows - but it should be an easy port to Macs and Linux, I just don't do those), and I'd like to know what people would want from such a library, before coding it.

What I want for my own purposes is the following:

a) It should grab frames/sound and draw at the right speed from within the draw method in Processing. Not very hard - but gives the programmer a lot of control over the drawn frame (so you could draw over it, manipulate it in OpenGL, or process a pImage - every frame in a pretty straightforward fashion).
b) It should be easy and fast to generate both PImages and an OpenGL texture (so far this seems fairly straightforward).
c) Obviously, if you just want to set up a video to playback and forget about it, this should also be possible (so it would run on a seperate Java thread, and just send events when it finished, or looped, or whatever).
d) It should be possible for mute videos to start immediately (harder for videos with sounds, as you have to fill the sound buffer), even with random access (obviously this will be partly dependent upon the codec).
e) Random frame access should be as fast as possible.
f) It should be easy to have the same video playing back at multiple locations, in sequence.

Nice to haves:
a) Support for FreeFrame plugins
b) Support for sequence generators (like Closures, or sequences in SuperCollider), that can control frame rate, frame access and possibly the texture (Alpha, Gama, Brightness, etc).
c) I guess OSC support.

Unlikely:
a) Encoding (Quicktime is fine for this)
b) Video Tracks like QuickTime

So what have I missed, or what kind of features would people here want.

would this allow us to get what is known as "DirectShowSource" footage frames?

if so, my world would be complete!!!!

edit: erm, for clarification see here: http://avisynth.org/mediawiki/First_script

avisynth lets you write scripts that "feed" frames into other editors, if I could feed such frames into processing... I could... i could... *spins in chair*

edit: here's another link of how you can treat .avs just as .avi, supposedly there's no problem: http://neuron2.net/www.math.berkeley.edu/benrg/avisynth-apps.html


I'm not suggesting that you use avisynth as the backbone of the frame-readers, but if it was one of the options than you'd take care of all the other "weird" formats, because a simple DirectShowSource call in an avisynth script would translate it into simple .avi pixels at real-time.

K, I'm fairly sure I'll be able to work with whatever you release (VFapi fake avi's wrapping avs's and whatnot)

So... *subscribes to topic*