Loading...
Logo
Processing Forum
I just corrected a stupid error and got my third attempt at realtime sketch capture to work. I haven't tested it much at all and the code is still a little raw, but I am very pleased with the performance and convenience.

My previous attempts:
attempt one
attempt two

This third version is much more convenient because it captures directly to compressed video instead of an image sequence and, now that I have OpenGL PBOs working, using it seems to cause very little framerate penalty. I have only tested it with 720 so far though.

The process involves adding a file to the sketch you would like to capture. The code creates a temporary file and continuously overwrites it with the raw pixel data of the most recent frame. The actual video encoding is done by ffmpeg which is run via another Processing sketch that reads the same temporary file that is being written to and pipes the data to the ffmpeg process. Your PATH environment variable has to inlude ffmpeg's bin folder for this to work.

capSend.pde (put this with the sketch you want to capture)
capEncode.pde (run your sketch using capSend.pde, then this to start capturing, click to stop)

The encoding sketch is basically an ffmpeg command line that I have tailored for Vimeo. The output video filenames are hard coded and it will overwrite them without asking unless you rename them. I have only tested this on Windows and Processing 1.5.1 because those are what I use most of the time (it wont work on the 2.0 betas). I'm kind of waiting for a stable 2.0 release before trying it with that. It shouldn't be hard to convert.

The command lines could probably be less hackish if this was run on Mac or Linux. I have the option to record the currently playing audio along with the video which should be possible with just one command line but I had to do them separately and combine them after the capture because the audio would not allow the process to end and close the file properly. On Linux I'm assuming you could send the process SIGINT with the kill command and it would close correctly. Also on Windows I'm using this audio capture thing. I don't know if you can capture line in via ffmpeg on Windows, but I think you can on Linux. Anyway I have the audio disabled by default in the code.

If you try it don't forget to make sure the width(ww) and height(hh) and save and temp locations match in both files. Width and height are set to 1280 and 720 by default. The temp location defaults to C:\p5temp and the save location to a folder called test in the sketch path. I'd be happy to hear if it does or doesn't work and how it performs on your system; Also, any suggestions for the code.

Replies(9)

I'm trying to get this working on a Mac.  I haven't been able to get the x264 library working on my system, so some changes have been made to the ffmpeg command:

    ffmpegInstall + "ffmpeg " +
    "-y " +
    "-f rawvideo -vcodec rawvideo -pix_fmt bgra " +
    "-s " + ww + "x" + hh + " -i - " +
    "-r 30 -vcodec mpeg4 " +
    "-rtbufsize 3686400 " +
    "-vf vflip " +
    saveLoc + "testv.mp4";

The saveLoc variable has been changed to the full path of my data folder.  I have also had to create a new variable to define the path of my ffmpeg installation.

Unfortunately, the video that this creates is only the left half of the sketch canvas, stretched to fill the entire canvas size.  I don't know enough about ffmpeg to tell if the changes I made are causing this problem; and, I don't understand the rest of the code well enough to know where I might look for a workaround.

I haven't yet found a Mac substitute for the audio feed.  Does anyone have a suggestion?


http://peace.dreadeye.com
I would suggest changing the codec (in the 5th line) to qtrle (you might also need to change the file extension to .mov). This will produce large videos, but I think it would be a good one to start with for testing on the Mac. As far as I know, mpeg4 is a container format not an actual codec. Also, you can uncomment the doResponse(...) lines in the capEncode.pde code to (hopefully) see the errors ffmpeg is giving.

Are you sure h264 doesn't work? I didn't need to do a separate install of the x264 library on Windows. It comes bundled with ffmpeg, but maybe its different for Mac.
I've tried, but h264 doesn't work for me on either Mac or Ubuntu.  I still don't know what is causing half of the sketch canvas to disappear on Mac.

On Ubuntu I am getting the full canvas and have managed to add line in audio:

    "ffmpeg " +
    "-y " +
    "-f alsa -ac 2 -ar 44100 -ab 128k -acodec pcm_s16le " +
    "-i pulse " +
    "-f rawvideo -vcodec rawvideo -pix_fmt bgra " +
    "-s " + ww + "x" + hh + " -i - " +
    "-r 30 -vcodec flv " +
    "-rtbufsi ze 3686400 " +
    saveLoc + "testv.flv";

I have only managed to get Flash video working so far.  By adjusting some of the settings, I can stream to justin.tv which also uses Flash:

    "ffmpeg " +
    "-y " +
    "-f alsa -ac 1 -ar 22050 -ab 64k -acodec pcm_s16le " +
    "-i pulse " +
    "-f rawvideo -vcodec rawvideo -pix_fmt bgra " +
    "-s " + ww + "x" + hh + " -i - " +
    "-r 10 -vcodec flv " +
    "-rtbufsi ze 3686400 " +
    "-f flv rtmp://live.justin.tv/app/***yourStreamKeyHere***";

My laptop is old and slow, so I have to shrink my sketch canvas size way down if I want to add audio.  If the stream starts to stutter or drop audio, I can adjust the following settings:  "-ac" for the number of audio channels, "-ar" for the audio sample rate, "-ab" for the audio bit rate, and "-r" for the outgoing frame rate.  If I decide to stream without audio (comment out the lines with "alsa" and "pulse"), the only settings to play with are the frame rate and canvas size.

Thanks for posting the alternate command line stuff. If you uncomment lines 45-47 in capEncode.pde, do you get any output on the PDE console when you run it on Mac? I'm pretty stumped on the Mac stuff. Its hard because I don't have one to play with. What model Mac is it?
I'm on a MacBook 2.4 GHz Core 2 Duo, 4 GB of memory, running Mac OSX 10.6.8 (Snow Leopard).  Here are the instructions I followed to install FFmpeg:


I have since tried a couple of times to add a library, but no luck.

Yes, I'm getting console output.  Any suggestions as to what to look out for?
I'd hate to start posting guesses, but I would like to have it work on Macs. Has anyone tried it on any other Macs? I've read about problems with Intel graphics devices, but they weren't the half frame stuff you reported. Maybe I'll PM with some guesses.
I would love to try this with Processing 2.0 sketches.  CapEncode seems just to be missing some java libraries, but capSend is more than I can tackle.
I've used it with the betas a little and just today got it to work with AA on 2.0b8. Maybe I'll post them tomorrow if I get a chance. Sorry I didn't get back to you about the Mac issues. Perhaps it will work better on your Mac with the beta version.
Here is
capSendBeta.pde
and
capEncodeBeta.pde.

The sender was registered to send on post() to ensure the frame was completely done drawing, but with the beta on my system for whatever reason AA doesn't work with that setup so the new one sends on draw(). The only problem with that is if you use registerDraw() (or whatever the new version of registered events is), I'm assuming the things you draw in the registered event might not get recorded because the capture code registers first and is called first.