Syncing video to audio

rtavakko

Hello everyone,

I recently started working on a project that creates visuals for music using the minim library. All is well until I export the video file using GSvideo and use windows movie maker to create my music video. The audio seems to be quite a bit behind and the visuals don't really make sense (I've tried different frames/s values and still the same issue). I'm using processing 2.0, tried using the GSvideo pipeline but I'm not sure how to export both audio and video at the same time this way (any examples you have would be very much appreciated). Does anyone have any experience with this subject? Any pointers would be great.

Thanks in advance

cadebe

I had similar problems a few weeks back. I tried to movie a minim sketch and spent a few frustrating days trying various options to resolve the audio-video syncing issue. At some stage even the GSVideo option would not work for me. In the end I used the Processing Movie Maker tool (using images created with each frame) and used the frame rate specified in the sketch, which was 60FPS for me (getting the correct frame rate is crucial). I then imported the audio and video files into Adobe After Effects and used Time > Time Stretch to increase the length of the video clip so that it +/- matched that of the audio track, and rendered the video file with the audio. It seemed fairly convincing, even if it is slightly empirical. I don't know if this might be an option for you. If you have managed to come up with a better solution in the meanwhile, do let me know.

rtavakko

Thanks for the post! That is a good solution for the time being, but my goal is to create visuals real-time so I would have to find a method to sync everything without the need for external software. There does not seem to be a good way currently but I will post if I find a better solution.

rtavakko

So the right approach so far to this problem is using Krister's ESS R2 instead of the minim library for FFT analysis. Simultaneously you make a video out of your frames using GSVideo; the crucial thing here is not to display the frames to prevent lag. Instead, you create a PGraphics object and save the frames to that. Once you have the video, you use an external editing software to put the video and audio together (I used moviemaker for this). Video quality is low right now, but I'm working on making HD videos. Will update once I figure that out.

Ben_Farmer

Hi I know this is an old post, but I'll put up my solution (for a Mac user). Like many, I wanted to create a music video to respond to audio frequencies via Minim FFT.

It's a quick, dirty, and free solution for a Mac user. Quicktime player has a screen recording feature that can record and save anything happening on your screen. The process involved recording my screen while the sketch was running, exporting this screen recording into iMovie along with the mp3 of my track, and trimming/matching up the video and audio there.

I've found that the quality of the end result depended on the size of my original sketch. So, I set the size of my sketch to (1280,720) so that it filled up most of the screen.

I exported the video from iMovie at "High Quality" as opposed to "Best Quality" (250 MB file size vs. 1.5 GB for a 2&1/2 minute video). The quality wasn't perfect but it wasn't terrible either.

Again, this is a quick and dirty solution for a Mac user!

Good luck!