Getting audio and video in sync

alisdt

Hi,

I have some code to play a video with different soundtracks (or sometimes no sound) and I can't seem to get them in sync. The sound plays noticeably behind the video, maybe by about a third to a half of a second. Any ideas?

import processing.video.*;
import processing.sound.*;

Movie video;
SoundFile audio;

void setup() {
  size(640, 360);
  background(0);
  video = new Movie(this, "test.mp4");
  audio = new SoundFile(this, "test.mp3");

  video.play();
  audio.play();

}

void movieEvent(Movie m) {
  m.read();
}

void draw() {
  image(video, 0, 0, width, height);
}

alisdt

I added some timings to my code:

void setup() {
  size(640, 360);
  background(0);

  t0 = millis();
  video = new Movie(this, "test.mp4");
  println("Video loaded at "+(millis() - t0));
  audio = new SoundFile(this, "test.mp3");
  println("Sound loaded at "+(millis() - t0));
  audio.play();
  println("Audio play returned at "+(millis() - t0));
  video.play();
  println("Video play returned at "+(millis() - t0));
}

This produced the result:

Video loaded at 4671
Sound loaded at 4718
Audio play returned at 4718
Video play returned at 5110

So it looks like there is a ~400ms delay in the call to video.play(), which is consistent with the delay in the timings.

Lord_of_the_Galaxy

Maybe pause the audio for the required time?
Or, if possible, run the audio in a seperate thread.(I don't know if it will work)

alisdt

I've concluded that Processing loads video progressively from the file, rather than having the option to buffer ahead of time so it is ready to play. I'll put this in as a feature request!

jeremydouglass

@alisdt -- great! Note that issues / feature requests for Processing-video are in a separate GitHub repository from the main Processing: https://github.com/processing/processing-video/issues

GoToLoop

Indeed all of those multimedia libraries should have some callback functions like audioStartEvent(), movieEndEvent(). :-@

alisdt

Looks like this issue in Video has already been reported:

https://github.com/processing/processing-video/issues/20

alisdt

I should say thanks all for your replies! For now I will try the workarounds suggested by @GoToLoop elsewhere:

https://forum.processing.org/two/discussion/14159/delay-when-playing-multiple-videos-at-the-same-time https://forum.processing.org/two/discussion/8109/prepare-a-second-video-to-play-without-slowing-down-the-first-video

Lord_of_the_Galaxy

I thought that loading a video file fully would be counterproductive as a lot of RAM would be used up, not to mention the extra amount of time to load large files.
Or are you suggesting that the library should load only a few seconds of buffer to take care of any delays?

alisdt

@Lord_of_the_Galaxy

True, I suppose I'm just asking for sufficient buffering to avoid the delay on startup! It would be good to have the option of how much of the video to preload/buffer. For our application (a presentation experiment with a short video) we'd be quite happy to just buy extra RAM and have the whole thing in memory, though I realise this is not the case for most people.

Lord_of_the_Galaxy

While there is no option for buffering in the library as far as I know, you could beforehand load the whole video file into your own custom PImage buffer array/ArrayList, but only if you have enough RAM.

alisdt

Good idea. For that purpose, the code at the end of this thread looks useful:

https://forum.processing.org/two/discussion/9651/an-arraylist-of-pimages

Thanks!

Lord_of_the_Galaxy

Is there any library that does support buffering? If so, it would be great.

alisdt

It looks like gstreamer does -- you can just set buffer_duration in playbin -- but this is not exposed through the Java interface, so far as I can tell.

Lord_of_the_Galaxy

gstreamer seems to be a bit under documented(the Java version) so I'm not sure if it a great choice.

neilcsmith_net

What you're trying to do is always going to be problematic. GStreamer has internal support for syncing to audio but you're bypassing that - you have a number of different clocks going on that will never keep in sync.

One option that might improve things is to start the audio in the first call back to movieEvent()?

alisdt

@neilcsmith_net

Thanks for that suggestion! In this case, it's possible to provide the required visuals and audio using Processing's graphics primitives and the sound package, so I'm going to do that rather than try to use pre-rendered video and audio.

For the benefit of others searching in the future -- do you know if there's a way to sync to audio via gstreamer from Processing?

neilcsmith_net

@alisdt - you should be able to put together a GStreamer pipeline that will do this, but I can't remember offhand whether the Processing library allows you to specify arbitrary GStreamer pipelines, or how it hooks them up. I don't use the Processing video library. I do maintain the GStreamer 1.x Java bindngs.