Getting audio and video in sync

edited November 2016 in Library Questions

Hi,

I have some code to play a video with different soundtracks (or sometimes no sound) and I can't seem to get them in sync. The sound plays noticeably behind the video, maybe by about a third to a half of a second. Any ideas?

import processing.video.*;
import processing.sound.*;

Movie video;
SoundFile audio;

void setup() {
  size(640, 360);
  background(0);
  video = new Movie(this, "test.mp4");
  audio = new SoundFile(this, "test.mp3");

  video.play();
  audio.play();

}

void movieEvent(Movie m) {
  m.read();
}

void draw() {
  image(video, 0, 0, width, height);
}

Answers

  • I added some timings to my code:

    void setup() {
      size(640, 360);
      background(0);
    
      t0 = millis();
      video = new Movie(this, "test.mp4");
      println("Video loaded at "+(millis() - t0));
      audio = new SoundFile(this, "test.mp3");
      println("Sound loaded at "+(millis() - t0));
      audio.play();
      println("Audio play returned at "+(millis() - t0));
      video.play();
      println("Video play returned at "+(millis() - t0));
    }
    

    This produced the result:

    Video loaded at 4671
    Sound loaded at 4718
    Audio play returned at 4718
    Video play returned at 5110
    

    So it looks like there is a ~400ms delay in the call to video.play(), which is consistent with the delay in the timings.

  • Maybe pause the audio for the required time?
    Or, if possible, run the audio in a seperate thread.(I don't know if it will work)

  • I've concluded that Processing loads video progressively from the file, rather than having the option to buffer ahead of time so it is ready to play. I'll put this in as a feature request!

  • @alisdt -- great! Note that issues / feature requests for Processing-video are in a separate GitHub repository from the main Processing: https://github.com/processing/processing-video/issues

  • Indeed all of those multimedia libraries should have some callback functions like audioStartEvent(), movieEndEvent(). :-@

  • Looks like this issue in Video has already been reported:

    https://github.com/processing/processing-video/issues/20

  • I thought that loading a video file fully would be counterproductive as a lot of RAM would be used up, not to mention the extra amount of time to load large files.
    Or are you suggesting that the library should load only a few seconds of buffer to take care of any delays?

  • @Lord_of_the_Galaxy

    True, I suppose I'm just asking for sufficient buffering to avoid the delay on startup! It would be good to have the option of how much of the video to preload/buffer. For our application (a presentation experiment with a short video) we'd be quite happy to just buy extra RAM and have the whole thing in memory, though I realise this is not the case for most people.

  • While there is no option for buffering in the library as far as I know, you could beforehand load the whole video file into your own custom PImage buffer array/ArrayList, but only if you have enough RAM.

  • Good idea. For that purpose, the code at the end of this thread looks useful:

    https://forum.processing.org/two/discussion/9651/an-arraylist-of-pimages

    Thanks!

  • Is there any library that does support buffering? If so, it would be great.

  • It looks like gstreamer does -- you can just set buffer_duration in playbin -- but this is not exposed through the Java interface, so far as I can tell.

  • gstreamer seems to be a bit under documented(the Java version) so I'm not sure if it a great choice.

  • What you're trying to do is always going to be problematic. GStreamer has internal support for syncing to audio but you're bypassing that - you have a number of different clocks going on that will never keep in sync.

    One option that might improve things is to start the audio in the first call back to movieEvent()?

  • @neilcsmith_net

    Thanks for that suggestion! In this case, it's possible to provide the required visuals and audio using Processing's graphics primitives and the sound package, so I'm going to do that rather than try to use pre-rendered video and audio.

    For the benefit of others searching in the future -- do you know if there's a way to sync to audio via gstreamer from Processing?

  • @alisdt - you should be able to put together a GStreamer pipeline that will do this, but I can't remember offhand whether the Processing library allows you to specify arbitrary GStreamer pipelines, or how it hooks them up. I don't use the Processing video library. I do maintain the GStreamer 1.x Java bindngs.

Sign In or Register to comment.