100% CPU utilisation when playing multiple videos using the video library

alzxul

Hi,

I'm trying to play some movies and sound in a processing sketch; there are 10 movies and 1 sound file - all the movies are fairly small [<=5MB]; I create them in setup and call .play() on them, and then in draw() I use image() on them, and I also use movieEvent() function to call .read() on them; whenever I run this the videos lag and my CPU spikes to 100%

Surely playing 10 concurrent movies isn't that onerous?

Does anyone have any ideas for how to improve the performance or can tell me if I'm doing something wrong or missing something?

Any help would be much appreciated, thanks

Thanks

GoToLoop

Asking for help w/o providing the defective code is like expecting us being clairvoyants!
Are they displayed at the same time as well?

alzxul

Yes they are all displayed concurrently.

Here is an example of my code.

Btw I am running a quad core i5 @ 3.3GHz and 8GB RAM.

The videos are all <1 min, <=~5MB (h264 mp4s) and 720x480

import processing.video.*;

Movie myMovie;

Movie myMovie1; 
Movie myMovie2; 
Movie myMovie3; 
Movie myMovie4; 
Movie myMovie5; 
Movie myMovie6; 
Movie myMovie7; 
Movie myMovie8; 
Movie myMovie9; 
Movie myMovie10;
Movie myMovie11;

void setup(){
  size(1920,1080);
  frameRate(30);

myMovie = new Movie(this, "movie.mp4");
  myMovie1 = new Movie(this, "movie1.mp4");
  myMovie2 = new Movie(this, "movie2.mp4");
  myMovie3 = new Movie(this, "movie3.mp4");
  myMovie4 = new Movie(this, "movie4.mp4");
  myMovie5 = new Movie(this, "movie5.mp4");
  myMovie6 = new Movie(this, "movie6.mp4");
  myMovie7 = new Movie(this, "movie7.mp4");

  myMovie8 = new Movie(this, "movie8.mp4");
  myMovie9 = new Movie(this, "movie9.mp4");
  myMovie10 = new Movie(this, "movie10.mp4");;
  myMovie11 = new Movie(this, "movie11.mp4");

myMovie.loop();
  myMovie1.loop();
  myMovie2.loop();
  myMovie3.loop();
  myMovie4.loop();
  myMovie5.loop();
  myMovie6.loop();
  myMovie7.loop();
  myMovie8.loop();
  myMovie9.loop();
  myMovie10.loop();
  myMovie11.loop();
}

void draw(){
    image(myMovie,  0, 0,   480, 360); 
    image(myMovie1, 0, 360, 480, 360);
    image(myMovie2, 0, 720, 480, 360);
    image(myMovie3, 480, 0,   480, 360);
    image(myMovie4, 480, 360, 480, 360);
    image(myMovie5, 480, 720, 480, 360);
    image(myMovie6, 960, 0,   480, 360);
    image(myMovie7, 960, 360, 480, 360);
    image(myMovie8, 960, 720, 480, 360);
    image(myMovie9,  1440, 0,   480, 360);
    image(myMovie10, 1440, 360, 480, 360);
    image(myMovie11, 1440, 720, 480, 360);

}

void movieEvent(Movie m) 
{
  if(m == myMovie) myMovie.read();
  if(m == myMovie1) myMovie1.read();
  if(m == myMovie2) myMovie2.read();
  if(m == myMovie3) myMovie3.read();
  if(m == myMovie4) myMovie4.read();
  if(m == myMovie5) myMovie5.read();
  if(m == myMovie6) myMovie6.read();
  if(m == myMovie7) myMovie7.read();
  if(m == myMovie8) myMovie8.read();
  if(m == myMovie9) myMovie9.read();
  if(m == myMovie10) myMovie10.read();
  if(m == myMovie11) myMovie11.read();
}

koogs

Surely playing 10 concurrent movies isn't that onerous?

reading a frame from a video is hideously complex because of the compression algorithm. look up the h264 spec sometime.

you're then redrawing them, and rescaling them, 30 times a second even if they haven't changed

also, that's 12 movies, not 10. and look into using arrays.

alzxul

reading a frame from a video is hideously complex because of the compression algorithm. look up the h264 spec sometime.

you're then redrawing them, and rescaling them, 30 times a second even if they haven't changed

also, that's 12 movies, not 10. and look into using arrays.

Can I use a less compressed video format then? Would that be faster?

With regards to them changing, I've tried calling .available() on the Movie in the draw method and only calling image() if that's true, but it hasn't changed the performance - that's only drawing it if there's a new frame right?

I've tried reducing the number of movies playing; when it's one or two it's fine, but anything like 4 or above (along with playing sound as well which I haven't even included in the sample code) has the same outcome, so whether it's 9, 10, 11 or 12 movies, it's still maxing out on CPU.

Arrays will make the code nicer, but won't be affecting the fundamental performance.

koogs

Can I use a less compressed video format then? Would that be faster?

it should help. as would resizing the videos to the target size (480x360) to skip the scaling.

but given that 4 or above videos are maxing out your processor and you want 300% of that then you may be SOL.

alzxul

Ok thanks - I'll try to change the videos to be the target size and use a less compressed format.

I was hoping there would be something more obvious like I was using the library in the wrong way! (though maybe the above counts lol)

GoToLoop

If you can, I'd suggest you to re-encode your vids to MPEG-2, the same used by the old DVD movies.
It generates much bigger files than MPEG-4 and everything else, but it's much lighter to decompress.

Also try to re-scale them all to fixed 480x360 rather than asking image() to do it on-the-fly.
Replacing image() w/ set() also helps a lot. Except when using an OpenGL-based renderer.

Also your movieEvent() is completely wrong! We only invoke read() over the passed Movie argument!

Anyways, here's my attempt. It's assuming that all videos are 480x360 & ".mp4"
And all of your vid files are named from "movie0" to "movie11" too.
Of course, change those constants in order to match your current configuration.

It's also daring to use the newest P3's renderer FX2D. Change it back to JAVA2D if you got any problems.

Last warning: I haven't tested this sketch, since I don't have those videos nor CPU power for it: :-\"

/**
 * Grid Movies (v1.0)
 * GoToLoop (2016-Jan-08)
 *
 * forum.Processing.org/two/discussion/14329/
 * 100-cpu-utilisation-when-playing-multiple-videos-using-the-video-library
 */

static final String FILE = "movie", EXT = ".mp4";

static final String RENDERER = FX2D;
//static final String RENDERER = JAVA2D;

static final int COLS = 4, ROWS = 3, QTY = COLS*ROWS;
static final int MW = 480, MH = 360, FPS = 30;
static final int W = MW*COLS, H = MH*ROWS;

import processing.video.Movie;
final Movie[] vids = new Movie[QTY];

void settings() {
  size(W, H, RENDERER);
  //fullScreen(RENDERER);
  noSmooth();
}

void setup() {
  frameRate(FPS);
  println(width, height, W, H);

  for ( int i = 0; i != QTY; 
    (vids[i] = new Movie(this, FILE + i++ + EXT)).loop() );
}

void draw() {
  for ( int row = 0; row != ROWS; ) {
    final int y = MH*row, rc = COLS*row++;
    for ( int col = 0; col != COLS; set(MW*col, y, vids[rc + col++]) );
  }
}

void movieEvent(final Movie m) {
  m.read();
}

koogs

sigh

koogs

here's mine.

the transcoding to mpg and the rescaling helped, it is a lot smoother than the original, but it's still ~100%. and the mpg is 5 times the size of the mp4.

(mjpeg format is older and even less compressed than mpeg2 and helps a tiny bit but is bigger still)

laptop is a dual core centrino, 1368x768, 2009 vintage...

// MovieWall
// acd 20160108

import processing.video.*;

static final int MOVIES = 12;
static final int W = 1368 / 4;  // 342
static final int H = 768 / 3;   // 256

Movie movies[] = new Movie[MOVIES];
boolean flag[] = new boolean[MOVIES];

void setup() {
  size(1366, 768);
  frameRate(30);
  for (int i = 0 ; i < MOVIES ; i++) {
    // movies are 342x256 mpeg2
    // and named movie00.mpg to movie11.mpg
    movies[i] = new Movie(this, "movie" + nf(i, 2) + ".mpg");
    movies[i].loop();
  }
}

void draw() {
  for (int i = 0 ; i < MOVIES ; i++) {
    // only redraw things that have changed
    if (flag[i]) {
      image(movies[i], (i % 4) * W, (i / 4) * H, W, H);
      flag[i] = false;
    }
  }
}

void movieEvent(Movie m) {
  m.read();
  // which movie updated?
  for (int i = 0 ; i < MOVIES ; i++) {
    if (m == movies[i]) {
      flag[i] = true;
      break;
    }
  }
}

alzxul

Hi guys,

Thanks so much - I changed the format of the videos to mpeg2 and resized them to 480x360 which is how they are being displayed on the sketch. I am also using the .set() method as well.

It consumes about 30-50% CPU and they videos and sound play as I expect them to now.

I did try using the FX2D renderer but I get an Exception at start up which says: "You need to use "Import Library" to add FX2D to your sketch" - I've tried searching for this and also how to use the FX2D renderer, but nothing meaningful has come up on it. The Exception messages suggests I would be able to add it from Sketch -> Import Library but when searching for FX2D or anything to do with renderers nothing comes up. I also wasn't able to find an example of someone importing the library/class in their sketch. If you could tell me what I'm missing with that it would be much appreciated.

However your solutions have solved my problem - my thanks for your speedy and informative answers!

GoToLoop

Nice performance ideas from @koogs. Forgotten about the % and integer / trick. =P~
An extra boolean[] can spare the CPU a little by avoiding unnecessary rendering updates.
Still, changing set() back to image() was a bad move! 8-|
Upgraded my own version to use those performance tricks as well: :ar!

/**
 * Grid Movies (v1.1)
 * GoToLoop (2016-Jan-08)
 *
 * forum.Processing.org/two/discussion/14329/
 * 100-cpu-utilisation-when-playing-multiple-videos-using-the-video-library
 */

static final String FILE = "movie", EXT = ".m2v";

static final String RENDERER = FX2D;
//static final String RENDERER = JAVA2D;

static final int COLS = 4, ROWS = 3, QTY = COLS*ROWS;
static final int MW = 480, MH = 360, FPS = 30;
static final int W = MW*COLS, H = MH*ROWS;

import processing.video.Movie;
final Movie[] vids = new Movie[QTY];
final boolean[] flags = new boolean[QTY];

void settings() {
  size(W, H, RENDERER);
  //fullScreen(RENDERER);
  noSmooth();
}

void setup() {
  frameRate(FPS);
  println(width, height, W, H);

  for (int i = 0; i != QTY; 
    (vids[i] = new Movie(this, FILE + i++ + EXT)).loop());
}

void draw() {
  for (int i = 0; i != QTY; ++i)  if (flags[i]) {
    flags[i] = false;
    set(i%4 * MW, (i>>2) * MH, vids[i]);
  }
}

void movieEvent(final Movie m) {
  m.read();

  for (int i = 0;; ++i)  if (m == vids[i]) {
    flags[i] = true;
    return;
  }
}

GoToLoop

"You need to use "Import Library" to add FX2D to your sketch"

Very strange error message! FX2D comes bundled w/ Processing 3. 8-X
Perhaps you're still using P2??? :-t

alzxul

That's what I gathered from your comment - I am definitely on 3! [3.0.1 to be exact]

GoToLoop

Although I'm still @ P v3.0, I can't see why v3.0.1 would fail to recognize FX2D renderer! :-O
Copy & paste my last code w/o any modifications and check whether the error shows up again. ~O)

https://raw.GitHubUserContent.com/processing/processing/master/build/shared/revisions.txt

dehyde

Hi

I've now run into a similar problem. I'm trying to make a "moving font", where each letter of the alphabet is represented by a video, while the videos loop and start and different spots.

Once I'm writing a sentence longer than 30-45 the frame rate drops from 29-30 to 4-5.

String str = "ב באודג הו ודד אא ";
int wordsNum = 1;

int t = 0;

int[] wordsLength;

float kerning = -130;
float leading = 0;
float charLocation = 0;
float lineLocation = 0;

float fontHeight = 122;
float fontWidth = 160;

float horMargin = 100;
float vertMargin = 100;

Movie[] currentLetter;


import processing.video.*;
void setup() {
  size(1200, 900);
  frameRate(30);
  // finding out how many words are there    (important for the wordsLength array)

  for (int i = 0; i<str.length(); i++) {
    if (str.charAt(i)==' ') {
      wordsNum++;
    }
  }
  println("number of words is: ", wordsNum);


  int letCount = 0;

  wordsLength = new int[wordsNum];
  for (int i = 0; i<str.length(); i++) {
    if (str.charAt(i)==' ') {
      wordsLength[t]=letCount;
      letCount=0;
      t++;
    } else {
      letCount++;
    }
  }
  println(wordsLength);

  currentLetter = new Movie[str.length()];
  for (int i=0; i<str.length(); i++) {
    if(str.charAt(i)!=' '){
    currentLetter[i] = new Movie(this, str.charAt(i)+".mov");
    currentLetter[i].loop();
    currentLetter[i].jump(random(currentLetter[i].duration()));
    }
    else{
    currentLetter[i] = new Movie(this, "space.mov");
    }

  }


  t=0;
}

void draw() {
  background(255);
  blendMode(DARKEST);
  charLocation = 0;
  lineLocation = 0;
  t = 0;
  for (int let = 0; let < str.length(); let++) {
    float letterxLocation = width-horMargin+charLocation-fontWidth;
    image(currentLetter[let], letterxLocation, vertMargin+lineLocation, fontWidth, fontHeight);

    // Character Location

    charLocation = charLocation - fontWidth - kerning;

    // checking if it's a new word

    if (str.charAt(let)==' ' && t < wordsNum) {

      // creatng a new line

      if (width-horMargin+charLocation-wordsLength[t]*(fontWidth+kerning) < horMargin) {
        charLocation = 0;
        lineLocation = lineLocation + fontHeight + leading;
      }
      t++;
    }

    //saveFrame("output/framer_####.png");

  }
        fill(0);
  textSize(16);
  text("Frame rate: " + int(frameRate), 10, 20);
}

void movieEvent(Movie m) {
  m.read();
}

jeremydouglass

What is the video content? You should probably use sprites for this, not movies. I would expect a major performance difference.

You can also optimize lookups by doing only 26 (or however many characters) passes. So, load the current sprite sheet state for the a animation, then draw every a on the screen (they are all synchronized, right?) then b, then c... now no matter how many characters you type, you are only doing 26 lookups. This is particularly easy if you use a fixed-width "font".

koogs

there's a resize involved as well

image(currentLetter[let], letterxLocation, vertMargin+lineLocation, fontWidth, fontHeight);

decoding n videos and then downsizing them is going to be terrible even for small values of n.

but this piggyback question already has it's own thread. i'm closing this one.

https://forum.processing.org/two/discussion/25858/associating-videos-to-characters-generatively