100% CPU utilisation when playing multiple videos using the video library

edited January 2016 in Library Questions

Hi,

I'm trying to play some movies and sound in a processing sketch; there are 10 movies and 1 sound file - all the movies are fairly small [<=5MB]; I create them in setup and call .play() on them, and then in draw() I use image() on them, and I also use movieEvent() function to call .read() on them; whenever I run this the videos lag and my CPU spikes to 100%

Surely playing 10 concurrent movies isn't that onerous?

Does anyone have any ideas for how to improve the performance or can tell me if I'm doing something wrong or missing something?

Any help would be much appreciated, thanks

Thanks

Answers

  • Asking for help w/o providing the defective code is like expecting us being clairvoyants!
    Are they displayed at the same time as well?

  • Yes they are all displayed concurrently.

    Here is an example of my code.

    Btw I am running a quad core i5 @ 3.3GHz and 8GB RAM.

    The videos are all <1 min, <=~5MB (h264 mp4s) and 720x480

    
    import processing.video.*;
    
    Movie myMovie;
    Movie myMovie1; Movie myMovie2; Movie myMovie3; Movie myMovie4; Movie myMovie5; Movie myMovie6; Movie myMovie7; Movie myMovie8; Movie myMovie9; Movie myMovie10; Movie myMovie11; void setup(){ size(1920,1080); frameRate(30); myMovie = new Movie(this, "movie.mp4"); myMovie1 = new Movie(this, "movie1.mp4"); myMovie2 = new Movie(this, "movie2.mp4"); myMovie3 = new Movie(this, "movie3.mp4"); myMovie4 = new Movie(this, "movie4.mp4"); myMovie5 = new Movie(this, "movie5.mp4"); myMovie6 = new Movie(this, "movie6.mp4"); myMovie7 = new Movie(this, "movie7.mp4");
    myMovie8 = new Movie(this, "movie8.mp4"); myMovie9 = new Movie(this, "movie9.mp4"); myMovie10 = new Movie(this, "movie10.mp4");; myMovie11 = new Movie(this, "movie11.mp4"); myMovie.loop(); myMovie1.loop(); myMovie2.loop(); myMovie3.loop(); myMovie4.loop(); myMovie5.loop(); myMovie6.loop(); myMovie7.loop(); myMovie8.loop(); myMovie9.loop(); myMovie10.loop(); myMovie11.loop(); } void draw(){ image(myMovie, 0, 0, 480, 360); image(myMovie1, 0, 360, 480, 360); image(myMovie2, 0, 720, 480, 360); image(myMovie3, 480, 0, 480, 360); image(myMovie4, 480, 360, 480, 360); image(myMovie5, 480, 720, 480, 360); image(myMovie6, 960, 0, 480, 360); image(myMovie7, 960, 360, 480, 360); image(myMovie8, 960, 720, 480, 360); image(myMovie9, 1440, 0, 480, 360); image(myMovie10, 1440, 360, 480, 360); image(myMovie11, 1440, 720, 480, 360);
    } void movieEvent(Movie m) { if(m == myMovie) myMovie.read(); if(m == myMovie1) myMovie1.read(); if(m == myMovie2) myMovie2.read(); if(m == myMovie3) myMovie3.read(); if(m == myMovie4) myMovie4.read(); if(m == myMovie5) myMovie5.read(); if(m == myMovie6) myMovie6.read(); if(m == myMovie7) myMovie7.read(); if(m == myMovie8) myMovie8.read(); if(m == myMovie9) myMovie9.read(); if(m == myMovie10) myMovie10.read(); if(m == myMovie11) myMovie11.read(); }
  • Surely playing 10 concurrent movies isn't that onerous?

    reading a frame from a video is hideously complex because of the compression algorithm. look up the h264 spec sometime.

    you're then redrawing them, and rescaling them, 30 times a second even if they haven't changed

    also, that's 12 movies, not 10. and look into using arrays.

  • reading a frame from a video is hideously complex because of the compression algorithm. look up the h264 spec sometime.

    you're then redrawing them, and rescaling them, 30 times a second even if they haven't changed

    also, that's 12 movies, not 10. and look into using arrays.

    Can I use a less compressed video format then? Would that be faster?

    With regards to them changing, I've tried calling .available() on the Movie in the draw method and only calling image() if that's true, but it hasn't changed the performance - that's only drawing it if there's a new frame right?

    I've tried reducing the number of movies playing; when it's one or two it's fine, but anything like 4 or above (along with playing sound as well which I haven't even included in the sample code) has the same outcome, so whether it's 9, 10, 11 or 12 movies, it's still maxing out on CPU.

    Arrays will make the code nicer, but won't be affecting the fundamental performance.

  • Answer ✓

    Can I use a less compressed video format then? Would that be faster?

    it should help. as would resizing the videos to the target size (480x360) to skip the scaling.

    but given that 4 or above videos are maxing out your processor and you want 300% of that then you may be SOL.

  • Ok thanks - I'll try to change the videos to be the target size and use a less compressed format.

    I was hoping there would be something more obvious like I was using the library in the wrong way! (though maybe the above counts lol)

  • edited January 2016 Answer ✓

    If you can, I'd suggest you to re-encode your vids to MPEG-2, the same used by the old DVD movies.
    It generates much bigger files than MPEG-4 and everything else, but it's much lighter to decompress.

    Also try to re-scale them all to fixed 480x360 rather than asking image() to do it on-the-fly.
    Replacing image() w/ set() also helps a lot. Except when using an OpenGL-based renderer.

    Also your movieEvent() is completely wrong! We only invoke read() over the passed Movie argument!

    Anyways, here's my attempt. It's assuming that all videos are 480x360 & ".mp4"
    And all of your vid files are named from "movie0" to "movie11" too.
    Of course, change those constants in order to match your current configuration.

    It's also daring to use the newest P3's renderer FX2D. Change it back to JAVA2D if you got any problems.

    Last warning: I haven't tested this sketch, since I don't have those videos nor CPU power for it: :-\"

    /**
     * Grid Movies (v1.0)
     * GoToLoop (2016-Jan-08)
     *
     * forum.Processing.org/two/discussion/14329/
     * 100-cpu-utilisation-when-playing-multiple-videos-using-the-video-library
     */
    
    static final String FILE = "movie", EXT = ".mp4";
    
    static final String RENDERER = FX2D;
    //static final String RENDERER = JAVA2D;
    
    static final int COLS = 4, ROWS = 3, QTY = COLS*ROWS;
    static final int MW = 480, MH = 360, FPS = 30;
    static final int W = MW*COLS, H = MH*ROWS;
    
    import processing.video.Movie;
    final Movie[] vids = new Movie[QTY];
    
    void settings() {
      size(W, H, RENDERER);
      //fullScreen(RENDERER);
      noSmooth();
    }
    
    void setup() {
      frameRate(FPS);
      println(width, height, W, H);
    
      for ( int i = 0; i != QTY; 
        (vids[i] = new Movie(this, FILE + i++ + EXT)).loop() );
    }
    
    void draw() {
      for ( int row = 0; row != ROWS; ) {
        final int y = MH*row, rc = COLS*row++;
        for ( int col = 0; col != COLS; set(MW*col, y, vids[rc + col++]) );
      }
    }
    
    void movieEvent(final Movie m) {
      m.read();
    }
    
  • Answer ✓

    here's mine.

    the transcoding to mpg and the rescaling helped, it is a lot smoother than the original, but it's still ~100%. and the mpg is 5 times the size of the mp4.

    (mjpeg format is older and even less compressed than mpeg2 and helps a tiny bit but is bigger still)

    laptop is a dual core centrino, 1368x768, 2009 vintage...

    // MovieWall
    // acd 20160108
    
    import processing.video.*;
    
    static final int MOVIES = 12;
    static final int W = 1368 / 4;  // 342
    static final int H = 768 / 3;   // 256
    
    Movie movies[] = new Movie[MOVIES];
    boolean flag[] = new boolean[MOVIES];
    
    void setup() {
      size(1366, 768);
      frameRate(30);
      for (int i = 0 ; i < MOVIES ; i++) {
        // movies are 342x256 mpeg2
        // and named movie00.mpg to movie11.mpg
        movies[i] = new Movie(this, "movie" + nf(i, 2) + ".mpg");
        movies[i].loop();
      }
    }
    
    void draw() {
      for (int i = 0 ; i < MOVIES ; i++) {
        // only redraw things that have changed
        if (flag[i]) {
          image(movies[i], (i % 4) * W, (i / 4) * H, W, H);
          flag[i] = false;
        }
      }
    }
    
    void movieEvent(Movie m) {
      m.read();
      // which movie updated?
      for (int i = 0 ; i < MOVIES ; i++) {
        if (m == movies[i]) {
          flag[i] = true;
          break;
        }
      }
    }
    
  • Hi guys,

    Thanks so much - I changed the format of the videos to mpeg2 and resized them to 480x360 which is how they are being displayed on the sketch. I am also using the .set() method as well.

    It consumes about 30-50% CPU and they videos and sound play as I expect them to now.

    I did try using the FX2D renderer but I get an Exception at start up which says: "You need to use "Import Library" to add FX2D to your sketch" - I've tried searching for this and also how to use the FX2D renderer, but nothing meaningful has come up on it. The Exception messages suggests I would be able to add it from Sketch -> Import Library but when searching for FX2D or anything to do with renderers nothing comes up. I also wasn't able to find an example of someone importing the library/class in their sketch. If you could tell me what I'm missing with that it would be much appreciated.

    However your solutions have solved my problem - my thanks for your speedy and informative answers!

  • edited January 2016

    Nice performance ideas from @koogs. Forgotten about the % and integer / trick. =P~
    An extra boolean[] can spare the CPU a little by avoiding unnecessary rendering updates.
    Still, changing set() back to image() was a bad move! 8-|
    Upgraded my own version to use those performance tricks as well: :ar!

    /**
     * Grid Movies (v1.1)
     * GoToLoop (2016-Jan-08)
     *
     * forum.Processing.org/two/discussion/14329/
     * 100-cpu-utilisation-when-playing-multiple-videos-using-the-video-library
     */
    
    static final String FILE = "movie", EXT = ".m2v";
    
    static final String RENDERER = FX2D;
    //static final String RENDERER = JAVA2D;
    
    static final int COLS = 4, ROWS = 3, QTY = COLS*ROWS;
    static final int MW = 480, MH = 360, FPS = 30;
    static final int W = MW*COLS, H = MH*ROWS;
    
    import processing.video.Movie;
    final Movie[] vids = new Movie[QTY];
    final boolean[] flags = new boolean[QTY];
    
    void settings() {
      size(W, H, RENDERER);
      //fullScreen(RENDERER);
      noSmooth();
    }
    
    void setup() {
      frameRate(FPS);
      println(width, height, W, H);
    
      for (int i = 0; i != QTY; 
        (vids[i] = new Movie(this, FILE + i++ + EXT)).loop());
    }
    
    void draw() {
      for (int i = 0; i != QTY; ++i)  if (flags[i]) {
        flags[i] = false;
        set(i%4 * MW, (i>>2) * MH, vids[i]);
      }
    }
    
    void movieEvent(final Movie m) {
      m.read();
    
      for (int i = 0;; ++i)  if (m == vids[i]) {
        flags[i] = true;
        return;
      }
    }
    
  • edited January 2016

    "You need to use "Import Library" to add FX2D to your sketch"

    Very strange error message! FX2D comes bundled w/ Processing 3. 8-X
    Perhaps you're still using P2??? :-t

  • That's what I gathered from your comment - I am definitely on 3! [3.0.1 to be exact]

  • edited January 2016

    Although I'm still @ P v3.0, I can't see why v3.0.1 would fail to recognize FX2D renderer! :-O
    Copy & paste my last code w/o any modifications and check whether the error shows up again. ~O)

    https://raw.GitHubUserContent.com/processing/processing/master/build/shared/revisions.txt

  • edited January 8

    Hi

    I've now run into a similar problem. I'm trying to make a "moving font", where each letter of the alphabet is represented by a video, while the videos loop and start and different spots.

    Once I'm writing a sentence longer than 30-45 the frame rate drops from 29-30 to 4-5.

    String str = "ב באודג הו ודד אא ";
    int wordsNum = 1;
    
    int t = 0;
    
    int[] wordsLength;
    
    float kerning = -130;
    float leading = 0;
    float charLocation = 0;
    float lineLocation = 0;
    
    float fontHeight = 122;
    float fontWidth = 160;
    
    float horMargin = 100;
    float vertMargin = 100;
    
    Movie[] currentLetter;
    
    
    import processing.video.*;
    void setup() {
      size(1200, 900);
      frameRate(30);
      // finding out how many words are there    (important for the wordsLength array)
    
      for (int i = 0; i<str.length(); i++) {
        if (str.charAt(i)==' ') {
          wordsNum++;
        }
      }
      println("number of words is: ", wordsNum);
    
    
      int letCount = 0;
    
      wordsLength = new int[wordsNum];
      for (int i = 0; i<str.length(); i++) {
        if (str.charAt(i)==' ') {
          wordsLength[t]=letCount;
          letCount=0;
          t++;
        } else {
          letCount++;
        }
      }
      println(wordsLength);
    
      currentLetter = new Movie[str.length()];
      for (int i=0; i<str.length(); i++) {
        if(str.charAt(i)!=' '){
        currentLetter[i] = new Movie(this, str.charAt(i)+".mov");
        currentLetter[i].loop();
        currentLetter[i].jump(random(currentLetter[i].duration()));
        }
        else{
        currentLetter[i] = new Movie(this, "space.mov");
        }
    
      }
    
    
      t=0;
    }
    
    void draw() {
      background(255);
      blendMode(DARKEST);
      charLocation = 0;
      lineLocation = 0;
      t = 0;
      for (int let = 0; let < str.length(); let++) {
        float letterxLocation = width-horMargin+charLocation-fontWidth;
        image(currentLetter[let], letterxLocation, vertMargin+lineLocation, fontWidth, fontHeight);
    
        // Character Location
    
        charLocation = charLocation - fontWidth - kerning;
    
        // checking if it's a new word
    
        if (str.charAt(let)==' ' && t < wordsNum) {
    
          // creatng a new line
    
          if (width-horMargin+charLocation-wordsLength[t]*(fontWidth+kerning) < horMargin) {
            charLocation = 0;
            lineLocation = lineLocation + fontHeight + leading;
          }
          t++;
        }
    
        //saveFrame("output/framer_####.png");
    
      }
            fill(0);
      textSize(16);
      text("Frame rate: " + int(frameRate), 10, 20);
    }
    
    void movieEvent(Movie m) {
      m.read();
    }
    
  • What is the video content? You should probably use sprites for this, not movies. I would expect a major performance difference.

    You can also optimize lookups by doing only 26 (or however many characters) passes. So, load the current sprite sheet state for the a animation, then draw every a on the screen (they are all synchronized, right?) then b, then c... now no matter how many characters you type, you are only doing 26 lookups. This is particularly easy if you use a fixed-width "font".

  • there's a resize involved as well

    image(currentLetter[let], letterxLocation, vertMargin+lineLocation, fontWidth, fontHeight);
    

    decoding n videos and then downsizing them is going to be terrible even for small values of n.

    but this piggyback question already has it's own thread. i'm closing this one.

    https://forum.processing.org/two/discussion/25858/associating-videos-to-characters-generatively

This discussion has been closed.