Best video solutions with p5.js

gergelysuto

Hello, I have been playing around with p5.sound and found it amazingly nice and simple to deal with any kind of mixing and timing issues. I wish there were comparably handy p5 tools for video management. What p5.dom offers doesn't seem by far as sophisticated, or it is just not well enough documented (or I am clumsy). Stuff that I'd be happy to be able to do is resizing the video (once with p5.dom I have managed to shrink it by accident but never to go fullscreen), set autostart, set sequences of diverse consecutive videos, and eventually go with superpositions and masking. I have looked into the other p5 libraries but haven't found anything pre-chewed yet. I have found solutions for some of these goals with native javascript (my level is very poor but I have found some code that I borrowed and adapted here and there) and probably some stuff can also be done directly in HTML.

But another solution that seems exciting would be to go with p5.js' native frame concept and split videos back into frames. But it would require a different sound management to fit the video (probably interesting but tedious to split the sound into chunks with crossfades), and most importantly: video compression must be amazingly efficient because I cannot seem to come even close in terms of size with separate images. As an example, a 3 seconds mp4 at 17 frames per second, size 640x360 takes 177 KB, the corresponding images in .jpg format (the smallest I manage) would take 1,5 MB, without the sound.

I'd love to have your opinions about this. cheers, gergely

blindfish

video compression must be amazingly efficient

In comparison to storing separate images of course; since it gets to apply compression over the entire collection of frames and not on a frame by frame basis - as is the case when saving the frames as individual images files.

I think you'd be digging yourself a very big hole if your original source media is video files with embedded audio and you try splitting into separate image and audio files...

In terms of the things you describe most seem relatively straightforward; particularly if you're using Video canvas:

• resizing the video
  Ideally resize source to target size (particularly if smaller) since this delivers minimal file size. Otherwise you can resize a DOM video element easily with CSS and I'd imagine if you're using canvas video you can simly resize the video object.
• set autostart
  Either play or loop the video on setup
• set sequences of diverse consecutive videos
  Store references to videos in an array and use the onended event to step to the next video...
• superpositions and masking
  Using Video canvas this appears straightforward - if I find time I'll put my assumption to the test and post a demo ;)

The one thing that isn't entirely trivial is "to go fullscreen": this is something browsers only let you activate through user interaction with DOM video since taking over the browser screen is considered not playing nice ;)

The alternative depends on how you choose to embed your video. If you use a DOM element then you'll either have to prompt the user or somehow hack the built-in option. If you're using a canvas element I suspect you'll have to make the canvas element go full screen. That is possible using some JS and CSS: I'm experimenting with a fullscreen, transparent and responsive canvas element on my website at the moment; though using PIXIJS library and not p5js...

blindfish

Here's a proof of concept that plays a queue of videos and demonstrates a couple of the things you're looking to do:

"use strict";

var videoSource = ['fingers1.mov', 'fingers2.mov'];
var videos = [];
var currentVideoIndex = 0;

// using instance mode
window['$p5'] = new p5(function (p) {
    p.preload = function() {
         videoSource.forEach(function(val, index) {
            videos[index] = p.createVideo(val);
            videos[index].hide();     
        });
    };

    p.setup = function() {
        p.createCanvas(340, 260);
        // autoplay on launch:
        playNextVideo();
        //resize the second video:
        videos[1].size(60,40);
    };

    p.draw = function() {
        p.background(0);
        if(currentVideoIndex < videos.length) {
            p.image(videos[currentVideoIndex], 10,10);    
        }
    };
});


function playNextVideo() {
    videos[currentVideoIndex].play();

    // attach event listener to catch end of video
    videos[currentVideoIndex].onended(function() {
        console.info("video " + currentVideoIndex + " ended.")
        currentVideoIndex ++;
        if(currentVideoIndex < videos.length) {
            playNextVideo();
        }
        else {
            console.info("reached end of queue");
        }

    });
}

To run this you'll need to download fingers.mov; create two copies in the sketch root; and rename each copy appropriately.

In practice I wouldn't use this code in production: I'd externalise the video management stuff into a pseudo-class and also implement deferred loading of videos: for example you could choose to only ever hold two videos in memory and load the next one in as you move to the next item in the queue...

Note that this demonstrates embedding video in the canvas. TBH if you want to embed in DOM I'd not use p5js and instead use a dedicated video library like video.js.

gergelysuto

Hello blindfish, thank you for the exhaustive answer and solutions. I am processing all the info and will come up with more questions, or, hopefully, with some own results.
...but just simple methods like "drawing" a movie's corresponding frames instead of just running the movie are so powerful since they allow to draw or manipulate on top of the very frame. Sweet :)