Process video (overlay, transparency)

darinb

I'm working on a video project where I have to do the same thing again and again to a set of files in Final Cut. I'm a beginning programmer but thought this might be a chance to write something to automate this process in Processing instead.

Does Processing have capabilities for processing video? I see how to capture and display video files, but I've found little on processing those files.

Specifically, I need to take a 4k file (any format you like) and overlay it with another file, at a certain position. I may want to overlay many files, each at a certain position (i.e. offset from the first frame by a specific number of frames with the offset being different for each overlay). I need to set the transparency for each overlay independently.

Can someone point me in the right direction for references, etc? If Processing is the wrong choice for this is there another "easy to use" solution out there?

Thanks,

--Darin

kfrajer

Check https://forum.processing.org/two/search?Search=videoexport

You can install the VideoExport library through the library manager in the Processing IDE. Check the provided examples, which you can access via Filles>>Examples and then under Contributed Libraries folder.

Kf

darinb

I'm sorry, kfrajer, I don'r understand. I don't see how to overlay video clips with VideoExport...do I have to learn ffmpeg?

kfrajer

@darinb If you want to but you don't have to. Take three movies and name them m1, m2, m3 with mp4 extension. Place them in your data folder of your brand new recently created sketch and add the code below there.

This only loads three movies and play them in the same sketch. Press the key 'q' to exit the sketch and save the movie. This new generated movie contains the three previous movies with offsets. For transparency, this might or might not work.

Kf

import com.hamoid.*;

VideoExport videoExport;
import processing.video.*;

final int NMOV=3;

String[] names = {"m1.mp4", "m2.mp4", "m3.mp4"};

Movie m[]=new Movie[NMOV];

// Press 'q' to finish saving the movie and exit.

// In some systems, if you close your sketch by pressing ESC, 
// by closing the window, or by pressing STOP, the resulting 
// movie might be corrupted. If that happens to you, use
// videoExport.endMovie() like you see in this example.

// In some systems pressing ESC produces correct movies
// and .endMovie() is not necessary.

void setup() {
  size(600, 600);

  for (int i=0; i<NMOV; i++) {
    m[i] = new Movie(this, names[i]);
    m[i].loop();
  }

  videoExport = new VideoExport(this);
  videoExport.startMovie();
}
void draw() {
  background(0);
  image(m[0], 0, 0, width/2, height/2);
  image(m[1], width/2, 0, width, height/2);
  image(m[2], width/4, height/2, 3*width/4.0, height);

  videoExport.saveFrame();
}

void movieEvent(Movie m) {
  m.read();
}

void keyPressed() {
  if (key == 'q') {
    videoExport.endMovie();
    exit();
  }
}

Keyword: multiple-movies

jeremydouglass

Different ways to approach this.

Does this need to be realtime, or can it be slower than realtime? If non-realtime, could you write a frame buffer to disk and them compile it into a movie?

Are there sound channels in one or more of the base video or overlays that need to be preserved or mixed-combined.

About how many simultaneous overlays? 2, 10, 100? About how many total? (10, 100, 1000?)

Do the transparency levels endure for each overlay, or are they ramped / crossfades?

darinb

@jeremydouglas Not real time--just need to process and save file. 30 layers in total, each offset by one frame from the one before. 30 frames per second. No audio. The transparency levels are constant for each overlay/layer but all are unique (more or less). The levels are known in advance (e.g. the second overlay has a transparency of x, the 3rd has a transparency of y, etc.).

Not a huge nightmare to do in Final Cut so I have to keep that in mind, but I was also hoping to use this project as an excuse to learn a little about the video capabilities of Processing--although it could be a little to advanced for me.

jeremydouglass

Well, the no audio, no realtime, and set transparency all make this sound easily doable in Processing as a simple project.

I think I understood everything except:

each offset by one frame from the one before

Do you mean: video 1 starts at frame 1 (1 layer), video 2 starts at frame 2 (2 layers), video 3 at frame 3 (3 layers) .. and frames 30+ each have 30 layers, with e.g.

at frame 40, video 1 is at frame 40 and video 30 is at frame 10. video 1 at frame 30 and video 30 at frame 1?

darinb

Sorry for the delay...

Yes, that's it. Basically a staircase shape if you draw it out, with each step being one frame.

--Darin

jeremydouglass

You almost certainly won't be able to open 30 different 4k videos simultaneously, and the opening and closing operations could get really slow if you were trying to open/close them each frame.

If you have a lot of scratch disk space, perhaps instead something like this:

• loop through videos 0-29
  • for each video n open and create an output folder n
    • for each frame in a given video up to MAXFRAMES, save to the output folder using the sum of the loop counters as the name.

Now you have a set of files like this:

/0/0.jpg
/0/1.jpg
/0/2.jpg
/0/3.jpg

/1/1.jpg
/1/2.jpg
/1/3.jpg

/2/2.jpg
/2/3.jpg

/3/3.jpg

• loop through the total frame counter
  • for each frame, check each output folder if it has an image of that name
    • if it does, layer it on at transparency [lookup level]
    • save the frame to /out/ e.g. using saveFrame()

Now you have a set of composited frames in your out directory:

/out/0.jpg   (0)
/out/1.jpg   (0+1)
/out/2.jpg   (0+1+2)
/out/3.jpg   (0+1+2+3)

and for example if your MAXFRAMES was 4 and you had only 5 videos:

/out/4.jpg     (1+2+3+4)
/out/5.jpg       (2+3+4+5)
/out/6.jpg         (3+4+5)
/out/7.jpg           (4+5)
/out/8.jpg             (5)

These will be loaded and compiled into your final video. You could do that final step in Processing as well, or with any framebuffer-to-video tool.

The nice thing about this process is that you only need to load one video at a time -- it is extremely RAM friendly. The bad part is that it can use a ton of disk space and processes n videos one after another (nothing in parallel).

darinb

O.K, I see. In other words, break it all down into still frames, composite what needs composited using still photography tools, then re-assemble into a video from those still frames.

Hmmm. Seems sort of straightforward now that you put it that way... :)

Thanks for the pointer(s)!

--Darin

jeremydouglass

Right -- and compositing a single frame once you have created all your frame buffers is something like:

background(0);
for(String fname : filesForThisFrame){
  PImage img = loadImage(fname);
  tint(255, getOpacity(fname) );
  image(img, 0, 0);
}
saveImage(frameName);

darinb

Great--thanks!

--Darin