Use a video as mask

martoo6

Hello everybody ! I've been trying to use a video as a dynamic mask. But for some reason the following code doesn't seem to work.

import processing.video.*;   

Movie mov; 
boolean cheatScreen; 
PImage img;

void setup() { 
  size(displayWidth, displayHeight);

  // Start capturing the images from the camera 
  mov = new Movie(this, "ad.mp4");
  mov.loop();
  mov.read();

  mov.volume(0); //Uncomment if you want to hear the audio

  img = loadImage("stars.jpeg");   
} 

void draw() {
  if (mov.available() == true) {
    background(0);
    mov.read();

    PImage tmpImg = img.get((width/2)+round(sin(frameCount*0.03)*width*0.25),0,width,height);     

    PImage tmpImg2 = createImage(width, height, RGB);
    tmpImg2.copy(mov,0,0, mov.width, mov.height, 0,0, width, height);//This does the resize itself
    tmpImg2.filter(THRESHOLD);
    tmpImg2.filter(INVERT);

    tmpImg.mask(tmpImg2);

//  PGraphics g = createGraphics(width,height);
//    mov.loadPixels();
//    tmpImg2.loadPixels();
//    tmpImg.loadPixels();
//    g.beginDraw();
//      g.loadPixels();
//        for(int i=0;i<g.pixels.length;i++) if(brightness(tmpImg2.pixels[i])<1) g.pixels[i] = tmpImg.pixels[i];
//      g.updatePixels();
//    g.endDraw();


    set(0, 0, tmpImg);
    if(cheatScreen) set(0, 0, mov);
  }
  text(frameRate, 20, 20);
} 

void keyPressed() { 
  switch (key) { 
  case 'g': 
    saveFrame(); 
    break; 
  case 'c': 
    cheatScreen = !cheatScreen; 
    break;
  }
}

I've also tried to apply a mask using a per pixel function, been the latter too slow for realtime rendering. Any Help would be appreciated !

GoToLoop

Dunno much about this Movie class. So it's just a risky bet of mine: :-SS

• Is there really a need to copy() a whole Movie object to a PImage 1? Aren't they related already?
• @ line #27, wouldn't be faster to instantiate a PImage once within setup() and re-use it?

martoo6

Thanks for answering @GoToLoop !

Movie class is actually part of the Video library.

*I have to copy the whole Movie object in order to resize it, and I can't change Mov object size as it results in a buffer overflow.

*I wouldn't care about instantiating PImage just once on the setup(), as overriding its value would mean harder work for the garbage collector, and using it inside draw makes the garbage collector free space as soon as draw() ends. Anyway, just 30 or 60 frames per second its not a huge overhead.

GoToLoop

I wouldn't care about instantiating PImage just once on the setup(),
as overriding its value would mean harder work for the garbage collector,...

PImage tmpImg2 = createImage(width, height, RGB);

That PImage is uselessly being recreated exactly the same all the time @ 60 FPS!
Since you need it around constantly, you can avoid the hassle of dynamically re-allocating it and spare Java's GB at the same time!
Just instantiate it once within setup()! :> Check it out my mod below. It's called imgMsk now:

//forum.processing.org/two/discussion/2661/use-a-video-as-mask

import processing.video.*;   

Movie mov;
PImage img, imgMsk;
boolean cheatScreen;

void setup() { 
  size(displayWidth, displayHeight);

  mov = new Movie(this, "ad.mp4");

  mov.loop();
  mov.read();
  mov.volume(0);

  img = loadImage("stars.jpeg");

  // Instantiate once, reuse many times:
  imgMsk = createImage(width, height, RGB);
}

void draw() {
  frame.setTitle("FPS: " + frameRate);

  if (!mov.available())  return;

  clear();

  mov.read();

  // Reusing imgMsk rather than instatiating a new 1 every time:
  imgMsk.copy(mov, 0, 0, mov.width, mov.height, 0, 0, width, height);
  imgMsk.filter(THRESHOLD);
  imgMsk.filter(INVERT);

  final PImage imgTmp = img.get(width/2 + 
    round(sin(frameCount*.03))*width/4, 0, width, height);

  imgTmp.mask(imgMsk);
  set(0, 0, imgTmp);

  if (cheatScreen)  set(0, 0, mov);
}

void keyPressed() {
  switch (key) {
  case 'g':
    saveFrame();
    break;
  case 'c':
    cheatScreen = !cheatScreen;
  }
}

GoToLoop

I have to copy the whole Movie object in order to resize it, and I can't change its size as it results in a buffer overflow.

I'm using 64-bit OpenJDK, and it seems incompatible to that video library. So I can't test it! :(
Nevertheless, I'm taking the risk of using a Movie object as it were a PImage 1! :ar!
Well, you can tell us all whether it can possibly work or not: :P

//forum.processing.org/two/discussion/2661/use-a-video-as-mask

import processing.video.*;   

Movie mov;
PImage img;
boolean cheatScreen;

void setup() { 
  size(displayWidth, displayHeight);

  mov = new Movie(this, "ad.mp4");

  mov.loop();
  mov.read();
  mov.volume(0);

  img = loadImage("stars.jpeg");
}

void draw() {
  frame.setTitle("FPS: " + frameRate);

  if (!mov.available())  return;

  mov.read();

  if (cheatScreen) {
    set(0, 0, mov);
    return;
  }

  // Directly using a Movie object rather than copying it to another PImage:
  mov.filter(THRESHOLD);
  mov.filter(INVERT);

  clear();

  final PImage imgTmp = img.get(width/2 + 
    round(sin(frameCount*.03))*width/4, 0, width, height);

  imgTmp.mask(mov);
  set(0, 0, imgTmp);
}

void keyPressed() {
  switch (key) {
  case 'g':
    saveFrame();
    break;
  case 'c':
    cheatScreen = !cheatScreen;
  }
}

martoo6

Oh, I can see your point ! Fool me..he he..

Still, the main point is that I can see the mask-video being resized, but it's like it can't be used as a mask.

Here are some snapshots:

The First one is with the mask applied.

This one is the video as mask.

martoo6

I'm afraid that the last code you posted doesn't work. As I said, images must be the same size. And resizing the Video results in:

java.nio.BufferUnderflowException at java.nio.DirectIntBufferU.get(Unknown Source) at java.nio.IntBuffer.get(Unknown Source) at processing.video.Movie.invokeEvent(Unknown Source) at processing.video.Movie$2.rgbFrame(Unknown Source) at org.gstreamer.elements.RGBDataAppSink$AppSinkNewBufferListener.newBuffer(RGBDataAppSink.java:162) at org.gstreamer.elements.AppSink$2.callback(AppSink.java:184) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:455) at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:485)

:-S

amnon

Realtime masking like this can be best done using a shader:

See: File > Examples > Topics > Shaders > ImageMask

TSH

So did somebody succeed in using a video as a mask with shaders?

Would love to know.

Thanks guys!

martoo6

Yeah, I made it..Do not expect a refined code from an GLSL programmer...anyway:

import processing.video.*; 
PShader maskShader, blur;
PImage srcImage;
PGraphics maskImage;

Movie mov; 

void setup() {
  srcImage = loadImage("carina-nebula.jpg");
  size(displayWidth, displayHeight, P2D);
  maskImage = createGraphics(displayWidth, displayHeight, P2D);
  mov = new Movie(this, "ad.mp4");
  mov.loop();
  mov.read();
  mov.volume(0); //Uncomment if you want to hear the audio

  maskShader = loadShader("mask.glsl");
  blur = loadShader("blur.glsl");
}

void draw() { 
  set(0, 0, srcImage);

  if(mov.available()) {
    mov.read();
    maskImage.beginDraw();
    maskImage.image(mov.get(),0,0,width,height);
    //maskImage.filter(blur);
    //maskImage.filter(THRESHOLD);
    maskImage.endDraw();

    maskShader.set("mask", maskImage);

  }

  filter(maskShader);    
}

mask.glsl:

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

#define PROCESSING_TEXTURE_SHADER

uniform sampler2D texture;
uniform sampler2D mask;

varying vec4 vertColor;
varying vec4 vertTexCoord;

void main() {
  vec4 texColor = texture2D(texture, vertTexCoord.st).rgba;
  vec4 maskColor = texture2D(mask, vec2(vertTexCoord.s, vertTexCoord.t)).rgba;
  //If a PImage is used instead of PGraphics, must invert Y axis
  //vec4 maskColor = texture2D(mask, vec2(vertTexCoord.s, 1.0-vertTexCoord.t)).rgba;
  if(maskColor.r==0){
    gl_FragColor = texColor;
  }else{
    gl_FragColor = vec4(0, 0, 0, 0);
  }
}

blur.glsl it's the same one used in the blur example in the shaders topic.

TSH

Nice, will try this out. Thanks!

TSH

So I've been having performance issues. Screen freezes mainly which I concluded are due to the garbage collector. Is there a way to avoid using the JVM GC as I have enough ram for the assets and data to stay in memory for the length of the sketch?

Any tricks to avoid these garbage collection performance issues?

Thanks!

martoo6

It works fairly good for me. If I set 1920*1080 it starts to lag a little bit, especially using the blur.glsl. But at lower resolutions it should work like a charm. Anyway, I realized there are some improvements over my code:

import processing.video.*;
PShader maskShader, blur;
PImage srcImage;
PGraphics maskImage;

Movie mov;

void setup() {
  srcImage = loadImage("carina-nebula.jpg");
  size(displayWidth, displayHeight, P2D);
  maskImage = createGraphics(displayWidth, displayHeight, P2D);
  mov = new Movie(this, "ad.mp4");
  mov.loop();
  mov.read();
  mov.volume(0); //Uncomment if you want to hear the audio

  maskShader = loadShader("mask.glsl");
  blur = loadShader("blur.glsl");

  maskShader.set("mask", maskImage); //Moved
}

void draw() {
  set(0, 0, srcImage);

  if(mov.available()) {
    mov.read();
    maskImage.beginDraw();
    maskImage.image(mov,0,0,width,height); //The get() wasn't necessary, and might be your bottleneck as it makes a copy of mov's buffer
    //maskImage.filter(blur);
    maskImage.endDraw();



  }

  filter(maskShader);   
}

On the other hand, regarding to you question about the GB. It's not possible to tell the VM to execute the GB when you want, as far as I know you can give it a hit, but the VM will do whatever it wants anyway. Another perfoance booster should be to decrease the mask's image size, as it has to go to the video card and that's a bottleneck as well. Check if the previous code gives you a performance boost and let me know. :)

martoo6

Oh, using the P3D renderer also gave me a performance boost about 6 fps.

avrahamcohenac

Hi,

I created example to how to mask a video & camera. It is very simple.

Have a look at: https://github.com/avrahamcohen/Processing---Video-Camera-Mask-Example..git