Use a video as mask

edited January 2014 in How To...

Hello everybody ! I've been trying to use a video as a dynamic mask. But for some reason the following code doesn't seem to work.

import processing.video.*;   

Movie mov; 
boolean cheatScreen; 
PImage img;

void setup() { 
  size(displayWidth, displayHeight);

  // Start capturing the images from the camera 
  mov = new Movie(this, "ad.mp4");
  mov.loop();
  mov.read();

  mov.volume(0); //Uncomment if you want to hear the audio

  img = loadImage("stars.jpeg");   
} 

void draw() {
  if (mov.available() == true) {
    background(0);
    mov.read();

    PImage tmpImg = img.get((width/2)+round(sin(frameCount*0.03)*width*0.25),0,width,height);     

    PImage tmpImg2 = createImage(width, height, RGB);
    tmpImg2.copy(mov,0,0, mov.width, mov.height, 0,0, width, height);//This does the resize itself
    tmpImg2.filter(THRESHOLD);
    tmpImg2.filter(INVERT);

    tmpImg.mask(tmpImg2);

//  PGraphics g = createGraphics(width,height);
//    mov.loadPixels();
//    tmpImg2.loadPixels();
//    tmpImg.loadPixels();
//    g.beginDraw();
//      g.loadPixels();
//        for(int i=0;i<g.pixels.length;i++) if(brightness(tmpImg2.pixels[i])<1) g.pixels[i] = tmpImg.pixels[i];
//      g.updatePixels();
//    g.endDraw();


    set(0, 0, tmpImg);
    if(cheatScreen) set(0, 0, mov);
  }
  text(frameRate, 20, 20);
} 

void keyPressed() { 
  switch (key) { 
  case 'g': 
    saveFrame(); 
    break; 
  case 'c': 
    cheatScreen = !cheatScreen; 
    break;
  }
}

I've also tried to apply a mask using a per pixel function, been the latter too slow for realtime rendering. Any Help would be appreciated !

Answers

  • Answer ✓

    Dunno much about this Movie class. So it's just a risky bet of mine: :-SS

    • Is there really a need to copy() a whole Movie object to a PImage 1? Aren't they related already?
    • @ line #27, wouldn't be faster to instantiate a PImage once within setup() and re-use it?
  • edited January 2014

    Thanks for answering @GoToLoop !

    Movie class is actually part of the Video library.

    *I have to copy the whole Movie object in order to resize it, and I can't change Mov object size as it results in a buffer overflow.

    *I wouldn't care about instantiating PImage just once on the setup(), as overriding its value would mean harder work for the garbage collector, and using it inside draw makes the garbage collector free space as soon as draw() ends. Anyway, just 30 or 60 frames per second its not a huge overhead.

  • edited January 2014

    I wouldn't care about instantiating PImage just once on the setup(),
    as overriding its value would mean harder work for the garbage collector,...

    PImage tmpImg2 = createImage(width, height, RGB);
    

    That PImage is uselessly being recreated exactly the same all the time @ 60 FPS!
    Since you need it around constantly, you can avoid the hassle of dynamically re-allocating it and spare Java's GB at the same time!
    Just instantiate it once within setup()! :> Check it out my mod below. It's called imgMsk now:

    //forum.processing.org/two/discussion/2661/use-a-video-as-mask
    
    import processing.video.*;   
    
    Movie mov;
    PImage img, imgMsk;
    boolean cheatScreen;
    
    void setup() { 
      size(displayWidth, displayHeight);
    
      mov = new Movie(this, "ad.mp4");
    
      mov.loop();
      mov.read();
      mov.volume(0);
    
      img = loadImage("stars.jpeg");
    
      // Instantiate once, reuse many times:
      imgMsk = createImage(width, height, RGB);
    }
    
    void draw() {
      frame.setTitle("FPS: " + frameRate);
    
      if (!mov.available())  return;
    
      clear();
    
      mov.read();
    
      // Reusing imgMsk rather than instatiating a new 1 every time:
      imgMsk.copy(mov, 0, 0, mov.width, mov.height, 0, 0, width, height);
      imgMsk.filter(THRESHOLD);
      imgMsk.filter(INVERT);
    
      final PImage imgTmp = img.get(width/2 + 
        round(sin(frameCount*.03))*width/4, 0, width, height);
    
      imgTmp.mask(imgMsk);
      set(0, 0, imgTmp);
    
      if (cheatScreen)  set(0, 0, mov);
    }
    
    void keyPressed() {
      switch (key) {
      case 'g':
        saveFrame();
        break;
      case 'c':
        cheatScreen = !cheatScreen;
      }
    }
    
  • edited January 2014

    I have to copy the whole Movie object in order to resize it, and I can't change its size as it results in a buffer overflow.

    I'm using 64-bit OpenJDK, and it seems incompatible to that video library. So I can't test it! :(
    Nevertheless, I'm taking the risk of using a Movie object as it were a PImage 1! :ar!
    Well, you can tell us all whether it can possibly work or not: :P

    //forum.processing.org/two/discussion/2661/use-a-video-as-mask
    
    import processing.video.*;   
    
    Movie mov;
    PImage img;
    boolean cheatScreen;
    
    void setup() { 
      size(displayWidth, displayHeight);
    
      mov = new Movie(this, "ad.mp4");
    
      mov.loop();
      mov.read();
      mov.volume(0);
    
      img = loadImage("stars.jpeg");
    }
    
    void draw() {
      frame.setTitle("FPS: " + frameRate);
    
      if (!mov.available())  return;
    
      mov.read();
    
      if (cheatScreen) {
        set(0, 0, mov);
        return;
      }
    
      // Directly using a Movie object rather than copying it to another PImage:
      mov.filter(THRESHOLD);
      mov.filter(INVERT);
    
      clear();
    
      final PImage imgTmp = img.get(width/2 + 
        round(sin(frameCount*.03))*width/4, 0, width, height);
    
      imgTmp.mask(mov);
      set(0, 0, imgTmp);
    }
    
    void keyPressed() {
      switch (key) {
      case 'g':
        saveFrame();
        break;
      case 'c':
        cheatScreen = !cheatScreen;
      }
    }
    
  • edited January 2014

    Oh, I can see your point ! Fool me..he he..

    Still, the main point is that I can see the mask-video being resized, but it's like it can't be used as a mask.

    Here are some snapshots:

    The First one is with the mask applied.

    Imgur

    This one is the video as mask.

    Imgur

  • I'm afraid that the last code you posted doesn't work. As I said, images must be the same size. And resizing the Video results in:

    java.nio.BufferUnderflowException at java.nio.DirectIntBufferU.get(Unknown Source) at java.nio.IntBuffer.get(Unknown Source) at processing.video.Movie.invokeEvent(Unknown Source) at processing.video.Movie$2.rgbFrame(Unknown Source) at org.gstreamer.elements.RGBDataAppSink$AppSinkNewBufferListener.newBuffer(RGBDataAppSink.java:162) at org.gstreamer.elements.AppSink$2.callback(AppSink.java:184) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:455) at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:485)

    :-S

  • Realtime masking like this can be best done using a shader:

    See: File > Examples > Topics > Shaders > ImageMask

  • So did somebody succeed in using a video as a mask with shaders?

    Would love to know.

    Thanks guys!

  • Yeah, I made it..Do not expect a refined code from an GLSL programmer...anyway:

    import processing.video.*; 
    PShader maskShader, blur;
    PImage srcImage;
    PGraphics maskImage;
    
    Movie mov; 
    
    void setup() {
      srcImage = loadImage("carina-nebula.jpg");
      size(displayWidth, displayHeight, P2D);
      maskImage = createGraphics(displayWidth, displayHeight, P2D);
      mov = new Movie(this, "ad.mp4");
      mov.loop();
      mov.read();
      mov.volume(0); //Uncomment if you want to hear the audio
    
      maskShader = loadShader("mask.glsl");
      blur = loadShader("blur.glsl");
    }
    
    void draw() { 
      set(0, 0, srcImage);
    
      if(mov.available()) {
        mov.read();
        maskImage.beginDraw();
        maskImage.image(mov.get(),0,0,width,height);
        //maskImage.filter(blur);
        //maskImage.filter(THRESHOLD);
        maskImage.endDraw();
    
        maskShader.set("mask", maskImage);
    
      }
    
      filter(maskShader);    
    }
    

    mask.glsl:

    #ifdef GL_ES
    precision mediump float;
    precision mediump int;
    #endif
    
    #define PROCESSING_TEXTURE_SHADER
    
    uniform sampler2D texture;
    uniform sampler2D mask;
    
    varying vec4 vertColor;
    varying vec4 vertTexCoord;
    
    void main() {
      vec4 texColor = texture2D(texture, vertTexCoord.st).rgba;
      vec4 maskColor = texture2D(mask, vec2(vertTexCoord.s, vertTexCoord.t)).rgba;
      //If a PImage is used instead of PGraphics, must invert Y axis
      //vec4 maskColor = texture2D(mask, vec2(vertTexCoord.s, 1.0-vertTexCoord.t)).rgba;
      if(maskColor.r==0){
        gl_FragColor = texColor;
      }else{
        gl_FragColor = vec4(0, 0, 0, 0);
      }
    }
    

    blur.glsl it's the same one used in the blur example in the shaders topic.

  • Nice, will try this out. Thanks!

  • So I've been having performance issues. Screen freezes mainly which I concluded are due to the garbage collector. Is there a way to avoid using the JVM GC as I have enough ram for the assets and data to stay in memory for the length of the sketch?

    Any tricks to avoid these garbage collection performance issues?

    Thanks!

  • It works fairly good for me. If I set 1920*1080 it starts to lag a little bit, especially using the blur.glsl. But at lower resolutions it should work like a charm. Anyway, I realized there are some improvements over my code:

    import processing.video.*;
    PShader maskShader, blur;
    PImage srcImage;
    PGraphics maskImage;
    
    Movie mov;
    
    void setup() {
      srcImage = loadImage("carina-nebula.jpg");
      size(displayWidth, displayHeight, P2D);
      maskImage = createGraphics(displayWidth, displayHeight, P2D);
      mov = new Movie(this, "ad.mp4");
      mov.loop();
      mov.read();
      mov.volume(0); //Uncomment if you want to hear the audio
    
      maskShader = loadShader("mask.glsl");
      blur = loadShader("blur.glsl");
    
      maskShader.set("mask", maskImage); //Moved
    }
    
    void draw() {
      set(0, 0, srcImage);
    
      if(mov.available()) {
        mov.read();
        maskImage.beginDraw();
        maskImage.image(mov,0,0,width,height); //The get() wasn't necessary, and might be your bottleneck as it makes a copy of mov's buffer
        //maskImage.filter(blur);
        maskImage.endDraw();
    
    
    
      }
    
      filter(maskShader);   
    }
    

    On the other hand, regarding to you question about the GB. It's not possible to tell the VM to execute the GB when you want, as far as I know you can give it a hit, but the VM will do whatever it wants anyway. Another perfoance booster should be to decrease the mask's image size, as it has to go to the video card and that's a bottleneck as well. Check if the previous code gives you a performance boost and let me know. :)

  • Oh, using the P3D renderer also gave me a performance boost about 6 fps.

  • Hi,

    I created example to how to mask a video & camera. It is very simple.

    Have a look at: https://github.com/avrahamcohen/Processing---Video-Camera-Mask-Example..git

Sign In or Register to comment.