Combining elements from video and webcam

arnolds112

Would it be possible to add elements from prerecorded video on certain fields over live web footage for example overlaying all moving objects with prerecorded footage? and what would be the best way to aproach this problem?

kfrajer

Do you have the live streaming part already? I ask for my own curiosity.

Check this: https://github.com/processing/p5.js/wiki/Positioning-your-canvas

Kf

Niels

I have been experimenting with video effects on live and prerecorded video.

The sketch below does not completely answer your question, but it uses a live video feed from my laptop's webcam (Mac Book Pro), processes each frame and then renders it on the screen.

The biggest downside is that the frame capture rate from the live video feed is not very high.

BTW: Left-click the mouse to switch between the video effects. ;)

// Video Camera Filter with Modes by Niels Johnson-Laird
// March 20th, 2018
//
// Mode 0 = Pixel Brightness to circle/dot (B&W TV Effect)
// Mode 1 = Pixel Brightness to color circle/dot (B&W TV Effect)
// Mode 2 = Semi-Mosiac effect
// Mode 3 = Color TV with Scanlines Effect
//
// Creative Commons Attribution ShareAlike [https://creativecommons.org/licenses/by-sa/3.0]
//
// See: https://processing.org/reference/libraries/video/index.html

import processing.video.*;

final int rasterScalingFactor = 2;
final int screenScalingFactor = 2;
final int rasterPatternSize = (4 * rasterScalingFactor) - 1;
final int ellipseDiameter = 2;
final int rectSize = 12;
final int mosaicSize = 4;

int modeSelect = 0;
Capture cam;
PImage img;

void settings() {
  size(640 * screenScalingFactor, 360 * screenScalingFactor);
}

void setup() {
  String[] cameras = Capture.list();

  if (cameras.length == 0) {
    println("There are no cameras available for capture.");
    exit();
  } else {
    println("Available cameras:");
    for (int i = 0; i < cameras.length; i++) {
      println("#" + i + " " + cameras[i]);
    }

    // The camera can be initialized directly using an element from the array returned by list():
    // Available cameras on MacBook Pro (Retina, 13-inch, Early 2015):
    //   #0 name=FaceTime HD Camera,size=1280x720,fps=30
    //   #1 name=FaceTime HD Camera,size=1280x720,fps=15
    //   #2 name=FaceTime HD Camera,size=1280x720,fps=1
    //   #3 name=FaceTime HD Camera,size=640x360,fps=30
    //   #4 name=FaceTime HD Camera,size=640x360,fps=15
    //   #5 name=FaceTime HD Camera,size=640x360,fps=1
    //   #6 name=FaceTime HD Camera,size=320x180,fps=30
    //   #7 name=FaceTime HD Camera,size=320x180,fps=15
    //   #8 name=FaceTime HD Camera,size=320x180,fps=1
    //   #9 name=FaceTime HD Camera,size=160x90,fps=30
    //   #10 name=FaceTime HD Camera,size=160x90,fps=15
    //   #11 name=FaceTime HD Camera,size=160x90,fps=1
    //   #12 name=FaceTime HD Camera,size=80x45,fps=30
    //   #13 name=FaceTime HD Camera,size=80x45,fps=15
    //   #14 name=FaceTime HD Camera,size=80x45,fps=1
    cam = new Capture(this, cameras[4]);
    cam.start();
  }

  textSize(36);
} 

void draw() {
  int c = 0;
  int xyOffset = 0;

  // Process a video frame when it is available?
  if (cam.available() == true) { 
    cam.read();
  } else { 
    return; // Nothing to process! :-(
  }

  PImage p = createImage(cam.width, cam.height, RGB);
  p.copy(cam, 0, 0, cam.width, cam.height, 0, 0, cam.width, cam.height);
  img = p;

  img.loadPixels();

  switch (modeSelect)
  {
  case 0: // Old Black and White TV
    background(180);
    xyOffset = ellipseDiameter / 2;

    for (int y = 0; y < img.height; y += ellipseDiameter) {
      for (int x = 0; x < img.width; x += ellipseDiameter) {
        c = int( brightness( img.pixels[(y * img.width + x)] ) );
        fill(c);
        ellipse(x * screenScalingFactor + xyOffset, y * screenScalingFactor + xyOffset, 
          ellipseDiameter * screenScalingFactor, ellipseDiameter * screenScalingFactor);
      } // end for - x
    } // end for - y

    fill(0, 0, 0);
    text("Old Black and White TV Effect", 10, 10, width, height);
    break;

  case 1: // Old Color TV
    background(180);
    xyOffset = ellipseDiameter / 2;

    for (int y = 0; y < img.height; y += ellipseDiameter) {
      for (int x = 0; x < img.width; x += ellipseDiameter) {
        c = int( img.pixels[(y * img.width + x)] );
        fill(c);
        ellipse(x * screenScalingFactor + xyOffset, y * screenScalingFactor + xyOffset, 
          ellipseDiameter * screenScalingFactor, ellipseDiameter * screenScalingFactor);
      } // end for - x
    } // end for - y

    fill(0, 0, 0);
    text("Old Color TV Effect", 10, 10, width, height);
    break;

  case 2: // Semi-Mosiac effect
    int r;

    background(180);

    for (int y = 0; y < img.height; y += mosaicSize) {
      for (int x = 0; x < img.width; x += mosaicSize) {
        c = int( img.pixels[(y * img.width + x)] );
        fill(c, 127); // fill(rgb, alpha) - alpha = opacity [0 to 255]
        r = int(red(c)) / 6;

        if (r < 5) { 
          r = r + int(random(mosaicSize, mosaicSize * 20));
        }
        noStroke();
        //ellipse(x * screenScalingFactor + xyOffset, y * screenScalingFactor + xyOffset, r, r);
        rect(x * screenScalingFactor + xyOffset, y * screenScalingFactor + xyOffset, r, r);
      }
    }

    fill(0, 0, 0);
    text("Semi-Mosiac Effect", 10, 10, width, height);
    break;

  case 3: // Color TV with Scanlines Effect
    background(31);
    for (int y = 0; y < img.height; y += 1) {
      if ( (y % 2) == 0) continue; // Skip frame lines

      for (int x = 0; x < img.width; x += 2) {
        c = int( img.pixels[(y * img.width + x)] );
        fill(c, 127); // fill(rgb, alpha) - alpha = opacity [0 to 255]
        noStroke();
        rect(x * screenScalingFactor + xyOffset, y * screenScalingFactor + xyOffset, 4, 2);
      }
    }

    fill(0, 0, 0);
    text("Color TV with Scanlines Effect", 10, 10, width, height);
    break;

  default:
    break;
  }
} // end draw()

void mouseClicked() {
  modeSelect = (modeSelect + 1) % 4;
}

// End of code.

kfrajer

@Niels

  PImage p = createImage(cam.width, cam.height, RGB);
  p.copy(cam, 0, 0, cam.width, cam.height, 0, 0, cam.width, cam.height);
  img = p;

Try this instead:

  img=cam.get();

Kf

Niels

@kfrajer Thanks! That should improve performance as it doesn't have to copy the entire frame every time through draw(). =D>