› ›

Getting video jitter from webcam Capture

prismspecsprismspecs
edited May 2015 in Library Questions

The following code results in a stutter type effect on the live video. If I take the 'scaler' var off (ie, make the Capture video and OpenCV instances the same dimensions) it works (but slower). It seems to be the temp = video.get(); that is throwing it. Any ideas? Here is a video of it in action if that helps.

import gab.opencv.*;
import processing.video.*;
import java.awt.*;

// for movie input
Movie myMovie;

// for live input
Capture video;
PImage temp;  // temporary pimage to copy video capture and resize
boolean live = true;  // are we using live vid or a pre-rendered?

// performing computer vision
OpenCV opencv;

// arraylist of trackbox instances
ArrayList<Trackbox> trackers = new ArrayList<Trackbox>();

int sloganID; // which slogan gets printed below face

int scaler = 2; // ratio of video to CV

int camW = 640;
int camH = 480;

void setup() {
  size(camW, camH, OPENGL);
  //smooth();
  frameRate(30);

  // for live video capture...
  if (live) {
    String[] cameras = Capture.list();

    if (cameras.length == 0) {
      println("There are no cameras available for capture.");
      exit();
    } else {
      println("Available cameras:");
      for (int i = 0; i < cameras.length; i++) {
        println(cameras[i]);
      }
    }

    video = new Capture(this, camW, camH);
    video.start();
  }

  opencv = new OpenCV(this, camW / scaler, camH / scaler);
  temp = new PImage(camW, camH);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);

  if (!live) {
    // load in an video
    myMovie = new Movie(this, "train.mov");
    myMovie.play();
  }
}

void draw() {
  //scale(2);
  if (live) {
    temp = video.get();
    image(video, 0, 0 );
  } else {
    temp = myMovie.get();
    image(myMovie, 0, 0 );
  }

  temp.resize(opencv.width, opencv.height);
  opencv.loadImage(temp);  

  // create array of rectangles around faces in video
  Rectangle[] faces = opencv.detect();

  // see if we need to add or subract trackers
  if (faces.length > trackers.size()) {
    // more faces needed
    for (int i = trackers.size(); i < faces.length; i++) {
      trackers.add(new Trackbox(faces[i].x, faces[i].y, faces[i].width, faces[i].height));
    }
  }
  if (faces.length < trackers.size()) {
    // fewer faces needed
    for (int i = trackers.size(); i > faces.length; i--) {
      trackers.remove(i - 1);
    }
  }

  for (int i = 0; i < trackers.size(); i++) {
    Trackbox t = trackers.get(i);
    t.update(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  }
}

void captureEvent(Capture c) {
  c.read();
}
Tagged:

Answers

  • GoToLoopGoToLoop
    • My advise is turn off auto draw() w/ noLoop().
    • Then use redraw = true inside captureEvent().
    • This way canvas is refreshed only when a new video feed frame arrives!
    • And we don't need get() in order to image() it AFAIK.
    • Take a look at my Movie example in this forum thread:
      http://forum.processing.org/two/discussion/10541/problems-with-video
  • prismspecsprismspecs

    This is great, thank you!

    I use the get() function so that I can shrink the temp PImage and have OpenCV do analysis on that instance rather than the full frame. Is there a better function than get?

  • GoToLoopGoToLoop
    edited May 2015
    • Seems like there are 2 sizes there: bigger for Capture and smaller for OpenCV.
    • In order not to slow down too much sketch's own "Animation" Thread, it seems like a good plan to place both get() & resize() inside captureEvent(), which is run from another Thread btW. ;)
    • Just tweaked my previous Movie to use Capture as well.
    • I don't have any of the hardware though. So check it out for yourself: ;;)

    /**
 * OpenCV Capture Example (v2.0)
 * by GoToLoop (2015/May/06)
 *
 * forum.processing.org/two/discussion/10654/
 * getting-video-jitter-from-webcam-capture
 *
 * forum.processing.org/two/discussion/10541/problems-with-video
 */

import gab.opencv.OpenCV;

import processing.video.Capture;
import processing.video.Movie;

import java.util.List;
import java.awt.Rectangle;

static final boolean LIVE = true;

static final String MOVIE = "train.mov";
static final String RENDERER = P2D;

static final int CAM_W = 640, CAM_H = 480, SCALER = 2;
static final int SMOOTH = 8, FPS = 50;

OpenCV  cv;
Capture ct;
Movie   mv;

volatile PImage  img;
volatile boolean isBusy;

void setup() {
  initFeed();
  size(LIVE? CAM_W : mv.width, LIVE? CAM_H : mv.height, RENDERER);
  noLoop();
  smooth(SMOOTH);
  frameRate(FPS);

  printArray(Capture.list());
  println();

  (cv = new OpenCV(this, width/SCALER, height/SCALER))
    .loadCascade(OpenCV.CASCADE_FRONTALFACE);
}

void draw() {
  background(LIVE? ct : mv);
  if (!isBusy)  thread("frameDetection");
}

void initFeed() {
  if (LIVE) {
    if (ct == null)  (ct = new Capture(this, CAM_W, CAM_H)).start();
  } else {
    if (mv == null)  (mv = new Movie(this, MOVIE)).loop();
    while (mv.width == 0)  delay(10);
  }
}

void resizeFrameImage(PImage frame) {
  PImage tmp = frame.get();
  tmp.resize(cv.width, cv.height);
  img = tmp;
}

void captureEvent(Capture c) {
  c.read();
  redraw = true;
  if (frameCount != 0)  resizeFrameImage(c);
}

void movieEvent(Movie m) {
  m.read();
  redraw = true;
  if (frameCount != 0)  resizeFrameImage(m);
}

void frameDetection() {
  isBusy = true;

  cv.loadImage(img);
  Rectangle[] faces = cv.detect();

  // ...

  isBusy = false;
}
  • prismspecsprismspecs

    This is great, thank you! One major problem though is that when I add the Thread stuff I get this error:

    OpenCV for Processing 0.5.2 by Greg Borenstein http://gregborenstein.com
Using Java OpenCV 2.4.5.0
Load cascade from: /Users/jargon/Documents/Processing/libraries/opencv_processing/library/cascade-files/haarcascade_frontalface_alt.xml
Cascade loaded: haarcascade_frontalface_alt.xml
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff8bf97db1, pid=6311, tid=89383
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build 1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libGL.dylib+0x1db1]  glGetError+0xd
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Applications/Processing.app/Contents/Java/hs_err_pid6311.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Could not run the sketch (Target VM failed to initialize).
For more information, read revisions.txt and Help ? Troubleshooting.
Could not run the sketch.
Finished.
  • prismspecsprismspecs

    Also, if I want to record the non-live movie with all my overlayed effects, is there a better way to do that than to just frameOut every draw loop and recombine JPGs?

  • GoToLoopGoToLoop
    edited May 2015

    As mentioned, I don't have the hardware! So anything I do is just an educated guess.

    If you say that thread() here: if (!isBusy) thread("frameDetection");
    is buggy, you can try w/o it: if (!isBusy) frameDetection();

    By using thread(), we've gotta make sure not to modify sketch's canvas while in it.
    Either you've included such code there or library OpenCV did it! :|

  • GoToLoopGoToLoop

    P.S.: Maybe the problem lies in class Trackbox.
    Since you haven't posted its implementation, it's a mystery whether it modifies sketch's canvas or not! :-\"

  • prismspecsprismspecs

    it does indeed modify the canvas,

    float lerpSpeed = .5; // how quick the box is. lower number is smoother but slower

    class Trackbox {

    //float newX, newY, newW, newH; // tweened values
    PVector coords, dims;

    // testing...
    float rot = 0;  // rotation for other shapes
    float rotSpeed = .1;
    float tSize = 30;

    Trackbox(int x, int y, int w, int h) {
        coords = new PVector(x * SCALER, y * SCALER);
        dims = new PVector(w * SCALER, h * SCALER);

        sloganID++;
        // inc slogan index and reset to 0 if needed
        if (sloganID > slogans.length) sloganID = 0;
    }

    void update(int x, int y, int w, int h) {
        coords.lerp(x * SCALER, y * SCALER, 0, lerpSpeed);
        dims.lerp(w * SCALER, h * SCALER, 0, lerpSpeed);

        // set up color and such
        noFill();
        stroke(255, 255, 0);
        strokeWeight(lineThickness);

        // rect around face
        if (drawRect) {
            rect(coords.x, coords.y, dims.x, dims.y);
        }

        // lines from corners
        if (cornerLines) {
            line(0, 0, coords.x, coords.y);
            line(0, height, coords.x, coords.y + dims.y);
            line(width, height, coords.x + dims.x, coords.y + dims.y);
            line(width, 0, coords.x + dims.x, coords.y);
        }

        // drawing cube
        if (drawBox) {
            pushMatrix();
            translate(coords.x + dims.x / 2, coords.y + dims.y / 2, 0);
            rotateY(rot += rotSpeed);
            rotateX(rot);
            box((dims.x + dims.y) / 2);
            popMatrix();
        }

        // copy face
        if (copyFace) {
            PImage face = get(int(coords.x), int(coords.y), int(dims.x), int(dims.y));
            image(face, 0, 0);
        }

        // filter face
        if (blurFace || threshFace) {
            PImage face = get(int(coords.x), int(coords.y), int(dims.x), int(dims.y));

            if (blurFace)
                face.filter(BLUR, 10);

            if (threshFace)
                face.filter(THRESHOLD, .5);

            image(face, coords.x, coords.y);
        }

        // meta cube
        if (meta) {
            PImage face = get(int(coords.x), int(coords.y), int(dims.x), int(dims.y + 20));
            TexturedCube(face);
        }

        // text
        if (drawText) {
            textAlign(CENTER);
            textSize(tSize);
            text(slogans[sloganID % slogans.length], coords.x + dims.x / 2, coords.y + dims.y + tSize);
        }

    }

    // beta... create a textured cube of face
    void TexturedCube(PImage tex) {

        pushMatrix();
        //tint(255,200);
        translate(coords.x + dims.x / 2, coords.y + dims.y / 2, 0);
        rotateY(rot += rotSpeed);
        rotateX(rot);
        scale((dims.x + dims.y) / 4);
        beginShape(QUADS);
        noStroke();
        textureMode(NORMAL);
        texture(tex);

        // +Z "front" face
        vertex(-1, -1,  1, 0, 0);
        vertex( 1, -1,  1, 1, 0);
        vertex( 1,  1,  1, 1, 1);
        vertex(-1,  1,  1, 0, 1);

        // -Z "back" face
        vertex( 1, -1, -1, 0, 0);
        vertex(-1, -1, -1, 1, 0);
        vertex(-1,  1, -1, 1, 1);
        vertex( 1,  1, -1, 0, 1);

        // +Y "bottom" face
        vertex(-1,  1,  1, 0, 0);
        vertex( 1,  1,  1, 1, 0);
        vertex( 1,  1, -1, 1, 1);
        vertex(-1,  1, -1, 0, 1);

        // -Y "top" face
        vertex(-1, -1, -1, 0, 0);
        vertex( 1, -1, -1, 1, 0);
        vertex( 1, -1,  1, 1, 1);
        vertex(-1, -1,  1, 0, 1);

        // +X "right" face
        vertex( 1, -1,  1, 0, 0);
        vertex( 1, -1, -1, 1, 0);
        vertex( 1,  1, -1, 1, 1);
        vertex( 1,  1,  1, 0, 1);

        // -X "left" face
        vertex(-1, -1, -1, 0, 0);
        vertex(-1, -1,  1, 1, 0);
        vertex(-1,  1,  1, 1, 1);
        vertex(-1,  1, -1, 0, 1);

        endShape();
        noTint();
        popMatrix();
    }
}
  • prismspecsprismspecs

    It works well enough, though. I think I'm more concerned now with capturing it as a video to playback later. Any thoughts on that? Also thanks so much for your hep so far :))

  • GoToLoopGoToLoop
    edited May 2015
    • Indeed, both Trackbox's update() & TexturedCube() methods can't be run outside sketch's "Animation" the way they are now!
    • Actually, Trackbox class can be more organized like separating the update() logic from the display() action!
    • Here's what I did to the class: 

    class Trackbox {
  static final float ROT_SPD  = .1, LERP_SPD = .3;
  static final short TXT_SIZE = 30;

  final PVector coords, dims;
  float rot;

  Trackbox(Rectangle r) {
    coords = new PVector(r.x * SCALER, r.y * SCALER);
    dims = new PVector(r.width * SCALER, r.height * SCALER);

    sloganID = (sloganID + 1) % slogans.length;
  }

  void update(Rectangle r) {
    coords.lerp(r.x * SCALER, r.y * SCALER, 0, LERP_SPD);
    dims.lerp(r.width * SCALER, r.height * SCALER, 0, LERP_SPD);
  }

  void display() {
    // canvas related drawings...
  }
}
    • So now you can call update() outside sketch's "Animation" Thread! \m/
    • Another problem is that you're using lotsa variables which weren't defined inside Trackbox class.
    • Generally we should avoid it b/c it breaks OOP separation of concerns.
    • But I believe you're doing that b/c they're related to some GUI sliders or something.
    • So practicality always trumps cohesiveness! >:)
    • Below is my template example updated to use Trackbox class.
    • Hopefully w/o crashing. Just complete it w/ your own code of course: O:-) 

    /**
 * OpenCV Capture Example (v2.43)
 * by GoToLoop (2015/May/06)
 *
 * forum.processing.org/two/discussion/10654/
 * getting-video-jitter-from-webcam-capture
 *
 * forum.processing.org/two/discussion/10541/problems-with-video
 */

import gab.opencv.OpenCV;

import processing.video.Capture;
import processing.video.Movie;

import java.util.List;
import java.awt.Rectangle;

static final boolean LIVE = false;

static final String MOVIE = "train" + ".mp4";
static final String RENDERER = JAVA2D;

static final short CAM_W = 640, CAM_H = 480;
static final short SCALER = 2, SMOOTH = 0, INTERVAL = 1 * 1000;
static final float FPS = 40.0, SPD = 1.0;

OpenCV  cv;
Capture ct;
Movie   mv;

volatile PImage img;

final List<Trackbox> trackers = new ArrayList<Trackbox>();

final int[] slogans = new int[10];
int sloganID;

void setup() {
  initFeed();
  size(LIVE? CAM_W : mv.width, LIVE? CAM_H : mv.height, RENDERER);

  noLoop();
  smooth(SMOOTH);
  frameRate(FPS);
  imageMode(CORNER);

  thread("initOpenCV");
}

void draw() {
  if (isGL())  image(LIVE? ct : mv, 0, 0);
  else         background(LIVE? ct : mv);

  frame.setTitle(round(frameRate) + ",   " + trackers.size());

  if (!trackers.isEmpty())  synchronized (trackers) {
    for (Trackbox t : trackers)  t.display();
  }
}

void initFeed() {
  if (LIVE) {
    printArray(Capture.list());
    println();
    if (ct == null)  (ct = new Capture(this, CAM_W, CAM_H)).start();
  } else if (mv == null) {
    (mv = new Movie(this, MOVIE)).loop();
    mv.speed(SPD);
    while (mv.width == 0)  delay(10);
  }
}

void initOpenCV() {
  (cv = new OpenCV(this, width/SCALER, height/SCALER))
    .loadCascade(OpenCV.CASCADE_FRONTALFACE);

  while (img == null)  delay(10);

  frameDetection();
}

void resizeFrameImage(PImage frame) {
  if (cv == null || cv.classifier == null)  return;

  PImage tmp = frame.get();
  tmp.resize(cv.width, cv.height);
  img = tmp;
}

void captureEvent(Capture c) {
  c.read();
  redraw = true;
  resizeFrameImage(c);
}

void movieEvent(Movie m) {
  m.read();
  redraw = true;
  resizeFrameImage(m);
}

void frameDetection() {
  for (;; delay(INTERVAL)) {
    cv.loadImage(img);
    Rectangle[] faces = cv.detect();
    int len = faces.length, qty = trackers.size();

    if        (qty < len)  synchronized (trackers) {
      for (int i = qty; i != len; trackers.add(new Trackbox(faces[i++])));
    } else if (qty > len)  synchronized (trackers) {
      for (int i = qty; i != len; trackers.remove(--i));
    }

    for (int i = 0; i != len; trackers.get(i).update(faces[i++]));
  }
}

class Trackbox {
  static final float ROT_SPD  = .1, LERP_SPD = .3;
  static final short TXT_SIZE = 30;

  final PVector coords, dims;
  float rot;

  Trackbox(Rectangle r) {
    coords = new PVector(r.x * SCALER, r.y * SCALER);
    dims = new PVector(r.width * SCALER, r.height * SCALER);

    sloganID = (sloganID + 1) % slogans.length;
  }

  void update(Rectangle r) {
    coords.lerp(r.x * SCALER, r.y * SCALER, 0, LERP_SPD);
    dims.lerp(r.width * SCALER, r.height * SCALER, 0, LERP_SPD);
  }

  void display() {
    // canvas related drawings...
  }
}
  • prismspecsprismspecs

    This is fantastic! Thank you so much for your help. I do have a final question of course ;-)

    When the input is not live, the goal is to output a movie file. I'm comfortable using saveFrames() and rebuilding with another program, but I'd love if there was a way to advance only a single frame at a time in the video so that the resulting video isn't effected by the system lag incurred from CV! Any ideas?

  • GoToLoopGoToLoop

    Perusing its reference: https://processing.org/reference/libraries/video/index.html
    I've found out speed(): https://processing.org/reference/libraries/video/Movie_speed_.html

    Check it out my latest v2.3 template above... B-)

  • GoToLoopGoToLoop
    edited May 2015
    • Since the whole OpenCV's detect() plus the Trackbox's update() is very heavy, another strategy is establishing an INTERVAL for that to execute.
    • For example, a static final int INTERVAL = 2 * 1000; along w/ delay(INTERVAL) would ease the sketch's burden for 2 whole seconds! \m/
    • Take a look at latest v2.4 for that extra feature. You can use both that and the speed(SPD) too. :D
  • prismspecsprismspecs
    edited May 2015

    Ah okay, I was thrown off by speed() not effecting the sound. This is great, thanks so much! In the spirit of continuing our little processing class here ... ;-)

    I'm getting this error only when it's in LIVE mode, any thoughts?

    java.lang.RuntimeException: java.util.ConcurrentModificationException
    at com.jogamp.common.util.awt.AWTEDTExecutor.invoke(AWTEDTExecutor.java:58)
    at jogamp.opengl.awt.AWTThreadingPlugin.invokeOnOpenGLThread(AWTThreadingPlugin.java:103)
    at jogamp.opengl.ThreadingImpl.invokeOnOpenGLThread(ThreadingImpl.java:206)
    at javax.media.opengl.Threading.invokeOnOpenGLThread(Threading.java:172)
    at javax.media.opengl.Threading.invoke(Threading.java:191)
    at javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:541)
    at processing.opengl.PJOGL.requestDraw(PJOGL.java:688)
    at processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:1651)
    at processing.core.PApplet.run(PApplet.java:2256)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
    at java.util.ArrayList$Itr.next(ArrayList.java:831)
    at tracking.draw(tracking.java:114)
    at processing.core.PApplet.handleDraw(PApplet.java:2386)
    at processing.opengl.PJOGL$PGLListener.display(PJOGL.java:862)
    at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:665)
    at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:649)
    at javax.media.opengl.awt.GLCanvas$10.run(GLCanvas.java:1289)
    at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1119)
    at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:994)
    at javax.media.opengl.awt.GLCanvas$11.run(GLCanvas.java:1300)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:302)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
    at java.awt.EventQueue.access$200(EventQueue.java:103)
    at java.awt.EventQueue$3.run(EventQueue.java:694)
    at java.awt.EventQueue$3.run(EventQueue.java:692)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
Could not run the sketch (Target VM failed to initialize).

    code:

    import gab.opencv.*;
import processing.video.*;
import java.awt.*;

import codeanticode.syphon.*;
SyphonServer server;

Movie mv;       // for movie input
Capture ct;     // webcam input
OpenCV cv;  // performing computer vision

// arraylist of trackbox instances
ArrayList<Trackbox> trackers = new ArrayList<Trackbox>();

// arraylist of game objects
ArrayList<GameThing> things = new ArrayList<GameThing>();

static final boolean LIVE = true;

static final String MOVIE = "train.mov";
static final String RENDERER = OPENGL;

static final int CAM_W = 640, CAM_H = 480, SCALER = 2;
static final int SMOOTH = 8, FPS = 30;
float TEXT_SIZE = 30;
static final float MOV_SPD = 0.2;
static final String THE_FONT = "inconsolata.vlw";

volatile PImage  img;     // img resized and passed to CV
volatile boolean isBusy;  // make sure we don't double dip

// graphics contexts
PGraphics gameLayer;
PGraphics pg; // draw everything to this... other than vid?

PFont font;

void setup() {

  initFeed();

  size(LIVE ? CAM_W : mv.width, LIVE ? CAM_H : mv.height, RENDERER);
  smooth(SMOOTH);
  frameRate(FPS);
  noLoop();

  // set up graphics contexts
  gameLayer = createGraphics(width, height, RENDERER);
  pg = createGraphics(width, height, RENDERER);

  cv = new OpenCV(this, width / SCALER, height / SCALER);
  cv.loadCascade(OpenCV.CASCADE_FRONTALFACE);
  img = new PImage(CAM_W, CAM_H);

  font = loadFont(THE_FONT);
  textFont(font, TEXT_SIZE);

  // Create syhpon server to send frames out.
  server = new SyphonServer(this, "Tracking");

  // set up line and font size to be relative to resolution
  lineThickness = .003125 * width;
  TEXT_SIZE = .0234375 * width;
}

void initFeed() {
  if (LIVE) {
    printArray(Capture.list());
    println();
    if (ct == null)  (ct = new Capture(this, CAM_W, CAM_H)).start();
  } else {
    if (mv == null)  (mv = new Movie(this, MOVIE)).play();
    while (mv.width == 0)  delay(10);
    mv.volume(0);
    mv.speed(MOV_SPD);
    println(mv.duration());
  }
}

void draw() {
  // draw background as an image so that it cuts off the
  // 3d shapes on the Z axis
  if (LIVE) {
    image(ct, 0, 0 );
    //pg.image(ct, 0, 0 );
  } else {
    image(mv, 0, 0 );
    //pg.image(mv, 0, 0 );

  }

  synchronized (trackers) {
    for (Trackbox t : trackers)  t.display();
  }

  //if (!isBusy)  frameDetection();


    // check for multiple trackers (add game thing)
    // create game things
    if (game && trackers.size() > 1) {
      Trackbox t1 = trackers.get(0);
      Trackbox t2 = trackers.get(1);

      PVector temp = new PVector(t2.coords.x + t2.dims.x / 2, t2.coords.y + t2.dims.y / 2);
      things.add(new GameThing(t1.coords, t1.dims, temp));
      //println("added game thing");
    }

    // update game things
    if (game) {
      gameLayer.beginDraw();
      //gameLayer.fill(0, 0);
      //gameLayer.rect(0, 0, gameLayer.width, gameLayer.height);
      gameLayer.fill(255, 255, 0);
      for (int i = 0; i < things.size(); i++) {
        GameThing g = things.get(i);
        g.update();
        if(g.pos.dist(g.dest) < 2)
          things.remove(i); // remove it if it's reached its destination
      }
      gameLayer.endDraw();
      image(gameLayer, 0, 0);
    }


  saveFrame("frames/#######.jpg");
  //server.sendImage(get());

}

// take in frame and resize so CV processing is less taxing
void resizeFrameImage(PImage frame) {
  PImage tmp = frame.get();
  tmp.resize(cv.width, cv.height);
  img = tmp;

  if (!isBusy)  frameDetection();
}

// webcam capture event
void captureEvent(Capture c) {
  c.read();
  redraw = true;  // force draw() since we arent auto looping
  if (frameCount != 0)  resizeFrameImage(c);
}

// called every time a new movie frame is available to read
void movieEvent(Movie m) {
  m.read();
  redraw = true;
  if (frameCount != 0)  resizeFrameImage(m);

  sceneStuff();
}
  • GoToLoopGoToLoop
    edited May 2015 

    synchronized (trackers) {
  for (Trackbox t : trackers)  t.display();
}
    • Protecting it against both add() & remove() from another Thread to happen at the same time it's being traversed in the "Animation" Thread: 

    if        (qty < len)  synchronized (trackers) {
  for (int i = qty; i != len; trackers.add(new Trackbox(faces[i++])));
} else if (qty > len)  synchronized (trackers) {
  for (int i = qty; i != len; trackers.remove(--i));
}
    • For that, they gotta share the same reference as their monitor gatekeeper.
    • In their case, I've chosen trackers as the reference object. But coulda been any other Object. ;)
    • Double-check your current code and try to spot where all those loops & size() changes are happening.
  • prismspecsprismspecs
    edited May 2015

    Thanks! sorry, I edited my post but it appears to have reverted.

    You are very right, it would seem. It's my trackers.remove() that it's not diggin...

    int faceCount = 0;

void frameDetection() {
  isBusy = true;

  cv.loadImage(img);

  // create array of rectangles around faces in video
  Rectangle[] faces = cv.detect();

  // SCENARIO 1: faceList is empty
  if (trackers.isEmpty()) {
    // Just make a Trackbox object for every face Rectangle
    for (int i = 0; i < faces.length; i++) {
      trackers.add(new Trackbox(faces[i]));
    }
    // SCENARIO 2: We have fewer Trackbox objects than face Rectangles found from OPENCV
  } else if (trackers.size() <= faces.length) {
    boolean[] used = new boolean[faces.length];
    // Match existing Trackbox objects with a Rectangle
    for (Trackbox f : trackers) {
      // Find faces[index] that is closest to face f
      // set used[index] to true so that it can't be used twice
      float record = 1000000;
      int index = -1;
      for (int i = 0; i < faces.length; i++) {
        float d = dist(faces[i].x, faces[i].y, f.coords.x, f.coords.y);
        if (d < record && !used[i]) {
          record = d;
          index = i;
        }
      }
      // Update Trackbox object location
      used[index] = true;
      f.update(faces[index]);
    }
    // Add any unused faces
    for (int i = 0; i < faces.length; i++) {
      if (!used[i]) {
        trackers.add(new Trackbox(faces[i]));
      }
    }
    // SCENARIO 3: We have more Trackbox objects than face Rectangles found
  } else {
    // All Trackbox objects start out as available
    for (Trackbox f : trackers) {
      f.available = true;
    }
    // Match Rectangle with a Trackbox object
    for (int i = 0; i < faces.length; i++) {
      // Find face object closest to faces[i] Rectangle
      // set available to false
      float record = 1000000;
      int index = -1;
      for (int j = 0; j < trackers.size (); j++) {
        Trackbox f = trackers.get(j);
        float d = dist(faces[i].x, faces[i].y, f.coords.x, f.coords.y);
        if (d < record && f.available) {
          record = d;
          index = j;
        }
      }
      // Update Trackbox object location
      Trackbox f = trackers.get(index);
      f.available = false;
      f.update(faces[i]);
    }

    // Start to kill any left over Trackbox objects
    for (Trackbox f : trackers) {
      if (f.available) {
        f.countDown();
        if (f.dead()) {
          f.delete = true;
        }
      }
    }
  }

  // Delete any that should be deleted
  for (int i = trackers.size () - 1; i >= 0; i--) {
    Trackbox f = trackers.get(i);
    if (f.delete) {
      trackers.remove(i);
    }
  }

  isBusy = false;

}
  • prismspecsprismspecs
    edited May 2015

    I tried editing that portion at the bottom where it removes a tracker to

      // Delete any that should be deleted
  synchronized (trackers) {
    for (int i = trackers.size () - 1; i >= 0; i--) {
      Trackbox f = trackers.get(i);
      if (f.delete) {
        trackers.remove(i);
      }
    }
  }

    but it still throws a concurrent modification exception. Is there some way to just catch this exception so the program won't stop at least?

    It seems to throw more exceptions when I have some modes enabled that use get(), like:

            // filter face
        if (blurFace || threshFace) {
            PImage face = get(int(coords.x), int(coords.y), int(dims.x), int(dims.y));

            if (blurFace)
                face.filter(BLUR, 10);

            if (threshFace)
                face.filter(THRESHOLD, .5);

            image(face, coords.x, coords.y);
        }
  • GoToLoopGoToLoop
    edited May 2015

    Concurrency is a very advanced subject in Java, unfortunately... ~O)

    ... but it still throws a ConcurrentModificationException.

    Since the used loop type there is the regular for ( ; ; ), and not the for ( : ) type, and therefore it's not based on a "fail-fast" Iterator, it can't by itself throw any ConcurrentModificationException. :-B

    However, since it uses remove(), it surely can cause such in some other for ( : ) loop happening in another Thread! :-S
    But be aware that remove() can also throw IndexOutOfBoundsException. :(|)

    For a solution, I dunno how much you figured out about how to use synchronized ().
    In my own example I did it in order to protect draw()'s for ( : ) loop against the "offender"
    for ( ; ; ) loops from frameDetection()'s Thread.
    B/c they were changing tracker's size() via add() & remove(). And that will surely cause ConcurrentModificationException! 8-X

  • GoToLoopGoToLoop
    edited May 2015

    It seems to throw more exceptions when I have some modes enabled that use get()...

    I'm not sure whether get() can be problematic when called outside "Animation"'s Thread.
    However, image() certainly is, b/c it alters sketch's canvas! #-o

    Either use them exclusively when running under "Animation"'s Thread...
    Or use PGraphics in another Thread.
    And later stamp them into canvas via image() when it's under "Animation"'s Thread. *-:)

  • prismspecsprismspecs

    Thank you! Initial tests seem to indicate that putting the synchronize armor around some of the for loops fixed it. I didn't realize add() could also cause these exceptions! GoToLoop, you are the best.

Sign In or Register to comment.