Why are the actual camera FPS different from captured FPS?

stefan123 · July 2015

Hi all

I'm using processing for object tracking and I've noticed that the actual frames per second at which video frames are captured is around 7 fps instead of the set value of either 15 fps or 30 fps (I've tried both). At first I thought that this may have to do with the time taken to complete each loop of the image analysis but I've checked and it's only around 23 milliseconds i.e. it should be possible to process at least 40 frames per second.

Does anybody know why this is happening and how to fix it? I've read that one of the problems with using webcams for image capturing is that they place an additional load on the CPU - the data I've extracted leads me to believe that the allocation for the webcam is insufficient while the allocation for running my sketch is more than sufficient. If this is the problem, is there a way to solve it?

Thanks Stefan

GoToLoop · July 2015

Some experiments I had about tweaking captureEvent() + pixels[] + thread("") + synchronized:
http://forum.processing.org/two/discussion/10986/cv-problems

stefan123 · July 2015

First, let me say a big thanks for your answer! Second, let me say that I think I should've said that I'm new to processing and programming in Java.

From the code and the links that you directed me to, am I correct that the "synchronized" statement controls the execution of the various parts of the sketch and "notify()" is used to relay when there is a new frame available? I'm having trouble getting my head around what exactly "void post()" does and what "try" and "catch" do.

I spent quite a bit of time last night trying to include the code from your sketch into my application and eventually decided that I should use a simpler sketch for figuring this out. I put together the sketch below (it doesn't work), the intention is that it uses "synchronized" along with a thread to copy the pixels from the camera frame into a PImage (if I can get this working then I should be able to apply it to my sketch with relative ease).

import processing.video.*;

Capture video;

PImage currFrame;

volatile boolean requestDraw;

void setup() {

  initFeed();

  size(video.width, video.height);

  currFrame = createImage(video.width, video.height, RGB);

  noLoop();

  float drawFPS = video.frameRate + 5;
  frameRate(drawFPS);

  registerMethod("post", this);
  thread("copyCurr");

}

void draw() {

  synchronized (currFrame) {

    background(currFrame);
    frame.setTitle(str(round(frameRate)));

  }

}

void post() {
  if (requestDraw)  requestDraw = !(redraw = true);
}

void captureEvent(Capture video) {

  synchronized(currFrame) {

    video.read();
    currFrame.notify();
    redraw = requestDraw = true;

  }

}

void initFeed() {

  if (video != null)  return;

  String[] cams = Capture.list();
  printArray(cams);
  println();

  (video = new Capture(this, cams[1]) ).start();
  video.updatePixels();
  while (video.width == 0)  delay(10);

}

void copyCurr() {
  synchronized (currFrame) {

    for (int i = 0; i<video.width * video.height; i++) try {

      currFrame.wait();

      currFrame.pixels[i] = video.pixels[i];
      currFrame.updatePixels();

    }

    catch (InterruptedException interrupt) {
      interrupt.printStackTrace();
    }

  }

}

GoToLoop · July 2015

... that the synchronized statement controls the execution of the various parts of the sketch...

This is actually pretty advanced stuff. For further info go here:
http://docs.Oracle.com/javase/tutorial/essential/concurrency/locksync.html

Main usage for synchronized blocks is to protect object states against Thread concurrent access.
The "B&W Capture" sketch got 3 Thread instances:

"Animation" which runs setup(), draw(), post() & initFeed();
"Capture" which runs captureEvent() only.
And the 1 I've created myself w/ thread("RGBtoBW"); which runs RGBtoBW().

Both "Animation" & "RGBtoBW" ends up accessing PImage bw's pixels[] at the same time.
The idea is that while "RGBtoBW" is busy modifying bw's pixels[] array w/ brightness(camPix[idx++]), "Animation" can't background(bw); and vice-versa!
Only 1 Thread can run synchronized blocks under the watch of the same "monitor" object in a given time.

... and notify() is used to relay when there is a new frame available?

After "RGBtoBW" Thread finishes modifying bw's pixels[] array, it is then put to "sleep" by issuing bw.wait();.
In order to wake it up, we need to issue bw.notify();.
Take notice that bw is the monitor for all those 3 synchronized blocks.
And that's exactly what "Capture" Thread does after receiving a new cam frame.
It also wakes "Animation" up w/ redraw = true. :)

I'm having trouble getting my head around what exactly void post() does...

It just assures that redraw keeps being true if requestDraw is still true.
It's just a little trick I've found out which raises the chance redraw() isn't skipped:
https://Processing.org/reference/redraw_.html

... and what try and catch () do.

There are many methods in Java which obliges us to use them, like wait() for example.
B/c calling them can throw exceptions that we have to catch when it happens: :-\"

http://docs.Oracle.com/javase/8/docs/api/java/lang/Object.html#wait-- https://Processing.org/reference/try.html
https://Processing.org/reference/catch.html

GoToLoop · July 2015

The good news is that since you just wanna display what was captured rather than modifying it 1st, the sketch can be much simpler! :bz
We can get rid of PImage bw and all those synchronized & try/catch () blocks! :-bd
Along w/ thread(""), wait() & notify() too! Just leaving registerMethod("post", this);.
Here's the same sketch but more trimmed up: :D

/**
 * Video Capture (v1.04)
 * by GoToLoop (2015/Jul/03)
 *
 * forum.Processing.org/two/discussion/11528/
 * why-are-the-actual-camera-fps-different-from-captured-fps
 */

import processing.video.Capture;
Capture cam;

volatile boolean requestDraw;
static final int FPS_ADJUST = 5, CAM = 1;

void setup() {
  initFeed();
  size(cam.width, cam.height, JAVA2D);

  noLoop();
  float canvasFPS = cam.frameRate + FPS_ADJUST;
  frameRate(canvasFPS);

  println("Cam's FPS:", cam.frameRate, "\t\tCanvas's FPS:", canvasFPS);
  print("Cam's size:", cam.width, 'x', cam.height, '\t');
  println("Canvas's size:", width, 'x', height);

  registerMethod("post", this);
}

void draw() {
  background(cam);
  frame.setTitle(str(round(frameRate)));
}

void post() {
  if (requestDraw)  requestDraw = !(redraw = true);
}

void captureEvent(Capture c) {
  c.read();
  redraw = requestDraw = true;
}

void initFeed() {
  if (cam != null)  return;

  String[] cams = Capture.list();
  printArray(cams);
  println("\nChosen Cam #" + CAM + ':', cams[CAM]);

  ( cam = new Capture(this, cams[CAM]) ).start();
  while ((cam.width & cam.height) == 0)  delay(5);
}

stefan123 · July 2015

Awesome, thanks so much for the detailed response! My actual code does need to modify what was displayed but the first of your two new posts cleared most things up for me!

I have one last question (hopefully), can you explain the use of the "for" loop in your RGBtoBW thread and how it works with the "while" loop? From some experimenting I've done, the "for" loop always yields 0 and without it, the thread only seems to run once?

Thank you again, I really appreciate the support.

GoToLoop · July 2015

... can you explain the use of the for loop in your RGBtoBW() Thread...

You mean this for (;;) loop at: for (int idx = 0;; idx = 0) try {?
Since its 2nd statement (conditional) is empty, it's an infinite loop like while (true) { is.
See its reference here: https://Processing.org/reference/for.html.
I've meant the whole RGBtoBW() Thread to be like an "eternal" service that is either modifying bw.pixels or is in standby w/ wait().
That's why I can't let RGBtoBW() reach the end of its scope; otherwise that Thread ends.
Notice that the try {} ... catch () {} block is inside that "infinite" for (;;) loop.
Thus even if an InterruptedException might be thrown, that loop will still resume!

... and how it works with the while loop?

The while (idx != len) relies on the iterator idx declared @ for (int idx = 0;; idx = 0).
If I had just written for (;;), the internal while () loop would need to become instead:
for (int idx = 0; idx != len; ) :D

stefan123 · July 2015

Yup, that's the for loop I was talking about. I think everything is clear for me now. Thanks a ton for all the assistance! I'm looking forward to getting this included in my code, I've been getting bogged-down for over 2 weeks now with this problem.

stefan123 · July 2015

Sorry, me again...

My code includes events that are triggered by pressing keys and now I'm having trouble including this. I put together the code below as a simple tester for myself. The idea is to use space bar to toggle between snow and the camera.

I had a few println()'s in the code while I was testing but no matter what I did, space bar always ended up being true. Your continued assistance would be much appreciated!!!

import processing.video.Capture;
Capture cam;
PImage bw = new PImage();

volatile boolean requestDraw;
static final int BW_THRESHOLD = 040, FPS_ADJUST = 5, CAM = 1;

boolean spaceBar = false;

int timer = millis();

void setup() {
  initFeed();

  size(cam.width, cam.height, JAVA2D);
  bw = createImage(width, height, RGB);

  noLoop();
  float canvasFPS = cam.frameRate + FPS_ADJUST;
  frameRate(canvasFPS);
  println("Cam's FPS:",cam.frameRate, "\t Canvas's FPS:", canvasFPS);

  registerMethod("post", this);
  thread("RGBtoBW");
}

void draw() {
  synchronized (bw) {
    background(bw);
    frame.setTitle(str(round(frameRate)));
  }
}

void post() {
  if (requestDraw) {
    requestDraw = !(redraw = true);
  }
}

void captureEvent(Capture c) {
  synchronized (bw) {
    c.read();
    bw.notify();
    redraw = requestDraw = true;
  }
}

void initFeed() {
  if (cam != null)  return;

  String[] cams = Capture.list();
  printArray(cams);
  println();

  ( cam = new Capture(this, cams[CAM]) ).start();
  cam.updatePixels();
  while (cam.width == 0)  delay(10);
}

void RGBtoBW() {

  synchronized (bw) {

    final color[] camPix = cam.pixels, bwPix = bw.pixels;
    final int len = camPix.length;

    for (int idx = 0;; idx = 0)  try { 

      bw.wait();

      for (int i = 0; i < len; i++) {

    if (spaceBar = false) {

      bwPix[i] = camPix[i];

    } else if (spaceBar = true) {

      int randnum = int(random(0, 255));
      bwPix[i] = color(randnum, randnum, randnum);

    }

      }

      bw.updatePixels();

    } catch (InterruptedException interrupt) {
      interrupt.printStackTrace();
    }
  }
}

void keyPressed() {

  // toggle snow by pressing spacebar
  // ===========================================

  if (keyPressed == true && key == ' ' && spaceBar == false) {

    spaceBar = true;

  } else if (keyPressed == true && key == ' ' && spaceBar == true) {

    spaceBar = false;

  }

}

GoToLoop · July 2015

Your keyPressed() seems too convoluted for just a simple task of toggling a spacebar!
Here's my tweaked version: O:-)

// forum.Processing.org/two/discussion/11528/
// why-are-the-actual-camera-fps-different-from-captured-fps

import processing.video.Capture;
Capture cam;
PImage snow = new PImage();

volatile boolean requestDraw, isSnowing;
static final int FPS_ADJUST = 5, CAM = 1;

void setup() {
  initFeed();

  size(cam.width, cam.height, JAVA2D);
  snow = createImage(width, height, RGB);

  noLoop();
  float canvasFPS = cam.frameRate + FPS_ADJUST;
  frameRate(canvasFPS);

  println("Cam's FPS:", cam.frameRate, "\t\tCanvas's FPS:", canvasFPS);
  print("Cam's size:", cam.width, 'x', cam.height, '\t');
  println("Canvas's size:", width, 'x', height);

  registerMethod("post", this);
  thread("makeItSnow");
}

void draw() {
  synchronized (snow) {
    background(isSnowing? snow : cam);
    frame.setTitle( str(round(frameRate)) );
  }
}

void keyPressed() {
  isSnowing ^= key == ' ';
}

void post() {
  if (requestDraw)  requestDraw = !(redraw = true);
}

void captureEvent(Capture c) {
  synchronized (snow) {
    c.read();
    snow.notify();
    redraw = requestDraw = true;
  }
}

void initFeed() {
  if (cam != null)  return;

  String[] cams = Capture.list();
  printArray(cams);
  println("\nChosen Cam #" + CAM + ':', cams[CAM]);

  ( cam = new Capture(this, cams[CAM]) ).start();
  while ( (cam.width & cam.height) == 0 )  delay(5);
}

void makeItSnow() {
  synchronized (snow) {
    final color[] snowPix = snow.pixels;
    final int len = snowPix.length;

    for (int idx = 0;; idx = 0)  try { 
      snow.wait();

      if (isSnowing)  while (idx != len)  snowPix[idx++]
        = color((int) random(0400));

      snow.updatePixels();
    }

    catch (InterruptedException interrupt) {
      interrupt.printStackTrace();
    }
  }
}

Howdy, Stranger!

Categories

In this Discussion

Why are the actual camera FPS different from captured FPS?

Best Answers

Answers