How to improve OpenCV performance on ARM?

Isaac96

Hi guys

I am making a face-tracking Nerf blaster using OpenCV on the Raspberry Pi. I am using a Microsoft LifeCam webcam for capture input and the SoftwareServo class for blaster control. However, my code runs at 1-2 FPS on the Pi (Pi 3 model B). I am currently using scale to improve performance, but the code still runs at 1 FPS. Additionally, the servos are extremely jittery. I am powering the servos using a 2A 5V regulator. The Pi is powered off a 2A USB supply. The grounds are connected. Does anyone know how to improve performance? Maybe a different CV library?

Thanks for input! Code:

import processing.io.*;
import gab.opencv.*;
import processing.video.*;
import java.awt.*; 

PImage img;
Rectangle[] faceRect; 

Capture cam;
OpenCV opencv; 
SoftwareServo panServo;
SoftwareServo trigServo;

int widthCapture=320; 
int heightCapture=240;
int fpsCapture=30; 
int panpos=90;
int firePos = 80;
int readyPos = 0;
long time;
int wait = 500;

int targetCenterX;
int targetCenterY;

int threshold = 20;
int thresholdLeft;
int thresholdRight;
int moveIncrement = 2;


int circleExpand = 20;
int circleWidth = 3;

boolean isFiring = false;
boolean isFound = false;
boolean manual = false;

void setup()
{ 
  size (320, 240); 
  frameRate(fpsCapture); 
  background(0);
  panServo = new SoftwareServo(this);
  trigServo = new SoftwareServo(this);
  panServo.attach(17);
  trigServo.attach(4);

  cam = new Capture(this, widthCapture, heightCapture);
  cam.start(); 

  opencv = new OpenCV(this, widthCapture, heightCapture); 
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);
}

void  draw() 
{
  if (millis() - time >= wait)
  {
    trigServo.write(readyPos);
    isFiring = false;
  }
  if (isFiring) 
  {
    trigServo.write(firePos);
    tint(255, 0, 0);
  } else
  {
    trigServo.write(readyPos);
    noTint();
  }
  if (cam.available() == true) 
  { 
    cam.read();  
    img = cam.get(); 

    opencv.loadImage(img);

    image(img, 0, 0);
    blend(img, 0, 0, widthCapture, heightCapture, 0, 0, widthCapture, heightCapture, HARD_LIGHT);
    faceRect = opencv.detect();
  }

  stroke(255, 255, 255);
  strokeWeight(1);
  thresholdLeft = (widthCapture/2)-threshold;
  thresholdRight =  (widthCapture/2)+threshold;

  stroke(255, 255, 255, 128);
  strokeWeight(1);
  line(thresholdLeft, 0, thresholdLeft, heightCapture); //left line
  line(thresholdRight, 0, thresholdRight, heightCapture); //right line

  if ((faceRect != null) && (faceRect.length != 0))
  {
    isFound = true;
    //Get center point of identified target
    targetCenterX = faceRect[0].x + (faceRect[0].width/2);
    targetCenterY = faceRect[0].y + (faceRect[0].height/2);    

    //Draw circle around face
    noFill();
    strokeWeight(circleWidth);
    stroke(255, 255, 255);
    ellipse(targetCenterX, targetCenterY, faceRect[0].width+circleExpand, faceRect[0].height+circleExpand);
    if (!manual) {
      //Handle rotation
      if (targetCenterX < thresholdLeft)
      {
        panpos -=  moveIncrement;
        //delay(70);
      }
      if (targetCenterX > thresholdRight)
      {
        panpos+=  moveIncrement;
        //delay(70);
      }

      //Fire
      if ((targetCenterX >= thresholdLeft) && (targetCenterX <= thresholdRight))
      {
        isFiring = true;
        println("Gotem");
        noFill();
      }
    }
  }
}
void keyPressed() {
  if (key == 'm') {
    manual = !manual;
    println("manual mode toggled");
    isFiring = false;
  } else if (key == 'a' && manual) {
    panpos-= moveIncrement;
    println("left");
  } else if (key == 'f' && manual) {
    isFiring = !isFiring;
  } else if (key == 'd' && manual) {
    panpos+= moveIncrement;
    println("right");
  } else if (key == 'c' )
  {
    panServo.write(90);
  } else {
    println(key);
  }
}

Isaac96

How in the world does one use code tags? :O

koogs

don't

edit post, remove any code tags you've already applied, highlight the code, press ctrl-o...

koogs

and leave a blank line above

koogs

(i've moved this from kinect category to something a bit vaguer. the poster wanted raspberry pi but i don't think the problem is pi-specific, more opencv specific and we don't have a category for that.)

opencv things are fiddly - it's a huge library with lots of expensive functions. face detection was the thing of science fiction when i was at university and now people expect it to run quickly on a £25 computer.

how often is the code in the block at line 74 run? line 77 does the face detection but the img and the blend that follows aren't exactly cheap operations either.

and the block at line 95 seems to run even if the camera image (and faces) haven't changed, which i guess is for some animation stuff. (haven't run the code, don't have opencv)

i guess the fundamental problem is just that the pi is underpowered.

koogs

have just looked at this and even with my whizzy laptop the .detect() method is taking 400ms which is 2.5fps

(camera is 640x480 which is the minimum supported but still)

Isaac96

Thanks for the answers. I have slightly restructured the img and blend loop and FPS is MASSIVELY improved.

  opencv.loadImage(cam);
  image(cam, 0, 0);
  blend(cam, 0, 0, widthCapture, heightCapture, 0, 0, widthCapture, heightCapture, HARD_LIGHT);
  faceRect = opencv.detect();

instead of

  if (cam.available() == true) 
  { 
    cam.read();  
    img = cam.get(); 

    opencv.loadImage(img);

    image(img, 0, 0);
    blend(img, 0, 0, widthCapture, heightCapture, 0, 0, widthCapture, heightCapture, HARD_LIGHT);
    faceRect = opencv.detect();
  }

jeremydouglass

@Isaac96 -- thanks for sharing your solution. If I understand what you are sharing:

1. don't check cam.available(), cam.read(), or copy the contents into a PImage img before drawing with image() and blend()
2. Instead, just pass the cam straight into image() and blend().

...do you think it is the available, the read, or the PImage copy that was taking all that extra time? Might be worth checking with println(millis()). I would be surprised if a single 320x200 PImage copy had such a huge performance hit.

koogs

i did exactly that, put System.getTimeMillis() around each of the lines in the block starting at line 73 (1-11 in that last code block) and they all returned within 10 millis except the detect which was 300-400 (depending on camera size). (and the next draw loop was pretty much immediate - ie the next frame was available by the time the detect had finished)

everything i've read says to check cam.available() (or use the event)

koogs

not seeing an improvement on my laptop.

jeremydouglass

I wonder if it @Isaac96 's specific model of Microsoft LifeCam is extremely slow to respond to cam.available()...?

Isaac96

Well, on my core 2 duo laptop I got massive FPS improvements (4-5 FPS) with the changes I made, not copying to PImage. Using millis() the detect() takes about 180ms. But switching over to the Pi again, the framerate was about 1.75 and detect() took 650ms. However, only one core on the Pi is used and I'm wondering whether it is possible to utilize more CPU? The camera is a 720p model. Thanks for the help!