Need Help With Pan/Tilt Facetracking with servos and Arduino

Doopsteep

Hello, my project intention is to track a face with OpenCV/Processing and try and center your face with two servos controlles by an arduino. I can confirm that communication between Processing and Arduino does work, my problem is that, suddenly when i run the processing sketch, there is no incoming video although a couple of hours ago it was working. I am using Processing 3.3.5 and Arduino 1.8.2. Also, i'd like to use Firmata as serial does not appear to work for me. I'd also like to add i am not too experienced with Processing nor Arduino.

[Code below]

Thank you in advance.

Doopsteep

I have made some changes to the code, now the video stream works but no response from servos.

Processing:

/**********************************************************************************************
* Pan/Tilt Face Tracking Sketch
* Written by Ryan Owens for SparkFun Electronics
* Uses the OpenCV real-time computer vision  framework from Intel
* Based on the OpenCV Processing Examples from ubaa.net
* This example is released under the Beerware License.
* (Use the code however you'd like, but mention us and by me a beer if we ever meet!)
*
* The Pan/Tilt Face Tracking Sketch interfaces with an Arduino Main board to control
* two servos, pan and tilt, which are connected to a webcam. The OpenCV library
* looks for a face in the image from the webcam. If a face is detected the sketch
* uses the coordinates of the face to manipulate the pan and tilt servos to move the webcam
* in order to keep the face in the center of the frame.
*
* Setup-
* A webcam must be connected to the computer.
* An Arduino must be connected to the computer. Note the port which the Arduino is connected on.
* The Arduino must be loaded with the ServoFirmata Sketch.
* Two servos mounted on a pan/tilt backet must be connected to the Arduino pins 4 and 7.
* The Arduino must be powered by a 9V external power supply.
* 
* Read this tutorial for more information:
**********************************************************************************************/
import gab.opencv.*;
import processing.video.*;
import java.awt.*;
import processing.serial.*;

Capture video;
OpenCV opencv;  //Create an instance of the OpenCV library.

Serial serial; // The serial port

//Variables for keeping track of the current servo positions.
char servoTiltPosition;
char servoPanPosition;
//The pan/tilt servo ids for the Arduino serial command interface.
char tiltChannel = 0;
char panChannel = 1;

//These variables hold the x and y location for the middle of the detected face.
int midFaceY=0;
int midFaceX=0;

//The variables correspond to the middle of the screen, and will be compared to the midFace values
int midScreenY = (height/2);
int midScreenX = (width/2);
int midScreenWindow = 10;  //This is the acceptable 'error' for the center of the screen. 

//The degree of change that will be applied to the servo each time we update the position.
int stepSize=1;

void setup() {
  size(640, 480);
  video = new Capture(this, 640/2, 480/2);
  opencv = new OpenCV(this, 640/2, 480/2);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  

  video.start();

  printArray(Serial.list());


  serial = new Serial(this, "COM3", 57600);

}



void draw() {
  scale(2);
  opencv.loadImage(video);

  image(video, 0, 0 );

  noFill();
  stroke(53, 204, 255);
  strokeWeight(3);
  Rectangle[] faces = opencv.detect();
  println(faces.length);

  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  }


  if(faces.length > 0){
    //If a face was found, find the midpoint of the first face in the frame.
    //NOTE: The .x and .y of the face rectangle corresponds to the upper left corner of the rectangle,
    //      so we manipulate these values to find the midpoint of the rectangle.
    midFaceY = faces[0].y + (faces[0].height/2);
    midFaceX = faces[0].x + (faces[0].width/2);

    //Find out if the Y component of the face is below the middle of the screen.
    if(midFaceY < (midScreenY - midScreenWindow)){
      if(servoTiltPosition >= 5)servoTiltPosition -= stepSize; //If it is below the middle of the screen, update the tilt position variable to lower the tilt servo.
    }
    //Find out if the Y component of the face is above the middle of the screen.
    else if(midFaceY > (midScreenY + midScreenWindow)){
      if(servoTiltPosition <= 175)servoTiltPosition +=stepSize; //Update the tilt position variable to raise the tilt servo.
    }
    //Find out if the X component of the face is to the left of the middle of the screen.
    if(midFaceX < (midScreenX - midScreenWindow)){
      if(servoPanPosition >= 5)servoPanPosition -= stepSize; //Update the pan position variable to move the servo to the left.
    }
    //Find out if the X component of the face is to the right of the middle of the screen.
    else if(midFaceX > (midScreenX + midScreenWindow)){
      if(servoPanPosition <= 175)servoPanPosition +=stepSize; //Update the pan position variable to move the servo to the right.
    }

  }
  //Update the servo positions by sending the serial command to the Arduino.
  serial.write(tiltChannel);      //Send the tilt servo ID
  serial.write(servoTiltPosition); //Send the updated tilt position.
  serial.write(panChannel);        //Send the Pan servo ID
  serial.write(servoPanPosition);  //Send the updated pan position.
  delay(50);
}



void captureEvent(Capture c) {

  c.read();
}

Arduino:

/*******************************************************
* SerialServoControl Sketch
* Written by Ryan Owens for SparkFun Electronics
* 7/15/11
*
* This sketch listens to serial commands and uses the data
* to set the position of two servos.
*
* Serial Command Structure: 2 bytes - [ID Byte][Servo Position byte]
* ID byte should be 0 or 1.
* Servo position should be a value between 0 and 180.
* Invalid commands are ignored
* The servo position is not error checked.
* 
* Hardware Setup
* Servos should be connected to pins 2 and 3 of the Arduino.
* 9V DC Power supply is recommended as USB can't always handle powering two servos
*/
#include <Servo.h>  //Used to control the Pan/Tilt Servos

//These are variables that hold the servo IDs.
char tiltChannel=0, panChannel=1;

//These are the objects for each servo.
Servo servoTilt, servoPan;

//This is a character that will hold data from the Serial port.
char serialChar=0;

void setup(){
  servoTilt.attach(2);  //The Tilt servo is attached to pin 2.
  servoPan.attach(3);   //The Pan servo is attached to pin 3.
  Serial.begin(57600);  //Set up a serial connection for 57600 bps.
}

void loop(){
  if(Serial.available() > 0);  //Wait for a character on the serial port.
  serialChar = Serial.read();     //Copy the character from the serial port to the variable
  if(serialChar == tiltChannel){  //Check to see if the character is the servo ID for the tilt servo
    while(Serial.available() <=0);  //Wait for the second command byte from the serial port.
    servoTilt.write(Serial.read());  //Set the tilt servo position to the value of the second command byte received on the serial port
  }
  else if(serialChar == panChannel){ //Check to see if the initial serial character was the servo ID for the pan servo.
    if(Serial.available() <= 0);  //Wait for the second command byte from the serial port.
    servoPan.write(Serial.read());   //Set the pan servo position to the value of the second command byte received from the serial port.
  }
  //If the character is not the pan or tilt servo ID, it is ignored.
}

Thank you in advance

koogs

Are those ; on the end of lines 37, 40, 44 of the arduino code meant to be there?

Doopsteep

No they aren't but know that i changed it, still no response from servos. I made a change to the processing sketch which fixed the video stream not working.

Processing: https://pastebin.com/TNMndFz1

Arduino: https://pastebin.com/DsuUMX8J

Doopsteep

Also koogs i accidentally pressed a button saying my problem was finished.

kfrajer

Can you please edit your post above and post an updated version of your code? To edit the post, click on the gear icon on the top right corner of your posts.

No response from servos... Have they operated in a stand alone fashion with a simple demo? Do they work? What have you tried? My first question is to make sure they work before we dig into the code. Specially, to make sure you have the right connections.

Also, if this is not your code, do you mind sharing the link of the site where you got the arduino code? This could be potentially helpful in this situation.

Kf