How do you implement java.awt.robot into a sketch?

fxmech

I have been using this code below in my sketch, but it causes processing to stop when it is combined with serial communication to an arduino... The serial part of my sketch works fine as long as I remove all of the code below... if I leave it in, it will error out about 4 out of 5 times... then work fine until I run it again... Is there a chance that I am not setting up the robot class correctly or that the serial class is wrong but somehow works because robot isnt there???

import java.awt.Robot;
import java.awt.AWTException;
Robot robby;
void setup()
{

//robot setup
   try
    {
      robby = new Robot();
    }
    catch (AWTException e)
    {
      println("Robot class not supported by your system!");
      exit();      
    }
  robby.mouseMove(screenResX/2, screenResY/2); //test line to move cursor
}

fxmech

the error I get from processing is: **Error, disabling serialEvent() for COM17 null ** The program does not shut down, but just hangs and doesnt receive serial data the mouse does move to the spot on the screen in setup() and stays there until I escape out of the sketch...

GoToLoop

There's no serialEvent() in your posted code. /:)
Besides, it's not even compilable either! :-@

fxmech

Sorry, here is the rest of it:

import java.awt.Robot;
import java.awt.AWTException;
import java.awt.event.InputEvent; 
import java.awt.Font;
import java.awt.datatransfer.*;
import java.awt.Toolkit;
import processing.serial.*;
import g4p_controls.*;
float roll  = 0.0F;
float pitch = 0.0F;
float yaw   = 0.0F;
float temp  = 0.0F;
float alt   = 0.0F;


int absoluteYawLeftRight;
int yawLeftRight;
int yawDelta;
int  rollDelta;
int  rollUpDown;
int absoluteRollUpDown;

int calibrationSamples=30; //number of samples to average;
int calibrationCounter=0;
int yawSampler;
int rollSampler;

boolean calibrationCompleted=false;


// Serial port state.
Serial       port;
final String serialConfigFile = "serialconfig.txt";
boolean      printSerial = true;

Robot robby;

void setup()
{
size(displayWidth, displayHeight);
  smooth();

  // Serial port setup.
  // Grab list of serial ports and choose one that was persisted earlier or default to the first port.
  int selectedPort = 0;
  String[] availablePorts = Serial.list();
  if (availablePorts == null) {
    println("ERROR: No serial ports available!");
    exit();
  }
  String[] serialConfig = loadStrings(serialConfigFile);
  if (serialConfig != null && serialConfig.length > 0) {
    String savedPort = serialConfig[0];
    // Check if saved port is in available ports.
    for (int i = 0; i < availablePorts.length; ++i) {
      if (availablePorts[i].equals(savedPort)) {
        selectedPort = i;
      } 
    }
  }



//robot setup
   try
    {
      robby = new Robot();
    }
    catch (AWTException e)
    {
      println("Robot class not supported by your system!");
      exit();      
    }
  robby.mouseMove(screenResX/2, screenResY/2); //test line to move cursor
setSerialPort("COM3");


}


void draw()
{
}


  void serialEvent(Serial p) 
{
  String incoming = p.readString();
 if (printSerial) {
  //println(incoming);
 }

  if ((incoming.length() > 8))
  {
    String[] list = split(incoming, " ");
    if ( (list.length > 0) && (list[0].equals("Orientation:")) ) 
    {
      leftClick = float(list[5]);
      rightClick = float(list[4]);
      roll  = float(list[3]); 
      pitch = float(list[2]); // Pitch = Y 
      yaw   = float(list[1]); // yawLeftRight

    }
  }
}

// Set serial port to desired value.
void setSerialPort(String portName) {
  // Close the port if it's currently open.
if (port != null) {
    port.stop();
}
  try {
    // Open port.
    port = new Serial(this, portName, 115200);
    port.bufferUntil('\n');
    // Persist port in configuration.
    saveStrings(serialConfigFile, new String[] { portName });
  }
  catch (RuntimeException ex) {
    // Swallow error if port can't be opened, keep port closed.
    port = null; 
  }
}

GoToLoop

• Still uncompilable! For example, where did you find variables screenResX & screenResY! @-)
• Also, it's somewhat not well indented. CTRL+T inside the PDE auto-indents our code. *-:)
• Are you sure your problem inside serialEvent() is due to class Robot? :-/
• I've got an old solution for serial multi-reading in this link below: O:-)
  https://forum.Processing.org/two/discussion/16618/processing-with-arduino-void-serialevent#Item_1

fxmech

Apologies, Its been years since I post on the boards and will do much better next time.... the sketch is quite large so I deleted the parts that simply compute numbers and execute mouse moves GUIs, etc... I did find that the error occurs when I incorporate the robot class... even without the rest of the code that executes robot commands.... If I remove anything related to robot, it has yet to fail... as soon as I uncomment the robot class I start getting the null error and closes my com port...

Here is the compilable processing code as well as my full arduino sketch the only error you will get is because of the missing config file... the text file just includes the line COM17

Processing:

import java.awt.Robot;
import java.awt.AWTException;
import java.awt.event.InputEvent; 
import java.awt.Font;
import java.awt.datatransfer.*;
import java.awt.Toolkit;
import processing.serial.*;
import g4p_controls.*;
float roll  = 0.0F;
float pitch = 0.0F;
float yaw   = 0.0F;
float temp  = 0.0F;
float alt   = 0.0F;
float rightClick   = 0.0F;
float leftClick   = 0.0F;
int screenResX= 1915;
int screenResY= 1075; 

int absoluteYawLeftRight;
int yawLeftRight;
int yawDelta;
int  rollDelta;
int  rollUpDown;
int absoluteRollUpDown;

int calibrationSamples=30; //number of samples to average;
int calibrationCounter=0;
int yawSampler;
int rollSampler;

boolean calibrationCompleted=false;


// Serial port state.
Serial       port;
final String serialConfigFile = "serialconfig.txt";
boolean      printSerial = true;

Robot robby;

void setup()
{
  size(displayWidth, displayHeight);
  smooth();

  // Serial port setup.
  // Grab list of serial ports and choose one that was persisted earlier or default to the first port.
  int selectedPort = 0;
  String[] availablePorts = Serial.list();
  if (availablePorts == null) {
    println("ERROR: No serial ports available!");
    exit();
  }
  String[] serialConfig = loadStrings(serialConfigFile);
  if (serialConfig != null && serialConfig.length > 0) {
    String savedPort = serialConfig[0];
    // Check if saved port is in available ports.
    for (int i = 0; i < availablePorts.length; ++i) {
      if (availablePorts[i].equals(savedPort)) {
        selectedPort = i;
      }
    }
  }



  //robot setup
  try
  {
    robby = new Robot();
  }
  catch (AWTException e)
  {
    println("Robot class not supported by your system!");
    exit();
  }
  robby.mouseMove(screenResX/2, screenResY/2); //test line to move cursor
  setSerialPort("COM3");
}


void draw()
{
}


void serialEvent(Serial p) 
{
  String incoming = p.readString();
  if (printSerial) {
    //println(incoming);
  }

  if ((incoming.length() > 8))
  {
    String[] list = split(incoming, " ");
    if ( (list.length > 0) && (list[0].equals("Orientation:")) ) 
    {
      leftClick = float(list[5]);
      rightClick = float(list[4]);
      roll  = float(list[3]); 
      pitch = float(list[2]); // Pitch = Y 
      yaw   = float(list[1]); // yawLeftRight
    }
  }
}

// Set serial port to desired value.
void setSerialPort(String portName) {
  // Close the port if it's currently open.
  if (port != null) {
    port.stop();
  }
  try {
    // Open port.
    port = new Serial(this, portName, 115200);
    port.bufferUntil('\n');
    // Persist port in configuration.
    saveStrings(serialConfigFile, new String[] { portName });
  }
  catch (RuntimeException ex) {
    // Swallow error if port can't be opened, keep port closed.
    port = null;
  }
}

Arduino:

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BNO055.h>
#include <utility/imumaths.h>

/* This driver uses the Adafruit unified sensor library (Adafruit_Sensor),
   which provides a common 'type' for sensor data and some helper functions.

   To use this driver you will also need to download the Adafruit_Sensor
   library and include it in your libraries folder.

   You should also assign a unique ID to this sensor for use with
   the Adafruit Sensor API so that you can identify this particular
   sensor in any data logs, etc.  To assign a unique ID, simply
   provide an appropriate value in the constructor below (12345
   is used by default in this example).

   Connections
   ===========
   Connect SCL to analog 5
   Connect SDA to analog 4
   Connect VDD to 3.3-5V DC
   Connect GROUND to common ground

   History
   =======
   2015/MAR/03  - First release (KTOWN)
*/

/* Set the delay between fresh samples */
#define BNO055_SAMPLERATE_DELAY_MS (100)

Adafruit_BNO055 bno = Adafruit_BNO055(55);

/**************************************************************************/
/*
    Displays some basic information on this sensor from the unified
    sensor API sensor_t type (see Adafruit_Sensor for more information)
*/
/**************************************************************************/
void displaySensorDetails(void)
{
  sensor_t sensor;
  bno.getSensor(&sensor);
  Serial.println("------------------------------------");
  Serial.print  ("Sensor:       "); Serial.println(sensor.name);
  Serial.print  ("Driver Ver:   "); Serial.println(sensor.version);
  Serial.print  ("Unique ID:    "); Serial.println(sensor.sensor_id);
  Serial.print  ("Max Value:    "); Serial.print(sensor.max_value); Serial.println(" xxx");
  Serial.print  ("Min Value:    "); Serial.print(sensor.min_value); Serial.println(" xxx");
  Serial.print  ("Resolution:   "); Serial.print(sensor.resolution); Serial.println(" xxx");
  Serial.println("------------------------------------");
  Serial.println("");
  delay(500);
}

/**************************************************************************/
/*
    Arduino setup function (automatically called at startup)
*/
/**************************************************************************/
void setup(void)
{
  Serial.begin(115200);
  pinMode(5, INPUT_PULLUP);
  pinMode(6, INPUT_PULLUP);
  // Serial.println("Orientation Sensor Test"); Serial.println("");

  /* Initialise the sensor */
  if (!bno.begin())
  {
    /* There was a problem detecting the BNO055 ... check your connections */
    Serial.print("Ooops, no BNO055 detected ... Check your wiring or I2C ADDR!");
    while (1);
  }

  delay(1000);

  /* Use external crystal for better accuracy */
  bno.setExtCrystalUse(true);

  /* Display some basic information on this sensor */
  // displaySensorDetails();
}

/**************************************************************************/
/*
    Arduino loop function, called once 'setup' is complete (your own code
    should go here)
*/
/**************************************************************************/
void loop(void)
{
  int rightClick = digitalRead(6);
  int leftClick = digitalRead(5);
  /* Get a new sensor event */
  sensors_event_t event;
  bno.getEvent(&event);

  /* Board layout:
         +----------+
         |         *| RST   PITCH  ROLL  HEADING
     ADR |*        *| SCL
     INT |*        *| SDA     ^            /->
     PS1 |*        *| GND     |            |
     PS0 |*        *| 3VO     Y    Z-->    \-X
         |         *| VIN
         +----------+
  */

  /* The processing sketch expects data as roll, pitch, heading */
  Serial.print(F("Orientation: "));
  Serial.print((float)event.orientation.x);
  Serial.print(F(" "));
  Serial.print((float)event.orientation.y);
  Serial.print(F(" "));
  Serial.print((float)event.orientation.z);
  Serial.print(F(" "));

  Serial.print((float)rightClick);
  Serial.print(F(" "));
  Serial.print((float)leftClick);
  Serial.println(F(""));
  /*
    // Also send calibration data for each sensor.
    uint8_t sys, gyro, accel, mag = 0;
    bno.getCalibration(&sys, &gyro, &accel, &mag);
    Serial.print(F("Calibration: "));
    Serial.print(sys, DEC);
    Serial.print(F(" "));
    Serial.print(gyro, DEC);
    Serial.print(F(" "));
    Serial.print(accel, DEC);
    Serial.print(F(" "));
    Serial.println(mag, DEC);
  */
  delay(BNO055_SAMPLERATE_DELAY_MS);
}

fxmech

I will try the example that you provided a link too... It would be extremely helpful to at least get an basic understanding of what is the root of my error in the sketch as it stands... the error message is very vague and my google searches haven't come up with a cause yet... I really appreciate the help and apologize for my incomplete initial post...

fxmech

I tried your code in place of my old serial method I now get this error instead:

null pointer exception [0] "COM17" null

fxmech

looks like adding a 1 second delay in the setup function of processing solved the null issue... Your code is so much simpler and efficient for serial.. Thank you for that!

GoToLoop

Looks like adding a 1 second delay in the setup function of processing solved the null issue...

It takes some time before serialEvent() is called back for the 1st time.
Given it's an asynchronous call, if you got anything which depends on serialEvent() to initialize it before usage, it may break your code if it happens before serialEvent() is executed.

However, it's unreliable to depend on some arbitrary delay() millisecond value to w8 for 1st serialEvent() execution.
In your current computer, 1 millisecond was enough. For another, it may need more or none at all.

Rather, at the end of setup(), you can add an infinite loop + delay(), which checks whether the problematic value was properly initialized within serialEvent().

Let's say you depend on pitch to be something else besides 0:
while (pitch == 0) delay(5);

So that statement in setup() is gonna halt the 1st execution of draw() until variable pitch got its proper initialization inside the 1st execution of serialEvent().

Just an extra forum trip: Replace all /* w/ /** in order to avoid the whole green post glitch. L-)