How to filter specific messages coming from arduino, and log them into an CSV?

luisandresgonzalez · January 2016

Hello, glad to meet you all, this is my first post. I'm trying to setup a CSV file logging, in order to record some reaction time responses coming from arduino. So far i've managed to tweak some codes i've found in the internets and managed to get them to work sepparately: the first one (arduino side) measures the reaction time and prints it in the serial ("Your reaction time was:"). A different, processing code receives some data (button pressed or not) and logs it into a CSV file. The problem is, that this code will write ANY line that comes from the serial, which leaves no room for me to send useful message to the user (like "you pressed the button too early!").

**tl;dr **

Is there a way that I can modify the processing code to filter just the seconds of the reaction time instead of every line that comes from the serial?

Below there are 3 blocks of code:

1) the arduino code that i want to implement,

2) the processing code,

3) and the original arduino code meant for the processing code.

Arduino code so far:

(more readable pastebin here http://pastebin.com/Ej6jt6RZ)


 /* REACTION TIME (with 2 leds) v1.1
  *  Luis Andrés Gonzalez
  * Reaction time original version from http://www.instructables.com/id/Arduino-Reaction-Time-Tester/?ALLSTEPS
  * Send data to processing via the Serial Port original from By Elaine Laguerta http://url/of/online/tutorial.cc
 */
 int switchPin = 6;  // pin where the button will be connected
 int ledPin1 = 2 ;   // LED that signals starting of the game
 int ledPin2 = 8 ;   // LED that lights to test the reaction time
 
 // declare some variables:
 boolean lastButton = LOW;
 boolean currentButton = LOW;
 boolean Started = false;
 boolean timer = false;
 long startTime;
 long endTime;
 int randomTime;
 long beginTime;
 float elapsedTime;
 
 
 void setup()
 {
   // Setup button and LEDs:
   pinMode(switchPin, INPUT);
   pinMode(ledPin1, OUTPUT);
   pinMode(ledPin2, OUTPUT);
 
   // Begin serial communication
   Serial.begin(9600);
 }
 boolean debounce(boolean last)
 {
   boolean current = digitalRead(switchPin);
   if(last != current)
   {
     delay(5);
     current = digitalRead(switchPin);
   }
   return current;
 }
 
 
 void loop()
 {
   // see if button pressed
   currentButton = debounce(lastButton);
   if(lastButton == LOW && currentButton == HIGH)
   {
     if(Started==false){
       Started=true;
       randomTime = random(4,10);
       randomTime = randomTime*1000;
       Blink();
       beginTime=millis(); 
       
     }
     else{
       if((millis()-beginTime)>=randomTime){
           Stop();
           Started=false; 
           timer=false;         
       }
       
       else{
         Started=false;
         timer=false;
         Serial.println("You pressed the button too soon !");
         for(int i=0; i<3; i++){
           Blink();
         }
       }
     }
   }
      
   lastButton = currentButton;
   
   if(Started == true && (millis()-beginTime)>=randomTime && timer==false){
     Serial.println("Start");
     timer=true;
     Start();
   }
 }
   
 
 
 
 
 
 void Start(){
   startTime = millis();
   digitalWrite(ledPin1, HIGH);
 }
 
 void Blink(){
   digitalWrite(ledPin2, HIGH);
   delay(100);
   digitalWrite(ledPin2, LOW);
   delay(100);
 }
 
 void Stop(){
   endTime = millis();
   elapsedTime = (endTime - startTime)+5;
   elapsedTime = elapsedTime/1000;
   Serial.print("Time Seconds: ");
   Serial.println(elapsedTime);
   digitalWrite(ledPin1, LOW);
    
 }

Processing code:

(http://pastebin.com/EjEcuqct)

 
 /* 
   Saving Values from Arduino to a .csv File Using Processing - Pseduocode
 
   This sketch provides a basic framework to read data from Arduino over the serial port and save it to .csv file on your computer.
   The .csv file will be saved in the same folder as your Processing sketch.
   This sketch takes advantage of Processing 2.0's built-in Table class.
   This sketch assumes that values read by Arduino are separated by commas, and each Arduino reading is separated by a newline character.
   Each reading will have it's own row and timestamp in the resulting csv file. This sketch will write a new file a set number of times. Each file will contain all records from the beginning of the sketch's run.  
   This sketch pseduo-code only. Comments will direct you to places where you should customize the code.
   This is a beginning level sketch.
 
   The hardware:
   * Sensors connected to Arduino input pins
   * Arduino connected to computer via USB cord
         
   The software:
   *Arduino programmer
   *Processing (download the Processing software here: https://www.processing.org/download/
   *Download the Software Serial library from here: http://arduino.cc/en/Reference/softwareSerial
 
   Created 12 November 2014
   By Elaine Laguerta
   http://url/of/online/tutorial.cc
 
 */
 
 import processing.serial.*;
 Serial myPort; //creates a software serial port on which you will listen to Arduino
 Table table; //table where we will read in and store values. You can name it something more creative!
 
 int numReadings = 5; //keeps track of how many readings you'd like to take before writing the file. 
 int readingCounter = 0; //counts each reading to compare to numReadings. 
  
 String fileName;
 void setup()
 {
   String portName = Serial.list()[1]; 
   //CAUTION: your Arduino port number is probably different! Mine happened to be 1. Use a "handshake" sketch to figure out and test which port number your Arduino is talking on. A "handshake" establishes that Arduino and Processing are listening/talking on the same port.
   //Here's a link to a basic handshake tutorial: https://processing.org/tutorials/overview/
   
   myPort = new Serial(this, portName, 9600); //set up your port to listen to the serial port
   table = new Table(); 
   table.addColumn("id"); //This column stores a unique identifier for each record. We will just count up from 0 - so your first reading will be ID 0, your second will be ID 1, etc. 
   
   //the following adds columns for time. You can also add milliseconds. See the Time/Date functions for Processing: https://www.processing.org/reference/ 
   table.addColumn("year");
   table.addColumn("month");
   table.addColumn("day");
   table.addColumn("hour");
   table.addColumn("minute");
   table.addColumn("second");
   
   //the following are dummy columns for each data value. Add as many columns as you have data values. Customize the names as needed. Make sure they are in the same order as the order that Arduino is sending them!
   table.addColumn("sensor1");  
 
 }
 
 void serialEvent(Serial myPort){
   String val = myPort.readStringUntil('\n'); //The newline separator separates each Arduino loop. We will parse the data by each newline separator. 
   if (val!= null) { //We have a reading! Record it.
     val = trim(val); //gets rid of any whitespace or Unicode nonbreakable space
     println(val); //Optional, useful for debugging. If you see this, you know data is being sent. Delete if  you like. 
     int sensorVals = int(val); //parses the packet from Arduino and places the valeus into the sensorVals array. I am assuming floats. Change the data type to match the datatype coming from Arduino. 
    
     TableRow newRow = table.addRow(); //add a row for this new reading
     newRow.setInt("id", table.lastRowIndex());//record a unique identifier (the row's index)
     
     //record time stamp
     newRow.setInt("year", year());
     newRow.setInt("month", month());
     newRow.setInt("day", day());
     newRow.setInt("hour", hour());
     newRow.setInt("minute", minute());
     newRow.setInt("second", second());
     
     //record sensor information. Customize the names so they match your sensor column names. 
     newRow.setInt("sensor1", sensorVals);
     
     
     readingCounter++; //optional, use if you'd like to write your file every numReadings reading cycles
     
     //saves the table as a csv in the same folder as the sketch every numReadings. 
     if (readingCounter % numReadings ==0)//The % is a modulus, a math operator that signifies remainder after division. The if statement checks if readingCounter is a multiple of numReadings (the remainder of readingCounter/numReadings is 0)
     {
       //saveTable(table, "data/new.csv");
       fileName = "data/" + str(year()) + str(month()) + str(day()) + str(table.lastRowIndex()) + ".csv"; //this filename is of the form year+month+day+readingCounter
       saveTable(table, fileName); //Woo! save it to your computer. It is ready for all your spreadsheet dreams. 
     }
    }
 }
 
 void draw()
 { 
    //visualize your sensor data in real time here! In the future we hope to add some cool and useful graphic displays that can be tuned to different ranges of values. 
 }

The original Arduino code meant for the processing code above:

(http://pastebin.com/qQF7YbYN)

/*
Sending Data to Processing via the Serial Port
This sketch provides a basic framework to send data from Arduino to Processing over a Serial Port. This is a beginning level sketch.

Hardware:
* Sensors connected to Arduino input pins
* Arduino connected to computer via USB cord

Software:
*Arduino programmer
*Processing (download the Processing software here: https://www.processing.org/download/

Additional Libraries:
*Read about the Software Serial library here: http://arduino.cc/en/Reference/softwareSerial

Created 12 November 2014
By Elaine Laguerta
http://url/of/online/tutorial.cc
*/

/*To avoid overloading the Arduino memory, and to encourage portability to smaller microprocessors, this sketch
does not timestamp or transform data. In this tutorial, timestamping data is handled on the processing side.

Whether you process data on the Arduino side is up to you. Given memory limitations of the Arduino, even a few computations and mapping of values can
max out the memory and fail. I recommend doing as little as possible on the Arduino board.*/

//#include SoftwareSerial.h

/*Declare your sensor pins as variables. I'm using Analog In pins 0 and 1. Change the names and numbers as needed
Pro tip: If you're pressed for memory, use #define to declare your sensor pins without using any memory. Just be careful that your pin name shows up NOWHERE ELSE in your sketch!
for more info, see: http://arduino.cc/en/Reference/Define
*/
int sensor1Pin = 6;


/*Create an array to store sensor values. I'm using floats. Floats use 4 bytes to represent numbers in exponential notation. Use int if you are representing whole numbers from -32,768 to 32,767.
For more info on the appropriate data type for your sensor values, check out the language reference on data type: http://arduino.cc/en/Reference/HomePage
Customize the array's size to be equal to your number of sensors.
*/
/*
 * Prueba de dejar esto afuera
 float sensorVals[] = {0,0,0}
*/
/*Pro tip: if you have a larger number of sensors, you can use a for loop to initialize your sensor value array. Here's sample code (assuming you have 6 sensor values):
float sensorVals[6];
int i;
for (i=0; i<6; i++)
{
sensorVals[i] = 0;
}
*/

void setup(){
Serial.begin(9600); //This line tells the Serial port to begin communicating at 9600 bauds
}

//
void loop(){
//read each sensor value. We are assuming analog values. Customize as nessary to read all of your sensor values into the array. Remember to replace "sensor1Pin" and "sensor2Pin" with your actual pin names from above!
int sensorVal = digitalRead(sensor1Pin);

/*If you are reading digital values, use digitalRead() instead. Here's an example:
sensorVal[0] = digitalRead(sensor1Pin);
*/

//print over the serial line to send to Processing. To work with the processisng sketch we provide, follow this easy convention: separate each sensor value with a comma, and separate each cycle of loop with a newline character.
//Remember to separate each sensor value with a comma. Print every value and comma using Serial.print(), except the last sensor value. For the last sensor value, use Serial.println()
//Serial.print(sensors[0]);
//Serial.print(",");
//Serial.print(sensors[1]);
//Serial.print(",");
Serial.println(sensorVal);
delay(1000);
}

Alex_Pr · January 2016

You should check the incoming string for keywords and if such keywords are found, print the string in the .csv

Something like if (incomingString.indexOf("keyword")!=-1) {print text} In this case, the "keyword" would be "Time Seconds:"

GoToLoop · January 2016

https://forum.Processing.org/two/discussion/8045/how-to-format-code-and-text

luisandresgonzalez · January 2016

Thanks for the answers so far.

What I've found out is that I can do a workaround by outputting the message via the processing console instead of the Arduino serial monitor.

Also as a workaround I've made up some "number codes" in the arduino code that signal specific events. These numbers (9090 or 1010) will never show up naturally in the timer, so I used those to signal processing the start of the game, or if the playerd responded to early. Also added some code to play sounds. I don't know if this is the most elegant or efficient way. What do you think?

Arduino:

/* REACTION TIME (with 2 leds) v1.0
 *  Luis Andrés Gonzalez
 * Reaction time original version from http://www.instructables.com/id/Arduino-Reaction-Time-Tester/?ALLSTEPS
 * Send data to processing via the Serial Port original from By Elaine Laguerta http://url/of/online/tutorial.cc
*/
int switchPin = 6;  // pin where the button will be connected
int ledPin1 = 2 ;   // LED that signals starting of the game
int ledPin2 = 8 ;   // LED that lights to test the reaction time

// declare some variables:
boolean lastButton = LOW;
boolean currentButton = LOW;
boolean Started = false;
boolean timer = false;
long startTime;
long endTime;
int randomTime;
long beginTime;
float elapsedTime;


void setup()
{
  // Setup button and LEDs:
  pinMode(switchPin, INPUT);
  pinMode(ledPin1, OUTPUT);
  pinMode(ledPin2, OUTPUT);

  // Begin serial communication
  Serial.begin(9600);
}
boolean debounce(boolean last)
{
  boolean current = digitalRead(switchPin);
  if(last != current)
  {
    delay(5);
    current = digitalRead(switchPin);
  }
  return current;
}


void loop()
{
  // see if button pressed
  currentButton = debounce(lastButton);
  if(lastButton == LOW && currentButton == HIGH)
  {
    if(Started==false){
      Started=true;
      randomTime = random(4,10);
      randomTime = randomTime*1000;
      Blink();
      Serial.println("9090"); // signal code for start sound
      beginTime=millis(); 
      
    }
    else{
      if((millis()-beginTime)>=randomTime){
          Stop();
          Started=false; 
          timer=false;         
      }
      
      else{
        Started=false;
        timer=false;
        Serial.println("1010"); // signal code for early response
        for(int i=0; i<3; i++){
          Blink();
        }
      }
    }
  }
     
  lastButton = currentButton;
  
  if(Started == true && (millis()-beginTime)>=randomTime && timer==false){
    //Serial.println("Start");
    timer=true;
    Start();
  }
}
  





void Start(){
  startTime = millis();
  digitalWrite(ledPin1, HIGH);
}

void Blink(){
  digitalWrite(ledPin2, HIGH);
  delay(100);
  digitalWrite(ledPin2, LOW);
  delay(100);
}

void Stop(){
  endTime = millis();
  elapsedTime = (endTime - startTime)+5;
  elapsedTime = elapsedTime/1000;
  //Serial.print("Time Seconds: ");
  Serial.println(elapsedTime);
  digitalWrite(ledPin1, LOW);
   
}

Processing:

/** 
 REACTION TIME GAME v 1.0
 Adapted by code made by Elaine Laguerta  
 by Luis Andrés Gonzalez, January 2016
 
 Reads reaction time from Arduino over the serial port and save it to .csv file on your computer.
 The .csv file will be saved in the same folder as your Processing sketch.
 This sketch assumes that values read in seconds by Arduino are separated by a newline character.
 Special code numbers are specified to signal the game start, success and failure of accomplishing the task. 
 Each reading will have it's own row and timestamp in the resulting csv file. This sketch will write a new file each day.   
 
 The hardware:
 * A pushbutton connected to Arduino input pins
 * Arduino connected to computer via USB cord
 
 The software:
 *Arduino IDE
 *Processing (download the Processing software here: https://www.processing.org/download/
 *The corresponding arduino code
 
 */

import processing.serial.*;
import ddf.minim.*;
import ddf.minim.ugens.*;

Minim minim;
AudioOutput out;

Serial myPort; //creates a software serial port on which you will listen to Arduino
Table table; //table where we will read in and store values. You can name it something more creative!

int numReadings = 1; //keeps track of how many readings you'd like to take before writing the file. 
int readingCounter = 0; //counts each reading to compare to numReadings. 

// Serial code numbers for specific events
int startCode = 9090; // code for game start
int earlyCode = 1010; // code for early button press

String fileName;

void setup() {
  String portName = Serial.list()[1]; 
  //CAUTION: your Arduino port number is probably different! Mine happened to be 1. Use a "handshake" sketch to figure out and test which port number your Arduino is talking on. A "handshake" establishes that Arduino and Processing are listening/talking on the same port.
  //Here's a link to a basic handshake tutorial: https://processing.org/tutorials/overview/

  // SOUND setup:
  minim = new Minim(this);
  out = minim.getLineOut();
  out.setTempo( 80 );
  out.pauseNotes();
  // end SOUND setup.

  myPort = new Serial(this, portName, 9600); //set up your port to listen to the serial port

  table = new Table(); 
  table.addColumn("id"); //This column stores a unique identifier for each record. We will just count up from 0 - so your first reading will be ID 0, your second will be ID 1, etc. 

  //the following adds columns for time. You can also add milliseconds. See the Time/Date functions for Processing: https://www.processing.org/reference/ 
  table.addColumn("year");
  table.addColumn("month");
  table.addColumn("day");
  table.addColumn("hour");
  table.addColumn("minute");
  table.addColumn("second");

  //the following are dummy columns for each data value. Add as many columns as you have data values. Customize the names as needed. Make sure they are in the same order as the order that Arduino is sending them!
  table.addColumn("sensor1");
}

void serialEvent(Serial myPort) {

  String val = myPort.readStringUntil('\n'); //The newline separator separates each Arduino loop. We will parse the data by each newline separator. 

  if (val!= null) { //We have a reading! Record it.
    val = trim(val); //gets rid of any whitespace or Unicode nonbreakable space

    if (float(val) == startCode) {
      out.playNote( 0.0, 0.4, "C3" );
      out.playNote( 0.2, 0.2, "C4" );
      out.resumeNotes();
    } else if (float(val) == earlyCode) {
      println("You pressed the button too early!");
      out.playNote( 0, 2, "A1");
      out.resumeNotes();
    } else {
      print("Your reaction time was: ");
      println(val); 
      out.playNote( 0, 0.2, "C4");
      out.playNote( 0.2, 0.2, "C4");
      out.playNote( 0.4, 0.2, "C4");
      out.resumeNotes();
    } 

    float sensorVal = float(val); //parses the packet from Arduino and places the valeus into the sensorVals array. I am assuming floats. Change the data type to match the datatype coming from Arduino. 

    if (sensorVal != startCode) { // avoids start code to get recorded
      TableRow newRow = table.addRow(); //add a row for this new reading
      newRow.setInt("id", table.lastRowIndex());//record a unique identifier (the row's index)

      //record time stamp
      newRow.setInt("year", year());
      newRow.setInt("month", month());
      newRow.setInt("day", day());
      newRow.setInt("hour", hour());
      newRow.setInt("minute", minute());
      newRow.setInt("second", second());

      //record sensor information. Customize the names so they match your sensor column names. 
      if (float(val) == earlyCode) {
        newRow.setString("sensor1", "early");
      } else {
        newRow.setFloat("sensor1", sensorVal);
      }

      readingCounter++; //optional, use if you'd like to write your file every numReadings reading cycles
    }
    //saves the table as a csv in the same folder as the sketch every numReadings. 
    if (readingCounter % numReadings ==0){//The % is a modulus, a math operator that signifies remainder after division. The if statement checks if readingCounter is a multiple of numReadings (the remainder of readingCounter/numReadings is 0)
      fileName = "data/" + str(year()) + str(month()) + str(day()) /*+" - " + str(table.lastRowIndex())*/ + ".csv"; //this filename is of the form year+month+day+readingCounter
      saveTable(table, fileName); //Woo! save it to your computer. It is ready for all your spreadsheet dreams.
    }
  }
}

void draw()
{ 
  //visualize your sensor data in real time here! In the future we hope to add some cool and useful graphic displays that can be tuned to different ranges of values.
}

Alex_Pr · January 2016

If I were you I'd use symbols as well! There are 30 available symbols to arrange in any order (of any length) to signal whatever message you want to processing. This way, you cannot mistake the signals for data (not integers) and at the same time do not waste number sequences which one can never know when they might be useful.

I've worked with arduino to processing communications before and that's how I did it. You can even combine symbols and numbers which offers even more possibilities in larger programs or more data points.

But that's just me ;)

luisandresgonzalez · January 2016

It's not a bad idea, @Alex_Pr. But since I'm a total noob (I'm a researcher in animal behavior, not a professional programmer) and I cannot trace what happens to different data types, I used just numbers. I think if I add symbols to the mix I'm gonna mess it big time. Since in future iterations the timer will go up to a max value 60 seconds and then stop, that freed up any number greater than that for me to use.

Howdy, Stranger!

Categories

In this Discussion

How to filter specific messages coming from arduino, and log them into an CSV?

Best Answer

Answers