how to extract random chunks of consecutive data

edited July 2017 in Questions about Code

Hi, I'm new to Processing and have run into a need that I can't work me head around. I am loading data from a csv into a Processing sketch. I want the sketch to go through the data and select, at random, a 10 seconds (consecutive) extract of the data (which is equivalent to 10 rows).

What is the best way to go on about this? Do I set "i" to be a random number (constrained to length of rows) and then ask it to select the 10 rows after that? Or is there another way?

The following is the code I have:

import processing.pdf.*;

String filename = "210717 at 172333.csv";
String [] rawData;
int [] Time = new int[10];
//accelerometer data
int [] ADataX = new int[10];
int [] ADataY = new int[10];
int [] ADataZ = new int[10];
int AMinX, AMaxX;
int AMinY, AMaxY;
int AMinZ, AMaxZ;


int margin, graphHeight;
float xSpacer;
//ACCELEROMETER
PVector [] APositionsX = new PVector[10];
PVector [] APositionsY = new PVector[10];
PVector [] APositionsZ = new PVector[10];


void setup() {
  size(800,300);
  processData();
  beginRecord(PDF, "Worldlines(X,Y,Z)-01.pdf"); 
}

void draw () {
  background(20);
  drawGUI();
  endRecord();
}

  void drawGUI(){

    //ACCELEROMETER
  for (int i=0; i<APositionsX.length; i++){
 stroke(200,100);
 line(APositionsX[i].x, margin, APositionsX[i].x, height-margin); // lines of same location

 if (i>0) {
   stroke(255);
 line(APositionsX[i].x, APositionsX[i].y, APositionsX[i-1].x, APositionsX[i-1].y); 
  line(APositionsY[i].x, APositionsY[i].y, APositionsY[i-1].x, APositionsY[i-1].y);
  line(APositionsZ[i].x, APositionsZ[i].y, APositionsZ[i-1].x, APositionsZ[i-1].y);
  }
  }
}

void processData () {

  rawData = loadStrings(filename); //load Strings from file
  //parse through data
  for(int i=1; i<rawData.length; i++) { 
  String [] thisRow = split(rawData[i],",");
  //convert to integers per column
  ADataX [i-1] = int(thisRow[0]);
  ADataY [i-1] = int(thisRow[1]);
  ADataZ [i-1] = int(thisRow[2]);

  }

 //ACCELEROMETER
 AMinX = min(ADataX); // find min value for XData
 AMaxX = max(ADataX); //find max value for XData
 AMinY = min(ADataY); // find min value for XData
 AMaxY = max(ADataY); //find max value for XData
 AMinZ = min(ADataZ); // find min value for XData
 AMaxZ = max(ADataZ); //find max value for XData

 margin = 50;
 graphHeight = (height-margin)-margin; // set height of y axis
 xSpacer = (width-margin)/Time.length;

  //ACCELEROMETER
  for (int i=0; i<ADataX.length;i++){
    float adjADataX = map(ADataX[i], AMinX, AMaxX, 0, graphHeight); // map values within graph
    float yPos = height-margin - adjADataX;    
    float xPos = margin + (xSpacer *i);
   APositionsX[i]= new PVector (xPos, yPos);
}
 for (int i=0; i<ADataY.length;i++){
    float adjADataY = map(ADataY[i], AMinY, AMaxY, 0, graphHeight); // map values within graph
    float yPosY = height-margin - adjADataY;    
    float xPosY = margin + (xSpacer *i);
    APositionsY[i]= new PVector (xPosY, yPosY);
}

 for (int i=0; i<ADataZ.length;i++){
    float adjADataZ = map(ADataZ[i], AMinZ, AMaxZ, 0, graphHeight); // map values within graph
    float yPosZ = height-margin - adjADataZ;    
    float xPosZ = margin + (xSpacer *i);
    APositionsZ[i]= new PVector (xPosZ, yPosZ);
  }

}

Any help or suggestions would be greatly appreciated! Thanks!

Answers

  • Answer ✓

    Yes, that is the correct approach: Pick a random row number between 1 and ((number of rows) - 10), and then start with that row and take the next ten rows.

    You'll need to use random(low,high) and cast the result of that to an int().

  • Ctrl-t in the editor will indent your code nicely.

    Variable names traditionally start with a lower case letter so aData is preferred to AData etc

  • Hi, after over a day trying to figure it out, I still can't. This is as far as I got:

    Table table;
    TableRow row;
    int RowCount; //total rows
    int [] Sample = new int [10]; //10 rows of data
    int index; // random starting point
    int end;
    
    void setup(){
    table=loadTable("210717.csv", "header");
    RowCount = table.getRowCount();
    println(RowCount);
    
    index = int(random(RowCount-10));
    println(index);
    end = index+10;
    println(end);
    }
    
    void draw () {
    
    }
    

    How do I cast the Rows between 'index' and 'end' onto Sample []? Or perhaps I am going about it the wrong way? Again, any help appreciated!

  • Can you post the header of your table? Now, Sample will hold only 10 consecutive values from one column, right?

    Kf

  • The header, as in title row in the csv? If so that would be "ax,ay,zy,ag,xg,zg,time". The idea is that 'Sample' would hold the 10 rows of data that the initial sketch would mark on the graph. I'll look into the post now. Many thanks!

  • Hi, I've just retaken this up and I still can't figure out how to chuck the data into an int[] so that I can plot that data into a graph. This is as far as I got, the code will select a random line on the csv, and the 9 consecutive ones after that, and it will print the values of the first column out ("ax"). But as lines, not as integers.

    Table table;
    int RowCount; //total rows
    int index; // random starting point
    int end;
    int []ax = new int [9];
    
    void setup() {
      processData();
    }
    
    void draw () {
    }
    
    void processData() {
      table=loadTable("28-10-17.csv", "header");
      RowCount = table.getRowCount();
      println(RowCount);
    
      index = int(random(RowCount-9));
      println(index);
      end = index+9;
      println(end);
    
      for  (int i=index; i<end; i++) {
        TableRow row = table.getRow(i);
        int ax = row.getInt("ax");
        println(ax);
      }
    }
    

    Any suggestions on how to get that data into an int array? Any help or suggestions appreciated. Thanks!

  • Answer ✓

    Check this example.

    Kf

    final int ASIZE=9;
    Table table;
    int RowCount; //total rows
    int index; // random starting point
    
    void setup() {
      processData();
    }
    
    
    void processData() {
      table=loadTable("data.csv", "header");
      RowCount = table.getRowCount();
      println(RowCount);
    
      index = int(random(RowCount-ASIZE));
      println("Start="+index+" end="+end);
      float[] intCol = float(table.getStringColumn("ax"));
      println(subset(intCol,index,ASIZE));
    
    }
    

    Sample data from csv:

    id,ax,ay,az
    1,42.07354924,10,34.5
    2,45.46487134,10,37
    3,7.056000403,10,39.5
    4,-37.84012477,10,42
    5,-47.94621373,0,44.5
    6,-13.97077491,0,47
    7,32.84932994,0,49.5
    8,49.46791233,0,52
    9,20.60592426,0,54.5
    10,-27.20105554,10,57
    11,-49.99951033,10,59.5
    12,-26.8286459,10,62
    13,21.00835184,10,64.5
    14,49.53036778,10,67
    15,32.51439201,0,69.5

  • Hi, thanks for that! It works a treat for getting the "ax" values from the "ax" column, but when I try to get more values ("ay" and "az"), it prints hundreds of values? I've had to add an int "end" for it was giving me an error and have changed float to int on the values printed. I'm trying to understand what you have done so that I can apply it to the rest of the columns but I'm lost and I don't know what I'm doing wrong. This is as far as I got:

    final int ASIZE=10;
    Table table;
    int RowCount; //total rows
    int index; // random starting point
    int end;
    
    void setup() {
      processData();
    }
    
    
    void processData() {
      table=loadTable("28-10-17.csv", "header");
      RowCount = table.getRowCount();
      println(RowCount);
    
      index = int(random(RowCount-ASIZE));
      end = index+9;
      println("Start="+index+" end="+end);
      int[] axValues = int(table.getStringColumn("ax"));
      //int[] ayValues = int(table.getStringColumn("ay"));
      println(axValues);
      //println(ayValues);
    }
    
  • edited November 2017

    Ignore me! I worked it out (I think). Thank you so much! On to the next problem... getting the min and max values from the selected data!

  • Answer ✓

    In my example, ax is made of floats. If you use int() to cast the String array into an int array, you will get all the elements to be 0. That is the reason I use float instead.

    You can load the data into and InList or FloatList object and use their min/max functions.

    https://processing.org/reference/IntList.html

    Kf

  • Thanks, I'll give that a try now.

Sign In or Register to comment.