how to extract random chunks of consecutive data

mfuster

Hi, I'm new to Processing and have run into a need that I can't work me head around. I am loading data from a csv into a Processing sketch. I want the sketch to go through the data and select, at random, a 10 seconds (consecutive) extract of the data (which is equivalent to 10 rows).

What is the best way to go on about this? Do I set "i" to be a random number (constrained to length of rows) and then ask it to select the 10 rows after that? Or is there another way?

The following is the code I have:

import processing.pdf.*;

String filename = "210717 at 172333.csv";
String [] rawData;
int [] Time = new int[10];
//accelerometer data
int [] ADataX = new int[10];
int [] ADataY = new int[10];
int [] ADataZ = new int[10];
int AMinX, AMaxX;
int AMinY, AMaxY;
int AMinZ, AMaxZ;


int margin, graphHeight;
float xSpacer;
//ACCELEROMETER
PVector [] APositionsX = new PVector[10];
PVector [] APositionsY = new PVector[10];
PVector [] APositionsZ = new PVector[10];


void setup() {
  size(800,300);
  processData();
  beginRecord(PDF, "Worldlines(X,Y,Z)-01.pdf"); 
}

void draw () {
  background(20);
  drawGUI();
  endRecord();
}

  void drawGUI(){

    //ACCELEROMETER
  for (int i=0; i<APositionsX.length; i++){
 stroke(200,100);
 line(APositionsX[i].x, margin, APositionsX[i].x, height-margin); // lines of same location

 if (i>0) {
   stroke(255);
 line(APositionsX[i].x, APositionsX[i].y, APositionsX[i-1].x, APositionsX[i-1].y); 
  line(APositionsY[i].x, APositionsY[i].y, APositionsY[i-1].x, APositionsY[i-1].y);
  line(APositionsZ[i].x, APositionsZ[i].y, APositionsZ[i-1].x, APositionsZ[i-1].y);
  }
  }
}

void processData () {

  rawData = loadStrings(filename); //load Strings from file
  //parse through data
  for(int i=1; i<rawData.length; i++) { 
  String [] thisRow = split(rawData[i],",");
  //convert to integers per column
  ADataX [i-1] = int(thisRow[0]);
  ADataY [i-1] = int(thisRow[1]);
  ADataZ [i-1] = int(thisRow[2]);

  }

 //ACCELEROMETER
 AMinX = min(ADataX); // find min value for XData
 AMaxX = max(ADataX); //find max value for XData
 AMinY = min(ADataY); // find min value for XData
 AMaxY = max(ADataY); //find max value for XData
 AMinZ = min(ADataZ); // find min value for XData
 AMaxZ = max(ADataZ); //find max value for XData

 margin = 50;
 graphHeight = (height-margin)-margin; // set height of y axis
 xSpacer = (width-margin)/Time.length;

  //ACCELEROMETER
  for (int i=0; i<ADataX.length;i++){
    float adjADataX = map(ADataX[i], AMinX, AMaxX, 0, graphHeight); // map values within graph
    float yPos = height-margin - adjADataX;    
    float xPos = margin + (xSpacer *i);
   APositionsX[i]= new PVector (xPos, yPos);
}
 for (int i=0; i<ADataY.length;i++){
    float adjADataY = map(ADataY[i], AMinY, AMaxY, 0, graphHeight); // map values within graph
    float yPosY = height-margin - adjADataY;    
    float xPosY = margin + (xSpacer *i);
    APositionsY[i]= new PVector (xPosY, yPosY);
}

 for (int i=0; i<ADataZ.length;i++){
    float adjADataZ = map(ADataZ[i], AMinZ, AMaxZ, 0, graphHeight); // map values within graph
    float yPosZ = height-margin - adjADataZ;    
    float xPosZ = margin + (xSpacer *i);
    APositionsZ[i]= new PVector (xPosZ, yPosZ);
  }

}

Any help or suggestions would be greatly appreciated! Thanks!

TfGuy44

Yes, that is the correct approach: Pick a random row number between 1 and ((number of rows) - 10), and then start with that row and take the next ten rows.

You'll need to use random(low,high) and cast the result of that to an int().

koogs

Ctrl-t in the editor will indent your code nicely.

Variable names traditionally start with a lower case letter so aData is preferred to AData etc

mfuster

Hi, after over a day trying to figure it out, I still can't. This is as far as I got:

Table table;
TableRow row;
int RowCount; //total rows
int [] Sample = new int [10]; //10 rows of data
int index; // random starting point
int end;

void setup(){
table=loadTable("210717.csv", "header");
RowCount = table.getRowCount();
println(RowCount);

index = int(random(RowCount-10));
println(index);
end = index+10;
println(end);
}

void draw () {

}

How do I cast the Rows between 'index' and 'end' onto Sample []? Or perhaps I am going about it the wrong way? Again, any help appreciated!

kfrajer

Can you post the header of your table? Now, Sample will hold only 10 consecutive values from one column, right?

Kf

kfrajer

This post could be used as a reference: https://forum.processing.org/two/discussion/22832/how-to-create-a-grid-of-colours-using-data-from-a-csv-file/p1

Kf

mfuster

The header, as in title row in the csv? If so that would be "ax,ay,zy,ag,xg,zg,time". The idea is that 'Sample' would hold the 10 rows of data that the initial sketch would mark on the graph. I'll look into the post now. Many thanks!

mfuster

Hi, I've just retaken this up and I still can't figure out how to chuck the data into an int[] so that I can plot that data into a graph. This is as far as I got, the code will select a random line on the csv, and the 9 consecutive ones after that, and it will print the values of the first column out ("ax"). But as lines, not as integers.

Table table;
int RowCount; //total rows
int index; // random starting point
int end;
int []ax = new int [9];

void setup() {
  processData();
}

void draw () {
}

void processData() {
  table=loadTable("28-10-17.csv", "header");
  RowCount = table.getRowCount();
  println(RowCount);

  index = int(random(RowCount-9));
  println(index);
  end = index+9;
  println(end);

  for  (int i=index; i<end; i++) {
    TableRow row = table.getRow(i);
    int ax = row.getInt("ax");
    println(ax);
  }
}

Any suggestions on how to get that data into an int array? Any help or suggestions appreciated. Thanks!

kfrajer

Check this example.

Kf

final int ASIZE=9;
Table table;
int RowCount; //total rows
int index; // random starting point

void setup() {
  processData();
}


void processData() {
  table=loadTable("data.csv", "header");
  RowCount = table.getRowCount();
  println(RowCount);

  index = int(random(RowCount-ASIZE));
  println("Start="+index+" end="+end);
  float[] intCol = float(table.getStringColumn("ax"));
  println(subset(intCol,index,ASIZE));

}

Sample data from csv:

id,ax,ay,az
1,42.07354924,10,34.5
2,45.46487134,10,37
3,7.056000403,10,39.5
4,-37.84012477,10,42
5,-47.94621373,0,44.5
6,-13.97077491,0,47
7,32.84932994,0,49.5
8,49.46791233,0,52
9,20.60592426,0,54.5
10,-27.20105554,10,57
11,-49.99951033,10,59.5
12,-26.8286459,10,62
13,21.00835184,10,64.5
14,49.53036778,10,67
15,32.51439201,0,69.5

mfuster

Hi, thanks for that! It works a treat for getting the "ax" values from the "ax" column, but when I try to get more values ("ay" and "az"), it prints hundreds of values? I've had to add an int "end" for it was giving me an error and have changed float to int on the values printed. I'm trying to understand what you have done so that I can apply it to the rest of the columns but I'm lost and I don't know what I'm doing wrong. This is as far as I got:

final int ASIZE=10;
Table table;
int RowCount; //total rows
int index; // random starting point
int end;

void setup() {
  processData();
}


void processData() {
  table=loadTable("28-10-17.csv", "header");
  RowCount = table.getRowCount();
  println(RowCount);

  index = int(random(RowCount-ASIZE));
  end = index+9;
  println("Start="+index+" end="+end);
  int[] axValues = int(table.getStringColumn("ax"));
  //int[] ayValues = int(table.getStringColumn("ay"));
  println(axValues);
  //println(ayValues);
}

mfuster

Ignore me! I worked it out (I think). Thank you so much! On to the next problem... getting the min and max values from the selected data!

kfrajer

In my example, ax is made of floats. If you use int() to cast the String array into an int array, you will get all the elements to be 0. That is the reason I use float instead.

You can load the data into and InList or FloatList object and use their min/max functions.

https://processing.org/reference/IntList.html

Kf

mfuster

Thanks, I'll give that a try now.