How to create a 3D array of data set.

torekndsen

Hello, im currently working on a data visualization project, where I pull collected data from a CSV dataset using a 2D array. However I would like to expand my sketch to include and pull data from several datasets, instead of just one. I have approx. ten CSV files of different datasets, and I would like to include them all at ones in my sketch, but i'm not sure how to do this. I have an idea that I can use a 3D array so it will be: [Dataset_x][column][row] but I'm not sure how to do this. Can anyone help me with this? Cheers!

Below you guys can see how I'm storing my data into a 2D array.

String csv[];
String myData[][];

 csv = loadStrings("dataset_1.CSV");
  myData = new String[csv.length][4];
  for(int i = 1; i < csv.length; i++){
    myData[i] = csv[i].split(",");
  } 

GoToLoop

If your ".csv" files have the same columns, you're better off defining a class to represent a row of them.

torekndsen

Unfortunately the columns varies alot.. only the rows is the same length.

GoToLoop

Unfortunately the columns vary a lot...

Then I suppose each ".csv" file is about different subjects.
Like 1 is about "math" and the other is about "astronomy", aren't they?

IMO it makes no much sense to group together disparate ".csv" files together.
But if you insist here's an example on how to group them as 1 String[][][]: 8-X

// forum.Processing.org/two/discussion/15710/
// how-to-create-a-3d-array-of-data-set

// GoToLoop (2016-Mar-27)

final String[] DATA_SETS = { "Math", "Physics", "Religion" };
final String EXT = ".csv";
final String[][][] csvs = new String[DATA_SETS.length][][];

for (int i = 0; i < DATA_SETS.length; ++i) {
  String[] csv = loadStrings(DATA_SETS[i] + EXT);
  String[][] lines = new String[csv.length][];

  for (int j = 0; j < csv.length; ++j)  lines[j] = split(csv[j], ',');
  csvs[i] = lines;
}

for (int i = 0; i < DATA_SETS.length; ++i) {
  println(ENTER, DATA_SETS[i] + EXT, ENTER);
  String[][] lines = csvs[i];

  for (int j = 0; j < lines.length; ++j) {
    println("Line #" + j + ":");
    for (String[] row : lines)  println(row);
  }
}

exit();

torekndsen

Thx for the answer, I will check it out! I undertand it can be difficult to see the logic, when you dont know the project :) Each CSV file contains GPS data from individual persons bicycle routes. I would like to display them all at ones in processing. Thats also why the columns varies.. because no route is identical.

GoToLoop

Indeed there's no way to blindly customize any app when the data shape is unknown.

If all of those variadic columns represent just bicycle routes, I believe my example should work alright.

However, if there are any other type of data mixed up, you're better off creating a class to represent exactly all of the columns of each row as I had mentioned at my 1st reply. L-)

Eeyorelife

Wouldn't it be more sensible to have one array for each route? So each new GPS coordinate will be an addition to the array of that specific route. That way you can manipulate the data a lot more than if all your data is in one giant string... ?

You can easily know how long the route is compared to the others, you can choose to display one after the other with little problem. I imagine the GPS coordinates are recorder at certain time intervals, so by comparing the difference between two sets of GPS coordinates you can calculate the speed. And by comparing the first and the last, you get the average speed of the trip.

GoToLoop

Wouldn't it be more sensible to have one array for each route?
If all of those variadic columns represent just bicycle routes, ...

If my assumption is correct, each 3rd dimension "row" is a complete route of variadic GPS coords. :-?

quark

Each CSV file contains GPS data from individual persons bicycle routes

Does that mean each file is for one person and each row represents a route and each column represent a GPS position along the route?

Although it would be possible to use a 3D array to hold the data it is not ideal because it makes it extremely difficult to create flexible and extensible access algorithms to process the data.

I would suggest that you create an object orientated solution. My initial thoughts would be to have three classes to represent the rider (Person), the route (Route) and the GPS locations (GpsPos).

If you don't know how to create and use classes then in the long run it would be worth your while learning how to. :)

torekndsen

Each CSV file is data from one route. Each columns is a new measurement which is done every two second. Each row consist of 4 separated values: Time, Longitude, Latitude, SensorMeasurement.

I have working with object oriented solution before, so I could give it a try. Working with the 2D array just worked really well, so i hoped I just could expand, to 3D without making major changes.

GoToLoop

Each columns is a new measurement which is done every two second.
Each row consist of 4 separated values: Time, Longitude, Latitude, SensorMeasurement.

It should be the opposite: each row 1 reading w/ 4 columns. ^#(^

torekndsen

ofcouse! my mistake. (I switched them by mistake when writing this post)

GoToLoop

If that's so now, it's no longer true what you had said before:

Unfortunately the columns vary a lot.. only the rows are the same length.

I have to conclude that each row got exactly 4 columns. And the # of rows varies. [-(

torekndsen

Yes Exactly.. each row is a measurement, and each measurement(row) got 4 columns of values(Time, Longitude, Latitude, SensorMeasurement). So only the row length varies. Haha sry for the mess..

GoToLoop

Can you post a sample of the GPS ".csv" file then? That is, 3 lines of it.
Does it have a header w/ the 4 column descriptions?
If you had done it from the beginning the "mess" would be found out much sooner. ~:>

torekndsen

Here is a sample:

time,latitude,longitude,ppm 14810,55.6702,12.5793,197.8048553466 14812,55.6702,12.5792,397.1301574707 14814,55.6703,12.5791,286.4534606933 14816,55.6703,12.5790,197.8048553466 14818,55.6704,12.5789,197.8048553466 14820,55.6704,12.5788,197.8048553466 14822,55.6705,12.5787,397.1301574707 14824,55.6705,12.5786,397.1301574707 14826,55.6706,12.5785,531.95184326177

quark

So the GpsPos class would have 4 float attributes for time, longitude, latitude and measurement. Then the Route class would have at least 1 attribute, an ArrayList of GpsPos objects in chronological order. Use an ArrayList here because the number of GpsPos objects vary for each route. Finally your application could have an ArrayList or an array of Route objects.

GoToLoop

/**
 * GPS Biker Map Table (v1.0)
 * GoToLoop (2016-Mar-27)
 *
 * forum.Processing.org/two/discussion/15710/
 * how-to-create-a-3d-array-of-data-set
 */

import java.util.Map;
import java.util.Map.Entry;
final Map<String, Table> bikers = new HashMap<String, Table>();

static final int[] COL_TYPES = {
  Table.INT, Table.FLOAT, Table.FLOAT, Table.DOUBLE };

import java.io.FilenameFilter;
static final FilenameFilter CSV_FILTER = new FilenameFilter() {
  final boolean accept(final File dir, final String name) {
    return name.toLowerCase().endsWith(".csv");
  }
};

void setup() {
  final String[] csvs = getFilenamesFromSketchFolders(CSV_FILTER);

  for (final String filename : csvs) {
    final Table csv = loadTable(filename, "header");
    bikers.put(removeExtFromFilename(filename), csv);
    csv.setColumnTypes(COL_TYPES);
  }

  for (final Entry<String, Table> csv : bikers.entrySet()) {
    final String biker = csv.getKey();
    final Table  table = csv.getValue();

    println(ENTER, biker, ENTER);
    println(table.getColumnTitles());

    for (final TableRow gps : table.rows())  println(
      gps.getInt(0), gps.getFloat(1), gps.getFloat(2), gps.getDouble(3));
  }

  exit();
}

String[] getFilenamesFromSketchFolders(final FilenameFilter filter) {
  final String[] dataPaths = dataFile("").list(filter);
  final String[] sketchPaths = sketchFile("").list(filter);
  return dataPaths != null? concat(dataPaths, sketchPaths) : sketchPaths;
}

static final String removeExtFromFilename(final String filename) {
  return filename.substring(0, filename.lastIndexOf('.'));
}

GoToLoop

A more simplified version which replaces Map<String, Table> w/ Table[]: *-:)

/**
 * GPS Biker Map Table II (v1.0)
 * GoToLoop (2016-Mar-27)
 *
 * forum.Processing.org/two/discussion/15710/
 * how-to-create-a-3d-array-of-data-set
 */

import java.io.FilenameFilter;
static final FilenameFilter CSV_FILTER = new FilenameFilter() {
  final boolean accept(final File dir, final String name) {
    return name.toLowerCase().endsWith(".csv");
  }
};

static final int[] COL_TYPES = {
  Table.INT, Table.FLOAT, Table.FLOAT, Table.DOUBLE };

Table[] bikers;

void setup() {
  final String[] csvs = getFilenamesFromSketchFolders(CSV_FILTER);
  bikers = new Table[csvs.length];

  for (int i = 0; i < bikers.length; ++i)
    (bikers[i] = loadTable(csvs[i], "header")).setColumnTypes(COL_TYPES);

  for (int i = 0; i < bikers.length; ++i) {
    final Table table = bikers[i];

    println(ENTER, "#" + i + ":", ENTER);
    println(table.getColumnTitles());

    for (final TableRow gps : table.rows())  println(
      gps.getInt(0), gps.getFloat(1), gps.getFloat(2), gps.getDouble(3));
  }

  exit();
}

String[] getFilenamesFromSketchFolders(final FilenameFilter filter) {
  final String[] dataPaths = dataFile("").list(filter);
  final String[] sketchPaths = sketchFile("").list(filter);
  return dataPaths != null? concat(dataPaths, sketchPaths) : sketchPaths;
}

torekndsen

Thx a lot, I will give it a try!