Array of Tables?

Aronnax

Hi, first time posting here and I have what I hope is a simple question.

I'm new to Processing, and just as new to Java, but have a little bit of programming experience. Most recently I've dabbled in Python which isn't really much of a help here.

I am working with data (xyz values) that I export as a csv file from a piece of test machinery we have here at work. The data comes out of it really sloppy, so I wrote on little sketch that cleans it all up really nice into a format that easy to work with. I wrote another sketch that reads the tabular data and plots it in 3d, that I can then rotate around and manipulate to analyze visually. I works really neat.

Right now I am working with with several csv files. My current example has 11, and each file can have tens of thousands of rows of coordinates, one even had a touch over 200k lines. My code currently has two nested for loops, one that reads a file from a "filenames array" (code I got off processing.org), and then another for loop that loops through the code, reading each row and plotting it, and when it's done it reads the next file.

The problem is that it's slow and clunky. My computer lags down and rotating the trace is choppy at best. I have a few possible solutions, that I think might speed it up and make the sketch run more efficient. One would to see if I can store all of the tables in an array. That way all the values would be in ram and it would be sending a command to read a file off of my hdd (and I have a sdd, don't know how much slower it would run if the files were on a network dirve). Another solution would be to append all of the files into one super long table. This would cause other problems that I would very much like to avoid. An array of tables would be much more elegant. A third solution which I will work on in conjunction with the other two is to figure out a way to shorten some of the longer files. I don't need 200k lines of data to plot an accurate trace, so I'll need to figure out some sort of algorithm that omits excessive, non-essential data.

In python I recall that you could create lists, and list of lists, and lists of lists of lists, nested deeper than I could even attempt to keep straight in my head. So in this case I would make one list that represented a row, x, y, z, then I'd make a list called "table", that had all the rows in it, then I'd make another list called tables that had all the tables in it. Just don't know how to do that in processing. Accessing the data would not be as simple as it is in processing, though.

Thanks...

GoToLoop

When requesting help, it really helps a lot to post a sample code of your attempts. Also a sample ".csv" file used there!
This way, we have a much clearer view of goals! Thus, better advises! /:)

koogs

reading each row and plotting it

does this imply you're reading all the files in the draw() loop? that's a terrible idea.

how are you reading the files? don't the normal methods return an array anyway?

you're short of details but i think you can use a global ArrayList, read all the files in setup() and add all the values to the ArrayList. then, in the draw(), draw everything.

post code, we'll fix it.

Aronnax

Yes- I am reading each file in the draw() loop. It is stupid, because I am already reading each file in the setup() to find some min/max values and to assign a few global variables.

I'm trying to post code but I can't seem to get it to work. I'm clicking the C button on the toolbar there but it doesn't seem to do anything. I've tried it both in FF and Chrome. Any help?

GoToLoop

You post 1st, highlight it, and then use the C button (or CTRL+K)! (~~)

Aronnax

ArrayList is where it's at. Thanks for that, took me a min to try and understand the syntax but my code is a million times better. Still if I reduce the amount of data it'd run even smoother, I'm sure...

And yeah, I don't know what my problem is. The C button and ctrl-K does nothing. I click it over and over again and nothing happens. Wait, is it just supposed ot indent the text four spaces?? If I highlight one line it surrounds the text in back-ticks, if I highlights a whole paragraph it just intents the first word four spaces.

Thanks for the help though.

(test)

Aronnax

Ok, now that I understand how it works here's how I load all of the csv files in my data folder. I copy/pasted most of that code from examples on processing.org.

ArrayList<Table> tables;

void setup() {
  tables = new ArrayList<Table>();
  java.io.File folder = new java.io.File(dataPath(""))
  java.io.FilenameFilter csvFilter = new java.io.FilenameFilter() {
    public boolean accept(File dir, String name) {
      return name.toLowerCase().endsWith(".csv");
    }
  };
  filenames = folder.list(csvFilter);

  for (int f = 0; f < filenames.length; f += 2) {
    Table table = loadTable(filenames[f],"header");
    tables.add(table);                              // builds my global array of tables

do stuff with the data here...

then in draw()-

void draw() {
  for (int t = 0; t < tables.size(); t++) {
  Table table = tables.get(t);

then I plot the data