http://processing.org/reference/loadStrings_.html or http://processing.org/reference/loadBytes_.html

first we'd need to know what data you have there, how is it saved? how big are your files?

depending on what you want to do with the graphic generated from the data the approach would be different. do you just want to save an image / pdf or is it going to be interactive?

normally i'd read the data into an array or object (depends on how complicated it is) at setup() then work with it in draw().

F

Thank you, that was most helpful. I noticed that you had to write your own percentile and average functions, not to mention an entire boxplot program.  I typically do this type of work in R, where the following code would produce a nice boxplot:

> xx<-rnorm(50)
> boxplot(xx)

If I wanted to get the average or the first quartile I could use the following:

> mean(xx)
> quantile(xx,0.25)

I'm wondering if there has been any work (or if there is even any interest) in producing standardized functions (like in R) that would facilitate manipulating and displaying data.

Getting data such as that example in is fairly straightforwards:

Code:

String[] data;
float[] Slength,SWidth,PLength,PWidth;
String[] Species;
void setup()
{
  //normal setup stuff..
  data=loadStrings("data.txt");
  
  //start from line 1 instead of line 0, since that's not data.
  for(int line=1;line<data.length;line++)
  {
    //separate out the parts...
    String[] fields=split(data[line], ',');
    if(fields.length!=5)
    {
	println("Bad data, line "+line);
    }
    else
    {
	//add each bit to the relevant array.
	SLength=append(SLength,float(fields[0]));
	SWidth=append(SWidth,float(fields[1]));
	PLength=append(PLength,float(fields[2]));
	PWidth=append(PWidth,float(fields[3]));
	//need to strip the "s off.
	String newSpecies=fields[4].substring(1,fields[4].length-2);
	Species=append(Species,newSpecies);
    }
  }
} 

And there, you have all the info into arrays, such that (for example) item 4 in each array corresponds to the 4th entry in your data file.

As a pre-emptive stumbling-block removal, if you come to want to find all "virginica" you need to use <string>.equals() instead of ==, e.g.:

Code:

for(int i=0;i<Species.length;i++)
{
  // *NOT* if(Species[i]=="virginica")
  if(Species[i].equals("virginica"))
  {
    //do something with all i-th entries in the arrays
  }
} 

As for built in functions, no processing doesn't have anything like R, since that's a dedicated graphing language, and processing is an all round tool.

processing will load them all into memory. That much data might be too much for processing to do any sort of "interactive" work with.

You might be better doing the statistics and analysis in something else, and then writing the results out to a file, (e..g bx position/extents/error bars) and then just showing that in processing, rather than trying to load the raw data, and do all the analysis in processing.

A couple of comments on this.

First, even if you do have the 100+ MB of free memory you'd need to do this, if you're using JohnG's code unaltered you're going to have a bit of a wait, because each of those "append()" calls actually allocates and fills a brand new array (which I'm sure he is quite aware of; for small data sets, his method is definitely the one to use).  By the time you reach the end of your dataset, you're allocating a new 1,000,000 element array for each additional data point, so I don't know if your loading will ever finish!

For large data sets like yours, you'd want to preallocate the array - assuming that there is one title row and the rest are valid data, you'd just say, for instance, float[] Slength = new float[data.length-1]; and then fill in the array as you traverse the String array (just replace each append() call with an assignment to the appropriate index).  If you don't know that each entry is valid you need to do something more complex, making a first pass through the array to calculate the number of valid entries, allocating the float[] array, and then making a second pass to actually populate that array.  Same two-pass method applies if you need to filter data.

On another note, though, if you don't actually need to use all of the data at any one time (this would _not_ be the case if you potentially needed random access to the array; in most cases you only need to process data in sequential order, and this is what I'm talking about), you can use the Java I/O functionality directly to do things like read the next character or line of data.  I'll refer you to http://java.sun.com/docs/books/tutorial/essential/io/ for more details on how to do this, as unfortunately Java's methods are not quite as simple to use as Processing's.  The main problem with this approach is that it necessarily binds the parsing phase to the calculation phase.  But this is usually preferable to hauling around thousands of megabytes of mostly irrelevant variables, so make what you will of it...