I need to parse data into multiple files based on the leading digits of a value, and naming the files on the leading digits. This is just the output declaration, but I go through this each step of the way. I have to create these files and write data directly to them as I am processing ~90 GB of data each time, so arrays are not going to work.

Currently the code looks like this:

output0 = createWriter(outPath + "0.txt");
output1 = createWriter(outPath + "1.txt");
output2 = createWriter(outPath + "2.txt");
output3 = createWriter(outPath + "3.txt");
output4 = createWriter(outPath + "4.txt");
output5 = createWriter(outPath + "5.txt");
output6 = createWriter(outPath + "6.txt");
output7 = createWriter(outPath + "7.txt");
output8 = createWriter(outPath + "8.txt");
output9 = createWriter(outPath + "9.txt");

I do this for the declaration, then the creatWriter, println, flush and close, so there are literally hundreds of lines of code that could be condensed via a loop.

I'd like to do this, but haven't gotten it to successfully create the output files, then write to them, etc.:

for (int i = 0; i < 10; i++) {
      output(i) = createWriter(outPath + i + ".txt");
}

Any help appreciated!

How can I get the filename of the current sketch? I want to write it in the header of my log file when it begins processing.

In the book "Visualizing Data" it mentions that BZIP (.bz) files can be read directly by the Processing API methods loadstrings(), createReader(), etc.

Does this also apply to .bz2 files? I've not been able to crack it, but this would be tremendously helpful. Any insights much appreciated.

To pre-process a huge data file for SQL loading, I have to read a text file and parse each line. Everything works great on a test file, but the real file is over a GB with millions of lines and generates OutOfMemoryError. I assume the array is too large...

Is there an alternative method of reading one line at a time instead of loading them all into an array?

I've searched and can't find documentation or forum topics around formatting output using print() or println().

I need to emulate various existing reports, so need to add commas, slashes, etc. to integers, dates, etc. I assume there are arguments to send but can't find mention of them anywhere.

This may be a Windows Vista issue, but I can't seem to figure out how to access a folder of files outside of the Sketch data folder.

This works:

1.   inputFolder= "data";
     File dataFolder = new File(sketchPath, inputFolder);
     String[] fileList = dataFolder.list( );
   

But how do I refer to another folder other than "data"? Such as:
"/Users/myname/Desktop/folderOfFiles/"

I have tried all the various combinations of forward/backward/double/single slashes.

I'm reading in log files to determine whether there are missing files in the set. They are consistently named, with servername at the front and hour at the end of the filenames. I need to check whether the current file is from the same server as the previous, and set "same" or "different" booleans to control the following actions. However, I it is resisting such determination and never recognizing the servers as "same". DRIVING ME NUTS. I'm sure there's a simple mistake in the code somewhere. If anyone can give it a few minutes, I'd appreciate your insights.

In comparefiles() I make the comparison, and act accordingly. The following data produces the following output:

Files being processed:

servername1_log-2010-0401-01
servername1_log-2010-0401-02
servername1_log-2010-0401-03
servername2_log-2010-0401-00
servername2_log-2010-0401-01

Output:

Different!
1 |servername1_log|0401| missed | first hour(s) 0 through 0
Different!
1 |servername1_log|0401| missed | last hour(s) 2 through 23
Different!
1 |servername1_log|0401| missed | last hour(s) 3 through 23
Different!
1 |servername1_log|0401| missed | last hour(s) 4 through 23
Different!
1 |servername2_log|0401| missed | last hour(s) 1 through 23
2 |servername2_log|0401| missed | last hour(s) 2 through 23

0401 had |5| files



My code:

1. // globals
   String filename = "";
   long fileCount = 0;
   String missed = "";
   
   String theServer;
   String theDay;
   String theHour;
   String oldServer = "none";
   
   int newHour;
   int oldHour = -1;
   
   boolean sameServer;
   
   String station = "";
   
   PrintWriter output;
   
   void setup() {
     String dateTimeStamp = "_" + year() + nf(month(),2) + nf(day(),2) + "_" + nf(hour(),2) + "_" + nf(minute(),2);
     output = createWriter("FileCheck" + dateTimeStamp + ".txt");
    
     File dataFolder = new File(sketchPath, "data");
     String[] fileList = dataFolder.list( );
   
     if (fileList != null) {
       for (int i = 0; i < fileList.length; i++) {
         filename = fileList[i];
         fileCount++;     
        
         oldServer = theServer;
         parsefilename(filename);
         comparefiles(theServer,oldServer,theDay,theHour,oldHour);
   
         oldHour = int(theHour);
       }
       if (oldHour != 23) {                                          // old server missed last hour(s)
         missed = " last hour(s) " + (oldHour+1) + " through 23";
       }
       station = "2 |";
       if (missed != "") {
         filestats(station,theServer,theDay,missed);   
       }
       stats(theDay,fileCount);
     } else {
     println("Could not access files.");
     }
     output.flush();
     output.close();
   }
   
   void comparefiles(String theServer,String oldServer,String theDay,String theHour, int oldHour) {
     String thisServer = "";
   
     if (theServer == oldServer) {
       println("Same!");
       sameServer = true;
     } else {
       println("Different!");
       sameServer = false;
     }
   
     if (!sameServer) {
       if ((oldHour != 23) && (oldHour != -1)) {            // old server missed last hour(s)
         thisServer = oldServer;
         missed = " last hour(s) " + (oldHour+1) + " through 23";
       } else {
       if (int(theHour) != 0) {                             // new server missed first hour(s)
         thisServer = theServer;
         missed = " first hour(s) 0 through " + (int(theHour)-1);
       }
       }
     } else {
       if (int(theHour) != (oldHour + 1)) {                 // same server missed mid hour(s)
         thisServer = theServer;
         missed = " mid hour(s) " + (oldHour+1) + " through " + (int(theHour)-1);
       }
     }
     if (thisServer != "") {
         station = "1 |";
         filestats(station,thisServer,theDay,missed);
     }
   }
   
   void filestats(String station,String thisServer,String theDay,String missed) {
       if (missed != "") {
         println(station + thisServer +"|"+ theDay +"| missed |"+ missed);
           missed = "";
       }
   }
   
   void stats(String theDay,long fileCount) {
       println();
       println(theDay +" had |"+ fileCount + "| files");
   }
   
   void parsefilename(String filename) {
     int p1 = filename.indexOf("-");
     int p2 = filename.indexOf("0401");
     theServer = filename.substring(0,p1);
     theDay = filename.substring(p2,p2 + 4);
     theHour = filename.substring(p2 + 5,p2 + 7);
   }
   

I have more than one Sketch open (on Windows Vista), and one of them is executing a job which is updating the console via println(). If I want do anything in the second Sketch, the println updates now show in the second console. Is there a way to isolate each of these? I'd like to be able to work on more than one Sketch at a time.

I haven't been able to find clear directions for sorting 2D arrays. I have to process data that is time based, but dependent on userIDs. What I am doing is putting userID, datetimestamp and a value in a 2D array. There are thousands of these rows:

userdata[i][1] = userID
userdata[i][2] = datetimestamp
userdata[i][3] = value

I need to sort on userID & datetimestamp in order to determine how to handle the associated values.

Any help appreciated.
roy