We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
Index › Programming Questions & Help › Programs › Reading sequentially from a file
‹ Previous Topic | Next Topic ›
Page Index Toggle Pages: 1
Reading sequentially from a file (Read 3866 times)
alphachapmtl
Reading sequentially from a file
Feb 6th, 2008, 7:26pm  
How can I read sequentially from a huge file without loading the whole file into memory?
Without using statement like
 loadStrings("positions.txt");
as is done in the Learning/Topics/Load File examples?
Back to top
 
 
eattheword
Re: Reading sequentially from a file
Reply #1 - Feb 6th, 2008, 8:53pm  
I wanted to do the same thing recently, so I did this:

Code:

final int capacity = 750 ; // constant
int numberOfVans = 0,
heaviestVan = 0,
parcelWeight,
payload ;
String fileName = dataPath("weights.txt") ; // tells Processing to look in the Data folder

try
  {
  BufferedReader file = new BufferedReader (new FileReader (fileName)) ;
  parcelWeight = Integer.parseInt(file.readLine()) ; 
  while (parcelWeight > 0)
    {
    payload = 0 ;
    while ((payload + parcelWeight <= capacity) && (parcelWeight > 0))
	{
	payload += parcelWeight ;
	parcelWeight = Integer.parseInt(file.readLine()) ; 
	}
    numberOfVans ++ ;
    println ("Van " + numberOfVans + " has payload of " + payload + " kg.") ;
    if (payload > heaviestVan)
	heaviestVan = payload ;
    }
  println ("Number of vans used was " + numberOfVans + ".") ;
  println ("Heaviest van was " + heaviestVan + " kg." ) ;
  }
catch (Exception e)
  {
  println ("Error" + e) ;
  }


The program needs a file called weights.txt in the sketch's Data folder. Each line of the file should be a single integer and the last line of the file should have an integer value of 0 or less on it.
Back to top
 
 
alphachapmtl
Re: Reading sequentially from a file
Reply #2 - Feb 7th, 2008, 7:50am  
Thanks for the code, I could not have written this myself.

Here is a small writefile/readfile demo based on your code.
I'm using two simultaneous readers here cause I need for this to work in my own program.
I'm using a 3rd reader which will detect the end of file, without having a zero sentinel at the end.

Using String filename="name.txt" is OK to write but not to read, so I use String filename=dataPath("name.txt") as you did.

Code:
/**
 * write integers to a file
 * read and print the file using 2 readers
 */

void setup() {
  size(100,100);
  background(153);
  noLoop();
}//setup()

void draw() {
  String filename=dataPath("testfile01.txt");
  //
  //writefile
  println("--writefile--");
  PrintWriter output=createWriter(filename);
  int num;
  for (int i=1; i<5; i++) {
    num=i*i;
    output.println(num);
    println(num);
  }
  output.flush(); // Flush data to file
  output.close(); // Close file
  //
  //readfile
  println("--readfile--");
  try {
    BufferedReader reader1 = new BufferedReader (new FileReader (filename)) ; 
    int num1;
    for (int i=1; i<5; i++) {
      num1 = Integer.parseInt(reader1.readLine()) ;
      println("1."+num1);
      if (i==2) {
        BufferedReader reader2 = new BufferedReader (new FileReader (filename)) ; 
        int num2;
        for (int j=1; j<5; j++) {
          num2 = Integer.parseInt(reader2.readLine()) ;
          println("   2."+num2);
        }
      }
    }
  println("-----------");
    BufferedReader reader3 = new BufferedReader (new FileReader (filename)) ; 
    int num3;
    while (true) {
	String sline=reader3.readLine();
	if (sline==null) break;
	num3 = Integer.parseInt(sline) ;
	println("3."+num3);
    }
  }
  catch (Exception e) {
    println("ERROR >> "+e);
  }
  //
  println("--itsdone--");
  //
}//draw()

Here is the output:
Code:
--writefile--
1
4
9
16
--readfile--
1.1
1.4
   2.1
   2.4
   2.9
   2.16
1.9
1.16
-----------
3.1
3.4
3.9
3.16
--itsdone--

Back to top
 
 
eattheword
Re: Reading sequentially from a file
Reply #3 - Feb 7th, 2008, 11:43am  
if you're using for loops to read the file, that suggests you know how many items are in the file, which means you don't need to explicitly test for end of file. If the for loops were just to show the nested readers, then that's different.

Another way to test for end of file is to use the bufferedReader's ready() method as the controlling condition of a while loop. This tests whether there is a data item to be read. If so, it returns true, if not then we are pointing to the end of file so it returns false.
Back to top
« Last Edit: Feb 8th, 2008, 11:07am by eattheword »  
 
alphachapmtl
Re: Reading sequentially from a file
Reply #4 - Feb 7th, 2008, 5:18pm  
OK, ready() is good!
It simplify that part of the code.

Code:
BufferedReader reader3 = new BufferedReader (new FileReader (filename)) ; 
    int num3;
    while (reader3.ready()) {
	num3 = Integer.parseInt(reader3.readLine()) ;
	println("3."+num3);
    }
    reader3.close();

Is the close() necessary?
Back to top
 
 
eattheword
Re: Reading sequentially from a file
Reply #5 - Feb 8th, 2008, 11:05am  
If you want to read from the file again later in the program starting at the beginning then I guess it ought to be closed in order not to leave the reader pointer pointing to the end of the file
Back to top
 
 
Page Index Toggle Pages: 1
‹ Previous Topic | Next Topic ›