We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
IndexProgramming Questions & HelpSyntax Questions › reading [very] large txt files
Page Index Toggle Pages: 1
reading [very] large txt files (Read 303 times)
reading [very] large txt files
Mar 8th, 2009, 5:15am
 
Does anyone have an idea of how I could read a 2.5GB txt file without running out of memory? Obviously, loadStrings won't do any good, so I would have to have some mechanism to stream the file or be able to read it in segments.
Thanks!
Re: reading [very] large txt files
Reply #1 - Mar 8th, 2009, 10:17am
 
Classical Java way, I let you adapt to Processing (probably usable as it):

Code:
  static String PATH = "E:/Some/Path/x.txt";

   BufferedReader reader = null;
   File f = new File(PATH);
   try
   {
     reader = new BufferedReader(new FileReader(f));
   }
   catch (FileNotFoundException e)
   {
     System.err.println("File not found: " + PATH);
     return;
   }
   try
   {
     String line = null;
     int lineNb = 0;

     line = reader.readLine();
     while (line != null)
     {
       // Here, do something with line
       System.out.println("Line " + lineNb + ": " + line);
       line = reader.readLine();
       lineNb++;
     }
   }
   catch (IOException e)
   {
     System.err.println("Error while reading file: " + e.getMessage());
   }
   finally
   {
     try { reader.close(); } catch (IOException e) {}
   }
Re: reading [very] large txt files
Reply #2 - Mar 8th, 2009, 5:53pm
 
Great. Thanks!
That pointed me to this page:
http://processing.org/reference/createReader_.html that has a similar implementation (though in Processing) to your code.
Cheers- wes
Re: reading [very] large txt files
Reply #3 - Mar 15th, 2009, 5:56pm
 
one additional way is to use memory-mapped IO ( C++ users wwill be familiar ) provided in java.nio package ( Java 1.5 )
Page Index Toggle Pages: 1