We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
Index › Programming Questions & Help › Programs › Load huge textfiles (> 200Mb)
‹ Previous Topic | Next Topic ›
Page Index Toggle Pages: 1
Load huge textfiles (> 200Mb) (Read 580 times)
extrapixel
Load huge textfiles (> 200Mb)
Sep 4th, 2006, 5:56pm  
Hi there,

I'm trying to read a big textfile into processing line by line. It's tab-separated strings, one record by line.

I keep getting the error message "Exception in thread "Thread-2" java.lang.OutOfmemoryError: Java heap space" when trying to use loadStrings() with that file.

The file is obviously too large, but is there a way to parse such files in processing? Is it possible to somehow stream the file, so it does not have to be in memory as a whole?

I set the memory settings in the preferences.txt and plist.info to 1024. I'm on a Apple G4 PPC-PowerBook, 2Gb of Ram. Running OSX 10.4.7 and Processing 0115.

Regards & Thanks,
Patrick
Back to top
 
http://www.extrapixel.ch
WWW  
cello
Re: Load huge textfiles (> 200Mb)
Reply #1 - Sep 4th, 2006, 7:08pm  
You could use FileReader and BufferedReader to stream the file in lines.

Check out this site for information on using Java's IO classes:
http://java.sun.com/docs/books/tutorial/essential/io/

Marcello
Back to top
 
WWW marcello3d  
extrapixel
Re: Load huge textfiles (> 200Mb)
Reply #2 - Sep 6th, 2006, 6:08pm  
Hi Marcello,

Thank you very much for this hint. I got it working, it's now streaming and parsing basically. However, I still have speed issues...
I go through the file in the draw loop. Do you think this is a good idea? I'm drawing a graph-like thing based on the data in the file. And because of the file size I want to see (and probably interact with) the graphs growing. (dont want to wait hours to see if my code works Wink)
Any better ideas?

ps: this is all about analyzing the AOL search data collection... just for fun!
Back to top
 
http://www.extrapixel.ch
WWW  
fry
Re: Load huge textfiles (> 200Mb)
Reply #3 - Sep 9th, 2006, 2:42pm  
you'd probably be better off deciding something that you're looking for in the file, and filtering based on that. munging through hundreds of megabytes inside draw() is gonna make for a slow or boring way of looking at things.

i'd recommend picking something that you want to search for, and having a separate function that handles running that search and loading resutls into memory. then in draw(), you render out those results and make things move.

that data set is enormous, so you've gotta think about ways to make it smaller so that you can play with it.
Back to top
 
WWW  
Page Index Toggle Pages: 1
‹ Previous Topic | Next Topic ›