Reading large text file with Processing - Processing Forum

Loading...

Processing Forum

Programming Questions

e45yyy

Reading large text file with Processing

in Programming Questions • 2 years ago

Hi.

I'm trying to create complex data visualisation program using Processing.

Everything is working fine, when my source data file is small, but when I try to read file larger than 800MB, I always run out of memory. It doesn't matter if i allocate more memory to Processing in preferences.

Hope someone can help me, because this is killing my project...

Thanks,

e45y

1

Replies(6)

calsign

calsign

Re: Reading large text file with Processing

2 years ago

Where do you load the file?
If you perform the load inside a loop, that would explain your memory error- and the larger the file, the quicker it would happen.

e45yyy

Re: Reading large text file with Processing

2 years ago

I read the file like this. I call openFile() inside Processing setup() function at the begining.

String[] lines;
boolean openFile(String filePath) {
println("Reading file...");
lines = loadStrings(filePath);
if (lines.length > 0) {
return true;
}
else {
return false;
}
}

And then later on, I use for loop to go through each line and parse it.

Regards,

e45y

Jesse Scott

Jesse Sco..

Re: Reading large text file with Processing

2 years ago

But is this being called within draw() - ie. multiple times ??

FYI, there have been a few threads about this recently... do a quick search and you might find some answers...

this one for example: https://forum.processing.org/topic/memory-management-10-8-2011

e45yyy

Re: Reading large text file with Processing

2 years ago

It's called inside setup() function, because i want to load data only at the begining.

Sorry, I posted this in a hurry and didn't look for any previous threads. Will look for other threads.

PhiLho

PhiLho

Re: Reading large text file with Processing

2 years ago

Note that you can replace 6 lines by one:
return lines.length > 0;

And if you don't need to have all the lines in memory at once, you can read the file line per line, for example.

e45yyy

Re: Reading large text file with Processing

2 years ago

Thank you phi.lho for your advice.

Don't know why I didn't think of this, because i really do not need all lines in the memory all the time.

Regards,

e45y

Top Reply