loadStrings() alternative for HUGE files

Programming Questions

roymond

loadStrings() alternative for HUGE files

in Programming Questions • 2 years ago

To pre-process a huge data file for SQL loading, I have to read a text file and parse each line. Everything works great on a test file, but the real file is over a GB with millions of lines and generates OutOfMemoryError. I assume the array is too large...

Is there an alternative method of reading one line at a time instead of loading them all into an array?

Replies(4)

PhiLho

Re: loadStrings() alternative for HUGE files

2 years ago

Quick answer: you have to use Java I/O to do that. Perhaps using createReader().
Otherwise, you have to look at BufferedReader and similar classes.

jonobr1

Re: loadStrings() alternative for HUGE files

2 years ago

I did this recently and found that BufferedReader works well. The csv was over 700mb — making it over 700million lines of text. Here's a sample pde:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.io.IOException;
void setup() {
File file = new File("test.txt");
StringBuffer contents = new StringBuffer();
BufferedReader reader = null;
try
{
reader = new BufferedReader(new FileReader(file));
String text = null;
// repeat until all lines is read
while ((text = reader.readLine()) != null)
{
contents.append(text)
.append(System.getProperty(
"line.separator"));
}
} catch (FileNotFoundException e)
{
e.printStackTrace();
} catch (IOException e)
{
e.printStackTrace();
} finally
{
try
{
if (reader != null)
{
reader.close();
}
} catch (IOException e)
{
e.printStackTrace();
}
}
// show file contents here
System.out.println(contents.toString());
}

Remember you're going to want to place "text.txt" in the data folder of your sketch.

—
http://jonobr1.com/

PhiLho

Re: Re: loadStrings() alternative for HUGE files

2 years ago

Good code. As minor improvements, I would put the line separator in a variable (before the loop) and I would use a more modern StringBuilder (supposedly slightly faster) even if it makes no real difference in speed (which is bound by disk I/O).

roymond

Re: loadStrings() alternative for HUGE files

2 years ago

Thanks, guys. For the short term I used TextPipe Standard (gotta love 30 day trials!) to create 13 files of 100,000 lines each, and processed them in sequence, but I will need to incorporate something like BufferedReader in the future as this is a regular challenge. I'll give it a try.

Much appreciated!

Top Reply