Handle a huge text file without error

TothProc

``Hi,

I have a huge text file with millions of point. (x,y,z). I need to transfer the data between two different software. But the format is not same as I need. (of course'' :-) ) I have a short code to change the file line by line. It works with approx.: 1 million point well. But if I want to test the final version, stucks somewhere. How I need to change the code to handle the huge file? The machine time is not important, but I need to be sure the code is running. (A chance of proper interrupt option to the code would be great. (ex.: close the file properly after the press a specific key) Thank you in advance.

` output = createWriter("result.txt"); }

void draw(){ String[] ln= loadStrings("liftgate test_tab.txt");
String[] content1 = new String[2]; String content2; int progress;

//for (int i=0; i < (ln.length-1) ;i++){ for (int i=1; i < (100) ;i++){ content1 =split(ln[i],","); content2=join(content1,"\t"); progress=(i); output.print(content2); output.print("\n"); println(progress);

} output.flush(); // Writes the remaining data to the file output.close(); // Finishes the file println("vege"); noLoop(); } `

Thanks!

koogs

loadStrings() will read in the entire file in one go. better to do it line by line.

(better not to use processing tbh)

koogs

sed "s/,/\t/g" file > output

TothProc

Koogs,

Thanks but what I can do with "sed" things? Please do not tell me I need to get a 100hr lessons to learn a new programming language. :-)

koogs

sed (stream editor) is a simple unix tool that has existed for decades and does exactly what you're trying to do here, which makes your processing code seem like overkill.

but it is possible. just read a line, transform, write a line.

and don't println() your progress every line - processing doesn't like too much console output. print every 1000th line, or every 10,000th.

TothProc

Thanks, Your comment regarding "sed" is good example to make me confused. I checked few pages about it. Looks promising. But where I can write that? Command prompt, which OS? .... blabalabla usual newbie staff.... Go back to processing. Your comment regarding println is something similar what I observed. So I gonna change this more rarely. Is there any way to read only one line from text with processing? (because the one line writing is in my code. Am I right?)

koogs

modulo operator

if (i % 10000 == 0) {
    println(i);
}

Chrisir

Is there any way to read only one line from text with processing?

Chrisir

in WINdows WordPad can read huge files (afaik 4 GB or so)

TothProc

I applied KOOGS solution, and wiped all the not important lines and voila'. the code runs perfectly within few seconds with 7 million lines.

koogs

Is there any way to read only one line from text with processing?

https://processing.org/reference/createReader_.html

(they had createWriter() in the code already so i figured that they already knew about createReader())