Loading a huge XYZ Point Cloud file

Programming Questions

tai_pinc

Loading a huge XYZ Point Cloud file

in Programming Questions • 2 months ago

Hello,

I have a simple Point Cloud .txt file, structured like this: "X Y Z r g b"

I've written the sketch to load it into an array class.

Problem is the file is 68 MB, and as soon as I hit "run" nothing happens. The sketch appears at the dock (mac) but there's no respond.

(by the way - the number of points is 1,778,686 (stringLength). might this number be outside the limits of integer?)

Any tips on how to do it, without reducing the Point Cloud file's size?

Thanks

Tai

Point[] pcloud;
int pointNum = 0;
void setup() {
String[] lines = loadStrings("lifta txt.txt");
pcloud = new Point[lines.length];
for (int i = 0; i < lines.length; i++) {
String[] pieces = splitTokens(lines[i]);
if (pieces.length == 6) {
pcloud[pointNum] = new Point(pieces);
pointNum++;
}
}
for (int i = 0; i < pointNum; i++) {
println(i);
}
}
class Point {
float x;
float y;
float z;
int r;
int g;
int b;
public Point(String[] pieces) {
x = float(pieces[0]);
y = float(pieces[1]);
z = float(pieces[2]);
r = int(pieces[3]);
g = int(pieces[4]);
b = int(pieces[5]);
}
}

Replies(7)

kooogy

Re: Loading a huge XYZ Point Cloud file

2 months ago

it's loading 1.8M rows, you won't see anything until it's done.

actually, given that you have no draw() then you won't ever see anything.

other things:

color can be an int, using an int for r, g and b separately is a waste.

loadStrings will create 63MB of strings. creating 1.8M Points after that might be asking too much. maybe use BufferedReader to read a line at a time.

if i was doing this i'd use processing 1.5, glgraphics and vbos.

tai_pinc

Re: Re: Loading a huge XYZ Point Cloud file

2 months ago

Thank you very much, This has been very helpful.

Although I don't understand everything I've done here, things seems to be working.

I could not get something has reader.length in order to create the pcloud array, so I loaded the text file a second time, as a string. Is there a better solution for it?

I also see the points in the sketch, but I'll have to do some map() and translate, because it's very tiny right now.

import processing.opengl.*;
Point[] pcloud;
int pointNum = 0;
BufferedReader reader;
String line;
void setup() {
size(500, 500, OPENGL);
smooth();
reader = createReader("lifta txt.txt");
String[] lines = loadStrings("lifta txt.txt");
pcloud = new Point[lines.length];
}
void draw() {
try {
line = reader.readLine();
} catch (IOException e) {
line = null;
}
if (line == null) {
noLoop();
} else {
String[] pieces = splitTokens(line);
if (pieces.length == 6) {
pcloud[pointNum] = new Point(pieces);
println(pointNum);
pcloud[pointNum].display();
pointNum++;
}
}
}
class Point {
float x;
float y;
float z;
color rgb;
public Point(String[] pieces) {
x = float(pieces[0]);
y = float(pieces[1]);
z = float(pieces[2]);
rgb = color(int(pieces[3]), int(pieces[4]), int(pieces[5]));
}
void display() {
stroke(rgb);
point(x, y, z);
}
}

dimkir

Re: Loading a huge XYZ Point Cloud file

2 months ago

Loading 1.6M of text lines into memory for java is not such a big deal. Actually, it's a piece-a-cake for Java to do that.

Here's my test results (I generated text file with 1_600_000 rows) and tried loading it with processing.

As you can see loading file to memory only takes 1.1 seconds. and parsing and populating Points[] array takes around 5.9 seconds.

What freezes your sketch is not loading of points, but doing println() on the points. What happens is that console cannot process so many messages and starts lagging. Thus making your sketch look like it's frozen. Printing out lines to console may take up to 2 minutes, and by that time you definately think that your sketch is frozen.

Also kooogy is right: you're not outputiing anything to the screen (graphically). So you need to start plotting your points.

As to performance when you actually will be graphically plotting your points: i don't know how well processing (P3D rendering mode) will work with so many points.

However there's a logical solution: why on earth you need to plot exactly ALL points??? (in 3D) when you have 2-3k points on the screen it's already hard to see what's there. I see little point plotting all of them. If I were you, I would only plot every 10th or every 100th point (or maybe whilst running first versions of sketch I would only plot every 1000th point). Then you can start increasing the amount of point, until one of the following:

a) processing crashes

b) you decide that screen is too messy and you don't need any more points.

c) you have all points on the screen (unlikely to happen in real world applications)

final String C_BIG_FILE = "c:\\tmp\\tmp\\myfile.txt";
//final String C_BIG_FILE = "lifta txt.txt";
Point[] pcloud;
int pointNum = 0;
void setup() {
println("Starting sketch...");
int timeStart = millis();
String[] lines = loadStrings(C_BIG_FILE);
int timeLoadingStrings = millis() - timeStart;
timeStart = millis();
pcloud = new Point[lines.length];
int timeInitializingPointsArray = millis() - timeStart;
timeStart = millis();
for (int i = 0; i < lines.length; i++) {
String[] pieces = splitTokens(lines[i]);
if (pieces.length == 6) {
pcloud[pointNum] = new Point(pieces);
pointNum++;
}
}
int timePopulatingPointsArray = millis() - timeStart;
println("Time loading strings to memory: " + timeLoadingStrings);
println("Time initializing ponits array: " + timeInitializingPointsArray);
println("Time populaintg points array : " + timePopulatingPointsArray);
// println is the slowest and takes 2-3 minutes and freezes your sketch.
// thus we comment it out.
// for (int i = 0; i < pointNum; i++) {
// println(i);
// }
}
class Point {
float x;
float y;
float z;
int r;
int g;
int b;
public Point(String[] pieces) {
x = float(pieces[0]);
y = float(pieces[1]);
z = float(pieces[2]);
r = int(pieces[3]);
g = int(pieces[4]);
b = int(pieces[5]);
}
}

tai_pinc

Re: Re: Loading a huge XYZ Point Cloud file

2 months ago

Hi Dimkir,

Thanks, it's seems to be working well.

As for how many points to load and print, I'll just have to experiment - I may want to zoom on the model and have as many as possible.

Now I need to be able to relocate the points on the sketch and to scale them so I can see them (they stack on just a few pixels at the edge of the screen).

How would you advice me to do that?

Thanks a lot to you both

Tai

tai_pinc

Re: Loading a huge XYZ Point Cloud file

2 months ago

Ok, things seems to be working very nice now.

I've zoomed in with camera() and translate on the model, loaded every 100th point at the moment and it's beautiful.

/**

Now my problem is that it seems there's some zoom limitation. I've uses my mouse to control the camera and I see that as soon as I reach a distance, the model's points begin to disappear. This is common off course in every 3d software. I could I extend my zoom at the model?

UPDATE: Never mind. got it. Perspective()!

Point[] pcloud;
int pointNum = 0;
int pointDraw = 0;
void setup() {
size(500, 500, OPENGL);
String[] lines = loadStrings("lifta txt.txt");
pcloud = new Point[lines.length];
for (int i = 0; i < lines.length; i++) {
String[] pieces = splitTokens(lines[i]);
if (pieces.length == 6) {
pcloud[pointNum] = new Point(pieces);
pointNum++;
}
}
}
void draw() {
background(0);
camera(mouseX, height/2, 50, width/2, height/2, 0, 0, 1, 0);
translate(250, 250, 0);
for (int i = 0; i < pcloud.length/100; i++) {
pcloud[i*100].display();
}
/**
pcloud[pointDraw].display();
println(pointDraw);
pointDraw++;
*/
}
class Point {
float x;
float y;
float z;
color rgb;
public Point(String[] pieces) {
x = float(pieces[0]);
y = float(pieces[1]);
z = float(pieces[2]);
rgb = color(int(pieces[3]), int(pieces[4]), int(pieces[5]));
}
void display() {
stroke(rgb);
point(x, y, z);
}
}

dimkir

Re: Loading a huge XYZ Point Cloud file

2 months ago

The PeasyCam http://mrfeinberg.com/peasycam/#about library should be helpful to you to navigate around your points space with mouse.

tai_pinc

Re: Loading a huge XYZ Point Cloud file

2 months ago

thanks dimkir, already found that on another post. works well

Top Reply