ArrayList of PVectors Setup and Display

schehe

Hi, I was wondering if anyone here can help me getting my head around PVectors. I am creating a visualization of points on Earth. For this I am creating an arrayList of PVectors, which get their data from a csv (Longitude, Latitute). My problem is the csv has over 88 000 lines and slows down the program when drawn in void draw() Therefore I wanted to initialize the PVectors in void createLocationPoints()) and only display them in draw with another funciton... For instance with giving the points a stroke weight. Can someone explain how I can display the function LocationVectors.add(new PVector(x, y, z)); in this example? I was thinking of something similar like Daniel Schiffmans simple particle example from the documentation but I can't quite get my head around it. Please be gentle, I am quite new to processing. Thanks a lot!

import peasy.*;

PeasyCam cam;

//Radius Earth
float radius = 6371000/50000; //global scale

//Tables
Table dataTable;
Table locationTable;

//Data lists
FloatList AffordabilityIndex;
FloatList LongituteAI;
FloatList LatituteAI;
FloatList AltituteAI;

//Locations Lists
FloatList settlementPoint;
FloatList Longitute;
FloatList Latitute;
FloatList Altitute;

float x;
float y;
float z;

PVector LocationVector;
//PShape points;

ArrayList<PVector> LocationVectors;

void setup() {
  size(800, 800, P3D);
  smooth();
  dataTable = loadTable("dataTable.csv", "header");
  locationTable = loadTable("locationTable.csv", "header");

  LocationVectors = new ArrayList<PVector>();

  cam = new PeasyCam(this, 4000);
  cam.setMinimumDistance(250); //0
  cam.setMaximumDistance(500); //8000

  ////LOCATIONS
  ////create location lists
  Longitute = new FloatList();
  Latitute = new FloatList();
  Altitute = new FloatList();
  //Read and store CSV
  for (int i = 0; i < locationTable.getRowCount(); i++) {
      TableRow rowLocationTable = locationTable.getRow(i);  
      Longitute.append(rowLocationTable.getFloat("Longitute")); 
      Latitute.append(rowLocationTable.getFloat("Latitute"));
      Altitute.append(rowLocationTable.getFloat("Altitute"));
    }


  //DATA
  //create data lists
  AffordabilityIndex = new FloatList();
  LongituteAI = new FloatList();
  LatituteAI = new FloatList();
  //Read and store CSV
  for (int i = 0; i < dataTable.getRowCount(); i++) {
    TableRow rowDataTable = dataTable.getRow(i);
    AffordabilityIndex.append(rowDataTable.getFloat("Affordability Index")*1);
    LongituteAI.append(rowDataTable.getFloat("Longitute"));
    LatituteAI.append(rowDataTable.getFloat("Latitute"));
    //println(Longitute, Latitute, AffordabilityIndex);
  }

  createLocationPoints();

}

void draw() {

  background(255);
  shape(points);
  displayLocationPoints();

  //DRAW DATA

  AltituteAI = AffordabilityIndex;

  strokeWeight(4);
  stroke(0);
  beginShape(POINTS);

  for (int i = 0; i < LongituteAI.size(); i++) {
    //Convert to Cartesian Coordinates
    //float a = pow( 1, 3);  // Sets 'a' to 1*1*1 = 1
    float f = 0; // flatening
    float lambda = atan(pow((1 - f), 2) * tan(radians(LatituteAI.get(i))));    // lambda

    float y = radius * cos(lambda) * cos(radians(LongituteAI.get(i))) + AltituteAI.get(i) * cos(radians(LatituteAI.get(i))) * cos(radians(LongituteAI.get(i))); //swaped x&y
    float x = radius * cos(lambda) * sin(radians(LongituteAI.get(i))) + AltituteAI.get(i) * cos(radians(LatituteAI.get(i))) * sin(radians(LongituteAI.get(i)));
    float z = radius * sin(lambda) + AltituteAI.get(i) * sin(radians(LatituteAI.get(i)));

    vertex(x, y, z);
  }
  endShape();

  noStroke();
  sphere(radius*0.98);

}

void createLocationPoints(){
    //DRAW ALL LOCATIONS
  //strokeWeight(1);
  //stroke(0);
  //beginShape(POINTS);
   //points = createShape();
   //points.beginShape();

  for (int i = 0; i < Longitute.size(); i++) {
    //Convert to Cartesian Coordinates
    //float a = pow( 1, 3);  // Sets 'a' to 1*1*1 = 1
    float f = 0; // flatening
    float lambda = atan(pow((1 - f), 2) * tan(radians(Latitute.get(i))));    // lambda

    float y = radius * cos(lambda) * cos(radians(Longitute.get(i))) + Altitute.get(i) * cos(radians(Latitute.get(i))) * cos(radians(Longitute.get(i))); //swaped x&y
    float x = radius * cos(lambda) * sin(radians(Longitute.get(i))) + Altitute.get(i) * cos(radians(Latitute.get(i))) * sin(radians(Longitute.get(i)));
    float z = radius * sin(lambda) + Altitute.get(i) * sin(radians(Latitute.get(i)));

    vertex(x, y, z);

    LocationVector = new PVector(x, y, z);
    LocationVectors.add(new PVector(x, y, z));

  }

  //endShape();
  //points.endShape();
  println(LocationVectors);
} 

void displayLocationPoints(){
  strokeWeight(10);
  stroke(0);

}

koogs

you can do the "flattening" in setup() after the reading and store the results, which will save you a bunch of calculations in draw()

and if the data doesn't change then you can create a PShape holding it all and just display that in draw().

koogs

FloatList AffordabilityIndex;
FloatList LongituteAI;
FloatList LatituteAI;
FloatList AltituteAI;

java naming standards have all variables starting with a lower case letter, so latitudeAI is preferred. (classes start with a Capital)

koogs

also

FloatList settlementPoint;
FloatList Longitute;
FloatList Latitute;
FloatList Altitute;

if all the values are connected then make a class and have an ArrayList of that class.

there are FAQs for this: https://forum.processing.org/two/discussion/8081/from-several-arrays-to-classes

schehe

thanks a lot. I should really work on my naming system. The draw() data doesn't change but it has a camera rotating around, so the display of the point array should be updating.

I tried it with a class and a PShape, but it doesn't give me any visual. Can createPShape() even do a collection of points/vertices?

Here's the code with a class and cleaned up (the previous had two data sets simultaneously). I am calling the drawPoints() function of the class in setup() and the display function in draw()

    float radius = 6371000/50000;

    Table dataTable;
    Table locationTable;
    //Data lists
    FloatList AffordabilityIndex;
    FloatList LongituteAI;
    FloatList LatituteAI;
    FloatList AltituteAI;
    //Locations Lists
    FloatList settlementPoint;
    FloatList Longitute;
    FloatList Latitute;
    FloatList Altitute;

    PointsLines AllData;
    PointsLines AllLocations;

    void setup() {
      size(800, 800, P3D);

      dataTable = loadTable("dataTable.csv", "header");
      locationTable = loadTable("locationTable.csv", "header");

      //TRANSFORM
      translate(width/2, height/2);

      //LOCATIONS
      //create location lists
      Longitute = new FloatList();
      Latitute = new FloatList();
      Altitute = new FloatList();
      //Read and store CSV
      for (int i = 0; i < locationTable.getRowCount(); i++) {
          TableRow rowLocationTable = locationTable.getRow(i);  
          Longitute.append(rowLocationTable.getFloat("Longitute")); 
          Latitute.append(rowLocationTable.getFloat("Latitute"));
          Altitute.append(rowLocationTable.getFloat("Altitute"));
    }

      //setup constructor with FloatList()
      AllData = new PointsLines(LongituteAI, LatituteAI, AltituteAI, radius);
      AllData.drawPoints(LongituteAI, LatituteAI, AltituteAI, radius);

      }


      void draw(){
          background(255);
    AllData.displayPoints();
    //println(LongituteAI);

      }


      // CLASS POINT ARRAY AND LINE ARRAY

       class PointsLines{

        PShape DataPoints;

        //data
        FloatList Longitute;
        FloatList Latitute;
        FloatList Altitute;
        float radius;   

        //constructor

        PointsLines(FloatList _Longitute, FloatList _Latitute, FloatList _Altitute, float _radius){
        Longitute = _Longitute; 
        Latitute = _Latitute;
        Altitute = _Altitute;
        radius = _radius;  

        }

      // DRAW POINT FUNCTION
      void drawPoints(FloatList Longitute, FloatList Latitute, FloatList Altitude, float radius) {


        for (int i = 0; i < Longitute.size(); i++){
        //float a = pow( 1, 3);  // Sets 'a' to 1*1*1 = 1
        float f = 0; // fLatituteAItening
        float lambda = atan(pow((1 - f), 2) * tan(Latitute.get(i)));    // lambda

        float y = radius * cos(lambda) * cos(Longitute.get(i)) + Altitude.get(i) * cos(Latitute.get(i)) * cos(Longitute.get(i)); //swap x&y
        float x = radius * cos(lambda) * sin(Longitute.get(i)) + Altitude.get(i) * cos(Latitute.get(i)) * sin(Longitute.get(i));
        float z = radius * sin(lambda) + Altitude.get(i) * sin(Latitute.get(i));
        DataPoints = createShape(POINT, x, y, z);
        DataPoints.beginShape();
        DataPoints.strokeWeight(4);
        DataPoints.stroke(0);

      //beginShape(POINTS);
        vertex(x, y, z);
        //endShape();

        DataPoints.endShape(CLOSE);

        }
       }

       void displayPoints(){
          shape(DataPoints);
       }
     }

koogs

DataPoints should be outside the for loop that starts on line 82, only the vertex call should be inside. Plus it needs to be DataPoints.vertex()

(This is untested - I don't have the data)

The draw() data doesn't change but it has a camera rotating around, so the display of the point array should be updating.

This is fine. With a pshape you define the data once. It is uploaded to the graphics card, once. Any camera changes are done by the graphics card in hardware and are lightning fast. The thing you avoid is calculations using the CPU and the data transfer.

schehe

Thanks for the ideas. I tried some different versions in the last days and to be honest I am starting to feel a bit hopeless about it.

Try1: Class: With PShape modified as you suggested. I think what the problem is that createShape() doesn't really do point clouds. The documentation says it is for geometries such as rectangles or custom shapes made from points etc. – https://processing.org/reference/createShape_.html

Try2: A Function called in setup() with PShape Group created from an array of PShapes:

it looks like this:

    //CREATE PSHAPE AND ADD TO ARRAY

    for (int i = 0; i < Longitute.size(); i = i+1) {

            //Spherical Abstraction
        float r = radius+Altitute.get(i);
        float y = r * cos(radians(Latitute.get(i))) * cos(radians(Longitute.get(i))); //swaped x&y
        float x = r * cos(radians(Latitute.get(i))) * sin(radians(Longitute.get(i)));
        float z = r * sin(radians(Latitute.get(i)));

        dataPoints.beginShape(POINT);
        dataPoints.vertex(x, y, z);
        dataPoints.endShape();

        dataPointsArray[i] = dataPoints;
      }

        //GROUP DATA POINTS
      for (int i = 0; i < Longitute.size(); i = i++) {
          dataPointsGroup.addChild(dataPointsArray[i]);
      }

It seems to work, the array has the right number of indices but with the grouping everything went blank. I suppose PShape group is not made for 88000 points.

Try3: an ArrayList of PVectors. It is actually the same as drawing all points in draw() because it still has to loop through all the points in each frame to display them.

The only idea that I still have is using the file ParticleSystePShape from the official examples and try to recreate it for my case. After all this program is creating a bizillion rectangles every frame. But to be honest I am lacking the brains to understand the example. I guess I would need help to even understand it.

koogs

does this work?

// point cloud
// acd 2016

import peasy.*;
PeasyCam cam;

static final int COUNT = 10000;
PShape s;

void setup() {
  size(600, 600, P3D);
  cam = new PeasyCam(this, 500);

  s = createShape();
  s.beginShape(POINTS);
  s.stroke(255, 0, 0);
  for (int i = 0 ; i < COUNT ; i ++) {
    // random points
    s.vertex(random(-100, 100), random(-100, 100), random(-100, 100));
  }
  s.endShape();
}

void draw() {
  background(0);
  shape(s);
}

koogs

(my 2009-vintage video card (Nvidia 130M) does 450,000 with very little cpu but .5M kills it)

koogs

but this, from 2011 and using Processing 1.5.1 and the GlGraphics library says i got 90,000 5 pointed stars (so 900,000 lines) before it started failing.

https://www.flickr.com/photos/31962137@N00/5966878196/in/photolist-a6gP1h

schehe

Thanks a lot! That's a really good example. Apparently my old Macbook Air can only go up to 100 000 after this it gives me weird glitches – 300 000 kill it.

Your example actually helped me figuring out what I did wrong. I needed to create an array of xyz coordinates and, when createShape() is called, only loop through the array to create the points. Apparently it didn't like the fact that I did the spherical coordinate conversion in the same loop. It is working now and the PShape method actually saves a lot of CPU in my case... totally worth it :) Thanks so much for your help!

Here's a screenshot of the WIP since you could never run it without the data: Screen Shot 2016-05-29 at 19.30.26

That's the code that worked:

in setup():

//Arrays
Float [] rL = new Float[Longitute.size()];
Float [] yL = new Float[Longitute.size()];
Float [] xL = new Float[Longitute.size()];
Float [] zL = new Float[Longitute.size()];

//GLOBAL LOCATIONS COORDINATES
for (int i = 0; i < Longitute.size(); i = i+1) {

  //Spherical Abstraction
  rL[i] = radius+Altitute.get(i);
  yL[i] = rL[i] * cos(radians(Latitute.get(i))) * cos(radians(Longitute.get(i))); //swaped x&y
  xL[i] = rL[i] * cos(radians(Latitute.get(i))) * sin(radians(Longitute.get(i)));
  zL[i] = rL[i] * sin(radians(Latitute.get(i)));
}

//CREATE SHAPE
s = createShape();
s.beginShape(POINTS);
for (int i = 0 ; i < xL.length ; i ++) {
  s.vertex(xL[i], yL[i], zL[i]);
}
s.endShape();

in draw():

  s.disableStyle();
  strokeWeight(1);
  stroke(0);
  shape(s);

koogs

Apparently it didn't like the fact that I did the spherical coordinate conversion in the same loop.

that shouldn't matter, really.

also, rL[i] isn't used outside the loop by the look of things so you don't need an array to store this value.

in short, this should work (untested)

s = createShape();
s.beginShape(POINTS);
for (int i = 0 ; i < Longitute.size() ; i++) {
  //Spherical Abstraction
  float r = radius + Altitute.get(i);
  float x = r * cos(radians(Latitute.get(i))) * sin(radians(Longitute.get(i)));
  float y = r * cos(radians(Latitute.get(i))) * cos(radians(Longitute.get(i)));
  float z = r * sin(radians(Latitute.get(i)));
  s.vertex(x, y, z);
}
s.endShape();     

GoToLoop

Some more extra advises: :-B

• Do not use Float reference datatype! Always favor primitive float as much as you can.
• In Java conventions, variables start w/ lowercase letters. The famous "camelCase". ~O)
• Moreover, it's better to choose plural or collective names for container variables.
• Thus, rather than Longitute, Latitute, Altitute; go w/ longitudes, latitudes, altitudes. :D
• General performance tip: use variables to cache repeatable results from expressions & functions.
• Instead of radians(longitudes.get(i)) over & over, why not float lon = DEG_TO_RAD * longitudes.get(i);?
• Those FloatList containers coulda been easily replaced by regular float[] arrays btW.
• Rather than longitudes = new FloatList();, better w/ longitudes = new float[locationTable.getRowCount()]; ;)
• To sum it all up, here's some template for it: :-bd

final Table locationTable = loadTable("locationTable.csv", "header");
final int len = locationTable.getRowCount();

final float[] longitudes = new float[len];
final float[] latitudes  = new float[len];
final float[] altitudes  = new float[len];

final float rad = random(100);

final PShape s = createShape();
s.beginShape(POINTS);

for (int i = 0; i < len; ++i) {
  final float lon = DEG_TO_RAD * longitudes[i];
  final float lat = DEG_TO_RAD * latitudes[i];
  final float r = rad + altitudes[i], radCosLat = r*cos(lat);

  s.vertex(radCosLat*sin(lon), radCosLat*cos(lon), r*sin(lat));
}

s.endShape();

GoToLoop

An even better idea: longitudes[], latitudes[], altitudes[] form 1 trio.
Why not get all those 3 together as 1 PVector[] as x, y & z fields? *-:)

// forum.Processing.org/two/discussion/16830/
// arraylist-of-pvectors-setup-and-display

// GoToLoop (2016-May-29)

final float RAD = 6371000/50000.;

final Table locationTable = loadTable("locationTable.csv", "header");
final PVector[] coords = new PVector[locationTable.getRowCount()];

int idx = 0;
for (final TableRow tr : locationTable.rows()) {
  final float lon = tr.getFloat("Longitute");
  final float lat = tr.getFloat("Latitute");
  final float alt = tr.getFloat("Altitute");

  coords[idx++] = new PVector(lon, lat, alt);
}

final PShape s = createShape();
s.beginShape(POINTS);

for (final PVector v : coords) {
  final float lon = DEG_TO_RAD * v.x, lat = DEG_TO_RAD * v.y;
  final float r = RAD + v.z, radCosLat = r*cos(lat);

  s.vertex(radCosLat*sin(lon), radCosLat*cos(lon), r*sin(lat));
}

s.endShape();

GoToLoop

And another alternative version where longitudes, latitudes & altitudes are already stored in PVector[] pre-converted to radians() plus pre-calculated w/ cos() + sin(): B-)

// forum.Processing.org/two/discussion/16830/
// arraylist-of-pvectors-setup-and-display

// GoToLoop (2016-May-29)

final float RAD = 6371000/50000.;

final Table locationTable = loadTable("locationTable.csv", "header");
final PVector[] coords = new PVector[locationTable.getRowCount()];

int idx = 0;
for (final TableRow tr : locationTable.rows()) {
  final float lon = DEG_TO_RAD * tr.getFloat("Longitute");
  final float lat = DEG_TO_RAD * tr.getFloat("Latitute");
  final float alt = RAD + tr.getFloat("Altitute");
  final float altCosLat = alt*cos(lat);

  coords[idx++] = new PVector(altCosLat*sin(lon), altCosLat*cos(lon), alt*sin(lat));
}

final PShape s = createShape();
s.beginShape(POINTS);
for (final PVector v : coords)  s.vertex(v.x, v.y, v.z);
s.endShape();

GoToLoop

If the Table or the PVector[] containers are not needed elsewhere within your sketch, why keep them around, given you'd only want the generated PShape? :-?

In this latest example, there's a customized createShape() which returns a PShape according to the Table file + some other attributes: :>

// forum.Processing.org/two/discussion/16830/
// arraylist-of-pvectors-setup-and-display

// GoToLoop (2016-May-29)

final float RAD = 6371000/50000.;
PShape s;

void setup() {
  size(800, 800, P3D);
  smooth(4);
  noLoop();

  s = createShape("locationTable.csv", RAD, 5.5, #FF0000);
}

void draw() {
  background(-1);
  shape(s);
}

PShape createShape(String tableFileName, float r, float big, color c) {
  final PShape s = createShape();
  s.beginShape(POINTS);

  s.strokeWeight(big);
  s.stroke(c);
  s.translate(width>>1, height>>1);

  for (final TableRow tr : loadTable(tableFileName, "header").rows()) {
    final float lon = DEG_TO_RAD * tr.getFloat("Longitute");
    final float lat = DEG_TO_RAD * tr.getFloat("Latitute");
    final float alt = r + tr.getFloat("Altitute");
    final float altCosLat = alt*cos(lat);

    s.vertex(altCosLat*sin(lon), altCosLat*cos(lon), alt*sin(lat));
  }

  s.endShape();
  return s;
}

koogs

8-X

schehe

:D Thanks GoToLoop, I cleaned my naming. Always appreciate some advice on naming conventions, as I am a bit of a noob. I tried your last version and it works perfectly fine. I just wonder if it brings any advantage on the CPU usage? I am not sure if it works for my final code because color and dataTable are set in setup(). As a next step I will have to lerp from one dataset to another and map it to the altitude. i guess in that case I have to call the createShape() under draw()? Or can the createShape function be influenced by draw, so that color and altitude can be modified?

Thanks guys for your help!