Loading...
Logo
Processing Forum
Hey everyone,

I'm working on a project at the moment with lots and lots of data, reading from a text file. The problem I'm having is really low framerates and then a slowdown - starting on 9fps and then getting slower and slower, tending towards 0fps. The points are being plotted with an opengl vertex buffer object (but this isn't the problem, the framerate was the same before I switched to opengl calls).

VisualVM blames my function drawData(). The code for the entire class that function is in is below. Is there a gaping problem here? Is a slowdown a sign of a bad loop or something, or does it just happen if it's all too much?

Copy code
  1. //This is GOD, the class for a Geospatially Overlaid Dataset.
  2. //It takes a data input, parses it and draws it.

  3. class GOD {

  4.   private String name;
  5.   private String source;
  6.   private String path;
  7.   private String filetype;
  8.   private String seperator;
  9.   private int fieldCount;
  10.   private HashMap metaValues;
  11.   private XMLElement metaData;
  12.   private String[][] godLayerData;
  13.   private int datasetSize;
  14.   FloatBuffer vertexBuffer;
  15.   FloatBuffer colourBuffer;

  16.   GOD(XMLElement GODLayer) {
  17.     metaData = GODLayer;
  18.     loadData();
  19.     initialiseOpenGLBuffer();
  20.   }

  21.   void getMetadata() {

  22.     name = metaData.getChild("name").getContent();
  23.     source = metaData.getChild("source").getContent();
  24.     path = metaData.getChild("path").getContent();
  25.     filetype = metaData.getChild("filetype").getContent();
  26.     seperator = metaData.getChild("seperator").getContent();
  27.     fieldCount = metaData.getChild("values").getChildCount();
  28.     metaValues = new HashMap(fieldCount, 1);
  29.     for (int i=0;i<fieldCount;i++) {
  30.       metaValues.put(metaData.getChild("values").getChild(i).getContent(), str(i));
  31.     }
  32.   }

  33.   void loadData() {

  34.     getMetadata();

  35.     if (filetype==null) {
  36.       println("filetype for "+name+" is either not set or caught");
  37.     }
  38.     else if (filetype!=null) {

  39.       if (filetype.equals("csv")) {
  40.         String[] rawLines = loadStrings(path);
  41.         int ii=rawLines.length;
  42.         datasetSize = ii-1;
  43.         godLayerData = new String[datasetSize][fieldCount];
  44.         for (int i=1;i<ii;i++) {
  45.           String[] splitLine = split(rawLines[i], seperator);
  46.           for (int j=0;j<fieldCount;j++) {
  47.             godLayerData[i-1][j] = splitLine[j];
  48.           }
  49.         }
  50.       }
  51.       else {
  52.         println("filetype '"+filetype+"' is not recognised");
  53.       }
  54.     }
  55.   }

  56.   void initialiseOpenGLBuffer() {
  57.     vertexBuffer = BufferUtil.newFloatBuffer(datasetSize*3); //space in the buffer for x,y,z positions
  58.     colourBuffer = BufferUtil.newFloatBuffer(datasetSize*4); //space in the buffer for r,g,b,a values
  59.     for (int i = 0; i < datasetSize; i++) {
  60.       //  r,g,b,a
  61.       colourBuffer.put(1);
  62.       colourBuffer.put(0);
  63.       colourBuffer.put(0);
  64.       colourBuffer.put(0.4);
  65.     }
  66.     colourBuffer.rewind();
  67.     PGraphicsOpenGL pgl = (PGraphicsOpenGL) g;        
  68.     GL gl = pgl.beginGL();
  69.     gl.glEnableClientState(GL.GL_VERTEX_ARRAY);
  70.     gl.glVertexPointer(3, GL.GL_FLOAT, 0, vertexBuffer);
  71.     gl.glEnableClientState(GL.GL_COLOR_ARRAY);
  72.     gl.glColorPointer(4, GL.GL_FLOAT, 0, colourBuffer);
  73.     gl.glPointSize(1);
  74.     pgl.endGL();
  75.   }


  76.   void drawData() {
  77.     int latitudeIndex = int(metaValues.get("latitude").toString());
  78.     int longitudeIndex = int(metaValues.get("longitude").toString());
  79.     for (int j=datasetSize-1;j>-1;j--) {
  80.       float latitude = float(godLayerData[j][latitudeIndex]);
  81.       float longitude = float(godLayerData[j][longitudeIndex]);

  82.       PVector spaceLoc = equator.returnGlobeCoordinate(latitude, longitude);
  83.       vertexBuffer.put(spaceLoc.x);
  84.       vertexBuffer.put(spaceLoc.y);
  85.       vertexBuffer.put(spaceLoc.z);
  86.     }
  87.     vertexBuffer.rewind();
  88.     PGraphicsOpenGL pgl = (PGraphicsOpenGL) g;
  89.     GL gl = pgl.beginGL();
  90.     gl.glColorPointer(4, GL.GL_FLOAT, 0, colourBuffer);                       
  91.     gl.glVertexPointer(3, GL.GL_FLOAT, 0, vertexBuffer);
  92.     gl.glDrawArrays(GL.GL_POINTS, 0, datasetSize);
  93.     pgl.endGL();
  94.   }
  95. }

The PVector function is also here below, maybe the problem is in the maths here:

Copy code
  1. PVector returnGlobeCoordinate(float latitude, float longitude) {

  2.   float lat = latitude;
  3.   float lng = longitude;
  4.   lat = radians(lat);
  5.   lng = radians(lng);

  6.   float pX = pRadius * cos(lat) * cos(lng);
  7.   float pY = pRadius * sin(lat);
  8.   float pZ = pRadius * cos(lat) * sin(lng);

  9.   PVector outputPVector = new PVector(int(pX), int(pY), int(pZ));
  10.   return outputPVector;
  11. }

datasetSize is about 2,000,000 for one of the datasets (20 for the only other one currently in there). Is that just too much for a loop to run every frame? But then why does it get slower and slower? Or is it just graphics? surely 2,000,000 points should be able to render faster than this, especially when I'm using a vertex buffer object?

Thanks so much!

Replies(6)

Does the location and color for each of these 2 million points change on each draw?

To test if the slowdown is caused by the maths (which is possible) just test it without the maths and see the fps:
Copy code
  1. PVector returnGlobeCoordinate(float latitude, float longitude) {
  2.   return new PVector(1,1,1);
  3. }
You could look into the possibilities of the GLGraphics library.
Hi Amnon,

Removing the maths like you suggested gives me an increase in framerate to approximately 20fps, and, more importantly, it stays there! The biggest thing baffling me is why the framerate keeps slowing down. I guess then it has something to do with my returnGlobeCoordinate function (which gets calculated every frame, but I'll see if I can just calculate it once and then manipulate it all with matrix transformations. The colours do not change

I'll look into the GLGraphics library too

Cheers,
Rafe
Sorry what I just said is actually wrong! I restricted the number of points to 100,000, and that gave me a framerate of about 20. Regardless of the maths, 2,000,000 points still starts on a framerate of 9.03 and then starts dropping rapidly
Ok, some suggestions:
  • Put everything in a GLModel and only set what changes. Don't set everything (positions, colors) on each frame like you are doing in your current code. You can change the position of the complete GLModel with no cost, if the GLModel itself (positions, colors) doesn't change. Even if/when some things change, you can do it. Just not everything on each draw. You are not only saving resources on the setting, but more importantly skipping all the calculations in the first place.
  • Try to work in radians everywhere, so you don't have to convert between degrees/radians. That is really wasteful calculation.
  • Your returnGlobeCoordinate() function does some costly calculations multiple times: pRadius * cos(lat)
  • A PVector is made from floats, so why turn all to int() when setting x,y,z? Seems wasteful.
  • The slowdown may be due to the fact you are creating millions of temporary PVector objects, which aren't removed by the garbage collector. Try setting all PVectors in a globally defined array. Then put the full array into the GLModel at once (I think this is possible in GLGraphics), instead of setting each xyz individually. So you have one single CPU-GPU transaction instead of 2 million. Perhaps this will help improve performance as well.
Thanks amnon, these are some good suggestions. I'll work on them and let you know what happen
Hi Amnon,

I've made all the changes you suggested and I definitely have some much better framerates now, upwards of 30. So thanks for your help and suggestions!

I have some other issues with GLModel and GLGraphics now, but I'll start a new thread