We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
IndexProgramming Questions & HelpPrograms › Huge datasets
Page Index Toggle Pages: 1
Huge datasets (Read 397 times)
Huge datasets
May 2nd, 2008, 6:44pm
 
Hi all,

I will have to visualize huge datasets in the near future and am looking at the different options.

How good is processing for pulling huge datasets from a database and then visualizing (I'm talking millions of datapoints here...)

Thanks.
jan.
Re: Huge datasets
Reply #1 - May 8th, 2008, 5:46am
 
It really all depends on the kind of visualization you're hoping to do.  As far as interfacing with a database, check out http://processing.org/reference/libraries/index.html#data_protocols, there is a MySQL library, so the database connection should be  fairly easy.  However, any sort of visualization involving millions of points is going to be quite processor intensive - you're not going to be able to plot a million points per frame in Java (at least on any normal computer), no matter how fast the libraries you use are!

Think about what exactly you need to display, and see if you can figure out a way to build it up incrementally without having to keep all the data around or process it all for each frame - I have used Processing to visualize millions of pieces of stock data before (I was calculating distributions and testing options valuation techniques based on the whole history of available daily data for every currently listed stock), so it's definitely possible, but you have to choose your algorithms very carefully with that much data.
Page Index Toggle Pages: 1