Large Files (csv or json)

edited September 2015 in p5.js

I'm working with a large file of data. I have the data in csv (27MB) and in json (52.1MB).

With a sample of 5MB in preload() - using loadTable() with csv or loadJSON() with json - I'm able to read and loop through with no problems, but when I load the original file it crashes. What is the best way to read and loop through big size files? I'm quite new with parsing data... thanks!


  • Break your file down into multiple smaller files and load them one at a time.

  • Assuming you don't want to break files into smaller chunks you'll need to stream large files to make them manageable and improve performance...

    It may or may not be possible to stream CSV/JSON using the built-in P5JS methods; but since it's running on JavaScript you should be able to implement this using a 3rd party JS library. I thought streaming of JSON was somewhat bleeding edge; but oboe claims to do the job. For CSV you could try PapaParse.

Sign In or Register to comment.