Howdy, Stranger!
We are about to switch to a new forum software. Until then we have removed the registration on this forum.
Categories
- All Categories 25.7K
- Announcements & Guidelines 13
- Common Questions 30
- Using Processing 22.1K
- Programming Questions 12.2K
- Questions about Code 6.4K
- How To... 4.2K
- Hello Processing 72
- GLSL / Shaders 292
- Library Questions 4K
- Hardware, Integration & Other Languages 2.7K
- Kinect 668
- Arduino 1K
- Raspberry PI 188
- Questions about Modes 2K
- Android Mode 1.3K
- JavaScript Mode 413
- Python Mode 205
- Questions about Tools 100
- Espanol 5
- Developing Processing 548
- Create & Announce Libraries 211
- Create & Announce Modes 19
- Create & Announce Tools 29
- Summer of Code 2018 93
- Rails Girls Summer of Code 2017 3
- Summer of Code 2017 49
- Summer of Code 2016 4
- Summer of Code 2015 40
- Summer of Code 2014 22
- p5.js 1.6K
- p5.js Programming Questions 947
- p5.js Library Questions 315
- p5.js Development Questions 31
- General 1.4K
- Events & Opportunities 288
- General Discussion 365
In this Discussion
- blindfish September 2015
- KevinWorkman September 2015
Large Files (csv or json)
I'm working with a large file of data. I have the data in csv (27MB) and in json (52.1MB).
With a sample of 5MB in preload() - using loadTable() with csv or loadJSON() with json - I'm able to read and loop through with no problems, but when I load the original file it crashes. What is the best way to read and loop through big size files? I'm quite new with parsing data... thanks!
Tagged:
Answers
Break your file down into multiple smaller files and load them one at a time.
Assuming you don't want to break files into smaller chunks you'll need to stream large files to make them manageable and improve performance...
It may or may not be possible to stream CSV/JSON using the built-in P5JS methods; but since it's running on JavaScript you should be able to implement this using a 3rd party JS library. I thought streaming of JSON was somewhat bleeding edge; but oboe claims to do the job. For CSV you could try PapaParse.