memory problems reading large XML files

Programming Questions

skeewbob

memory problems reading large XML files

in Programming Questions • 3 years ago

I'm having trouble reading medium to large XML files using processing's built-in xml library. When I edit out part of the XML to make it a smaller file size I can get it to load into memory, but the original XML causes a "NullPointerException".

For example, a 230KB xml file would not load, but it will load when I cut it down to 172KB. I'd like to be able to load files up to 10,000KB.

Are there any easy ways to sequentially load and work on just a small portion of the XML file to prevent memory problems? I've tried increasing the maximum available memory preference in Processing but that didn't solve my problem.

Is there an easy way to do this using proXML? I'm just learning Processing so the solution needs to be simple

Thanks in advance for any advice!

-Bob

Replies(2)

PhiLho

Re: memory problems reading large XML files

3 years ago

Processing's XML reader relies on NanoXML which is a Sax reader, ie. it reads the file linearly, generating events for each element it finds, without memorizing the past elements nor building its own in-memory structure (unlike a Dom parser). It just find elements and pass them to your code.

Moreover, 230KB of XML isn't that much. By default, Processing allows to use up to 256MB!

So I think the issue is more in the way you use the data you read than in the XML parser itself.

skeewbob

Re: memory problems reading large XML files

3 years ago

Thanks for the help phi.lho! Your suggestion made me look at other potential problems in my code rather than focus on a memory issue and I found a bug in how I was processing my XML data. I can now read in XML files of at least 12,000KB without a problem.

-Bob

Top Reply