Are you sure you do only that? I am suspicious because of the data(gl) line which won't compile...

Anyway, I just took your code, changed to a path to a 200MB file, and got a 200MB binary file in the output. My Processing max memory settings are set to 1024MB. Running 1.5 on Win7 with JDK 1.6.0_25

Workaround: read the file by chunks and write them on the fly.

Hi again, I'm still no closer to a working solution. Can anyone who has encountered this issue and is aware of a process which would, as phi.lho previously suggested, "read the file by chunks and write them on the fly" please provide some more information on how to achieve this for the sketch above.

I've not been able to track down much helpful information about how Processing/Java deals with memory, so the best I've managed so far on my own is to break down the process into pseudo code but how I turn that in to a working set of instructions is still beyond me. Without really understanding enough about how Processing handles memory and data I'm really just guessing it will work similar to the way I have suggested below so please excuse me if I'm way off the mark in my mental approach to the process.

If the mp3 is larger than 200MB
Read only 200MB of the file at a time (to avoid the OutOfMemoryError)
Run through the code applying the process to the first 200MB portion of the file
(At this stage would you write this first chunk of the input file out to a 200MB temp file that's joined at the end?/or would it be better to create an empty output file the same size as the input file which the various 200MB parts of the original input file are then sequentially added in to as they are processed?)
Once the process has been applied to the first 200MB portion of the file
Apply the same process to the remaining 200MB chunks of the file until the entire file has been processed
(At this point if the temp file merge route is most appropriate then this is the stage that the final output file would be created by merging all of the 200MB temp file chunks of the original input file)

I'd really appreciate any links to relevant information, help or suggestions that anyone may have.

Many Thanks.

Here is some code copying a binary file of any size, chunk by chunk. You can add you altering code in it:

1. final int BUFFER_SIZE = 4096;
2. String fileNameIn = "G:/Tmp/LargeImage.jpg";
3. String fileNameOut = "G:/Tmp/LargeImageA.jpg";
4. 
   
5. byte[] buffer = new byte[BUFFER_SIZE];
6. InputStream in = null;
7. OutputStream out = null;
8. try
9. {
10.   in = new BufferedInputStream(new FileInputStream(fileNameIn));
11.   out = new BufferedOutputStream(new FileOutputStream(fileNameOut));
12.   int n = in.read(buffer, 0, BUFFER_SIZE);
13.   while (n > 0)
14.   {
15.     out.write(buffer);
16.     n = in.read(buffer, 0, BUFFER_SIZE);
17.   }
18. }
19. catch (IOException e)
20. {
21.   println("Error: " + e);
22. }
23. finally
24. {
25.   try
26.   {
27.     if (in != null)
28.       in.close();
29.     if (out != null)
30.       out.close();
31.   }
32.   catch (IOException e) {}
33. }
34. exit();

It is quite easy to find such information on Internet (if you search for Java), that's why I didn't feel necessary to expand.
The code above just duplicate the image, I can open it to verify it is unchanged. It should work for mp3 as well.

This method of sprinkling random bytes over a binary file is interesting, but IMHO should be applied only on uncompressed data. On compressed data (be it Jpeg image, MP3 music, etc.), it can just crash the reader (if not robust), or stop reading the file (on detected error), etc., because compressed data follows a precise encoding, so altering it can lead to errors.

Yes, kooogy, that's the point of the original post, OP just has to change a bit the algorithm to a stream version of the data, either by pre-computing where in the stream the random bytes must overwrite data and watch the position in the file to trigger the replacement, or by sprinkling the random data on each chunk.

phi.lho and koogy, thank you both greatly for your replies, you are indeed correct in your observations. I am attempting to generate as many errors as possible in the output files. I am quite happy to manually wade through the problematic output files to discover the few which have survived the process, retaining the errors but without crashing the reader during playback in the case of video/audio formats or rendering the file useless in the case of image files.

I am applying an iterative process of corruption, whereby I select the output file which has survived each stage of the corruption process with the most interesting visual artifacts, then I re-apply the process over several hundred generations, manually selecting the most favourable survivors. After about a hundred or so generations, having slowly introduced tens of thousands of errors to the original file, some of the traits that emerge in these files are downright bizarre and, to my eyes, beautiful.

koogy I wish I was able to translate your short and sweet pseudo, but I'm an utter noobatron! My brain just turns to jelly when I try to think about how to convert your generous instructions into working code.

>>if you're copying the file anyway, why not introduce the errors then?<<

I wouldn't know how to, but I'd love to hear an explanation of why this might be a better approach and in what way your approach differs both from my original process and the method that phi.lho has suggested above.

>>in C i'd just open (a copy of) the file for random access, seek to the required position and replace the byte with garbage.<<

I'd love to see this in action, you make it sound so easy!

Thank you also for the link you provided to the Java random access equivalent. Unfortunately at this stage of my learning, much of what is being described in the link is currently beyond my comprehension.

phi.lho I am as of yet unable to integrate my own lines of code into the sketch you have provided. I don't really understand what's going on in the code yet but I'll endeavour to make it work on my own through trial and error for the next few days before I consider returning to request another spoonfull of your immense knowledge!

>>OP just has to change a bit the algorithm to a stream version of the data, either by pre-computing where in the stream the random bytes must overwrite data and watch the position in the file to trigger the replacement, or by sprinkling the random data on each chunk.<<

I'll now have to partially contradict my previous statement of intended abstinence from quizzing your brain to request further insight into what you mean by this. Once again I'm employing guesswork as my grasp of the ideas you express are minimal so apologies if I've misunderstood or my questions don't make much sense!.. What type of data is the current algorithm handling/generating if not a stream version of the data? Does this apply to isolating favourable corruption traits in order to reproduce them? i.e. If I am "pre-computing where in the stream the random bytes must overwrite data", will this process of pre-computing involve an awareness of the location in memory and/or the data stream where a favourable corruption trait has been previously observed (by using println to output to the terminal details of the location(s) in memory/the data stream where the causes of these favourable traits were first observed), and once observed and noted, require me to add instructions into the code which will refer to these locations in subsequent sketch run's to consistently reproduce these traits?

Hopefully my queries are at least travelling along the right lines! I look forward to your replies.

Many Thanks.

Just a quick answer (lack of time), ask for more later if you need.
First, thanks for explaining your idea, indeed if you filter out the many attempts, it is viable, just a work of patience...

Second, the idea is to take the file length, generate a few (offset, value) pairs, and when reaching a position in the file, to alter the bytes. Or, you can change just a byte per buffer load, which can introduce more errors, depending on the size of the buffer. Or alter one byte per randomly selected buffer (ie. take a probably that a buffer is altered or not).