out of memory error on loadImage() in Draw loop

jeffmarc

So the Java garbage collection and memory management works in the background . In post from 2016 from Henri , "PImage won't release memopry" KevinWorkman says the Java garbage collection and memory management works in the background so "Out of Memory" error should never happen. Wonder if it is fixed in later versions or what cause I get it in the following scenerio.

Using 2.2.1 and getting out of memory when i run sketch. basically it asks the user to grab an image. The image is then loaded with loadImage() After 1000's of draw loops , i get out of memory and sketch halts with error messages in console. Is the auto memory management in Java the same in this version or better in later versions of Processing? The program uses an event counter to allow only one iteration of each condition in the draw loop. 

  Step 1 - wait for user to request image grab 
  Step 2 - loadImage and display with image() 
  Step 3- perform image processing 
  Step 4- report numeric results ( count all pixels below threshold) 
  Reset event counter to 1

  I inserted a wait time of 1 second to auto initiate the loop and it runs for hours and finally gives out of memory 
 So is the garbage collector not working in theis version or what?

Thanks

Chrisir

show code or mcve

kfrajer

Why not to continue in your previous post? https://forum.processing.org/two/discussion/26006/out-of-memory-error-on-loadimage#latest

Kf

TfGuy44

You keep saying that you have to use loadImage() inside draw(), and we keep telling you that you probably don't, but we can't help you rewrite it because you haven't actually shown us the code of that yet.

Post. Your. Code!

jeffmarc

I use open to calla program that snaps a image form a camera and stores it as .png file. THis image is different each time in the draw loop and must be loaded for processing by an algorithm. All this works fine for 1000's of grabs until i get a out of memory. in the post i reference KevenWorkman writes: Also note that according to the Processing source code, the image cache is a WeakHashMap, so you don't need to do anything special to clear the cache. It will be cleared if you're running low on memory. The whole program is big and it works except for this focused problem that always halts the program with a out of memory at the same place, in the tempimg=loadImage() statement. setting tempimg to null at the start of the loop deosn't stop out of memory either.

I put a counter in the grab sequence and it does execute about 6 times before the draw loop goes to next event , which triggers the processing of the image and then the next event which is results written to screen.

kfrajer

You can create a MVCE demonstrating your approach. It is hard to provide any feedback without seeing some code. Also, cross-reference the prev kevin's post for completeness.

**** EDIT: https://forum.processing.org/two/discussion/18762/can-t-release-pimage-memory

@jeffmarc Keep all related conversations in a single post. Otherwise you are going to create duplicate answers, wasted efforts and lots of confusion.

Kf

jeffmarc

ok. question: PImage img above setup creates a global image object for life of program correct? So every time I reference img in the draw loop, as in img=loadImage("newimage.png") doesn't that just use the memory for that one object over and over, loading a new image into that memory? The image is always 640x480 it never changes size. The file name is the same , it is overwritten by the grab program. I delete the file each time before grabbing and creating a new one. I check the new one "exists" in a while loop before continuing. All these have been the result of forum answers , and i still get out of memory within 24 hours of the sketch operation, always at the same line, the tempimg=loadImage("newimage.png")

TfGuy44

Again, you probably are calling loadImage() much, MUCH more often that you need to. AGAIN, this is hard to say with out SEEING YOUR CODE. POST. YOUR. CODE! The draw() function tries to run about 60 times a SECOND, so unless you are doing some sort of check to limit the amount of times you are reloading the image, you are loading that image 60 times a SECOND. If the image only changes every minute, you have 359 calls to loadImage() that you just DON'T NEED - no wonder you run out of memory!

POST.

YOUR.

CODE.

if I say it again, will you do it?

POST!!!!

YOUR!!!!

CODE!!!!!!

jeffmarc

As i stated innpost above i have proven it is only called 6 times after user requests new image so it is not loading 300 times

jeremydouglass

i have proven

That is not what "proven" means. "Proven" means demonstrating that something is true by sharing evidence.

For example, sharing sample code that demonstrates a sketch which calls loadImage six times per request, yet runs out of memory.

You have claimed this, but have not proven this, causing frustration for those trying to help you.

jeffmarc

So just answer me this.

Is this NOT true?

My initial reasoning was that :

PImage img;

was defined as a global variable, and so each time I used:

img = loadImage("newimg.png");

the new image would be loaded into the img variable memory space, not take new memory!

But it does use more memory each time it is loaded even though it is using only one variable name declared gloabally?

So the simple solution would be to "release" these memory references So my simple question is there a way to do this in Processing or not?

I know in C i can release memory back to the memory pool, but is this not possible in Processing?

koogs

You don't know what loadImage is doing. It could be allocating new memory and passing back a pointer to this new object, leaving the old, unreferenced memory to the garbage collector.

Which should be simple enough to verify.

jeffmarc

What do you mean you don't know what loadImage is doing?

I really don't understand Java and Processing then, in C,C++ there is code that defines functions like loadImage and they behave a certain way all the time so you can determine how to use them or program a work around.

If it's simple then how do I verify what is happening memory wise ? I tried monitoring memory but it didn't show any abnormal usage.

Does anybody know how loadImage really works inside draw?

KevinWorkam claims that according to the Processing source code, the image cache is a WeakHashMap, so you don't need to do anything special to clear the cache. It will be cleared if you're running low on memory.

I have found several posts with similar issues with using loadImage. It just makes sense that if you declare a global PImage then loading different images into it should not use up memory, but apparently it does.

Putting PImage in draw it doesn't.

reference :

https://forum.processing.org/two/discussion/18762/can-t-release-pimage-memory

kfrajer

@jeffmarc Great you cross-linked to the previous post. I will strongly suggest you continue any other discussions here.

Related to your code. Are you calling loadImage() in draw()? Additionally, are you using the standard or P3D renderer?

To check what loadImage() does, visit: https://github.com/processing/processing/blob/master/core/src/processing/core/PApplet.java#L5370

If you are working with a png type of file, go to: https://github.com/processing/processing/blob/master/core/src/processing/core/PApplet.java#L5425

I really don't understand Java and Processing then, in C,C++ there is code that defines functions like loadImage and they behave a certain way all the time so you can determine how to use them or program a work around.

If you come from C/C++ background, then it might work as you said. If you are using a library or framework, you can only guess what they do unless you check under the hood. If the library is widely used, either other people could have found a problem, it gets reported and it is solved.

Processing is not any different. The loadImage is performant. When a bug is found, well it is bad and it is good. Bad because it affects your code. Good bc you found it and you could report it. However...

When you find a bug, the first instance you should take is: Is it really a bug or is it something else.... like "me". In other words, for a something that smells like a bug and walks like a bug, it needs to be a bug. But you need to prove first it is a bug. There are two ways:

1. You go into the source code, you trace the code and you show the problem. You can go the extra mile and propose a solution. You can fork the repo and re-generated the jar files for a modified version of Processing where you fixed the bug. This way you demonstrate not only that the bug is real but you also have a solution. Notice a solution is not the solution unless it doesn't break any other features in Processing. Otherwise a more careful solution is required.
2. Another way is to provide a sample code and then other community members can run the sample code and confirm the bug. You see, this approach is more convenient for the layman because the work to be done is minimal. The community validates the problem and then you can proceed to open a ticket in github. This step is also convenient because it also shows your approach, or maybe there is a better way to do the things you want to do.

Related to loadImage() using the same memory space. Not sure if that is true. I hope Java is smart enough to do that. The JVM suppose to be able to optimized your code after it is run few times. I cannot say much about code optimization as it is out of my realm. However, you need to remember there is an extra layer between your code and the JVM and that is the Processing layer. At this point one can assume Processing is doing the right thing unless proven otherwise.

Once again, an MCVE is needed. If you open a ticket in github related to this issue, the first thing they will ask if for some code demonstrating the problem. People can tell you so many times what is needed here but if you don't collaborate, everybody's efforts are just and simply wasted. So before you claim something and based this problem on an assumption from your side, show some code.

Kf

jeffmarc

Still like to know how to verify what Koog was saying regarding memory left over for garbage collector

Thks

kfrajer

This is my attempt to create and read the same image using independent threads. The next code is jut a demo with plenty room of improvement specially in the concurrency side.

Parameters of interest:
* Set how often an image is re-generated/re-created
* Set how often an image is loaded
* Choose if loading the image only if file has changed
* The size of the image as it will have more impact on resources for larger images

I tested the code in JAVA2D and P3D and I can see my mem consumption going up but they always peak. I used Windows's task manager to keep an eye on RAM resources.

Kf

//REFERENCE: https://forum.processing.org/two/discussion/comment/73362/#Comment_73362
//REFERENCE: http://www.rgagnon.com/javadetails/java-0490.html
//REFERENCE: https://docs.oracle.com/javase/tutorial/essential/concurrency/index.html  
//REFERENCE: https://stackoverflow.com/questions/18223931/how-can-a-dead-thread-be-restarted


//INSTRUCTIONS: [Version 2.0]
//         *--  This code create and load images at different rates using independent threads.
//         *--  The creating of a radom image is done using its own thread
//         *--  Loading the image can be done either by consistenly loading the image
//         *--  every so often OR by loading the image only if it has changed. To 
//         *--  choose what loading method to use, use CHECK_IF_EXIST_BEFORE_LOADING
//         *--  field. Notice than loading an image is using requestImage() as it creates
//         *--  its own thread to load the image.
//         *--  

//===========================================================================
// IMPORTS:
import java.util.*;
import java.io.*;


//===========================================================================
// FINAL FIELDS:
final String FILENAME="image.jpg";
final boolean CHECK_IF_EXIST_BEFORE_LOADING=true;  

final int SYSTEM_FRAMERATE=30;

final int FRAMECOUNT_WRITING_RATE=200;
final int FRAMECOUNT_READING_RATE=60;
final int VERBOSITY_RATE=30;



//===========================================================================
// GLOBAL VARIABLES:
PImage mainImg;
boolean imgReady=false;
TimerTask readTaskSupervised;
Timer timer;
TimerTask aWriter;


//===========================================================================
// PROCESSING DEFAULT FUNCTIONS:
void setup() {
  size(400, 400);
  frameRate(SYSTEM_FRAMERATE);

  aWriter=new WriteImage();
  aWriter.run();  //Run once to create image locally

  readTaskSupervised = new FileWatcher( new File(sketchPath(FILENAME)) ) {

    protected void onChange( File file ) {

      if (!file.exists()) {
        println("File is not ready yet!!!");
        return;
      }
      // here we code the action on a change
      println( "File "+ file.getName() +" have change !" );
      mainImg=requestImage(FILENAME);
      imgReady=false;
    }
  };

  timer = new Timer();

  //LOAD image #1: Loading only if file has changed
  if (CHECK_IF_EXIST_BEFORE_LOADING) {
    int timeInSecs=int( (FRAMECOUNT_READING_RATE*1.0/SYSTEM_FRAMERATE)*1000 );
    timer.schedule( readTaskSupervised, new Date(), timeInSecs);
  }

  //TASK to create an image
  int timeInSecs=int( (FRAMECOUNT_WRITING_RATE*1.0/SYSTEM_FRAMERATE)*1000 );
  timer.schedule(aWriter, new Date(), timeInSecs);
}



void draw() {
  background(0);


  //LOAD image #2: Continuous loading 
  if (!CHECK_IF_EXIST_BEFORE_LOADING && (frameCount%FRAMECOUNT_READING_RATE==0) ) {
    mainImg=requestImage(FILENAME);
    imgReady=false;
  }

  if (mainImg==null || mainImg.width<=0) {
    if (frameCount%VERBOSITY_RATE==0) println("Waiting for image @"+frameCount);
  } else {

    if (imgReady==false) {
      mainImg.resize(width, height);
      imgReady=true;
      println("Resizing done @"+frameCount);
    }

    image(mainImg, 0, 0);
  }
}




//===========================================================================
// OTHER FUNCTIONS:

class WriteImage extends TimerTask { 

  void run() {
    create();
  }

  void create() {

    PImage img;
    int w=4000;
    int h=3000;

    if (w+h<100) {
      println("WARNING: Invalid dimensions: w+h must be greater than 100");
      return;
    }

    //File f = new File(sketchPath(FILENAME));
    //if(f.exists()){
    //  f.delete();
    //  delay(500);  //Safety net
    //}
    img=createImage(w, h, ARGB);
    img.loadPixels();
    int green=int(random(5, 250));
    int blue=int(random(5, 250));
    random(50, 200);
    for (int i=0; i<w*h; i++)
      img.pixels[i]=color( (i%w)*1.0/w*255, green, blue );  // A random gradient across the width of the iamge 
    img.updatePixels();
    img.save(FILENAME);
    img=null;
  }
}



//=================



public abstract class FileWatcher extends TimerTask {
  private long timeStamp;
  private File file;

  public FileWatcher( File file ) {
    this.file = file;
    this.timeStamp = file.lastModified();
  }

  public final void run() {
    long timeStamp = file.lastModified();

    if ( this.timeStamp != timeStamp ) {
      this.timeStamp = timeStamp;
      onChange(file);
    }
  }

  protected abstract void onChange( File file );
}

koogs

Verifying it is 4 lines of code. But I'm not posting any code until you do 8)

Think about what you're saying though, that it reuses the memory. I load a 1x1 png into a global called img and then load a 1000x1000 png into the same global... Or the other way around.

Or I call malloc() twice and assign it to the same global variable without freeing it in between. What happens to the first block of memory?

neilcsmith_net

@jeffmarc there's some interesting misunderstanding in this thread. But if you come from a C/C++ background the easiest way to understand this is that a PImage field is a pointer. All references in Java are pointers. When you call loadImage() new memory is used and the pointer changed to that. You cannot manually free() - when no live pointers remain then memory can be garbage collected. This happens asynchronously - not at the time you change the reference. This is why memory use will go up (and then down again) when loading new images.

You can use a tool like VisualVM to take a snapshot of the heap and prove existence of memory leaks (using GC root to find pointers that shouldn't still be live). I have done this in the development of Praxis LIVE, which is more affected by this than a typical Processing sketch would be, and reported / helped fix a few leaks in the OpenGL renderer. There is still one I know of, but not image related.

jeffmarc

Thanks for the explanation. The more i understand HOW things work then i know where to look. Today i was running the sketch with the minimal I/O , no usb to rs-232 connected. The sketch uses a usb image device to grab an image from a camera. THis is the result of discovering that the video lib fro processing despite recognizing the device and the camera input being in the list, it will not work with the vid lib. So i use a usb device and trigger it to grab an imageto a file with open(); A usb to serial controller is used to turn a light on and off as needed for the test. I unplugged the serial device from the usb last night and started the program again. To my surprise it ran over night without getting out of memory, and is still running. It is running the same sketch with no changes on the same PC. THe only thing different is no serial port. IN the program in setup if no serial ports are found then the program does not tryto open a serial port at all, talk to it in any way. In the if's that use the serial port i use stop() to turn them off after sending data. I open the serial port, send data and stop(); they are not looping with draw 60 times a sec, they only ruj when the if is true, twice in the draw loop per grab sequence. The rest of the time the draw loop is idleing waiting for a mouse pressed condition to initiate the grab. So any clue how the serial lib could be leaving mem behind ? I use it as recommended in other posts to eliminate it even causing a serial port busy error. It is one port, with myport=new Serial(this,Serial.list()[0],9600) before writes and myport.stop() after each time. println statements to console prove it is only called twice per iteration of draw per mousepressed.

So in my experience debugging it seems the serial calls are the problem causing out of memory that then manifests itself on a loadImage statement the next time memory is called for.

jeffmarc

Opening each serial port after checking they were seen with serial list, and stopping them only on program exit cured the out of memory errors. I have no run over 30,000 interactions in 3 different applications that all exhibited the problem. They all work for days now without out of memory errors I don't know why , i just find solutions by deduction and trust me , something with opening new Serial(this and then stop() withing draw loop causes out of memory.

kfrajer

Great, thank you for sharing your solution. May I ask why you were calling stop()? I have done arduino and serial stuff before and I didn't call this at all before. Where you following some reference that was advising you to do this?

Kf

jeffmarc

I was having issues with the program not starting as an .exe but icouldn't tell what was hanging it up. When i removed serial support it worked. It just made sense to only open the serial port i needed and use it to output and close ( stop ) it. so there would be no chance of a open serial port when the program closed. Apparently this method used memory every time i opened the ports again and stop does not clean up after itself with regard to memory.

kfrajer

I guess this post capture this problem well. Can you comment if you were using the Processing libraries for this or your own (external) Java functions to access the serial line? This is important detail related to this issue before it is closed.

Kf

jeffmarc

I imported the serial libraries in processing for install of ver 2.2.1

import processing.serial.*;