Memory leak or misunderstanding? (IPCapture)

tohox

Hi,

The program below grabs frames from an MJPEG camera stuffs them in a circular buffer and displays them with a predefined delay.

While visually this behaves as expected the problem I have is that memory usage continuously increases while the program runs and eventually stops when it reaches the limit.

The IPcapture library example grabs the frames by reading the cam1 instance directly as so: buffer[x] = cam1;

But I've had to change this slightly because it would always show the last frame regardless of where in the buffer I was reading from. My guess is that assigning this way is similar to using a pointer to the actual IPcapture object and not a copy of its image data. So I did this instead: buffer[x] = cam1.get();

This works as expected making a copy of the frame and storing it in the PImage buffer. The problem I have then is that it seems to be making a copy of the whole IPcapture object and not just the image data taking up more memory every time I copy it to the buffer.

Is this a problem with the way the library works or am I not doing this properly? Any input appreciated. Thanks!

import ipcapture.*;

IPCapture cam1;

// Circular buffer
PImage[] cam1_buffer; 

// This determines the size of the circular buffe
int nFrames = 500;
// These are used to iterate through the buffer while writing and reading
// the read counter should always be at least one frame ahead of the write counter
int iWrite = 0, iRead = 1;

void setup() {
  fullScreen();

  // Set the cam's URL and start it
  cam1 = new IPCapture(this, "http://" + "192.168.0.251:88/cgi-bin/CGIStream.cgi?cmd=GetMJStream&usr=root&pwd=root", "root", "root");
  cam1.start();
  cam1.pixelWidth = 640;
  cam1.pixelHeight = 480;

  // Declare a frame buffer sized accordingly to our desired number of frames
  cam1_buffer = new PImage[nFrames];
}

void draw() {
  // Gets a frame from the camera and store it in our buffer 
  if (cam1.isAvailable()) {
    cam1.read();
    cam1_buffer[iWrite] = cam1.get();
    // Reads a frame from our buffer
    if(cam1_buffer[iRead] == null){
      // Display this text if the buffer isn't full yet.
      clear();
      fill(255);
      text("BUFFERING " + str(iWrite) +" / " + str(nFrames),300,10);
    }
    else{
      // Here we display our delayed frames
      image(cam1_buffer[iRead],0,0);
    }    

    // Increment write and read counters
    iWrite++;
    iRead++;

    // Start writing over the begining of our buffer every time we reach the end
    if(iRead >= nFrames-1){
      iRead = 0;
    }

    if(iWrite >= nFrames-1){
      iWrite = 0;
    }
  }
}

koogs

it seems to be making a copy of the whole IPcapture object

this says otherwise: http://www.stefanobaldan.com/projects/ipcapture/reference/index.html

and you'd get a class cast exception if it was returning an IPCapture.

your problem might just be the 500 pimages in your circular buffer, that's quite a lot.

there was another thread recently about pimage internals not freeing up memory. have a read: https://forum.processing.org/two/discussion/16362/possible-memory-leak

tohox

Hi Koogs,

Thanks for your quick response!

I've looked at the IPcapture reference but what makes me think I'm actually making copies of the complete IPcapture instance and not just its image data is that when I debug and look at the contents of my circular buffer I can see all the members of the IPcapture object and not only those of a Pimage object...

If the IPcapture is only exposing a Pimage when I copy it to my buffer why then do I see all these objects passed along : urlString, user, pass, httpIn, curFrame etc. ?

I had also looked at the thread you mention and in another version of my program I tried using removeCache() by overloading the image() method as so but it had no effect:

@ Override
public void image(PImage img, float a, float b, float c, float d) 
{
    super.image(img, a, b, c, d);

    PGraphics pg = getGraphics();

    Object cache = pg.getCache(img);
    if (cache instanceof Texture)
        ((Texture) cache).disposeSourceBuffer();

    pg.removeCache(img);
}

As for the number of Pimages in my buffer it only seems to influence the memory inflation rate (the larger the buffer the faster the memory usage increases) but the problem is present no matter what size I define my buffer.

koogs

when I debug and look at the contents of my circular buffer I can see all the members of the IPcapture object

like what?

import ipcapture.*;

IPCapture ipc;
PImage p;

void setup() {
  ipc = new IPCapture(this);
}

void draw() {
  p = ipc.get();
  noLoop();
}

a breakpoint on the noLoop shows me p as a PImage and ipc as a IPCapture and no unexpected fields. PImage contains a parent PApplet which has access to the IPCapture but...

koogs

mind you, PImage is hardly lightweight...

https://github.com/processing/processing/blob/master/core/src/processing/core/PImage.java

tohox

Indeed... Is there a simpler object representing an image which I could extract from the original IPCapture and more easily store in great numbers? In fact this is what I thought .get() would accomplish by copying only a bunch of pixel colors.

The ultimate goal being to store up to 30 minutes of MJPEG frames from 6 simultaneous streams...

tohox

This is what my buffer looks like when assigned from a get() which as you mentioned above is consistant with a PImage:

This is what my buffer looks like when assigned directly from the IPcapture object as suggested by the library examples:

The first example gives the result I want but causes the memory inflation. The second example doesn't cause the memory to inflate but all elements in my array seem to point to the live feed instead of being copies of frames captured previously.

tohox

Update:

I repeated the same example as above but with the Capture object instead of the IPcapture object and got the exact same behaviour. buffer[x]=cam works ok but doesn't store the frames while buffer[x]=cam.get() works as expected but memory goes through the roof. So the problem doesn't seem to be related to the IPcapture but indeed with the way I'm storing the frames.

Also same behaviour when debugging with the PImage buffer elements showing the Capture fields this time.

GoToLoop

https://forum.Processing.org/two/discussion/16435/how-to-make-the-webcam-capture-window-go-away-or-put-images-over-it

tohox

Thanks GoToLoop, I like the way you Capture.read() only when a frame is actually available as this currently takes a lot of time inside each of my draw iterations. Unfortunately I can't seem to find an equivalent event for IPCapture... Perhaps run a separate thread checking for available frames and reading only when necessary?

Anyway, going back to my inflating memory problem I found that forcing the garbage collection immediately after a frame has been read from my circular buffer managed to keep the memory usage stable. This comes at at significant performance cost however...

I'm not sure if the problem is due to the garbage collector not having enough time to do its job "naturally" or if I should be doing something to explicitly release some resources.

GoToLoop

captureEvent() isn't necessary. Just return when not isAvailable(): if (!cam.isAvailable()) return;

In that sketch, background() is used in place of image().
The advantage of both background() & set() is that the PImage isn't cached.

tohox

Thanks, unfortunately I'm making use of tint() in the real application so I assumed image() is necessary and set() or background() won't do.

tohox

To conclude this thread, here is what I learned from koogs and GoToLoop's helpful comments:

• Storing hundreds of PImages rapidly is generally not a good idea. They take up a lot of space fast.
• If you use the image() method, a cached version of the PImage you pass to it gets created and isn't necessarily dealt with automatically by the garbage collector. You may need to use removeCache() each time you use image() to keep memory usage from creeping up.
• If you're not using Tint() you could use Set() or Background() to display your PImage instead of Image().

Thanks guys!

GoToLoop

and isn't necessarily dealt with automatically by the garbage collector.

Actually it is eventually dealt w/ b/c the cache is a WeakHashMap:
http://docs.Oracle.com/javase/8/docs/api/java/util/WeakHashMap.html

However, if the rate the images are cached is too high, the cache isn't emptied fast enough.
That's why a forced removeCache() is needed for such cases. 8-|

GoToLoop

You may try to have a PImage[] array w/ about 1000 length.
Once filled up, pause() the Capture, save() everything; removeCache() and null the whole array.
Then start(), rinse and repeat the whole process. *-:)

koogs

toxi used to use a cache (specifically whirlycache) to cache his pimages when doing something like this. it would allow instant(ish) access to images without having to worry (that much) about memory.

this was 2010 though and all the links are dead and i can't google up the code anywhere...
https://forum.processing.org/one/topic/getting-multiple-images-to-queue-and-load.html

koogs

although that's probably more valid for stored files than for video frames.