Out of memory error using PImage ArrayList

lohjianhui

hi everyone, I am creating an app that can capture images from a webcam and display them back to the user. In the Processing window, there is a main screen that will show the live cam feed and the most recent two images. Below the main screen, there is a preview slider panel that shows the captured images in a smaller size. I use the spacebar to capture the images into an arraylist of PImages.

The problem that I have been trying to solve is that I keep getting an out of memory error when I reach about 30 frames or so. I already tried increasing the heap space size to about 1 gig using the "-Xms512M -Xmx1024M" in Eclipse run configuration. Theoretically, since the maximum size of my arraylist is 150 PImages, with each PImage of about 1280x720 pixels, my programme should never hit anywhere near the max limit of 1 gigabyte.

I suspect that there is a memory leak somewhere but I don't know what is causing it. The error message in the Processing IDE says that I am perhaps trying to display thousands of images at once but I am only trying to display around ten images at any point of time.

I have isolated the portion of my code with this issue into a small Processing sketch which can be cut and paste from here:-

https://dl.dropboxusercontent.com/u/16657513/heapSpaceArraylistTest_140317a/heapSpaceArraylistTest_140317a.pde

Do note that to run the code, you need to change the cameraNameString to that of your webcam, preferably around 1280x720 resolution also.

lohjianhui

I'm using Java 1.7 and Processing 2.1.1. Windows 7.

PhiLho

150*1280*720*4/1024/1024 = 527 MB

That's the base memory used by your images. Processing has a cache for the images it displays, so it can double the memory usage, plus internal memory used by Java.

GoToLoop

And we're forgetting about the pixels[] image representation! Each of its bytes is a pixel too!
AFAIK, a PImage instantiated through loadImage() already got pixels[] active, doubling its memory waste!
Dunno whether pixels = null would reclaim that extra space though!

150*1280*720*4/1024/1024 = 527 MB

@PhiLho , what is that *4 about btW? :-??

PhiLho

"Each of its bytes is a pixel too!"
No, it takes 4 bytes to make a pixel. ARGB. That's where the * 4 comes from!

GoToLoop

Oh that's right! 8-} It's an int, 4 bytes or 32 bits! How could I forget that? :-$
Anyways, it means it's 2 * 527 MB if we also count the pixels[]! So it's actually more than 1 GB!!! :-O

zeroisnan

Hi @lohjianhui, why are you storing 150 frames? What frame rate are you trying to target? At 30fps, you're keeping 5s worth of video, is that required?

Also, for this kind of application, it is usually better to use a FIFO structure like Queue or CircularArrayList

lohjianhui

@zeroisnan 150 frames is the max size of my arraylist. It is due to the project specifications that the number of frames would never exceed 150. I am glad it is a small number due to the out-of-memory problems I am having. Just for background info, the project requirements also stipulate that the video will be played back at 5fps for a maximum duration of 30sec for 150 frames.

Also, I don't think I can use a Queue or CircularArrayList because my program is not a FIFO application. The use must be able to remove or insert elements at any position in the data structure.

@PhiLho Is the 527MB quoted by you in a separate cache for images or is it part of the -Xmx1024M allocated to Processing? If it is part of the Xmx1024M, then should I limit the number of images used in the PImage ArrayList to a number less than 150?

lohjianhui

@GoToLoop I instantiate all my images once in setup() using loadImage and not in draw(), but is image(myPImageArrayList.get(index), screenXPos, screenYPos) considered a form of loadImage as well?

Also, correct me if I am wrong, but shouldn't pixels be automatically null unless you do a image.loadPixels() in order to access the pixels?

When I tried setting pixels = null in setup(), the image didn't display at all in draw().

PhiLho

It is part of the mx specification, indeed, in the main memory. What GoToLoop meant is that you have at least the memory used to display the image on screen in addition to your storage, plus perhaps some copy in memory for internal processing, etc.

If you don't care about speed (apparently not), perhaps you can clear the cache after each drawing. Ie., after image(img, ...); call g.removeCache(img);

GoToLoop

What GoToLoop meant is that...

I was recently checking on PJS's implementation of PImage. And it seemed not to depend too much on pixels[].
It used another internal variable for image representation it seemed.

But after taking a more careful look at P5's actual version, it became clear that even though there are some other array fields too,
field pixels[] was indeed the real image representation:

https://github.com/processing/processing/blob/master/core/src/processing/core/PImage.java

The other arrays are -> private int srcBuffer[], blurKernel[], blurMult[][];

Therefore we can't issue pixels = null. At least in the Java's implementation! Sorry! b-(

Is image(myPImageArrayList.get(index), screenXPos, screenYPos) considered a form of loadImage() as well?

If that method get() is from a PImage object, that would surely instantiate a clone() of it!
However I strongly believe that get() actually refers to a List's method version instead.
Then that would result in a PImage existing reference. And finally be rendered on the canvas PGraphics g. (*)

lohjianhui

OK I think I have an idea of why I am getting the out-of-memory error. Thanks to PhiLho, I understand that each PImage is taking up around 3.5MB in raw format in my memory even though it is only 125Kb when saved as a jpeg image on the disk drive (because of the jpeg compression ratio of about 20:1). 150 PImages in raw format in my RAM would take up slightly more than 0.5Gb which is too much space for my program's own good.

On a related note, displaying ten PImages from my memory in its raw format slowed my frameRate from 20fps to 10fps which is unacceptable. I am now guessing that this is due to the large PImages in my memory and the processor is struggling to render all these large images onto the Processing screen.

Now that I understand why I am getting this error, I will try exploring potential solutions such as ... erm... I will have to think about it.

Thanks everyone for pointing me in the right direction.

lohjianhui

Just to add on about the slow framerate caused by displaying the PImages:-

Even after using the P2D renderer, the framerate was about 8fps when I display about 10 raw image files in draw(). The reason, which I just found out, was caused by resizing images on-the-fly. Resizing images on-the-fly in draw() such as image(myPImage, xPos, yPos, newResizedWidth, newResizedHeight) is very processor-intensive apparently. Therefore, it is advisable to display PImages as their original size if possible, i.e. image(myPImage, xPos, yPos).

Just adding this on in case someone else bumps into the same problem as me. Cheers.

GoToLoop

You can resize() those PImage while in setup(). Or at least create resized versions of them!
Actually, we gotta strive to configure anything static inside setup()! /:)