We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
IndexDiscussionGeneral Discussion,  Status › memory increase
Pages: 1 2 
memory increase (Read 5218 times)
memory increase
Jan 8th, 2010, 7:24pm
 
Dear All,

i am using processing with sunflow on a super computer that has 16 processors and 16gb of memory. the problem is that on  a windows server edition, i can only increase the memory to 1.5gb. it would be great if there is a way to utilize the other memory of such computer.

next i wanted to try is to use a super computer with linux, but i guess i would again have to stick to around 2gb memory limit.
since the sketches we run are memory intense, lot of the times the sketch cannot run and gives me "outofmemoryerror.. try increasing it.."
when i increase it, then processing won't compile it.

is there are a way to increase the memory to lets say 12gb?

thanks.
t
Re: memory increase
Reply #1 - Jan 8th, 2010, 9:10pm
 
I'm not sure that's a Processing issue so much as a Java Virtual Machine issue.  Take a look at this thread.. little old but maybe it will provide an answer.
http://www.theserverside.com/discussions/thread.tss?thread_id=26347

Also, I don't know that Processing can utilize multiple processors... I think it's single threaded, unless you specifically create new threads.  So your supercomputer may not be any more powerful for Processing than a new 500 dollar desktop with a decent video card.  I could be wrong here though.
Re: memory increase
Reply #2 - Jan 8th, 2010, 9:33pm
 
Dear Jeffg,

thanks for your response.

when i run a processing sketch, under task manager i can see that 8 processors activity jumps from 0 to maximum, so it looks like multithreading is happening. although it cuts down rendering time, but it is not really radical, it is probably about one third of how long does it take to render stuff on mbp.

the main issue now is that i can't even run some sketches because they are only using 1.5gb memory allocation in processing on windows server edition (supercomputer) and on my MBP i run them on 2gb memory allocation.

has anyone been able to run processing with higher memory allocations - like 10gb and on what OS?
tx!
t
Re: memory increase
Reply #3 - Jan 9th, 2010, 2:08am
 
The question might be silly, but is your Windows Server Edition a 64bit system? If not, I think you are limited by the system.
The thread pointed out by jeffg is interesting, it shows there are many pitfalls at using lot of memory in (not so) old systems...

If your sketch isn't real time (unlikely with Sunflow, no?), perhaps you could compute smaller images by fragmenting the big one, then stitch them together.
Re: memory increase
Reply #4 - Jan 9th, 2010, 4:18am
 
yes it is a windows server 2003 64bit edition. i am running processing 1.0.9, so i guess it is not very easy to change the memory settings?

is there any 'easy' way to compile processing for such system, so that memory can be increased. also i don't understand how can i setup 2gb memory size for processing on mac book pro(with 4gb ram) and only 1.5gb on windows server with 16gb ram..


...

t
Re: memory increase
Reply #5 - Jan 9th, 2010, 6:43am
 
You running 64bit Java?  It should work... Processing maintains it's memory setting in the preferences, by checking "Increase maximum available memory", which for windows just compiles essentially a batch file launching the jar file using the "-Xmx2000m", so you could play around with manually launching it to see if it changes anything.
Re: memory increase
Reply #6 - Feb 24th, 2010, 12:17am
 
now, i am trying to work with red hat server edition, and first there are some issues with gluegen that doesn't allow us to compile opengl, so i must use p5sunflow to render. p3D compiles, but gives me some mistakes - kind of random...

the sad thing is, i have access to 64 processors super computer unit, 4.6ghz each, with huge amount of ram memory(technician mentioend 240gb). and when i run a sketch, it takes 10 minutes to get one frame. on macbook pro it is around 50% slower..

processing uses, from 3-8 processors, and it cannot use more. memory usage is around 0.4 % of the total 240gb...

i increased my memory to close to 10gb in processing preferences. it compiles the sketches, but still, it doesn't give me speed.. and it doesn't let me compile with opengl..

does anyone know if i move this thread to implementation section, people might be able to help? (and how can i move it?)

thanks!

t


i am not sure if there is any version that can utilize cloud computing better...
Re: memory increase
Reply #7 - Feb 24th, 2010, 1:48am
 
I would move the topic to Integration, instead... Smiley But I doubt you will find a better audience.
The topic is interesting. Your config is quite rare, but multi-core processors become common and 64bit systems too.
I suppose Java is able to split tasks between processors automatically (is it). Explicit parallel processing is hard to do, lot of people study that but the topic is far from being extensively known...
That's why I think that moving the thread to Processing implementation wouldn't be useful, as I doubt the authors would spend time optimizing multi-processor work (unless they have personal interest on the topic!).

Note: as I wrote above, perhaps you can optimize the code yourself, spawning multiple threads dividing the rendering task, I think Java will spawn the threads on multiple processors.
Somehow, Processing is a "toy", optimizing rendering like you want would need a good level of Java knowledge...

I am not a specialist of the topic, alas. I know there are researches on the field (Java, multi-threading, multi-processors, clouds...), see for example Terracotta.
Re: memory increase
Reply #8 - Feb 24th, 2010, 3:41am
 
Hi PhiLho,

thanks for the answer... yes, it is an interesting topic, but with my limited knowledge of the topic and time, i think it would be difficult to use the full power of cloud computing for now... maybe in future..

yes java, is splitting the jobs... on one node i am getting 8 processors,(win server) and on linux close to that...

if i get any results, i will update this post...

tx
t
Re: memory increase
Reply #9 - Feb 24th, 2010, 5:48am
 
I have a friend that was having memory limitation issues on Java.  He switched to Oracles JRockit and it resolved them.  Not sure this is relevant but I thought I would mention it.
Re: memory increase
Reply #10 - Feb 26th, 2010, 3:44am
 
thanks jeffg,

again, issues that might be difficult to resolve, since i don't have admin privileges on the machine..

when we were running it on linux server, i remember running it with close to 10gb ram memory allocation.

now i switched to windows that has two java's installed 1.6 and 1.4... it still reports that it cannot run a sketch with more than 2gb.

is there anyway to force processing to use one of these java versions?

this system has 48gb ram, but i can only use 1.5gb

thanks.

t
Re: memory increase
Reply #11 - Feb 26th, 2010, 5:23am
 
another maybe minor thing...

since i noticed that my opengl sketches didn't act nicely and were giving me 1/3rd of an image(in the form of a centered rectangle(all the rest was a black frame)), i found out that there is no opengl hardware acceleration if you run 32bit applications on 64bit platform for windows.
this is because in 64bit windows there is only an opengl emulator...

http://support.microsoft.com/kb/835563

so, i can use P3D or p5sunflow instead. on linux server i didn't have problems with opengl.

anyone running opengl on a windows 64bit machine?

best,
t

Re: memory increase
Reply #12 - Feb 26th, 2010, 5:44am
 
The windows .exe for the processing sketch is essentially a batch file that calls the jar.  So you can create your own batch file which calls the jar and specifies the memory and the specific JRE exe.  Also, you should not need admin rights.  While you can install java on the machine, it is portable and requires no installation.  To make my sketch portable, I just copy the JRE folder into my sketch and then directly execute that JRE against my JAR.

Something like "./jre/javaw.exe -Xmx2000m -jar ../lib/sketch.jar"

I'm pretty sure you will need 64bit java and 64bit Windows to use more than 2GB of memory.
Re: memory increase
Reply #13 - Mar 17th, 2010, 4:04am
 
jeffg wrote on Feb 24th, 2010, 5:48am:
I have a friend that was having memory limitation issues on Java.  He switched to Oracles JRockit and it resolved them.  Not sure this is relevant but I thought I would mention it.

I tried JRockit last week and hated it.
Re: memory increase
Reply #14 - Mar 28th, 2010, 9:17pm
 
wow! im a newbie here. I experienced the same thing a couple of months ago, I tried switching from oracle as well. If only I had entered this thread earlier, I would have known better.
Pages: 1 2