Yes, Java has memory settings that can be set on the command line.
When I export for Windows, for example, (using the PDE here) I get an application.windows folder, with a lib folder in it. Inside, you will find an args.txt file. The first line, looking like -Xms64m -Xmx1024m, are these memory settings.
The ms value is the initial memory allocated at the start of the program, here 64MB. The mx one is the maximum memory that will be used by Java, here 1GB, which is the setting I have set in the PDE preferences.

You might also want to investigate why the memory usage is increasing so much...

thanks a lot! I will check the memory settings. It is strange as I left it to run on my computer for 24 hour, compiled or from the editor and there are no issues with it. Problems only happen with the kiosk computer with the compiled exe, which is actually quite old.

" but is there any way to run a sketch with PDE via a single command line?"
I don't understand the question. Is that PDE in the sense of the Processing environment, or do you mean .pde?

Anyway, you can run an exported sketch named SketchFoo, for example, by going where the jars are, and using the command line java -cp core.jar;SketchFoo.jar SketchFoo (include other jars, like libraries, if there are.
You add the command line switches you need, of course. But the SketchFoo.exe runner does the same thing, using the indicated file, with the given parameters.

I had similar issues with an installation piece, where it would run great from the ide, but the exported exe would have a number of issues; memory, framerate, sometimes even crashing the jvm. phi.lho has it right with the command line option, and the memory setting of -Xms and -Xmx. This was the only solution that worked in the end for me.

Thanks phi.lho and akiersky! I meant PDE in the sense of the Processing environment, as in the ide. Now I get a sense of what needs to be done. Will report when there is good news, thanks!