Look how Processing app. runs the sketches: that's a Java application running applets... Although the analogy is half good only, as it runs .class files after the compilation phase, not the .jars.

But basically, I think it just spawns a separate JVM instance with the right set of parameters.
You probably should look at ProcessBuilder, there are some good tutorials on its usage, and even how to get its output.