Sorry this post is long.  Please skip to the bottom if you prefer the short version.

I am packaging a Processing-based release of the arduino.cc software, which I'm targeting for Gentoo Linux, primarily for intent of my own use on that platform but also to contribute to the community and increase the number of quality available packages in that distro.

In source-based distros like Gentoo linux, all dependencies must be available in source form, to be compiled at install time on the user's computer by the package management system (portage).

In my packaging effort, I notice that there are a number of binary dependencies that are being distributed with the Processing software, in various formats such as jar or so or dll, without accompanying source or an authoritative upstream reference.

There may be others, but the ones I've encountered include:

• librxtxSerial.so and RXTXcomm.jar (found at the rxtx project at http://rxtx.org/, LGPL)
• antlr.jar (found at http://www.antlr2.org/, ANTLR license)
• oro.jar (found at the Apache foundation, Apache license)
• mrj.jar (the Apple Mac Runtime for Java, apparently no longer supported, license unknown, authoritative site unknown)

The only one truly holding me up right now is MRJ.  I've been able to reference stable upstream versions of all of the other dependencies (those affecting me so far, at least) in source form.  But for MRJ, I am having a bear of a time finding sources to build it.

The Gentoo maintainers don't accept ebuilds that provide only binaries for dependencies, so I have no choice here if I am to meet my goal of serving the community "properly".  Anyone who currently installs Processing-based software on Gentoo must do so using unsupported install scripts or manual steps that install binary librares not in the package management system, and it is my goal to correct that.  Be assured, my results will shared here as well as with Gentoo.  As I mentioned, I've accounted for quite a few so far with the exception of MRJ.  I hope I don't encounter others, but if I do, I'll address them in turn.

So, I ask, where is the authoritative source for the MRJ package, or at least java source files necessary to build it?  My googling skills are good but I haven't found it yet.  Apple themselves seem to have given up on it, in favor of what I have not been able to determine.

I would strongly encourage the Processing maintainers to publish all dependency sources, in SVN at the very least, for the convenience of many users as well as to retain the ability to fix upstream problems without the delays of their involvement.  But naturally I have no basis to demand it (and probably wouldn't even if I did).

Really, I just need the MRJ source java files (or a pointer thereto) most urgently.  I think I've found the rest.  Can someone please advise from whence they come.

Best regards,
IWTGTT

The MRJ dependencies are pervasive in the source and are invoked even when not on a Mac platform.  That was my first attempt, and it turns out that the entire package really does depend on it, even when running on other platforms.  Why it wasn't built on straight Swing, I can only speculate to either be because of "pretty" or because someone was used to MRJ.

The jad-type kludge is a possibility if necessary.  I hope not, though.

It's all good if someone knows where the MRJ sources can be found.

sbrudenell wrote on Mar 7^th, 2009, 9:06pm:

There are a couple of bugs at bugs.gentoo.org but they're stale.  I'm going to propose a new ebuild when filing mine.

The ebuild I'm currently working on is arduino-specific, but my work may be of help to other Processing users on Gentoo as well.  There is an arduino-0011 ebuild marked unstable, and it's completely broken.  There's no ebuild for Processing itself currently in portage.  As you know, the Gentoo herd strongly opposes ebuilds that reference binaries without accompanying source (and I strongly agree with that philosophy).  This is what I'm currently reconciling.

I've put the IDE jar issues aside for the moment while I resolve some command-line and system library dependency issues.  When I get back to the jar issues, I may well decide to split off the Processing package into its own ebuild, which I suspect might help others here like yourself.

Progress is continuing, I'll keep you posted.

Thanks, I will certainly do that.

I don't know (until I dig a little more) just how much the arduino folks have forked the Processing core for their use, but removing a dependency for which there are no available sources is a Good Thing (tm) in my book.

How handy that you were able to chat him up about it!

So I finally got a chance to look over the processing build system. It looks like it should be fairly simple to convert (almost) everything from their existing make.sh script to ant, with a grey-area exception being the perlified synchronization of the PGraphics methods to PApplet.

However, there are other obstacles ...

Looking at processing in depth more, it uses third-party libraries for pretty much everything: rxtx for serial communication, dxf for dxf files, jogl for opengl, log4j for logging, minim for audio i/o, itext for pdf interaction, etc. All of these projects are provided in prewrapped jars in the processing source, and the expectation is that they'll live in the processing directory at both compile time and runtime.

This isn't a problem by itself, as it's relatively straightforward to rejigger the build system and runtime to depend on system jars rather than ones packaged with processing ... though it is going to take a long time to get all these projects into portage; I think this needs to be done for jogl, minim and dxf at least.

The BIG problem is:

When exporting a sketch, processing packages up all these third-party packages and depends on their static location to do so. This is to ensure that someone running a processing sketch need only have the JRE installed, and nothing else, and the sketch should run. This is a pretty important feature of processing, philosophically ... but it's diametrically opposed to Gentoo's philosophy of never including static libraries with anything.

This is more complicated than just making the "export sketch" code aware of dynamic jar paths; currently, the only way exporting works is because a static list of special magic auxiliary files (the export.txt files) are included with every third-party library -- this includes stuff like dlls and .so files which get loaded via JNI. It's all very brittle and static today.

As I see it, a "real" solution to the "export problem" would be to figure out all the classes loaded by a sketch, resolve them to jars via the current ClassLoader somehow, figure out whether any classes in the dependency tree need any JNI libraries, somehow (this would be the big mystery) figure out in a dynamic fashion what JNI libraries would be needed for the sketch to run on other platforms, mask out stuff included in the JRE, and include all this with the exported sketch. I know all this is at least possible (except the cross-OS support stuff) ... the internal build system at Amazon (where I used to work) did something along these lines and it worked quite well.

Obviously all this is incredibly ambitious and long-term. In the short-term, since Gentoo is obviously not going to let any static anything occur in any package and the export functionality is incredibly brittle without the assumption that everything is static, I think the best thing to do is just disable exporting "the Gentoo version" of processing for the time being. This sucks, but it at least enables progress.

For the time being, I'm going to focus on getting the dependencies into Gentoo and getting processing building with ant.