This has got me to a point where I can have a transparent window with no decoration, thanks to
frame.removeNotify();
frame.setUndecorated(true);
and
AWTUtilities.setWindowOpaque(frame, false);
and I can then set all the pixels in the application to be transparent in draw() with
loadPixels();
for ( int i = 0 ; i < pixels.length ; i++ ) pixels[i] = 0;
updatePixels();
But anything else I try to draw after this, say a rectangle that follows the mouse, leaves a trail, even though I reset all the pixels to "0" in the draw() loop before drawing the rectangle each time.