Check out: GLGraphics Examples > Textures > DynamicMask

Yes I've used that one as a base. Or is there a difference in drawing on offscreen and a JAVA2D surface? I've used the JAVA2D part of the example.

Sure you can use a JAVA2D-based PGraphics (or any other PImage/PGraphics for that matter).

It is important to know what how each mask works. I think the most crucial thing to realise here is that the Processing mask function uses the blue color channel while the GLGraphics mask filter uses the alpha channel.

Using a greyscale image in Processing would thus work in the sense that it will only look at the blue color channel (which is in a greyscale image the same as red and green anyway) and use that as a mask plus replace the alpha channel of the final image by the alpha channel of the mask. Also see this thread. When you use the mask filter in GLGraphics the color channels (R, G and B) do not matter, they are not used. It only looks at the alpha channel to mask.

To prove the above point via the code below, I have changed the background from black (in the original) to white. For Processing masking this would have a huge impact. In GLGraphics this does not affect masking at all, since it is not the color channels, but the alpha channel that is used. To solve 1 pixel uglyness I have added noStroke() and smooth() to the example (you can see the difference of adding these better without the blur filter). In line with your code I have added the super-slow blur-x12 filter, since all of this is just to explain a point, so right now it's not about framerate. The code that is affecting the mask directly is in line 27 where the transparency of the ellipse changes and thus the alpha channel of the image.

There are a lot of options with regard to PImage, PGraphics, GLTexture and GLGraphicsOffscreen. Basically all could be used. Also, blur is probably the slowest thing in whole of Processing ;-) So either search for fast blur in the forum or indeed rely on the blur shaders from GLGraphics.

Code Example (uses the resources from the GLGraphics DynamicMask example)

1. import processing.opengl.*;
2. import codeanticode.glgraphics.*;
3. 
   
4. GLTexture imgTex;
5. GLTexture maskTex;
6. GLTexture maskedTex;
7. GLTextureFilter maskFilter;
8. 
   
9. PGraphics pg;
10. 
    
11. void setup() {
12.   size(256, 256, GLConstants.GLGRAPHICS);
13.  
14.   imgTex = new GLTexture(this, "beach.jpg");
15.   maskTex = new GLTexture(this, 256, 256);   
16.   maskedTex = new GLTexture(this, 256, 256);
17.   maskFilter = new GLTextureFilter(this, "Mask.xml");
18.  
19.   pg = createGraphics(256, 256, JAVA2D);
20. }
21. 
    
22. void draw() {
23.   pg.beginDraw();
24.   pg.noStroke();
25.   pg.smooth();
26.   pg.background(255, 0);
27.   pg.fill(255, (frameCount*10) % 255);
28.   pg.ellipse(mouseX, mouseY, 150, 150);
29.   pg.endDraw();
30.   pg.filter(BLUR, 12); // super slow
31.   maskTex.putImage(pg); 
32.  
33.   maskFilter.setParameterValue("mask_factor", 0.0f);
34.   maskFilter.apply(new GLTexture[]{imgTex, maskTex}, maskedTex);
35. 
    
36.   background(0);
37.   stroke(255);
38.   for (int x = 0; x < width; x += 10) line(x, 0, x, height);
39.   for (int y = 0; y < width; y += 10) line(0, y, width, y);
40. 
    
41.   image(maskedTex, 0, 0);
42. }

Thank you for the info and example Stupid that I didn't think of that. Just expected that it would work with a black/white image and didn't think of alpha.

Great. Thank you, that's exactly what I needed.

Below the result with this masking applied. I still keep seeing a hairline if I draw 2 masked images on top of each other. If I draw just one triangle at a time on the yellow background I don't see it. 

this might be a bit late, but check the background of your original image.

if you set it to (0,0) and have antialiazing on, the edges will be calculated to fade into black.
you get me?
so better turn antializing off and slightly blur it afterwards.

How do I turn anti-aliasing of? Is that the smooth() function in Processing? Or is this an OpenGL command?