apply shader on PImage directly

clankill3r

Is something like this possible?

testImg = loadImage("someImage.jpg");
testImg.filter(shader);

The reason I ask is cause getting the color of an image directly is insane fast

int c = img.get(mouseX, mouseY);

Like 500 frames a second. But if I draw to the main canvas first (g) then it's really slow. Like:

int c =get(mouseX, mouseY);

benja

Not sure if there is better solution, but you could draw the source image, apply a filter and then copy the result to a new PImage.

src = loadImage("example.jpg");
image(src, 0, 0);
filter(shader);
dest = get(0,0, src.width, src.height);

clankill3r

I'm looking for something faster. The code above will create a new PImage every time which is slow. I'm hoping for something that doesn't leave GPU.

Poersch

Well, get(int, int) just returns the color value saved in the pixels array. For PImage.get() this is pretty fast, because pixels is always uptodate. For PGraphics.get() (actually only the OpenGL based PGraphics) this is reeaally slow, because every time get() is called, PGraphics will attempt to flush (render every buffered vertex) and copy the whole image from the VRAM texture object to the backing pixels array in your RAM.

I'm looking for something faster. The code above will create a new PImage every time which is slow. I'm hoping for something that doesn't leave GPU.

It has to leave the GPU or you won't be able to interact with it in Processing anymore. So that's not possible, but if you only need to get the color value for one or a handful of pixels you could try to request the data directly from your graphics card:

import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import java.nio.ByteOrder;

PGL pgl;
IntBuffer colorBuffer;
PImage someImage;

void setup() {
    size(600, 600, P2D);
    noStroke();
    someImage = loadImage("http" + "://forum.processing.org/two/uploads/userpics/970/n4IUAUUTANWJL.png");

    // Get PGL reference once (no need to do this every frame)
    pgl = g.beginPGL();
    g.endPGL();

    // Allocate direct byte buffer as int buffer (size = 4 byte = 1 int),
    // needed to communicate with PGL (OpenGL)
    colorBuffer = ByteBuffer.allocateDirect(4).order(ByteOrder.nativeOrder()).asIntBuffer();

}

void draw() {
    image(someImage, 0, 0, width, height);
    fill(getColor(mouseX, mouseY));
    rect(0, 0, 80, 80);
}

// Rewing int buffer so it can be used and request pixel data
// from PGL (OpenGL - your graphics card memory directly)
int getColor(int x, int y) {
    colorBuffer.rewind();
    pgl.readPixels(x, height - y - 1, 1, 1, PGL.RGBA, PGL.UNSIGNED_BYTE, colorBuffer);
    return colorBuffer.get();
}

Anyways, applying a PShader to an PImage directly isn't possible - in Processing. While it is possible in OpenGL, it's more complicated than you might think: You'd have to have two texture objects, one as input (sampler2D) and one as output (Frame Buffer Object) and use ping pong rendering (from A to B, the next frame B to A, etc.) to be efficient. Plus, using a texture as render target can be more slowly than using a default render buffer (depending on the hardware).

clankill3r

Thanks that might help. I need to jump a lot. Like jump 63 pixels forward, and again and again... So it might help. After that I will dive into CUDA.

Poersch

No problem. :)

clankill3r

Is there a way to get the integers like blendMode(REPLACE) was used. I'm getting different values at the moment. Which is most likely due the alpha channel adjusting the RGB values again.

Poersch

Blend mode works like it should, but Processing returns the color value in BGRA instead of the chosen RGBA... :/

The following code should fix the problem:

int getColor(int x, int y) {
    colorBuffer.rewind();
    pgl.readPixels(x, height - y - 1, 1, 1, PGL.RGBA, PGL.UNSIGNED_BYTE, colorBuffer);
    int c = colorBuffer.get();
    return (c & 0xff) << 16 | c & 0xff00 | (c & 0xff0000) >> 16 | c & 0xff000000;
}

clankill3r

Once again thanks :)

clankill3r

I looked into it. It's really slow! Even for 1 pixel my frameRate goes from around 400 to 100. If i check like 60 pixels my frameRate is around 20. Damn...

Do you know any other way apart from CUDA?

Poersch

Yeah because calling the graphics card directly is always slow... I'm currently at work, but it should be possible to update your canvas' pixels array just once per frame and then get the color values from the CPU sided pixels array instead of requesting it from the GPU again and again.

Poersch

The following code will update the pixels array only once a frame - only one graphics card call, this is as fast as it gets (with Processing):

import com.jogamp.opengl.GL2GL3;
import java.nio.IntBuffer;

PGL pgl;
IntBuffer pixelBuffer;
PImage someImage;

void setup() {
    size(600, 600, P2D);
    noStroke();
    someImage = loadImage("http" + "://forum.processing.org/two/uploads/userpics/970/n4IUAUUTANWJL.png");

    // Get PGL reference and call the modified loadPixels once
    pgl = g.beginPGL();
    g.endPGL();
    loadPixels();

}

void draw() {

    image(someImage, 0, 0, width, height);

    // Update pixels array once a frame
    updatePixels();

    // Get the color like usual
    fill(get(mouseX, mouseY));
    rect(0, 0, 80, 80);

}

void loadPixels() {
    pixelBuffer = IntBuffer.wrap(pixels = new int[width * height]);
}

void updatePixels() {
    pixelBuffer.rewind();
    pgl.readPixels(0, 0, width, height, GL2GL3.GL_BGRA, PGL.UNSIGNED_BYTE, pixelBuffer);
}

// The "strange" index calculation is a result from having to flip the Y, but not
// the X axis. Just create and render an image with different colored corners to
// test if the code is returning the correct index into the flipped pixels array.
int get(int x, int y) {
    return pixels[(height - 1 - y) * width + x];
}

I modified Processing's loadPixels() and updatePixels() as they seem to be broken (in 3.0 at least). I also modified get() to use the pixels array and removed all unnecessary overhead (like calling loadPixels() again and again and needless bounding checks).

Btw.: I know this is still slow, updatePixels() lowers my framerate from 1000 (can't set a higher rate) to around 330, but the following get() calls are basically for free.

clankill3r

Nice! This is really cool! I tested and this is faster. Now I have a framerate of 60 again. And there is a loot of room for optimization in my shaders, but I will do that as last.

clankill3r

I think this is bugged:

int get(int x, int y) {
    return pixels[(height - 1 - y) * width + x];
}

Let's say I have an image of 800*800. That will be an array of 640.000 pixels.

If I get the coordinate at 0,0:

int x = 0;
int y = 0;

int w = 800;
int h = 800;

println((h - 1 - y) * w + x);

Then it prints 639200, so where 800 short!

Edit #3 This seems to be ok:

int x = 799;
int y = 799;

int w = 800;
int h = 800;

int l = w*h;

int index = l - 1 - (y * w + x);
println(index);

x = 0;
y = 0;

index = l - 1 - (y * w + x);
println(index);

clankill3r

In a shader, if I want to check one pixel higher do I use:

+texOffset.y

or

 -texOffset.y

?

clankill3r

Correction:

I think this is correct:

    int index = pg.pixels.length - 1 - (y * pg.width + (pg.width - x));

Poersch

Nope, it's correct. If width & height equal 800, get(799, 799) will return the color from the index 799. get(799, 0) will return the color from the index 0. That strange behavior is caused by the flipped Y axis of the OpenGL texture.

Poersch

Just use the following image to test the code. In the top-right corner is a green pixel, in the bottom-right a red one and in the bottom-left corner a blue pixel. You wouldn't even be able to "hit" them with the mouse if the index calculation would be off by 1 pixel or more:

And if you don't believe your mouse:

• get(599, 0) will return green
• get(599, 599) will return red
• get(0, 599) will return blue

clankill3r

Making a test like that was the first thing I wanted to do this morning :)

Just to be sure, this is correct right?

int tx = index % width;
int ty = height - 1 - ((index - tx) / width);

(If it's not I have a bug somewhere else :) )

Poersch

Jup, but you don't really need to subtract tx from index, dividing it by width will floor the value anyways (in this case (index - tx) / width == index / width). Oh, and try to avoid using modulo, it's a pretty slow operator. This should be a little faster:

ty = height - 1 - index / width;
tx = index - ty * width;

clankill3r

Your tx calculation makes no sense but I can't figure out why apart from that it's pretty late :) .

int h = 32;
int w = 32;

for (int i = 0; i < w*h; i++) {
  int index = i;
  int ty = h - 1 - index / w;
  int tx = index - ty * w;
  println(tx, ty);
}

Poersch

Yeah, I guess it was pretty late for me too! :D

My code just worked for the extreme values (0,0 / 599,599 / etc.) your tx calculation is right. This should do the trick:

void setup() {
    size(640, 480, P2D);
}

void draw() {

    int index = (height - mouseY - 1) * width + mouseX;

    int tx = index / width; // This way we only devide once
    int ty = height - 1 - tx;
    tx = index - tx * width; // index % width == index - (tx / width) * width

    background(0);
    fill(255);
    text("X " + tx, 10, height - 22);
    text("Y " + ty, 10, height - 10);

}

clankill3r

Cool. For now I got rid of getting the pixels with pgl. What i'm doing is already complex and with the reversed array it made thing so much more complex. And reversing the array results in a nasty crash.