Performance of Filter()

joySeeing

I was thinking of better ways to make the previous frame's image disppear rather than starting each off with a blank by doing background(0); or something. I slapped a filter(BLUR); at the end to make each previous frame diminish smoothly but it literally brought my fps from 50-60 down to under 5... just that one line. Is that supposed to be right?

I'm using Processing 2.2.1 with P3D and fullscreen. What techniques do you use to do the above?

clankill3r

You should use a shader for the blur. By default it goes on the CPU. Look at examples > topics> shaders > sepBlur

That should help a lot.

Poersch

clankill3r is right, BLUR is just an int variable "pointing" to a CPU based filter. Use a hardware accelerated PShader instead.

In case of gaussian blurring I'd recommend to use a two pass approach. This way you reduce the quadratic sample kernel (9 samples) to two linear sample kernels (2 * 3 samples) while getting the same effect:

PShader blurH, blurV;

void setup() {
    size(600, 600, P2D);
    initBlurH();
    initBlurV();
}

void draw() {
    rect(mouseX - 20, mouseY - 20, 40, 40);
    filter(blurH);
    filter(blurV);
}

void initBlurH() {
    String[] vertSource = {
        "uniform mat4 transform;",

        "attribute vec4 vertex;",
        "attribute vec2 texCoord;",

        "varying vec2 vertTexCoord;",

        "void main() {",
            "vertTexCoord = texCoord;",
            "gl_Position = transform * vertex;",
        "}"
    };
    String[] fragSource = {
        "uniform sampler2D texture;",
        "uniform vec2 texOffset;",

        "varying vec2 vertTexCoord;",

        "void main() {",
            "gl_FragColor  = 0.27901 * texture2D(texture, vec2(vertTexCoord.x - texOffset.x, vertTexCoord.y));",
            "gl_FragColor += 0.44198 * texture2D(texture, vertTexCoord);",
            "gl_FragColor += 0.27901 * texture2D(texture, vec2(vertTexCoord.x + texOffset.x, vertTexCoord.y));",
        "}"
    };
    blurH = new PShader(this, vertSource, fragSource);
}

void initBlurV() {
    String[] vertSource = {
        "uniform mat4 transform;",

        "attribute vec4 vertex;",
        "attribute vec2 texCoord;",

        "varying vec2 vertTexCoord;",

        "void main() {",
            "vertTexCoord = texCoord;",
            "gl_Position = transform * vertex;",
        "}"
    };
    String[] fragSource = {
        "uniform sampler2D texture;",
        "uniform vec2 texOffset;",

        "varying vec2 vertTexCoord;",

        "void main() {",
            "gl_FragColor  = 0.27901 * texture2D(texture, vec2(vertTexCoord.x, vertTexCoord.y - texOffset.y));",
            "gl_FragColor += 0.44198 * texture2D(texture, vertTexCoord);",
            "gl_FragColor += 0.27901 * texture2D(texture, vec2(vertTexCoord.x, vertTexCoord.y + texOffset.y));",
        "}"
    };
    blurV = new PShader(this, vertSource, fragSource);
}

In case of box blurring I'd stick to a single pass approach that samples between the actual pixels. This practically quadruples the actual sample count and almost creates the same visual effect as the gaussian blur:

PShader blur;

void setup() {
    size(600, 600, P2D);
    initBlur();
}

void draw() {
    rect(mouseX - 20, mouseY - 20, 40, 40);
    filter(blur);
}

void initBlur() {
    String[] vertSource = {
        "uniform mat4 transform;",

        "attribute vec4 vertex;",
        "attribute vec2 texCoord;",

        "varying vec2 vertTexCoord;",

        "void main() {",
            "vertTexCoord = vec2(texCoord.x, 1.0 - texCoord.y);",
            "gl_Position = transform * vertex;",
        "}"
    };
    String[] fragSource = {
        "uniform sampler2D texture;",
        "uniform vec2 texOffset;",

        "varying vec2 vertTexCoord;",

        "void main() {",
            "gl_FragColor  = texture2D(texture, vertTexCoord - 0.5 * texOffset);",
            "gl_FragColor += texture2D(texture, vertTexCoord - 0.5 * vec2(texOffset.x, -texOffset.y));",
            "gl_FragColor += texture2D(texture, vertTexCoord + 0.5 * vec2(texOffset.x, -texOffset.y));",
            "gl_FragColor += texture2D(texture, vertTexCoord + 0.5 * texOffset);",
            "gl_FragColor *= 0.25;",
        "}"
    };
    blur = new PShader(this, vertSource, fragSource);
}

While the gaussian blur is in general more "pleasing" to the eye, the optimized box blur is much faster and creates almost the same visual effect. I'd stick to the faster box blur.

Edit: Changed the first line in the filter's main() method to gl_FragColor = something in case the graphics card driver isn't initializing the value correctly.

clankill3r

Great, I will try the box blur out :)

joySeeing

Thanks for your answers. But as someone who doesn't know much about filtering... How does one apply these examples to a sketch which which blurs the canvas? PImage theCanvas = get()

Poersch

@clankill3r: Nice! Hope it works on your Mac. ;)

@joySeeing: Just like in my code snippet, load the PShaders in setup() or some dedicated method called by setup(), render your scene in draw() and then call filter():

// ...
PShader myFilter; // Define some global var for the filter

void setup() {
    // ...
    myFilter = new PShader(/* ... */); // Create/load the filter
}

void draw() {
    // ...
    filter(myFilter); // Use the filter on your main canvas
}

clankill3r

@Poersch

About the box blur. Why the 0.5? Why is it not 1. So you look at the next pixel. Now you look at half a pixel away. Is it so you can use the filter in such a little amount that it is hardly visible?

Also changing the value to 4 for example and run the filter multiple times looks nice :)

clankill3r

@Poersch

I did update it for my mac :) Using in and out and texture instead of texture2D in case your interested in writing my Mac compatible shaders :)

Poersch

@clankill3r:

In case of box blurring I'd stick to a single pass approach that samples between the actual pixels. This practically quadruples the actual sample count and almost creates the same visual effect as the gaussian blur

The reason why I use "half offsets" is because it will force the graphics card to interpolate between 4 pixel values instead of using 1 (actually it always interpolates a little because of the low UV precision, so the performance is lost anyways). That means my second blur approach practically samples 16 pixels instead of just 4. Plus, using offsets equal to or greater than 1 will cause artifacts (at least if you use the filter in an accumulative manner) as you are not sampling your current pixel anymore. While this can look good and be exactly what you are looking for, it's not really a blur anymore.

0.5 pixel offset:

1.0 pixel offset:

In my real life projects I always depend on at least GLSL version 1.30 (in/out/texture/etc.), but on the forums I always try to use GLSL version 1.10 because it is (by specification) supported by any OpenGL hardware - don't know why apple seems to ignore the official specs though.

clankill3r

Ah cool that is done really smart :)

And if I learned one thing over the years, big companies != quality.

joySeeing

@clankill3r

How do you re-write his examples for Mac exactly? That's why I was confused... was just getting video noise...

clankill3r

I only did the box blur.

PShader box_blur;
PImage img;

void settings() {
  size(512, 512, P2D);
}

void setup() {
  box_blur = loadShader("box_blur.frag", "box_blur.vert");
  frameRate(999);
  img = loadImage("http://bellard.org/bpg/lena30.jpg");
}


void draw() {
  background(255);

  fill(255, 0, 0);
  ellipse(width/2, height/2, width, height);

  image(img, 0, 0);

  int n = (int) map(mouseX, 0, width, 0, 20);
  for (int i = 0; i < n; i++) {
    filter(box_blur);
  }


  surface.setTitle(""+frameRate);
}

vert:

#version 410

uniform mat4 transform;

in vec4 vertex;
in vec2 texCoord;

out vec2 vertTexCoord;

void main() {
    vertTexCoord = vec2(texCoord.x, 1.0 - texCoord.y);
    gl_Position = transform * vertex;
}

frag:

#version 410

uniform sampler2D texture;
uniform vec2 texOffset;

in vec2 vertTexCoord;

out vec4 fragColor;

void main() {
    fragColor += texture(texture, vertTexCoord - 0.5 * texOffset);
    fragColor += texture(texture, vertTexCoord - 0.5 * vec2(texOffset.x, -texOffset.y)); 
    fragColor += texture(texture, vertTexCoord + 0.5 * vec2(texOffset.x, -texOffset.y));
    fragColor += texture(texture, vertTexCoord + 0.5 * texOffset);
    fragColor *= 0.25;
}

joySeeing

@clankill3r

With a standard 2015 Macbook Pro with Intel Iris graphics card, I get Cannot compile vertex shader: ERROR: 0:1: '' : version '410' is not supported

Thanks for your help.

Poersch

Video noise..? Could you post your Code?

joySeeing

@Poersch

Whenever I copy your code directly to my Macbook Pro Processing 2.2.1 I get this:

Using @clankill3r 's Mac adjusted code gives me a different error.

clankill3r

Poersch had some code to find what GLSL version your drivers supported but I can't find it back. That might help. I have set it for 410 in my case since that's what I have and I'm not sure if the shaders I port work < 410.

joySeeing

Ok, well that's fine. Finally, one thing missing that blur.glsl doesn't do that BLUR does is make things eventually fade into nothingness (or the illusion of this anyawys). Any clever recommendations to make that happen?

clankill3r

Just to make clear: By fading into nothingness, you meant that each frame the previous blur result got blurred again right?

joySeeing

Probably. Such that (simple example), if you drew a picture of a square and did not draw it again on the next frame, it would become progressively harder to see the square outline as the blurring would continue.

Poersch

@joySeeing: Looking at your screenshot it seems like your graphics card driver isn't initializing gl_FragColor with vec4(0.0). Updated my examples, please try them again.

clankill3r

@Poersch 4-o-clock?

gl_FragCoord or did the update not get threw?

Poersch

@clankill3r: Argh! These typos! :D (changed it)

joySeeing

@Poersch

Thanks. And actually. I was playing around with the numbers and if I moved .5 (on second example) up to .99, everything begins to fade away quickly. Perfect!

Thanks

joySeeing

Actually, I don't really know what I'm talking about. Not sure what I did to make it do this.. Hmm...

shawnlau

The filter blur is very slow. Do your own one direction at a time with a "window" of pixels. That way the the speed of 3 pixel blur is about the same as a 256 pixel blur. With the window of pixels, after its filled its just one on and one off and the sum in the middle remains constant. So instead of 256 operations on a 256 pixel blur, you're doing 3. The 256 pixel blur I do on a 1000 x 1000 image takes 34 ms.

Poersch

@joySeeing: Yeah, that will modify the blur weighting but the image will never reach the average color (because of rounding errors) you'll need a bigger blur kernel for that. Try the 2 pass gaussian blur with a far biffer blur kernel.

@shawnlau: Yeah, but a shader based blur is pretty fast. Shouldn't take more than a few thousand nanoseconds.