Chaining Shaders and Feedback

toneburst

Hi,

I'm now here, so let me introduce myself.

I'm Alex, from sunny London, in the UK.

I'm new to Processing, but have some programming experience, in various languages. I've also used GLSL shaders quite a lot in Quartz Composer, but the particular job I'm working on at the moment seems difficult to achieve in QC, so I'm thinking of giving Processing a try, instead, which brings me to my questions:

1. Is it possible to chain several 2D fragment shaders together, so that the result of one shader is passed into the next shader as a texture? My particular scenario involves a further complication:

2. Is it possible to chain 2 shaders as above, and have the second shader feed back into the first, and also go through a third shader, which renders the result to the screen? I've attempted to illustrate what I have in mind below.

|| [shader 0] > [shader 1] || > [shader 2] > [screen] || || === < [prev. frame] < ===

I'm aware this is kinda throwing myself in at the deep end, but any advice would be much appreciated.

Thanks guys,

a|x

toneburst

I think this might help with the chaining, perhaps. https://github.com/cansik/processing-postfx I'm not sure if it would help with the feedback part, though.

a|x

nabr

Hello

i just skip trough some pictures in QuartComposer seems more Node based (Visual Programing). If you enter the world of Processing, you have to write code. With a custom request, their is not way around, than read one or the other topic on how computer graphics work.

Processing is driven by JOGL

1. you can ask @cansik if he can spend 5minutes more to add more functionality to his awesome postfx lib.

2. Search in this Forum.

3. Just hack, explore, their is always more then way to do something (one implemention is just faster, other qualiltes then the other)

@Lord_of_the_Galaxy hava a nice solution to write the previous frame to texture

I'm not realy sure, but you can read it between the lines, some computation is already set up in the background, when you start https://github.com/processing/processing/wiki/Advanced-OpenGL#vertex-coordinates-are-in-model-space

Can't find in the source code right now, but i think, they render the Framebuffer to a texture per default.

Here is my attempt to write and store data without access to a low level api

String[] vertSource={"#version 150"
  , "in vec2 position;"
  , "out vec2 uv;"
  , "void main() {"
  , "gl_Position = vec4(position,0.,1.);"
  , "uv=position;"
  , "}"
};
String[] firstPass={"#version 150"
  , "in vec2 uv;"
  , "uniform float t;"
  , "out vec4 fragColor;"
  , "void main() {"
  //a line with animation
  , "fragColor = vec4(vec3(.2/length(abs(uv.x)*sin(t))),1.);"
  , "}"
};
String[] secondPass={"#version 150"
  , "in vec2 uv;"
  , "uniform sampler2D s;"
  , "out vec4 fragColor;"
  , "void main() {"
  //just a texture
  , "fragColor = texture(s,uv-vec2(.04));"
  , "}"
};
PShader shdr;
void setup() {
  size(640, 360, P3D);
  noStroke(); //no Stroke i guess
  noSmooth(); //no antialiasing (hopyfully)
  //load vert/frag Source
  shdr =new PShader(this,vertSource,firstPass);
  //compile to FrameBuffer
  shader(shdr);
}
void draw(){
  //clear buffer for redraw
  background(0); 
  //set time uniform
  shdr.set("t",millis()*.001f);
  //paint the shader on a surface
  rect(0,0,width,height);
  //reset? free the the Object (again i have to idea of java) 
  shdr=null; 
  //create a secondPass (todo write a global function, i feel - should)
  shdr =new PShader(this,vertSource,secondPass); 
  //copy pixels from the firstPass
  shdr.set("s",get()); 
  //draw back to FrameBuffer
  filter(shdr); 

  //note that this is not an endless loop. the light line have a time uniform variable 
  //the animation stops becourse we are only in the secondPass without a time uniform
  //a global function would help, if needed.
}

Or Look at the source the truth is somewhere out there :)

https://github.com/processing/processing/blob/master/core/src/processing/opengl/FrameBuffer.java#L56

Lord_of_the_Galaxy

Yes that's possible. Just look through some of the code on this forum, you should find it.
I can't help now, but if this problem remains till next week, I'll post my own already tested code here. Just drop me a PM or a mention on this thread.

toneburst

Thanks very much, guys, I'll follow up those tips.

@nabr QC is node-based, but also has programmable nodes (patches, in QC terminology) for GLSL shaders, OpenCL and Core Image Filter kernels and JavaScript.

I've used them quite a lot (albeit long ago), so I have a reasonable grasp of shader basics.

I've been messing around with code since the mid 90s, but never used Java, so more advanced Processing code looks unfamiliar to me. I'm sure I'll get the hang of it, though.

Thanks again, guys.

a|x

nabr

@tonebust

Okay, great! Just few notes, - i'm also new here, use processing for quick scetches and of course learning. P3D is made more for students and beginners (just to have a great time), so i feel their a overhead, like i what just hint(EVERYTHING) especially for shadertoy mode https://github.com/processing/processing/blob/master/core/src/processing/opengl/PGraphicsOpenGL.java#L1853

reason why most of the guys here use P2D. the problem is the vertex of position is not right they use QUAD per default (cant find the source right now) instead of

gl.glBufferData(GL.GL_ARRAY_BUFFER, 32, GLBuffers.newDirectFloatBuffer(new float[]{-1,-1,1,-1,1,1,-1,1}), GL.GL_STATIC_DRAW);
        gl.glDrawArrays(PGL.TRIANGLE_STRIP, 0, 8); /*shadertoy here we go*/

so you have to push the vertices back

gl_FragCoord =vec4(position.xy-1,0,1.); //i put *.5-1 negative numbers always feels wrong, just a personal preference

Processing is also made to run on, as much as possible variarity, of devices, if you on a WIN32_LEAN_AND_MEAN machine and updated your graphic drives the last 7 years you can bypass the most of the stuff also write a texture direct from fragment without a FBO

https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_shader_image_load_store.txt (sorry Mac) downside you usually end up with 500 line of OpenGL state machine.

your example from above pseudocode

void setup(){
   //...
   loadStuff();
}

int mix(shader a,shader b) {
return a+b;
}

void draw(){
backgroud(0)

shader2.Image(mix());
}

toneburst

@nabr Cool, thanks for that.

I think I can combine two of the shaders into one, which should simplify things somewhat.

So I need to render the shader to an FBO, then that FBO can be piped into the final shader which is displayed.

Is that correct?

I've never used FBOs before, but as I understand it, they're essentially textures that are rendered offscreen but can be used in other shaders, or fed back into the original shader for iterative effects.

Or do I need two FBOs, if I'm going to feed the result of the first shader back into the shader on the next frame?

a|x

toneburst

@nabr do you mean texture coords with P2D go from -1 to 1? That does seem odd, if so..

It makes sense for vertex coords to have 0, 0, 0 at the approximate centre of the mesh, for easier rotation.

a|x

nabr

@toneburst

when i do this

shdr.set("s",get()); 
filter(shdr); //POSTPROCESSING

I should land here somewhere, (would be interessting to debug) https://github.com/processing/processing/blob/master/core/src/processing/opengl/PGraphicsOpenGL.java#L1601-L1702

at least here https://github.com/processing/processing/blob/master/core/src/processing/opengl/PGraphicsOpenGL.java#L1679

10min reading time: https://learnopengl.com/#!Advanced-OpenGL/Framebuffers

nabr

String[] vertSource={"#version 150"
  , "in vec4 position;"
  , "in vec2 texCoord;"
  , "uniform mat4 transform ;"
  , "out vec4 vertTexCoord;"
  , "void main() {"
  , "gl_Position =transform*position;"
  , "vertTexCoord.xy = texCoord;"
  , "}"
};
String[] fragSource={"#version 150"
  , "in vec4 vertTexCoord;"
  , "uniform sampler2D tex;"
  , "out vec4 fragColor;"
  , "void main() {"
  , "fragColor =texture(tex,vertTexCoord.xy)*.83;"  // alpha baby yeah       
  , "}"
};
PShader shdr;
PImage img;
void setup() {
  size(400,400,P3D);
  noStroke();
  String https="https://";
  img = loadImage(https+"i0.wp.com/vanillicon.com/b5e4e568a6d3ac0022536d83a2b3124e.png");
  shdr=new PShader(this, vertSource, fragSource);
  shader(shdr);
}
void draw() {
    float t = millis()*.001f;


    //theirs a texture bug in this scetch
    //github.com/processing/processing/blob/master/core/src/processing/opengl/Texture.java#L586
    //((PGraphicsOpenGL)g).textureSampling(3); 
    //hint(ENABLE_TEXTURE_MIPMAPS);


    image(img,0,0,width,height);

    shdr.set("tex", img); 
    shdr.set("tex", get()); 
    background(255/2);
    translate(width/2, height/2);
    rotateY(t);
    rotateX(t);
    //bug fixed by adding nummbers perv. height/2
    box(200);
}

toneburst

Thanks @nabr I will have a read.

a|x

nabr

//you welcome
import javax.sound.sampled.*;
void setup(){
  size(640, 360, P3D);
  byte[]pcm_data=new byte[500000];
  for(int t=0;pcm_data.length>t;)pcm_data[t++]=(byte)(t%256-128-64|t%126);
  try {
    Clip clip=AudioSystem.getClip() ;
    clip.open(new AudioFormat(22050,1,1,1<0,1<0), pcm_data, 0, pcm_data.length-1);
    clip.start();
  }catch(Exception e) {e.printStackTrace();}
}
void draw(){
 float t = millis()*.001f;
 new PShader(this,
      new String[]{"#version 150 \n"+
        "in vec2 position;"+
        "out vec2 v;"+
        "void main() {"+
        "gl_Position = vec4(position.xy,0,1.);v=position;"+
        "}"
      },
      new String[]{"#version 150 \n"+
          "in vec2 v;"+
          "uniform float t;"+
          "out vec4 fragColor;"+
          "void main() {"+
          "fragColor = vec4(vec3(v,sin(t)*.5+.5),1.);"+
          "}"
      })
      {
        void clearColor(float time){
            this.set("t", time);
            filter(this);
          }
      }.clearColor(t);

     translate(width/2, height/2);
     rotateY(t);rotateX(t);
     noFill();
     sphereDetail(12);
     sphere(180);

     if(mousePressed||keyPressed)exit();
}

cansik

So your first request to chain multiple shaders is implemented in my library. Check out the Custom Shader part of the readme, to add your own shaders.

The second problem you mention sounds like a loop shader? Right? You then just have to store the textures you want to keep alive over one frame. Or am I wrong with my understanding?

@nabr I'm going to extend my library with more shaders. Are there other features you would like to add? :)

toneburst

Hi @cansik!

That's cool. Will check out your library.

Re. feedback; yes, that's correct. I want to chain two shaders, then pass the result on to a third shader for rendering to the output, and also back into the first shader for feedback effects.

a|x

nabr

@cansik I think, im more the path tracer guy. Also already tones of shaders on the web. I like your lib! We keep in touch.

@toneburst you now 1 week into processing, you have to share some code :) so the comunity can catch up on your progress and help.

nabr

im experimenting, maybe it's useful, i leave it here.

PShader shdr;
void setup(){
size(500,500,P3D);

filter(new PShader(this,new String[]{"#version 150 \n"
    + "in vec4 position;"
    + "in vec2 texCoord;"
    + "uniform mat4 transform ;"
    + "out vec4 vertTexCoord;"
    + "void main() {"
    + "gl_Position =transform*position;"
    + "vertTexCoord.xy = texCoord;"
    + "}"
   },new String[]{"#version 150 \n"
      + "in vec4 vertTexCoord;"
      + "uniform sampler2D tex;"
      + "out vec4 fragColor;"
      + "void main() {"
      + "fragColor =texture(tex,vertTexCoord.xy);"  
      + "}"
    }){
      PImage _img (){
        return loadImage("https://"+"processing.org/img/processing-web.png");
       }
      PShader _draw(PShader overload){
          print("Loading Image \n");
          if(!_img().isLoaded()) print("Done! Letz go \n");
          this.set("tex",_img());
          shader(overload);  
          return this;
        }
    }._draw(
        shdr=FirstPass(new PShader(this,new String[]{"#version 150 \n"
        + "in vec4 position;"
        + "in vec2 texCoord;"
        + "uniform mat4 transform ;"
        + "out vec4 vertTexCoord;"
        + "void main() {"
        + "gl_Position =transform*position;"
        + "vertTexCoord.xy = texCoord;"
        + "}"
       },new String[]{"#version 150 \n"
          + "in vec4 vertTexCoord;"
          + "uniform sampler2D tex;"
          + "uniform float t;"
          + "out vec4 fragColor;"
          + "void main() {"
          + "fragColor =vec4(vec3(vertTexCoord.xy,sin(t)*.5+.5),1.).brag;"  
          + "}"
          }
      )))
    );
}
void draw(){
  float t = millis()*.001f;
  shdr.set("t",t);
  translate(width/2,height/2);
  rotateX(t); rotateZ(t);
  box(100);
}
PShader FirstPass(PShader A){
print("Compiling FirstPass shader ...\n");
return A;
}

change some values see what they do

PShader shdr;
void setup(){
size(500,500,P3D);

shdr=MainShdr(new PShader(this,new String[]{"#version 150 \n"
    + "in vec4 position;"
    + "in vec2 texCoord;"
    + "uniform mat4 transform ;"
    + "out vec4 vertTexCoord;"
    + "void main() {"
    + "gl_Position =transform*position;"
    + "vertTexCoord.xy = texCoord;"
    + "}"
   },new String[]{"#version 150 \n"
      + "in vec4 vertTexCoord;"
      + "uniform sampler2D tex;"
      + "out vec4 fragColor;"
      + "void main() {"
      + "fragColor =texture(tex,vertTexCoord.xy);" 
      + "}"
    }){
      PShader _draw(PShader overload){
         this.set("tex",loadImage("https://"+"processing.org/img/processing-web.png"));
          shader(overload);  
          return this;
        }
    }._draw(shdr=MainShdr(new PShader(this,new String[]{"#version 150 \n"
        + "in vec4 position;"
        + "in vec2 texCoord;"
        + "uniform mat4 transform ;"
        + "out vec4 vertTexCoord;"
        + "void main() {"
        + "gl_Position =transform*position;"
        + "vertTexCoord.xy = texCoord;"
        + "}"
       },new String[]{"#version 150 \n"
          + "in vec4 vertTexCoord;"
          + "uniform sampler2D tex;"
          + "uniform float t;"
          + "out vec4 fragColor;"
          + "void main() {"
          + "fragColor =vec4(vec3(vertTexCoord.xy,sin(t)*.5+.5),1.).rgab;" 
          + "}"
          }
         )
        )
       )
      );

}
void draw(){
  float t = millis()*.001f;
  if(shdr.bound())shdr.set("t",t);
  filter(shdr);
  translate(width/2,height/2);
  rotateX(t); rotateZ(t);
  box(100);
}
PShader MainShdr(PShader A){
print("Compiling shader ... \t"+A+"\n");
return A;
}

toneburst

Hi,

got sidetracked and ended up trying to do this in Quartz Composer, failed, so I'm now back in Processing...

@cansik, can I access the previous frame's texture using ppixels when using postFX?

a|x

nabr

@toneburst this is a nice technique by @Lord_of_the_Galaxy
offscreen render

ppixels = backbuffer
http://glslsandbox.com/e#207.3

toneburst

@nabr ah, so ppixels will only work if the shader renders to the canvas, presumably..

a|x

nabr

@toneburst
this is what i do: i try different things out, and start with something very simpel.

//processing.org/reference/PShader.html

PShader conway;

void setup() {
  size(640, 360, P2D);
  // Shaders files must be in the "data" folder to load correctly
  conway = loadShader("https://"+"pastebin.com/raw/w0cjNYZN"); 

  //set uniforms that dont need to be updated
  conway.set("resolution",float(width),float(height));

  //compile shader
  shader(conway);

  //no outline for the rect
  noStroke();

  //processing.org/reference/rectMode_.html
  rectMode(RADIUS);
}
float t =0.f;
void draw() {

  //frameCount works best in processing internal 
  //otherwise you really have to know what you are doing
  t=frameCount*.01f;

  //set uniforms

  //time
  conway.set("time", t); 

  //mouse processing.org/reference/map_.html
  conway.set("mouse",map(mouseX, 0, width, 0, 1),map(mouseY, 0, height, 1, 0));

  //draw the shader to a rectangle
  rect(0, 0, width, height);
}

toneburst

I managed add my own shader to the Conway example in the Processing examples. I also added a brush-size control using the controlsP5 library.

import controlP5.*;
ControlP5 cp5;
float brushSize;

// GLSL version of Conway's game of life, ported from GLSL sandbox:
// http://glsl.heroku.com/e#207.3
// Exemplifies the use of the ppixels uniform in the shader, that gives
// access to the pixels of the previous frame.
PShader conway;
PGraphics pg;

void setup() {

  size(640, 480, P3D);

  // Add controls
  cp5 = new ControlP5(this);
  cp5.addSlider("brushSize")
    .setPosition(40, 40)
    .setSize(100, 20)
    .setRange(0.01, 0.05)
    .setValue(0.025)
    .setColorCaptionLabel(color(200,200,200));

  pg = createGraphics(640, 480, P2D);
  pg.noSmooth();
  conway = loadShader("conway.glsl");
  conway.set("resolution", float(pg.width), float(pg.height)); 
}

void draw() {
  conway.set("time", millis()/1000.0);
  float x = map(mouseX, 0, width, 0, 1);
  float y = map(mouseY, 0, height, 1, 0);
  conway.set("brushsize", brushSize); 
  conway.set("mouse", x, y);  
  pg.beginDraw();
  pg.background(0);
  pg.shader(conway);
  pg.rect(0, 0, pg.width, pg.height);
  pg.endDraw();  
  image(pg, 0, 0, width, height);

}

Fragment Shader:

// Conway's game of life

#ifdef GL_ES
precision highp float;
#endif

#define PROCESSING_COLOR_SHADER

uniform float time;
uniform vec2 mouse;
uniform vec2 resolution;
uniform sampler2D ppixels;
uniform float brushsize;

float cellstates = 100.;
uniform float a0, a1, a2, a3, a4, a5, a6, a7, a8;
uniform float d0, d1, d2, d3, d4, d5, d6, d7, d8;

vec4 live = vec4(1.,1.,1.,1.);
vec4 dead = vec4(0.,0.,0.,1.);

float singlestate = 1. / (cellstates - 1);

vec2 pixel = 1. / resolution;

float rand(vec2 pos) {
    return mod(fract(sin(dot(pos + time * 0.001, vec2(14.9898,78.233))) * 43758.5453), 1.0);
}

float val(vec2 pos, vec2 offset) {
    return (texture2D(ppixels, fract(pos + (pixel * offset))).r > 0.) ?
        1. :
        0.;
}

void main( void ) {

    //
    vec2 rules[9];
    rules[0] = vec2(a0, d0);
    rules[1] = vec2(a1, d1);
    rules[2] = vec2(1., d2);
    rules[3] = vec2(a3, 1.);
    rules[4] = vec2(a4, d4);
    rules[5] = vec2(a5, d5);
    rules[6] = vec2(a6, d6);
    rules[7] = vec2(a7, d7);
    rules[8] = vec2(a8, d8);

    // Normalised texture coordinates
    vec2 position = ( gl_FragCoord.xy / resolution.xy );

    // Paintbrush
    if (length(position-mouse) < brushsize) {
        gl_FragColor = mix(live, dead, step(rand(position), 0.5));
    } else {

        // Neighbouring cells
        float sum = 0.;
        sum += val(position, vec2(-1., -1.));
        sum += val(position, vec2(-1.,  0.));
        sum += val(position, vec2(-1.,  1.));
        sum += val(position, vec2( 1., -1.));
        sum += val(position, vec2( 1.,  0.));
        sum += val(position, vec2( 1.,  1.));
        sum += val(position, vec2( 0., -1.));
        sum += val(position, vec2( 0.,  1.));

        // Current pixel
        float me = texture2D(ppixels, position).r;

        // Lookup into rules array
        vec2 r = rules[int(sum)].rg;

        float state = me * r.x + r.y;

        vec4 color = vec4(state,state,state,1.);

        gl_FragColor = color;
    }
}

Still very much a WIP.

I'm currently trying to do all the various parts of the projects in separate sketches, before trying to stitch them together in some way.

a|x

nabr

@toneburst Ctrl+O will format Code :)
https://en.wikipedia.org/wiki/Markdown
.PDE file?

toneburst

Thanks @nabr. It's more nicely-formatted now :)

a|x

toneburst

I've made some progress.

// PostFX shader-chaining library
import ch.bildspur.postfx.builder.*;
import ch.bildspur.postfx.pass.*;
import ch.bildspur.postfx.*;
// GLSL version of Conway's game of life
PShader conway;
PGraphics canvas;

// Custom shader pass class
PostFX fx;
//PostFXSupervisor supervisor;
//FeedbackPass feedbackPass;
NegatePass negatePass;

// GUI library
import controlP5.*;
ControlP5 cp5;

float brushSize;

///////////////////////////
// SETUP //////////////////
///////////////////////////

void setup() {

  size(640, 480, P2D);

  // Add controls
  cp5 = new ControlP5(this);
  cp5.addSlider("brushSize")
    .setPosition(40, 40)
    .setSize(100, 20)
    .setRange(0.01, 0.05)
    .setValue(0.025)
    .setColorCaptionLabel(color(200,200,200));

  // Init shader stuff
  canvas = createGraphics(width, height, P3D);
  canvas.noSmooth();

  conway = loadShader("conway.glsl");
  conway.set("resolution", float(canvas.width), float(canvas.height));

  // PostFX pass stuff
  //supervisor = new PostFXSupervisor(this);
  fx = new PostFX(this);
  //feedbackPass = new FeedbackPass();
  negatePass = new NegatePass();
}

///////////////////////////
// DRAW ///////////////////
///////////////////////////

void draw() {

  // Shader uniforms
  conway.set("time", millis()/1000.0);
  float x = map(mouseX, 0, width, 0, 1);
  float y = map(mouseY, 0, height, 1, 0);
  conway.set("brushsize", brushSize); 
  conway.set("mouse", x, y);  

  // Draw shader
  canvas.beginDraw();
  canvas.background(0);

  canvas.pushMatrix();

  canvas.shader(conway);
  canvas.rect(0, 0, canvas.width, canvas.height);

  canvas.popMatrix();
  canvas.endDraw();

  //
  blendMode(BLEND);
  fx.render(canvas)
    .bloom(0.5, 20, 40)
    //.custom(negatePass)
    .compose();

}

This nicely applies a bloom effect to the output of the conway shader.

My hope is to replace the negate shader, from the PostFX CustomShaderEffect example with my own, but for the moment, I'm just using the negate shader from the example.

Unfortunately, if I uncomment the line

.custom(negatePass)

I get an OpenGL error

ERROR: 0:12: Regular non-array variable 'vertColor' may not be redeclared

Is there a way to fix this, @cansik?

a|x

toneburst

Oops, I did accidentally redeclare the varying 'vertColor' in negateFrag.glsl.

I think I was messing around with it, and didn't realise I'd saved it.

Now it works :) Time to try my own shader.

a|x

toneburst

So, now I have:

// PostFX shader-chaining library
import ch.bildspur.postfx.builder.*;
import ch.bildspur.postfx.pass.*;
import ch.bildspur.postfx.*;
// GLSL version of Conway's game of life
PShader conway;
PGraphics canvas;

// Custom shader pass class
PostFX fx;
//PostFXSupervisor supervisor;
FeedbackPass feedbackPass;
NegatePass negatePass;

// GUI library
import controlP5.*;
ControlP5 cp5;

float brushSize;

///////////////////////////
// SETUP //////////////////
///////////////////////////

void setup() {

  size(640, 480, P2D);

  // Add controls
  cp5 = new ControlP5(this);
  cp5.addSlider("brushSize")
    .setPosition(40, 40)
    .setSize(100, 20)
    .setRange(0.01, 0.05)
    .setValue(0.025)
    .setColorCaptionLabel(color(200,200,200));

  // Init shader stuff
  canvas = createGraphics(width, height, P3D);
  canvas.noSmooth();

  conway = loadShader("conway.glsl");
  conway.set("resolution", float(canvas.width), float(canvas.height));

  // PostFX pass stuff
  //supervisor = new PostFXSupervisor(this);
  fx = new PostFX(this);
  feedbackPass = new FeedbackPass();
  negatePass = new NegatePass();
}

///////////////////////////
// DRAW ///////////////////
///////////////////////////

void draw() {

  // Shader uniforms
  conway.set("time", millis()/1000.0);
  float x = map(mouseX, 0, width, 0, 1);
  float y = map(mouseY, 0, height, 1, 0);
  conway.set("brushsize", brushSize); 
  conway.set("mouse", x, y);  

  // Draw shader
  canvas.beginDraw();
  canvas.background(0);

  canvas.pushMatrix();

  canvas.shader(conway);
  canvas.rect(0, 0, canvas.width, canvas.height);

  canvas.popMatrix();
  canvas.endDraw();

  //
  blendMode(BLEND);
  fx.render(canvas)
    .bloom(0.5, 20, 40)
    .custom(feedbackPass)
    .compose();

}

And my shader:

// Mix between input texture and previous frame pixels

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

uniform sampler2D texture;
uniform sampler2D ppixels;
uniform float feedback;
varying vec4 vertTexCoord;

void main( void ) {
    vec2 position = vertTexCoord.st;
    gl_FragColor = mix(texture2D(texture, position), texture2D(ppixels, position), feedback);
}

Which doesn't work. The ppixel texture in the feedback shader seems to contain nothing, so changing the feedback amount just has the effect of mixing between the output of the first pass and black, and making the image darker. I'd intended to create a kind of 'trails' effect.

Does this mean I have to manage my own FBOs manually, in order to get feedback working?

I'm able to use ppixels for feedback, while I do the initial drawing of the Conway shader, before I apply the PostFX passes.

a|x

toneburst

Aha.. if I comment out the line

supervisor.clearPass(pass);

in my feedbackPass class declaration, it works...

I've added a function to my feedbackPass class to allow me to change one of the shader uniforms from the draw loop in the main sketch. No idea if this is the correct way to do this, as I'm unfamiliar with Java (and OOP in general, to my shame).

FeedbackPass.pde:

class FeedbackPass implements Pass
{
  PShader shader;
  float feedbackLevel;

  public FeedbackPass()
  {
    shader = loadShader("feedback.glsl");
  }

  @Override
    public void prepare(Supervisor supervisor) {
    shader.set("feedback", 0.75);
  }

  @Override
    public void apply(Supervisor supervisor) {
    PGraphics pass = supervisor.getNextPass();
    //supervisor.clearPass(pass);

    shader.set("feedback", feedbackLevel);
    pass.beginDraw();
    pass.shader(shader);
    pass.image(supervisor.getCurrentPass(), 0, 0);
    pass.endDraw();
  }

  public void setfeedback(float feedback)
  {
    feedbackLevel = feedback;
  }
}

And the main sketch .pde:

// PostFX shader-chaining library
import ch.bildspur.postfx.builder.*;
import ch.bildspur.postfx.pass.*;
import ch.bildspur.postfx.*;
// GLSL version of Conway's game of life
PShader conway;
PGraphics canvas;

// Custom shader pass class
PostFX fx;
FeedbackPass feedbackPass;

// GUI library
import controlP5.*;
ControlP5 cp5;

float brushSize;
float feedback;

///////////////////////////
// SETUP //////////////////
///////////////////////////

void setup() {

  size(640, 480, P2D);

  // Add controls
  cp5 = new ControlP5(this);
  cp5.addSlider("brushSize")
    .setPosition(40, 40)
    .setSize(100, 20)
    .setRange(0.01, 0.05)
    .setValue(0.025)
    .setColorCaptionLabel(color(200,200,200));

  cp5.addSlider("feedback")
    .setPosition(40, 70)
    .setSize(100, 20)
    .setRange(0., 0.95)
    .setValue(0.5)
    .setColorCaptionLabel(color(200,200,200));

  // Init shader stuff
  canvas = createGraphics(width, height, P3D);
  canvas.noSmooth();

  conway = loadShader("conway.glsl");
  conway.set("resolution", float(canvas.width), float(canvas.height));

  // PostFX pass stuff
  //supervisor = new PostFXSupervisor(this);
  fx = new PostFX(this);
  feedbackPass = new FeedbackPass();
}

///////////////////////////
// DRAW ///////////////////
///////////////////////////

void draw() {

  // Shader uniforms
  conway.set("time", millis()/1000.0);
  float x = map(mouseX, 0, width, 0, 1);
  float y = map(mouseY, 0, height, 1, 0);
  conway.set("brushsize", brushSize); 
  conway.set("mouse", x, y);  

  // Draw shader
  canvas.beginDraw();
  canvas.background(0);

  canvas.pushMatrix();

  canvas.shader(conway);
  canvas.rect(0, 0, canvas.width, canvas.height);

  canvas.popMatrix();
  canvas.endDraw();

  image(canvas, 0, 0);

  // Set feedback level for feedback pass shader
  feedbackPass.setfeedback(feedback);

  // Apply passes
  blendMode(BLEND);
  fx.render()
    //.bloom(0.5, 20, 40)
    .custom(feedbackPass)
    .compose();

}

a|x

toneburst

@cansik though I've sort of got feedback to work, I'd really like to be able to access previous frames data from inside the custom pass class instance, so I'm not relying on the ppixels class.

You mentioned saving the texture for the next frame. That's exactly what I need to do :)

I guess I'd change the shader to something like:

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

uniform sampler2D texture;
uniform sampler2D oldTexture;
uniform float feedback;
varying vec4 vertTexCoord;

void main( void ) {
    vec2 position = vertTexCoord.st;
    gl_FragColor = mix(texture2D(texture, position), texture2D(oldTexture, position), feedback);
}

Any tips on how I'd do that in my custom pass class definition?

class FeedbackPass implements Pass
{
  PShader shader;
  float feedbackLevel;


  PGraphics previousTex;

  public FeedbackPass()
  {
    shader = loadShader("feedback.glsl");
  }

  @Override
    public void prepare(Supervisor supervisor) {
    shader.set("feedback", 0.75);
  }

  @Override
    public void apply(Supervisor supervisor) {
    PGraphics pass = supervisor.getNextPass();
    //supervisor.clearPass(pass);

    shader.set("feedback", feedbackLevel);
    pass.beginDraw();
    pass.shader(shader);
    pass.image(supervisor.getCurrentPass(), 0, 0);
    pass.endDraw();
  }

  public void setfeedback(float feedback)
  {
    feedbackLevel = feedback;
  }
}

I've added a new PGraphics, as I guess I'll need to render to this at some stage, and bind it to the old_texture uniform in my shader. Not sure how I'd go about that though.

Any tips for a processing noob gratefully accepted.

a|x

toneburst

I have this updated custom pass definition:

class FeedbackPass implements Pass
{
  private PShader shader;

  private float feedbackLevel;

  private PGraphics previousTex;

  /////////////////
  // Constructor //
  /////////////////

  public FeedbackPass()
  {
    shader = loadShader("feedback.glsl");
    previousTex = createGraphics(width, height, P3D);
    shader.set("oldTexture", previousTex);
  }

  @Override
    public void prepare(Supervisor supervisor) {
    shader.set("feedback", 0.75);
  }

  @Override
    public void apply(Supervisor supervisor) {
    PGraphics pass = supervisor.getNextPass();
    //supervisor.clearPass(pass);

    shader.set("feedback", feedbackLevel);
    pass.beginDraw();
    pass.shader(shader);
    pass.image(supervisor.getCurrentPass(), 0, 0);
    pass.endDraw();

    // Update previous texture
    previousTex.loadPixels();
    // Don't know how to get pixels from pass
    //arrayCopy(pass.image.pixels, previousTex.pixels); 
    previousTex.updatePixels(); 

  }

  ///////////////////// 
  // Mutator methods //
  /////////////////////

  public void setfeedback(float feedback)
  {
    this.feedbackLevel = feedback;
  }
}

I think I just need to work out how to copy the pass pixels to the background texture on line 38. Not sure how to go about that, though.

a|x

toneburst

Too easy...

This seems to work :)

/*
  ////////////////////////////////////////
  PostFX custom pass class definition

  Dependencies:
  PostFX library
  ////////////////////////////////////////
*/

class FeedbackPass implements Pass
{
  private PShader shader;

  private float feedbackLevel;

  private PGraphics previousTex;

  /////////////////
  // Constructor //
  /////////////////

  public FeedbackPass()
  {
    shader = loadShader("feedback.glsl");
    previousTex = createGraphics(width, height, P3D);
  }

  @Override
    public void prepare(Supervisor supervisor) {
       shader.set("oldTexture", previousTex);
  }

  @Override
    public void apply(Supervisor supervisor) {
    PGraphics pass = supervisor.getNextPass();
    //supervisor.clearPass(pass);

    shader.set("feedback", feedbackLevel);
    pass.beginDraw();
    pass.shader(shader);
    pass.image(supervisor.getCurrentPass(), 0, 0);
    pass.endDraw();

    // Update previous texture
    previousTex = pass;  
  }

  ///////////////////// 
  // Mutator methods //
  /////////////////////

  public void setfeedback(float feedback)
  {
    this.feedbackLevel = feedback;
  }
}

I have a feeling that setting 'oldTex' to 'pass' like this (line 45) may not be the fastest way to do it.

I tried arrayCopy, but got a nullPointer exception.

a|x

toneburst

incidentally, is there a way to stop Markdown attempting to insert links whenever there's an "@Override" line in my code?

a|x

Lord_of_the_Galaxy

@toneburst @cansik Any chance you guys know what's happening here - https://forum.processing.org/two/discussion/22385/reaction-diffusion-using-glsl-works-different-from-normal

toneburst

OK, next step: I need to get an image from a webcam into a shader.

It doesn't look like an instance of the Capture object can have a shader attached to it directly, in the same way a PGraphics instance can.

I guess this means I need to copy the pixels from the capture to a PGraphics instance. Sounds like this might be expensive, though. Do I literally have to loop through all the pixel array of the Capture, copying to the PGraphics?

Maybe there's a better way to do the copy, all in one go?

a|x

toneburst

Got image capture working :)

/*
  ////////////////////////////////////////////////////
  ////////////////////////////////////////////////////

  A-Life

  Alex Drinkwater 2017

  ////////////////////////////////////////////////////
  ////////////////////////////////////////////////////
*/

//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// LIBRARIES /////////////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

// Video library  
import processing.video.*;

// PostFX shader-chaining library
// https://github.com/cansik/processing-postfx
import ch.bildspur.postfx.builder.*;
import ch.bildspur.postfx.pass.*;
import ch.bildspur.postfx.*;

// ControlP5 GUI library
// https://github.com/sojamo/controlp5
import controlP5.*;

//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// GLOBAL VARIABLES //////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

Capture cam;
PGraphics camFrame;

// Custom shader pass class (requires PostFX)
PostFX fx;
FeedbackPass feedbackPass;
ConwayPass conwayPass;

PGraphics canvas;

// GUI library
ControlP5 cp5;
color guiColor = color(200,200,200);

float brushSize;
float feedback;
boolean runFX;

//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// SETUP /////////////////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

void setup() {

  size(1280, 720, P2D);

  //////////////////
  // Init Capture //
  //////////////////

  String[] cameras = Capture.list();

  if (cameras.length == 0) {
    println("There are no cameras available for capture.");
    exit();
  } else {
    println("Available cameras:");
    for (int i = 0; i < cameras.length; i++) {
      println(cameras[i]);
    }   
    // The camera can be initialized directly using an 
    // element from the array returned by list():
    cam = new Capture(this, cameras[0]);
    cam.start();     
  }

  camFrame = createGraphics(width, height, P2D);

  ///////////////////////
  // PostFX pass stuff //
  ///////////////////////

  fx            = new PostFX(this);
  feedbackPass  = new FeedbackPass();
  conwayPass    = new ConwayPass();

  //////////////////
  // Add controls //
  //////////////////

  cp5 = new ControlP5(this);
  cp5.addSlider("brushSize")
    .setPosition(40, 40)
    .setSize(100, 20)
    .setRange(0.01, 0.05)
    .setValue(0.025)
    .setColorCaptionLabel(guiColor);

  cp5.addSlider("feedback")
    .setPosition(40, 70)
    .setSize(100, 20)
    .setRange(0., 0.95)
    .setValue(0.80)
    .setColorCaptionLabel(guiColor);

   cp5.addToggle("runFX")
     .setPosition(40, 100)
     .setSize(20, 20)
     .setColorCaptionLabel(guiColor)
     .setValue(false);
}

//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// DRAW //////////////////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

void draw() {

  // Set feedback level for feedback pass shader
  feedbackPass.setfeedback(feedback);

  conwayPass.setStartFX(runFX);
  conwayPass.setMouse(map(mouseX, 0, width, 0, 1), map(mouseY, 0, height, 1, 0));
  conwayPass.setBrushSize(brushSize);

  if (cam.available() == true) {
    cam.read();
  }

  cam.loadPixels();
  camFrame.loadPixels();
  arrayCopy(cam.pixels, camFrame.pixels);
  cam.updatePixels();
  camFrame.updatePixels();

  image(camFrame, 0, 0);

  // Apply passes
  blendMode(BLEND);
  fx.render()
    //.custom(conwayPass)
    .custom(feedbackPass)
    //.bloom(0.5, 20, 40)
    .compose();
}

a|x

toneburst

Still having issues with feedback within the custom shader passes, though. I'm going to post about that in a new thread.

a|x

toneburst

@Lord_of_the_Galaxy I'm a complete Processing noob, I'm afraid. I've messed around with Grey Scott RD shaders in the past, but using Quartz Composer. I'm struggling with implementing feedback effects properly in Processing at the moment, so we're probably in similar positions.

Hope you get the problem sorted.

a|x

nabr

@toneburst

I went few pages back in this thread, i don't know how all of you guys live with this copy paste attitude and then setup .pde for testing (already 10 minutes pass)

It's 2017 can you zip it ?

So i (someone) can unzip it and start with debug? Sorry, it's just me, and i'm lazy as hell. I mean, if some1 fear to get a virus or something, github, mediafire, googledrive etc. has now a shared folder, so the user can preview the files befor they get zipped.

Best

toneburst

Good points. I'm new to this forum, so don't know how things are done here.

https://www.dropbox.com/s/fkgh0zu2yhrpxdi/Conway_Webcam.zip?dl=0

Here's a zip of the file.

a|x

toneburst

It runs very slowly on my old MacBook Pro. I'm pretty sure it's the Graphic instance copying that's required for texture feedback that's slowing things to a crawl. A very similar setup would run at 60fps in Quartz Composer, even on my ancient laptop.

a|x

toneburst

Also, it doesn't handle different camera output formats well.

a|x

nabr

@toneburst

"don't know how things are done here." You the first one who zip it. Im also new to this forum, and it makes me sick to copy paste stuff til i get something runnig.

Thank you man!

I mean the users here used to do it for the past 10 years. PHP or whatever script to zip me a source file will take 10 lines of code ?

toneburst

No problem! I like to keep things in separate files, which makes it more difficult to paste the whole thing into a post here. It's getting a bit long for that, anyway, as you observed ;)

a|x

toneburst

I'm kinda thinking I should rewrite the whole thing to use native OpenGL textures and FBOs wherever possible. That way, maybe the whole thing can be kept on the GPU.

No real idea if this is possible, though.

a|x

toneburst

I'm also quite interested in Geometry Shaders. I've seen examples of their use in Processing, but don't know how easy it is to achieve.

a|x

nabr

@toneburst

Yes, their a bug in your code. @cansik know's a quick fix i think

It was not working, on my PC i changed few lines to make it work

https://processing.org:8443/tutorials/video/

You have different declarations like P2D and then P3D. I found it's best, while debuging your stetch, when you use nummers in the size(nummer, nummer, P3D)
- and then createGraphic(nummer,nummer) >instead of width, height. Thouse values can evaluate druing the "startup" and "miss" the setup. default is 100x100 px so you would run a createGraphic, or a box(size) Object at a lower res ...etc.

   //
  ////////////////////////////////////////////////////
  ////////////////////////////////////////////////////

  //A-Life

 //Alex Drinkwater 2017

  ////////////////////////////////////////////////////
  ////////////////////////////////////////////////////
  //

// Video library
import processing.video.*;

// PostFX shader-chaining library
// https://github.com/cansik/processing-postfx
import ch.bildspur.postfx.builder.*;
import ch.bildspur.postfx.pass.*;
import ch.bildspur.postfx.*;

// ControlP5 GUI library
// https://github.com/sojamo/controlp5
import controlP5.*;

//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// GLOBAL VARIABLES //////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

Capture cam;
PGraphics camFrame;

// Custom shader pass class (requires PostFX)
PostFX fx;
FeedbackPass feedbackPass;
ConwayPass conwayPass;

PGraphics canvas;

// GUI library
ControlP5 cp5;
color guiColor = color(200, 200, 200);

float brushSize;
float feedback;
float channelSpread;
boolean runFX;

//changed
void captureEvent(Capture video) {
  video.read();
}
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// SETUP /////////////////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

void setup() {

  //changed
  size(640, 480, P3D);

  //////////////////
  // Init Capture //
  //////////////////

  //changed
  camFrame = createGraphics(640, 480, P3D);

  ///////////////////////
  // PostFX pass stuff //
  ///////////////////////

  fx            = new PostFX(this);
  feedbackPass  = new FeedbackPass();
  conwayPass    = new ConwayPass();

  //////////////////
  // Add controls //
  //////////////////

  cp5 = new ControlP5(this);
  cp5.addSlider("brushSize")
    .setPosition(40, 40)
    .setSize(100, 20)
    .setRange(0.01, 0.05)
    .setValue(0.025)
    .setColorCaptionLabel(guiColor);

  cp5.addSlider("feedback")
    .setPosition(40, 70)
    .setSize(100, 20)
    .setRange(0.0, 0.90)
    .setValue(0.80)
    .setColorCaptionLabel(guiColor);

  cp5.addSlider("channelSpread")
    .setPosition(40, 100)
    .setSize(100, 20)
    .setRange(0.00, 0.07)
    .setValue(0.0)
    .setColorCaptionLabel(guiColor);

  cp5.addToggle("runFX")
    .setPosition(40, 130)
    .setSize(20, 20)
    .setColorCaptionLabel(guiColor)
    .setValue(false);

  //changed   
  cam = new Capture(this, 640, 480);
  cam.start();
}

//////////////////////////////////////////////////////
//////////////////////////////////////////////////////
// DRAW //////////////////////////////////////////////
//////////////////////////////////////////////////////
//////////////////////////////////////////////////////

void draw() {

  // Set feedback level for feedback pass shader
  feedbackPass.setFeedback(feedback);
  feedbackPass.setChannelSpread(channelSpread);

  conwayPass.setStartFX(runFX);
  conwayPass.setMouse(map(mouseX, 0, width, 0, 1), map(mouseY, 0, height, 1, 0));
  conwayPass.setBrushSize(brushSize);

  // Copy capture pixels to PGraphics instance
  cam.loadPixels();
  camFrame.loadPixels();
  arrayCopy(cam.pixels, camFrame.pixels);
  cam.updatePixels();
  camFrame.updatePixels();

  image(camFrame, 0, 0);

  //OpenGL error 1282 at top endDraw(): invalid operation
  //Apply passes
  blendMode(BLEND);
   fx.render()
   .custom(conwayPass)
   .custom(feedbackPass)
   //.bloom(0.5, 20, 40)
   .compose();

  //enable for debug
  //image(cam, 0, 0);
}

nabr

but i looks already crazy and fun. keep digging it!

toneburst

@nabr Hmm.. thanks for that.

I'm not sure it's improved things on my system, though it's quite a cool glitch effect. The camera feed stutters and 'sticks' in quite a nice way, though I think it might give me a bit of a headache if I looked at it for too long.

It's cool there's a way to force the captured image to a particular size, though. That's what I was looking for, in fact.

I'll have a look tomorrow, and see if it works better on my (much more recent) work machine.

Thanks again!

a|x

nabr

@toneburst

yes,

so only this part works, but their is something wrong with the "filter". Run FX gives me the conwayPass only, on/off will switch between cam output and Conway shader

void draw() {

  // Set feedback level for feedback pass shader
  feedbackPass.setFeedback(feedback);
  feedbackPass.setChannelSpread(channelSpread);

  conwayPass.setStartFX(runFX);
  conwayPass.setMouse(map(mouseX, 0, width, 0, 1), map(mouseY, 0, height, 1, 0));
  conwayPass.setBrushSize(brushSize);

  // Copy capture pixels to PGraphics instance
  cam.loadPixels();
  camFrame.loadPixels();
  arrayCopy(cam.pixels, camFrame.pixels);
  cam.updatePixels();
  camFrame.updatePixels();


  image(camFrame, 0, 0);


  //OpenGL error 1282 at top endDraw(): invalid operation
  //Apply passes
  blendMode(BLEND);
   fx.render()
   .custom(conwayPass)
   .custom(feedbackPass)
   //.bloom(0.5, 20, 40)
   .compose();
}

toneburst

@nabr thanks. It's for a student at my work, in fact. She has an idea for an installation for her final project, and I'm trying to implement it in Processing, since that's what her group are taught.

a|x

nabr

@toneburst

the feedback looks great! what is the goal here to make the conway shader work on top of the video output ?

toneburst

@nabr thanks re. the feedback! I got a bit sidetracked into messing around with that for a while. Still have a little idea I'd like to integrate into it.

Am I right in thinking that copying the pixels in the custom PostFX pass definition is a performance bottleneck, do you think? I can't imagine it's done on the GPU.

a|x