Chaining Shaders and Feedback

Hi,

I'm now here, so let me introduce myself.

I'm Alex, from sunny London, in the UK.

I'm new to Processing, but have some programming experience, in various languages. I've also used GLSL shaders quite a lot in Quartz Composer, but the particular job I'm working on at the moment seems difficult to achieve in QC, so I'm thinking of giving Processing a try, instead, which brings me to my questions:

  1. Is it possible to chain several 2D fragment shaders together, so that the result of one shader is passed into the next shader as a texture? My particular scenario involves a further complication:

  2. Is it possible to chain 2 shaders as above, and have the second shader feed back into the first, and also go through a third shader, which renders the result to the screen? I've attempted to illustrate what I have in mind below.

|| [shader 0] > [shader 1] || > [shader 2] > [screen] || || === < [prev. frame] < ===

I'm aware this is kinda throwing myself in at the deep end, but any advice would be much appreciated.

Thanks guys,

a|x

«1

Answers

  • edited May 2017

    I think this might help with the chaining, perhaps. https://github.com/cansik/processing-postfx I'm not sure if it would help with the feedback part, though.

    a|x

  • edited May 2017

    Hello

    i just skip trough some pictures in QuartComposer seems more Node based (Visual Programing). If you enter the world of Processing, you have to write code. With a custom request, their is not way around, than read one or the other topic on how computer graphics work.

    Processing is driven by JOGL

    1. you can ask @cansik if he can spend 5minutes more to add more functionality to his awesome postfx lib.

    2. Search in this Forum.

    3. Just hack, explore, their is always more then way to do something (one implemention is just faster, other qualiltes then the other)

    @Lord_of_the_Galaxy hava a nice solution to write the previous frame to texture

    I'm not realy sure, but you can read it between the lines, some computation is already set up in the background, when you start https://github.com/processing/processing/wiki/Advanced-OpenGL#vertex-coordinates-are-in-model-space

    Can't find in the source code right now, but i think, they render the Framebuffer to a texture per default.

    Here is my attempt to write and store data without access to a low level api

    String[] vertSource={"#version 150"
      , "in vec2 position;"
      , "out vec2 uv;"
      , "void main() {"
      , "gl_Position = vec4(position,0.,1.);"
      , "uv=position;"
      , "}"
    };
    String[] firstPass={"#version 150"
      , "in vec2 uv;"
      , "uniform float t;"
      , "out vec4 fragColor;"
      , "void main() {"
      //a line with animation
      , "fragColor = vec4(vec3(.2/length(abs(uv.x)*sin(t))),1.);"
      , "}"
    };
    String[] secondPass={"#version 150"
      , "in vec2 uv;"
      , "uniform sampler2D s;"
      , "out vec4 fragColor;"
      , "void main() {"
      //just a texture
      , "fragColor = texture(s,uv-vec2(.04));"
      , "}"
    };
    PShader shdr;
    void setup() {
      size(640, 360, P3D);
      noStroke(); //no Stroke i guess
      noSmooth(); //no antialiasing (hopyfully)
      //load vert/frag Source
      shdr =new PShader(this,vertSource,firstPass);
      //compile to FrameBuffer
      shader(shdr);
    }
    void draw(){
      //clear buffer for redraw
      background(0); 
      //set time uniform
      shdr.set("t",millis()*.001f);
      //paint the shader on a surface
      rect(0,0,width,height);
      //reset? free the the Object (again i have to idea of java) 
      shdr=null; 
      //create a secondPass (todo write a global function, i feel - should)
      shdr =new PShader(this,vertSource,secondPass); 
      //copy pixels from the firstPass
      shdr.set("s",get()); 
      //draw back to FrameBuffer
      filter(shdr); 
    
      //note that this is not an endless loop. the light line have a time uniform variable 
      //the animation stops becourse we are only in the secondPass without a time uniform
      //a global function would help, if needed.
    }
    

    Or Look at the source the truth is somewhere out there :)

    https://github.com/processing/processing/blob/master/core/src/processing/opengl/FrameBuffer.java#L56

  • Yes that's possible. Just look through some of the code on this forum, you should find it.
    I can't help now, but if this problem remains till next week, I'll post my own already tested code here. Just drop me a PM or a mention on this thread.

  • Thanks very much, guys, I'll follow up those tips.

    @nabr QC is node-based, but also has programmable nodes (patches, in QC terminology) for GLSL shaders, OpenCL and Core Image Filter kernels and JavaScript.

    I've used them quite a lot (albeit long ago), so I have a reasonable grasp of shader basics.

    I've been messing around with code since the mid 90s, but never used Java, so more advanced Processing code looks unfamiliar to me. I'm sure I'll get the hang of it, though.

    Thanks again, guys.

    a|x

  • edited May 2017

    @tonebust

    Okay, great! Just few notes, - i'm also new here, use processing for quick scetches and of course learning. P3D is made more for students and beginners (just to have a great time), so i feel their a overhead, like i what just hint(EVERYTHING) especially for shadertoy mode https://github.com/processing/processing/blob/master/core/src/processing/opengl/PGraphicsOpenGL.java#L1853

    reason why most of the guys here use P2D. the problem is the vertex of position is not right they use QUAD per default (cant find the source right now) instead of

    gl.glBufferData(GL.GL_ARRAY_BUFFER, 32, GLBuffers.newDirectFloatBuffer(new float[]{-1,-1,1,-1,1,1,-1,1}), GL.GL_STATIC_DRAW);
            gl.glDrawArrays(PGL.TRIANGLE_STRIP, 0, 8); /*shadertoy here we go*/
    

    so you have to push the vertices back

    gl_FragCoord =vec4(position.xy-1,0,1.); //i put *.5-1 negative numbers always feels wrong, just a personal preference

    Processing is also made to run on, as much as possible variarity, of devices, if you on a WIN32_LEAN_AND_MEAN machine and updated your graphic drives the last 7 years you can bypass the most of the stuff also write a texture direct from fragment without a FBO

    https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_shader_image_load_store.txt (sorry Mac) downside you usually end up with 500 line of OpenGL state machine.

    your example from above pseudocode

    void setup(){
       //...
       loadStuff();
    }
    
    int mix(shader a,shader b) {
    return a+b;
    }
    
    void draw(){
    backgroud(0)
    
    shader2.Image(mix());
    }
    
  • edited May 2017

    @nabr Cool, thanks for that.

    I think I can combine two of the shaders into one, which should simplify things somewhat.

    So I need to render the shader to an FBO, then that FBO can be piped into the final shader which is displayed.

    Is that correct?

    I've never used FBOs before, but as I understand it, they're essentially textures that are rendered offscreen but can be used in other shaders, or fed back into the original shader for iterative effects.

    Or do I need two FBOs, if I'm going to feed the result of the first shader back into the shader on the next frame?

    a|x

  • edited May 2017

    @nabr do you mean texture coords with P2D go from -1 to 1? That does seem odd, if so..

    It makes sense for vertex coords to have 0, 0, 0 at the approximate centre of the mesh, for easier rotation.

    a|x

  • edited May 2017
    String[] vertSource={"#version 150"
      , "in vec4 position;"
      , "in vec2 texCoord;"
      , "uniform mat4 transform ;"
      , "out vec4 vertTexCoord;"
      , "void main() {"
      , "gl_Position =transform*position;"
      , "vertTexCoord.xy = texCoord;"
      , "}"
    };
    String[] fragSource={"#version 150"
      , "in vec4 vertTexCoord;"
      , "uniform sampler2D tex;"
      , "out vec4 fragColor;"
      , "void main() {"
      , "fragColor =texture(tex,vertTexCoord.xy)*.83;"  // alpha baby yeah       
      , "}"
    };
    PShader shdr;
    PImage img;
    void setup() {
      size(400,400,P3D);
      noStroke();
      String https="https://";
      img = loadImage(https+"i0.wp.com/vanillicon.com/b5e4e568a6d3ac0022536d83a2b3124e.png");
      shdr=new PShader(this, vertSource, fragSource);
      shader(shdr);
    }
    void draw() {
        float t = millis()*.001f;
    
    
        //theirs a texture bug in this scetch
        //github.com/processing/processing/blob/master/core/src/processing/opengl/Texture.java#L586
        //((PGraphicsOpenGL)g).textureSampling(3); 
        //hint(ENABLE_TEXTURE_MIPMAPS);
    
    
        image(img,0,0,width,height);
    
        shdr.set("tex", img); 
        shdr.set("tex", get()); 
        background(255/2);
        translate(width/2, height/2);
        rotateY(t);
        rotateX(t);
        //bug fixed by adding nummbers perv. height/2
        box(200);
    }
    
  • Thanks @nabr I will have a read.

    a|x

  • edited May 2017
    //you welcome
    import javax.sound.sampled.*;
    void setup(){
      size(640, 360, P3D);
      byte[]pcm_data=new byte[500000];
      for(int t=0;pcm_data.length>t;)pcm_data[t++]=(byte)(t%256-128-64|t%126);
      try {
        Clip clip=AudioSystem.getClip() ;
        clip.open(new AudioFormat(22050,1,1,1<0,1<0), pcm_data, 0, pcm_data.length-1);
        clip.start();
      }catch(Exception e) {e.printStackTrace();}
    }
    void draw(){
     float t = millis()*.001f;
     new PShader(this,
          new String[]{"#version 150 \n"+
            "in vec2 position;"+
            "out vec2 v;"+
            "void main() {"+
            "gl_Position = vec4(position.xy,0,1.);v=position;"+
            "}"
          },
          new String[]{"#version 150 \n"+
              "in vec2 v;"+
              "uniform float t;"+
              "out vec4 fragColor;"+
              "void main() {"+
              "fragColor = vec4(vec3(v,sin(t)*.5+.5),1.);"+
              "}"
          })
          {
            void clearColor(float time){
                this.set("t", time);
                filter(this);
              }
          }.clearColor(t);
    
         translate(width/2, height/2);
         rotateY(t);rotateX(t);
         noFill();
         sphereDetail(12);
         sphere(180);
    
         if(mousePressed||keyPressed)exit();
    }
    
  • So your first request to chain multiple shaders is implemented in my library. Check out the Custom Shader part of the readme, to add your own shaders.

    The second problem you mention sounds like a loop shader? Right? You then just have to store the textures you want to keep alive over one frame. Or am I wrong with my understanding?

    @nabr I'm going to extend my library with more shaders. Are there other features you would like to add? :)

  • edited May 2017

    Hi @cansik!

    That's cool. Will check out your library.

    Re. feedback; yes, that's correct. I want to chain two shaders, then pass the result on to a third shader for rendering to the output, and also back into the first shader for feedback effects.

    a|x

  • @cansik I think, im more the path tracer guy. Also already tones of shaders on the web. I like your lib! We keep in touch.

    @toneburst you now 1 week into processing, you have to share some code :) so the comunity can catch up on your progress and help.

  • edited May 2017

    im experimenting, maybe it's useful, i leave it here.

    PShader shdr;
    void setup(){
    size(500,500,P3D);
    
    filter(new PShader(this,new String[]{"#version 150 \n"
        + "in vec4 position;"
        + "in vec2 texCoord;"
        + "uniform mat4 transform ;"
        + "out vec4 vertTexCoord;"
        + "void main() {"
        + "gl_Position =transform*position;"
        + "vertTexCoord.xy = texCoord;"
        + "}"
       },new String[]{"#version 150 \n"
          + "in vec4 vertTexCoord;"
          + "uniform sampler2D tex;"
          + "out vec4 fragColor;"
          + "void main() {"
          + "fragColor =texture(tex,vertTexCoord.xy);"  
          + "}"
        }){
          PImage _img (){
            return loadImage("https://"+"processing.org/img/processing-web.png");
           }
          PShader _draw(PShader overload){
              print("Loading Image \n");
              if(!_img().isLoaded()) print("Done! Letz go \n");
              this.set("tex",_img());
              shader(overload);  
              return this;
            }
        }._draw(
            shdr=FirstPass(new PShader(this,new String[]{"#version 150 \n"
            + "in vec4 position;"
            + "in vec2 texCoord;"
            + "uniform mat4 transform ;"
            + "out vec4 vertTexCoord;"
            + "void main() {"
            + "gl_Position =transform*position;"
            + "vertTexCoord.xy = texCoord;"
            + "}"
           },new String[]{"#version 150 \n"
              + "in vec4 vertTexCoord;"
              + "uniform sampler2D tex;"
              + "uniform float t;"
              + "out vec4 fragColor;"
              + "void main() {"
              + "fragColor =vec4(vec3(vertTexCoord.xy,sin(t)*.5+.5),1.).brag;"  
              + "}"
              }
          )))
        );
    }
    void draw(){
      float t = millis()*.001f;
      shdr.set("t",t);
      translate(width/2,height/2);
      rotateX(t); rotateZ(t);
      box(100);
    }
    PShader FirstPass(PShader A){
    print("Compiling FirstPass shader ...\n");
    return A;
    }
    

    change some values see what they do

    PShader shdr;
    void setup(){
    size(500,500,P3D);
    
    shdr=MainShdr(new PShader(this,new String[]{"#version 150 \n"
        + "in vec4 position;"
        + "in vec2 texCoord;"
        + "uniform mat4 transform ;"
        + "out vec4 vertTexCoord;"
        + "void main() {"
        + "gl_Position =transform*position;"
        + "vertTexCoord.xy = texCoord;"
        + "}"
       },new String[]{"#version 150 \n"
          + "in vec4 vertTexCoord;"
          + "uniform sampler2D tex;"
          + "out vec4 fragColor;"
          + "void main() {"
          + "fragColor =texture(tex,vertTexCoord.xy);" 
          + "}"
        }){
          PShader _draw(PShader overload){
             this.set("tex",loadImage("https://"+"processing.org/img/processing-web.png"));
              shader(overload);  
              return this;
            }
        }._draw(shdr=MainShdr(new PShader(this,new String[]{"#version 150 \n"
            + "in vec4 position;"
            + "in vec2 texCoord;"
            + "uniform mat4 transform ;"
            + "out vec4 vertTexCoord;"
            + "void main() {"
            + "gl_Position =transform*position;"
            + "vertTexCoord.xy = texCoord;"
            + "}"
           },new String[]{"#version 150 \n"
              + "in vec4 vertTexCoord;"
              + "uniform sampler2D tex;"
              + "uniform float t;"
              + "out vec4 fragColor;"
              + "void main() {"
              + "fragColor =vec4(vec3(vertTexCoord.xy,sin(t)*.5+.5),1.).rgab;" 
              + "}"
              }
             )
            )
           )
          );
    
    }
    void draw(){
      float t = millis()*.001f;
      if(shdr.bound())shdr.set("t",t);
      filter(shdr);
      translate(width/2,height/2);
      rotateX(t); rotateZ(t);
      box(100);
    }
    PShader MainShdr(PShader A){
    print("Compiling shader ... \t"+A+"\n");
    return A;
    }
    
  • Hi,

    got sidetracked and ended up trying to do this in Quartz Composer, failed, so I'm now back in Processing...

    @cansik, can I access the previous frame's texture using ppixels when using postFX?

    a|x

  • edited May 2017

    @toneburst this is a nice technique by @Lord_of_the_Galaxy
    offscreen render

    ppixels = backbuffer
    http://glslsandbox.com/e#207.3

  • @nabr ah, so ppixels will only work if the shader renders to the canvas, presumably..

    a|x

  • edited May 2017

    @toneburst
    this is what i do: i try different things out, and start with something very simpel.

    //processing.org/reference/PShader.html
    
    PShader conway;
    
    void setup() {
      size(640, 360, P2D);
      // Shaders files must be in the "data" folder to load correctly
      conway = loadShader("https://"+"pastebin.com/raw/w0cjNYZN"); 
    
      //set uniforms that dont need to be updated
      conway.set("resolution",float(width),float(height));
    
      //compile shader
      shader(conway);
    
      //no outline for the rect
      noStroke();
    
      //processing.org/reference/rectMode_.html
      rectMode(RADIUS);
    }
    float t =0.f;
    void draw() {
    
      //frameCount works best in processing internal 
      //otherwise you really have to know what you are doing
      t=frameCount*.01f;
    
      //set uniforms
    
      //time
      conway.set("time", t); 
    
      //mouse processing.org/reference/map_.html
      conway.set("mouse",map(mouseX, 0, width, 0, 1),map(mouseY, 0, height, 1, 0));
    
      //draw the shader to a rectangle
      rect(0, 0, width, height);
    }
    
  • edited May 2017

    I managed add my own shader to the Conway example in the Processing examples. I also added a brush-size control using the controlsP5 library.

    import controlP5.*;
    ControlP5 cp5;
    float brushSize;
    
    // GLSL version of Conway's game of life, ported from GLSL sandbox:
    // http://glsl.heroku.com/e#207.3
    // Exemplifies the use of the ppixels uniform in the shader, that gives
    // access to the pixels of the previous frame.
    PShader conway;
    PGraphics pg;
    
    void setup() {
    
      size(640, 480, P3D);
    
      // Add controls
      cp5 = new ControlP5(this);
      cp5.addSlider("brushSize")
        .setPosition(40, 40)
        .setSize(100, 20)
        .setRange(0.01, 0.05)
        .setValue(0.025)
        .setColorCaptionLabel(color(200,200,200));
    
      pg = createGraphics(640, 480, P2D);
      pg.noSmooth();
      conway = loadShader("conway.glsl");
      conway.set("resolution", float(pg.width), float(pg.height)); 
    }
    
    void draw() {
      conway.set("time", millis()/1000.0);
      float x = map(mouseX, 0, width, 0, 1);
      float y = map(mouseY, 0, height, 1, 0);
      conway.set("brushsize", brushSize); 
      conway.set("mouse", x, y);  
      pg.beginDraw();
      pg.background(0);
      pg.shader(conway);
      pg.rect(0, 0, pg.width, pg.height);
      pg.endDraw();  
      image(pg, 0, 0, width, height);
    
    }
    

    Fragment Shader:

    // Conway's game of life
    
    #ifdef GL_ES
    precision highp float;
    #endif
    
    #define PROCESSING_COLOR_SHADER
    
    uniform float time;
    uniform vec2 mouse;
    uniform vec2 resolution;
    uniform sampler2D ppixels;
    uniform float brushsize;
    
    float cellstates = 100.;
    uniform float a0, a1, a2, a3, a4, a5, a6, a7, a8;
    uniform float d0, d1, d2, d3, d4, d5, d6, d7, d8;
    
    vec4 live = vec4(1.,1.,1.,1.);
    vec4 dead = vec4(0.,0.,0.,1.);
    
    float singlestate = 1. / (cellstates - 1);
    
    vec2 pixel = 1. / resolution;
    
    float rand(vec2 pos) {
        return mod(fract(sin(dot(pos + time * 0.001, vec2(14.9898,78.233))) * 43758.5453), 1.0);
    }
    
    float val(vec2 pos, vec2 offset) {
        return (texture2D(ppixels, fract(pos + (pixel * offset))).r > 0.) ?
            1. :
            0.;
    }
    
    void main( void ) {
    
        //
        vec2 rules[9];
        rules[0] = vec2(a0, d0);
        rules[1] = vec2(a1, d1);
        rules[2] = vec2(1., d2);
        rules[3] = vec2(a3, 1.);
        rules[4] = vec2(a4, d4);
        rules[5] = vec2(a5, d5);
        rules[6] = vec2(a6, d6);
        rules[7] = vec2(a7, d7);
        rules[8] = vec2(a8, d8);
    
        // Normalised texture coordinates
        vec2 position = ( gl_FragCoord.xy / resolution.xy );
    
        // Paintbrush
        if (length(position-mouse) < brushsize) {
            gl_FragColor = mix(live, dead, step(rand(position), 0.5));
        } else {
    
            // Neighbouring cells
            float sum = 0.;
            sum += val(position, vec2(-1., -1.));
            sum += val(position, vec2(-1.,  0.));
            sum += val(position, vec2(-1.,  1.));
            sum += val(position, vec2( 1., -1.));
            sum += val(position, vec2( 1.,  0.));
            sum += val(position, vec2( 1.,  1.));
            sum += val(position, vec2( 0., -1.));
            sum += val(position, vec2( 0.,  1.));
    
            // Current pixel
            float me = texture2D(ppixels, position).r;
    
            // Lookup into rules array
            vec2 r = rules[int(sum)].rg;
    
            float state = me * r.x + r.y;
    
            vec4 color = vec4(state,state,state,1.);
    
            gl_FragColor = color;
        }
    }
    

    Still very much a WIP.

    I'm currently trying to do all the various parts of the projects in separate sketches, before trying to stitch them together in some way.

    a|x

  • @toneburst Ctrl+O will format Code :)
    https://en.wikipedia.org/wiki/Markdown
    .PDE file?

  • Thanks @nabr. It's more nicely-formatted now :)

    a|x

  • edited May 2017

    I've made some progress.

    // PostFX shader-chaining library
    import ch.bildspur.postfx.builder.*;
    import ch.bildspur.postfx.pass.*;
    import ch.bildspur.postfx.*;
    // GLSL version of Conway's game of life
    PShader conway;
    PGraphics canvas;
    
    // Custom shader pass class
    PostFX fx;
    //PostFXSupervisor supervisor;
    //FeedbackPass feedbackPass;
    NegatePass negatePass;
    
    // GUI library
    import controlP5.*;
    ControlP5 cp5;
    
    float brushSize;
    
    ///////////////////////////
    // SETUP //////////////////
    ///////////////////////////
    
    void setup() {
    
      size(640, 480, P2D);
    
      // Add controls
      cp5 = new ControlP5(this);
      cp5.addSlider("brushSize")
        .setPosition(40, 40)
        .setSize(100, 20)
        .setRange(0.01, 0.05)
        .setValue(0.025)
        .setColorCaptionLabel(color(200,200,200));
    
      // Init shader stuff
      canvas = createGraphics(width, height, P3D);
      canvas.noSmooth();
    
      conway = loadShader("conway.glsl");
      conway.set("resolution", float(canvas.width), float(canvas.height));
    
      // PostFX pass stuff
      //supervisor = new PostFXSupervisor(this);
      fx = new PostFX(this);
      //feedbackPass = new FeedbackPass();
      negatePass = new NegatePass();
    }
    
    ///////////////////////////
    // DRAW ///////////////////
    ///////////////////////////
    
    void draw() {
    
      // Shader uniforms
      conway.set("time", millis()/1000.0);
      float x = map(mouseX, 0, width, 0, 1);
      float y = map(mouseY, 0, height, 1, 0);
      conway.set("brushsize", brushSize); 
      conway.set("mouse", x, y);  
    
      // Draw shader
      canvas.beginDraw();
      canvas.background(0);
    
      canvas.pushMatrix();
    
      canvas.shader(conway);
      canvas.rect(0, 0, canvas.width, canvas.height);
    
      canvas.popMatrix();
      canvas.endDraw();
    
      //
      blendMode(BLEND);
      fx.render(canvas)
        .bloom(0.5, 20, 40)
        //.custom(negatePass)
        .compose();
    
    }
    

    This nicely applies a bloom effect to the output of the conway shader.

    My hope is to replace the negate shader, from the PostFX CustomShaderEffect example with my own, but for the moment, I'm just using the negate shader from the example.

    Unfortunately, if I uncomment the line

    .custom(negatePass)

    I get an OpenGL error

    ERROR: 0:12: Regular non-array variable 'vertColor' may not be redeclared

    Is there a way to fix this, @cansik?

    a|x

  • edited May 2017

    Oops, I did accidentally redeclare the varying 'vertColor' in negateFrag.glsl.

    I think I was messing around with it, and didn't realise I'd saved it.

    Now it works :) Time to try my own shader.

    a|x

  • edited May 2017

    So, now I have:

    // PostFX shader-chaining library
    import ch.bildspur.postfx.builder.*;
    import ch.bildspur.postfx.pass.*;
    import ch.bildspur.postfx.*;
    // GLSL version of Conway's game of life
    PShader conway;
    PGraphics canvas;
    
    // Custom shader pass class
    PostFX fx;
    //PostFXSupervisor supervisor;
    FeedbackPass feedbackPass;
    NegatePass negatePass;
    
    // GUI library
    import controlP5.*;
    ControlP5 cp5;
    
    float brushSize;
    
    ///////////////////////////
    // SETUP //////////////////
    ///////////////////////////
    
    void setup() {
    
      size(640, 480, P2D);
    
      // Add controls
      cp5 = new ControlP5(this);
      cp5.addSlider("brushSize")
        .setPosition(40, 40)
        .setSize(100, 20)
        .setRange(0.01, 0.05)
        .setValue(0.025)
        .setColorCaptionLabel(color(200,200,200));
    
      // Init shader stuff
      canvas = createGraphics(width, height, P3D);
      canvas.noSmooth();
    
      conway = loadShader("conway.glsl");
      conway.set("resolution", float(canvas.width), float(canvas.height));
    
      // PostFX pass stuff
      //supervisor = new PostFXSupervisor(this);
      fx = new PostFX(this);
      feedbackPass = new FeedbackPass();
      negatePass = new NegatePass();
    }
    
    ///////////////////////////
    // DRAW ///////////////////
    ///////////////////////////
    
    void draw() {
    
      // Shader uniforms
      conway.set("time", millis()/1000.0);
      float x = map(mouseX, 0, width, 0, 1);
      float y = map(mouseY, 0, height, 1, 0);
      conway.set("brushsize", brushSize); 
      conway.set("mouse", x, y);  
    
      // Draw shader
      canvas.beginDraw();
      canvas.background(0);
    
      canvas.pushMatrix();
    
      canvas.shader(conway);
      canvas.rect(0, 0, canvas.width, canvas.height);
    
      canvas.popMatrix();
      canvas.endDraw();
    
      //
      blendMode(BLEND);
      fx.render(canvas)
        .bloom(0.5, 20, 40)
        .custom(feedbackPass)
        .compose();
    
    }
    

    And my shader:

    // Mix between input texture and previous frame pixels
    
    #ifdef GL_ES
    precision mediump float;
    precision mediump int;
    #endif
    
    uniform sampler2D texture;
    uniform sampler2D ppixels;
    uniform float feedback;
    varying vec4 vertTexCoord;
    
    void main( void ) {
        vec2 position = vertTexCoord.st;
        gl_FragColor = mix(texture2D(texture, position), texture2D(ppixels, position), feedback);
    }
    

    Which doesn't work. The ppixel texture in the feedback shader seems to contain nothing, so changing the feedback amount just has the effect of mixing between the output of the first pass and black, and making the image darker. I'd intended to create a kind of 'trails' effect.

    Does this mean I have to manage my own FBOs manually, in order to get feedback working?

    I'm able to use ppixels for feedback, while I do the initial drawing of the Conway shader, before I apply the PostFX passes.

    a|x

  • edited May 2017

    Aha.. if I comment out the line

    supervisor.clearPass(pass);

    in my feedbackPass class declaration, it works...

    I've added a function to my feedbackPass class to allow me to change one of the shader uniforms from the draw loop in the main sketch. No idea if this is the correct way to do this, as I'm unfamiliar with Java (and OOP in general, to my shame).

    FeedbackPass.pde:

    class FeedbackPass implements Pass
    {
      PShader shader;
      float feedbackLevel;
    
      public FeedbackPass()
      {
        shader = loadShader("feedback.glsl");
      }
    
      @Override
        public void prepare(Supervisor supervisor) {
        shader.set("feedback", 0.75);
      }
    
      @Override
        public void apply(Supervisor supervisor) {
        PGraphics pass = supervisor.getNextPass();
        //supervisor.clearPass(pass);
    
        shader.set("feedback", feedbackLevel);
        pass.beginDraw();
        pass.shader(shader);
        pass.image(supervisor.getCurrentPass(), 0, 0);
        pass.endDraw();
      }
    
      public void setfeedback(float feedback)
      {
        feedbackLevel = feedback;
      }
    }
    

    And the main sketch .pde:

    // PostFX shader-chaining library
    import ch.bildspur.postfx.builder.*;
    import ch.bildspur.postfx.pass.*;
    import ch.bildspur.postfx.*;
    // GLSL version of Conway's game of life
    PShader conway;
    PGraphics canvas;
    
    // Custom shader pass class
    PostFX fx;
    FeedbackPass feedbackPass;
    
    // GUI library
    import controlP5.*;
    ControlP5 cp5;
    
    float brushSize;
    float feedback;
    
    ///////////////////////////
    // SETUP //////////////////
    ///////////////////////////
    
    void setup() {
    
      size(640, 480, P2D);
    
      // Add controls
      cp5 = new ControlP5(this);
      cp5.addSlider("brushSize")
        .setPosition(40, 40)
        .setSize(100, 20)
        .setRange(0.01, 0.05)
        .setValue(0.025)
        .setColorCaptionLabel(color(200,200,200));
    
      cp5.addSlider("feedback")
        .setPosition(40, 70)
        .setSize(100, 20)
        .setRange(0., 0.95)
        .setValue(0.5)
        .setColorCaptionLabel(color(200,200,200));
    
      // Init shader stuff
      canvas = createGraphics(width, height, P3D);
      canvas.noSmooth();
    
      conway = loadShader("conway.glsl");
      conway.set("resolution", float(canvas.width), float(canvas.height));
    
      // PostFX pass stuff
      //supervisor = new PostFXSupervisor(this);
      fx = new PostFX(this);
      feedbackPass = new FeedbackPass();
    }
    
    ///////////////////////////
    // DRAW ///////////////////
    ///////////////////////////
    
    void draw() {
    
      // Shader uniforms
      conway.set("time", millis()/1000.0);
      float x = map(mouseX, 0, width, 0, 1);
      float y = map(mouseY, 0, height, 1, 0);
      conway.set("brushsize", brushSize); 
      conway.set("mouse", x, y);  
    
      // Draw shader
      canvas.beginDraw();
      canvas.background(0);
    
      canvas.pushMatrix();
    
      canvas.shader(conway);
      canvas.rect(0, 0, canvas.width, canvas.height);
    
      canvas.popMatrix();
      canvas.endDraw();
    
      image(canvas, 0, 0);
    
      // Set feedback level for feedback pass shader
      feedbackPass.setfeedback(feedback);
    
      // Apply passes
      blendMode(BLEND);
      fx.render()
        //.bloom(0.5, 20, 40)
        .custom(feedbackPass)
        .compose();
    
    }
    

    a|x

  • edited May 2017

    @cansik though I've sort of got feedback to work, I'd really like to be able to access previous frames data from inside the custom pass class instance, so I'm not relying on the ppixels class.

    You mentioned saving the texture for the next frame. That's exactly what I need to do :)

    I guess I'd change the shader to something like:

    #ifdef GL_ES
    precision mediump float;
    precision mediump int;
    #endif
    
    uniform sampler2D texture;
    uniform sampler2D oldTexture;
    uniform float feedback;
    varying vec4 vertTexCoord;
    
    void main( void ) {
        vec2 position = vertTexCoord.st;
        gl_FragColor = mix(texture2D(texture, position), texture2D(oldTexture, position), feedback);
    }
    

    Any tips on how I'd do that in my custom pass class definition?

    class FeedbackPass implements Pass
    {
      PShader shader;
      float feedbackLevel;
    
    
      PGraphics previousTex;
    
      public FeedbackPass()
      {
        shader = loadShader("feedback.glsl");
      }
    
      @Override
        public void prepare(Supervisor supervisor) {
        shader.set("feedback", 0.75);
      }
    
      @Override
        public void apply(Supervisor supervisor) {
        PGraphics pass = supervisor.getNextPass();
        //supervisor.clearPass(pass);
    
        shader.set("feedback", feedbackLevel);
        pass.beginDraw();
        pass.shader(shader);
        pass.image(supervisor.getCurrentPass(), 0, 0);
        pass.endDraw();
      }
    
      public void setfeedback(float feedback)
      {
        feedbackLevel = feedback;
      }
    }
    

    I've added a new PGraphics, as I guess I'll need to render to this at some stage, and bind it to the old_texture uniform in my shader. Not sure how I'd go about that though.

    Any tips for a processing noob gratefully accepted.

    a|x

  • edited May 2017

    I have this updated custom pass definition:

    class FeedbackPass implements Pass
    {
      private PShader shader;
    
      private float feedbackLevel;
    
      private PGraphics previousTex;
    
      /////////////////
      // Constructor //
      /////////////////
    
      public FeedbackPass()
      {
        shader = loadShader("feedback.glsl");
        previousTex = createGraphics(width, height, P3D);
        shader.set("oldTexture", previousTex);
      }
    
      @Override
        public void prepare(Supervisor supervisor) {
        shader.set("feedback", 0.75);
      }
    
      @Override
        public void apply(Supervisor supervisor) {
        PGraphics pass = supervisor.getNextPass();
        //supervisor.clearPass(pass);
    
        shader.set("feedback", feedbackLevel);
        pass.beginDraw();
        pass.shader(shader);
        pass.image(supervisor.getCurrentPass(), 0, 0);
        pass.endDraw();
    
        // Update previous texture
        previousTex.loadPixels();
        // Don't know how to get pixels from pass
        //arrayCopy(pass.image.pixels, previousTex.pixels); 
        previousTex.updatePixels(); 
    
      }
    
      ///////////////////// 
      // Mutator methods //
      /////////////////////
    
      public void setfeedback(float feedback)
      {
        this.feedbackLevel = feedback;
      }
    }
    

    I think I just need to work out how to copy the pass pixels to the background texture on line 38. Not sure how to go about that, though.

    a|x

  • edited May 2017

    Too easy...

    This seems to work :)

    /*
      ////////////////////////////////////////
      PostFX custom pass class definition
    
      Dependencies:
      PostFX library
      ////////////////////////////////////////
    */
    
    class FeedbackPass implements Pass
    {
      private PShader shader;
    
      private float feedbackLevel;
    
      private PGraphics previousTex;
    
      /////////////////
      // Constructor //
      /////////////////
    
      public FeedbackPass()
      {
        shader = loadShader("feedback.glsl");
        previousTex = createGraphics(width, height, P3D);
      }
    
      @Override
        public void prepare(Supervisor supervisor) {
           shader.set("oldTexture", previousTex);
      }
    
      @Override
        public void apply(Supervisor supervisor) {
        PGraphics pass = supervisor.getNextPass();
        //supervisor.clearPass(pass);
    
        shader.set("feedback", feedbackLevel);
        pass.beginDraw();
        pass.shader(shader);
        pass.image(supervisor.getCurrentPass(), 0, 0);
        pass.endDraw();
    
        // Update previous texture
        previousTex = pass;  
      }
    
      ///////////////////// 
      // Mutator methods //
      /////////////////////
    
      public void setfeedback(float feedback)
      {
        this.feedbackLevel = feedback;
      }
    }
    

    I have a feeling that setting 'oldTex' to 'pass' like this (line 45) may not be the fastest way to do it.

    I tried arrayCopy, but got a nullPointer exception.

    a|x

  • incidentally, is there a way to stop Markdown attempting to insert links whenever there's an "@Override" line in my code?

    a|x

  • OK, next step: I need to get an image from a webcam into a shader.

    It doesn't look like an instance of the Capture object can have a shader attached to it directly, in the same way a PGraphics instance can.

    I guess this means I need to copy the pixels from the capture to a PGraphics instance. Sounds like this might be expensive, though. Do I literally have to loop through all the pixel array of the Capture, copying to the PGraphics?

    Maybe there's a better way to do the copy, all in one go?

    a|x

  • edited May 2017

    Got image capture working :)

    /*
      ////////////////////////////////////////////////////
      ////////////////////////////////////////////////////
    
      A-Life
    
      Alex Drinkwater 2017
    
      ////////////////////////////////////////////////////
      ////////////////////////////////////////////////////
    */
    
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // LIBRARIES /////////////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    // Video library  
    import processing.video.*;
    
    // PostFX shader-chaining library
    // https://github.com/cansik/processing-postfx
    import ch.bildspur.postfx.builder.*;
    import ch.bildspur.postfx.pass.*;
    import ch.bildspur.postfx.*;
    
    // ControlP5 GUI library
    // https://github.com/sojamo/controlp5
    import controlP5.*;
    
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // GLOBAL VARIABLES //////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    Capture cam;
    PGraphics camFrame;
    
    // Custom shader pass class (requires PostFX)
    PostFX fx;
    FeedbackPass feedbackPass;
    ConwayPass conwayPass;
    
    PGraphics canvas;
    
    // GUI library
    ControlP5 cp5;
    color guiColor = color(200,200,200);
    
    float brushSize;
    float feedback;
    boolean runFX;
    
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // SETUP /////////////////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    void setup() {
    
      size(1280, 720, P2D);
    
      //////////////////
      // Init Capture //
      //////////////////
    
      String[] cameras = Capture.list();
    
      if (cameras.length == 0) {
        println("There are no cameras available for capture.");
        exit();
      } else {
        println("Available cameras:");
        for (int i = 0; i < cameras.length; i++) {
          println(cameras[i]);
        }   
        // The camera can be initialized directly using an 
        // element from the array returned by list():
        cam = new Capture(this, cameras[0]);
        cam.start();     
      }
    
      camFrame = createGraphics(width, height, P2D);
    
      ///////////////////////
      // PostFX pass stuff //
      ///////////////////////
    
      fx            = new PostFX(this);
      feedbackPass  = new FeedbackPass();
      conwayPass    = new ConwayPass();
    
      //////////////////
      // Add controls //
      //////////////////
    
      cp5 = new ControlP5(this);
      cp5.addSlider("brushSize")
        .setPosition(40, 40)
        .setSize(100, 20)
        .setRange(0.01, 0.05)
        .setValue(0.025)
        .setColorCaptionLabel(guiColor);
    
      cp5.addSlider("feedback")
        .setPosition(40, 70)
        .setSize(100, 20)
        .setRange(0., 0.95)
        .setValue(0.80)
        .setColorCaptionLabel(guiColor);
    
       cp5.addToggle("runFX")
         .setPosition(40, 100)
         .setSize(20, 20)
         .setColorCaptionLabel(guiColor)
         .setValue(false);
    }
    
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // DRAW //////////////////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    void draw() {
    
      // Set feedback level for feedback pass shader
      feedbackPass.setfeedback(feedback);
    
      conwayPass.setStartFX(runFX);
      conwayPass.setMouse(map(mouseX, 0, width, 0, 1), map(mouseY, 0, height, 1, 0));
      conwayPass.setBrushSize(brushSize);
    
      if (cam.available() == true) {
        cam.read();
      }
    
      cam.loadPixels();
      camFrame.loadPixels();
      arrayCopy(cam.pixels, camFrame.pixels);
      cam.updatePixels();
      camFrame.updatePixels();
    
      image(camFrame, 0, 0);
    
      // Apply passes
      blendMode(BLEND);
      fx.render()
        //.custom(conwayPass)
        .custom(feedbackPass)
        //.bloom(0.5, 20, 40)
        .compose();
    }
    

    a|x

  • Still having issues with feedback within the custom shader passes, though. I'm going to post about that in a new thread.

    a|x

  • @Lord_of_the_Galaxy I'm a complete Processing noob, I'm afraid. I've messed around with Grey Scott RD shaders in the past, but using Quartz Composer. I'm struggling with implementing feedback effects properly in Processing at the moment, so we're probably in similar positions.

    Hope you get the problem sorted.

    a|x

  • edited May 2017

    @toneburst

    I went few pages back in this thread, i don't know how all of you guys live with this copy paste attitude and then setup .pde for testing (already 10 minutes pass)

    It's 2017 can you zip it ?

    So i (someone) can unzip it and start with debug? Sorry, it's just me, and i'm lazy as hell. I mean, if some1 fear to get a virus or something, github, mediafire, googledrive etc. has now a shared folder, so the user can preview the files befor they get zipped.

    Best

  • Good points. I'm new to this forum, so don't know how things are done here.

    https://www.dropbox.com/s/fkgh0zu2yhrpxdi/Conway_Webcam.zip?dl=0

    Here's a zip of the file.

    a|x

  • It runs very slowly on my old MacBook Pro. I'm pretty sure it's the Graphic instance copying that's required for texture feedback that's slowing things to a crawl. A very similar setup would run at 60fps in Quartz Composer, even on my ancient laptop.

    a|x

  • Also, it doesn't handle different camera output formats well.

    a|x

  • edited May 2017

    @toneburst

    "don't know how things are done here." You the first one who zip it. Im also new to this forum, and it makes me sick to copy paste stuff til i get something runnig.

    Thank you man!

    I mean the users here used to do it for the past 10 years. PHP or whatever script to zip me a source file will take 10 lines of code ?

  • No problem! I like to keep things in separate files, which makes it more difficult to paste the whole thing into a post here. It's getting a bit long for that, anyway, as you observed ;)

    a|x

  • edited May 2017

    I'm kinda thinking I should rewrite the whole thing to use native OpenGL textures and FBOs wherever possible. That way, maybe the whole thing can be kept on the GPU.

    No real idea if this is possible, though.

    a|x

  • I'm also quite interested in Geometry Shaders. I've seen examples of their use in Processing, but don't know how easy it is to achieve.

    a|x

  • edited May 2017

    @toneburst

    Yes, their a bug in your code. @cansik know's a quick fix i think

    It was not working, on my PC i changed few lines to make it work

    https://processing.org:8443/tutorials/video/

    You have different declarations like P2D and then P3D. I found it's best, while debuging your stetch, when you use nummers in the size(nummer, nummer, P3D)
    - and then createGraphic(nummer,nummer) >instead of width, height. Thouse values can evaluate druing the "startup" and "miss" the setup. default is 100x100 px so you would run a createGraphic, or a box(size) Object at a lower res ...etc.

       //
      ////////////////////////////////////////////////////
      ////////////////////////////////////////////////////
    
      //A-Life
    
     //Alex Drinkwater 2017
    
      ////////////////////////////////////////////////////
      ////////////////////////////////////////////////////
      //
    
    // Video library
    import processing.video.*;
    
    // PostFX shader-chaining library
    // https://github.com/cansik/processing-postfx
    import ch.bildspur.postfx.builder.*;
    import ch.bildspur.postfx.pass.*;
    import ch.bildspur.postfx.*;
    
    // ControlP5 GUI library
    // https://github.com/sojamo/controlp5
    import controlP5.*;
    
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // GLOBAL VARIABLES //////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    Capture cam;
    PGraphics camFrame;
    
    // Custom shader pass class (requires PostFX)
    PostFX fx;
    FeedbackPass feedbackPass;
    ConwayPass conwayPass;
    
    PGraphics canvas;
    
    // GUI library
    ControlP5 cp5;
    color guiColor = color(200, 200, 200);
    
    float brushSize;
    float feedback;
    float channelSpread;
    boolean runFX;
    
    //changed
    void captureEvent(Capture video) {
      video.read();
    }
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // SETUP /////////////////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    void setup() {
    
      //changed
      size(640, 480, P3D);
    
      //////////////////
      // Init Capture //
      //////////////////
    
      //changed
      camFrame = createGraphics(640, 480, P3D);
    
      ///////////////////////
      // PostFX pass stuff //
      ///////////////////////
    
      fx            = new PostFX(this);
      feedbackPass  = new FeedbackPass();
      conwayPass    = new ConwayPass();
    
      //////////////////
      // Add controls //
      //////////////////
    
      cp5 = new ControlP5(this);
      cp5.addSlider("brushSize")
        .setPosition(40, 40)
        .setSize(100, 20)
        .setRange(0.01, 0.05)
        .setValue(0.025)
        .setColorCaptionLabel(guiColor);
    
      cp5.addSlider("feedback")
        .setPosition(40, 70)
        .setSize(100, 20)
        .setRange(0.0, 0.90)
        .setValue(0.80)
        .setColorCaptionLabel(guiColor);
    
      cp5.addSlider("channelSpread")
        .setPosition(40, 100)
        .setSize(100, 20)
        .setRange(0.00, 0.07)
        .setValue(0.0)
        .setColorCaptionLabel(guiColor);
    
      cp5.addToggle("runFX")
        .setPosition(40, 130)
        .setSize(20, 20)
        .setColorCaptionLabel(guiColor)
        .setValue(false);
    
      //changed   
      cam = new Capture(this, 640, 480);
      cam.start();
    }
    
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    // DRAW //////////////////////////////////////////////
    //////////////////////////////////////////////////////
    //////////////////////////////////////////////////////
    
    void draw() {
    
      // Set feedback level for feedback pass shader
      feedbackPass.setFeedback(feedback);
      feedbackPass.setChannelSpread(channelSpread);
    
      conwayPass.setStartFX(runFX);
      conwayPass.setMouse(map(mouseX, 0, width, 0, 1), map(mouseY, 0, height, 1, 0));
      conwayPass.setBrushSize(brushSize);
    
      // Copy capture pixels to PGraphics instance
      cam.loadPixels();
      camFrame.loadPixels();
      arrayCopy(cam.pixels, camFrame.pixels);
      cam.updatePixels();
      camFrame.updatePixels();
    
      image(camFrame, 0, 0);
    
      //OpenGL error 1282 at top endDraw(): invalid operation
      //Apply passes
      blendMode(BLEND);
       fx.render()
       .custom(conwayPass)
       .custom(feedbackPass)
       //.bloom(0.5, 20, 40)
       .compose();
    
      //enable for debug
      //image(cam, 0, 0);
    }
    
  • but i looks already crazy and fun. keep digging it!

  • edited May 2017

    @nabr Hmm.. thanks for that.

    I'm not sure it's improved things on my system, though it's quite a cool glitch effect. The camera feed stutters and 'sticks' in quite a nice way, though I think it might give me a bit of a headache if I looked at it for too long.

    It's cool there's a way to force the captured image to a particular size, though. That's what I was looking for, in fact.

    I'll have a look tomorrow, and see if it works better on my (much more recent) work machine.

    Thanks again!

    a|x

  • edited May 2017

    @toneburst

    yes,

    so only this part works, but their is something wrong with the "filter". Run FX gives me the conwayPass only, on/off will switch between cam output and Conway shader

    void draw() {
    
      // Set feedback level for feedback pass shader
      feedbackPass.setFeedback(feedback);
      feedbackPass.setChannelSpread(channelSpread);
    
      conwayPass.setStartFX(runFX);
      conwayPass.setMouse(map(mouseX, 0, width, 0, 1), map(mouseY, 0, height, 1, 0));
      conwayPass.setBrushSize(brushSize);
    
      // Copy capture pixels to PGraphics instance
      cam.loadPixels();
      camFrame.loadPixels();
      arrayCopy(cam.pixels, camFrame.pixels);
      cam.updatePixels();
      camFrame.updatePixels();
    
    
      image(camFrame, 0, 0);
    
    
      //OpenGL error 1282 at top endDraw(): invalid operation
      //Apply passes
      blendMode(BLEND);
       fx.render()
       .custom(conwayPass)
       .custom(feedbackPass)
       //.bloom(0.5, 20, 40)
       .compose();
    }
    
  • @nabr thanks. It's for a student at my work, in fact. She has an idea for an installation for her final project, and I'm trying to implement it in Processing, since that's what her group are taught.

    a|x

  • @toneburst

    the feedback looks great! what is the goal here to make the conway shader work on top of the video output ?

  • edited May 2017

    @nabr thanks re. the feedback! I got a bit sidetracked into messing around with that for a while. Still have a little idea I'd like to integrate into it.

    Am I right in thinking that copying the pixels in the custom PostFX pass definition is a performance bottleneck, do you think? I can't imagine it's done on the GPU.

    a|x

Sign In or Register to comment.