GLSL and video Texture

vjjv

hello everyone. Im trying to pass a video as a texture into a fragment shader. but the sketch crash when i run it.

here is the code:

  import processing.video.*;
import peasy.*;
import peasy.org.apache.commons.math.*;
import peasy.org.apache.commons.math.*;

PeasyCam cam;
PShader sh;
float count;
Movie mov;
PGraphics p;

void setup (){

size(1440, 900, P3D );
mov = new Movie(this, "osc_noc.mov");
mov.play();
p = createGraphics(width,height);
cam = new PeasyCam(this, 500);
sh = loadShader("basicShaderFrag.glsl", "basicShader.glsl");
}

void movieEvent(Movie m) {
  m.read();
}

void draw(){
background(0);
shader(sh);
count +=0.09;
sh.set("u_time", count);

sphere(100);
p.beginDraw();
p.background(0);
p.image(mov, 0, 0, 200, 200);
p.endDraw();
sh.set("tex",p);
// image(p, 5, 260 ,200, 200);
}

#version 150

uniform mat4 transform;
uniform sampler2D tex;

in vec4 position;
in vec2 texCoord;
in vec3 normal;

out vec2 TexCoord;

void main(){

  TexCoord = texCoord;
  gl_Position = transform * position;
}


#ifdef GL_ES
precision mediump float;
#endif

#define PI 3.14

in vec2 TexCoord;
uniform float u_time;
uniform sampler2D tex;

void main(){
  vec2 uv = TexCoord;

  gl_FragColor = vec4(texture(tex, TexCoord));
}

a white screen appears, and next it close. the console just say: "Finished". it may be a bug? i could pass a PImage as a texture. but when i link the fragment and shader program into the sketch folder then crash. ..

nabr

What i can say, the video lib for capture frames from webcam is a bit buggy.
You have to experiment, go through trail and error.

https://forum.processing.org/two/discussion/comment/97331/#Comment_97331

The link +code still works for me, but now i have a second software installed. The first camera aviable in the array is shutdown trough software emulated cam, i need this setup and changed it manually.

So i have to print all modes as an array

//
import processing.video.Capture;
for (String name : (String[])Capture.list()) println(name);

Then i know that i can work on my system eg.
name= "someName" , size=160x120, fps=30

String[] cameras = Capture.list();
cam = new Capture(this, cameras[2]);

The most important part when work with shaders is to wait, check when the next frame is available

//
if (cam.available() == true) {
    cam.read();
    shader.set("textur", cam);
     }

If everything goes wrong look at the source

Reference for Capture

EDIT 1: sorry man totally misread your question:
it works here. https://github.com/gohai/processing-glvideo/tree/master/examples/VideoMappingWithShader

vjjv

Thanks. finally i could pass a video as a texture, but directly, avoiding PGraphics ..dont know why, but PGraphics + video + shaders crash!

kfrajer

So your current code plus nabr suggested changes means your code works and the question is solved?

This next link is from P2. Not sure if it could help: https://forum.processing.org/one/topic/how-to-use-a-fragment-shader-on-a-pgraphics-in-processing-2-0xx.html

What I infer from your code: you are projecting a texture but you are trying to use the code from another source. Is that right? Because the shaders tutorial show how to do something similar.

Kf

cansik

If you create a P3D scene (defined in the size() method), then all of your PGraphics objects created by createGraphics() are of that kind. But if you are working with video files, you have to use a P2D PGraphics object, because video is usually 2-dimensional.

I think you have to read more about 2D and 3D offscreen canvas and when to use which one.

Here is an example on how to use an incoming movie image, render it to a offscreen canvas with shading and then using the result as texture for a shape in a 3D onscreen canvas. You can also download the whole sketch (with example files here).

import processing.video.*;
import peasy.*;

PShader sobelShader;
PGraphics buffer;

Movie mov;

PeasyCam cam;

float canvasWidth = 480;
float canvasHeight = 360;

void setup()
{
  size(500, 500, P3D);
  cam = new PeasyCam(this, 500);

  // load movie
  mov = new Movie(this, "city.mov");
  mov.loop();

  sobelShader = loadShader("sobelFrag.glsl");
  buffer = createGraphics(mov.width, mov.height, P2D);
  buffer.shader(sobelShader);
}

void draw()
{
  // shade the incoming movie image
  buffer.beginDraw();
  buffer.background(0, 0);
  buffer.image(mov, 0, 0);
  buffer.endDraw();

  // create 3d scene
  background(0);

  pushMatrix();

  // rotate for more interesting 3d magic
  rotateX(radians(frameCount % 360));
  rotateZ(radians(frameCount % 360));

  // use shaded video as texture in the 3d scene
  rectMode(CENTER);
  beginShape();
  textureMode(IMAGE);
  texture(buffer);
  vertex(canvasWidth / -2f, canvasHeight / -2f, 0, 0);
  vertex(canvasWidth / 2f, canvasHeight / -2f, buffer.width, 0);
  vertex(canvasWidth / 2f, canvasHeight / 2f, buffer.width, buffer.height);
  vertex(canvasWidth / -2f, canvasHeight / 2f, 0, buffer.height);
  endShape();

  popMatrix();
}

void movieEvent(Movie m) {
  m.read();
}

nabr

@cansik nice!

but i wont run on my machine :/

// won't run on windows 10 processing 3.3.6
buffer = createGraphics(mov.width, mov.height, P2D);

---

// possible solution

mov = new Movie(this, "city.mov");
// github.com/processing/processing-video/blob/master/examples/Movie/Frames/Frames.pde
mov.play();
mov.jump(0);
mov.loop();

print(mov.width); // 960

sobelShader = loadShader("sobelFrag.glsl");
buffer = createGraphics(mov.width, mov.height, P2D);
// etc.

mov.width is zero at init time i guess.
(sobelFrag.glsl has to be in the data folder, just saying)

@cansik NP.

update:

mov = new Movie(this, "city.mov");    
mov.read(); //  grab the first frame     
mov.loop();  

GoToLoop

https://Forum.Processing.org/two/discussion/23389/resizing-a-movie-possible-bug#Item_4

cansik

@nabr oh yeah that is because I have the pre-release of the video library installed (2.0-beta1). This is based on gstreamer 1.0 and runs much smoother.

Thank you for the comment :)