GSVideo with alpha channel

andres

Re: GSVideo with alpha channel

2 years ago

It seems not not easy to do with gsvideo/gstreamer. There is a gstreamer plugin that supposedly allows to read the alpha channel, alphacolor, but didn't work for me.

The only way I found is to use a GSPipeline object to get the raw byte video buffers without any color conversion (just after they are decoded) and process these buffers directly in the sketch. Something like this:

import codeanticode.gsvideo.*;
GSPipeline pipeline;
PImage img;
void setup() {
size(200, 200);
pipeline = new GSPipeline(this, "filesrc location=" + dataPath("z.mov") + " ! qtdemux ! ffdec_png", GSVideo.RAW);
pipeline.loop();
img = createImage(200, 200, ARGB);
}
void pipelineEvent(GSPipeline p) {
p.read();
}
void draw() {
if (pipeline.data != null) {
println("Data size: " + pipeline.data.length);
println("Data caps: " + pipeline.dataCaps);
background(255, 0, 0);
line(0, height/2, width, height/2);
line(width/2, 0, width/2, height);
img.loadPixels();
byte[] data = pipeline.data;
for (int i = 0; i < width * height; i++) {
// The data array contains byte values in the range
// [-128, 127], so we need to convert to [0, 255]:
int b = int(data[4 * i + 0]);
int g = int(data[4 * i + 1]);
int r = int(data[4 * i + 2]);
int a = int(data[4 * i + 3]);
img.pixels[i] = color(r, g, b, a);
}
img.updatePixels();
image(img, 0, 0);
}
}

I tested this sketch (on Linux) with a video including alpha channel, which I generated from a series of png images using these instructions. Note that you have to explicitly indicate the video demuxer and decoder elements, qtdemux and ffdec_png in this case. Also, this method is not very fast because of the per-pixel operations, although it could be made a little more efficient I guess.

The support for videos with alpha channel in JMCVideo seems better.

dawouta

Re: Re: GSVideo with alpha channel

2 years ago

Hey Andres.

That's awesome thank you for your reply, I will certainly give this option a go. Unfortunately the MT4j framework uses GSVideo for video textures and I'm already bound into this library so JMCVideo is not really an option at this point. Despite saying I wouldn't use png sequences I gave it a go ( for lack of any other options ) and found this to be a suitable workaround. The only thing is that loading the images on the fly is a bit slow - so now I just cache all required sequences at startup - given that it's an installation and only needs to be started up once a day - it works ok for me.

I must say that excluding an easy option for including an alpha channel seems to me to be somewhat of an oversight. Anyway thanks once again for the reply - it certainly is very helpful and the first legitimate response I've seen to this question.

akiersky

Re: GSVideo with alpha channel

2 years ago

Hi Everybody!

It looks like the new GSVideo (0.99) http://codeanticode.wordpress.com/2011/05/16/gsvideo-09-release/#more-1303 enables alpha in your video files! and it works great with hd video!

Side note, Andres, it looks like the example listed on your site above uses a movieEvent, which as far as I can tell does not get called. I also get this warning twice at initialization, but it seems to still work after that:

** (GSVideo:1752): WARNING **: gstvideo: failed to get caps of pad playsink0:video_raw_sink

here's how I am using it as an external class:

import processing.opengl.*;
import codeanticode.gsvideo.*;
import codeanticode.glgraphics.*;
public class BGvideo {
GSMovie myMovie;
GLTexture tex;
public BGvideo (String videoFile, PApplet app) {
app.registerDraw(this);
myMovie = new GSMovie(app, videoFile );
tex = new GLTexture(app);
myMovie.setPixelDest(tex);
myMovie.loop();
}
public void draw () {
myMovie.read();
if (tex.putPixelsIntoTexture()) {
background(255, 0, 0);
image(tex, 0, 0, 1920, 487);
}
}
}

and I call it simply like this:

BGvideo bgv = new BGvideo("Untitled.mov", app);

Works great! thanks for all the great work making a fantastic video lib!

ak

andres

Re: GSVideo with alpha channel

2 years ago

Interesting, I didn't try videos with alpha using the new GSVideo-GLGraphics mode. Great to know that they work!

Thanks for the code!

dawouta

Re: GSVideo with alpha channel

2 years ago

Wow! That is fantastic news indeed and hopefully will make my life a lot easier! Thanks for letting us know.

4ndr01d3

Re: GSVideo with alpha channel

2 years ago

Hey There....

Thanks for the info... it is really cool library, full of possibilities!

I did a couple of modifications to the code of @akierski and it runs nicely! See code below. Thanks! However there I'm force to use GLConstants.GLGRAPHICS as the rendering engine. the problem is that I want to include something like this into MT4j and the allowed modes there are OPENGL and P3D. When I change in the code below o use OPENGL it does not work and the whole texture is transparent. Is there any way to make it work with OPENGL mode?

Thanks,

Gustavo.

import processing.core.PApplet;
import codeanticode.glgraphics.GLConstants;
import codeanticode.glgraphics.GLTexture;
import codeanticode.gsvideo.GSMovie;

public class GSVideoLoop extends PApplet {

GSMovie myMovie;
private GLTexture movieTexture;

public void setup() {
size(1100, 600, GLConstants.GLGRAPHICS);
myMovie = new GSMovie(this, "video.mov");
movieTexture = new GLTexture(this);
myMovie.setPixelDest(movieTexture);
myMovie.loop();
}

public void movieEvent(GSMovie myMovie) {
myMovie.read();
}

public void draw() {
if (movieTexture.putPixelsIntoTexture()) {
background(25, 0, 200);
image(movieTexture, 0,0);
}
}

public static void main(String args[]) {
PApplet.main(new String[] { GSVideoLoop.class.getName() });
}

}

andres

Re: GSVideo with alpha channel

2 years ago

In principle, the gsvideo-glgraphics integration requires the glgraphics renderer to function properly, since it uses some functionality not available in the opengl renderer.

On the other hand, glgraphics is just an extension of the o pengl renderer, and should behave in exactly the same way. So one solution for this problem would be to modify MT4j so it accepts glgraphics as a renderer.

Another option is to extract from GLTexture the code that handles the buffering mechanism, and put it in a custom class like the BGVideo suggested by ak. For example:

import processing.opengl.*;
import javax.media.opengl.*;
import codeanticode.gsvideo.*;
import java.lang.reflect.*;
import java.nio.*;
GLMovie glm;
void setup() {
size(640, 480, OPENGL);
background(0);
glm = new GLMovie("station.mov", this);
}
void draw() {
}
public class GLMovie {
GSMovie mov;
int[] texID = {0};
PGraphicsOpenGL pgl;
GL gl;
Method disposePixelsMethod = null;
Object diposePixelsHandler = null;
LinkedList<PixelData> pixelBuffer = null;
int maxBuffSize = 3;
boolean disposeFramesWhenPixelBufferFull = false;
public GLMovie (String videoFile, PApplet app) {
app.registerDraw(this);
mov = new GSMovie(app, videoFile);
mov.setPixelDest(this);
mov.loop();
pgl = (PGraphicsOpenGL) app.g;
}
public void draw () {
gl = pgl.beginGL();
if (mov.ready()) {
mov.read();
if (putPixelsIntoTexture()) {
gl.glViewport(0, 0, width, height);
gl.glClear(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT);
gl.glMatrixMode(GL.GL_PROJECTION);
gl.glPushMatrix();
gl.glLoadIdentity();
gl.glOrtho(0.0, width, 0.0, height, -10, 10);
gl.glMatrixMode(GL.GL_MODELVIEW);
gl.glPushMatrix();
gl.glLoadIdentity();
gl.glEnable(GL.GL_TEXTURE_2D);
gl.glBindTexture(GL.GL_TEXTURE_2D, texID[0]);
gl.glBegin(GL.GL_QUADS);
gl.glTexCoord2f(0.0, 1.0); gl.glVertex2f(0.0, 0.0);
gl.glTexCoord2f(1.0, 1.0); gl.glVertex2f(width, 0.0);
gl.glTexCoord2f(1.0, 0.0); gl.glVertex2f(width, height);
gl.glTexCoord2f(0.0, 0.0); gl.glVertex2f(0.0, height);
gl.glEnd();
gl.glBindTexture(GL.GL_TEXTURE_2D, 0);
gl.glMatrixMode(GL.GL_PROJECTION);
gl.glPopMatrix();
gl.glMatrixMode(GL.GL_MODELVIEW);
gl.glPopMatrix();
}
}
pgl.endGL();
}
/**
* Sets obj as the pixel source for this texture. The object must have
* a public method called disposeBuffer(Object obj) that the texture will
* use to release the int buffers after they are copied to the texture.
*/
public void setPixelSource(Object obj) {
diposePixelsHandler = obj;
try {
disposePixelsMethod = obj.getClass().getMethod("disposeBuffer", new Class[] { Object.class});
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* If there are frames stored in the buffer, it removes the last
* and copies the pixels to the texture. It returns true if that
* is the case, otherwise it returns false.
*/
public boolean putPixelsIntoTexture() {
if (pixelBuffer != null && 0 < pixelBuffer.size() && disposePixelsMethod != null) {
PixelData data = pixelBuffer.remove(0);
gl.glEnable(GL.GL_TEXTURE_2D);
if (texID[0] == 0) {
gl.glGenTextures(1, texID, 0);
gl.glBindTexture(GL.GL_TEXTURE_2D, texID[0]);
gl.glTexParameteri(GL.GL_TEXTURE_2D, GL.GL_TEXTURE_MIN_FILTER, GL.GL_LINEAR);
gl.glTexParameteri(GL.GL_TEXTURE_2D, GL.GL_TEXTURE_MAG_FILTER, GL.GL_LINEAR);
gl.glTexImage2D(GL.GL_TEXTURE_2D, 0, GL.GL_RGBA, data.w, data.h, 0, GL.GL_RGBA, gl.GL_UNSIGNED_BYTE, null);
} else {
gl.glBindTexture(GL.GL_TEXTURE_2D, texID[0]);
}
gl.glTexSubImage2D(GL.GL_TEXTURE_2D, 0, 0, 0, data.w, data.h, GL.GL_RGBA, GL.GL_UNSIGNED_BYTE, data.rgbBuf);
gl.glBindTexture(GL.GL_TEXTURE_2D, 0);
return true;
}
return false;
}
/**
* Sets the size of the pixel buffer, in number of frames. When this size is
* reached, new frames are dropped.
*/
public void setPixelBufferSize(int n) {
maxBuffSize = n;
}
/**
* Returns how many frames are currently stored in the pixel buffer.
*/
public int getPixelBufferUse() {
return pixelBuffer.size();
}
/**
* Determines the behavior of the buffering. When the argument is true,
* then new frames are disposed when the buffer is already full, otherwise
* they are not. In the later case, and depending on how the buffer generation
* method works, might result in the buffers being stored somewhere else
* and being resent later.
*/
public void delPixelsWhenBufferFull(boolean v) {
disposeFramesWhenPixelBufferFull = v;
}
/**
* This is the method used by the pixel source object
* to add frames to the buffer.
*/
public void addPixelsToBuffer(Object natBuf, IntBuffer rgbBuf, int w, int h) {
if (pixelBuffer == null) {
pixelBuffer = new LinkedList<PixelData>();
}
if (pixelBuffer.size() + 1 <= maxBuffSize) {
pixelBuffer.add(new PixelData(natBuf, rgbBuf, w, h));
} else if (disposeFramesWhenPixelBufferFull) {
// The buffer reached the maximum size, so we just dispose the new frame.
try {
disposePixelsMethod.invoke(diposePixelsHandler, new Object[] { natBuf });
} catch (Exception e) {
e.printStackTrace();
}
}
}
protected class PixelData {
int w, h;
Object natBuf; // Native buffer object.
IntBuffer rgbBuf; // Buffer viewed as int.
PixelData(Object nat, IntBuffer rgb, int w, int h) {
natBuf = nat;
rgbBuf = rgb;
this.w = w;
this.h = h;
}
void dispose() {
try {
// Disposing the native buffer.
disposePixelsMethod.invoke(diposePixelsHandler, new Object[] { natBuf });
natBuf = null;
rgbBuf = null;
} catch (Exception e) {
e.printStackTrace();
}
}
}
}

4ndr01d3

Re: GSVideo with alpha channel

2 years ago

Thanks a lot Andres!

I will carefully study the code that you shared and let you guys know if that solve my issue...

Cheers,

Gustavo.

4ndr01d3

Re: GSVideo with alpha channel

2 years ago

Hey there,

I try a different approach and is almost working. The thing is that we are creating the videos we can actually create a video that contains the alpha information and then we can use its frames as a mask for each image of the video I want to display. Here is my code:

package sfda.view.components.video;

import org.mt4j.components.TransformSpace;

import org.mt4j.components.visibleComponents.shapes.MTRectangle;

import codeanticode.gsvideo.GSMovie;

import processing.core.PImage;

import processing.core.PApplet;

public class VideoPlayer3 extends MTRectangle {

private PApplet app;

private GSMovie myMovie,myAlpha;

private PImage movieTexture,alphaTexture;

///////////////////////////////////////////////////////////////////////////////////////

public VideoPlayer3 ( PApplet app, String moviePath,String alphaPath) {

super( app, 400, 400 );

this.setNoFill(true);

this.setNoStroke(true);

this.app = app;

myMovie = new GSMovie(app, moviePath);

movieTexture = new PImage();

myMovie.setPixelDest(movieTexture);

myMovie.setEventHandlerObject(this);

myAlpha = new GSMovie(app, alphaPath);

alphaTexture = new PImage();

myAlpha.setPixelDest(alphaTexture);

myAlpha.setEventHandlerObject(this);

myAlpha.play();

myMovie.loop();

}

///////////////////////////////////////////////////////////////////////////////////////

public void movieEvent(GSMovie myMovie2) {

if (myMovie2==myMovie){

if (myAlpha!=null){

myAlpha.jump((int)myMovie.frame());

myAlpha.read();

}

myMovie.read();

}

}

///////////////////////////////////////////////////////////////////////////////////////

@Override

public void destroy (){

if( myAlpha != null ){

myAlpha.dispose();

myAlpha.delete();

}

myMovie.dispose();

myMovie.delete();

super.destroy();

}

public void updateComponent(long timeDelta){

this.setWidthLocal(myMovie.getSourceWidth());

this.setHeightLocal(myMovie.getSourceHeight());

movieTexture = myMovie.get();

alphaTexture.removeCache(this);

alphaTexture = myAlpha.get();

if(movieTexture.width==alphaTexture.width && movieTexture.height==alphaTexture.height)

movieTexture.mask(alphaTexture);

app.image(movieTexture, this.getPosition(TransformSpace.RELATIVE_TO_PARENT).getX()-this.getWidthXYRelativeToParent()*0.5f, this.getPosition(TransformSpace.RELATIVE_TO_PARENT).getY()-this.getHeightXYRelativeToParent()*0.5f);

}

}

And I said I almost got it because I still haven't been able of fully synchronise both videos... there is a couple of frames of diference in between, here I am using the method jump to go to the same frame when reading the video... but still a sync issue there. Any ideas???

Thanks,

Gustavo.

Ben Purdy

Re: GSVideo with alpha channel

1 year ago

I know this thread is kind of old but I just spent many hours getting this to work so hopefully someone else out there will find my experience helpful!

I'm creating video content in after effects and using processing to layer individual video elements for a live performance, so I need video with alpha channel support. After lots of dead-ends and frustrating trial/error, I finally got it working. In short, getting the compression settings figured out was the key. Several different video formats allow "RGB + Alpha" in the AfterEffects output settings, but only one has worked for me with GSVideo.

I'm using AferEffects CS5, Windows 7 x64, GSVideo 1.0.0, and GLGraphics 1.0.0. When rendering from AfterEffects, I configure things like this:

1) Set the format to "QuickTime"

2) Open the "Format Options..." dialog and then select the option called "Animation"

3) Back in the main dialog, choose "RGB + Alpha" from the "channels" list.

Note #1: After reading this thread I first tried using "PNG" from the "format options" list but that didn't work for me.

Note #2: Due to a (possible?) bug regarding GLGraphicsOffscreen and the treatment of transparent pixels, rendering partially transparent video to an offscreen canvas will give you inconsistent results. See this thread: https://forum.processing.org/topic/glgraphics-offscreenbuffer-aplha-semitransparency-bug

Replies(10)

Re: GSVideo with alpha channel

Re: Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Re: GSVideo with alpha channel

Statistics

Tags

Actions

Related Posts