... it should not be any reason in based on which the CPU consume grows:

The video library is simply a binding wrapper for GStreamer, a framework written in C: :-B
https://en.Wikipedia.org/wiki/GStreamer

Java's GC can't touch it! Instead, in order to actually release its hardware resources, we need to explicitly command it via method dispose(). :-$

PraxisLIVE is an open-source hybrid visual IDE and runtime for (live) creative coding. It offers a powerful alternative environment for working with the Processing library. Easily create projections, interactive spaces, custom VJ tools, sonic performance instruments, media for theatre; use it as a live digital sketchbook, test out ideas, experiment with code ...

Features include -

• Intuitive graphical node-based patching, where every component is similar to a Processing sketch - mix multiple sketches together without the overhead.
• Wide range of pre-built components.
• Real-time live programming - rewrite any component as it is running, without missing a frame, for live coding or faster creative development.
• Integrated GStreamer 1.x support with re-codeable playback and capture components.
• Unlimited number of pipelines, running locally or remotely - run in multiple processes for improved performance or crash-free multiple windows; run across multiple machines for large projects or live-coding a Pi.
• Audio library with JACK support, and ability to re-code synthesis down to the per-sample level.
• Integrated GUI, MIDI, TinkerForge, OSC and Syphon/Spout bindings.
• Threading done right - entirely lock-free communication between pipelines, locally and remotely; background resource loading and management.
• Compositing fixed - pre-multiplied alpha ensures correct blending across multiple transparent surfaces.
• Professional IDE base (Apache NetBeans) with full code-completion and support for Java 8 features.
• Fully FLOSS stack, including bundled OpenJDK on Windows and macOS.

Recent changes include -

• PraxisCORE runtime is now LGPL, with support for standalone projects.
• Support for using any Java libraries, and access to underlying Processing PApplet, images and graphics for library integration (eg. HE_Mesh above).
• Data root pipelines and improved reference handling for building custom background tasks, IO, passing around geometry data, etc.
• Most built-in components now re-codeable, including video composite, mixer, player and capture.
• Improved UI, with start page tasks for installing examples and additional components; updated graph UI (with bezier curves!)

Download from - https://www.praxislive.org
Online documentation (still being updated for v4) - http://praxis-live.readthedocs.io

The Processing pre-processor does two things, structural changes and syntax changes. It is the syntax changes (float/double and color, etc) that are a PITA, because they make it difficult to keep up-to-date with Java changes. In PraxisLIVE I have some similar structural support, automatically wrapping in a class, making things accessible, etc. However, I don't support the syntax stuff and therefore supporting Java 8 is easy - and lambdas / method references are great - can really lead to some easier to understand code!

Java 10/11 introduce type inference eg. var. And raw / multi-line Strings are also coming!

Java 12/13 next year are intended to support record (data) types, value types, expressions in switch statements, etc. I have a photo of a slide from recent JAlba unconference (below) which had a session on this. So, really the only thing Kotlin has that I'd like and isn't coming to Java in the next year is better null support. Spoke to a couple of other Java devs recently who'd started with Kotlin and were now finding it frustrating, although I probably know more who love it! :-)

My primary project, PraxisLIVE, is based on Apache NetBeans (which I'm also involved with), and I'm waiting for the full release of NetBeans v9.0 with Java 9+ support before completing migration there. I only intend to support Java LTS releases anyway (eg. 11 and 14), which I believe is the same as Processing's plan. Other libraries such as GStreamer bindings which Processing Video also uses are working / tested on Java 9+, but the code is not even being migrated beyond Java 1.6 syntax just yet.

Hi, thank you very much for this infos. I will check the GStreamer PlayBin, I hope I will understand it.

I have to try :-)

@tom_tm - the workarounds I was thinking (aside from caching some images as others have described)

• look at using the ImageIO library in Java directly. Keep a single TYPE_INT_RGB BufferedImage, and pass that into a PImage - this should wrap the same pixels array - see https://github.com/processing/processing/blob/master/core/src/processing/core/PImage.java#L299 draw each image you load into the BufferedImage, probably with a call to PImage loadPixels() updatePixels() either side

• look for a faster Java JPEG loader (Google suggests a few options) and do the same trick to update the pixels array directly.

• use the video library - the GStreamer PlayBin backing the Video component is quite happy to play JPEG files as much as movies! It's pretty fast too.

@KMat - cool, glad you're sorted. GStreamer 0.10 is a little on its last legs and hasn't been updated in years, so not entirely surprised! The Java bindings have also had a lot of fixes since then.

Yes, if you are interested in trying with Praxis LIVE too, that error is probably because you haven't got a Java 8 JDK installed. Unlike Processing, it doesn't ship with an included JDK ... yet - inbound v4 will.

@KMat - if you haven't already, have a look at the beta version of Processing Video v2 as that is also using GStreamer 1.x like the Pi library and might improve matters.

Also, I'd be interested if you get better performance in Praxis LIVE - I'm using a different, and potentially better performing, way to get the GStreamer video data into P2D/P3D. If it is any smoother then there may be some stuff I can contribute.

Finally note that one of the reasons the Pi is so smooth is that the GL Video library is doing direct transfer of GPU decoded video to OpenGL. Support is not there yet in the underlying GStreamer Java bindings for this to work across platforms, but it is something I'm looking into.

I have no idea why .avi works and .mov and .mp4 not. I already have similar problem and it was libgstreamer and gstreamer package that were out of date. Late april there is a nieuw ubuntu long time support. It will be good to focus on it, maybe gstreamer package change and everything is to restart.

@matthieu - glad you've got something working. However, I'd still be really interested in finding out why that file isn't working if it's something you can share or point me at? I should have said, but I'm the maintainer of the underlying GStreamer bindings, so would really like to track down whether there's an issue there or in how the Processing Video library is using them.

That doesn't look good! (well, it kind of does ;-) )

Can you try some different video files / formats? Is the file somewhere public? I wouldn't mind testing it in the underlying GStreamer bindings. Does it play fine in the OS video player?

Pinging @gohai too!

You need to use the beta of version 2 of the video library that supports GStreamer 1.x. Not sure if there's an automatic install for that yet - see https://github.com/processing/processing-video/releases/tag/r3-v2.0-beta1

It's a lot bigger than it needs to be for you because of the included Windows and Mac binaries. :-/

The 32 bit message is an error in this case - that's not actually the problem at all.

Hi, I'm trying to read a video on linux with the processing video library without result. The output says that it run on 32 bits and I'm of course on 64. I'm trying to install gstreamer 0.10 but on ubuntu 17.10, it's no avaible anymore. What must I do ?

Also might be worth trying the beta of v2 of the video library https://github.com/processing/processing-video/releases using GStreamer 1.x. Vague recollections of an old GStreamer 0.10 bug with upside down AVI's. Specific to certain colour formats IIRC.

@Yaser

Thank you for your post and interest in GSoC w/ Processing! I believe the Video library is a great project, and the Processing community can greatly benefit from the fruits of your work.

Just a couple quick thoughts on the points you mentioned:

1- Device listing

The problem here is that GStreamer itself currently does not implement the necessary functionality to do capture device enumeration on macOS. I am sure they would not mind patches, but this would (a) require quite some involved Objective C, and (b) we have no control over their release timeline.

2- OpenGL

I would strongly advise against implementing any of this before getting at least one release of the Video library based on modern GStreamer out of the door.

I believe that making this switch, while not causing any unnecessary regressions, and fixing as many bugs as possible on the way, should be the goal really. Otherwise we'll end up with yet another video library, and a bunch of users having to stick with the "legacy" one because something does not quite work the way they expect it to in the new / OpenGL one.

@codeanticode @gohai @neilcsmith_net

Thanks for all of your valuable thoughts and discussions. After reading your comments and considering my limited time until proposal submission period, I've decided to put my energy into the video library for this year. I'm trying to summarize what I've learned so far and laid out a plan for myself. I've itemized the features needed to be implemented and want to clarify steps to approach them.

1- Make the device listing stable and make sure capturing works across all platform.

I've been reading GStreamer tutorials and playing with examples for a while. The command line tool seems to provide very detailed and helpful feedback. I thought along with other efforts to make capturing works across all platforms, providing appropriate feedback from GStreamer to Processing console would be helpful, at least for debugging issues we receive from other users with different devices and platforms.

For the purpose of testing, I have access to Win 8.1 and Win 10 and multiple Linux version through VM. Also, I have an RPi which can be used for testing.

2- Improve the OpenGL integration for better playback performance

After a bit of study, I understand that OpenGL integration would be beneficial especially if we want to do any transformation on the video frame. It seems that the video library is not intended to do anything with video stream other than playing and capturing. Then using OpenGL, we are just uploading and downloading frames to GPU for no good reason. I don't have knowledge about how OpenGL currently integrated into the Processing, but if we want to play/capture a stream, transform and then render it, it might be useful to streamline the steps in the GPU and let OpenGL does some steps through GStreamer.

3- As @neilcsmith_net suggested, providing a mechanism to download and extract OS-Specific GStreamer libraries. Also, when exporting sketches, put only required GStreamer plugins to a have a smaller output.

I don't know how or even if possible to set a mechanism to download and extract OS-Specific libraries at the time of installing a new library within the Processing application. But about recognizing a subset of plugins to export with a sketch, I've watched this presentation (thanks @gohai for mentioning) to get a sense of how we can use debugging feature of GStreamer.

Please comment if you see I'm in a wrong direction in any part.

As it doesn't works with processing, I'm thinking about gstreamer. Is there something to install ?

It's definitely not a issue with Java, as the GLVideo library uses GStreamer for video decoding and handling of the audio

I would forgo OpenGL at this point - the GStreamer API for it isn't even stable.

@codeanticode - bindings for OpenGL API's in GStreamer are targeted for 1.0 of gst1-java-core. There are also ways of speeding up the current texture upload, particularly if full OpenGL is available. There are definitely some issues around threading and pipeline control in the current video library that could do with addressing.

In terms of the video library size, I still think that a mechanism for downloading and extracting the OS-specific libs directly from GStreamer would be worth looking at, and then some mechanism for making a cross-platform sketch export with only the required plugins? Generic parts of that would definitely be something that could live / be maintained upstream.

I'm not sure LWJGL3 is as purely focused on games these days, despite the name! ;-) But have you also given any thought to abstracting over libGDX btw? Just a thought (I used to have a ~75% P2D renderer based on code forked from libGDX over LWJGL2) - it could reduce the amount of work required.

@gohai - RPi support with LWJGL would be a major requirement for me too, so happy to help with that in any way I can!

I agree with @codeanticode - a bit of a general clean-up (or potentially re-implementation while keeping the same interfaces) would be a good idea. There is some historic code in there that is no longer used, which made it a bit hard to understand at times. GStreamer also has moved a lot in the mean time since certain workarounds were developed, so it might be worth revisiting them, and seeing if this is still the best available way. And having Capture and Movie derive from a shared Video class (or similar) sounds good as well!

Regarding RPi support for LWJGL: this is something I'd be happy to help with, so this shouldn't be a deterrent ;)