Interactive video with sound input

komats

Hi,

I will soon have to make interactive video. As I know a little bit of Processing language, think I should use it for my project.

I have been searching for inspiration and found one of ideas (that might not be as complex for me as a beginer).

So the main question is this:

Is it possible in Processing to code smth like this:

The main idea is to make some black&white background "come to life" with the sounds. The input sounds make random different blurry color splashes that stays for a while, but doesn't disappear very fast.

Would like to make it a bit complex by having those colorful flashes on top of some video.

Any tuts/videos on mind, where I could learn something similar?

kfrajer

This is totally doable. For the sound effect, you could use either amplitude over time or frequency over time. The latter requires using FFT function to extract the frequency components of each sound buffer which is concurrently being played.

If you are a beginner, i will stick to the amplitude vs. time concept as it is easier to implement. I dunno how much experience you have in processing. So I encourage you to write your first sketch and demonstrate what can you do and show what parts you require some guidance.

For this project, you need to use:

• Minim: Read the input from a mic or play a sound file and access the current sound buffer. You need to also need to get the amplitude of the current buffer (all provided by minim) which it is useful to detect sound. When you detect some sound, you can draw it, exactly as demonstrated in the video you provided.
• Key terms to review in Processing: PGraphics(), mask(), tint(), lerpColor() and map()
• You need the video library to load and play your video.
• You will also need to have an array list to keep track of the objects drawn in the sketch
• Finally you will need to work with classes to simplify the processing of color over time. This is related to the color fading effect, possibly and effect of using time+a pgraphics buffer+tint.

Check the reference for those key terms and don't forget to check the provided examples accessible through the PDE.

Kf

komats

Huge thanks, kfrajer! Will look up your tips. After a week I will start my project, then will update how it went.

komats

So I tried to stick together two codes - one for webcam input --> black&white output, other from library - the Audio Input, that draws colorful circle depending on the sound input.

Its not the main idea, but I just wanted to sea wether it will work - making something colorful on to an video with the sound. And no surprise, the code didn't work. What should I change? Or isn't it even possible to draw over a video that is capturing at the same time?

So this is it (I bolded the parts of sound input, without them the code is working fine )

import processing.video.*; **import processing.sound.*;

AudioIn input; Amplitude rms; int scale=1;**

color black = color(0); color white = color(255); int numPixels; Capture video;

void setup() { size(640, 480); // Change size to 320 x 240 if too slow at 640 x 480 strokeWeight(5);

// This the default video input, see the GettingStartedCapture // example if it creates an error video = new Capture(this, width, height);

// Start capturing the images from the camera video.start();

numPixels = video.width * video.height; noCursor(); smooth();

**// FOR SOUND //Create an Audio input and grab the 1st channel input = new AudioIn(this, 0);

// start the Audio Input
input.start();

// create a new Amplitude analyzer
rms = new Amplitude(this);

// Patch the input to an volume analyzer
rms.input(input);**

}

void draw() { if (video.available()) { video.read(); video.loadPixels(); int threshold = 120; // Set the threshold value float pixelBrightness; // Declare variable to store a pixel's color // Turn each pixel in the video frame black or white depending on its brightness loadPixels(); for (int i = 0; i < numPixels; i++) { pixelBrightness = brightness(video.pixels[i]); if (pixelBrightness > threshold) { // If the pixel is brighter than the pixels[i] = white; // threshold value, make it white } else { // Otherwise, pixels[i] = black; // make it black } } updatePixels();

**// FOR SOUND // adjust the volume of the audio input input.amp(map(mouseY, 0, height, 0.0, 1.0));

// rms.analyze() return a value between 0 and 1. To adjust
// the scaling and mapping of an ellipse we scale from 0 to 0.5
scale=int(map(rms.analyze(), 0, 0.5, 1, 350));
noStroke();

fill(255,0,150);
// We draw an ellispe coupled to the audio analysis
ellipse(width/2, height/2, 1*scale, 1*scale);**

}

koogs

edit post, highlight code, press ctrl-o to format.

you cannot have the code looking like code AND parts of the code in bold. however, when it's formatted as code then you can refer to it using the line numbers.

komats

Ha, finally smth is going! Had to study long time to realize that sound library doesn't work, I had to download minim as you said. But yet I haven't figured out how to make those colorful circles. Now I just put the monitoring sound on top of the video to see if it actually works. Somehow the video isn't looping. Even though I take of video.jump. Any advice, examples how to put some colorful low opacity color splashes as making sound into microphone? :)

import processing.video.*;
import ddf.minim.*;

    Minim minim;
    AudioInput in;
    Movie video;

    void setup(){
      size(640,360);
      frameRate(25);
      video=new Movie(this, "gliemezis.mov");
      video.loop();
      video.speed(4);
      minim = new Minim(this);
      in = minim.getLineIn();

    }

    void movieEvent (Movie video) {
      video.read();
    }

    void mousePressed(){
      video.jump(3);
    }

    void draw(){
    // float r= map(mouseX,0,width,0,6);
     image(video,0,0, width,height);
     //video.speed(r);

     stroke(255);

      // draw the waveforms so we can see what we are monitoring
      for(int i = 0; i < in.bufferSize() - 1; i++)
      {
        line( i, 50 + in.left.get(i)*50, i+1, 50 + in.left.get(i+1)*50 );
        line( i, 150 + in.right.get(i)*50, i+1, 150 + in.right.get(i+1)*50 );
      }

    }

koogs

Please don't post duplicates. Or triplicates.

kfrajer

https://forum.processing.org/two/discussion/24357/is-it-possible-to-adjust-tint-on-a-movie

https://forum.processing.org/two/discussion/17803/how-to-control-tint-of-image-in-processing-with-audio-output-volume

Task 1: Can you draw ellipses on top of your movie and those ellipses to shift from right to left over time?

Task 2: Can you at read the amplitude of the current sound detected by your mic?

Task 3: Can you map this sound amplitude to the height of the screen?

Task 4: Can you generate ellipses every 5 frames (for example) and assign it the current Y position mapped from the detected sound amplitude?

Extra: Paly with the alpha field in color to get some fading effects, or change colors using lerpColor.

Kf

Kf