Sound reactive visual, swapping movie clips

edited August 2017 in Library Questions


I am trying to figure out a way to create a sound reactive installation in which depending on the volume of the input sound from the computer mic it displays a specific movie clip. For example when the volume is low it plays the square graphic, medium level for the circle and swapping to a triangle graphic for the high volume. I looked around but the only samples I found are computer generative which applies effects to the image such as color and size conversion.

If you can please point me to the right path I would really appreciate it. Thank you!



  • You have a problem with two parts. A) work out whether a sound is low, medium or high volume and B) show something different in each case.

    You can do the second without worrying about the first - the loudness will just be a parameter into the display code. So write that first.

    The first part is just one of the minim library examples.

  • @koogs===

    i agree with your hints; yet i am not sure: that is "theory" because the sound "loudness" can change very quickly from min to max, too quickly for the video play stop pause toggle. Of course i am wrong if you set yourself the main level of each sound with sound.volume() but is it what is expected???

  • I found a sample AnalyzeSound in the Minim library as below. Would this be a good place to start?

    /** * This sketch demonstrates how to use an FFT to analyze * the audio being generated by an AudioPlayer. *


    p> * FFT stands for Fast Fourier Transform, which is a * method of analyzing audio that allows you to visualize * the frequency content of a signal. You've seen * visualizations like this before in music players * and car stereos. *


    p> * For more information about Minim and additional features, * visit */

    import ddf.minim.analysis.*; import ddf.minim.*;

    Minim minim; AudioPlayer jingle; FFT fft;

    void setup() { size(512, 200, P3D);

    minim = new Minim(this);

    // specify that we want the audio buffers of the AudioPlayer // to be 1024 samples long because our FFT needs to have // a power-of-two buffer size and this is a good size. jingle = minim.loadFile("jingle.mp3", 1024);

    // loop the file indefinitely jingle.loop();

    // create an FFT object that has a time-domain buffer // the same size as jingle's sample buffer // note that this needs to be a power of two // and that it means the size of the spectrum will be half as large. fft = new FFT( jingle.bufferSize(), jingle.sampleRate() );


    void draw() { background(0); stroke(255);

    // perform a forward FFT on the samples in jingle's mix buffer, // which contains the mix of both the left and right channels of the file fft.forward( jingle.mix );

    for(int i = 0; i < fft.specSize(); i++) { // draw the line for frequency band i, scaling it up a bit so we can see it line( i, height, i, height - fft.getBand(i)*8 ); } }

  • What I am trying to achieve is something like this.

  • To format code edit the post, highlight the code and press Ctrl-o.

  • Yes to the volume dynamics, but there are ways around that. And if the codes written well you can play with that when the image side is done.

    Besides, I don't think the images on the linked video are themselves video clips but just 3 or 4 separate images.

  • Instead of playing the mp3 file I suppose I need to change it to audio in from mic.

    I know there is AudioIn() and I need to start the audio in stream, but I don't know where to begin... Do I need to set up a function like this?

    var mic; function setup(){ mic = new AudioIn() mic.start(); }

    Please advise. Thank you a bunch,

  • var mic;

    is this javascript?

  • did you read the bit where i said not to worry about the sound for the moment and concentrate on getting the images working?

    did you read the bit about how to format code?

  • @Yoshiko===

    in my mind it could be possible only with PImage[] or little gif, not with video

  • I agree with @akenaton that using image arrays or Sprites for this a better idea than using video.

    e.g. see the S4P Sprites library by @quark --

Sign In or Register to comment.