How to make a generative visual react to sound

Mxrx

Hello, I am kind of new to Processing. What I am trying to do is to make this generative visual react to sound. For now the visual is reacting to the mouse movements, but I would like it to take input from the sound captured by the microphone and generate the visuals. The code for now is this. I guess I will have to use the minim library and the fft audio analyzer but I don't really know how to integrate it to this sketch. Thank you in advance for any help

float[] x;
float[] y;
color[] col;
float s = 0.001;
float depth = 0.5;
PImage img; 

void setup() {
  size(1000, 1000);
  background(255);
  int n = 1000;
  x = new float[n];
  y = new float[n];
  col = new color[n];
  img = loadImage("imageName0.jpg");
  img.resize(width, height);
  img.loadPixels();
  for (int i = 0; i < x.length; i++) {
    x[i]= random(0, width);
    y[i]= random(0, height);
    int loc = int(x[i]) + int(y[i])*width;
    col[i] = img.pixels[loc];
  }
}

void draw() {
  noStroke();
  depth = map(mouseY, 0, height, 0.5, 1.5);
  //fill(255, 4); //Uncomment if you don't want to use an image;
  for (int i = 0; i < x.length; i++) {
    float alpha = customNoise(x[i] * s, y[i] * s)*2*PI;
    x[i]+= depth * cos(alpha); // + random(-0.4, 0.4);
    y[i]+= depth * sin(alpha); // + random(-0.4, 0.4);
    if (y[i] > height) {
      y[i] = 0;
      x[i] = random(0, width);
    }
    x[i]= x[i]%width;
    fill(col[i], 4); //Comment if you don't want to use an image;
    ellipse(x[i], y[i], 2, 2);
  }
}


float customNoise(float x, float y) {
  return pow(sin(0.9*x + noise(x, y)*map(mouseX, 0, width, 0, 5)*y), 3);
}

kfrajer

Check the following posts: https://forum.processing.org/two/search?Search=audioIn

For example:
https://forum.processing.org/two/discussion/comment/81611/#Comment_81611
https://forum.processing.org/two/discussion/18392/advice-on-how-to-convert-sound-into-camera-distortion/p1

I am not sure if you want to work with FFT. In your case, you can start by looking at the amplitude of the received signal as shown in the previous posts.

Kf

Mxrx

Thank you for your answer, the links were kinda useful, but the problem is that I really don't know where to start as I have never worked with sound in Processing, I feel stupid. Would anyone be able to offer an example of how to integrate sound in this sketch so that the generative design would animate following the sound input?

kfrajer

how to integrate sound in this sketch so that the generative design would animate following the sound input

Maybe you should be more specific. What do you want to do?

If you provide some code, ppl will see your approach and your programming level and provide relevant and efficient feedback.

If you have something in mind, or something you want to reproduce, provide a clear description, examples either as images or links to relevant resources including video.

You should check the sound tutorial in the processing web page. You can get ideas of some projects done in the past and basic concepts immediately available to you.

Kf

Mxrx

Hi kfrajer thank you for the answer. For now I only managed to come up with this bit of code, this is not at all what I want to achieve in the end, but at least I managed to generate some lines using sound input

import ddf.minim.analysis.*;
import ddf.minim.*;

//global variables
Minim minim;          //minim object
AudioInput in;     //object for realtime audio
FFT fft;

float x1,x2; // We define the variables

void setup() {
 fullScreen();
 minim = new Minim (this);
  in = minim.getLineIn (Minim.STEREO, 512);
  fft = new FFT (in.bufferSize(), in.sampleRate());
  fft.logAverages(60,7);
  background(0,0,0);
  noCursor();

}

void draw() {

  fft.forward(in.mix);
  stroke(255,255,255,100);
  x1 = width/2+1000; // We initialise them
  x2 = width/2-1000; // which means we affect them for the first time
  for(int i=0; i< fft.specSize(); i ++){
  line( x1, 1000, x2, fft.getBand(i)*100);
}
}

What I actually want to achieve is something like this

float x1 = 300;
float y1 = 400;

void setup(){
size(595,842);
smooth();
}

void draw(){
float x2 = x1 + random(-10,10);
float y2 = y1 + random(-10,10);

line(x1,y1,x2,y2);

x1 = x2;
y1 = y2;
 }

but obviously I would like the line to be drawn if sound is playing ( I mean if sound is picked up from the microphone), any idea on how I could do it? I don't really want the line to do anything specific when sound is playing, just randomly draw stuff on the canvas, and once sound is not playing, i would like it to stop.

kfrajer

There are different ways to do it. here is a relevant post:

https://forum.processing.org/two/discussion/558/creating-a-next-page-button

You can create different pages. In one page you run your draw without interacting with sound. In another page you draw while interacting with input sound.

The concept is more related to states. In your draw function you do something based on the current state. The state is controlled by a global variable. In your case, you want to switch states every time you call play or stop on your sound source. i like to use the previous post as an example that fits most situations. Try to implement it yourself and if you get stuck, ask and post your code below.

Kf

Mxrx

I managed to do what I wanted, It wasn't so complicated as it seemed, this is the final code

import processing.sound.*;


float x1 = 700;
float y1 = 400;
//float a1 = 300;
//float b1 = 500;

Amplitude amp;
AudioIn in;
float ampt;

void setup(){
fullScreen();
//size (800,800);
smooth();
background(0);

 amp = new Amplitude(this);
  in = new AudioIn(this, 0);
  in.start();
  amp.input(in);

}

void draw(){
  ampt = amp.analyze();
  println(ampt);
  if (ampt>0.030) {

float x2 = x1 + random(-20,20);
float y2 = y1 + random(-20,20);
//float a2 = a1 + random(-20,20);
//float b2 = b1 +random(-20,20);

line(x1,y1,x2,y2);
//line (a1,b1,a2,b2);
stroke (255);

x1 = x2;
y1 = y2;
//a1= a2;
//b1=b2;
  }
  // Constrain all points to the screen
  x1 = constrain(x1, 0, width);
  y1 = constrain(y1, 0, height);
 }

Mxrx

Now I'd like the line, instead of being black, to get the color from the pixel of an image. I have written this, but it is not working because the line is still being drawn black instead of using the colors of the image I have loaded, can somebody tell me what I am doing wrong?

import processing.sound.*;

PImage img;
float x1 = 700;
float y1 = 400;


Amplitude amp;
AudioIn in;
float ampt;

void setup(){
fullScreen();
//size (800,800);
smooth();
background(255);

img = loadImage("43181369_HD.jpg");

 amp = new Amplitude(this);
  in = new AudioIn(this, 0);
  in.start();
  amp.input(in);

}

void draw(){
  ampt = amp.analyze();
  println(ampt);
  if (ampt>0.030) {

float x2 = x1 + random(-20,20);
float y2 = y1 + random(-20,20);

color c= img.get(int(x1),int(y1));
fill(c);
line(x1,y1,x2,y2);

//stroke (255);

x1 = x2;
y1 = y2;

  }
  // Constrain all points to the screen
  x1 = constrain(x1, 0, width);
  y1 = constrain(y1, 0, height);
 }

kfrajer

To change the color of the line, use stroke instead of fill in line 36.

Kf