Turn audio input sound into color

kmooney

Hello all,

I am currently having some major trouble attempting to turn input sound into color (using minim fft).

I am hoping to use bandpass filters to set three frequency ranges and then use the numbers that come out of the analysis of the audio input waves to determine the r,g,b fill values of a square.

(Also I am open to any other suggestions for ways to turn input audio into colors.)

Any help is MUCH appreciated! Thank you so much in advance! Cheers!

Katherine

Lord_of_the_Galaxy

Please tell us some details about the problem. What specifically is the trouble? Do you have any code yet? Remember to format your code.

kfrajer

This is a sample code demonstrating the effect. To run it, you need to install the minin library using Processing's library manager. You also need to have your image in your data folder of your sketch. I hope this helps,

Kf

import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;
PImage img;

Minim minim;
AudioInput in;

void setup()
{
  size( 540, 687 );
  img = loadImage("fig.jpg");

}

{   
  minim = new Minim( this );
  in = minim.getLineIn( Minim.STEREO, 512 );
}

void draw()
{
   image(img, 0, 0);
  float p = 0;
  for ( int i = 0; i < in.bufferSize(); i++ ) {
    p += abs( in.mix.get( i ) ) * 1;
  }
  loadPixels();

  for (int i=0; i<(width*height); i++) {
    float r = red(pixels[i]);
    float g = green(pixels[i]);
    float b = blue(pixels[i]);

    pixels[i] = color(r*p, g/p, b+p);
  }
  updatePixels();
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

Lord_of_the_Galaxy

Working code from @kfrajer, hope you understood it.

kmooney

Thank you for your reply! I am quite new to Processing, so here is my very basic code. I am trying to change the color of the rectangle according to three different frequencies of the input sound but I'm not sure how to set up three separate bandpass filters. I want the first filter to detect the low range Hz of the input, the second filter to detect the mid range Hz of the input, and the third filter to detect the high range Hz of the input. Then, I want to fill the rect by feeding the low range number into red, the mid range number into green, and the high range number into blue.

import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;



Minim minim;

//Audio signal
AudioInput input;
FFT fftLog;


//Effect Filter
BandPass band;


void setup(){
  size (100, 100, P3D);
  minim = new Minim(this);
  input = minim.getLineIn(Minim.STEREO, 2048, 192000.0); 
  fftLog=new FFT(input.bufferSize(),input.sampleRate()); 

}

  void draw(){
    background(100); 
    fftLog.forward(input.mix);
    rect(30, 20, 55, 55);
    fill(fftLog.getAvg(0)); 

   // Take the frequency values returned by the audio input
   // and connected them with red, green, and blue fill values for rect()

  }

  class Block{
  int bandNumber;
  float baseSize;
  float level; 

  Block(int _bandNumber,float _baseSize,float _level){ 
    bandNumber=_bandNumber;
    baseSize=_baseSize;
    level=_level;
  }

  void display(){
    fftLog.forward(input.mix);
    rect(30, 20, 55, 55);
    fill(level*fftLog.getAvg(bandNumber),baseSize,baseSize); 
  }
}

  // Stops the program
  void stop(){
    input.close();
    minim.stop();
    super.stop();
}

kmooney

Thank you for any help!

Lord_of_the_Galaxy

OK, I'll mention again, please format your code.
Click on the gear icon to edit your post. Select your code and press ctrl + o to indent. Leave a line above and below (the code). Markdown will do the rest for you.

kmooney

Thank you, kfrajer! That is helpful. I will try to apply it to my code now!

kmooney

So this is what I have now, but I'm hoping to make the color change effect happen only to a rectangle. Also, I'm not sure if there is a way to make the color change happen more fluidly or more frequently? Any suggestions are much appreciated!

import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;

Minim minim;
AudioInput in;

void setup()
{
  size( 540, 687 );

}

{   
  minim = new Minim( this );
  in = minim.getLineIn( Minim.STEREO, 512 );
}

void draw()
{

  float p = 0;
  for ( int i = 0; i < in.bufferSize(); i++ ) {
    p += abs( in.mix.get( i ) ) * 1;
  }
  loadPixels();

  for (int i=0; i<(width*height); i++) {
    float r = red(pixels[i]);
    float g = green(pixels[i]);
    float b = blue(pixels[i]);

    pixels[i] = color(r*p, g/p, b+p);
  }
  updatePixels();
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

kfrajer

@kmooney

You are doing a good approach. Testing your code generating the color change separate from your filter calculations. Your life will be easier if you figure the projects in steps.

Your line 34 above needs some consideration:

pixels[i] = color(r*p, g/p, b+p);

You need to check the reference and become familiar with colorMode. In a nutshell, the default color mode is RGB and each color is a number from 0 to 255, a total of 256 values. Now when you multiply, let's say red by your value stored in p, r [times]p can be bigger than the color limit of 255. For example if red is 250 and p is 200, r[times]p will give you 500000. But reds are defined from 0 to 255, inclusive! :-?? There are different ways to solve this problem.

One way:

float r = red(pixels[i]);
r = r/2.0;
r=r*p;

Then the value of p needs to be constrained from 0 to 2. How to do this? Use the map function: https://processing.org/reference/map_.html

p=map(p,min,max,0,2);

where min and max are the minimum and max values p can be. To calculate p, you will need to understand where you are getting this value from. In your code, from line 24. In this case you will need to understand what values p can take in order to define min and max. A good guess for min is 0 (aka zero). Max depends on the summation of values of buffer size.

On a side note, for the green color, g/p needs two considerations. First, you need to watch for zero divisions. Second, you have to consider the previous definition of p for red doesn't work for green because for green (and blue) you are using another operation (division instead of multiplication). Values for green and blue needs to also be constrained to 255.

There are other options, but it really depends to what effect you want to obtained at the end. For example, https://processing.org/reference/constrain_.html

There are other questions to resolve:

1. How to change the color only in an area of the sketch
2. How to use fft on sound data
3. How to use the filters

First, get the color concept down.

Kf

kfrajer

Here is a sample code for you to consider based on your previous post.

Kf

import ddf.minim.*;
import ddf.minim.signals.*; 
import ddf.minim.analysis.*; 
import ddf.minim.effects.*;

Minim minim;
AudioInput in;

PGraphics pgBase;

void settings() {
  size( 540, 687 );
}

void setup()
{
  pgBase=createGraphics(width, height);
  shapeSketch0(pgBase);  
  minim = new Minim( this );
  in = minim.getLineIn( Minim.STEREO, 512 );
}

void draw()
{
  PImage pg=pgBase.copy();
  image(pg, 0, 0, width, height);
  float p = 0;
  for ( int i = 0; i < in.bufferSize(); i++ ) {
    p += abs( in.mix.get( i ) ) * 1;
  }
  println("P values="+p);
  loadPixels();

  p=constrain(p, 0, 400);    //Optional as map will take care of constraining the value anyways.


  // ************************************
  //OPTION: Next try either of both lines
  p=map(p, 0, 400, 0, 2);  //Changing from 0 to 2. You need to make lots of noise to see the figure
  //p=map(p, 0, 400, 1, 2);    //Changing from 1 to 2. You always see the image. Noise just makes it whiter.


  for (int i=0; i<(width*height); i++) {
    float r = red(pixels[i]);
    float g = green(pixels[i]);
    float b = blue(pixels[i]);

    // ************************************
    // TRY commenting out any of these next three lines to see effects on single basic colors

    //Halves my color so when I multiply by the max value (aka 2), it is still within the color range
    r=r/2.0*p;
    g=g/2.0*p;  
    b=b/2.0*p;

    pixels[i] = color(r, g, b);
  }
  updatePixels();

  //Draws the original image in the lower right quadrant for comparison purposes
  image(pg, width/2, height/2, width, height);
}

public void stop()
{
  in.close();
  minim.stop();
  super.stop();
}


void shapeSketch0(PGraphics ap) {
  ap.beginDraw();
  ap.background(211, 38, 38);
  ap.rect(110.0f, 69.0f, 194.0f, 194.0f);
  ap.fill(38, 42, 211);
  ap.rect(0, 0, 239.0f, 167.0f);
  ap.rect(176.0f, 336.0f, 0.0f, 0.0f);
  ap.ellipse(346.0f, 305.0f, 210.0f, 210.0f);
  ap.line(371.0f, 142.0f, 34.0f, 408.0f);
  ap.line(148.0f, 552.0f, 482.0f, 96.0f);
  ap.strokeWeight(5);
  ap.line(180.0f, 559.0f, 492.0f, 108.0f);
  ap.line(25.0f, 382.0f, 368.0f, 126.0f);
  ap.endDraw();
}

kmooney

Thank you so much, kfrajer! That code is very helpful! I'll update you when I figure out how to use it correctly. You're so great for helping!

MGlolenstine

Sorry for necrobumping this old thread, but I'm trying to make my app work with a mic, but there is the following problem... I have a dummy microphone input on my Linux, but I have no idea how I can switch between 2 or more devices with minim.

koogs

it's also a different question completely - start a new one

that said, i think the answer is - choose microphone using system settings.