Processing 1.0 - Processing Discourse - Minim reading lines from image

evolcdc

Minim reading lines from image
Jan 21^st, 2008, 9:28am

I am trying to assemble the pieces to create a synth that "plays" the vertical lines of pixels from an image. Can Minim handle 400 addSignals? is there a better way to go about this? Thanks in advance!

import ddf.minim.*;
import ddf.minim.signals.*;

AudioOutput sound;
MuzikMaker[] muzik = new MuzikMaker[400];
PImage file;
int[][] pixel;
float[] redC, greenC, blueC;
int x = 0;

void setup()
{
size(600, 400);
Minim.start(this);
frameRate(10);
sound = Minim.getLineOut(Minim.STEREO, 512);
for (int m=0; m < height; m++)
{
muzik[m] = new MuzikMaker();
muzik[m].update(m);
sound.addSignal(muzik[m]);
}
pixel = new int[width][height];
file = loadImage("mountain.jpg");
for(int i=0; i<width; i++) {
for(int j=0; j<height; j++) {
pixel[i][j] = file.pixels[i + width*j];
}
}
}

void draw()
{
image(file, 0, 0);
for(int i = 0; i<height; i++){
redC[i] = red(pixel[500][i]);
greenC[i] = green(pixel[500][i]);
blueC[i] = blue(pixel[500][i]);
println(redC[i]);
println(greenC[i]);
println(blueC[i]);
// x ++;
}
}

class MuzikMaker implements AudioSignal
{
int xcode;
void update(int xtemp) {
xcode = xtemp;
}
void generate(float[] samp)
{
float range = map(redC[xcode], 0, 255, 0, 1);
float peaks = map(blueC[xcode], 0, 255, 1, 20);
float inter = float(samp.length) / peaks;
for ( int i = 0; i < samp.length; i += inter )
{
for ( int j = 0; j < inter && (i+j) < samp.length; j++ )
{
samp[i + j] = map(j, 0, inter, -range, range);
}
}
}

// this is a stricly mono signal
void generate(float[] left, float[] right)
{
generate(left);
generate(right);
}
}

void stop()
{
// always closes audio I/O classes
sound.close();
super.stop();
}

taifunbrowser

Re: Minim reading lines from image
Reply #1 - Jan 21^st, 2008, 4:19pm

The audio part of your code was fine... Just some simple mistakes, and I was too curious about the minim thing to avoid just going ahead and fixing the code myself.

You just weren't instantiating redC and greenC and blueC, so you got nullpointerexceptions...

And on my computer, 400 samples was no problem.

Code:


AudioOutput sound;
MuzikMaker[] muzik = new MuzikMaker[400];
PImage file;
int[][] pixel;
float[] redC, greenC, blueC;  
int x = 0;

void setup()
{
Minim.start(this);
frameRate(10);
sound = Minim.getLineOut(Minim.STEREO, 512);
file = loadImage("mountains.png"); //Change
size(file.width,file.height);
pixel = new int[width][height];
file.loadPixels();
for(int i=0; i<width; i++) {
for(int j=0; j<height; j++) {
pixel[i][j] = file.pixels[i + width*j];
}
}
redC = new float[height];
greenC = new float[height];
blueC = new float[height];
for(int i = 0; i<height; i++){
redC[i] = red(pixel[0][i]);
greenC[i] = green(pixel[0][i]);
blueC[i] = blue(pixel[0][i]);
System.out.println(redC[i]);
System.out.println(greenC[i]);
System.out.println(blueC[i]);
// x ++;
}
for (int m=0; m < height; m++)  
{
muzik[m] = new MuzikMaker();
muzik[m].update(m);
sound.addSignal(muzik[m]);
}
noLoop();
}

void draw()  
{
image(file, 0, 0);
}

class MuzikMaker implements AudioSignal
{
int xcode;
void update(int xtemp) {
xcode = xtemp;
}
void generate(float[] samp)
{
float range = map(redC[xcode], 0, 255, 0, 1);
float peaks = map(blueC[xcode], 0, 255, 1, 20);
float inter = (float)(samp.length) / peaks;
for ( int i = 0; i < samp.length; i += inter )
{
for ( int j = 0; j < inter && (i+j) < samp.length; j++ )
{
samp[i + j] = map(j, 0, inter, -range, range);
}
}
}

// this is a stricly mono signal
void generate(float[] left, float[] right)
{
generate(left);
generate(right);
}
}


void stop()
{
// always closes audio I/O classes
sound.close();
super.stop();
}

ddf

Re: Minim reading lines from image
Reply #2 - Jan 21^st, 2008, 4:26pm

Well, that will work, but iterating through all 400 signals every time a new buffer of samples needs to be created will probably be too slow. Also, your loop in draw where you print out the pixel color values every frame is really going to slow your sketch down.

Another thing to consider is that even if all 400 of your signals process in a reasonable amount of time, you are generating the same very short, dense chunk of audio for every audio buffer request. You might instead consider having a single signal that reads successive lines from an image. So, for example, if you have a 1024x1024 image, you could read rows starting from the top. Each time you are asked to generate audio, you read the next row. Then the image would result in 1024 sample buffers that are 1024 samples long, which will have a much more audible and distinct sound than every line of the image sounding simultaneously.

http://www.compartmental.net

evolcdc

Re: Minim reading lines from image
Reply #3 - Jan 21^st, 2008, 6:48pm

I understand what you are hinting at, and I was going for all 400 sounds to play for a second before shifting to the next line, just trying to get one line working first.
I think what you might also be suggesting is that each pixel's frequency from a given line could play out as part of a sample, and then these could change as I move through the image. In fact what I would like to accomplish is a reverse FFT, where a channel from the image provides the input.
Say, pixels at the top of the image provide the amplitude for the high frequencies, and lower pixels provide bass amplitudes, however I could not decipher your (amazingly helpful and useful) library well enough to do this. Any hints? What object would be most effective for this?

ddf

Re: Minim reading lines from image
Reply #4 - Jan 22^nd, 2008, 4:12am

Well, what you'd want to use would be an FFT object:

http://code.compartmental.net/minim/javadoc/ddf/minim/analysis/FFT.html

But looking now, I see that I did not provide a way to set the spectrum values directly. I will rectify this in the next release, for sure. As for deciphering the library, the manual, which is easier to read than the javadoc (I hope), is finished and can be found here:

http://code.compartmental.net/tools/minim/manual-introduction/

http://www.compartmental.net

evolcdc YaBB Newbies Offline Posts: 5	Re: Minim reading lines from image Reply #5 - Jan 22^nd, 2008, 4:31am Hmm. So I would do an fft.forward on ??? then use fft.setBand(position of pixel, color of pixel); to fill the ???buffer with new signals and then do an fft.inverse to get it to play the new sound? Do I need to have a sound file for the space holder, or is there another way to declare a sound buffer?

ddf

Re: Minim reading lines from image
Reply #6 - Jan 23^rd, 2008, 6:03am

Ok, right, you could use setBand to set the bands one at a time. If you do that, you don't need to do a forward transform first. What I meant by "no way to set the spectrum directly" is that there's no way to pass in a array of floats to set it all at once. It's kind of gross that you have to use setBand over and over. But, basically, you could do something like:

Code:


void generate(float[] samps)
{
  for(int i = 0; i < fft.specSize(); i++)
  {
    fft.setBand(i, redC[i]);
  }

  fft.inverse(samps);
}

In this case, you just have to make sure that fft.timeSize() is equal to the length of the buffers begin passed into generate.

http://www.compartmental.net

evolcdc

Re: Minim reading lines from image
Reply #7 - Jan 24^th, 2008, 3:16am

I have gotten very close at this point, but the sound only plays for a moment and then goes quiet, like it played one line and then stopped. I have tried putting println s everywhere and things are updating, but not sonicly. Any help appreciated...I am trying to solve this for my students tommorrow.

import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;

AudioOutput sound;
FFT fft;
MuzikMaker muzik;
PImage file;
int[][] pixel;
float[] redC, greenC, blueC;
int x = 0;

void setup()
{
size(600, 513);

file = loadImage("17_dark_clouds.jpeg");
file.loadPixels();
pixel = new int[width][height];
for(int i=0; i<width; i++) {
for(int j=0; j<height; j++) {
pixel[i][j] = file.pixels[i + width*j];
}
}
redC = new float[height];
greenC = new float[height];
blueC = new float[height];
Minim.start(this);
sound = Minim.getLineOut(Minim.STEREO, 1024);
fft = new FFT(1024, 44100);
muzik = new MuzikMaker();

sound.addSignal(muzik);
frameRate(5);
}

void draw()
{
image(file, 0, 0);
for(int i = 0; i<height; i++){
redC[i] = red(pixel[x][i]);
greenC[i] = green(pixel[x][i]);
blueC[i] = blue(pixel[x][i]);
//println(redC[i]);
}
if (x<width) x++;
else x = 0;

}

class MuzikMaker implements AudioSignal
{

void generate(float[] samp)
{
for(int i = 0; i < fft.specSize(); i++)
{
fft.setBand(i, redC[i]);
//println(i);
}
fft.inverse(samp);
//println("inversed");
}

// this is a stricly mono signal
void generate(float[] left, float[] right)
{
generate(left);
generate(right);
}
}

void stop()
{
sound.close();
Minim.stop();
super.stop();
}

rory

Re: Minim reading lines from image
Reply #8 - Feb 18^th, 2008, 8:16pm

ddf wrote on Jan 23^rd, 2008, 6:03am:

Ok, right, you could use setBand to set the bands one at a time. If you do that, you don't need to do a forward transform first. What I meant by "no way to set the spectrum directly" is that there's no way to pass in a array of floats to set it all at once.

ddf, Is there some way we could use the FFT in conjunction with the Convolver to achieve something like this effect

I don't totally understand the Convolver class but it seems like it could be useful here.

Thanks,
Rory

« Last Edit: Feb 18^th, 2008, 11:31pm by rory »

ddf

Re: Minim reading lines from image
Reply #9 - Feb 19^th, 2008, 3:35am

I'm not sure what you mean by that. The Convolver works in the time-domain and requires a "kernel" to convolve the signal with. FFT is a way to convert audio from the time-domain to the frequency domain. If you want to do convolution, it's actually a much better idea to do it in the frequency domain by simply multiplying the spectra of the two signal you want to convolve and then doing the inverse transform.

http://www.compartmental.net

rory

Re: Minim reading lines from image
Reply #10 - Feb 24^th, 2008, 12:21am

Hi ddf,

ok, I think I understand what you are saying. I was thinking that perhaps instead of calling setBand repeatedly, the previous poster could repeatedly generate kernels and use the Convolver. But maybe I'm misunderstanding what the Convolver does.

I should actually move my discussion to a new thread since I'm trying to do something totally different:
http://processing.org/discourse/yabb_beta/YaBB.cgi?board=Sound;action=display;num=1203808804

Thanks!