Speech Visualisation Project

General Discussion Other

andyg

in General Discussion • Other • 10 months ago

Hi guys, I'm pretty new to the processing world. As part of a college project, I am hoping to create a system that will visualize speech.

"This idea would be to create system that would allow people to talk into a microphone that would transfer data from the sound like velocity, pitch, tone etc. This data could be then be used to create visuals on a display/installation.

These visuals could either be graphic visualization or just the words reacting in the form of scale and opacity. I have also thought that the tone of the speech could influence colour of words. It could consider the mood of the voice and represent these moods: anger with red visuals or calm with yellow visuals etc. This idea would incorporate the use of a microphone with an arduino board and processing software to create the visuals from the sound data."

I was just wondering if anyone could point me in the right direction towards what I would need to undertake this project.

Replies(4)

artsnova

Re: Speech Visualisation Project

10 months ago

Of the sound libraries that are used with Processing it seems that Minim is the most popular. I would begin by investigating Minim's functionality so that you can determine if its capabilities meet your needs.

Note that there is a collection of Minim sample programs that you can access from your Processing installation via the menu path Files -> Examples.

You can find a list of the other sound libraries that are available for Processing here:
http://www.processing.org/reference/libraries/#sound

Best Regards, Jim

andyg

Re: Speech Visualisation Project

10 months ago

Here is some code I been working on that takes audio input, and then makes text appear corresponding to the input velocity. I was wondering could any help me with the code to make it adjust the size of the text corresponding to the audio input. Any help or other ideas would be most appreciated.

/**
* Get Line In
* by Damien Di Fede.
*
* This sketch demonstrates how to use the <code>getLineIn</code> method of
* <code>Minim</code>. This method returns an <code>AudioInput</code> object.
* An <code>AudioInput</code> represents a connection to the computer's current
* record source (usually the line-in) and is used to monitor audio coming
* from an external source. There are five versions of <code>getLineIn</code>:
* <pre>
* getLineIn()
* getLineIn(int type)
* getLineIn(int type, int bufferSize)
* getLineIn(int type, int bufferSize, float sampleRate)
* getLineIn(int type, int bufferSize, float sampleRate, int bitDepth)
* </pre>
* The value you can use for <code>type</code> is either <code>Minim.MONO</code>
* or <code>Minim.STEREO</code>. <code>bufferSize</code> specifies how large
* you want the sample buffer to be, <code>sampleRate</code> specifies the
* sample rate you want to monitor at, and <code>bitDepth</code> specifies what
* bit depth you want to monitor at. <code>type</code> defaults to <code>Minim.STEREO</code>,
* <code>bufferSize</code> defaults to 1024, <code>sampleRate</code> defaults to
* 44100, and <code>bitDepth</code> defaults to 16. If an <code>AudioInput</code>
* cannot be created with the properties you request, <code>Minim</code> will report
* an error and return <code>null</code>.
*
* When you run your sketch as an applet you will need to sign it in order to get an input.
*
* Before you exit your sketch make sure you call the <code>close</code> method
* of any <code>AudioInput</code>'s you have received from <code>getLineIn</code>.
*/
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
String windowName;
void setup()
{
size(512, 200, P2D);
minim = new Minim(this);
minim.debugOn();
// get a line in from Minim, default bit depth is 16
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
textFont(createFont("SanSerif", 16));
textAlign(CENTER);
windowName = "None";
}
void draw()
{
background(0);
stroke(255);
// draw the waveforms
fft.forward(in.mix);
for (int i = 0; i < fft.specSize(); i++)
{
int val=0;
float val2=0;
val = (int)(height - fft.getBand(i)*50) ;
val2 = (int)(fft.getBand(i)*50) ;
// draw the line for frequency band i, scaling it by 4 so we can see it a bit better
line(i, height, i, height-fft.getBand(i)*100);
//println(val);
println(val2);
if (val2 >10.0 ) {
text("HELLO", width/2, height/2);
}
}
fill(255);
// keep us informed about the window being used
text("The window being used is: " + windowName, 5, 20);
}
void keyReleased()
{
if ( key == 'w' )
{
// a Hamming window can be used to shape the sample buffer that is passed to the FFT
// this can reduce the amount of noise in the spectrum
fft.window(FFT.HAMMING);
windowName = "Hamming";
}
if ( key == 'e' )
{
fft.window(FFT.NONE);
windowName = "None";
}
}
void stop()
{
// always close Minim audio classes when you are done with them
in.close();
minim.stop();
super.stop();
}