We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
Pages: 1 2 
Singing to MIDI (Read 6743 times)
Singing to MIDI
Apr 12th, 2008, 6:06pm
 
Hello!
I have a project. I have to convert singing (from mic) to midi  note signals... I know I should use FFT for getting the sounds amplitude and frequency.
I read through the forum and i saw several apis like Sonia, minim ess...
What api should I use for the task, and where should I start?
Thaks for the further answers, and sorry for my English!
Janos
Re: Singing to MIDI
Reply #1 - Apr 15th, 2008, 9:13am
 
I was thinking of something like this the other day.
I'm not sure how to get the notes from the frequencies.

Based on, and combination of, some of the Minim examples. It will highlight the frequency with the highest amplitude with a blue line.

Quote:



import ddf.minim.*;
import ddf.minim.analysis.*;

AudioInput in;
FFT fft;
int highest=0, bsize=512;
//float[] strings = new float[6];


void setup() {
 size(512, 200);
 
 // always start Minim before you do anything with it
 Minim.start(this);
 
 // get a line in from Minim, default bit depth is 16
 in = Minim.getLineIn(Minim.MONO, bsize, 44100);
 
 //Create a new FFT
 fft = new FFT(in.left.size(), 44100);
 
 fft.window(FFT.HAMMING); //Choose a windowing method
 //fft.window(FourierTransform.NONE);
}


void draw() {
 background(0);
 stroke(255);
 
 
 fft.forward(in.left);
 highest=0;
 for(int n = 0; n < fft.specSize(); n++) {
   // draw the line for frequency band n, scaling it by 4 so we can see it a bit better
   line(n, height, n, height - fft.getBand(n)*4);
   
   //find frequency with highest amplitude
   if (fft.getBand(n)>fft.getBand(highest))
     highest=n;
 }
 
 //Highlight frequency with highest amplitude
 stroke(0,0,255);
 line(highest, height, highest, height - fft.getBand(highest)*4);
 
 /* Guitar strings from howstuffworks.com
 strings[0] = fft.getFreq(82.4); //E
 strings[1] = fft.getFreq(110.0); //A
 strings[2] = fft.getFreq(146.8); //D
 strings[3] = fft.getFreq(196.0); //G
 strings[4] = fft.getFreq(246.9); //B
 strings[5] = fft.getFreq(329.6); //E
 */
 
 // draw the audio waveforms
 for(int i = 0; i < in.bufferSize() - 1; i++) {
   line(i, 50 + in.left.get(i)*50, i+1, 50 + in.left.get(i+1)*50);
   line(i, 150 + in.right.get(i)*50, i+1, 150 + in.right.get(i+1)*50);
 }
 
}


void stop() {
 // always close Minim audio classes when you are done with them
 in.close();
 super.stop();
}


Re: Singing to MIDI
Reply #2 - Apr 15th, 2008, 4:15pm
 
hello!
Thanks for the code! I worked out something like this with sonia. Smiley
After a while I realized that the easiest way to get the notes (frequencies) would be: Setting the frequency bands like the musical notes frequencies so, if we get a note for eg. 440hz A note, there would be only one band for that notes frequency range.
Is this possible with the minim logAverages function?
Re: Singing to MIDI
Reply #3 - Apr 16th, 2008, 7:33am
 
I'm not sure if the logAverage would get what you're looking for (I don't mean it wouldn't, I simply don't know). I do know that you can request the amplitude at a specific frequency.
Quote:

a = fft.getFreq(440);


Then maybe compare this with a table of notes.
Re: Singing to MIDI
Reply #4 - Apr 16th, 2008, 4:29pm
 
Ive had the same idea for a long time and actually just came across Processing when I saw some stuff Flight404 did mentioned on boingboing.

Anyways, after doing a little research about frequency and pitch I was able to throw something together that coverts the detected sound input through microphone to the nearest semitone and then sends the note through midi out.

I used Minim for sound lib and promidi.

I used FFT analysis and then found the band index with the maximum sound power. Converted the band to frequency and then frequency to the midi pitch. and then used the midi pitch to figure out the octave and semi tone.

These are my formulas:

maximumI is the band index with the most power. bufferSize is the FFT band resolution. This formula gives the middle frequency of the specific FFT band.

freq = maximumI / float(in.bufferSize()) * in.sampleRate();

I round the pitch here to snap the frequency to the closest semitone. This is the formula for converting freq to midi pitch. A440 = midipitch 69. middle C4 = midipitch 60.

pitch = round(69 + 12 * (log(freq/440.0) / log(2.0)));

octave = pitch / 12 - 1;

semitone = pitch - (octave + 1) * 12;

Hope this helps. I can post the full source if you would like but keep in mind I have only been using Processing for 5 days now so its a little rough.
Re: Singing to MIDI
Reply #5 - Apr 16th, 2008, 4:52pm
 
Also something important to note is that your bufferSize or resolution must be set high, 512 wont cut it as frequency to pitch mapping is logarithmic.

ie. from a Deep C4 to C4# you jump in frequency from 262Hz to 278Hz a distance of 16Hz. Yet in the whistle Register between octaves 6 and 7 the frequency distance is a lot higher. C6 to C6# there is a jump from 1047Hz to 1109Hz, distance of 62Hz.

at 44100Hz sample rate and a 512 buffer size you are getting 86Hz per band. So basically your band is not fine enough to detect the difference between middle C4 and C4#.

I use a buffer size of 4096 so I can detect resonably down to Octave 3. Or any semitones with frequency distances greater then 10Hz. Every time you double your buffer size you should be able to detect another octave lower. for instance buffer of 8192 should be able to get Octave 2 reliably.

Now depending on what you are doing you may run up against performance constraints with higher and higher band resolutions. You could also try experimenting with lower sample rates.

Re: Singing to MIDI
Reply #6 - Apr 16th, 2008, 4:54pm
 
http://www.youtube.com/watch?v=prIOW1gKFtw

heres a youtube of a test i did. No sound but you see what happens when i whistle or play the piano.
Re: Singing to MIDI
Reply #7 - Apr 16th, 2008, 5:00pm
 
Sorry for all the replies but Im excited that someone else is interested in doing this.

Here is a mp3 of me whistling the super mario theme and recording the outputted midi in reason.

http://schf.uc.org/livepitch-mario.mp3

I am really interested how you will detect the note or chords while singing since the voice has a lot of extra harmonics which will make the FFT go crazy. To be accurate with my testing I use a sine wave since sines have zero harmonics they produce a fine reading on the FFT. Whistling also produces a sine with very few harmonics since you arent using your voice box.

It would be nice to detect chords and label them.
Re: Singing to MIDI
Reply #8 - Apr 16th, 2008, 8:00pm
 
Hello!
Thanks for the replies! Smiley
Very nice job on making the code, I tried and worked fine for whistling. Smiley But after a while it crashes and goes to nullpointer exception.
I think less sample rate would be enough, 22050 or might as well 11025 because human voice input not really uses frequencies above 5500 hz so the algorithm would be faster, with a lot of bands too.
Now I don't really know how to get rid of harmonics while geting singin signal, just an idea: If I can make statistics about the  occurences of frequencies in a time range that equals with the time of for eg. a 32th note in 120bpm (I should investigate how "fast" can sing a person normally and getting the least notetime for "unity") and in the midi conversion I could add the same frequency note times for eg to a 16th note...
After getting a note that longer than the "unity" I could filter the inhuman frequency jumps (maybe harmonics)...

Maybe i will stick to whistling, because the project needs only human input converted to midi signals.

Now I can' hear the mp3, due to some personal sound problems, but tomorrow i will Smiley
Thanks for the posts, and codes once again:)
And Sorry for my English! Smiley
Re: Singing to MIDI
Reply #9 - Apr 17th, 2008, 11:24am
 
corban wrote on Apr 16th, 2008, 5:00pm:
Sorry for all the replies but Im excited that someone else is interested in doing this.

Here is a mp3 of me whistling the super mario theme and recording the outputted midi in reason.

http://schf.uc.org/livepitch-mario.mp3

I am really interested how you will detect the note or chords while singing since the voice has a lot of extra harmonics which will make the FFT go crazy. To be accurate with my testing I use a sine wave since sines have zero harmonics they produce a fine reading on the FFT. Whistling also produces a sine with very few harmonics since you arent using your voice box.

It would be nice to detect chords and label them.


Hello!
I checked out the mp3, and it is very very impressive Smiley
Re: Singing to MIDI
Reply #10 - Apr 29th, 2008, 7:49pm
 
Wow, this looks great.  Any chance you could post your code?
Re: Singing to MIDI
Reply #11 - May 9th, 2008, 2:37am
 
I would like to adapt this so that I can plug my guitar into the computer during a jam session to record all the notes/timing I'm playing, for those occasions when I produce some golden music, but never seem to remember it afterwards. A copy of your code would certainly help in this process.
Much thanks music lover
Re: Singing to MIDI
Reply #12 - May 9th, 2008, 8:59am
 
fazzuk wrote on May 9th, 2008, 2:37am:
I would like to adapt this so that I can plug my guitar into the computer during a jam session to record all the notes/timing I'm playing, for those occasions when I produce some golden music, but never seem to remember it afterwards. A copy of your code would certainly help in this process.
Much thanks music lover


Hello!
I tried mine code which looks like Corban's with guitar input, and, sounds about accurate, but it needs a lot of work too.
I working in Netbeans, so I extended the PApplet, minim and promidi classes. You need that stuff to run the thing.
Re: Singing to MIDI
Reply #13 - May 21st, 2008, 7:19pm
 
Sorry for taking so long to reply and posting my code.

http://schf.uc.org/LivePitch3.pde

Commands:

D - turns on sounds dampening. to filter out ambient noise. (This is a work in progress.. not working correctly now but you can see how it will look) Just hold down the D button until all peaks are filtered.

F - turns off sound dampening.

UP and DOWN arrow - raise and lower the threshhold trigger.
Re: Singing to MIDI
Reply #14 - Oct 20th, 2008, 11:17am
 
Hello Corban!
It was a while I was here. Just want to ask what's about your sing-to midi project? Have you, or anyone else developed something interesting?
Cheers!
Pages: 1 2