There is currently not a terribly good way to do this. But there are other people on the board who are doing this kind of audio/visual sync and hopefully they will chime in with their solutions.

Its worth using ESS for this very feature..  I haven't had any problems with it either.  Give it a try.  

The other solution would be to FFT the audio data yourself.   Retreive 1470 samples at a time (1 frame's worth for 44.1 khz at 30FPS) and FFT it with a 2048 bin FFT (Because it must be a power of two.  The extra slots can be zero, it won't affect the FFT.  This is called Zero-padding.) I have general FFT code for processing if you want it.

Ah, but I have another perspective! There are numerous benefits to the Minim solutions:

1.) Scan the entire audio to a spectrum file (make up your own file format, fairly simple to do). These really don't actually get as big as you think! 60 channels / frame, I got a 10 meg file (my file format was really verbose) for a 3:00 song.

2.) Scan ahead in, say, 5 second intervals (pausing the track, playing, reading fft as it plays, pausing the track, rendering occuring to the data you gathered over those 5 seconds). I like the file better, but this is also doable.

Now, for most purposes you may appreciate the simplicity of just using Ess. But, if you make this spectrum data file you have added capabilities:

bi-directional smoothing: You can 'smooth' the fft results in both directions! So, if you think a drumbeat is affecting your visualization too much, you can look ahead and anticipate future changes! There's so much potential here. I really reccomend not looking for the "get a sample, decode the fft, and render" because the file format will give you more options and in the end I think it will save you rendering time, though its not a big deal.

Just my opinion, as I have made a few FFT music videos in the past few months.

That code looks great, but how are you manually advancing AudioInput forward? The code looks like you're just constantly re-reading the input's current buffer... and actually, it looks like you're trying to analyze of off the microphone input stream O.o

How are you "pushing" say, a .wav file's sound buffer into AudioInput in?

thx for that code mavax! i like the fact you can scale your fft result the way you see fit, definitely come in handy when needing to pronounce a certain range in the spectrum.

before your post i actually went down the path of exporting the fft data into a text file in a seperate sketch and reading in those values during rendering. its worked as far as i can see, there was a little lag but easily adjusted by shifting up  the audio layer over the video layer. still need to test this technique over a long audio file to make sure it doesn't lag any further.

think your approach is more stable, going try plugging it in into minim.

thx again.

I have found this whole thread really interesting and the non real time mapping of audio analysis data to an anim is something I am experimenting with.

However - processing and audio are areas I have limited knowledge developing with and am only starting to play so I would really appreciate feedback on the following script I developed based on what i have read and learnt.

I still need to test synching anims with the data file.

Code:

/////////////////////////////////////////////////////////////////////
/**
 * FFT Logarithmic Averages to TextFile
 * by gLASSHOPPER*
 *  
 * It is an experiment - updated for cleaner logging!!!
 */
/////////////////////////////////////////////////////////////////////  
 
/////////////////////////////////////////////////////////////////////
// VARIABLES //
import ddf.minim.analysis.*;
import ddf.minim.*;
 
Minim minim;
AudioPlayer audio;
AudioMetaData meta;
FFT fft;
PrintWriter output;
 
int yi = 15;
 
int tFrameRate = 25;
String audioFileName = "test.mp3";
 
int setupMilli; int playMilli; int playSecs;
int curFrame = 0; int logFrame = 0; String audioLog = "";
 
int minHz = 20;
int maxHz = 500;
int bands = 2;
 
int durSecs; int durMilli;
 
float[] arrSpectrum;
 
 
/////////////////////////////////////////////////////////////////////
// MAIN //
 
// SETUP //
void setup() {
  size(512, 200, P3D);
  minim = new Minim(this);
 
  textFont( loadFont("Helvetica.vlw") );
  textMode(SCREEN);
 
  audio = minim.loadFile(audioFileName, 2048);
   
  // get metadata for audio
  meta = audio.getMetaData();
 
  // create FFT object  
  fft = new FFT(audio.bufferSize(), audio.sampleRate());
 
  // Sets the window to use on the samples  
  // before taking the forward transform.
  fft.window(FFT.HAMMING);
   
  // calculate averages based on a miminum octave width of minHz
  // split each octave into bands
  fft.logAverages(minHz, bands);
   
  // setup arrSpectrum to contain each avg value from spectrum
  arrSpectrum = new float[fft.avgSize()];
  rectMode(CORNERS);
 
  // Create a new output file in the sketch directory
  output = createWriter(meta.fileName()+".txt");  
 
  // get track dur from metadata associated with file	
  durSecs = meta.length()/1000;
  durMilli = meta.length()-(meta.length()/1000)*1000;
 
  // log the amount of time setup took to complete  
  println("Setup Complete: "+millis());	
  println("File Name: "+meta.fileName()+" Length: "+durSecs+":"+durMilli);
  setupMilli = millis();	    
  // play the file
  audio.play();  
}


// DRAW //
void draw() {
  background(0);
  fill(255);
   
  // reset text pos to yi
  int y = yi;
 
  // perform a forward FFT on the samples in audio's mix buffer
  fft.forward(audio.mix);
  fft.calcAvg(minHz, maxHz);
  int w = int(width/fft.avgSize());
  
 
  // draw a rectangle for each average
  for(int i = 0; i < fft.avgSize(); i++) {
    rect(i*w, height, i*w + w, height - fft.getAvg(i)*5);
    
    // update the average value in array
    arrSpectrum[i] = round(fft.getAvg(i));
  }
 
  // update playMilli to current play duration minus setupMilli
  playMilli = millis() - setupMilli;
  playSecs = (playMilli/1000);
   
  // remap milliseconds for current second between 1 + tFrameRate
  curFrame = round(map((playMilli-(1000*playSecs)),0,1000,1,tFrameRate));
   
  // if curFrame is not the same as last logged frame call logger
  // ensure we log only based on the framerate
  if(curFrame != logFrame) { logSpectrum(); }
 
  // update text output so user can sse what is happening
  text("File Name: " + meta.fileName(), 5, y);
  text("Length: " + durSecs+":"+durMilli, 5, y+=yi);  
  text("Timer: " + (playSecs)+":"+curFrame, 5, y+=yi);
   
  // exit case if we are beyond the duration of the track
  if (millis()/1000 > meta.length()/1000) {  
    end();  
  }  
}
 
 
// STOP // final output to console and EXIT ...
void stop() {
  println("DONE: "+(playMilli/1000)+":"+curFrame);
  exit();
}
 
 
/////////////////////////////////////////////////////////////////////  
// FUNCTIONS //
 
// LOGSPECTRUM // logs spectrum data at current frame to audioLog
void logSpectrum() {
  logFrame = curFrame;
  String joinedSpectrum = join(nf(arrSpectrum, 3, 0), ",")+"\n";  
  audioLog +=  nf(playSecs,2)+":"+nf(curFrame,2)+"\t"+joinedSpectrum;
  //println(audioLog);
}
 
 
// END // cleans up minim classes write log file + forces close
void end() {
  // always close Minim audio classes when you finish with them
  audio.close();
  minim.stop();
 
  println(audioLog);
  
  //String logLines new String[lo
  saveStrings(meta.fileName()+".txt", split(audioLog,"\n"));
 
  super.stop();  
} 
 

I've done a fair amount of messing around with offline rendering and sync issues.  I've been meaning to post back here in the forum (since i don't blog) in case anyone else might benefit, so here are some further thoughts, fwiw...

There are 3 (4?) general approaches to capture sync'd audio/video:

1A) do everything real-time, play audio, draw stuff, save video frames to disk, merge audio after-the-fact
1B) do everything real-time, play audio, draw stuff, record both simultaneously with a camera (or external software)
2) do the audio analysis real-time, log the results, but render video offline by reading the log, merge audio after-the-fact
3) do everything offline, fft the audio one video frame at a time, render that frame & save it, merge audio after-the-fact

Method 1A is practically guaranteed to be laggy and loose audio sync, due to the inexact video framerate of Processing. (and, of course, if you have really complex visuals that further drop the requested video framerate, then it's just that much worse)

Method 1B suffers the same video lag as 1A, but due to the realtime recording will make up for it when played back.  (camera will simply record multiple frames of a single rendered frame as necessary until you get around to rendering the next one)  So that even if you can't render *every* video frame at least the frames you do render are recorded synchronously with the audio currently playing.  (this is essentially just recording a live performance, a non-issue as far as i'm concerned, it works, done)

Method 2 will likely have minor lags, again due to the inexact video framerate of Processing (versus the audio samplerate which is *exact*).  So on video frame N you can't be sure that audio is exactly playing at N*samplerate/framerate, so your logged results will have lag built into them.

Method 3 has no lag and can attain "perfect" sync.

Consider if you had an *exact* 30fps video rate:  playing a 44.1Hz audio requires that 1470 (44100/30) samples are analyzed during each video frame.  On frame #1, you cue to sample #0 and fft the next 1470 samples.  On frame #2, you cue to sample #1470 and fft the next 1470.  On frame #3, you cue to sample #2940 and fft the next 1470.  et cetera till end of audio.

(technical aside: in order to accomplish an fft of 1470 samples, you'd need to zero pad a buffer of size 2048)

so, if you had audio exactly 60 seconds long, you'd do exactly 1800 fft's (60*30) spanning exactly 79380000 samples (1800*44100) and render exactly 1800 video frames, each of which "looked at" an fft of exactly 1470 samples.  That's the sort of math you'd like to see for "perfect" sync.

btw, that's a good test of any rendering method - create an audio snippet of exactly 60 seconds, process it, then ask:  did i render exactly 1800 video frames? or log exactly 1800 fft frames?  if not, then you'll loose sync to some degree when you eventually merge the audio with video.

Basically, if you do your FFT based on audio that's *playing* realtime (even if logging it for later rendering), there's really no way using the draw() thread to capture perfectly sync'd audio/video (other than some variant of method 1B with an external live recorder).  A better approach for method 2 "logging" variants would be to start a separate thread, and so doing the fft asynchronously with the draw() thread.  That should get you a lot closer, use the sleep(long,int) version to get 1000/30=33 millis + 333333 nanos. (and i'd still check it, did it log exactly 1800 fft frames on your 60 second test audio?)

Method 3 gets around all of that by never actually playing the audio, just fft-ing the raw wave data at specific cue points that match up exactly with video frames.  And it's really, really easy once you "get" how it works.  I've done pieces >15 minutes that retained perfect frame-accurate sync throughout - hard to do with other methods!  (or, at least, it was hard for me!! :-D)  Hope that helps.