Is it possible to perform FFT with FilePlayer object? [Minim]

SnailPropulsionLabs

I'm trying to affect the playback speed of an mp3 based on the positions of the bars.

The "vanilla" code (from the analyzeSound.pde example) below works with an AudioPlayer object but when I try to combine it with the tickRate example, it says "the function 'bufferSize()' does not exist" and "the global variable 'mix' does not exist".

I don't understand Minim enough to know how to remedy this.

Is it possible to do it this way?

Thanks.

Vanilla:

import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioPlayer player;
FFT fft;

float spectrumAvg;

void setup() {
  fullScreen();
  //size(512, 200);
  minim = new Minim(this);
  selectInput("Select an audio file:", "fileSelected");
}

void fileSelected(File selection) {
  String audioFileName = selection.getAbsolutePath();
  player = minim.loadFile(audioFileName);
  fft = new FFT(player.bufferSize(), player.sampleRate());
  player.play();
}

void draw()
{
  background(0);
  stroke(255);

  if (player != null) {
    if (fft != null) {
      fft.forward(player.mix);

      for (int i = 0; i < fft.specSize(); i++) {
        float lineStrength = height - fft.getBand(i)*height/2;
        spectrumAvg += lineStrength;
        line(i, height, i, lineStrength);
      }
      spectrumAvg = spectrumAvg / fft.specSize();
      println(spectrumAvg);
    }
  }
}

Combined with tickRate:

import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.spi.*; // for AudioRecordingStream
import ddf.minim.ugens.*;


Minim minim;
FilePlayer player;
FFT fft;

float spectrumAvg;

void setup() {
  fullScreen();
  //size(512, 200);
  minim = new Minim(this);
  selectInput("Select an audio file:", "fileSelected");
}

void fileSelected(File selection) {
  String audioFileName = selection.getAbsolutePath();
  player = new FilePlayer(minim.loadFileStream(audioFileName));
  fft = new FFT(player.bufferSize(), player.sampleRate()); //error
  player.play();
}

void draw()
{
  background(0);
  stroke(255);

  if (player != null) {
    if (fft != null) {
      fft.forward(player.mix); //error

      for (int i = 0; i < fft.specSize(); i++) {
        float lineStrength = height - fft.getBand(i)*height/2;
        spectrumAvg += lineStrength;
        line(i, height, i, lineStrength);
      }
      spectrumAvg = spectrumAvg / fft.specSize();
      println(spectrumAvg);
    }
  }
}

kfrajer

One example uses a FilePlayer object and in the first one an AudioPlayer object. They are different and they have different features. You need to check the documentation page to become familiar with their respective features.

Try exploring minim AudioPlayer associated with rate or speed. You are in the right track working directly from the documentation. Also explore the examples provided in the library. These libraries have really good starting material for most projects.

Kf

SnailPropulsionLabs

So as far as I can see in the documentation:

AudioPlayer: No speed or rate method I can find.

FilePlayer: Has a tickRate to affect playback speed but doesn't work with fft due to not having a buffersize or mix method. What would need to be passed instead of the bufferSize? Or how do I access the bufferSize of the FilePlayer Object?

Am I missing something? I hope it's not a matter of FFT not able to work with a FilePlayer object.

kfrajer

Check the following posts.

http://stackoverflow.com/questions/8448859/fast-sound-analysis-in-processing-minim-library

https://forum.processing.org/one/topic/playing-a-sample-at-different-speeds-frequencies.html

It looks like one option is to use beads. From this post, it seems you already have experience using it? https://forum.processing.org/two/discussion/comment/93566/#Comment_93566

Related to the first option, you read the FFT data in setup and then you could play it at a different rate maybe sending it to an AudioOut object? If you play it faster, then you will have to make sure you play multiple buffers.

Kf

SnailPropulsionLabs

I looked at the tickrate example and realised that it accesses the bufferSize through the output, so I gave it a try in my code below.

It plays the sound but nothing appears on screen and the

fft.forward(player.mix); 

doesn't work. Is this on the right track, or am I grasping at straws?

import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.spi.*; 
import ddf.minim.ugens.*;

Minim minim;
TickRate rateControl;
FilePlayer player;
AudioOutput out;
FFT fft;

float spectrumAvg;

void setup() {
  //fullScreen();
  size(512, 200);
  minim = new Minim(this);
  selectInput("Select an audio file:", "fileSelected");
}

void fileSelected(File selection) {
  String audioFileName = selection.getAbsolutePath();
  player = new FilePlayer(minim.loadFileStream(audioFileName));
  out = minim.getLineOut();
  fft = new FFT(out.bufferSize(), player.sampleRate()); 
  rateControl = new TickRate(1.f);
  player.patch(rateControl).patch(out);
  rateControl.setInterpolation(true);
  player.loop(0);
}

void draw(){
  background(0);
  stroke(255);

  if (player != null) {
    if (fft != null) {
      fft.forward(player.mix); //error

      for (int i = 0; i < fft.specSize(); i++) {
        float lineStrength = height - fft.getBand(i)*height/2;
        spectrumAvg += lineStrength;
        line(i, height, i, lineStrength);
      }
      spectrumAvg = spectrumAvg / fft.specSize();
      println(spectrumAvg);
    }
  }
}

That thread you link to was when I was trying to work out how to speed up playback, your suggestion therein lead me to Beads but Beads is unable to play my 90min+ mp3s.

kfrajer

90min+ mp3s.

Can you break down your mp3 in multiple files? And then play one after the other one?

Have you tested loading this file in processing before you do this implementation? just curious...

Kf

SnailPropulsionLabs

Beads just hangs when loading it even without any complex implementations, no error. Minim plays it without issue.

Breaking it up would really be a last resort as I'd need to break it into 1440 segments and ffmpeg rounds to the nearest second.

SnailPropulsionLabs

Solved, to get it working all it took was fff.forward(out.mix); Facepalm, so simple.

kfrajer

Solved, to get it working all it took was fff.forward(out.mix); Facepalm, so simple.

Just to clarify, you use an FFT to adjust the speed of your file since there was not speed available in the AudioPlayer object, right? And is the out object an instance of AudioPlayer?

Kf

SnailPropulsionLabs

This is what I ended up with

import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.spi.*; 
import ddf.minim.ugens.*;

Minim minim;
TickRate rateControl;
FilePlayer player;
AudioOutput out;
FFT fft;

float spectrumAvg;

void setup() {
  fullScreen();
  minim = new Minim(this);
  selectInput("Select an audio file:", "fileSelected");
}

void fileSelected(File selection) {
  String audioFileName = selection.getAbsolutePath(); //loading the selected file
  player = new FilePlayer(minim.loadFileStream(audioFileName)); //initialising the filePlayer object
  out = minim.getLineOut(); //initialising the audioOut object, no out, no sound.
  fft = new FFT(out.bufferSize(), player.sampleRate()); //initialising the FFT object, setting the out buffersize to the selected file samplerate
  rateControl = new TickRate(1.f); //initialising the tickRate object
  player.patch(rateControl).patch(out); //building the UGen chain, patching the player through the tickRate and into the out object
  rateControl.setInterpolation(true); //stops the audio from being "scratchy" lower speeds
  player.loop(0);
}

void draw() {
  background(0);
  stroke(255);

  if (player != null) {
    if (fft != null) {
      fft.forward(out.mix); //combining the left and right channels

      for (int i = 0; i < fft.specSize(); i++) {
        float lineStrength = height - fft.getBand(i)*height/2;
        spectrumAvg += lineStrength;
        line(i, height, i, lineStrength);
      }
      spectrumAvg = spectrumAvg / fft.specSize();
      println(spectrumAvg);
      strokeWeight(5);
      stroke(255, 125, 0);
      line(0, spectrumAvg, width, spectrumAvg);

      line(0, height/5+height/5*3, width, height/5+height/5*3);
      line(0, height/5+height/5*2, width, height/5+height/5*2);
      line(0, height/5+height/5, width, height/5+height/5);
      line(0, height/5, width, height/5);

      if(spectrumAvg < height && spectrumAvg > height/5+height/5*3)              rateControl.value.setLastValue(3.f);
      if(spectrumAvg < height/5+height/5*3 && spectrumAvg > height/5+height/5*2) rateControl.value.setLastValue(6.f);
      if(spectrumAvg < height/5+height/5*2 && spectrumAvg > height/5+height/5)   rateControl.value.setLastValue(12.f);
      if(spectrumAvg < height/5+height/5 && spectrumAvg > 0)                     rateControl.value.setLastValue(24.f);
    }
  }
}

AudioPlayer is self contained and as such doesn't seem to be able to be used in a UGen chain, this is why I had to use FilePlayer, all the same methods available and able to be used in a UGen chain.

The FFT is used to get the spectrum data(frequency bands) from the input file, I used the bufferSize of the AuidioOutput for this purpose as the FilePlayer object doesn't have a bufferSize method.

I took the average of the FFT and am using this in conjunction with tickRate to control playback speed.

AudioOut, FilePlayer, TickRate & FFT are UGens. AudioPlayer is not.

Hope that's clearer, Some of this might not be correct as I was teaching myself as I did this. Those examples are wonderful.

jeremydouglass

@SnailPropulsionLabs -- thanks so much for sharing your solution with the forum.

SnailPropulsionLabs

np, It makes it a more valuable resource.