How to sync calls to music manually

ryan71 · June 2015

Hi Everyone,

I’m hoping to get some advice on a Processing graphics/music project that I’d like to build. What I’d like to do is sequence Processing function calls to music. And to clarify, I want to sequence the calls manually, not use any kind of beat detection or spectrum analysis. The problem is that I need to make several hundred calls or more during a song and coding this manually will be tedious. I need something that will help automate (as much as possible) the process of sequencing the calls.

Ideally, I’d like to have some kind of software that shows the whole timeline of a song (possibly with a waveform or other analysis superimposed) and let’s me graphically choose (pre-written) Processing functions to call at any point in the timeline. The functions are then somehow associated with the chosen timestamps and then recorded for use with a sketch. What I don’t want is to hand write timestamps for every call (or the calls themselves for that matter).

I’ve researched various VJ software packages, but I haven’t found anything yet that can do this. Also, most VJ software includes functionality I don’t need.

Does anyone know of software that can do what I want? Perhaps there is something similar out there. Or are there any other suggestions or methods I might think about? I know that even with some automation, the job is still going to be quite tedious. I’m with okay with that. This is part of a larger art project I’m working on.

Any help is most appreciated! Thanks!

Ryan

colouredmirrorball · June 2015

Do you already have the effects made? What is your experience level with coding?

You could program a timeline where you can place effects and adjust parameters. The effects would need to be uniformly written for this to work. The problem is that timeline editing works at runtime while effect creation would happen in code. You could either:

(a) Create the timeline environment with a file saving feature which saves all Events on the timeline and an identifier (like event name). When you want to create a new effect, code it as a TimelineEvent and add the entry to the input file parser. Upside is that you can use Processing code for your effects, downside is that you have to recompile and reopen the show when creating/editing an effect

(b) Figure out a way to create effects at runtime (this can be very complicated)

Something like this is on my list of things I want to do but maybe not immediately.

Chrisir · June 2015

I understand that you have one wave-file like 4 min long and that at certain points you want to show a graphical event

yeah, I'd go with (a)

you have two programs:

one program A to set the points (in millis from start) and to save them with a name (or index, but be aware that can become clumsy when you insert new points) each as file F1
another program B to load F1, play the song and work through F1

actually I think this is very easy:

for A just use an ArrayList and fill it with mouseClicks map mouseX to millis

eventMillis = map (mouseX, 0, width, 0, songLengthInMillis);

;-)

Chrisir · June 2015

//import 
import ddf.minim.*;
import ddf.minim.analysis.*;

ArrayList<Event> events;

// song stuff
Minim minim;
AudioPlayer song;
// AudioMetaData meta;

int songLength = 0;

// Other vars
// frequence display 
FFT fft;

boolean showFFT      = true;   // show fft yes/no

// --------------------------------------

void setup() {
  size(displayWidth-40, 200);
  //  frameRate(10);
  smooth();
  noStroke();
  background(255);

  minim = new Minim(this);

  events = new ArrayList();

  song = minim.loadFile( "test.mp3" ); 
  songLength=song.length();

  // an FFT needs to know how
  // long the audio buffers it will be analyzing are
  // and also needs to know
  // the sample rate of the audio it is analyzing
  fft = new FFT(song.bufferSize(), song.sampleRate());

  song.play();
  //  millisStarted = millis();
} // func 

void draw() {
  background(255);

  fill(222, 222, 2);
  rect(0, height-30, width, 30); 

  fill(0); 
  text("Click in the yellow zone to cue. \n"
    +"Click above it to place a grapical event. Press s to save (OVERWRITING).\n\n"
    +"Small line shows song, other line shows mouse.", 20, 20);

  // line mouse 
  stroke(25, 24, 0);
  line (mouseX, 150, mouseX, height);

  // line song playing 
  stroke(2, 2, 255);
  int songMillis = int (map (song.position(), 0, songLength, 0, width));
  line (songMillis, 170, songMillis, height);

  // for-loop
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    event.display();
  } // for 

  // a new for-loop to delete Events
  // (it is better to have 2 separate for-loops)
  for (int i = events.size ()-1; i >= 0; i--) {
    Event event = events.get(i);
    if (event.isDead) {
      events.remove(i);
    } // if
  } // for 
  //  println(Events.size());
  showOtherScreenElements();
} // func 

void mousePressed() {

  if (mouseY>height-30) {  
    // if we are in the yellow zone : 
    // choose a position to cue to based on where the user clicked.
    // the length() method returns the length of recording in milliseconds.
    int position = int( map( mouseX, 0, width, 0, song.length() ) );
    song.cue( position );
  } else 
  {
    // else : 
    int xPos = 10;
    int yPos = 0;

    int rectWidth = 10; 
    int rectHeight = 10; 

    float rCol, gCol, bCol;

    int eventMillis = int (map (mouseX, 0, width, 0, songLength));
    String name = str(events.size()); 

    xPos = mouseX; 
    yPos = 150;

    rCol = random(255);
    gCol = random(255);
    bCol = random(255);

    events.add(new Event(xPos, yPos, 
    rectWidth, rectHeight, 
    rCol, gCol, bCol, 
    eventMillis, 
    name));

    println("placed " + name + " at " 
      +  eventMillis );
  }
} // func 

void keyPressed() {
  //
  if (key=='s') {
    String[] target = new String[events.size()];
    // for-loop
    for (int i = 0; i < events.size (); i++) { 
      Event event = events.get(i);
      target[i] = new String(); 
      target[i] = event.name+"#"+str(event.eventMillis) ;
    } // for
    saveStrings ("events.txt", target);
    println ("saved");
  } // if
} // func 


void showOtherScreenElements() {
  // when we listen to a song
  // 
  // first perform a forward fft on one of song's buffers
  // I'm using the mix buffer
  // but you can use any one you like
  if (showFFT) {
    if (!(fft==null)) {
      fft.forward(song.mix);

      stroke(255, 0, 0, 128);
      // draw the spectrum as a series of vertical lines
      // I multiple the value of getBand by 4
      // so that we can see the lines better
      for (int i = 0; i < fft.specSize (); i++)
      {
        line(i, height, i, height - fft.getBand(i)*4);
      } // for
    } // if
  } // if  
  //
  fill(255);
  try {
    // show played 
    String text1 ="Played " 
      + strFromMillis(song.position())  
      + " of " 
        + strFromMillis ( songLength )  
        + "."; 
    fill(0); 
    text ( text1, width - (textWidth(text1))-40, 30 );
    // show pause 
    if (!song.isPlaying()) {
      fill(255);
      text ("pause", width/2-17, 54);
    } // if
  } // try 
  catch (Exception e) {
    // e.printStackTrace();
    // do nothing
  }
  //
} // func 

String strFromMillis ( int m ) {
  // returns a string that represents a given amount of millis "m" 
  // as hours:minutes:seconds 
  // (can't do days etc.)
  float sec;
  int min;
  //
  sec = m / 1000;
  min = floor(sec / 60);  
  sec = floor(sec % 60);

  // over one hour? 
  if (min>59) { 
    int hrs = floor(min / 60);
    min = floor (min % 60); 
    return  hrs+":"
      +nf(min, 2)+":"
      +nf(int(sec), 2);
  } else 
  {
    return min+":"+nf(int(sec), 2);
  }
} // func 

// ==================================================

class Event {  

  // pos
  float x;
  float y;

  // size
  float w;
  float h;

  float r, g, b;

  boolean isDead = false; 

  int eventMillis;
  String name; 

  // constr 
  Event(float tempX, float tempY, 
  float tempW, float tempH, 
  float tempR, float tempG, float tempB, 
  int eventMillis_, 
  String name_ ) {
    x = tempX;
    y = tempY;
    w = tempW;
    h = tempH;
    //    beginPos = tempBegin;
    //    stopPos = tempStop;
    r = tempR;
    g = tempG;
    b = tempB;

    eventMillis = eventMillis_; 
    name        = name_;
  } // constr 

  void display() {
    if (!isDead) {
      fill(r, g, b);
      rect(x, y, w, h);
    }
  }
  //
} // class
//

Chrisir · June 2015

just did program A

Chrisir · June 2015

new version with full edit for name and crs to move events

Chrisir · June 2015

//imports 
import ddf.minim.*;
import ddf.minim.analysis.*;
import controlP5.*;

// global scope 

// Input name
ControlP5 cp5;  
Textfield myTextfield;

// list of events  
ArrayList<Event> events;
// which event from this list is selected 
int selected = -1;  // -1 means none 

// song stuff
Minim minim;
AudioPlayer song;
int songLength = 0;

// Other vars
// frequence display 
FFT fft;
// FFT on / off
boolean showFFT = true;   // show fft yes/no

// --------------------------------------
// core funcs 

void setup() {
  size(displayWidth-40, 200);
  //  smooth();
  background(255);

  events = new ArrayList();

  initializeTexts();

  // song 
  minim = new Minim(this);
  song = minim.loadFile( "test.mp3" ); 
  songLength=song.length();

  // an FFT needs to know how
  // long the audio buffers it will be analyzing are
  // and also needs to know
  // the sample rate of the audio it is analyzing
  fft = new FFT(song.bufferSize(), song.sampleRate());

  // play it 
  song.play();
} // func 

void draw() {
  background(255);

  // yellow zone 
  fill(222, 222, 2);
  noStroke();
  rect(0, height-30, width, 30); 

  // text 
  fill(0); 
  text("Click in the yellow zone to cue. \n"
    +"Click above it to place a grapical event.\n"
    +"Press s to save (OVERWRITING).Press f to toggle FFT. "
    +"Click on an event to select it (use crs and DEL for delete and n for edit then).\n\n"
    +"Small line shows song, other line shows mouse.", 20, 20);

  // line mouse 
  stroke(0, 0, 0); // black  
  line (mouseX, 150, mouseX, height);

  // line song playing 
  stroke(0, 0, 255); // blue 
  int songMillis = int (map (song.position(), 0, songLength, 0, width));
  line (songMillis, 170, songMillis, height);

  // for-loop
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    event.display();
  } // for 

  // a new for-loop to delete Events
  // (it is better to have 2 separate for-loops)
  for (int i = events.size ()-1; i >= 0; i--) {
    Event event = events.get(i);
    if (event.isDead) {
      events.remove(i);
    } // if
  } // for 

  showOtherScreenElements();
} // func 

// ----------------------------------------------
// Inputs 

void mousePressed() {

  // either select a event 
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    if (event.over()) {
      selected = i;
      unselectAll(); 
      event.isSelected = true;
      return;
    }
  }   

  // or cue the song 
  if (mouseY>height-30) {  
    // if we are in the yellow zone : 
    // choose a position to cue to based on where the user clicked.
    // the length() method returns the length of recording in milliseconds.
    int position = int( map( mouseX, 0, width, 0, song.length() ) );
    song.cue( position );
    selected=-1;
    unselectAll();
  } else 
  {
    // else : above : 
    // make a new event 
    int rectWidth = 10; 
    int rectHeight = 10; 

    int eventMillis = int (map (mouseX, 0, width, 0, songLength));
    String name = str(events.size()); 

    int xPos = mouseX; 
    int yPos = 150;

    color col1 = color(random(255), random(255), random(255));

    events.add(new Event(xPos, yPos, 
    rectWidth, rectHeight, 
    col1, 
    eventMillis, 
    name));

    selected=-1;
    unselectAll();

    println("placed " 
      + name + " at " 
      + eventMillis);
  }
} // func 

void keyPressed() {
  //
  if (key==CODED) {
    // key is coded 
    switch(keyCode) {
      // cursor keys 
    case LEFT:
      if (selected>-1) {
        Event event = events.get(selected);
        event.x--;
        if (event.x<=0)
          event.x=0;
        event.eventMillis = int (map (event.x, 0, width, 0, songLength));
      } // if 
      break; 
    case RIGHT:
      if (selected>-1) {
        Event event = events.get(selected);
        event.x++;
        if (event.x>width)
          event.x=width;
        event.eventMillis = int (map (event.x, 0, width, 0, songLength));
      } // if
      break;
    } // switch
  } // if 
  else {
    // key is not coded
    // eval normal key here  
    if (key=='s') {
      String[] target = new String[events.size()];
      // for-loop
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        target[i] = new String(); 
        target[i] = event.name+"#"+str(event.eventMillis) ;
      } // for
      saveStrings ("events.txt", target);
      println ("saved");
    } // if
    else if (key =='f') {
      // toggle 
      showFFT=!showFFT;
    } // else if
    else if (key =='n') {
      //  
      if (selected>-1) {
        myTextfield.setFocus(true);
        myTextfield.setVisible(true);
      }
    } // else if
    else if (key==DELETE) {
      // kill it 
      if (selected>-1) {
        Event event = events.get(selected);
        selected=-1;
        event.isDead=true;
      } // if
    } // else if
  } // else
} // func 

// --------------------------------------------------------
// minor tools 

void unselectAll() {
  for (int i2 = 0; i2 < events.size (); i2++) {
    Event event2 = events.get(i2);
    event2.isSelected=false;
  }
}

void showOtherScreenElements() {
  // when we listen to a song
  // 
  // first perform a forward fft on one of song's buffers
  // I'm using the mix buffer
  // but you can use anyone you like
  if (showFFT) {
    if (!(fft==null)) {
      fft.forward(song.mix);

      stroke(255, 0, 0, 128);
      // draw the spectrum as a series of vertical lines
      // I multiple the value of getBand by 4
      // so that we can see the lines better
      for (int i = 0; i < fft.specSize (); i++)
      {
        line(i, height, i, height - fft.getBand(i)*4);
      } // for
    } // if
  } // if  

  fill(255);
  try {
    // show played 
    String text1 ="Played " 
      + strFromMillis(song.position())  
      + " of " 
        + strFromMillis ( songLength )  
        + "."; 
    fill(0); 
    text ( text1, width - (textWidth(text1))-40, 30 );
    // show pause 
    if (!song.isPlaying()) {
      fill(255);
      text ("pause", width/2-17, 54);
    } // if
  } // try 
  catch (Exception e) {
    // e.printStackTrace();
    // do nothing
  }
  //
} // func 

String strFromMillis ( int m ) {
  // returns a string that represents a given amount of millis "m" 
  // as hours:minutes:seconds 
  // (can't do days etc.)
  float sec;
  int min;
  //
  sec = m / 1000;
  min = floor(sec / 60);  
  sec = floor(sec % 60);

  // over one hour? 
  if (min>59) { 
    int hrs = floor(min / 60);
    min = floor (min % 60); 
    return  hrs+":"
      +nf(min, 2)+":"
      +nf(int(sec), 2);
  } else 
  {
    return min+":"
      +nf(int(sec), 2);
  }
} // func 

void initializeTexts() {

  //  String textValue = "";
  // Textarea myTextarea;

  PFont font = createFont("arial", 20);  

  cp5 = new ControlP5(this);
  cp5.getTooltip().setDelay(500);

  myTextfield = cp5.addTextfield("input")
    .setPosition( width-(width-10), height-60 )
      .setSize(width-20, 40)
        .setFont(font)
          .setFocus(false)
            .setColor(color(225, 225, 225))
              .setVisible(false);
}
//
public void input(String theText) {
  // automatically receives results from controller input
  println("a textfield event for controller 'input' : "+theText);
  String textValue = theText;
  Event event = events.get(selected);
  event.name = textValue ;
  myTextfield.setFocus(false);
  myTextfield.setVisible(false);
}



// ==================================================

class Event {  

  // pos
  float x;
  float y;

  // size
  float w;
  float h;

  color eventColor;

  boolean isDead = false; 

  int eventMillis;
  String name; 

  boolean isSelected = false; 

  // constr 
  Event(float tempX, float tempY, 
  float tempW, float tempH, 
  color eventColor_, 
  int eventMillis_, 
  String name_ ) {
    x = tempX;
    y = tempY;
    w = tempW;
    h = tempH;

    eventColor  = eventColor_;
    eventMillis = eventMillis_; 
    name        = name_;
  } // constr 

  boolean over() {
    return mouseX>x&&mouseY>y&&
      mouseX<x+w&&mouseY<y+h;
  }

  void display() {
    // quit? 
    if (isDead) 
      return; 
    // When seleted     
    if (isSelected) {
      fill(0);
      text (name, x, y-h);
      text (eventMillis, x, y-h-h-2);
      stroke(0);
      fill(eventColor);
      rect(x, y, w, h);
    } // if 
    else {
      if (over())
        stroke(0);
      else
        noStroke();
      fill(eventColor);
      rect(x, y, w, h);
    } // else
  } // method
  //
} // class
//

Chrisir · June 2015

new version

same features, you can edit existing event markers etc.

//imports 
import ddf.minim.*;
import ddf.minim.analysis.*;
import controlP5.*;

// global scope 

String helpText = "Click in the yellow zone to cue. \n"
+"Click above it to place a grapical event.\n"
+"Press s to save (OVERWRITING). Press f to toggle FFT.\n"
+"Click on an event to select it: Then use cursor l/r and DEL for delete and n for edit.\n\n"
+"The small line shows song, the other line shows mouse."; 

final int stateNormal    = 0;
final int stateInputtext = 1;
final int stateBeforeStateInputtext = 3; 
int state = stateNormal; 

// Input name
ControlP5 cp5;  
Textfield myTextfield;

// list of events  
ArrayList<Event> events = new ArrayList();
// which event from this list is selected 
int selected = -1;  // -1 means none 

// song stuff
Minim minim;
AudioPlayer song;
int songLength = 0;

// Other vars
// frequence display 
FFT fft;
// FFT on / off
boolean showFFT = true;   // show fft yes/no

// --------------------------------------
// core funcs 

void setup() {
  size(displayWidth-40, 200);
  background(255);

  initializeTextfield();

  // song 
  minim = new Minim(this);
  song = minim.loadFile( "test.mp3" ); 
  songLength=song.length();

  // an FFT needs to know how
  // long the audio buffers it will be analyzing are
  // and also needs to know
  // the sample rate of the audio it is analyzing
  fft = new FFT(song.bufferSize(), song.sampleRate());

  // play it 
  song.play();
} // func 

void draw() {
  switch (state) {
  case stateNormal:
    drawForStateNormal(); 
    break; 
  case stateInputtext:
    // drawForStateInputtext(); 
    // same as above: 
    drawForStateNormal(); 
    break;
  case stateBeforeStateInputtext:
    // init the next state 
    drawForStateBeforeStateInputtext(); 
    break;
  } // switch
} // func 

// --------------------------------------
// state funcs 

void drawForStateNormal() {

  background(255); // white 

  // yellow zone 
  fill(222, 222, 2);
  noStroke();
  rect(0, height-30, width, 30); 

  // text 
  fill(0);  // black    
  text(helpText, 20, 20);

  // line mouse
  if (state==stateNormal) { 
    stroke(0, 0, 0); // black  
    line (mouseX, 150, mouseX, height);
  }

  // line song playing 
  stroke(0, 0, 255); // blue 
  int songMillis = int (map (song.position(), 0, songLength, 0, width));
  line (songMillis, 170, songMillis, height);

  // for-loop
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    event.display();
  } // for 

  // a new for-loop to delete Events
  // (it is better to have 2 separate for-loops)
  for (int i = events.size ()-1; i >= 0; i--) {
    Event event = events.get(i);
    if (event.isDead) {
      events.remove(i);
    } // if
  } // for 

  showOtherScreenElements();
} // func 

//void drawForStateInputtext() {
//
//} // func 

void drawForStateBeforeStateInputtext() {
  // init the next state 
  myTextfield.setFocus(true);
  myTextfield.setVisible(true);
  state = stateInputtext;
}

// ----------------------------------------------
// Inputs 

void mousePressed() {

  // only state normal is allowed in this func 
  if (state!=stateNormal)
    return; 

  // either select an existing event 
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    if (event.over()) {
      selected = i;
      unselectAll(); 
      event.isSelected = true;
      return;
    }
  }  // for 

  // or cue the song 
  if (mouseY>height-30) {  
    // if we are in the yellow zone : 
    // choose a position to cue to based on where the user clicked.
    // the length() method returns the length of recording in milliseconds.
    int position = int( map( mouseX, 0, width, 0, song.length() ) );
    song.cue( position );
    selected=-1;
    unselectAll();
  } else 
  {
    // else : above : 
    // or make a new event 
    int rectWidth = 10; 
    int rectHeight = 10; 

    int eventMillis = int (map (mouseX, 0, width, 0, songLength));
    String name = str(events.size()); 

    int xPos = mouseX; 
    int yPos = 150;

    color col1 = color(random(255), random(255), random(255));

    events.add(new Event(xPos, yPos, 
    rectWidth, rectHeight, 
    col1, 
    eventMillis, 
    name));

    selected=-1;
    unselectAll();

    println("placed " 
      + name + " at " 
      + eventMillis);
  } // else
} // func 

void keyPressed() {
  if (state==stateNormal) {
    keyPressedForStateNormal();
  } else if (state==stateInputtext) {
    if (key==ESC) {
      key=0;
      // close the text field 
      myTextfield.setFocus(false);
      myTextfield.setVisible(false);
      state=stateNormal;
    }//if
  }// state
}// func 

void keyPressedForStateNormal() {  
  //
  if (key==CODED) {
    // key is coded 
    switch(keyCode) {
      // cursor keys 
    case LEFT:
      if (selected>-1) {
        Event event = events.get(selected);
        event.eventMillis--;
        if (event.eventMillis<=0)
          event.eventMillis=0;
        // event.eventMillis = int (map (event.x, 0, width, 0, songLength));
        event.x = map (event.eventMillis, 0, songLength, 0, width );
      } // if 
      break; 
    case RIGHT:
      if (selected>-1) {
        Event event = events.get(selected);
        event.eventMillis++;
        if (event.eventMillis>songLength)
          event.eventMillis=songLength;
        //event.eventMillis = int (map (event.x, 0, width, 0, songLength));
        event.x = map (event.eventMillis, 0, songLength, 0, width );
      } // if
      break;
    } // switch
  } // if 
  // -------------------------------------------------------------
  else {
    // key is not coded
    // eval normal key here  
    if (key=='s') {
      String[] target = new String[events.size()];
      // for-loop
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        target[i] = new String(); 
        target[i] = event.name+"#"+str(event.eventMillis) ;
      } // for
      saveStrings ("events.txt", target);
      println ("saved");
    } // if
    else if (key =='f') {
      // toggle 
      showFFT=!showFFT;
    } // else if
    else if (key =='n') {
      //  get name 
      if (selected>-1) {
        key=0;
        // assign the old event name to the text field
        Event event = events.get(selected);
        myTextfield.setValue(event.name); 
        state = stateBeforeStateInputtext;
      }
    } // else if
    else if (key==DELETE) {
      // kill it 
      if (selected>-1) {
        Event event = events.get(selected);
        selected=-1;
        event.isDead=true;
      } // if
    } // else if
    else if (key==ESC) {
      // kill esc anyway 
      key=0;
    } else if (key=='X') {
      exit();
    }
  } // else
} // func 

// --------------------------------------------------------
// minor tools 

void unselectAll() {
  for (int i2 = 0; i2 < events.size (); i2++) {
    Event event2 = events.get(i2);
    event2.isSelected=false;
  }
} // func 

void showOtherScreenElements() {
  // when we listen to a song
  // 
  // first perform a forward fft on one of song's buffers
  // I'm using the mix buffer
  // but you can use anyone you like
  if (showFFT) {
    if (!(fft==null)) {
      fft.forward(song.mix);

      stroke(255, 0, 0, 128);
      // draw the spectrum as a series of vertical lines
      // I multiple the value of getBand by 4
      // so that we can see the lines better
      for (int i = 0; i < fft.specSize (); i++)
      {
        line(i, height, i, height - fft.getBand(i)*4);
      } // for
    } // if
  } // if  

  fill(255);
  try {
    // show played time 
    String text1 ="Played " 
      + strFromMillis(song.position())  
      + " of " 
        + strFromMillis ( songLength )  
        + "."; 
    fill(0); 
    text ( text1, width - (textWidth(text1))-40, 30 );
    // show pause 
    if (!song.isPlaying()) {
      fill(255);
      text ("pause", width/2-17, 54);
    } // if
  } // try 
  catch (Exception e) {
    // e.printStackTrace();
    // do nothing
  }
  //
} // func 

String strFromMillis(int m) {
  // returns a string that represents a given amount of millis "m" 
  // as hours:minutes:seconds 
  // (can't do days etc.).
  float sec;
  int min;
  //
  sec = m / 1000;
  min = floor(sec / 60);  
  sec = floor(sec % 60);

  // over one hour? 
  if (min>59) { 
    int hrs = floor(min / 60);
    min = floor (min % 60); 
    return  hrs+":"
      +nf(min, 2)+":"
      +nf(int(sec), 2);
  } else 
  {
    return min+":"
      +nf(int(sec), 2);
  }
} // func

// --------------------------------------------------
// tools for the textfield 

void initializeTextfield() {

  PFont font = createFont("arial", 20);  

  cp5 = new ControlP5(this);
  cp5.getTooltip().setDelay(500);

  myTextfield = cp5.addTextfield("input")
    .setPosition( 40, 60 )
      .setSize(320, 40)
        .setFont(font)
          .setFocus(false)
            .setColor(color(225, 225, 225))
              .setVisible(false);
} // func

public void input(String theText) {
  // automatically receives results from controller input
  // after hitting return 
  println("a textfield event for controller 'input' : "+theText);
  String textValue = theText;
  if (textValue!=null && !textValue.equals("")) {
    Event event = events.get(selected);
    event.name = textValue;
  }
  // close the text field
  myTextfield.setFocus(false);
  myTextfield.setVisible(false);
  state=stateNormal;
}

// ==================================================

class Event {  

  // pos
  float x;
  float y;

  // size
  float w;
  float h;

  color eventColor;

  boolean isDead = false; 

  int eventMillis;
  String name; 

  boolean isSelected = false; 

  // constr 
  Event(float tempX, float tempY, 
  float tempW, float tempH, 
  color eventColor_, 
  int eventMillis_, 
  String name_ ) {
    x = tempX;
    y = tempY;
    w = tempW;
    h = tempH;

    eventColor  = eventColor_;
    eventMillis = eventMillis_; 
    name        = name_;
  } // constr 

  boolean over() {
    return mouseX>x&&mouseY>y&&
      mouseX<x+w&&mouseY<y+h;
  }

  void display() {
    // quit? 
    if (isDead) 
      return; 
    // When seleted     
    if (isSelected) {
      fill(0);
      text (name, x, y-h);
      text (eventMillis, x, y-h-h-2);
      stroke(0);
      fill(eventColor);
      rect(x, y, w, h);
    } // if 
    else {
      if (over())
        stroke(0);
      else
        noStroke();
      fill(eventColor);
      rect(x, y, w, h);
    } // else
  } // method
  //
} // class
//

Chrisir · June 2015

this was all program A now I show program B

(a lot of this is just show, e.g. the yellow bar etc. )

//imports 
import ddf.minim.*;
import ddf.minim.analysis.*;

boolean fileHasBeenSelected = false; // marker 
String waitText = "Select a event list file to process.";

String [][] grid;

String eventText = ""; 

// song stuff
Minim minim;
AudioPlayer song;
int songLength = 0;

// Other vars
// frequence display 
FFT fft;
// FFT on / off
boolean showFFT = true;   // show fft yes/no


// --------------------------------------
// core funcs 

void setup() {
  size(displayWidth-40, 200);
  background(111); 

  // song 
  minim = new Minim(this);
  song = minim.loadFile( "test.mp3" ); 
  songLength=song.length();

  // an FFT needs to know how
  // long the audio buffers it will be analyzing are
  // and also needs to know
  // the sample rate of the audio it is analyzing
  fft = new FFT(song.bufferSize(), song.sampleRate());

  println (sketchPath(""));
  String sp1 = sketchPath("");

  File f = new File(sp1+"*.txt");

  selectInput("Select a file to process: ", "fileSelected", f);
} // func

void draw() {
  background(111); 
  // evaluate the marker 
  if (!fileHasBeenSelected) {
    // wait
    fill(0); 
    text (waitText, 30, 30);
  } else {
    // main part
    drawWhenEventFileHasBeenLoaded();
  } // else
} // func 

// ----------------------------------------------------

void drawWhenEventFileHasBeenLoaded() {

  // main part

  // yellow zone 
  fill(222, 222, 2);
  noStroke();
  rect(0, height-30, width, 30); 

  // line song playing 
  stroke(0, 0, 255); // blue 
  int songMillis = int (map (song.position(), 0, songLength, 0, width));
  line (songMillis, 170, songMillis, height);

  fill(222, 222, 2);
  text (eventText, 100, 70);

  for  (int i = 0; i < grid.length; i++) {
    // display the rects 
    float eventX = map (int(grid[i][1]), 0, songLength, 0, width );
    fill (2, 3, 222);
    rect (eventX, 100, 10, 10);
    // check if the position matches any events  
    if (!grid[i][2].equals("X") && song.position() >= int(grid[i][1])) {
      // yes 
      grid[i][2]="X"; // mark as done 
      evaluateCurrentEvent(grid[i][0]);
    } // if
  } // for
} // func 

void evaluateCurrentEvent(String eventName) {

  if (eventName.equals("1")) {
    eventText="ONE";
  } else if (eventName.equals("2")) {
    eventText="TWO";
  } // if
  // ..............
  else {
    eventText="Unkown event occured. It had the name "
      + eventName + ".";
  }
}

void fileSelected(File selection) {
  if (selection == null) {
    println("Window was closed or the user hit cancel.");
    waitText="Window was closed or the user hit cancel.";

    //    File f = new File("*.txt");
    //    selectInput("Select a file to process:    ", "fileSelected", f);
  } else {
    println("User selected " + selection.getAbsolutePath());
    String theNameThatHasBeenEntered  = selection.getAbsolutePath(); 
    // img = loadImage( theNameThatHasBeenEntered );

    String[] events;

    events = loadStrings (theNameThatHasBeenEntered);
    for (String s1 : events)  
      printArray(s1); 

    //create grid array based on # of rows and columns in file
    grid = new String [events.length][3];
    //
    int count=0;
    //parse values into 2d array
    for (int i=0; i < events.length; i++) {
      String [] temp;
      temp = split(events[i], '#');
      for (int j=0; j < temp.length; j++) {
        grid[i][j]=temp[j];
        grid[i][2]="";
        count++;
      }
    } // for

    println ("Strings in total: " + count );

    // set marker 
    fileHasBeenSelected = true;
    // play it 
    song.play();
  }
}

//

Chrisir · June 2015

file F1 looks like this btw.

ryan71 · June 2015

Hi colouredmirrorball,

Thanks for your response. I’m pretty to new to Processing. I’ve made my way through several tutorials and parts of books. I’ve played around with Minim a bit. But, I’m just now starting to work on the graphics calls. I wanted to first make sure, that there wasn’t some code/framework/software package already existing that does what I want. Based on responses so far, it doesn’t seem like it (this includes a post to vjforums.info).

Yes, as chrisir has already confirmed, I assumed that there would be a separate program where events are placed/sequenced. I wasn’t sure if Processing would be good for that, but chrisr seems to have answered that.

Hi chrisir,

Wow! Thanks so much for this! It’s pretty much what I was envisioning. How long did it take you? Are you wanting/needing a similar tool? Or into it for the exercise?

I need to add a bunch of stuff, so I’ll take your code and run with it. Here’s a few things off the top of my head, in case you want to chime in about any of them (btw, I'm not expecting you to code any of this):

Since I need to cram so many calls into the song timeline, I’m gonna reduce the little squares to probably single or double width lines. Also, I’ll want add parallel rows (to your initial row of squares) where I can add calls for, say, specific instruments or sounds. This would allow me to separate instruments/sound effects into their own lines—which would add some organisation and ease of navigation to the calls.
I also need to have some ability to run the music at various speeds (ie., slow it way down) and easily start/stop and step through it, to make precise call placement easier.
A display of current time in the song (in ms) will help with this too.
I'd like the ability to read-in specially marked functions (eg. specifically formatted comments could be used for this) from another sketch, then upon placement of calls on the timeline, I am prompted to choose which function I want. This would then be written to the text file.

I've already played around with your sketches and looked at the text file. I think I might whip-up a simple sketch that reads the file and does some graphics….just as a proof-of-concept, before I get too far into developing.

Thanks again for this! It's really nice start to have. Much appreciated!

Ryan

Chrisir · June 2015

here is a new version of sketch A which produces a short passage of source code to paste into sketch B

the source code just comes via println

// this writes events in a file 

// this file can then be read by 
//          Ball_ArraylistMouseReadEvents1

//imports 
import ddf.minim.*;
import ddf.minim.analysis.*;

import controlP5.*;

// global scope 

String helpText = "Click in the yellow zone to cue. \n"
+"Click above it to place a grapical event.\n"
+"Press s to save (OVERWRITING). Press f to toggle FFT.\n"
+"Click on an event to select it: Then use cursor l/r and DEL for delete and n for edit.\n\n"
+"The small line shows song, the other line shows mouse."; 

final int stateNormal    = 0;
final int stateInputtext = 1;
final int stateBeforeStateInputtext = 3; 
int state = stateNormal; 

// Input name
ControlP5 cp5;  
Textfield myTextfield;

// list of events  
ArrayList<Event> events = new ArrayList();
// which event from this list is selected 
int selected = -1;  // -1 means none 

// song stuff
Minim minim;
AudioPlayer song;
int songLength = 0;

// Other vars
// frequence display 
FFT fft;
// FFT on / off
boolean showFFT = true;   // show fft yes/no

// --------------------------------------
// core funcs 

void setup() {
  size(displayWidth-40, 200);
  background(255);

  initializeTextfield();

  // song 
  minim = new Minim(this);
  song = minim.loadFile( "test.mp3" ); 
  songLength=song.length();

  // an FFT needs to know how
  // long the audio buffers it will be analyzing are
  // and also needs to know
  // the sample rate of the audio it is analyzing
  fft = new FFT(song.bufferSize(), song.sampleRate());

  // play it 
  song.play();
} // func 

void draw() {
  switch (state) {
  case stateNormal:
    drawForStateNormal(); 
    break; 
  case stateInputtext:
    // drawForStateInputtext(); 
    // same as above: 
    drawForStateNormal(); 
    break;
  case stateBeforeStateInputtext:
    // init the next state 
    drawForStateBeforeStateInputtext(); 
    break;
  } // switch
} // func 

// --------------------------------------
// state funcs 

void drawForStateNormal() {

  background(255); // white 

  // yellow zone 
  fill(222, 222, 2);
  noStroke();
  rect(0, height-30, width, 30); 

  // text 
  fill(0);  // black    
  text(helpText, 20, 20);

  // line mouse
  if (state==stateNormal) { 
    stroke(0, 0, 0); // black  
    line (mouseX, 150, mouseX, height);
  }

  // line song playing 
  stroke(0, 0, 255); // blue 
  int songMillis = int (map (song.position(), 0, songLength, 0, width));
  line (songMillis, 170, songMillis, height);

  // for-loop
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    event.display();
  } // for 

  // a new for-loop to delete Events
  // (it is better to have 2 separate for-loops)
  for (int i = events.size ()-1; i >= 0; i--) {
    Event event = events.get(i);
    if (event.isDead) {
      events.remove(i);
    } // if
  } // for 

  showOtherScreenElements();
} // func 

//void drawForStateInputtext() {
//
//} // func 

void drawForStateBeforeStateInputtext() {
  // init the next state 
  myTextfield.setFocus(true);
  myTextfield.setVisible(true);
  state = stateInputtext;
}

// ----------------------------------------------
// Inputs 

void mousePressed() {

  // only state normal is allowed in this func 
  if (state!=stateNormal)
    return; 

  // either select an existing event 
  for (int i = 0; i < events.size (); i++) { 
    Event event = events.get(i);
    if (event.over()) {
      selected = i;
      unselectAll(); 
      event.isSelected = true;
      return;
    }
  }  // for 

  // or cue the song 
  if (mouseY>height-30) {  
    // if we are in the yellow zone : 
    // choose a position to cue to based on where the user clicked.
    // the length() method returns the length of recording in milliseconds.
    int position = int( map( mouseX, 0, width, 0, song.length() ) );
    song.cue( position );
    selected=-1;
    unselectAll();
  } else 
  {
    // else : above : 
    // or make a new event 
    int rectWidth = 10; 
    int rectHeight = 10; 

    int eventMillis = int (map (mouseX, 0, width, 0, songLength));
    String name = str(events.size()); 

    int xPos = mouseX; 
    int yPos = 150;

    color col1 = color(random(255), random(255), random(255));

    events.add(new Event(xPos, yPos, 
    rectWidth, rectHeight, 
    col1, 
    eventMillis, 
    name));

    selected=-1;
    unselectAll();

    println("placed " 
      + name + " at " 
      + eventMillis);
  } // else
} // func 

void keyPressed() {
  if (state==stateNormal) {
    keyPressedForStateNormal();
  } else if (state==stateInputtext) {
    if (key==ESC) {
      key=0;
      // close the text field 
      myTextfield.setFocus(false);
      myTextfield.setVisible(false);
      state=stateNormal;
    }//if
  }// state
}// func 

void keyPressedForStateNormal() {  
  //
  if (key==CODED) {
    // key is coded 
    switch(keyCode) {
      // cursor keys 
    case LEFT:
      if (selected>-1) {
        Event event = events.get(selected);
        event.eventMillis--;
        if (event.eventMillis<=0)
          event.eventMillis=0;
        // event.eventMillis = int (map (event.x, 0, width, 0, songLength));
        event.x = map (event.eventMillis, 0, songLength, 0, width );
      } // if 
      break; 
    case RIGHT:
      if (selected>-1) {
        Event event = events.get(selected);
        event.eventMillis++;
        if (event.eventMillis>songLength)
          event.eventMillis=songLength;
        //event.eventMillis = int (map (event.x, 0, width, 0, songLength));
        event.x = map (event.eventMillis, 0, songLength, 0, width );
      } // if
      break;
    } // switch
  } // if 
  // -------------------------------------------------------------
  else {
    // key is not coded
    // eval normal key here  
    if (key=='s') {
      // save ---
      String[] target = new String[events.size()];
      println(); 
      println("source code - please remove the first \"} else\".");
      println("please paste source code into the program that reads the event file in function evaluateCurrentEvent");
      println();  
      // 
      boolean firstTime = true; 
      // for-loop over all events 
      for (int i = 0; i < events.size (); i++) {
        // preparing output file  
        Event event = events.get(i);
        target[i] = new String(); 
        target[i] = event.name+"#"+str(event.eventMillis);
        // preparing source code 
        if (firstTime) {
          println ( "if (eventName.equals(\"" + event.name +"\")) {" ) ;
          firstTime=false;
        } else {
          println ( "} else if (eventName.equals(\"" + event.name +"\")) {" ) ;
        }
        println ( "      // .... ");  
        println ( "} // if");
      } // for
      println(); 
      println(); 
      // writing the file 
      saveStrings ("events.txt", target);
      println ("saved");
    } // if
    else if (key =='f') {
      // toggle 
      showFFT=!showFFT;
    } // else if
    else if (key =='n') {
      //  get name 
      if (selected>-1) {
        key=0;
        // assign the old event name to the text field
        Event event = events.get(selected);
        myTextfield.setValue(event.name); 
        state = stateBeforeStateInputtext;
      }
    } // else if
    else if (key==DELETE) {
      // kill it 
      if (selected>-1) {
        Event event = events.get(selected);
        selected=-1;
        event.isDead=true;
      } // if
    } // else if
    else if (key==ESC) {
      // kill esc anyway 
      key=0;
    } else if (key=='X') {
      exit();
    } // else
  } // else
} // func 

// --------------------------------------------------------
// minor tools 

void unselectAll() {
  for (int i2 = 0; i2 < events.size (); i2++) {
    Event event2 = events.get(i2);
    event2.isSelected=false;
  }
} // func 

void showOtherScreenElements() {
  // when we listen to a song
  // 
  // first perform a forward fft on one of song's buffers
  // I'm using the mix buffer
  // but you can use anyone you like
  if (showFFT) {
    if (!(fft==null)) {
      fft.forward(song.mix);

      stroke(255, 0, 0, 128);
      // draw the spectrum as a series of vertical lines
      // I multiple the value of getBand by 4
      // so that we can see the lines better
      for (int i = 0; i < fft.specSize (); i++)
      {
        line(i, height, i, height - fft.getBand(i)*4);
      } // for
    } // if
  } // if  

  fill(255);
  try {
    // show played time 
    String text1 ="Played " 
      + strFromMillis(song.position())  
      + " of " 
        + strFromMillis ( songLength )  
        + "."; 
    fill(0); 
    text ( text1, width - (textWidth(text1))-40, 30 );
    // show pause 
    if (!song.isPlaying()) {
      fill(255);
      text ("pause", width/2-17, 54);
    } // if
  } // try 
  catch (Exception e) {
    // e.printStackTrace();
    // do nothing
  }
  //
} // func 

String strFromMillis(int m) {
  // returns a string that represents a given amount of millis "m" 
  // as hours:minutes:seconds 
  // (can't do days etc.).
  float sec;
  int min;
  //
  sec = m / 1000;
  min = floor(sec / 60);  
  sec = floor(sec % 60);

  // over one hour? 
  if (min>59) { 
    int hrs = floor(min / 60);
    min = floor (min % 60); 
    return  hrs+":"
      +nf(min, 2)+":"
      +nf(int(sec), 2);
  } else 
  {
    return min+":"
      +nf(int(sec), 2);
  }
} // func

// --------------------------------------------------
// tools for the textfield 

void initializeTextfield() {

  PFont font = createFont("arial", 20);  

  cp5 = new ControlP5(this);
  cp5.getTooltip().setDelay(500);

  myTextfield = cp5.addTextfield("input")
    .setPosition( 40, 60 )
      .setSize(320, 40)
        .setFont(font)
          .setFocus(false)
            .setColor(color(225, 225, 225))
              .setVisible(false);
} // func

public void input(String theText) {
  // automatically receives results from controller input
  // after hitting return 
  println("a textfield event for controller 'input' : "+theText);
  String textValue = theText;
  if (textValue!=null && !textValue.equals("")) {
    Event event = events.get(selected);
    event.name = textValue;
  }
  // close the text field
  myTextfield.setFocus(false);
  myTextfield.setVisible(false);
  state=stateNormal;
}

// ==================================================

class Event {  

  // pos
  float x;
  float y;

  // size
  float w;
  float h;

  color eventColor;

  boolean isDead = false; 

  int eventMillis;
  String name; 

  boolean isSelected = false; 

  // constr 
  Event(float tempX, float tempY, 
  float tempW, float tempH, 
  color eventColor_, 
  int eventMillis_, 
  String name_ ) {
    x = tempX;
    y = tempY;
    w = tempW;
    h = tempH;

    eventColor  = eventColor_;
    eventMillis = eventMillis_; 
    name        = name_;
  } // constr 

  boolean over() {
    return mouseX>x&&mouseY>y&&
      mouseX<x+w&&mouseY<y+h;
  }

  void display() {
    // quit? 
    if (isDead) 
      return; 
    // When seleted     
    if (isSelected) {
      fill(0);
      text (name, x, y-h);
      text (eventMillis, x, y-h-h-2);
      stroke(0);
      fill(eventColor);
      rect(x, y, w, h);
    } // if 
    else {
      if (over())
        stroke(0);
      else
        noStroke();
      fill(eventColor);
      rect(x, y, w, h);
    } // else
  } // method
  //
} // class
//

ryan71 · June 2015

Running the new version now--until my battery dies.

I see you already put in some fine tuning with the l/r keys. Nice.

Chrisir · June 2015

yes, can change the millis now with cursors

also, with n, you can give each event another name

Howdy, Stranger!

Categories

In this Discussion

How to sync calls to music manually

Best Answers

Answers