How to sync calls to music manually

edited June 2015 in Library Questions

Hi Everyone,

I’m hoping to get some advice on a Processing graphics/music project that I’d like to build. What I’d like to do is sequence Processing function calls to music. And to clarify, I want to sequence the calls manually, not use any kind of beat detection or spectrum analysis. The problem is that I need to make several hundred calls or more during a song and coding this manually will be tedious. I need something that will help automate (as much as possible) the process of sequencing the calls.

Ideally, I’d like to have some kind of software that shows the whole timeline of a song (possibly with a waveform or other analysis superimposed) and let’s me graphically choose (pre-written) Processing functions to call at any point in the timeline. The functions are then somehow associated with the chosen timestamps and then recorded for use with a sketch. What I don’t want is to hand write timestamps for every call (or the calls themselves for that matter).

I’ve researched various VJ software packages, but I haven’t found anything yet that can do this. Also, most VJ software includes functionality I don’t need.

Does anyone know of software that can do what I want? Perhaps there is something similar out there. Or are there any other suggestions or methods I might think about? I know that even with some automation, the job is still going to be quite tedious. I’m with okay with that. This is part of a larger art project I’m working on.

Any help is most appreciated! Thanks!

Ryan

Answers

  • Do you already have the effects made? What is your experience level with coding?

    You could program a timeline where you can place effects and adjust parameters. The effects would need to be uniformly written for this to work. The problem is that timeline editing works at runtime while effect creation would happen in code. You could either:

    (a) Create the timeline environment with a file saving feature which saves all Events on the timeline and an identifier (like event name). When you want to create a new effect, code it as a TimelineEvent and add the entry to the input file parser. Upside is that you can use Processing code for your effects, downside is that you have to recompile and reopen the show when creating/editing an effect

    (b) Figure out a way to create effects at runtime (this can be very complicated)

    Something like this is on my list of things I want to do but maybe not immediately.

  • edited June 2015 Answer ✓

    I understand that you have one wave-file like 4 min long and that at certain points you want to show a graphical event

    yeah, I'd go with (a)

    you have two programs:

    • one program A to set the points (in millis from start) and to save them with a name (or index, but be aware that can become clumsy when you insert new points) each as file F1

    • another program B to load F1, play the song and work through F1

    actually I think this is very easy:

    • for A just use an ArrayList and fill it with mouseClicks map mouseX to millis

      eventMillis = map (mouseX, 0, width, 0, songLengthInMillis);

    ;-)

  • Answer ✓
    //import 
    import ddf.minim.*;
    import ddf.minim.analysis.*;
    
    ArrayList<Event> events;
    
    // song stuff
    Minim minim;
    AudioPlayer song;
    // AudioMetaData meta;
    
    int songLength = 0;
    
    // Other vars
    // frequence display 
    FFT fft;
    
    boolean showFFT      = true;   // show fft yes/no
    
    // --------------------------------------
    
    void setup() {
      size(displayWidth-40, 200);
      //  frameRate(10);
      smooth();
      noStroke();
      background(255);
    
      minim = new Minim(this);
    
      events = new ArrayList();
    
      song = minim.loadFile( "test.mp3" ); 
      songLength=song.length();
    
      // an FFT needs to know how
      // long the audio buffers it will be analyzing are
      // and also needs to know
      // the sample rate of the audio it is analyzing
      fft = new FFT(song.bufferSize(), song.sampleRate());
    
      song.play();
      //  millisStarted = millis();
    } // func 
    
    void draw() {
      background(255);
    
      fill(222, 222, 2);
      rect(0, height-30, width, 30); 
    
      fill(0); 
      text("Click in the yellow zone to cue. \n"
        +"Click above it to place a grapical event. Press s to save (OVERWRITING).\n\n"
        +"Small line shows song, other line shows mouse.", 20, 20);
    
      // line mouse 
      stroke(25, 24, 0);
      line (mouseX, 150, mouseX, height);
    
      // line song playing 
      stroke(2, 2, 255);
      int songMillis = int (map (song.position(), 0, songLength, 0, width));
      line (songMillis, 170, songMillis, height);
    
      // for-loop
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        event.display();
      } // for 
    
      // a new for-loop to delete Events
      // (it is better to have 2 separate for-loops)
      for (int i = events.size ()-1; i >= 0; i--) {
        Event event = events.get(i);
        if (event.isDead) {
          events.remove(i);
        } // if
      } // for 
      //  println(Events.size());
      showOtherScreenElements();
    } // func 
    
    void mousePressed() {
    
      if (mouseY>height-30) {  
        // if we are in the yellow zone : 
        // choose a position to cue to based on where the user clicked.
        // the length() method returns the length of recording in milliseconds.
        int position = int( map( mouseX, 0, width, 0, song.length() ) );
        song.cue( position );
      } else 
      {
        // else : 
        int xPos = 10;
        int yPos = 0;
    
        int rectWidth = 10; 
        int rectHeight = 10; 
    
        float rCol, gCol, bCol;
    
        int eventMillis = int (map (mouseX, 0, width, 0, songLength));
        String name = str(events.size()); 
    
        xPos = mouseX; 
        yPos = 150;
    
        rCol = random(255);
        gCol = random(255);
        bCol = random(255);
    
        events.add(new Event(xPos, yPos, 
        rectWidth, rectHeight, 
        rCol, gCol, bCol, 
        eventMillis, 
        name));
    
        println("placed " + name + " at " 
          +  eventMillis );
      }
    } // func 
    
    void keyPressed() {
      //
      if (key=='s') {
        String[] target = new String[events.size()];
        // for-loop
        for (int i = 0; i < events.size (); i++) { 
          Event event = events.get(i);
          target[i] = new String(); 
          target[i] = event.name+"#"+str(event.eventMillis) ;
        } // for
        saveStrings ("events.txt", target);
        println ("saved");
      } // if
    } // func 
    
    
    void showOtherScreenElements() {
      // when we listen to a song
      // 
      // first perform a forward fft on one of song's buffers
      // I'm using the mix buffer
      // but you can use any one you like
      if (showFFT) {
        if (!(fft==null)) {
          fft.forward(song.mix);
    
          stroke(255, 0, 0, 128);
          // draw the spectrum as a series of vertical lines
          // I multiple the value of getBand by 4
          // so that we can see the lines better
          for (int i = 0; i < fft.specSize (); i++)
          {
            line(i, height, i, height - fft.getBand(i)*4);
          } // for
        } // if
      } // if  
      //
      fill(255);
      try {
        // show played 
        String text1 ="Played " 
          + strFromMillis(song.position())  
          + " of " 
            + strFromMillis ( songLength )  
            + "."; 
        fill(0); 
        text ( text1, width - (textWidth(text1))-40, 30 );
        // show pause 
        if (!song.isPlaying()) {
          fill(255);
          text ("pause", width/2-17, 54);
        } // if
      } // try 
      catch (Exception e) {
        // e.printStackTrace();
        // do nothing
      }
      //
    } // func 
    
    String strFromMillis ( int m ) {
      // returns a string that represents a given amount of millis "m" 
      // as hours:minutes:seconds 
      // (can't do days etc.)
      float sec;
      int min;
      //
      sec = m / 1000;
      min = floor(sec / 60);  
      sec = floor(sec % 60);
    
      // over one hour? 
      if (min>59) { 
        int hrs = floor(min / 60);
        min = floor (min % 60); 
        return  hrs+":"
          +nf(min, 2)+":"
          +nf(int(sec), 2);
      } else 
      {
        return min+":"+nf(int(sec), 2);
      }
    } // func 
    
    // ==================================================
    
    class Event {  
    
      // pos
      float x;
      float y;
    
      // size
      float w;
      float h;
    
      float r, g, b;
    
      boolean isDead = false; 
    
      int eventMillis;
      String name; 
    
      // constr 
      Event(float tempX, float tempY, 
      float tempW, float tempH, 
      float tempR, float tempG, float tempB, 
      int eventMillis_, 
      String name_ ) {
        x = tempX;
        y = tempY;
        w = tempW;
        h = tempH;
        //    beginPos = tempBegin;
        //    stopPos = tempStop;
        r = tempR;
        g = tempG;
        b = tempB;
    
        eventMillis = eventMillis_; 
        name        = name_;
      } // constr 
    
      void display() {
        if (!isDead) {
          fill(r, g, b);
          rect(x, y, w, h);
        }
      }
      //
    } // class
    //
    
  • just did program A

  • new version with full edit for name and crs to move events

  • //imports 
    import ddf.minim.*;
    import ddf.minim.analysis.*;
    import controlP5.*;
    
    // global scope 
    
    // Input name
    ControlP5 cp5;  
    Textfield myTextfield;
    
    // list of events  
    ArrayList<Event> events;
    // which event from this list is selected 
    int selected = -1;  // -1 means none 
    
    // song stuff
    Minim minim;
    AudioPlayer song;
    int songLength = 0;
    
    // Other vars
    // frequence display 
    FFT fft;
    // FFT on / off
    boolean showFFT = true;   // show fft yes/no
    
    // --------------------------------------
    // core funcs 
    
    void setup() {
      size(displayWidth-40, 200);
      //  smooth();
      background(255);
    
      events = new ArrayList();
    
      initializeTexts();
    
      // song 
      minim = new Minim(this);
      song = minim.loadFile( "test.mp3" ); 
      songLength=song.length();
    
      // an FFT needs to know how
      // long the audio buffers it will be analyzing are
      // and also needs to know
      // the sample rate of the audio it is analyzing
      fft = new FFT(song.bufferSize(), song.sampleRate());
    
      // play it 
      song.play();
    } // func 
    
    void draw() {
      background(255);
    
      // yellow zone 
      fill(222, 222, 2);
      noStroke();
      rect(0, height-30, width, 30); 
    
      // text 
      fill(0); 
      text("Click in the yellow zone to cue. \n"
        +"Click above it to place a grapical event.\n"
        +"Press s to save (OVERWRITING).Press f to toggle FFT. "
        +"Click on an event to select it (use crs and DEL for delete and n for edit then).\n\n"
        +"Small line shows song, other line shows mouse.", 20, 20);
    
      // line mouse 
      stroke(0, 0, 0); // black  
      line (mouseX, 150, mouseX, height);
    
      // line song playing 
      stroke(0, 0, 255); // blue 
      int songMillis = int (map (song.position(), 0, songLength, 0, width));
      line (songMillis, 170, songMillis, height);
    
      // for-loop
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        event.display();
      } // for 
    
      // a new for-loop to delete Events
      // (it is better to have 2 separate for-loops)
      for (int i = events.size ()-1; i >= 0; i--) {
        Event event = events.get(i);
        if (event.isDead) {
          events.remove(i);
        } // if
      } // for 
    
      showOtherScreenElements();
    } // func 
    
    // ----------------------------------------------
    // Inputs 
    
    void mousePressed() {
    
      // either select a event 
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        if (event.over()) {
          selected = i;
          unselectAll(); 
          event.isSelected = true;
          return;
        }
      }   
    
      // or cue the song 
      if (mouseY>height-30) {  
        // if we are in the yellow zone : 
        // choose a position to cue to based on where the user clicked.
        // the length() method returns the length of recording in milliseconds.
        int position = int( map( mouseX, 0, width, 0, song.length() ) );
        song.cue( position );
        selected=-1;
        unselectAll();
      } else 
      {
        // else : above : 
        // make a new event 
        int rectWidth = 10; 
        int rectHeight = 10; 
    
        int eventMillis = int (map (mouseX, 0, width, 0, songLength));
        String name = str(events.size()); 
    
        int xPos = mouseX; 
        int yPos = 150;
    
        color col1 = color(random(255), random(255), random(255));
    
        events.add(new Event(xPos, yPos, 
        rectWidth, rectHeight, 
        col1, 
        eventMillis, 
        name));
    
        selected=-1;
        unselectAll();
    
        println("placed " 
          + name + " at " 
          + eventMillis);
      }
    } // func 
    
    void keyPressed() {
      //
      if (key==CODED) {
        // key is coded 
        switch(keyCode) {
          // cursor keys 
        case LEFT:
          if (selected>-1) {
            Event event = events.get(selected);
            event.x--;
            if (event.x<=0)
              event.x=0;
            event.eventMillis = int (map (event.x, 0, width, 0, songLength));
          } // if 
          break; 
        case RIGHT:
          if (selected>-1) {
            Event event = events.get(selected);
            event.x++;
            if (event.x>width)
              event.x=width;
            event.eventMillis = int (map (event.x, 0, width, 0, songLength));
          } // if
          break;
        } // switch
      } // if 
      else {
        // key is not coded
        // eval normal key here  
        if (key=='s') {
          String[] target = new String[events.size()];
          // for-loop
          for (int i = 0; i < events.size (); i++) { 
            Event event = events.get(i);
            target[i] = new String(); 
            target[i] = event.name+"#"+str(event.eventMillis) ;
          } // for
          saveStrings ("events.txt", target);
          println ("saved");
        } // if
        else if (key =='f') {
          // toggle 
          showFFT=!showFFT;
        } // else if
        else if (key =='n') {
          //  
          if (selected>-1) {
            myTextfield.setFocus(true);
            myTextfield.setVisible(true);
          }
        } // else if
        else if (key==DELETE) {
          // kill it 
          if (selected>-1) {
            Event event = events.get(selected);
            selected=-1;
            event.isDead=true;
          } // if
        } // else if
      } // else
    } // func 
    
    // --------------------------------------------------------
    // minor tools 
    
    void unselectAll() {
      for (int i2 = 0; i2 < events.size (); i2++) {
        Event event2 = events.get(i2);
        event2.isSelected=false;
      }
    }
    
    void showOtherScreenElements() {
      // when we listen to a song
      // 
      // first perform a forward fft on one of song's buffers
      // I'm using the mix buffer
      // but you can use anyone you like
      if (showFFT) {
        if (!(fft==null)) {
          fft.forward(song.mix);
    
          stroke(255, 0, 0, 128);
          // draw the spectrum as a series of vertical lines
          // I multiple the value of getBand by 4
          // so that we can see the lines better
          for (int i = 0; i < fft.specSize (); i++)
          {
            line(i, height, i, height - fft.getBand(i)*4);
          } // for
        } // if
      } // if  
    
      fill(255);
      try {
        // show played 
        String text1 ="Played " 
          + strFromMillis(song.position())  
          + " of " 
            + strFromMillis ( songLength )  
            + "."; 
        fill(0); 
        text ( text1, width - (textWidth(text1))-40, 30 );
        // show pause 
        if (!song.isPlaying()) {
          fill(255);
          text ("pause", width/2-17, 54);
        } // if
      } // try 
      catch (Exception e) {
        // e.printStackTrace();
        // do nothing
      }
      //
    } // func 
    
    String strFromMillis ( int m ) {
      // returns a string that represents a given amount of millis "m" 
      // as hours:minutes:seconds 
      // (can't do days etc.)
      float sec;
      int min;
      //
      sec = m / 1000;
      min = floor(sec / 60);  
      sec = floor(sec % 60);
    
      // over one hour? 
      if (min>59) { 
        int hrs = floor(min / 60);
        min = floor (min % 60); 
        return  hrs+":"
          +nf(min, 2)+":"
          +nf(int(sec), 2);
      } else 
      {
        return min+":"
          +nf(int(sec), 2);
      }
    } // func 
    
    void initializeTexts() {
    
      //  String textValue = "";
      // Textarea myTextarea;
    
      PFont font = createFont("arial", 20);  
    
      cp5 = new ControlP5(this);
      cp5.getTooltip().setDelay(500);
    
      myTextfield = cp5.addTextfield("input")
        .setPosition( width-(width-10), height-60 )
          .setSize(width-20, 40)
            .setFont(font)
              .setFocus(false)
                .setColor(color(225, 225, 225))
                  .setVisible(false);
    }
    //
    public void input(String theText) {
      // automatically receives results from controller input
      println("a textfield event for controller 'input' : "+theText);
      String textValue = theText;
      Event event = events.get(selected);
      event.name = textValue ;
      myTextfield.setFocus(false);
      myTextfield.setVisible(false);
    }
    
    
    
    // ==================================================
    
    class Event {  
    
      // pos
      float x;
      float y;
    
      // size
      float w;
      float h;
    
      color eventColor;
    
      boolean isDead = false; 
    
      int eventMillis;
      String name; 
    
      boolean isSelected = false; 
    
      // constr 
      Event(float tempX, float tempY, 
      float tempW, float tempH, 
      color eventColor_, 
      int eventMillis_, 
      String name_ ) {
        x = tempX;
        y = tempY;
        w = tempW;
        h = tempH;
    
        eventColor  = eventColor_;
        eventMillis = eventMillis_; 
        name        = name_;
      } // constr 
    
      boolean over() {
        return mouseX>x&&mouseY>y&&
          mouseX<x+w&&mouseY<y+h;
      }
    
      void display() {
        // quit? 
        if (isDead) 
          return; 
        // When seleted     
        if (isSelected) {
          fill(0);
          text (name, x, y-h);
          text (eventMillis, x, y-h-h-2);
          stroke(0);
          fill(eventColor);
          rect(x, y, w, h);
        } // if 
        else {
          if (over())
            stroke(0);
          else
            noStroke();
          fill(eventColor);
          rect(x, y, w, h);
        } // else
      } // method
      //
    } // class
    //
    
  • new version

    same features, you can edit existing event markers etc.

    //imports 
    import ddf.minim.*;
    import ddf.minim.analysis.*;
    import controlP5.*;
    
    // global scope 
    
    String helpText = "Click in the yellow zone to cue. \n"
    +"Click above it to place a grapical event.\n"
    +"Press s to save (OVERWRITING). Press f to toggle FFT.\n"
    +"Click on an event to select it: Then use cursor l/r and DEL for delete and n for edit.\n\n"
    +"The small line shows song, the other line shows mouse."; 
    
    final int stateNormal    = 0;
    final int stateInputtext = 1;
    final int stateBeforeStateInputtext = 3; 
    int state = stateNormal; 
    
    // Input name
    ControlP5 cp5;  
    Textfield myTextfield;
    
    // list of events  
    ArrayList<Event> events = new ArrayList();
    // which event from this list is selected 
    int selected = -1;  // -1 means none 
    
    // song stuff
    Minim minim;
    AudioPlayer song;
    int songLength = 0;
    
    // Other vars
    // frequence display 
    FFT fft;
    // FFT on / off
    boolean showFFT = true;   // show fft yes/no
    
    // --------------------------------------
    // core funcs 
    
    void setup() {
      size(displayWidth-40, 200);
      background(255);
    
      initializeTextfield();
    
      // song 
      minim = new Minim(this);
      song = minim.loadFile( "test.mp3" ); 
      songLength=song.length();
    
      // an FFT needs to know how
      // long the audio buffers it will be analyzing are
      // and also needs to know
      // the sample rate of the audio it is analyzing
      fft = new FFT(song.bufferSize(), song.sampleRate());
    
      // play it 
      song.play();
    } // func 
    
    void draw() {
      switch (state) {
      case stateNormal:
        drawForStateNormal(); 
        break; 
      case stateInputtext:
        // drawForStateInputtext(); 
        // same as above: 
        drawForStateNormal(); 
        break;
      case stateBeforeStateInputtext:
        // init the next state 
        drawForStateBeforeStateInputtext(); 
        break;
      } // switch
    } // func 
    
    // --------------------------------------
    // state funcs 
    
    void drawForStateNormal() {
    
      background(255); // white 
    
      // yellow zone 
      fill(222, 222, 2);
      noStroke();
      rect(0, height-30, width, 30); 
    
      // text 
      fill(0);  // black    
      text(helpText, 20, 20);
    
      // line mouse
      if (state==stateNormal) { 
        stroke(0, 0, 0); // black  
        line (mouseX, 150, mouseX, height);
      }
    
      // line song playing 
      stroke(0, 0, 255); // blue 
      int songMillis = int (map (song.position(), 0, songLength, 0, width));
      line (songMillis, 170, songMillis, height);
    
      // for-loop
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        event.display();
      } // for 
    
      // a new for-loop to delete Events
      // (it is better to have 2 separate for-loops)
      for (int i = events.size ()-1; i >= 0; i--) {
        Event event = events.get(i);
        if (event.isDead) {
          events.remove(i);
        } // if
      } // for 
    
      showOtherScreenElements();
    } // func 
    
    //void drawForStateInputtext() {
    //
    //} // func 
    
    void drawForStateBeforeStateInputtext() {
      // init the next state 
      myTextfield.setFocus(true);
      myTextfield.setVisible(true);
      state = stateInputtext;
    }
    
    // ----------------------------------------------
    // Inputs 
    
    void mousePressed() {
    
      // only state normal is allowed in this func 
      if (state!=stateNormal)
        return; 
    
      // either select an existing event 
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        if (event.over()) {
          selected = i;
          unselectAll(); 
          event.isSelected = true;
          return;
        }
      }  // for 
    
      // or cue the song 
      if (mouseY>height-30) {  
        // if we are in the yellow zone : 
        // choose a position to cue to based on where the user clicked.
        // the length() method returns the length of recording in milliseconds.
        int position = int( map( mouseX, 0, width, 0, song.length() ) );
        song.cue( position );
        selected=-1;
        unselectAll();
      } else 
      {
        // else : above : 
        // or make a new event 
        int rectWidth = 10; 
        int rectHeight = 10; 
    
        int eventMillis = int (map (mouseX, 0, width, 0, songLength));
        String name = str(events.size()); 
    
        int xPos = mouseX; 
        int yPos = 150;
    
        color col1 = color(random(255), random(255), random(255));
    
        events.add(new Event(xPos, yPos, 
        rectWidth, rectHeight, 
        col1, 
        eventMillis, 
        name));
    
        selected=-1;
        unselectAll();
    
        println("placed " 
          + name + " at " 
          + eventMillis);
      } // else
    } // func 
    
    void keyPressed() {
      if (state==stateNormal) {
        keyPressedForStateNormal();
      } else if (state==stateInputtext) {
        if (key==ESC) {
          key=0;
          // close the text field 
          myTextfield.setFocus(false);
          myTextfield.setVisible(false);
          state=stateNormal;
        }//if
      }// state
    }// func 
    
    void keyPressedForStateNormal() {  
      //
      if (key==CODED) {
        // key is coded 
        switch(keyCode) {
          // cursor keys 
        case LEFT:
          if (selected>-1) {
            Event event = events.get(selected);
            event.eventMillis--;
            if (event.eventMillis<=0)
              event.eventMillis=0;
            // event.eventMillis = int (map (event.x, 0, width, 0, songLength));
            event.x = map (event.eventMillis, 0, songLength, 0, width );
          } // if 
          break; 
        case RIGHT:
          if (selected>-1) {
            Event event = events.get(selected);
            event.eventMillis++;
            if (event.eventMillis>songLength)
              event.eventMillis=songLength;
            //event.eventMillis = int (map (event.x, 0, width, 0, songLength));
            event.x = map (event.eventMillis, 0, songLength, 0, width );
          } // if
          break;
        } // switch
      } // if 
      // -------------------------------------------------------------
      else {
        // key is not coded
        // eval normal key here  
        if (key=='s') {
          String[] target = new String[events.size()];
          // for-loop
          for (int i = 0; i < events.size (); i++) { 
            Event event = events.get(i);
            target[i] = new String(); 
            target[i] = event.name+"#"+str(event.eventMillis) ;
          } // for
          saveStrings ("events.txt", target);
          println ("saved");
        } // if
        else if (key =='f') {
          // toggle 
          showFFT=!showFFT;
        } // else if
        else if (key =='n') {
          //  get name 
          if (selected>-1) {
            key=0;
            // assign the old event name to the text field
            Event event = events.get(selected);
            myTextfield.setValue(event.name); 
            state = stateBeforeStateInputtext;
          }
        } // else if
        else if (key==DELETE) {
          // kill it 
          if (selected>-1) {
            Event event = events.get(selected);
            selected=-1;
            event.isDead=true;
          } // if
        } // else if
        else if (key==ESC) {
          // kill esc anyway 
          key=0;
        } else if (key=='X') {
          exit();
        }
      } // else
    } // func 
    
    // --------------------------------------------------------
    // minor tools 
    
    void unselectAll() {
      for (int i2 = 0; i2 < events.size (); i2++) {
        Event event2 = events.get(i2);
        event2.isSelected=false;
      }
    } // func 
    
    void showOtherScreenElements() {
      // when we listen to a song
      // 
      // first perform a forward fft on one of song's buffers
      // I'm using the mix buffer
      // but you can use anyone you like
      if (showFFT) {
        if (!(fft==null)) {
          fft.forward(song.mix);
    
          stroke(255, 0, 0, 128);
          // draw the spectrum as a series of vertical lines
          // I multiple the value of getBand by 4
          // so that we can see the lines better
          for (int i = 0; i < fft.specSize (); i++)
          {
            line(i, height, i, height - fft.getBand(i)*4);
          } // for
        } // if
      } // if  
    
      fill(255);
      try {
        // show played time 
        String text1 ="Played " 
          + strFromMillis(song.position())  
          + " of " 
            + strFromMillis ( songLength )  
            + "."; 
        fill(0); 
        text ( text1, width - (textWidth(text1))-40, 30 );
        // show pause 
        if (!song.isPlaying()) {
          fill(255);
          text ("pause", width/2-17, 54);
        } // if
      } // try 
      catch (Exception e) {
        // e.printStackTrace();
        // do nothing
      }
      //
    } // func 
    
    String strFromMillis(int m) {
      // returns a string that represents a given amount of millis "m" 
      // as hours:minutes:seconds 
      // (can't do days etc.).
      float sec;
      int min;
      //
      sec = m / 1000;
      min = floor(sec / 60);  
      sec = floor(sec % 60);
    
      // over one hour? 
      if (min>59) { 
        int hrs = floor(min / 60);
        min = floor (min % 60); 
        return  hrs+":"
          +nf(min, 2)+":"
          +nf(int(sec), 2);
      } else 
      {
        return min+":"
          +nf(int(sec), 2);
      }
    } // func
    
    // --------------------------------------------------
    // tools for the textfield 
    
    void initializeTextfield() {
    
      PFont font = createFont("arial", 20);  
    
      cp5 = new ControlP5(this);
      cp5.getTooltip().setDelay(500);
    
      myTextfield = cp5.addTextfield("input")
        .setPosition( 40, 60 )
          .setSize(320, 40)
            .setFont(font)
              .setFocus(false)
                .setColor(color(225, 225, 225))
                  .setVisible(false);
    } // func
    
    public void input(String theText) {
      // automatically receives results from controller input
      // after hitting return 
      println("a textfield event for controller 'input' : "+theText);
      String textValue = theText;
      if (textValue!=null && !textValue.equals("")) {
        Event event = events.get(selected);
        event.name = textValue;
      }
      // close the text field
      myTextfield.setFocus(false);
      myTextfield.setVisible(false);
      state=stateNormal;
    }
    
    // ==================================================
    
    class Event {  
    
      // pos
      float x;
      float y;
    
      // size
      float w;
      float h;
    
      color eventColor;
    
      boolean isDead = false; 
    
      int eventMillis;
      String name; 
    
      boolean isSelected = false; 
    
      // constr 
      Event(float tempX, float tempY, 
      float tempW, float tempH, 
      color eventColor_, 
      int eventMillis_, 
      String name_ ) {
        x = tempX;
        y = tempY;
        w = tempW;
        h = tempH;
    
        eventColor  = eventColor_;
        eventMillis = eventMillis_; 
        name        = name_;
      } // constr 
    
      boolean over() {
        return mouseX>x&&mouseY>y&&
          mouseX<x+w&&mouseY<y+h;
      }
    
      void display() {
        // quit? 
        if (isDead) 
          return; 
        // When seleted     
        if (isSelected) {
          fill(0);
          text (name, x, y-h);
          text (eventMillis, x, y-h-h-2);
          stroke(0);
          fill(eventColor);
          rect(x, y, w, h);
        } // if 
        else {
          if (over())
            stroke(0);
          else
            noStroke();
          fill(eventColor);
          rect(x, y, w, h);
        } // else
      } // method
      //
    } // class
    //
    
  • Answer ✓

    this was all program A now I show program B

    (a lot of this is just show, e.g. the yellow bar etc. )

    //imports 
    import ddf.minim.*;
    import ddf.minim.analysis.*;
    
    boolean fileHasBeenSelected = false; // marker 
    String waitText = "Select a event list file to process.";
    
    String [][] grid;
    
    String eventText = ""; 
    
    // song stuff
    Minim minim;
    AudioPlayer song;
    int songLength = 0;
    
    // Other vars
    // frequence display 
    FFT fft;
    // FFT on / off
    boolean showFFT = true;   // show fft yes/no
    
    
    // --------------------------------------
    // core funcs 
    
    void setup() {
      size(displayWidth-40, 200);
      background(111); 
    
      // song 
      minim = new Minim(this);
      song = minim.loadFile( "test.mp3" ); 
      songLength=song.length();
    
      // an FFT needs to know how
      // long the audio buffers it will be analyzing are
      // and also needs to know
      // the sample rate of the audio it is analyzing
      fft = new FFT(song.bufferSize(), song.sampleRate());
    
      println (sketchPath(""));
      String sp1 = sketchPath("");
    
      File f = new File(sp1+"*.txt");
    
      selectInput("Select a file to process: ", "fileSelected", f);
    } // func
    
    void draw() {
      background(111); 
      // evaluate the marker 
      if (!fileHasBeenSelected) {
        // wait
        fill(0); 
        text (waitText, 30, 30);
      } else {
        // main part
        drawWhenEventFileHasBeenLoaded();
      } // else
    } // func 
    
    // ----------------------------------------------------
    
    void drawWhenEventFileHasBeenLoaded() {
    
      // main part
    
      // yellow zone 
      fill(222, 222, 2);
      noStroke();
      rect(0, height-30, width, 30); 
    
      // line song playing 
      stroke(0, 0, 255); // blue 
      int songMillis = int (map (song.position(), 0, songLength, 0, width));
      line (songMillis, 170, songMillis, height);
    
      fill(222, 222, 2);
      text (eventText, 100, 70);
    
      for  (int i = 0; i < grid.length; i++) {
        // display the rects 
        float eventX = map (int(grid[i][1]), 0, songLength, 0, width );
        fill (2, 3, 222);
        rect (eventX, 100, 10, 10);
        // check if the position matches any events  
        if (!grid[i][2].equals("X") && song.position() >= int(grid[i][1])) {
          // yes 
          grid[i][2]="X"; // mark as done 
          evaluateCurrentEvent(grid[i][0]);
        } // if
      } // for
    } // func 
    
    void evaluateCurrentEvent(String eventName) {
    
      if (eventName.equals("1")) {
        eventText="ONE";
      } else if (eventName.equals("2")) {
        eventText="TWO";
      } // if
      // ..............
      else {
        eventText="Unkown event occured. It had the name "
          + eventName + ".";
      }
    }
    
    void fileSelected(File selection) {
      if (selection == null) {
        println("Window was closed or the user hit cancel.");
        waitText="Window was closed or the user hit cancel.";
    
        //    File f = new File("*.txt");
        //    selectInput("Select a file to process:    ", "fileSelected", f);
      } else {
        println("User selected " + selection.getAbsolutePath());
        String theNameThatHasBeenEntered  = selection.getAbsolutePath(); 
        // img = loadImage( theNameThatHasBeenEntered );
    
        String[] events;
    
        events = loadStrings (theNameThatHasBeenEntered);
        for (String s1 : events)  
          printArray(s1); 
    
        //create grid array based on # of rows and columns in file
        grid = new String [events.length][3];
        //
        int count=0;
        //parse values into 2d array
        for (int i=0; i < events.length; i++) {
          String [] temp;
          temp = split(events[i], '#');
          for (int j=0; j < temp.length; j++) {
            grid[i][j]=temp[j];
            grid[i][2]="";
            count++;
          }
        } // for
    
        println ("Strings in total: " + count );
    
        // set marker 
        fileHasBeenSelected = true;
        // play it 
        song.play();
      }
    }
    
    //
    
  • file F1 looks like this btw.

    0#20558
    1#12124
    2#27938
    3#35317
    
  • Hi colouredmirrorball,

    Thanks for your response. I’m pretty to new to Processing. I’ve made my way through several tutorials and parts of books. I’ve played around with Minim a bit. But, I’m just now starting to work on the graphics calls. I wanted to first make sure, that there wasn’t some code/framework/software package already existing that does what I want. Based on responses so far, it doesn’t seem like it (this includes a post to vjforums.info).

    Yes, as chrisir has already confirmed, I assumed that there would be a separate program where events are placed/sequenced. I wasn’t sure if Processing would be good for that, but chrisr seems to have answered that.

    Hi chrisir,

    Wow! Thanks so much for this! It’s pretty much what I was envisioning. How long did it take you? Are you wanting/needing a similar tool? Or into it for the exercise?

    I need to add a bunch of stuff, so I’ll take your code and run with it. Here’s a few things off the top of my head, in case you want to chime in about any of them (btw, I'm not expecting you to code any of this):

    1. Since I need to cram so many calls into the song timeline, I’m gonna reduce the little squares to probably single or double width lines. Also, I’ll want add parallel rows (to your initial row of squares) where I can add calls for, say, specific instruments or sounds. This would allow me to separate instruments/sound effects into their own lines—which would add some organisation and ease of navigation to the calls.

    2. I also need to have some ability to run the music at various speeds (ie., slow it way down) and easily start/stop and step through it, to make precise call placement easier.

    3. A display of current time in the song (in ms) will help with this too.

    4. I'd like the ability to read-in specially marked functions (eg. specifically formatted comments could be used for this) from another sketch, then upon placement of calls on the timeline, I am prompted to choose which function I want. This would then be written to the text file.

    I've already played around with your sketches and looked at the text file. I think I might whip-up a simple sketch that reads the file and does some graphics….just as a proof-of-concept, before I get too far into developing.

    Thanks again for this! It's really nice start to have. Much appreciated!

    Ryan

  • here is a new version of sketch A which produces a short passage of source code to paste into sketch B

    the source code just comes via println

    // this writes events in a file 
    
    // this file can then be read by 
    //          Ball_ArraylistMouseReadEvents1
    
    //imports 
    import ddf.minim.*;
    import ddf.minim.analysis.*;
    
    import controlP5.*;
    
    // global scope 
    
    String helpText = "Click in the yellow zone to cue. \n"
    +"Click above it to place a grapical event.\n"
    +"Press s to save (OVERWRITING). Press f to toggle FFT.\n"
    +"Click on an event to select it: Then use cursor l/r and DEL for delete and n for edit.\n\n"
    +"The small line shows song, the other line shows mouse."; 
    
    final int stateNormal    = 0;
    final int stateInputtext = 1;
    final int stateBeforeStateInputtext = 3; 
    int state = stateNormal; 
    
    // Input name
    ControlP5 cp5;  
    Textfield myTextfield;
    
    // list of events  
    ArrayList<Event> events = new ArrayList();
    // which event from this list is selected 
    int selected = -1;  // -1 means none 
    
    // song stuff
    Minim minim;
    AudioPlayer song;
    int songLength = 0;
    
    // Other vars
    // frequence display 
    FFT fft;
    // FFT on / off
    boolean showFFT = true;   // show fft yes/no
    
    // --------------------------------------
    // core funcs 
    
    void setup() {
      size(displayWidth-40, 200);
      background(255);
    
      initializeTextfield();
    
      // song 
      minim = new Minim(this);
      song = minim.loadFile( "test.mp3" ); 
      songLength=song.length();
    
      // an FFT needs to know how
      // long the audio buffers it will be analyzing are
      // and also needs to know
      // the sample rate of the audio it is analyzing
      fft = new FFT(song.bufferSize(), song.sampleRate());
    
      // play it 
      song.play();
    } // func 
    
    void draw() {
      switch (state) {
      case stateNormal:
        drawForStateNormal(); 
        break; 
      case stateInputtext:
        // drawForStateInputtext(); 
        // same as above: 
        drawForStateNormal(); 
        break;
      case stateBeforeStateInputtext:
        // init the next state 
        drawForStateBeforeStateInputtext(); 
        break;
      } // switch
    } // func 
    
    // --------------------------------------
    // state funcs 
    
    void drawForStateNormal() {
    
      background(255); // white 
    
      // yellow zone 
      fill(222, 222, 2);
      noStroke();
      rect(0, height-30, width, 30); 
    
      // text 
      fill(0);  // black    
      text(helpText, 20, 20);
    
      // line mouse
      if (state==stateNormal) { 
        stroke(0, 0, 0); // black  
        line (mouseX, 150, mouseX, height);
      }
    
      // line song playing 
      stroke(0, 0, 255); // blue 
      int songMillis = int (map (song.position(), 0, songLength, 0, width));
      line (songMillis, 170, songMillis, height);
    
      // for-loop
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        event.display();
      } // for 
    
      // a new for-loop to delete Events
      // (it is better to have 2 separate for-loops)
      for (int i = events.size ()-1; i >= 0; i--) {
        Event event = events.get(i);
        if (event.isDead) {
          events.remove(i);
        } // if
      } // for 
    
      showOtherScreenElements();
    } // func 
    
    //void drawForStateInputtext() {
    //
    //} // func 
    
    void drawForStateBeforeStateInputtext() {
      // init the next state 
      myTextfield.setFocus(true);
      myTextfield.setVisible(true);
      state = stateInputtext;
    }
    
    // ----------------------------------------------
    // Inputs 
    
    void mousePressed() {
    
      // only state normal is allowed in this func 
      if (state!=stateNormal)
        return; 
    
      // either select an existing event 
      for (int i = 0; i < events.size (); i++) { 
        Event event = events.get(i);
        if (event.over()) {
          selected = i;
          unselectAll(); 
          event.isSelected = true;
          return;
        }
      }  // for 
    
      // or cue the song 
      if (mouseY>height-30) {  
        // if we are in the yellow zone : 
        // choose a position to cue to based on where the user clicked.
        // the length() method returns the length of recording in milliseconds.
        int position = int( map( mouseX, 0, width, 0, song.length() ) );
        song.cue( position );
        selected=-1;
        unselectAll();
      } else 
      {
        // else : above : 
        // or make a new event 
        int rectWidth = 10; 
        int rectHeight = 10; 
    
        int eventMillis = int (map (mouseX, 0, width, 0, songLength));
        String name = str(events.size()); 
    
        int xPos = mouseX; 
        int yPos = 150;
    
        color col1 = color(random(255), random(255), random(255));
    
        events.add(new Event(xPos, yPos, 
        rectWidth, rectHeight, 
        col1, 
        eventMillis, 
        name));
    
        selected=-1;
        unselectAll();
    
        println("placed " 
          + name + " at " 
          + eventMillis);
      } // else
    } // func 
    
    void keyPressed() {
      if (state==stateNormal) {
        keyPressedForStateNormal();
      } else if (state==stateInputtext) {
        if (key==ESC) {
          key=0;
          // close the text field 
          myTextfield.setFocus(false);
          myTextfield.setVisible(false);
          state=stateNormal;
        }//if
      }// state
    }// func 
    
    void keyPressedForStateNormal() {  
      //
      if (key==CODED) {
        // key is coded 
        switch(keyCode) {
          // cursor keys 
        case LEFT:
          if (selected>-1) {
            Event event = events.get(selected);
            event.eventMillis--;
            if (event.eventMillis<=0)
              event.eventMillis=0;
            // event.eventMillis = int (map (event.x, 0, width, 0, songLength));
            event.x = map (event.eventMillis, 0, songLength, 0, width );
          } // if 
          break; 
        case RIGHT:
          if (selected>-1) {
            Event event = events.get(selected);
            event.eventMillis++;
            if (event.eventMillis>songLength)
              event.eventMillis=songLength;
            //event.eventMillis = int (map (event.x, 0, width, 0, songLength));
            event.x = map (event.eventMillis, 0, songLength, 0, width );
          } // if
          break;
        } // switch
      } // if 
      // -------------------------------------------------------------
      else {
        // key is not coded
        // eval normal key here  
        if (key=='s') {
          // save ---
          String[] target = new String[events.size()];
          println(); 
          println("source code - please remove the first \"} else\".");
          println("please paste source code into the program that reads the event file in function evaluateCurrentEvent");
          println();  
          // 
          boolean firstTime = true; 
          // for-loop over all events 
          for (int i = 0; i < events.size (); i++) {
            // preparing output file  
            Event event = events.get(i);
            target[i] = new String(); 
            target[i] = event.name+"#"+str(event.eventMillis);
            // preparing source code 
            if (firstTime) {
              println ( "if (eventName.equals(\"" + event.name +"\")) {" ) ;
              firstTime=false;
            } else {
              println ( "} else if (eventName.equals(\"" + event.name +"\")) {" ) ;
            }
            println ( "      // .... ");  
            println ( "} // if");
          } // for
          println(); 
          println(); 
          // writing the file 
          saveStrings ("events.txt", target);
          println ("saved");
        } // if
        else if (key =='f') {
          // toggle 
          showFFT=!showFFT;
        } // else if
        else if (key =='n') {
          //  get name 
          if (selected>-1) {
            key=0;
            // assign the old event name to the text field
            Event event = events.get(selected);
            myTextfield.setValue(event.name); 
            state = stateBeforeStateInputtext;
          }
        } // else if
        else if (key==DELETE) {
          // kill it 
          if (selected>-1) {
            Event event = events.get(selected);
            selected=-1;
            event.isDead=true;
          } // if
        } // else if
        else if (key==ESC) {
          // kill esc anyway 
          key=0;
        } else if (key=='X') {
          exit();
        } // else
      } // else
    } // func 
    
    // --------------------------------------------------------
    // minor tools 
    
    void unselectAll() {
      for (int i2 = 0; i2 < events.size (); i2++) {
        Event event2 = events.get(i2);
        event2.isSelected=false;
      }
    } // func 
    
    void showOtherScreenElements() {
      // when we listen to a song
      // 
      // first perform a forward fft on one of song's buffers
      // I'm using the mix buffer
      // but you can use anyone you like
      if (showFFT) {
        if (!(fft==null)) {
          fft.forward(song.mix);
    
          stroke(255, 0, 0, 128);
          // draw the spectrum as a series of vertical lines
          // I multiple the value of getBand by 4
          // so that we can see the lines better
          for (int i = 0; i < fft.specSize (); i++)
          {
            line(i, height, i, height - fft.getBand(i)*4);
          } // for
        } // if
      } // if  
    
      fill(255);
      try {
        // show played time 
        String text1 ="Played " 
          + strFromMillis(song.position())  
          + " of " 
            + strFromMillis ( songLength )  
            + "."; 
        fill(0); 
        text ( text1, width - (textWidth(text1))-40, 30 );
        // show pause 
        if (!song.isPlaying()) {
          fill(255);
          text ("pause", width/2-17, 54);
        } // if
      } // try 
      catch (Exception e) {
        // e.printStackTrace();
        // do nothing
      }
      //
    } // func 
    
    String strFromMillis(int m) {
      // returns a string that represents a given amount of millis "m" 
      // as hours:minutes:seconds 
      // (can't do days etc.).
      float sec;
      int min;
      //
      sec = m / 1000;
      min = floor(sec / 60);  
      sec = floor(sec % 60);
    
      // over one hour? 
      if (min>59) { 
        int hrs = floor(min / 60);
        min = floor (min % 60); 
        return  hrs+":"
          +nf(min, 2)+":"
          +nf(int(sec), 2);
      } else 
      {
        return min+":"
          +nf(int(sec), 2);
      }
    } // func
    
    // --------------------------------------------------
    // tools for the textfield 
    
    void initializeTextfield() {
    
      PFont font = createFont("arial", 20);  
    
      cp5 = new ControlP5(this);
      cp5.getTooltip().setDelay(500);
    
      myTextfield = cp5.addTextfield("input")
        .setPosition( 40, 60 )
          .setSize(320, 40)
            .setFont(font)
              .setFocus(false)
                .setColor(color(225, 225, 225))
                  .setVisible(false);
    } // func
    
    public void input(String theText) {
      // automatically receives results from controller input
      // after hitting return 
      println("a textfield event for controller 'input' : "+theText);
      String textValue = theText;
      if (textValue!=null && !textValue.equals("")) {
        Event event = events.get(selected);
        event.name = textValue;
      }
      // close the text field
      myTextfield.setFocus(false);
      myTextfield.setVisible(false);
      state=stateNormal;
    }
    
    // ==================================================
    
    class Event {  
    
      // pos
      float x;
      float y;
    
      // size
      float w;
      float h;
    
      color eventColor;
    
      boolean isDead = false; 
    
      int eventMillis;
      String name; 
    
      boolean isSelected = false; 
    
      // constr 
      Event(float tempX, float tempY, 
      float tempW, float tempH, 
      color eventColor_, 
      int eventMillis_, 
      String name_ ) {
        x = tempX;
        y = tempY;
        w = tempW;
        h = tempH;
    
        eventColor  = eventColor_;
        eventMillis = eventMillis_; 
        name        = name_;
      } // constr 
    
      boolean over() {
        return mouseX>x&&mouseY>y&&
          mouseX<x+w&&mouseY<y+h;
      }
    
      void display() {
        // quit? 
        if (isDead) 
          return; 
        // When seleted     
        if (isSelected) {
          fill(0);
          text (name, x, y-h);
          text (eventMillis, x, y-h-h-2);
          stroke(0);
          fill(eventColor);
          rect(x, y, w, h);
        } // if 
        else {
          if (over())
            stroke(0);
          else
            noStroke();
          fill(eventColor);
          rect(x, y, w, h);
        } // else
      } // method
      //
    } // class
    //
    
  • Running the new version now--until my battery dies.

    I see you already put in some fine tuning with the l/r keys. Nice.

  • yes, can change the millis now with cursors

    also, with n, you can give each event another name

Sign In or Register to comment.