I'm thinking about syncing Processing's frameRate/draw function with an incoming MIDI signal because I would love to create a visualization in sync with music coming from Ableton.

With the MidiBus Library and through the IAC Driver on my Mac I got the Sync/Clock MIDI Message from Ableton into Processing but then I wasn't able to link those two things together.

This is my code so far but I'm not sure it's working properly. Any help would be greatly appreciated.

import themidibus.*; //Import the library
import javax.sound.midi.MidiMessage; //Import the MidiMessage classes http://java.sun.com/j2se/1.5.0/docs/api/javax/sound/midi/MidiMessage.html
import javax.sound.midi.SysexMessage;
import javax.sound.midi.ShortMessage;

MidiBus myBus; // The MidiBus

int i = 1;

void setup() {
  size(400, 400);
  background(0);
  frameRate(60);
  MidiBus.list();
  myBus = new MidiBus(this, 0, 1);
}

void draw() {
 if(frameCount % 60 == 0) {
   println("min");
   i = 1;
 }
}

void midiMessage(MidiMessage message) {
  print(i + " ");
  i++;
}

I am using Kinect with Windows library (https://github.com/ThomasLengeling/KinectPV2) in processing and detecting all the skeleton joints and positions. What I want to do is to combine sounds and music according to movements of the skeleton joints. (e.g. control pitch or volume of a sound according to the x-position of the left hand). Something like this (). As I don't have any experience with music software I would like to ask you which method you recommend in order to implement this. A possible approached that I've found to combine with processing is by using Max7 (). But I am not sure if Max is the most suitable approach as I am aware that there are many other software (Ableton, virtual midi controllers). In respect to Max7, I am thinking that I could do that by controlling a slider using the skeleton joint position(e.g as the hand moves right the slider value will increase but I don't know how I could do that - assign the position values from processing to a Max's slider). Any guidelines would be appreciated!

Michael

So my code is now advancing but it's still laggy. It receives the data in Ableton but it's very temperamental. I've avoided using delay functions as it delays the entire thing. If anyone has any suggestions or can see any anomalies it would be much appreciated! Thank you :)

import gab.opencv.*;
import processing.video.*;
import ipcapture.*;
import org.opencv.core.Rect;
import themidibus.*; //Import the library


OpenCV opencv;
IPCapture cam;
MidiBus midi; 
int lastMillis;
int [] timestamps = new int [128];

void setup() {
  size(640, 480);
  //video = new Movie(this, "street.mov"); MUTED
  cam = new IPCapture(this, "http://153.201.66.43:60001/mjpg/video.mjpg?resolution=640x480", "", "");
  cam.start();
  opencv = new OpenCV(this, 640, 480);

  opencv.startBackgroundSubtraction(5, 3, 0.5);
  MidiBus.list();
  midi = new MidiBus(this, -1, "Bus 1");

  //video.loop();
  //video.play();
}

void draw() {
  if (cam.isAvailable()) {
    cam.read();
    image(cam, 0, 0);  
    opencv.loadImage(cam);
    opencv.updateBackground();

    opencv.dilate();
    opencv.erode();

    noFill();
    stroke(255);
    strokeWeight(1);

    int rx = 300; 
    int ry = 150;
    int rw = 30;
    int rh = 15;
    rect(rx, ry, rx+rw, ry+rh);

    int rx2 = 10; 
    int ry2 = 10;
    int rw2 = 15;
    int rh2 = 15;
    rect(rx2, ry2, rx2+rw2, ry2+rh2);


    stroke(255, 0, 0);
    strokeWeight(10);
    for (Contour contour : opencv.findContours()) {


      println(contour.getBoundingBox().x);
      println(contour.getBoundingBox().y);
      int x = contour.getBoundingBox().x;
      int y = contour.getBoundingBox().y;

      contour.draw(); 
      {

        int [] numbers = new int [128];

        if (( x > rx && x < rx + rw) && (y > ry && y < ry + rh))

          for (int i = -1; i < timestamps.length; i++)


            midi.sendNoteOn(0, 62, 127);

        timestamps[62] = millis();


        if (millis()>lastMillis+40)
          midi.sendNoteOff(0, 62, 127);

        println(timestamps);
      }
    }
  }
}

Finally cracked my new project of triggering midi in Ableton via webcam movement but the midi signal is very glitchy and intermittent, despite my console telling me is it sending out signals. I feel it might be an issue with buffer size as when I'm sending a small amount of data, it works okay. Anything bigger and it glitches.

If you need a screen recording of what is happening, let me know. Otherwise I've posted the code underneath. Thanks!

import gab.opencv.*;
import processing.video.*;
import ipcapture.*;
import org.opencv.core.Rect;
import themidibus.*; //Import the library

OpenCV opencv;
IPCapture cam;
MidiBus midi; 


void setup() {
  size(640, 480);
  //video = new Movie(this, "street.mov"); MUTED
  cam = new IPCapture(this, "http://" + "100.38.83.153:8081/mjpg/video.mjpg?resolution=640x480", "", "");
  cam.start();
  opencv = new OpenCV(this, 640, 480);

  opencv.startBackgroundSubtraction(5, 3, 0.5);
  MidiBus.list();
  midi = new MidiBus(this, -1, "IAC Bus 1");

  //video.loop();
  //video.play();
}

void draw() {
  if (cam.isAvailable()) {
    cam.read();
    image(cam, 0, 0);  
    opencv.loadImage(cam);
    opencv.updateBackground();

    opencv.dilate();
    opencv.erode();

    noFill();
    stroke(255, 0, 0);
    strokeWeight(5);
    for (Contour contour : opencv.findContours()) {


      println(contour.getBoundingBox().x);
      println(contour.getBoundingBox().y);
      int x = contour.getBoundingBox().x;
      int y = contour.getBoundingBox().y;

      contour.draw();

      if ((x < 700) && (x > 650)) 
        println("x1");
      midi.sendNoteOn(0, 60, 127);
      midi.sendNoteOff(0, 60, 127);

      if ((x < 649) && (x > 600)) 
        println("x2");
      midi.sendNoteOn(0, 61, 127);
      midi.sendNoteOff(0, 61, 127);


      if ((x < 599) && (x > 550)) 
        println("x3");
      midi.sendNoteOn(0, 62, 127);
      midi.sendNoteOff(0, 62, 127);

      if ((x < 549) && (x > 500)) 
        println("x4");
      midi.sendNoteOn(0, 63, 127);
      midi.sendNoteOff(0, 63, 127);


      if ((x < 499) && (x > 450)) 
        println("x5");
      midi.sendNoteOn(0, 64, 127);
      midi.sendNoteOff(0, 64, 127);

      if ((x < 449) && (x > 400)) 
        println("x6");
      midi.sendNoteOn(0, 65, 127);
      midi.sendNoteOff(0, 65, 127);

      if ((x < 399) && (x > 350)) 
        println("x7");
      midi.sendNoteOn(0, 66, 127);
      midi.sendNoteOff(0, 66, 127);


      if ((x < 349) && (x > 300)) 
        println("x8");
      midi.sendNoteOn(0, 67, 127);
      midi.sendNoteOff(0, 67, 127);

      if ((x < 299) && (x > 250)) 
        println("x9");
      midi.sendNoteOn(0, 68, 127);
      midi.sendNoteOff(0, 68, 127);

      if ((x < 249) && (x > 200)) 
        println("x10");
      midi.sendNoteOn(0, 69, 127);
      midi.sendNoteOff(0, 69, 127);


      if ((x < 199) && (x > 150)) 
        println("x11");
      midi.sendNoteOn(0, 69, 127);
      midi.sendNoteOff(0, 69, 127);

      if ((x < 149) && (x > 100)) 
        println("x12");
      midi.sendNoteOn(0, 69, 127);
      midi.sendNoteOff(0, 69, 127);

      if ((x < 99) && (x > 50)) 
        println("X13");
      midi.sendNoteOn(0, 69, 127);
      midi.sendNoteOff(0, 69, 127);

      if ((x < 49) && (x > 1)) 
        println("x14");
      midi.sendNoteOn(0, 69, 127);
      midi.sendNoteOff(0, 69, 127);


      if ((y < 700) && (y > 650)) 
        println("y1");
      midi.sendNoteOn(1, 60, 127);
      midi.sendNoteOff(1, 60, 127);

      if ((y < 649) && (y > 600)) 
        println("x2");
      midi.sendNoteOn(1, 61, 127);
      midi.sendNoteOff(1, 61, 127);


      if ((y < 599) && (y > 550)) 
        println("y");
      midi.sendNoteOn(1, 62, 127);
      midi.sendNoteOff(1, 62, 127);

      if ((y < 549) && (y > 500)) 
        println("x4");
      midi.sendNoteOn(1, 63, 127);
      midi.sendNoteOff(1, 63, 127);


      if ((y < 499) && (y > 450)) 
        println("y5");
      midi.sendNoteOn(1, 64, 127);
      midi.sendNoteOff(1, 64, 127);

      if ((y < 449) && (y > 400)) 
        println("y6");
      midi.sendNoteOn(1, 65, 127);
      midi.sendNoteOff(1, 65, 127);

      if ((y < 399) && (y > 350)) 
        println("y7");
      midi.sendNoteOn(1, 66, 127);
      midi.sendNoteOff(1, 66, 127);


      if ((y < 349) && (y > 300))
        println("y8");
      midi.sendNoteOn(1, 67, 127);
      midi.sendNoteOff(1, 67, 127);

      if ((y < 299) && (y > 250)) 
        println("y9");
      midi.sendNoteOn(1, 68, 127);
      midi.sendNoteOff(1, 68, 127);

      if ((y < 249) && (y > 200)) 
        println("y10");
      midi.sendNoteOn(1, 69, 127);
      midi.sendNoteOff(1, 69, 127);


      if ((y < 199) && (y > 150)) 
        println("y11");
      midi.sendNoteOn(1, 69, 127);
      midi.sendNoteOff(1, 69, 127);

      if ((y < 149) && (y > 100)) 
        println("y12");
      midi.sendNoteOn(1, 69, 127);
      midi.sendNoteOff(1, 69, 127);

      if ((y < 99) && (y > 50)) 
        println("y13");
      midi.sendNoteOn(1, 69, 127);
      midi.sendNoteOff(1, 69, 127);

      if ((y < 49) && (y > 1)) 
        println("y14");
      midi.sendNoteOn(1, 69, 127);
      midi.sendNoteOff(1, 69, 127);
    }



    // void movieEvent(Movie m) {
    //  m.read();
  }
}

EDIT. Sorry, being stupid and can't get the code quoted properly! Edit 2, Got it ;)

A quick run down - I took a module in Processing at uni years ago and thoroughly enjoyed my time with it. I haven't done any sort of programming since but really want to get back into creative coding.

Essentially, the ultimate goal is to create live abstract visualisations of audio (the audio being part pre-programmed and part live) and project it onto myself whilst 'gigging'. Alongside learning Processing, I'll be learning Ableton as I'm bored of Cubase and from what I've heard Ableton is a lot more flexible.

So I guess the main quiestion is this... Is it possible/relatively painless to route multiple tracks of audio from Ableton into Processing and then use data from those tracks to trigger visuals? I'm guessing I'd have to get some sort of virtual I/O box and send the audio to both the speakers and this virtual box, which Processing would pick up/analyse from there? I know it's possible to do this with MIDI data so I assume it should be the same for audio?

Also to note, I'm aware that as I'll be learning Ableton I could probably just use MaxMSP/Jitter and make life easier for myself. The aim though is to teach myself to code - I don't mind jumping through a few hoops to get it up and running, as long as my CPU dosen't explode under the load...

Many thanks all anyway, have a good evening!

At the moment the arduino talks to processing via serial sending messages like so <Param1=Param2=> etc etc where < and > are start and end chars.

I was hoping to use the new ableton live connection kit to get the arduino to send some POT values to ableton. I was also hoping the arduino could send ableton the other messages like <Param1=Param2=> etc so that ableton could almost act like a broker.

I.e it could take the pot values to influence some tracks and use the param1 param2 values as well but forward on any messages that aren't the pot values.. i.e. the messages contained in the greater than less than chars..

I was thinking of using the connection kit + firmata and then somehow maybe creating a custom max patch with some regex to identify any messages in the format < .. . . . > and then maybe forward them on to processing via osc?

Has anyone done anything similar and had ableton as the message broker generating arbitrary messages out via osc?

Thanks :)