Music visuals that don't hog the cpu!!!

rtavakko

Hi all, so I've made a real-time music visualizer that works very well. Except that it is very slow due to the large number of operations it does on the pixels and objects in the program. Any pointers or suggestions as to how I can speed this up?

Cheers

GoToLoop

http://forum.Processing.org/two/discussions/tagged?Tag=thread()

koogs

Without seeing the code? Not really.

Try opengl mode?

rtavakko

Hi thanks for the reply. The code is too long to post here, but I have somewhere around 50000 pvector objects moving at all times plus an FFT object. I have my main functions which calculate object movement running on separate threads, but the output is still laggy at 30 fps. Not sure if there is a good way to use multi-threading for such a high volume of calculations.

GoToLoop

• As long as you know how to identify parts of your code which can act isolated, you can easily place them in their own Thread.
• Shared objects are the real problem. In order to have more than 1 Thread acting upon them, we need to synchronized () their access.
• However, too much of it can end up defeating much or all performance gained via threading! :-SS
• W/o studying the relationship between the objects used in your app, it's almost impossible to advise for a specific threading approach. ^#(^

koogs

Again, without the code we are guessing. Screenshot? Video?

koogs

And threading is all very well but the processor has a maximum and it doesn't matter how you share the work out it won't go beyond that.

rtavakko

Alright, so here is the part off the code that is giving me the most trouble. It is supposed to be a simulation of a river and it will react to the bass frequencies of a minim audio input.

rtavakko

class Creek
{
  //Nodes: original centers of the points that represent water drops
  //Pos: current position of a droplet
  //Rad: radius of movement of each droplet around its node
  //Vel: speed of each droplet
  PVector[][] nodes;
  PVector[][] pos;
  PVector[][] rad;  
  PVector[][] vel;

  //Dimensions of the creek
  int w,l;

  //This is used to calculate sine and cosine values
  float angle;

  //
  float k;
  float p;

  float r;
  float t;

  //These are parameters for an additional wave matrix with each
  //point on the matrix affecting a square of points on the main
  //grid
  //
  //Wave_cent: center position of the waves
  //Wave_speed: two-directional speed value which determines
  //where a wave is headed
  //Wave_amp: amplitude of each wave; this is what determines the 
  //amplitude of the points on the grid affected by a wave
  PVector[][] wave_cent;
  PVector[][] wave_speed;
  float[][] wave_amp;

  //These values determine how big of an area on the main matrix
  //is affected by each wave 
  int wave_rad_x, wave_rad_z;

  //These parameters specify the dimensions of the wave matrix
  //the wave matrix contains less points than the main matrix
  int size_x, size_z;

  Creek(int wid, int len)
  {
    w = wid;
    l = len;

    angle = 0;

    nodes = new PVector[w][l];
    pos = new PVector[w][l];
    rad = new PVector[w][l];
    vel = new PVector[w][l];

    k = 5;
    p = 1;

    //t: this is the equivalent of time in x = v*t. It specifies how much a droplet         moves each frame  
    //r: radius of movement of each water droplet around its center
    t = 0.125;
    r = 7.3;

    wave_rad_x = int(w/24);
    wave_rad_z = int(wave_rad_x*2);

    size_x = int(w/4);
    size_z = int(l/2);

    wave_cent = new PVector[size_x][size_z];
    wave_speed = new PVector[size_x][size_z];
    wave_amp = new float[size_x][size_z];

    for(int x = 0; x < size_x; x+=1)
    {
      for(int z = 0; z < size_z; z+=1)
      {
        //Initially, there are no waves
        //The center position of each wave is mapped to a point
        //on the main matrix of droplets
        wave_speed[x][z] = new PVector(0,0,0);
        wave_cent[x][z] = new PVector(map(x,0,size_x,0,w),0,map(x,0,size_x,0,l));
        wave_amp[x][z] = 0;
      }
    }

    for(int x = 0; x < w; x+=1)
    {
      for(int z = 0; z < l; z+=1)
      {
        //The droplets must move at all times, so the initial radius
        //and velocity values are chosen randomly to simulate water
        //movement
        nodes[x][z] = new PVector(10*x,0,10*z);
        pos[x][z] = new PVector(nodes[x][z].x + random(-w/6,w/6), nodes[x][z].y + random(-w/6,w/6), nodes[x][z].z + random(-l/6,l/6)); 
        rad[x][z] = new PVector(r,0,r); 
        vel[x][z] = new PVector(random(-1,1),0,random(-1,1)); 
      }
    }
  }

  void display()
  {  
    for(int x = 0; x < w; x+= 1)
    {
      for(int z = 0; z < l; z+= 1)
      {
        //Some color randomization
        stroke(random(18,165),random(130,205),255);    

        point(pos[x][z].x, pos[x][z].y*sin(angle), pos[x][z].z);
      }
    }
  }

  void run()
  {
    for(int i = 0; i < size_x; i+=1)
    {
      for(int j = 0; j < size_z; j+=1)
      {
        //This determines how quickly a wave dies down
        wave_cent[i][j].y *= 0.35;

        //Move the center position of the wave based on its speed
        wave_cent[i][j].x += wave_speed[i][j].x;
        wave_cent[i][j].z += wave_speed[i][j].z;

        //If the center position of a wave exceeds the bounds of the main droplet matrix
        //we will reverse its direction of movement
        if((wave_cent[i][j].z - wave_speed[i][j].z >= l) || (wave_cent[i][j].z + wave_speed[i][j].z <= 0))
        {
          wave_speed[i][j].z *= -1;
        }
        if((wave_cent[i][j].x - wave_speed[i][j].x >= w) || (wave_cent[i][j].x + wave_speed[i][j].x <= 0))
        {
          wave_speed[i][j].x *= -1;
        }

        //This is just an optional safety measure to keep wave centers within bounds
        wave_cent[i][j].x = constrain(wave_cent[i][j].x,0,w);
        wave_cent[i][j].z = constrain(wave_cent[i][j].z,0,l);      

        //These loops are taking the center position of each wave and moving the amplitude 
        //of each droplet within its boundaries based on the amplitude of the wave
        //The main movement of the creek object is coming from here
        for(int x = int(constrain(wave_cent[i][j].x - wave_rad_x,0,w)); x < int(constrain(wave_cent[i][j].x + wave_rad_x,0,w)); x++)
        {
          for(int z = int(constrain(wave_cent[i][j].z - wave_rad_z,0,l)); z < int(constrain(wave_cent[i][j].z + wave_rad_z,0,l)); z++)
          {
            //pos[x][z].y += wave_cent[i][j].y*cos(x*angle)*sin(z*angle);
            pos[x][z].y += wave_cent[i][j].y + 0.5*wave_cent[i][j].y*cos(z*angle);
          }
        }
      }
    }
    //These are not scary as they look, they are only individual particle movement calculations. They could be simplified quite a bit.


    for(int x = 0; x < w; x++)
    {
      for(int z = 0; z < l; z++)
      {    
        pos[x][z].y += random(-0.1,0.1)*sin(angle/4);

        vel[x][z].x *= 0.85;
        vel[x][z].z *= 0.85;

        pos[x][z].y *= 0.2;

        if(z != 0 && z != l-1 && x != 0 && x != w-1)
        {
          pos[x][z].y += 0.55*pos[x][z-1].y;

          vel[x][z].x += k*(1/pow((pos[x+1][z].x - pos[x][z].x),p)
          + 1/pow((pos[x-1][z].x - pos[x][z].x),p)
          + 1/pow((pos[x+1][z+1].x - pos[x][z].x),p)
          + 1/pow((pos[x+1][z-1].x - pos[x][z].x),p)
          + 1/pow((pos[x-1][z-1].x - pos[x][z].x),p)
          + 1/pow((pos[x-1][z+1].x - pos[x][z].x),p));

          vel[x][z].z += k*(1/pow((pos[x][z+1].z - pos[x][z].z),p)
          + 1/pow((pos[x][z-1].z - pos[x][z].z),p)
          + 1/pow((pos[x+1][z+1].z - pos[x][z].z),p)
          + 1/pow((pos[x+1][z-1].z - pos[x][z].z),p)
          + 1/pow((pos[x-1][z+1].z - pos[x][z].z),p)
          + 1/pow((pos[x-1][z-1].z - pos[x][z].z),p));
        }
        else
        {
          if(z == 0 && x == 0)
          {
            vel[x][z].x = k*(1/pow((pos[x+1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x+1][z+1].x - pos[x][z].x),p));

            vel[x][z].z = k*(1/pow((pos[x][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x+1][z+1].z - pos[x][z].z),p));
          }
          else if(z == 0 && x == w-1)
          {
            vel[x][z].x = k*(1/pow((pos[x-1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x-1][z+1].x - pos[x][z].x),p));

            vel[x][z].z = k*(1/pow((pos[x][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x-1][z+1].z - pos[x][z].z),p));     
          }
          else if(z == l-1 && x == w-1)
          {
            vel[x][z].x = 1/pow((pos[x-1][z].x - pos[x][z].x),p)      
            + 1/pow((pos[x-1][z-1].x - pos[x][z].x),p);

            vel[x][z].z = k*(1/pow((pos[x][z-1].z - pos[x][z].z),p)           
            + 1/pow((pos[x-1][z-1].z - pos[x][z].z),p));
          }
          else if(z == l-1 && x == 0)
          {
            vel[x][z].x = k*(1/pow((pos[x+1][z].x - pos[x][z].x),p)          
            + 1/pow((pos[x+1][z-1].x - pos[x][z].x),p));

            vel[x][z].z = k*(1/pow((pos[x][z-1].z - pos[x][z].z),p)     
            + 1/pow((pos[x+1][z-1].z - pos[x][z].z),p));        
          } 
          else if(z == 0 && x != w-1 && x != 0)
          {
            vel[x][z].x = k*(1/pow((pos[x+1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x-1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x+1][z+1].x - pos[x][z].x),p)
            + 1/pow((pos[x-1][z+1].x - pos[x][z].x),p));

            vel[x][z].z = k*(1/pow((pos[x][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x+1][z+1].z - pos[x][z].z),p)    
            + 1/pow((pos[x-1][z+1].z - pos[x][z].z),p));        
          }
          else if(z == l-1 && x != w-1 && x != 0)
          {
            vel[x][z].x = k*(1/pow((pos[x+1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x-1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x+1][z-1].x - pos[x][z].x),p)
            + 1/pow((pos[x-1][z-1].x - pos[x][z].x),p));       

            vel[x][z].z = k*(1/pow((pos[x][z-1].z - pos[x][z].z),p)  
            + 1/pow((pos[x+1][z-1].z - pos[x][z].z),p)   
            + 1/pow((pos[x-1][z-1].z - pos[x][z].z),p));
          }
          else if(x == 0 && z != l-1 && z !=0)
          {
            vel[x][z].x = k*(1/pow((pos[x+1][z].x - pos[x][z].x),p)
            + 1/pow((pos[x+1][z+1].x - pos[x][z].x),p)
            + 1/pow((pos[x+1][z-1].x - pos[x][z].x),p));

            vel[x][z].z = k*(1/pow((pos[x][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x][z-1].z - pos[x][z].z),p)
            + 1/pow((pos[x+1][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x+1][z-1].z - pos[x][z].z),p));
          }
          else
          {
            //x = l - 1 && z in between
            vel[x][z].x = k*(1/pow((pos[x-1][z].x - pos[x][z].x),p)         
            + 1/pow((pos[x-1][z-1].x - pos[x][z].x),p)
            + 1/pow((pos[x-1][z+1].x - pos[x][z].x),p));

            vel[x][z].z = k*(1/pow((pos[x][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x][z-1].z - pos[x][z].z),p)
            + 1/pow((pos[x-1][z+1].z - pos[x][z].z),p)
            + 1/pow((pos[x-1][z-1].z - pos[x][z].z),p));
          }
        }

        if(pos[x][z].x >= nodes[x][z].x + rad[x][z].x ||
        pos[x][z].x <= nodes[x][z].x - rad[x][z].x)
        {
          vel[x][z].x = (nodes[x][z].x - pos[x][z].x)/(4*t);
        }

        if(pos[x][z].z >= nodes[x][z].z + rad[x][z].z ||
        pos[x][z].z <= nodes[x][z].z - rad[x][z].z)
        {
          vel[x][z].z = (nodes[x][z].z - pos[x][z].z)/(4*t);
        }

        pos[x][z].x += vel[x][z].x*t;
        pos[x][z].z += vel[x][z].z*t;

        pos[x][z].x = constrain(pos[x][z].x, nodes[x][z].x - rad[x][z].x,nodes[x][z].x + rad[x][z].x);
        pos[x][z].z = constrain(pos[x][z].z, nodes[x][z].z - rad[x][z].z,nodes[x][z].z + rad[x][z].z);

      }
    }
    angle += PI/12;
  }

  //This is used to create random movement
  void flow()
  {
    for(int x = 0; x < w; x++)
    {
      vel[x][0].x += sin(angle);
      vel[x][0].z -= 0.5*cos(angle);

      vel[x][0].x = constrain(vel[x][0].x,-4,4);
      vel[x][0].z = constrain(vel[x][0].z,-4,4);
    }
  }

  //This function starts inserts the initial waves into the 
  //creek based on the different frequencies in the music
  void ripple(float max)
  {
    //Max is the maximum amplitude that is allowed based on
    //a specific frequency in the music that is being played
    max = constrain(max, 0, 100);

    for(int i = 0; i < size_x; i++)
    {
      for(int j = 0; j < int(0.3*size_z); j++)
      {
        //wave_speed[i][j] = new PVector(random(-w/12,w/12),0,random(-l/6,l/6));
        wave_speed[i][j] = new PVector(0.1*wave_speed[i][j].x + random(-w/12,w/12),0,0.1*wave_speed[i][j].z + random(-l/6,l/6));
        wave_cent[i][j].y += 0.2*max + random(-j*max/24,j*max/36) + random(-i*max/24,i*max/48);       
      }
    }

    float tmp = 0;
    for(int x = 0; x < w; x++)
    {
      pos[x][0].y += 0.1*max*sin(angle) ;      

      vel[x][0].x += 2*sin(angle);
      vel[x][0].z -= 2*cos(angle);

      for(int z = 0; z < l; z+=int(0.05*l))
      {
        tmp = random(0,10);

        pos[x][z].y += random(0.5*max,max)*sin(2*angle) + random(-z*max/96,z*max/48);      

        vel[x][z].x += 2*sin(angle);
        vel[x][z].z -= 2*cos(angle);

        if(tmp >=7)
        {
          pos[x][z].y += random(0.3,0.6)*max*cos(0.3*angle);      

          vel[x][z].x += 0.3*sin(angle);
          vel[x][z].z -= 0.4*cos(angle);
        }      
        pos[x][z].y  = constrain(pos[x][0].y,-150,150);      
      }
    }
  }
}

rtavakko

import ddf.minim.*;
import ddf.minim.analysis.*;
import processing.opengl.*;
import com.hamoid.*;

Minim minim;

//Creek is the object for the river
Creek creek;

//This object creates a video of the sketch
VideoExport videoExport;

//Out: an averaged version of the fft outputs
//Cut_offs: specifies the frequency bands for Out
//Out_size: size of Out

FFT fft;
AudioInput inLine;

float[] out;
float[] cut_offs;

int fps = 30;

int out_size;

//This is used to calculate the angle of sine and cosine functions
float osc_ang;

//These parameters are for camera movement
//Eye: Where the camera is originally located
//Cam: Where the camera is currently located
//Lim: specifies camera movement boundaries
//Inc: How much the camera moves each frame
float eye_x,eye_y,eye_z;
float cam_x,cam_y;
float lim_x, lim_y;
float inc_x, inc_y;

void setup()  {

  size(displayWidth,displayHeight,OPENGL);
  frame.setSize(displayWidth,displayHeight);

  osc_ang = 0;

  //Initialize camera variables and objects
  eye_x = 0.55*width;
  eye_y = height/2;
  eye_z = 190;

  cam_x = eye_x;
  cam_y = eye_y;

  lim_x = width/50;
  lim_y = height/32;

  inc_x = lim_x/100;
  inc_y = -lim_y/100;
  //

  // set up our FFT
  minim = new Minim(this);
  inLine = minim.getLineIn(Minim.STEREO,2048);
  fft = new FFT(inLine.bufferSize(), inLine.sampleRate());

  frameRate(30);         

  out_size =3;

  out = new float[out_size];
  cut_offs = new float[] {0,200,16000,44100/2}; 

  videoExport = new VideoExport(this, "out.mp4");
  videoExport.setFrameRate(fps);

  //Width X Depth
  creek = new Creek(50,100);
}

void draw()  {

  background(0);
  //Move the camera position
  camera(cam_x,cam_y,eye_z,width/2,height/2,0,0,1,0);

  //process_audio: get fft array
  //process_values: average fft array and put values in Out
  process_audio();
  process_values();

  lights();

  rotateY(25.5);
  rotateX(12.33);

  //This is where all the threading happens  
  thread("creek_run");
  thread("creek_flow");
  thread("creek_ripple");

  pushMatrix();
  translate((width-10*creek.w)/2,height/2,-5*creek.l);  
  creek.display();
  popMatrix();

  videoExport.saveFrame();

  //Reset the cos / sin angle every 2*PI
  osc_ang = (osc_ang >= TWO_PI) ? 0:osc_ang + PI/36;

  cam_x += inc_x;
  cam_y += inc_y;

  inc_x = (cam_x >= eye_x + lim_x || cam_x <= eye_x -lim_x) ? -inc_x:inc_x;
  inc_y = (cam_y >= eye_y + lim_y || cam_y < eye_y) ? -inc_y:inc_y;
}

void process_audio()
{
  fft.forward(inLine.mix);
}

void process_values()
{       
  for(int i = 0; i < out_size ; i++)
  {
    get_average(cut_offs[i], cut_offs[i+1], i);
  }
}

void get_average(float f1, float f2, int index)
{
  int tmp_bin1 = fft.freqToIndex(f1);
  int tmp_bin2 = fft.freqToIndex(f2);

  out[index] = 0;

  for(int i = tmp_bin1; i < tmp_bin2; i++)
  {
    out[index] += fft.getBand(i);
  }

  out[index] /= (tmp_bin2 - tmp_bin1);
}

public void creek_ripple()
{
  //Out[0] is the bass average of  the fft
  creek.ripple(out[0]);
}

public void creek_run()
{
  creek.run();
}

public void creek_flow()
{
  creek.flow();
}

rtavakko

The code is a bit messy, let me know if a part of it doesnt make sense and I'll add comments

GoToLoop

Yikes, just did a quick glance there! :-&
1 thing I noticed you're relying on point(), which is very slow! ~:>
You should replace it w/ pixels[] instead: https://Processing.org/reference/pixels.html

rtavakko

Hey, I will add more comments and try to clean up the code. How do I use pixel[] for a 3D simulation?

farm402

You should definitely test the different renderers. I also wrote a programme for visualising music (https://vimeo.com/136952485) and even all my objects are basic 2D graphics I use the P3D renderer because it’s way faster for me.

clankill3r

Could you benchmark something for me.

Make your own vector with only a x, y and z. (methods and constructors are also ok) Processing has an array in it that is never used. This makes the object larger in memory. I'm wondering how much benefit you will gain.

GoToLoop

This makes the object larger in memory.

Indeed, removing that NEVER used array field, each PVector object would drop from the current implementation of 32 bytes down to 24 bytes! It's 8 bytes saved for each 1! 8->

I'm wondering how much benefit you will gain.

All those years I'm here, never seen any1 calling the unknown method array()!

For larger containers like IntList, Table, JSONArray, etc., it is indeed somewhat beneficial to cache the array the 1st time it's requested and keep returning it for future later on requests.

However, PVector is a very light container which are regularly instantiated several times.

It is so in total contrast to the heavier 1s, where most sketches merely instantiate a couple of them.
So it's not a big deal whether they happen to have some extra cache fields. ^#(^

colouredmirrorball

I'm curious, if the JVM detects the array of a PVector is never accessed in code, will it remove it?

GoToLoop

• Unfortunately I'm afraid not! All fields are initiated w/ their corresponding type's default value.
• As we all know for all reference types, including arrays & strings, they start off as null.
• In Java 32-bit, reference types consume 4 bytes.
• In Java 64-bit, for not so big apps, they also use 4 bytes. Otherwise 8 bytes each.

rtavakko

I tried P3D which made very little difference. As far as making my own 3D vector class I don't think that will make a big difference either; I tried converting one of my 2D visualizers into a real-time application and it has the same behavior. In this sketch, I use PShape objects which are rendered during setup and only scaled in realtime so I'm wondering if there is a problem with how I'm analyzing the audio and calculating fft values that's slowing things down?

koogs

Is this for live visuals?

I think you're going to have to chose between real time rendering and video exporting - if it's live you'll have to skip the recording, if it's not live them you can preprocess the fft data and use that as input - not real time but the resulting video will look as though it is

rtavakko

Yes its for live visuals. I have a separate template for offline visuals and the video export object is from there; never bothered to take it out lol. I'll try it without that tonight to see if it makes a difference [-O<

colouredmirrorball

On my five year old laptop the above sketch runs at 30 fps with video export disabled. I'd say that is decent.

Maybe you need to get into shaders and related magic so you can use the GPU instead of the CPU?

rtavakko

Thanks for trying it! Yes, 30 fps is decent. I've looked at some tutorials for shaders and OpenGL; its still a bit confusing but its on my list. Any tutorial suggestions?