Polluting pixels with sound

quark

This discussion was created from comments split from: How to first translate my idea into a processing sketch?.

Patricia_Brsk

Hello everyone!

I am desperately trying to find out how to do this little experiment I found online with Processing.

Can anyone help me out please? It'd be really appreciated.

Note : I am a beginner in Processing.

Thank you all in advance.

Pat

P.s. you can also contact me via email.

quark

@Patricia_Brsk this comment was tacked onto another discussion about something completely different. This is bad etiquette do NOT repeat this again.

Patricia_Brsk

OK, thank you. @quark

Chrisir

ok break your idea down in steps:

1 display movie

2 distort image

3 receive sound

write three separate sketches 1st, then make them work, then join them

• now for 1 : look at the libraries page 1 section, movie

• for 2 : get random pos , get its color and paint a rect there eg.

• for 3 : look at library sound or minim

;-)

Patricia_Brsk

Thank you a lot. I'll try what you said @Chrisir

For example, for step 1 : shouldn't I try the Capture code instead of Movie? So I can capture the webcam picture as in the experiment i linked up here

If I have questions, can I ask you here?

Thanks again.

Chrisir

sure

just post the three sketches here when they work

then join them

Patricia_Brsk

Sketch 1

import processing.video.*;

Capture cam;

void setup() {
  size(640, 480);

  String[] cameras = Capture.list();

  if (cameras.length == 0) {
    println("There are no cameras available for capture.");
    exit();
  } else {
    println("Available cameras:");
    for (int i = 0; i < cameras.length; i++) {
      println(cameras[i]);
    }

    // The camera can be initialized directly using an 
    // element from the array returned by list():
    cam = new Capture(this, cameras[0]);
    cam.start();     
  }      
}

void draw() {
  if (cam.available() == true) {
    cam.read();
  }
  image(cam, 0, 0);
  // The following does the same, and is faster when just drawing the image
  // without any additional resizing, transformations, or tint.
  //set(0, 0, cam);
}

Sketch 2

https://processing.org/tutorials/pixels/

is this the code i need?

void setup() {
  size(200, 200);
  img = loadImage("sunflower.jpg");
}

void draw() {
  loadPixels(); 
  // Since we are going to access the image's pixels too  
  img.loadPixels(); 
  for (int y = 0; y < height; y++) {
    for (int x = 0; x < width; x++) {
      int loc = x + y*width;

      // The functions red(), green(), and blue() pull out the 3 color components from a pixel.
      float r = red(img.pixels[loc]);
      float g = green(img.pixels[loc]);
      float b = blue(img.pixels[loc]);

      // Image Processing would go here
      // If we were to change the RGB values, we would do it here, before setting the pixel in the display window.

      // Set the display pixel to the image pixel
      pixels[loc] =  color(r,g,b);          
    }
  }
  updatePixels();
}

Patricia_Brsk

I also found this, maybe it is what i am looking for. what do you think?

http://www.learningprocessing.com/examples/chapter-16/example-16-7/

Chrisir

that looks great

Patricia_Brsk

When I run the code I linked from Shiffman it shows me just a black canvas :(

Chrisir

does your cam work?

Patricia_Brsk

yes, but when i run this code it doesn't turn on.

when i run other codes involving camera it works. i dunno :(

Chrisir

can you run this

https://www.processing.org/reference/libraries/video/Capture.html

?

Patricia_Brsk

Yes i can

Patricia_Brsk

I use a macbook pro 13" so i use its webcam

Chrisir

then look at the differences of the sketches and try to repair shiffmans

bilmor

Looking at your Sketch 1 and Shiffman's example I see a line missing from the latter. Try inserting video.start()just after the new Capture line.

Patricia_Brsk

I DID IT! It works!

Btw thank you guys!!!

Now comes the cool part, what about the sound?

@Chrisir you said i probably need minim for this, right? I already downloaded minim

Chrisir

basically this could be a starting point for you

(see examples)

http://code.compartmental.net/minim/fft_class_fft.html

on this page you'll find BeatDetect as well (lower right corner)

http://code.compartmental.net/minim/index_analysis.html

Patricia_Brsk

I'm trying some of these codes here https://processing.org/reference/libraries/sound/

but it says "No library found for processing.sound Libraries must be installed in a folder named 'libraries' inside the 'sketchbook' folder." How comes I don't have this library?

Trying with Minim now, i have it in my Libraries folder.

Chrisir

here you start with a song and use it as data for a graphic

apply this to the video thing you have

when the video moves according to the song, good

next, change the sound source from the source being a song to the source being the Mic

Chrisir

and write shiffman pls he needs to fix his sketch

;-)

Chrisir

quote

but it says "No library found for processing.sound Libraries must be installed in a folder named 'libraries' inside the 'sketchbook' folder." How comes I don't have this library?

there is a lib installer in menu Tools or so

install lib sound there iirc

Patricia_Brsk

No, I found out Shiffman did update his sketch, I didn't check the comments down below where people had the same problem i had with the code. ("black screen") Then he linked the github's page with all the example codes updated and it works now.

Ok, I'm trying with what you said, hope I'm getting something done :-S

Thanks ;)

Chrisir

;-)

Patricia_Brsk

I only see 3 sound libs, and i downloaded them. Anyways i'm trying with minim and see what happens

Chrisir

weird

quote

The Video and Sound libraries need to be downloaded through the Library Manager. Select "Add Library..." from the "Import Library..." submenu within the Sketch menu.

is it already installed?

do you have the current version of processing?

Patricia_Brsk

Yes I do. Processing 2.2.1 (19 May 2014)

Chrisir

skip lib sound

move on to minim

Patricia_Brsk

;)

Chrisir

this is a simple sound reader

//


/**
 * Get Line In
 * by Damien Di Fede.
 *  
 * This sketch demonstrates how to use the getLineIn (or other) method of 
 * Minim. This method returns an AudioInput object. 
 * An <code>AudioInput</code> represents a connection to the computer's current 
 * record source (usually the line-in) and is used to monitor audio coming 
 * from an external source. There are five versions of <code>getLineIn</code>:
 * <pre>
 * getLineIn()
 * getLineIn(int type) 
 * getLineIn(int type, int bufferSize) 
 * getLineIn(int type, int bufferSize, float sampleRate) 
 * getLineIn(int type, int bufferSize, float sampleRate, int bitDepth)  
 * </pre>
 * The value you can use for <code>type</code> is either <code>Minim.MONO</code> 
 * or <code>Minim.STEREO</code>. <code>bufferSize</code> specifies how large 
 * you want the sample buffer to be, <code>sampleRate</code> specifies the 
 * sample rate you want to monitor at, and <code>bitDepth</code> specifies what 
 * bit depth you want to monitor at. <code>type</code> defaults to <code>Minim.STEREO</code>,
 * <code>bufferSize</code> defaults to 1024, <code>sampleRate</code> defaults to 
 * 44100, and <code>bitDepth</code> defaults to 16. If an <code>AudioInput</code> 
 * cannot be created with the properties you request, <code>Minim</code> will report 
 * an error and return <code>null</code>.
 * 
 * When you run your sketch as an applet you will need to sign it in order to get an input. 
 * 
 * Before you exit your sketch make sure you call the <code>close</code> method 
 * of any <code>AudioInput</code>'s you have received from <code>getLineIn</code>.
 */

import ddf.minim.*;

Minim minim;
// AudioInput in;

int cellsize = 2; // Dimensions of each cell in the grid

AudioPlayer in;

int i=0; 

// ----------------------------------

void setup() {

  size(800, 600); 
  println( "Start of setup():" );


  minim = new Minim(this);
  //  minim.debugOn();

  // get a line in from Minim, default bit depth is 16
  // in = minim.getLineIn(Minim.STEREO, 800);
  in = minim.loadFile("test.mp3", 2048);
  in.loop();
  println( "End of setup()." );
}

void draw() {
  background(0);
  // Begin loop for columns

  int x ; // x position
  int y ; // y position

  color c = color(random(255)); // the color

  // use map()
  float val1 = map (abs(in.left.get(i)), 0, 5, 0, width);      
  float val2 = map (abs(in.right.get(i)), -4, 4, 0, height);

  // Calculate a position 
  x = int(val1); // 
  y = int(val2); // 

  // Translate to the location, set fill and stroke, and draw the rect
  pushMatrix();
  translate(x+220, y);
  fill(c, 204);
  noStroke();
  rectMode(CENTER);
  rect(0, 0, cellsize*3, cellsize*3);
  popMatrix();

  i++;
} // func 

void stop()
{
  // always close Minim audio classes when you are done with them
  in.close();
  minim.stop();

  super.stop();
}
//

Patricia_Brsk

What about MonitorInput? Minim > MonitorInput

Chrisir

or getLineIn()?

Patricia_Brsk

You are right :P

Patricia_Brsk

I am trying to merge the pixelation video code with getLineIn(). Is it right?

Then I want what the mic reads to be outputted to the speakers as in the example on youtube.

Chrisir

I am trying to merge the pixelation video code with getLineIn(). Is it right?

yes

Then I want what the mic reads to be outputted to the speakers as in the example on youtube.

why? What is said in the room is loud anyway.

when you play it over the speaker, you'll likely get a feedback loop. Very bad.

just display the distorted video?

I mean you're the boss. Just try it

Chrisir

Audio and video systems can demonstrate positive feedback. If a microphone picks up the amplified sound output of loudspeakers in the same circuit, then howling and screeching sounds of audio feedback (at up to the maximum power capacity of the amplifier) will be heard, as random noise is re-amplified by positive feedback and filtered by the characteristics of the audio system and the room. Microphones are not the only transducers subject to this effect.

https://en.wikipedia.org/wiki/Positive_feedback#In_electronics

Patricia_Brsk

Ok i understand.

Here it is, first time merging tho, so be clement :-S

import processing.video.*;

 // Size of each cell in the grid, ratio of window size to video size
 int videoScale = 8;
 // Number of columns and rows in our system
 int cols, rows;
 // Variable to hold onto Capture object
 Capture video;

 import ddf.minim.*;

 Minim minim;
 AudioInput in;

 void setup() {
   size(640, 480);

   // Initialize columns and rows
   cols = width / videoScale;
   rows = height / videoScale;
   video = new Capture(this, 80, 60);
   video.start();
 }

 void captureEvent(Capture video) {
   // Read image from the camera
   video.read();
 }

 {
   size(512, 200, P3D);

   minim = new Minim(this);
   minim.debugOn();

   // get a line in from Minim, default bit depth is 16
   in = minim.getLineIn(Minim.STEREO, 512);
 }

 void draw() {

   video.loadPixels();

      // Begin loop for columns
      for (int i = 0; i < cols; i++) {
      // Begin loop for rows
      for (int j = 0; j < rows; j++) {

      // Where are we, pixel-wise?
      int x = i * videoScale;
      int y = j * videoScale;
      // Looking up the appropriate color in the pixel array
      color c = video.pixels[i + j * video.width];
      fill(c);
      stroke(0);
      rect(x, y, videoScale, videoScale);
    }
 {
  background(0);
  stroke(255);
  // draw the waveforms
  for(int i = 0; i < in.bufferSize() - 1; i++)
  {
    line(i, 50 + in.left.get(i)*50, i+1, 50 + in.left.get(i+1)*50);
    line(i, 150 + in.right.get(i)*50, i+1, 150 + in.right.get(i+1)*50);
  }
 void stop()
 {
   // always close Minim audio classes when you are done with them
   in.close();
   minim.stop();

   super.stop();
 }

Prob : it says "unexpected token: void", highligthning : void stop()

Patricia_Brsk

I obviously made something wrong.

I think I have to put only one size command :-S Gosh i dunno.

Chrisir

before void }

Chrisir

in processing please use ctrl-t to auto-format the code

all { and } must match

same in number { and }

Chrisir

learn how to post code here in the forum please

Patricia_Brsk

I matched all the { and } and it seems ok for that now, but it says "Duplicate local variables i" highlitning "for (int i = 0; i < in.bufferSize () - 1; i++)"

Do you know what it means? Too many variables named 'i'?

Patricia_Brsk

Sorry for posting the code uncorrectly in the forum, i pressed ctrl+o before it anyways.....

:/

Chrisir

regarding i :

• yes, you must have a 2nd i somewhere (in the same scope) which is not allowed

regarding formatting code in the forum:

• leave 2 empty lines before and after the code please,

• then select the code with the mouse (e.g.)

• and hit ctrl-o

Patricia_Brsk

Fixed the code in the previous post so it's easier to check it out now, thanks.

By the way I corrected it and now here is what i've came up to so far :

import processing.video.*;

// Size of each cell in the grid, ratio of window size to video size
int videoScale = 8;
// Number of columns and rows in our system
int cols, rows;
// Variable to hold onto Capture object
Capture video;

import ddf.minim.*;

Minim minim;
AudioInput in;

void setup() {
  size(640, 480);

  // Initialize columns and rows
  cols = width / videoScale;
  rows = height / videoScale;
  video = new Capture(this, 80, 60);
  video.start();
}

void captureEvent(Capture video) {
  // Read image from the camera
  video.read();
}

{
  size(512, 200, P3D);

  minim = new Minim(this);
  minim.debugOn();

  // get a line in from Minim, default bit depth is 16
  in = minim.getLineIn(Minim.STEREO, 512);
}

void draw() {

  video.loadPixels();

  // Begin loop for columns
  for (int i = 0; i < cols; i++) {
    // Begin loop for rows
    for (int j = 0; j < rows; j++) {

      // Where are we, pixel-wise?
      int x = i * videoScale;
      int y = j * videoScale;
      // Looking up the appropriate color in the pixel array
      color c = video.pixels[i + j * video.width];
      fill(c);
      stroke(0);
      rect(x, y, videoScale, videoScale);
    }
    {
      background(0);
      stroke(255);
      // draw the waveforms
      for (int i = 0; i < in.bufferSize () - 1; i++)
      {
        line(i, 50 + in.left.get(i)*50, i+1, 50 + in.left.get(i+1)*50);
        line(i, 150 + in.right.get(i)*50, i+1, 150 + in.right.get(i+1)*50);
      }
    }
  }
}
void stop()
{
  // always close Minim audio classes when you are done with them
  in.close();
  minim.stop();

  super.stop();
}

Do I have to rename the int variable with another letter?

Chrisir

yes

Chrisir

in line 62 instead of i use

int iBuffer e.g.

not all the i's are iBuffer though ;-)

// draw the waveforms
      for (int iBuffer = 0; iBuffer < in.bufferSize () - 1; iBuffer++)
      {
        line(i, 50 + in.left.get(iBuffer)*50, i+1, 50 + in.left.get(iBuffer+1)*50);
        line(i, 150 + in.right.get(iBuffer)*50, i+1, 150 + in.right.get(iBuffer+1)*50);
      }

Patricia_Brsk

I changed line 62 as you suggested adding iBuffer to int, and to the other lines below it.

Now there's another error:

java.lang.RuntimeException: java.lang.NullPointerException at processing.core.PApplet.runSketch(PApplet.java:10713) at processing.core.PApplet.main(PApplet.java:10525) Caused by: java.lang.NullPointerException at processing.core.PApplet.size(PApplet.java:1762) at processing.core.PApplet.size(PApplet.java:1742) at glitch_pixel_sound.(glitch_pixel_sound.java:50) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at processing.core.PApplet.runSketch(PApplet.java:10711) ... 1 more

Chrisir

did you change the 2 line statements

I wrote for you above?