Howdy, Stranger!
We are about to switch to a new forum software. Until then we have removed the registration on this forum.
Categories
- All Categories 25.7K
- Announcements & Guidelines 13
- Common Questions 30
- Using Processing 22.1K
- Programming Questions 12.2K
- Questions about Code 6.4K
- How To... 4.2K
- Hello Processing 72
- GLSL / Shaders 292
- Library Questions 4K
- Hardware, Integration & Other Languages 2.7K
- Kinect 668
- Arduino 1K
- Raspberry PI 188
- Questions about Modes 2K
- Android Mode 1.3K
- JavaScript Mode 413
- Python Mode 205
- Questions about Tools 100
- Espanol 5
- Developing Processing 548
- Create & Announce Libraries 211
- Create & Announce Modes 19
- Create & Announce Tools 29
- Summer of Code 2018 93
- Rails Girls Summer of Code 2017 3
- Summer of Code 2017 49
- Summer of Code 2016 4
- Summer of Code 2015 40
- Summer of Code 2014 22
- p5.js 1.6K
- p5.js Programming Questions 947
- p5.js Library Questions 315
- p5.js Development Questions 31
- General 1.4K
- Events & Opportunities 288
- General Discussion 365
In this Discussion
- jeremydouglass December 2017
- LokiBear December 2017
3D/Binaural audio mixing
https://youtube.com/watch?v=nLJzY2LE4g0
Click the link above for a small demo of what is described below. Please wear headphones!
What is this?
This is a small example scene showcasing some code I've written for positioning sound sources in 3D space around the listener (the sketch actually starts when you see the head - at the start you Audacity to illustrate how panning affects the mono signal (and that it's truly mono)). The sound sources are just mono audio files which are represented by the sphere rotating around a human's head. You can place them anywhere in the space and animate them as well (currently just by hand) to rotate around the listener. The project makes extensive use of the Minim library and extends some of the unit generators, notably adding a Convolver and FadeIn/FadeOut.
How it works?
The code is based on an HRIR database provided by CIPIC. Essentially, our ears work as a filter that shapes the incoming sound - this alteration of the sound can be captured via an impulse response which we call head related impulse response. We can use this HRIR data to convolve it with the original sound source and so make it sound as if it is coming from a certain position. A HRIR database per subject can contain thousands of points of data - CIPIC provides us with 1250 points per point cloud (1250 positions), and almost 50 subjects. These points are collected at the same distance from the head's "center." The demo above demonstrates the use of HRIR for Subject 003, but there are two things to note: 1. We have different ears and so filter the sound differently. Hence why there are so many subjects when collecting data. Studies have shown, though, that you can learn how to localize sounds using someone else's ears over 2-3 weeks, which is pretty cool since you keep them in memory! 2. There is a limited amount of points in the point cloud, but an unlimited amount of positions. What we are doing here is using Delaunay triangulation (with some custom tinkering) to create a kind of a spherical mesh from which we could interpolate data based on into which triangle our defined point falls into where the vertices of that triangle are 3 know HRIR measurements - in such a way, we can cover quite well all the points in 3D space around the person.
Where is it going and why the post here?
I am looking to expand this to firstly include all the subjects from CIPIC and then add more functionality to also include the distance and different trajectories plus rendering so the files could be easily saved. I am using Processing to both quickly prototype and learn about this subject and while it may very well happen that at some point it will leave Processing and even get rewritten in another language, I am quite interested to hear from the community if there would be interest to add this as a library to Processing, along with all the upcoming results of experiments? The project already extends the Minim audio library a fair bit and I was considering sending the new ugens to Minim as well - I will quite likely build upon them a lot more, but if properly wrapped up they could be useful to others.
Who am I?
I'm a second-year computer science and music student. I have quite a lot of random programming experience mostly in Unity using c# but never ventured into programming for audio. Now that it is pretty much a part of my degree, I decided to tackle making a "virtual" binaural microphone, since I have made physical ones in the past and using a combination of both forms a part of my final year project!
Comments
Nice user interface. I would be very interested in this being available as a Processing Library!
Thank you, but the interface in the video you see at the start is just Audacity! I'll have to clarify that in the post. I'm showing Audacity just to show how it sounds when it's pure mono and so you could see (and hear) panning only changing the volume. I am looking for a nice way of making an interface for animating the sounds in 3D, because right now I just move the sphere using the arrows keys. Glad to hear it'd be a fitting thing to be a library! :) Guess I really need to think harder how to make it more presentable so people would know exactly what's going on quickly.
No, not Audacity -- I use Audacity myself. I meant the head in the gyroscope is a nice visual UI. ... I also understand that every piece of software using the library would not be interacting with it using a floating head.