3D/Binaural audio mixing

edited December 2017 in Share Your Work

https://youtube.com/watch?v=nLJzY2LE4g0

Click the link above for a small demo of what is described below. Please wear headphones!

What is this?

This is a small example scene showcasing some code I've written for positioning sound sources in 3D space around the listener (the sketch actually starts when you see the head - at the start you Audacity to illustrate how panning affects the mono signal (and that it's truly mono)). The sound sources are just mono audio files which are represented by the sphere rotating around a human's head. You can place them anywhere in the space and animate them as well (currently just by hand) to rotate around the listener. The project makes extensive use of the Minim library and extends some of the unit generators, notably adding a Convolver and FadeIn/FadeOut.

How it works?

The code is based on an HRIR database provided by CIPIC. Essentially, our ears work as a filter that shapes the incoming sound - this alteration of the sound can be captured via an impulse response which we call head related impulse response. We can use this HRIR data to convolve it with the original sound source and so make it sound as if it is coming from a certain position. A HRIR database per subject can contain thousands of points of data - CIPIC provides us with 1250 points per point cloud (1250 positions), and almost 50 subjects. These points are collected at the same distance from the head's "center." The demo above demonstrates the use of HRIR for Subject 003, but there are two things to note: 1. We have different ears and so filter the sound differently. Hence why there are so many subjects when collecting data. Studies have shown, though, that you can learn how to localize sounds using someone else's ears over 2-3 weeks, which is pretty cool since you keep them in memory! 2. There is a limited amount of points in the point cloud, but an unlimited amount of positions. What we are doing here is using Delaunay triangulation (with some custom tinkering) to create a kind of a spherical mesh from which we could interpolate data based on into which triangle our defined point falls into where the vertices of that triangle are 3 know HRIR measurements - in such a way, we can cover quite well all the points in 3D space around the person.

Where is it going and why the post here?

I am looking to expand this to firstly include all the subjects from CIPIC and then add more functionality to also include the distance and different trajectories plus rendering so the files could be easily saved. I am using Processing to both quickly prototype and learn about this subject and while it may very well happen that at some point it will leave Processing and even get rewritten in another language, I am quite interested to hear from the community if there would be interest to add this as a library to Processing, along with all the upcoming results of experiments? The project already extends the Minim audio library a fair bit and I was considering sending the new ugens to Minim as well - I will quite likely build upon them a lot more, but if properly wrapped up they could be useful to others.

Who am I?

I'm a second-year computer science and music student. I have quite a lot of random programming experience mostly in Unity using c# but never ventured into programming for audio. Now that it is pretty much a part of my degree, I decided to tackle making a "virtual" binaural microphone, since I have made physical ones in the past and using a combination of both forms a part of my final year project!

Comments

  • Nice user interface. I would be very interested in this being available as a Processing Library!

  • Thank you, but the interface in the video you see at the start is just Audacity! I'll have to clarify that in the post. I'm showing Audacity just to show how it sounds when it's pure mono and so you could see (and hear) panning only changing the volume. I am looking for a nice way of making an interface for animating the sounds in 3D, because right now I just move the sphere using the arrows keys. Glad to hear it'd be a fitting thing to be a library! :) Guess I really need to think harder how to make it more presentable so people would know exactly what's going on quickly.

  • edited December 2017

    No, not Audacity -- I use Audacity myself. I meant the head in the gyroscope is a nice visual UI. ... I also understand that every piece of software using the library would not be interacting with it using a floating head.

Sign In or Register to comment.