Raytraced Audio Occlusion

Specialization

Introduction

The most basic form of audio is by just playing the sound directly to the speakers. The next step would be adding some 2d/3d stereo sound to get a feeling of oriantation. What I wanted to do was to simulate how sound would interact with the environment to get a more realistic sound.

To make this possible I decided to shoot rays from the audio source in random directions and letting them bounce and interact with the environment.

Step 1

The first step was to decide what I wanted the end result to be, as there are thousands of effects that can change a sound.

So the main focus was occlusion, where the sound gets lower when going through objects.

Step 2

Secondly I made a basic sound occlusion test in Unity. Where I send 9 rays between the listener to the sound source in a W pattern and checking if they hit a object.

In the first picture we can see the sound source, listener and the rays between them. The green line represent rays that have not hit a object, while the red lines are the rays that have hit a object.

We then take the rays that hit a object and divide them by the total ray count to get a value of how much we should occlude the sound. The occlusion parameter is predetermined in FMOD where we in this example just lower the volume.

Step 3

The occlusion we have so far would work in most cases, as it occludes the sound when there is a object between the source and listener.

The problem with this implementation is that it doesn't simulate real life well. For example as seen in the picture if we stand behind a wall we get fully occluded and won't here any sound, wheres in real life the sound would have bounced on the wall (yellow lines) and we would still be able to hear sound.

To fix this problem one solution would be to implement a more advanced raytracer where we let the sound bounce around the enviorment.

Step 4

In the first iteration of the raytracer we shoot X amount of rays and let them bounce Y amount of times. If a ray hit the listener it gets added to the final calculation.

In the UpdateRays function we shoot X amount of rays in random directions and pass them to the BounceRay function together with a bounce amount.

In the BounceRay function we check if the ray hit a object and if so we reduce the absorb value. The absorb varible represent how much that single ray got absorbed by the material of the object. After we absorb we reflect a new ray using the last rays direction and the hit surface normal. We then let this continue until either we hit the listener or we hit the max allowed bounces.

After we have sent out all rays and let them bounce around we gather the rays that hit the listener and add all rays absorbation. We then divide the final absorbation value by the rays to get the average. Then in FMOD we then simply lower the volume

Step 5

The implementation in step 4 still has a problem where if there is no object to bounce against we lose all sound. To fix this we can add one more ray and let it go through the object instead of reflecting it.

In this new implementation we continue to send rays in random directions but when bouncing them we create one ray that will reflect, just like in the last implementation. But this time we also create one ray that will continue through the object in the same direction as its parent ray.

In the Bounce function we save the type of ray to give more control over how different types of rays should change the sound. In FMOD we then choose what the different parameters should do with the sound.

Conclusion

With this implementation we get a pretty good simulation of occlusion. In the video we have the reflectiv rays in red and the occlusion rays in yellow.

The end result wasn't quite what I wanted, so what would I change if I did it again.

First problem is the sound itself. Because I'm not a sound designer I was unsure what effects in FMOD I should use and how to make the sound reflect a real sound wave hitting objects.

Second problem is performance. When sending 1000 rays and letting them bounce 10 time with 35+ colliders I have around 11 fps and thats for one sound source. This is not optimal and would not work in a game.
To fix this I started to implement the raytracing to a compute shader and let the GPU handle the heavy calculations. This would work pretty good as every ray only interacts with them selfs, so calcutating them in pararell would not be a problem. Sadly I did not manage to finish it within the time frame I had.

To get a even more realistic and nice sounding sound, there was some features I would have liked to add. One of these were reverb, I think calculating reverb in my raytracer would have improved the final result alot by giving the player a sense of the size of the space their in.

Because of being very sick for 6 out of the 8 weeks of my specialization, I wasn't able to finish where I had planed. For example I only made an implementation in Unity and not in our in-house c++ engine which I had planed for. Despite this I'm quite happy with the end result and what I managed to do.