Algorithmic Procedural Audio + Unity Game

Introduction: This is a description of my Assessment Project made for Audio for Games 2 Module at Abertay University (Dundee, UK). Task: Use PureData as Audio Engine (middleware) in Unity. Produce PureData Patches for Wind, Water, Fire, Vocalisations, Music, and Extras (this is part where my project shows the most advanced scripting). Apply advanced event and parameter management through scripting.

During the semester we were provided with template game level in Unity but I was working on my own much more complex level so I decided to work on my own game level.

Video of final game:

* Sounds that are not created in PD are: Footsteps and Orbs.

Project execution description:

Pure Data: PD Extended 0.43.4

Unity: 2017.3.1f1

Implementation method: LibPD

All Pure Data Patches were implemented into a basic game made by myself which include additional sound sources like footsteps implemented apart from Assessment tasks to improve the experience. I began with Heavy as integration to Unity but later I’ve moved to Lib Pd which was easier and faster to work with. Also, I had problems with Heavy as implemented Patches in a working build of the game worked on my laptop (Win10) but they didn’t work at all on my desktop (Win7) which makes me worried about compatibility issues.

Wind:

First Patch Wind took me a lot of time to make because it was my first patch that I was working on in PD. The approach for it was to use multiple [noise] objects as the source and modulate it with BP and HP filters. 3 [noise] objects were used to create three different states of wind: from slow to whistling fast wind. At the earliest stage, I followed instructions from YouTube tutorials but later I found that some objects like [drunk] cannot be implemented in Heavy so I had to find other solutions to create randomization as e.g. LFO is not a good source of modulation of wind. Even that I was able to implement a working solution it took me much more time to fully understand the logic behind it. Later during the course, we were shown how to implement wind and I have extended my patch by use of some of the examples. It was my first and last patch that I used help from other peers in our group. Because I wanted to use Wind Patch more than once in Unity project I decided to reveal more than few parameters in Unity, so I could reuse the same Patch and change behavior and timbre of wind to match to surroundings like snowy storm area with obstacles like mountains and big rocks or open space like a desert. I’m satisfied with the results as Wind Patch is quite complex, the sound is authentic and can be used many times for different environments. As a result, (except Default Patch which is slow and calm) Wind Patch is used in 5 areas in the game.

Storm (rapid, loud)
Sandstorm (desert, turbulent)
Desert dust (calm)
Forest/Park (mild)

Fire:

Fire Audio is not controlled by Unity’s build-in distance curve attenuator. For this Patch's different approach has been used following physics. Low frequencies are not absorbed as well as high frequencies over a distance, so I took an opportunity that Fire patch has three elements: two crackle sounds which have only high frequencies and one with low noise. I made one parameter to control levels of these three elements where crackles are attenuated earlier than low-end noise. In the opposite situation when a player is coming closer to a fire object, crackles are audible when the player is very close. This way it’s closer to the situation in the real world. The only one minus of this solution is that the curve between player and fire object is linear as opposed to real-world situations where the attenuation curve is not perfectly linear. Because I don’t know how to calculate the distance at the game level I had to experiment while calculating attenuation over distance in the script.

Water:

This Patch is almost fully based on an example from Lecture/Tutorial. I tried to extend the number of bubbles believing that a huge amount of these makes them less distinct from each other to minimize the effect of boiling water. But I stopped at the amount of 10 when I realized that it doesn’t change sound enough and probably more than 100 of different [bubble] object would be needed to begin to hear a difference which would use too much of resources for just one audio source. By consultation with other peers, I got to the conclusion that it’s impossible to create a realistic sound of running water in Pure Data. So, my further approach was to find a video of a river and to make notes of the sound characteristics. I found that a bit of noise can make use, but it must be gentle to avoid sounding like a wind. By frequency analyze of river sound from a video that I found I have created three different noise settings for noise in the range of 0-100Hz, 160-500Hz and 1000-3200Hz. Gentle randomization has been added to vary noise levels to make them sound more natural. Noise levels are low. I’m not satisfied with the result of this Patch. It doesn’t sound realistic for running water/river but may be used in situations where water is not running and is in the small pool (for example I’m pretty sure that I could hear almost the same type of sound in GTA V). Water Patch has been implemented in Unity by following instructions in the tutorial. As I’ve had river created in-game level already I had to change it to match one of the axes to avoid too much complexity which I wasn’t ready to deal with at that point. Like in the tutorial additional object have been created to follow a player (because it’s 3rd person game) on the Z-axis and at some point, also on the X-axis (Lake next to Waterfall). Because the river in my level is not visible by the whole level length I’ve had to use a bit of additional coding to limit movement of the object/audio source to match visuals in the game. The river can be found close to the ‘Waterfall’ area. 3rd alteration of Water Patch is assigned to ‘Ocean’ (sea) which surrounds the game level. This one also follows the player's position.

Vocalisations:

For this patch, I took an approach from Synthesis and Sonic Art Module of Vowel synthesis. Filter Bank has five vowels: A, E, I, O and U. These Filters are changing rapidly controlled by a randomized [metro] object. Saw wave that is a source of the sound is modulated by another Sine Wave Oscillator which has a randomized pitch to simulate speech. To make this sound even more ‘alien’ like I’ve decided to add additional global Amplitude Modulation where Modulator is additionally modulated by FM. At this point, I sent this sound to my friend who draws an alien character that we both created years ago. Drawing has been imported to Unity following tutorial steps. I had issues with a Shader because my version of Unity didn’t import settings properly and the original Shader type wasn’t available. But after tweaking I found a solution to have alien characters without visual glitches. After Lecture about Vocalisation, I decided to implement the approach with Arrays as Frequency Modulators to previously created Patch. I wanted to make these Arrays random, so I found the best solution that didn’t demand to write hundreds of numbers as messages. The solution is writing Array settings to text files and then use of [tabwrite] to write these settings on the fly. For each Carrier, Modulator and Index sub-patches I made 4 different presets/text files that are selected randomly with help of [moses] objects which give 64 different combinations. To make vocalizations dynamic, proximity effect was used. The script in Unity control an additional two parameters in Pd Patch over a distance between Player and Alien. These parameters are Carrier Pitch (closer to Alien = higher Pitch) and Index Amount (closer to Alien = smaller Amount). In addition to this is Pitchshifter Patch (provided in Tutorial) which is also assigned to Carrier. Alien can be found on Desert close to the giant machine with light scanners.

Music:

Chords: is the first element that has been implemented. The basic idea was to generate chords progression by use of weighted randomness. 6 basic chords have been selected and the percentage of each one was calculated basing on Canon Music chord progression: (I – V – VI – III – IV – I – IV – V). [moses] object was used to split chords by percentage of probability. To each [moses] object three different frequencies were assigned following three oscillators: 1xSine 2x Square PWM. 1st Sine Wave has additional FM which parameters are revealed in Unity, so they can distort sound in areas like close to the alien character, for example, to make spooky mood FM Index parameter is attenuating the other two oscillators to just keep distorted noise which generates by itself. Two other oscillators are controlled by a chain of filters which later are modulated by LFO’s and Phaser in between. The whole purpose was to animate sound and avoid the static sound of basic square waves. LFO of filters can be controlled in Unity and from the main GUI of the instrument. Different note length/tempo is available to set based on 60BPM as I couldn’t synchronize them with the main BPM Counter. Both oscillators have additional Reverb object [freeverb~] set as send and Room Size and Amount can be controlled from the main GUI and in Unity. Chords are double note length with slow amplitude.
Melody: Markov Chain is the main element of this sub-patch. Chain generates a melody from the pull of 4 notes and rest. Each note has a length of 1/16th. The real issue was to set the Markov Chain Matrix in the way to avoid absolutely random non-melodic sequences which would be annoying in the game. Depeche Mode – Strangelove melody was used as a source of notes and Matrix preset. Unfortunately, the result wasn’t rewarding. G note was replaced with Rest and preset settings were changed so fewer notes are played and while the melody is still random now it’s not that annoying and is more in control.
The sound source is Square Wave Oscillator with PWM. It’s animated with LPF LFO. As PWM led signal to go out of phase (or maybe DC was generated) so the chain of two HPF is set to 150Hz at the end of the chain to bring it back 0 phase area and to make space for [Chords]. To make it more pleasant additional Delay Effect has been created as send effect. Parameters can be controlled on the main GUI and in Unity. The tempo is not synchronized with the main BPM Counter at this moment as I’m not sure how to do it.
Kick: same as Melody, Kick is controlled by Markov Chain but instead of notes, this time 8th note Drum Sequencer is controlled. User can change which steps are switched ‘On’ in the main GUI. Sound Source is basic Kick Drum Patch generated from [osc~] with Pitch Envelope.
Snare: another module that is controlled by Markov Chain like Kick. The only difference is its signal source is an audio sample
Hihat: This one simply follows the main step sequencer controlled by BPM Counter. It’s simply white noise with short amplitude envelope and the main purpose of this element is to create a static rhythm to help the player to the fell tempo which would be hard to archive with only randomly played instruments.
Drumloop: this module use [tabread4] object to read audio sample which was made in DAW with additional virtual synthesizers and sampler. It has a different character than Melody and in Game both play in different areas. The purpose of this module is to diversify the character and style of the music.

Mixer: is visible on the main GUI window. The volume of each element can be controlled here. Also, Mute and Default [bang] objects are assigned here. With this parameter revealed in Unity, it’s possible to run or stop different elements in different areas in the game. I found a trick with the use of [lop~] object to slow down the reaction of parameters. So now Mute or quick change of parameter needs a few seconds to take full effect. This way it’s possible to [bang] Mute as a trigger in a game and instead of rapidly muting sound it will fade away slowly. This way coding in Unity will be much faster and simpler. I had a lot of issues with Markov Chain elements because they’re used for Melody and Drums and they conflict with each other. It took me quite a lot of time to somehow get control over them. I’m not sure if it’s the best solution but certain sub-patches were duplicated and reused for every instrument separately. Issues like overlooking that number Boxes receive messages make the whole process that long. Also understanding the logic behind Markov Chain mechanics in PD took me a long time to understand and I’m not sure if I fully understand it even now. The in-game music is altered twice: one is when the player is coming close to Alien. Everything except Chords is attenuated and Chords are modulated with FM to create sci-fi sound effects. Latter is when the player is in ‘Storm’ area and Music is distorted with the center point in the middle of the path with barriers.

Extras:

The Synthesizer (Synthezziser): The concept of this patch was to give the player the possibility to tweak the sound in the game and put this as part of the game mechanics. For this purpose, I created the synthesizer patch in PD. It’s split into 4 sections:

FM: Sine and Square PWM Modulators + Mix to morph between waveforms which may result as new type of waveform
Oscillator: 1x Sine + 1x Saw + 1x Noise. All three with additional Random Pitch/Filter
AM: Sine Modulator + Mix section
Filter: LPF + HPF + Mix between Serial and Parallel order.

In the game, the player can find 4 artifacts (communication devices). When the player gets close to any of them he will be able to tweak essential parameters. Getting close to them fire up GUI with sliders and with the name of parameters and synthesizer section but the font is hard to read for purpose so even experienced synth user may have to guess what is tweaked. Some parameters are mixed under one slider like 2nd Osc Pitch + LPF. It’s also possible to check out the sound anywhere on the map location by pressing R key but it’s not possible to tweak any of the parameters. The player has to tweak these parameters to unblock barriers that block access to other areas of the map. At this moment only the first two sliders must be set in the specific position. In a further development, synth may be re-programmed for new game mechanics. For example, by simplifying sound patches so the player can tweak easily more parameters (right now e.g. frequency modulation change sound drastically with slightest movement). In that scenario, it may be possible to create game mechanics where the player must recreate sound or pattern which is played in the background (or synchronize to). The whole idea of this Patch came to me in the last six days before submission, that’s why it’s still in an experimental stage.

Orbiting Planet: it’s the first Pd Patch that I have implemented in Unity. It’s really simple one 50Hz oscillator which level is modulated by LFO to create low pitch wobbling sound. It’s attached to orbiting the planet over the desert.

Machine: It’s patch created for the giant machine with tentacles that can be found on a desert close to the alien character. The idea was to create unpleasant, spooky, alien sound. For this purpose, I’ve gone into Amplitude Modulation. First, there’s an [noise] object with randomized BPF around 30Hz to create rumble and it’s distorted by multiply of volume 10 times and clipping by [clip~] object. Then more noises are generated by another sine oscillator with a randomized pitch around 40Hz. Its amplitude is modulated by [phasor~] oscillator which has pitch randomized between 10-100Hz. It's filtered by [vcf~] object with randomized Cut Off around 1kHz. Another signal source is [phasor~] which together with filter create vowel A which is added to a patch of the previous signal source and both are assigned to LPF which can be controlled in Unity and is revealed as ‘Distance’ to filter out high frequencies based on the distance between object and player. At this point sound was hollow and spooky and because it’s assigned to the giant machine I’ve decided to add low-end rumble. Two sine wave oscillators were used for this purpose which one is set to 48Hz and another to 44Hz which when added together they create Beat Frequency (some believe that it has impact on human brain and perception, but I have used it with a bit of skepticism as it’s not officially proved to have any impact). Anyway, because this machine (sound source) is an alien object my idea with Beat Frequency was to make the sound like a kind of sonic weapon to control the player's mind by aliens.

Ball: patch in construction is similar to KickDrum made from Sine Wave. 5 different settings are made for amplitude envelope and these settings are triggered randomly. The signal is split to ‘clean’ for strong low end and ‘distorted’ with [clip] object to simulate impact when the huge ball hit the sandy ground.Crickets: This Patch was made by following general information on the internet how to set parameters for Ring Modulator of Sine Wave like C7 as base frequency. RingMod parameters were set to get close enough to get something similar to cricket’s sound. Amplitude Envelope was set with random Decay. Chirp is made by very fast triggering by a random generator that uses prime numbers. Over that is another random generator that simply turns the volume on and off with slight decay to simulate crickets making sound from time to time. Crickets can be heard in Park (Forest).Crickets2: House Cricket. This Patch is briefly based on the tutorial of the same sound done in Absynth 5 Synthesizer. The base is a Sine Wave modulated by another wave that was drawn in Array (Waveshaping). Then [counter] was made to count 17 times per second. It triggers a very short exponential curve of an envelope and also Oscillator Pitch Modulation. The curve of Pitch Modulation is exponentially-rising. At the final stage, the level is modulated by Pulse Wave LFO to add short breaks between chirps. Crickets can be heard in Park (Forest). These are located in a few places with a small range of proximity.

Birds: It’s a simple patch with 13 audio samples of birds that I recorded a few months ago. Samples play randomly by [moses] objects and [metro] which tempo is randomized. Also, the volume of each sample is randomized each time the sample is triggered. It was easy to do but was hard to implement in Unity as some of the objects that were used for the first time doesn’t work in LibPD. So, it was necessary to make modifications to randomization mechanics. Depends on ‘time’ of the day, the quantity of birds is changing. When the sun (object) is in the highest position (y) then the quantity of singing birds is most intensified. When the sun is moving away from the highest position then the number of birds also decreases. Crickets and Birds are part of the Day/Night Cycle process. It was made by simply triggering GameObjects in Unity to active and inactive. The indicator for the cycle is Sphere Object ‘Sun’ which rotates around the map. One minute is equivalent to one hour in the game.Scheme for Day/Night Cycle cycle:

Waterfall: patch has to be made with Pink Noise as frequency characteristics of Pink Noise matched almost perfectly to the audio sample of Waterfall. Unfortunately, LibPD cannot read [pink~] object (or for any other reason Unity doesn’t receive a signal from this object) so I had to use White Noise. To get close to Pink Noise some profiling was made with LP filters and three instances of [noise~] were used to create full-range, low-range and hi-range of noise to match the audio sample of a waterfall. A bit of randomness is added to the filter of a full-range object. The main LP filter is controlled by Unity over distance. When the Player is close to waterfall then the filter is open, and the further away Player is then the filter is closing down to 100Hz. I had issues with spatialization for a long time to find out that all I need to do for noise-based patches is to set audio to 3D and unchecking ‘spatialize’ in Unity. With ‘spatialize’ on, the behavior of sound was very unpleasant and unnatural on loudspeakers.

Menu Music: for Menu, music has to be calm so the idea was to generate a simple random melody played by Sine Wave Oscillator. The output is transposed by Pitchshifter simple Patch. It’s transposed that way for purpose. [window] of grains is set to 442.29 to create artifacts that normally are unwanted but, in this patch, it makes it more interesting as it adds variety to the melody. From time to time one note of additional Sine Osc plays in a lower octave. These are two [osc~] objects where the second one is transposed 40Hz down to create Beats Audio. Finally, a signal was sent to [freeverb~] object but for some reason, it doesn’t work in Unity (even that it works in Music Patch). So, reverb was removed and one in Unity is used. Level Fade out in Patch is triggered when the ‘Play’ button is pressed in Game’s Main Menu.

#PureData #Middleware #audioimplementation #algorithmic #procedural #unity #abertay #university

Algorithmic Procedural Audio + Unity Game

Recent Posts

Commenti