Synaesthesia VR Application BA (Hons) Project: ABSTRACT

As new and final year of 'BA (Hons) Sound and Music for Games' already happened I finally started to work on my Honours Project. A bit late I know but for the first time I'm confident that I can do it in time (thanks to previous experience) and of course I have also my side projects and I finally got PS4 and Horizon Zero Dawn so you should understand...

Basic idea came to my mind out of the blue while walking back from mall. Synthesizer app in VR was my thought. Sound module but without typical knobs and faders but abstract controllers like boxes, balls and environment elements that you can step on, throw, punch was my thought. Then week later in a study room I got a revelation: Synaesthesia! Whenever I'm saying that word in my mind Thrillseekers - Synaesthesia (En-Motion) music track is coming to my mind

But anyway, as much as I love sound and 'it's my one and only true love' blah blah blah... I love visuals no less. So why not combine both? Of course it would be nice to have someone who can program shaders and do these visuals but I think that I will end up with much more abstract result like 'imagine if this app would be make by a team of talented people, not one guy who can do simple c# or BP programming'.

I have to learn about: VR (my whole experience is that I tested it twice for few minutes). How the system works, how to use it and how to work with it. Then do the research, read interviews etc. to find out what are the common audio-visual relationships. I have to choose 2 or 3 of them that are representative to keep it as close to real experience as possible. Then find a way to translate sound into these visuals. It should be pretty easy to do sound when I learn more details in that department. Implementation to visuals may be much more problematic.

I'm pretty sure that I learn a lot about Synaesthesia during next few months. Also my skills in audio implementation and programming should jump up a level or two. I'm not sure which platform I'll choose: Unity+PureData or UE4+Wwise. It all depends on what I find during the research.

While doing just most basic and simple research I found that I have problem with 'trivial' things like how to call Synaesthesia. Disease, issue, condition, phenomenon? Nice beginning.

So at this point I made my Abstract. Of course it's WIP and everything can change in final version of my proposal, especially after proofreading.

Ok here it is:

ABSTRACT

The aim of this Honours Project is to create VR application which will allow for first-hand experience and better understanding of Auditory Visual Synaesthesia through Virtual Reality technology.

Synaesthesia is a neurological trait where sensations, normally experienced separately are joined together which result in unparalleled experience. While many combinations of the mixing of the senses are possible, this project will focus on specific part of this phenomenon in which stimulation of hearing senses evoke visual experience. While most if not all of us enjoy audio-visual experiences like films, synaesthesia is involuntary, consistent and could not be experienced by person not affected by this condition, till now. Virtual Reality is emerging and still new technology which can serve as educational and entertainment tool at the same time in a direct, emerging way that no other audio/visual medium can. As every human affected by synaesthesia may experience it differently and time and resources for this project are limited, it is necessary to analyse interviews, personal reports and books to find common factors which can be used as competent examples. Further investigation on the subject of neurological processes that determine relationship between sound and visual outcome, together with studies and practice of working in 3D Engine will serve to create 3D application where user not only can play with audio-based 3D game but simultaneously learn and experience synaesthesia first-hand.