PhD Research

Soma: live performance where congruent musical, visual, and proprioceptive stimuli fuse to form a combined aesthetic narrative

Starting in 2007, I did my PhD at University College London’s Lottolab, in the disciplines of Human Computer Interaction & Music Technology, successfully defending my thesis in 2010.

Please refer to the relevant research papers in the publications section, and take a look at the videos on Vimeo:

And of course, do not hesitate to contact me if you want any further information on this work!


Artists and scientists have long had an interest in the relationship between music and visual  art.  Today,  many  occupy  themselves  with  correlated  animation  and  music, called ‘visual music’. Established tools and paradigms for performing live visual music however, have several limitations:

Virtually  no  user  interface  exists,  with  an  expressivity  comparable  to  live  musical performance.

Mappings between music and visuals are typically reduced to the music‘s beat and amplitude  being  statically  associated  to  the  visuals,  disallowing  close  audiovisual congruence, tension and release, and suspended expectation in narratives.

Collaborative  performance,  common  in  other  live  art,  is  mostly  absent  due  to technical limitations.

Preparing  or  improvising  performances  is  complicated,  often  requiring  software development.

This  thesis  addresses  these,  through  a  transdisciplinary  integration  of  findings  from several  research  areas,  detailing  the  resulting  ideas,  and  their  implementation  in  a novel system:

Musical instruments are used as the primary control data source, accurately encoding all musical gestures of each performer. The advanced embodied knowledge musicians have  of  their instruments,  allows  increased  expressivity,  the  full  control  data bandwidth  allows  high  mapping  complexity,  while  musicians‘  collaborative performance familiarity may translate to visual music performance.

The  conduct  of  Mutable  Mapping,  gradually  creating,  destroying  and  altering mappings, may allow for a narrative in mapping during performance. The  art  form  of  Soma,  in  which  correlated  auditory,  visual  and  proprioceptive stimulus  form  a  combined  narrative,  builds  on  knowledge  that  performers  and audiences  are  more  engaged  in  performance  requiring  advanced  motor  knowledge, and when congruent percepts across modalities coincide.

Preparing  and  improvising  is  simplified,  through  re-adapting  the  Processing programming  language  for  artists  to  behave  as  a  plug-in  API,  thus  encapsulating complexity in modules, which may be dynamically layered during performance.

Design research methodology is employed during development and evaluation, while introducing  the  additional  viewpoint  of  ethnography  during  evaluation,  engaging musicians, audience and visuals performers.