Res, a matter.

I’m delighted to announce the publication of two books on art and science to which I’ve contributed two different essays. These draw upon my recent doctoral research on corporeality and computation in sound performance. A multidimensional practice-based research which weaves together resources from cultural studies of the body, performing arts and human-computer interaction.

In “Experiencing the Unconventional: Science in Art”, I propose and discuss the notion of “configuration” of human bodies and machines. I do so by linking philosophy of human individuation, body performativity and the use of sound to mediate human physiology with machine circuitry.

“Do we control our body properties, or are they controlled by the media that condition the body in the first place? In our physical interaction with technology, are our body capacities the object or the subject of the action? The answer lies in understanding human subjectivity and technological individuality as two sides of the same iterative process. Human beings create technological artifacts which influence the formation of human subjectivity. In turn, the shifting characterisation of human subjectivity prompts new directions for technical development. It is a self-organisational feedback loop. From this viewpoint, we can move beyond the idea where identity is controlled, and embrace a notion where human subjectivity emerges through the configuration of human bodies with technological bodies. The term ‘configuration’ will be used in this essay to indicate not a mere pairing of machine and human bodies, but the their arrangement in particular forms and for specific purposes.”

Read the full article or buy the book.

In the second book, “Meat, Metal and Code \ Contestable Chimeras – STELARC”, I examine the practice of pioneering body artist Stelarc. By comparing, through the lens of Deleuze philosophy of sensation, the bodies in Francis Bacon’s painting with those in Stelarc’s performances, I elaborate on the notions of fluid flesh and rhythmic skin that link their works.

“According to Deleuze, when Bacon first draws a head and then scrubs the eyes and the mouth with a brush, he deforms the Figure by exerting directional forces on the canvas. The physical movement of the brush is the force that makes the drawing of the head mutate into something else, something that is not a human nor an animal face. This is the moment when the sensation is brought to light. By looking at the amorphous Figure, one can feel the sensation of those forces, their rhythm. Similarly, in Stelarc’s work, the physical tension of the fish hooks, the ropes and the gravitational field are the forces that make the body mutate into something else, a stretched mesh of skin emptied of its flesh and bones. By looking at Stelarc’s suspended body one can feel the sensation of those forces. The body is motionless to the eyes,
but the rhythm of the forces that pull the body downward and upward is clearly expressed by the deformation of the skin. The skin becomes rhythm.”

Full article can be read here, and the book is available at this page.

Enjoy the reading!

We (the EAVI research group at Goldsmiths, University of London) just got back from the SIGCHI Conference on Computer-Human Interaction in Seoul, Korea. CHI is one of the largest conference in the field, counting this year over 3000 attendees.

The CHI experience is as overwhelming as exciting. With 15 parallel tracks, there’s always something interesting to see and something equally interesting you are going to miss. To add to the thrill, this year the conference was hosted in a massive multipurpose complex, the COEX, which includes a mall, restaurants and other conferences all in the same venue. I leave the rest to your imagination.

My contribution to the conference was twofold. Over the weekend I joined the workshop “Collaborating with Intelligent Machines” and during the week we presented a long paper on our latest research on using bimodal muscle sensing to understand expressive gesture.

Organised by consortium members of the GiantSteps research project, Native Instruments (DE), STEIM (NL) and Dept. of Computational Perception, Johannes Kepler University (AT), the workshop run for a full day and involved several researchers working on embodied musical interaction, music information retrieval and instruments design.

Led by Kristina Andersen, Florian Grote and Peter Knees, we first went through brief presentations of personal research, including a keynote by Byungjun Kwon, then engaged in a brainstorming on the possibilities of future music machines, and finally went on realising (im)possible musical instruments using props, like cardboard, scissors, tape and plastic cups.

Eventually, we closed the workshop discussing the ideas emerged throughout the day, and in the evening we joined local experimental musicians for a sweet concert and some drinks.

On Monday the conference started at full speed. Dodging rivers of attendees, we managed to walk our way into the keynote venue, and started hooking up with colleagues from around the world.

On Thursday, we presented a long paper entitled “Understanding Gesture Expressivity through Muscle Sensing”. The paper, by Baptiste Caramiaux, myself and Atau Tanaka, is actually a journal article which we have published in a recent issue of the Transactions on Computer-Human Interaction (TOCHI).

Our contribution focuses on expressivity as a visceral capacity of the human body. In the article, we argue that to understand what makes a gesture expressive, one needs to consider not only its spatial placement and orientation, but also its dynamics and the mechanisms enacting them.

We start by defining gesture and gesture expressivity, and then present fundamental aspects of muscle activity and ways to capture information through electromyography (EMG) and mechanomyography (MMG). We present pilot studies that inspect the ability of users to control spatial and temporal variations of 2D shapes and that use muscle sensing to assess expressive information in gesture execution beyond space and time.

This leads us to the design of a study that explores the notion of gesture power in terms of control and sensing. Results give insights to interaction designers to go beyond simplistic gestural interaction, towards the design of interactions that draw upon nuances of expressive gesture.

Eventually, we showed a small excerpt from a new performance I’ll be previewing at the upcoming NIME conference in Louisiana (see below, and yes, that’s a sneaky preview!). Here I have implemented the feature extraction system described in the article, modifying and adapting the system to the more fuzzy requirements of a live performance.

The talk was very well received and prompted some interesting questions for future work. Some pointed to the use of our system together with posture recognition systems to enrich user’s input, and others questioned whether subtle tension and force levels can be examined with our methodology. Food for thought!

To conclude, here some personal highlights of the conference:

• “The transfer of learning as HCI similarity: Towards an objective assessment of the sensory-motor basis of naturalness”
• “Advancing muscle-computer interfaces with high-density electromyography”
• “Proprioceptive interaction”
• “From user-centered to adoption-centered design: a case study of an HCI research innovation becoming a product”
• “Collaborative accessibility: How blind and sighted companions co-create accessible home spaces”
• “I’d hide you: Performing live broadcasting in public”
• “As light as your footsteps: Altering walking sounds to change perceived body weight, emotional state and gait”

Useless to say, Seoul was surprising and heartwarming as usual, so… ’til the next time!

Here it comes, it’s that moment when you have to write your PhD thesis in less than 300 words. You know, it’s one of those awkward yet incredibly helpful challenges that, as a PhD candidate, you better face sometime sooner than later.

As we all know, any advice is more easily suggested than put into practice, you will be my proofreader. So, after two years of research, here it goes. On a second attempt I made it at 247 words.

“The subject of the research is the human body in the live performance of electronic music. By using notions selected from the field of body theory, I discuss the practice of established performers in the field. I look at how they combined human bodies and technological ones and what aesthetic and technical results their practice has led to. In so doing, I develop an argument for the importance to elaborate a model of human-machine interaction that goes beyond a notion of control. A model where the performer’s physical engagement with the instrument is paramount and the relation between the performer and the instrument is one of mutual dependence.

In order to build such a model, I use an interdisciplinary methodology which investigates the physiological and physical qualities of a perfomer’s motion. Namely, I look at the ways in which those qualities can affect the functioning of a digital musical instrument, and vice versa, how the functioning of a digital musical instrument can affect the physiological and physical qualities of a performer’s motion. This theoretical and technical knowledge will be used to develop performances to be exhibited publicly and evaluated through peer-review, so as to feed back into the research process.

This work, it is hoped, will contribute a methodology and a set of computational tools, which on one hand, produce novel ways to understand and design musical expression with body technologies, and on the other, encourage performance practices where human and technological bodies complement each other’s capabilities.”

The core research questions are:

Q1: Can notions from body theory inform the design of digital musical instruments?
Q2: If so, does the performance of live electronic music change and how?
Q3: Can physiological sensing provide an entry point to a performance model that goes beyond control?

If you have comments, please don’t hesitate to write me, I’m eager to discuss things and thoughts.

“Winding Road” by Steve Maloney. Picture by Brian Kennedy on Flickr.

The following is the introduction of an essay entitled “Taken Apart and Put Together: Human, Machine and Sound Technologies”. The title draws on a passage by Donna Haraway. I’ve just completed the essay for an upcoming book on art science, computation and life. The text is also part of my ongoing research into the combination of body theory and machine learning for biophysical musical instruments and performance strategies. In this sense, it gives a glimpse of the path I’m following. Hope the text can give you some ideas. Enjoy the reading…

UPDATE: Added some useful references at the bottom of the article.

One of the unique characteristics of sound lies in that, among all the energy waves that surround us, acoustic waves actively influence the human body at different levels, from intangible emotion arousal to physical induction of resonances, from pleasure and melancholy, to stillness and dance. From another point of view, for the human being sound is sound only when it is heard. It is in our ears that acoustic waves become a discernible event that we can feel, listen to and define as a sound. Because the experience of sound is intrinsically linked to the human body, the performance of music and the design of musical instrument have, for the most, been grounded in body motion. An obvious example is provided by the centuries-old history of traditional musical instrument making. String, wind and percussion instruments are all based on an interaction model whereby the kinetic energy of the human body excite the body of the instrument which then produces sound. With the arrival of analog synthesisers, the role of the human body in the performance of sound technologies shifted from sourcing acoustic energy to the instrument, to controlling its modulation parameters. Switching knobs, plugging and removing electrical cables, or tapping on a switch became understood as another gesture vocabulary. The idea of the body as a means to control sound technologies was then strengthen by the coming of digital computation, in the form of laptop computers and digital music software. Orchestras of ‘virtual synthesisers’ are operated by simply typing on a keyboard and moving a mouse. The performance of laptop music minimises the impact of the player’s physical effort on sound production in exchange for an easy access to a virtually infinite number of musical parameters and operations. With the increasing accessibility of technologies that gather information on a user’s gestural motion, and with the rise on the market of portable devices, the performance of sound technologies have gone through a re-evaluation of the player’s physical engagement in the production of sound. This is the first core concern of this essay.

How to leverage a player’s physical engagement with sound and computational technologies to enable the creation of new sounds and exciting performance strategies?
In which ways can we move beyond a metaphor of control towards an open-ended and mutual mediation of the player and the instrument?

It would be misleading however to discuss the shifting characterisation of the body in the performance of electronic music without adding to the picture the equally mutable characterisation of the body in the cultural milieu. The human body is cultural and political. It is cultural in that it is the active means by which public cultural artifacts are conceived and made. It is political because it is the subject and object of social strategies involving power, ethics and belief. It follows that the cultural understanding of the human body has important, yet often overlooked, repercussions on the understanding of the player’s body in a public musical performance.

Before the establishment of a dedicated field of studies, the body has been discussed as merely an operational part of the overarching structure of society. This approach has prompted a reductive understanding of the human body that have been rendered most notably through the views of essentialism and social constructivism. The former view, indebted with Descartes philosophy of dualism, sees the human being as an entity split between the higher activity of the mind and the lower fleshly mechanisms of the body. Social constructivism on the other hand, sees human beings as individuals whose subjectivity is constructed through the social making of culture. Thereby social constructivism does not account the body nor the world we live in as means of production of the social, but rather as results of it. Public critical discourses addressing directly the body have first arised from the second feminist wave during the ’60s and the ’70s. The feminist critique has addressed the objectification of the female body by a male-dominated power structure. Specifically, a broad public debate has focused on issues like family, sexuality, workplace conditions, and reproductive rights. Since then, a sociology of the body, or body theory have been established. Sociologists, philosophers, and scientists have investigated the body as the site where and through which identity, gender, religion, and knowledge are nurtured. These perspectives have in turn fostered the cultural analysis of the relation between the machine and the human body. Namely, the humanities have elaborated new definitions of what means to be human in the view of the increasingly intimate integration of the human body and machinic, computational, and biological technologies. This is the second core issue of the present work.

How can we discuss critically the integration of man and machine at a cultural and political level?
How can such understanding inform the way we design technological musical instruments and the related performance strategies?

The term ‘integration’ is used above purposely to indicate not a mere pairing of the machine and the human body, but the intermixing of two things that have been long segregated. The integration of the human and the machine is not intended at a metaphysical level, but at a practical one. Biomedical technologies, or biotechs in short, have provided us an entry point to a still largely unexplored territory where human and technological bodies blend together. DNA cells are used to do computations in test tubes, the entire human genome heritage is stored and categorised in digital database accessible via the Web, artificial organs are used to replace malfunctioning human organs, electrical and mechanical signals produced by physiological processes are channeled through the circuits of robotic prosthesis that enable human beings to restore their bodies. This is the result of a long history of research in the field of biomedical engineering and its branches, which include neural, genetic and tissue engineering, medical implants and prosthetics, and biomedical equipment design. Far from wanting to center this work on biomedical engineering, it will be shown how the knowledge produced in that field has been re-appropriated by performance art practice towards the critical questioning of human physicality and biology.

To summarise, the previous paragraphs have mapped out the broad areas of study that this essay will touch upon, namely a) performance of sound technologies, b) body theory, and c) biomedical engineering. Within those areas we have identified the core point of interest that are i) an open-ended and mutual mediation of a player and the instrument that leverages physical engagement, ii) the cultural and political nuances of the integration of human and machine and its relevance to musical instrument design. In order to elaborate a perspective that embraces such diverse topics we will make use of two key concepts: unfinishedness and biomediation. Unfinishedness relates to the incomplete nature of the human being and the ways such nature make us bound to extend our bodies through technologies. Biomediation refers to the intermix of human and machines at the physical and biological level, and to the modalities whereby this extends the expressive capacity of both the human and the technological bodies. In the remainder of this essay we review the origins and most relevant developments of those two key concepts in philosophy and cultural studies, and then discuss their applications in artistic performance, with a special focus on the role of sound technologies.

Some core references the full essay draws upon:

Bedau, M. and P. Humphreys
2008. Emergence: Contemporary Readings in Philosophy and Science. MIT Press.

Braidotti, R.
2013. The Posthuman. Cambridge, UK, Malden, MA, USA: Polity Press.

Clough, P. T.
2008. The Affective Turn: Political Economy, Biomedia and Bodies. Theory, Culture & Society, 25(1):1–22.

Gray, C. H., H. Figueroa-Sarriera, and S. Mentor
1995. The Cyborg Handbook. New York and London: Routledge.

Haraway, D.
2003. The Companion Species Manifesto: Dogs, People, and Significant Otherness. Prickly Paradigm Press.

Hayles, K. N.
1999. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. University Of Chicago Press.

Massumi, B.
2002. The Evolutionary Alchemy of Reason. In Parables for the Virtual:
Movement, Affect, Sensation. Durham: Duke University Press.

Maturana, H. R. and F. J. Varela
1980. Autopoiesis and Cognition: The Realization of the Living. Springer.

Mulhall, S.
1996. Heidegger and Being and Time. London and New York: Routledge.

Shilling, C.
2012. The Body and Social Theory. Third Edition. Sage Publications Ltd.

Simondon, G.
1992. The Genesis of the Individual. In Incorporations (Zone 6), J. Crary and S. Kwinter, eds., Pp. 297–319. Zone Books.

Stelarc
2002. Towards a Compliant Coupling: Pneumatic Projects, 1998-2001. In The Cyborg Experiments: The Extension of the Body in the Media Age, Pp. 73–77. Bloomsbury Academic

Tanaka, A.
2011. BioMuse to Bondage: Corporeal Interaction in Performance and Exhibition BioMuse. In Intimacy Across Visceral and Digital Performance, M. Chatzichristodoulou and R. Zerihan, eds., Pp. 1–9. Basingstoke: Palgrave Macmillan.

Thacker, E.
2003. What is Biomedia? Configurations, 11(1):47–79.

Turner, B. S.
1992. Regulating Bodies: Essays in Medical Sociology. Taylor & Francis.

Varela, F. J., E. Rosch, and E. Thompson
1991. The embodied mind: Cognitive Science and Human Experience. MIT Press.

Waisvisz, M.
2006a. Panel Discussion moderated by Michel Waisvisz. Manager or Musician? About virtuosity in live electronic music. In International Conference on New Interfaces for Musical Expression, May, p. 415, Paris, France.

In an attempt to map out the origins of the different perspectives on the technological body in the performing arts, I began compiling different timelines, looking at music technology, performance art, sociology of the body, system theory, natural sciences, and history. Now I’m in the process of grasping core theories in those disciplines and contextualising them in the their historical context. The next step is to find links and connections that, by sweeping across disciplines, will reveal the cultural, technological and social modalities by which the body, in the technological performance of art, has formed in the way we know it today. It is a complex research and it is proving difficult to keep focused on the subject matter, yet it is incredibly fascinating to gain a detailed map of this topic, and I’m confident it has the potential to be an interesting contribution both to the community and my own practice.
I’ll be updating the blog with further notes and thoughts as they emerge. Click the image below to enlarge.

This picture pretty much sums it all up. Given the interest of motion-gesture researchers in using mostly cognitive science to understand sound-making motion, there is room to explore a  politically engaged perspective that straight-forwardly address the (player’s) body.

The body is a political entity and body-based music performance produces political messages. A body is political for it touches the aspirations and the “open wounds” of everyone’s cultural and emotional background. Think not only of gender, identity, self-acceptance, who I am, who I want to be, who I pretend I am, but also everyday life as a part of a larger society of bodies. We are bodies that interact with and affect other bodies countless times a day, and together we put forth our world. What a body can do? What multiple bodies can do? How our bodies are modified by the instruments we use? and how this comes down to the design of new musical instruments?

The politics of physical musical performance might not be as evident as for performance art, yet, it is enacted with or without the player will. This is valid for the overall practice of music performance, and is particularly true when it comes to works that augment the body with technology. The question is: How can we make the design of physical musical instrument and performance with (bio)technologies politically significant?

An interesting perspective is forming by drawing upon the notions below. The summer will be the time for me to write a related literature review and thus combine this knowledge into an original form.

effort, a body whose physicality defines and mediates with the musical instrument (Ryan, 1991);
emergence, a body always in process (Massumi, 2002);
enactment, a body that brings forth a world by knowing (Maturana and Varela, 1992);
biomediation, a body reconfigured by media practices that condition its biology (Thacker, 2003).

References:
Maturana, H.R., and Varela, F. J. 1992, The tree of knowledge. Shambala.
Massumi, B. 2002. Parables of the virtual: Movement, affect, sensation. Durham N.C: Duke University Press.
Ryan, J. 1991. Some remarks on musical instrument design at STEIM. Contemporary Music Review, 6(1):3-17.
Thacker, E. 2003. What is Biomedia? Configurations, 11(1):47-49.