Sonically Touchable
Sonically Touchable is an interactive documentation of a new musical instrument that aims to explore touch as a main source of inspiration.
produced by: Christina Karpodini
Introduction
Sonically Touchable is an interactive documentation of a new musical instrument. The main element of this piece is a projection of a remote performer exploring different ways of touch like they have been imposed by someone. The piece is developed around the idea of the new musical instrument, a sonic jacket that gives touch a sonic character, however, the sound character and the way of performance can change by the choice of the audience.
Concept and background research
Very often we do not appreciate something we have until it’s gone. This can be said about touch and contact with other people. But as technology is developing, new ways are invented to replace this loss. What if sound could be a way to substitute the feeling of someone else’s touching? On the other hand, are all ways of touch desirable and acceptable? What about the times when this sense is violating our personal space and our boundaries? The initial idea regarding the concept of distant touch has been inspired by the collaboration with Dr. Christina Kapadocha and her research project “From haptic deprivation to haptic possibilities”. The piece aims to explore the intersection between the ways of touch but also the affectiveness of the digital touch as well as the affectiveness of sonic feedback.
Sonically Touchable is a multi-perspective piece that has been conceptually and theoretically explored and developed through the exploration of multiple relevant pieces of literature. These works will be referred to in the following paragraphs of the technical description.
Focusing on the main subject of this piece which is the musical instrument made with haptic technology, there are many projects that influenced its progress. Instrumented Bodies by Joseph Malloch and Ian Hattwick is one of the main pieces that motivate my instrument making (Hattwick, Mallock, 2014). Although it is not haptic, it is wearable - a prosthetic- that highlights the meaning of the embodiment. The dancers took a big role in the design of the instrument by giving feedback and they are an important piece of the performance of this instrument. Another piece that I have been related to is tangible but not wearable, this piece is the Tactus by James Bulley which is addressed to a blind audience that can experience sound by touch (Bulley, 2018). The piece is formed as a poetic arrangement of sound and that gives the audience the chance to create their own narrative or explore the sound similar to what it is trying to be achieved in Sonically Touchable interaction.
Technical
The process of making this piece has been through many stages. The first stage has been the making of the sonic jacket itself and the sound design. Following was the test and filming of the performance and finally designing the presentation and the interaction.
The sonic jacket
The sonic jacket is a hoodie jacket that has been made interactive with the use of conductive ribbon tape. Small parts of the ribbon have been placed in parallel forming long stripes across the sleeves and the chest of the jacket. The ribbon has been connected to the electronics with conductive fabric and wires. The whole number of the ribbons that work as on the sensor of touch in the jacket is 12 for each side, right and left, therefore 24 sensors in total. They are connected to the microcontroller of adafruit MPR121 12-key capacitive touch sensor. The control is sending the data to the computer via the Adafruit Feather Huzzah. The technology of the Feather Huzzah was very efficient as it allows the wireless communication of the jacket with the computer via wifi/OSC communication.
The sound
The sound design takes a very important role in the message that this piece wants to convey. Three sound engines have been designed in max/MSP describing three different interpretations of touch according to the artist’s aesthetics and experiences.
The main sound material for the composition of the sounds was a recording of a poem recitation. The poem that is being heard in the instrument performance is the Astrophil and Stella, Sonnet 9 by Phillip Sydney (1586). In this poem, we can find a variety of emotions around touch ( Smith, 2020).
Using speech as a material for making sounds is a referral to Micheal Chion and Pierre Shaeffer semantic listening mode which uses language to convey a message focusing on the meaning of the words rather than the quality of the sound itself. (Chion, 1994)
- Mode One (smooth)
These sounds have been designed having in mind the touch as a form of showing caring and love to someone. The recitation recording is being used in a clear form where the instrument controls the segment from the whole recording that we are going to listen on the one side of the body and on the other the speed in which we are going to listen to this segment.
- Mode Two (exploration)
This performance mode explores touch as a way to explore what touch is for ourselves and how it feels when we don’t have the touch of other people. Smaller segments of the recording are spread out to the sensors of the instrument along with some random percussion sounds that create some sonic breaks.
- Mode Three (violence)
This last mode of the instrument has been designed to express how violence in touch could be sonically expressed. The most disturbing processing of the sample would be the only way to express something that outrageous as the violent touch. Here the recorded sample is granulated at a fast speed with small grains that are being played at various speeds creating a chaotic and disturbing soundscape. The one hand of the jacket controls the segment that is being granulated. In parallel with this soundscape, we are listening to a distorted FM synthesizer. Both sides of the instrument are ( micro) controlling elements of this synthesized sound.
In all three modes, we are listening to a synthesized sound based on Additive Synthesis. This background sound has been controlled by the FFT analysis of the sample. the amplitude of each band of the FFT analysis is controlling the amplitude of one sinewave. (Tristan Jehan's Max/MSP stuff, 2020)
The FM synthesizer in the third sound is also controlled by a sound analysis but this time by frequency detection. Using the object fiddle~ we are detecting different frequencies of the speech and we are sending them as the frequency of the synthesizer. (Downloads | volker böhm, 2020)
The presentation
The way this piece has been presented was part of the conceptual fulfillment. The restrictions that concerned the physical exhibition, the dynamic of the audience affected by being split in both physical and online helped in the development of this idea and the formation of the final presentation of the piece.
Interactive installation
Sonically Touchable is the title of the specific documentation performance with the sonic jacket. During the exhibition that took place on the 17th to 20th of September at St Hatchams Church the sonic jacket presented through a projection of a pre-filmed performance as a demonstration of something that a lot of people have been through, the distance and the inability to touch someone. Aiming to offer to the audience the experience of digital distant touch but also the exploration of the different ways of touch and how that feels being in the position of the one who is causing each way, a digital interface has been designed to enable this interaction with the documented performance. This interface was made in Processing and it was running in the exhibition through a raspberry pi using an iPad and a Virtual Network Computing system. The raspberry pi was communicating via OSC with an openFrameworks program that runs the video and was changing its start point according to the received data.
website
Similar to the installation interaction, a website has been designed to let the audience from remote locations to access and experience the piece. The website has been designed with the help of p5.js and the stylistic elements as well as the written content with the help of HTML and CSS. On the website on the left side visitors can see the video performance and on the right the same interface as the one of the exhibition. By clicking the mouse on the shapes of the interface users can interact with the video.
Experience the website in the following address:
https://ckarpo.github.io/touchProject/
Performance
The online presentation of the piece took place on Sunday the 20th of September at the official website of the degree show. In this presentation, the audience that attended the online exhibition could experience an uninterrupted performance with the sonic jacket. This presentation of the piece aims to offer a holistic view of the artist's sonic interpretation of touch and the performer's interpretation of the instrument. The fact that this performance was only presented online and not physically highlights the distance and the incapability of physical touch that this piece has been built around.
Future development
Future development is focusing on the musical instrument itself. Exploration of different designs and ways of using the conductive fabric will be the primary goal. Following that, this instrument is a digital instrument that can change sounds at any time. Experimentation with different techniques that facilitates equally different concepts like the aforementioned Sonically Touchable, is the next step. One of the first sonic compositional processes to be explored is the concatenative synthesis and the use of the instrument as a 2D template for navigation.
Finally, the physical computing technology behind the instrument is something to be developed and researched as well. There are many ideas for adding pressure sensors and mini speakers and boards that can contain sound engines so the jacket can be a stand-alone instrument without the use of a computer.
Self-evaluation
Making a new musical instrument, to begin with is a challenge. The most important lesson during this process is the multiple testing with the performer, something that did not happen due to social distancing restrictions at the time. The process of observing the performer trying to understand and explore the instrument possibilities is something that helps enormously the development and the evaluation of the instrument design.
Going back to the process of building this project during the summer, I look back to all the things I learned through the things that never worked but I found an alternative and the things that work but in the end, they did not offer something to the piece and they have been removed. Some of the things that did not work were to build a simple corpus-based synthesizer that shorts out the sound grains by frequency or amplitude, this attempt although it worked practically in computing and in numbers, it did not sound as I was imaging it therefore it couldn't progress in the final piece. Some other technologies concern the interaction and involve the building of an android app and the building of two websites that will communicate with each other. The last two were technologies that were aiming for the avoidance of the use of a communal device for the interaction.
References
Theoretical:
- Bulley, James, 2018. Sounding Materiality : Explorations In Resonant Practice.
- Chion, M., Gorbman, C. & Murch, W., 1994. Audio-vision : sound on screen, New York: Columbia University Press.
- Hattwick, Ian, Malloch, Joseph & Wanderley, Marcelo, 2014. Forming Shapes To Bodies: Design For Manufacturing In The Prosthetic Instruments.
- Smith, J., 2020. Astrophil And Stella, Sonnet 9 | Blogging Sidney's Sonnets. [online] Blogs.hanover.edu. Available at:
- Hogg, B. & Norman, S.J., 2013. Resistant Materials in Musical Creativity. Contemporary Music Review: Resistant Materials in Musical Creativity, 32(2-3), pp.115–118.
Technical:
- Openprocessing.org. 2020. 191227 - Openprocessing. [online] Available at:
- Adafruit Learning System. 2020. Adafruit Feather HUZZAH ESP8266. [online] Available at:
- Adafruit Learning System. 2020. Adafruit HUZZAH32 - ESP32 Feather. [online] Available at:
- Vboehm.net. 2020. Downloads | Volker Böhm. [online] Available at:
- desertbot.io. 2020. Headless Raspberry Pi 4 SSH Wifi Setup (Mac + Windows). [online] Available at:
- Parrish, A., 2020. Working With Video (Creative Coding). [online] Creative-coding.decontextualize.com. Available at:
- Web.media.mit.edu. 2020. Tristan Jehan's Max/MSP Stuff. [online] Available at:
Special Thanks to:
Dr. Christina Kapadocha for performing, recording the poem that is being used in the piece, and mentoring the conceptual progress of the project.
Rupert Earl, Natalia Kokkinos, and Emmanouil Kokkinos for the help in documenting this project.