How Do You Feel Today?
A speculative materialization of emotions.
produced by: Christina Karpodini
Introduction
Emotions are difficult to be expressed but also difficult to be detected and acknowledged. The purpose of this paper is to present a speculative design that measures emotions and materialize them as a reflection of individuals’ emotional statement. This installation aims to help participants to involve the process of self-exploration and acceptance of their emotions in their lives. For the needs of this installation, I worked on a software that combines data from self-report of emotions, and physiological measurement and translates them into an artistic audiovisual representation. This paper discusses the design of this installation from the perspective of feminist technoscience. It explores the affective technology and the sonification in parallel with the role of the use of emotions as material for artistic output.
Section One
Emotions, why should we materialize them.
One may think, what are the emotions in a first place and how we can measure them but most importantly, why to measure them. Defining emotions is really complicated and always a subject open to revision. Neurobiological or behavioral expression are some of the approaches that form the definition of emotions (Evans 2008). However, the most important is not to find or know the definition of emotion but to understand the importance of it in our lives and in the second phase, to explore how we can measure it for the needs of our research development.
Similar to the definition, exploring the importance of emotions is vast and difficult to structure. Back in the previous century, anthropologists supported that emotions are related to cultures, therefore people from different places of the world experience emotions in different ways. According to Paul Ekmen studies (1960), this is not true, in fact, there are differences in the experiences but there are these Basic Emotions that are common for everyone regardless of their culture or ethnicity(Evans 2008). Basic Emotions are shorted –lived and intense, and many researchers disagree about the number of them or even their name (eg 6 or 8, sadness or distress).
Emotions proved to be vital for the evolution of our kind. Scientists support that emotions give us intelligence in our action. For example without the emotion of fear we wouldn’t be able to understand the danger and therefore to protect our selves(Evans 2008). There are many ways that our body gives us messages about our emotional states and from those messages; we abstract the ways to measure them.
We get signals from our body, like sweat, but the main place where all begins is the limbic system. This is a very special part of our brain which structure makes us different from the rest of the creatures on this planet. However, it worth mention that mammals have limbic systems but their neocortex (1) is much smaller than ours and therefore they feel the basic emotions of fear and anger (Evans 2008).
More specifically, there are four ways that we can measure emotions. To begin with, Dorman suggests the self-report way of measuring emotions. People express verbally what they feel and with no need for extra equipment we have a report of emotion. Although this method sounds very convenient, it has a couple of drawbacks. The drawbacks of this method of measurement according to Mauss and Robinson is that people often are unwilling to express their emotions or unaware of them (Artino and Naismith 2015). These drawbacks have been noticed in tests of my installation design and they will be presented in the third section of this paper.
In addition to that, according to Winters and Wanderley, self-report can be misleading like the aforementioned occasions, unavailable in case of a disability that can not give us a verify emotion, and inappropriate when the self-report is complex (Winters and Wanderley 2014).
Another report is also not very accurate is behavioral. Signs like the voice tone, facial expressions or posture can give us indications about the emotional statement. My project is based on the recall of emotion, thus signals like these are difficult to repeat. Also, not everyone’s emotional reaction based on behavioral responses, thus, this method is not reliable especially for the purpose of this project.
The next two types of measurement are those that are used from most of the artists that want to materialize emotions, as they can be quite accurate and rich in data. Firstly, there are ways to measure emotion from physiological measurements such us galvanic skin response, heart rate, blood pressure. These are efficient methods and they demand simple equipment (Lopez 2017).
Finally, we can measure emotions directly from the source, the brain. Using equipment like fMRI, EEG, PET which measure the brain activity we can be even more accurate about the true emotional statement (Lopez 2017). This equipment though is not very accessible and it is expensive. EEG is the most approachable method but gets in order to be accurate it will need to be an expensive and complex EEG system. In addition, they demand basic knowledge around the neuroscience field or collaboration with equivalent scientists.
The last way of measurement raises a question regarding the accessibility of the data for the materialization of emotions, in particular. The materialization of data is a powerful way to communicate information through art and it intrigues many artists to work on this field. However, there are limitations on how much an artist can do especially if he is not supported or accommodated by an academic environment. Thus, some measurements that demand expensive and sophisticated equipment are available to a limited number of artists.
Emotions as material for making art.
In 2012 the artistic collective Blok from Canada created the installation “not my self today”, a wall in the streets of Toronto with pins of emotion names. In partnership with Partners for Mental Health, they aim to raise the awareness of mental health issues by calling people to choose their own pin ("Not Myself Today Campaign By Blok Design | RGD" 2019). More recently, Mark Verstand created an audiovisual experience, Aura, which materializes emotions. He is measuring them with physiological measurements and brainwaves measurements and it reflects an individual’s emotions("Nick Verstand Website" 2019).
Both examples used emotions as their main data that will form their work. Being quite different but yet so similar on their purpose, these installations aim to communicate mental health issues through art. Focusing more on computational art, it worth mentioning a project that has a lot of common aspects with the one of Mark Verstand. The pioneer Alvin Lucier, composer of electroacoustic music, in 1965, he used the technology of brain activity measurement to make music but in contrast with the aforementioned project he didn’t do it with any other intention rather than the artistic exploration(Straebel and Thoben 2014).
Sonification and aesthetics
This research project aims to explore the continuous reflective relation of emotions and sound with an ulterior motive to raise the awareness of the importance of self –evaluate our emotions and the communication of them.
In parallel with art conveying emotions, many types of research have been undertaken to prove the strong relation between sound and emotions. Sound incurs emotions in our everyday life even if we choose to listen to music or by unwanted sounds we hear in our soundscape (Zentner, Grandjean and Scherer 2008). In particular, Music Emotion Recognition (MIR), Music Psychology and Music Information Retrieval (MIR) have studied the relation between emotions and music. However, there are no many disciplines that have studied the emotions that occur from sound events (Drossos, Kotsakis, Kalliris, Floros) 2013.
In between scientific exploration of emotions and art, there are many projects that sonification emotions combining in this way science, technology, and art. Sonification of emotion can be characterized as affective technology as it is a way to communicate information about an individual’s emotional state and facilitate the communication of it (Winters and Wanderley 2014). The use of data from physiological measurements of emotions (e.g. galvanic skin response, heart rate, skin temperature) and brain activity measures (eg EEG, fMRI, PET)(Lopez 2017), are some examples of sonification procedures, a sound product that is more relevant to sound events material rather than musical material (2).
Although sonification is a great way to communicate information about data, it has been questioned for its aesthetics. Therefore, during the years of its development artists introduced techniques inspired by traditional electroacoustic music such us the music of Luigi Russolo, Pierre Schaeffer, John Cage, Edgare Varese introducing this way a more open creatively approach to sonification design(Hermann, Hunt, Neuhoff 2011). Considering this approach, sonification of emotion can be related to Pierre Schaeffer’s listening modes as it concerns aesthetics. Sonification of emotions, in particular, suggests the semantic listening mode, which is when the sound has a specific meaning for the listener and therefore makes the sonification affective (affective computing). As well as, reduced listening, which is the perception of the sound based on its characteristics rather than the meaning or the source (causal listening)(Chion 1994). This mode is the most relevant to sonification aesthetics as an artistic way to communicate data.
Taking these theories in mind, for the needs of this research project, I designed my sonification of emotion according to the new aesthetics and very much influenced by the use of sound events like in music concrete (sound objects)(3). More details for this design will be discussed in section 2.
(1)“the limbic system includes the hippocampus, the cingulate gyrus, the anterior thalamus, and the amygdala. All these structures are tucked away in the centre of the brain, underneath the outer layer of neural tissue known as the neocortex.”(Evans 2008)
(2) By Musical Material, we mean the Western Tonal Music.
(3) Sound Objects according to Pierre Schaeffer are recorded sounds that used in music concrete as a material for the composition like traditional composers have used the instruments or voice.
Section 2
Installation design
Measuring emotions is a complicated procedure; it is not always guaranty that will give the desirable data for the materialization process. Many projects have been based on long-term research, many experiments, and collaborations with the department of neuroscience("Projects:Emotionsonification [Input Devices And Music Interaction Laboratory (IDMIL)]" 2019). For the needs of this research and given the time limit, I created a speculative design that evolves self–report of emotions and arousal measurement via galvanic skin response. The participants of this installation will have to take a moment in advance and think some of the strongest moments of their days and recall their emotions, then, they will have to choose the emotions among a list of 8 basic emotions according to Robert Plutchik (Plutchik 1991). The emotions appeared on an iPad screen in a grid of different colors for each category (image1) They disappear from the grid when someone chose them, triggering that way the specific design sound. The audio is distributed in 8 channels, which will surround the participant (image2), each speaker related to one emotion. At the same time, participants are wearing a specially designed glove in which is attached to the GSR device. This device will measure the arousal of the participant, which will give us the density of the emotion. The role of this device in the installation is to bring the participants even closer to their emotions as this device will either confirm their emotions or refute them. The increased arousal will give a message of a stronger emotion and it will amplify the chosen audio.
Each emotion has a specific sound design. All emotions share the same two sources of sound, piano, and violin contemporary techniques processed with principals of sound design in order to be related to what they express. Following the aforementioned theory regarding sonification aesthetics, the sounds have been processed in a way to form textures and rhythms according to the emotion they represent. The goal of this installation is to reflect the chosen emotion creating in a way a virtuous circle of giving and taking emotions with a centric axis the affective technology.
In parallel with the audio feedback, a visual representation will appear above the speaker with the chosen emotion and the GSR data will also influence the shape of the visual representation of the emotion. Visuals will be generated and be individual so they help participants allocate their emotions in the sonic space. The visual design gives a lot of prospects for developing this project. However, the combination of visual and audio is something that needs to be examined and tested before the completion of the design of this installation.
Section 3
The Feminist Technoscience approach
“How do you feel today?” is an on-going research project which aims to provide a computational reflection of individuals’ emotions. Considering the role affective technology has in our lives, the need for seeking mental support in computational applications is more than predictable. A huge percentage of the developed world own a smartphone or wearable technology, many people have already a smart speaker like Alexa at their houses. Despite how useful these devices are they still are not able to understand human’s emotions. Thus, the difficulty of making the computer feel makes still psychologies irreplaceable. Although AI has made huge steps in coming closer to human and affective computing developing more and more day by day, it is not yet ready to help humans directly with their mental health needs.
This project aims to present a speculative design that will bring technology a little closer to humans’ emotions by reflecting them like a mirror reflect our image. Interacting with this installation will give people the opportunity to express their emotions and get an audio-visual representation of them. Getting this feedback they will understand the importance of the expression of their emotions but also listening to them in blend with others emotions they will understand that they are not alone, they are part of a community that “feels”.
Getting involved with affective technology, the developing process of this project should also involve the self –criticism and the ethical exploration of it. Affective technology is a very ambitious discipline as it aims to make computers feel and understand our emotions however it can be used by transgressing individuals privacy as. After all, emotions will become data that will be transmitted to a server in order to be processed and give feedback by the computer(Brigham 2017).
In this perspective, affective technology conceals risks if commercial industries use it for their own benefit. Therefore, my project is designed based on Feminist technoscience approach. The aim of the project is purely anthropocentric by the meaning of caring human and it’s mental health statement. I apply science in technology in a non-commercial way and with the main objective the care of people(Åsberg, Lykke 2010).
Evaluation of process
In the process of this research, I came across several new questions. The main question was regarding the interaction. How I will approach and convince people to express their emotions through my installation, considering the fact that we already refer regarding self-report of emotions having a big chance to face the unwillingness of participants. Being very optimistic about my work I decided to exhibit a demo of part of my installation, which included the self-report design and the audio feedback through headphones. After examining the documentation of the exhibition I was lucky enough to realize the problem of my design and the self-report system itself. People were tending to chose many, if not all, the given emotions in order just to hear the sound feedback and not to express their emotions. Therefore, I realized the limitations of this design and the need for differentiation of it and adding other ways of measurement. Another aspect I also learned from this example was that for this project it will be necessary to proceed in many tests/demos in order to understand the needs of the participants and the way they are willing to interact in order to express their emotions.
Conclusion
“How do you feel today” is a project that started its journey in project-based research discipline approached by the feminist technoscience theoretical framework. It remains a speculative design that aims to assist emotional expression by giving audio-visual feedback. The theoretical research and personal development through studying academic literature along with the support of my lectures’ advise took the project further than the first proposed idea. Scientific ways of measurement emotion have been added to the installation design but also the structure of the original idea had been changed in order to accommodate the new parameters that emerged through the research process. This procedure led to a deeper understanding of the theoretical framework and raised more questions for future development. Hence, in my future aspirations for this project, I would like to consider more the participants' interaction through the sociopolitical perspective and how they influence the design of the installation. Finally, after the realization of the project, and the development of its audiovisual aesthetics, I would also try to communicate with local communities and councils in order to work with people from different backgrounds for the purpose of further research but mainly to make this project functional and available to a broader audience.
References
· Artino, Anthony R, and Laura M Naismith. 2015. "‘But How Do You Really Feel?’ Measuring Emotions In Medical Education Research". Medical Education 49 (2): 140-142. doi:10.1111/medu.12642.
· Åsberg, Cecilia, and Nina Lykke. "Feminist Technoscience Studies." European Journal of Women's Studies 17, no. 4 (2010): 299-305.
· Brigham, Tara J. 2017. "Merging Technology And Emotions: Introduction To Affective Computing". Medical Reference Services Quarterly 36 (4): 399-407. doi:10.1080/02763869.2017.1369289.
· Chion, Michel, Claudia Gorbman, and Walter Murch. Audio-vision : Sound on Screen. New York: Columbia University Press, 1994.
· Drossos, Konstantinos, Rigas Kotsakis, George Kalliris, and Andreas Floros. "Sound Events and Emotions: Investigating the Relation of Rhythmic Characteristics and Arousal." IISA 2013, 2013, 1-6.
· Evans, Dylan. 2008. Emotion. Oxford: Oxford University Press.
· Hermann, Thomas., Andy Hunt, and John G. Neuhoff. The Sonification Handbook. Berlin: Logos Verlag, 2011
· Lopez, Gerardo. 2017. Unheard Sounds: A Manual for the Sonification of Emotional States in an Artistic Space (Undergraduate honors thesis, University of Redlands). Retrieved from https://inspire.redlands.edu/cas_honors/160
· "Not Myself Today Campaign By Blok Design | RGD". 2019. Rgd.Ca. https://www.rgd.ca/2012/06/01/not-myself-today-campaign-by-blok-design.php.
· "Nick Verstand Website". 2019. http://www.nickverstand.com/.
· Plutchik, Robert. The Emotions. Rev. ed. Lanham ; London: University Press of America, 1991.
· “Projects:Emotionsonification [Input Devices And Music Interaction Laboratory (IDMIL)]". 2019. Idmil.Org. http://www.idmil.org/projects/emotionsonification.
· Straebel, Volker, and Wilm Thoben. 2014. "Alvin Lucier's Music For Solo Performer: Experimental Music Beyond Sonification". Organised Sound 19 (1): 17-29. doi:10.1017/s135577181300037x.
· Winters, R. Michael, and Marcelo M. Wanderley. 2014. "Sonification Of Emotion: Strategies And Results From The Intersection With Music". Organised Sound 19 (1): 60-69. doi:10.1017/s1355771813000411.
· Zentner, Marcel, Didier Grandjean, and Klaus R. Scherer. 2008. "Emotions Evoked By The Sound Of Music: Characterization, Classification, And Measurement.". Emotion 8 (4): 494-521. doi:10.1037/1528-3542.8.4.494.
Annotated Bibliography
Hermann, Thomas., Andy Hunt, and John G. Neuhoff. The Sonification Handbook. Berlin: Logos Verlag, 2011
This book contains technical information about sonification, the technique that transmutes data into sound. Sonification is an interdisciplinary field as it used in a big range of applications. For this reason, in this book, the authors display and analyze all these fields among these, Interactive Sonification which refers to the practice of sonification while humans can interact and alter the data.
Evans, Dylan. 2008. Emotion. Oxford: Oxford University Press.
This book is a short introduction to the notion of emotion. In its chapters, it gives a brief explanation of various theories of scientists around emotions from the beginning of the previous century until today. The author presents why emotions are important to us and how they have helped us evolved as a kind. Finally, he is discussing the relation of emotions with the computer and all the recent news regarding the progress of affective technology.
Winters, R. Michael, and Marcelo M. Wanderley. 2014. "Sonification Of Emotion: Strategies And Results From The Intersection With Music". Organised Sound 19 (1): 60-69. doi:10.1017/s1355771813000411.
This article is about Sonification of Emotion, It gives information about the sonification as an art form that communicates information and its relation with the music making process. The author explores the ways we can measure emotions and he analyses them from the perspective of affective technology. In the main body of this article, he is presenting two ways of sonification of emotions according to the ways of measuring them.