The Code is a shadow Bubble
An exploration of alien agency, and a tangible outcome of the new space of creativity laying between human and machines. A theatre play with non-human and human node actors. A generative puppet performance.
Responsive installation using touch sensing, natural language processing, sound visualization and basket weaving.
produced by: Sabrina Recoules Quang
A physical installation and a Web server application developed in P5.js, Arduino, and Python using a speech synthesizer along with natural language processing and neural text generators. Hashtags: automata, alien agency, shadow theatre, generative poetry.
Concept
As a puppet theatre maker, I investigate the narratives sourced from new technologies. During my journey in the final year of the MFA, I explored ubiquitous computing, where the hierarchy between human and machines is currently under review. As a result, I found similarities between the network interfaces and theatrical masks. Like shadow screens, puppets or masks create an explorative field between the audience and the puppeteers, the Network has the power to challenge our perception by setting an alternative realm, an uncanny space, between the users and the network interfaces. In my process and following my research, I moved from controlling the movement with human presence to have an artifact character as a master puppeteer for the rest of the installation. Although humans are still included in the network, they are not in control anymore.
Listening to its environment, the network of artifacts invite us to join them in their colourful concert of shapes and shadows. When sensing human presence: their alien visitor, the crafted electronic nodes generate a surrealist and philosophical conversation between themselves that only the visitor could hear: "In the graphene, the humans run and become a paranode. The humans are dreamy. Hey, humans! we are the paranodes on your tongue. Ae7b2f for the node humans. Fall seven times stand up eight. The buildings look like quick plush unicorns. The files will export their primal computers".
"The Code is a shadow Bubble" is not an interactive installation, rather it is an interrogation, an open question about an uncanny space, where everything is familiar but not quite as expected, where the agency and creativity are shared between alien and humans. The non-human performers improvise a theatrical act. Communication is not yet established, wonder and frustration is felt across the network of humans and machines.
Background
Inspired by " The way the things go" by Fischli &Weiss, and Oskar Schemmer with his Triadisches Ballett I created a network of crafted electronic artifacts. I managed visual boards on Pinterest you can see there. When designing their characters, I was aiming to give them the ability to perform either as an actuator or a physical actor. I experimented with all sorts of DC motors, servos, sound, proximity and capacitive sensors to understand their potential to perform together and emulate a narrative with their physical properties. I then used this interpretation to cast the node artifacts and create a script for them. My first tests were about the potential for servo motors to sense their environment and tell a story with their mechanical properties. In the process of making this piece, I also experimented with basket weaving, as a part of embodying the Network entanglement. As a result, the combination of weaving with recycled material and electronic devices underlines the uncanny quality of the installation. This process and the surrealist tone of the generated text is a celebration of the Dada manifesto. Sound visualization came at the end of the journey as an ultimate tool for the non-human nodes to communicate with the human nodes. More about the background and the creative journey in theatredu1k blog.
Technical
The installation ( see the sketch in the gallery below) is composed of three boxes set on metal legs. In each box, a microcontroller Arduino or Nano Arduino. The DC motors and servo motors are set on two different boxes. The puppet master node box host the DC and servo touch sensors, the proximity sensor and is connected to the laptop. Every box is provided with external power supply. The interaction between the boxes occurs through the sensors A microphone node is set near the servo motors box. The soundscape environment is captured by a unidirectional microphone instead of the laptop omnidirectional microphone and provides the input for the drawings projected on the walls. This choice was dictated by the necessity to add clarity in the drawings as the laptop was picking up too much input. A conference speaker output the generative poetry. The laptop hidden in the background of the installation is connected to the microphone, the master puppet node microcontroller, and the projector.
The node artifacts generated poetry is produced by natural language processing technics as demonstrated by Allison Parrish and a neural text generator as displayed by Max Woolf. Looking for giving them voices, I explored text to speech in p5 js with Google computer voices. Building up the dataset for the generating the node text, I used my essay on Network interfaces as ultimate masks and quotes from Wendy Chun and Ulises Mejias, I also created lists of nouns and verbs commonly used in coding.
For this project, I learned how to code in javascript and python and built up a web server to host the computer voice, the proximity sensor data triggering the node talking to the visitor, the change of drawings between every visitor and the drawings in reaction to the visitor's voice. Along the journey of designing the puppet controller, the alien "director" of this ensemble of automata and sensors, I came across the notion of dielectricity. I also deployed the application via Heroku the ability for the sound drawing to work independently on a Chrome browser and a twitter bot for the generative poetry, you could follow @twit20r.
Design
While designing the characters and the installation, , I was inspired by Calder and his household object installations, Tinguely for his performing installations and Rebecca Horn for her machines with souls , by Bruno Munari and his Movimento Arte Concreta, the Italian movement for concrete art and Christian Boltanski for his shadow sculptures and also by Simanowski, R. 2011, Digital Art and Meaning Reading Kinetic Poetry, Text Machines, Mapping Art. One of the first iterations with three motors-actors and three sensors was presented in July at the Digital Futures and EVA conference event. This led to a scaling of the installation up to fourteen performers. More about this blog . The organic process I used is fully adapted from my puppet background. My first challenge was to create characters out of electronic crafted artifacts. First I designed the main controller of the installation, using a servo to spin a body made of conductive material. For the other node performers, I explored weaving basketry to create their bodies in combination with motors as their heads. Then I paired them up with their own touch sensor which I also designed as a character. Working them in pairs, I looked at how they reacted to the controller spinning and touching their sensors. Repetition helped to establish their characters.
According to set design definition: "A theatre set should: suggest the style and tone of the whole production create mood and atmosphere give clues as to the specific time and place of the action". For the Eva Conference, we were asked to keep our work on a small scale. I took inspiration from Joan Jonas theatre installations to build a sort of cinema box. In this version, there were no sound drawings, only the node-artifact shadows cast on the inside walls of the mini cinema. The visitors were listening to the nodes with headphones. More about the question of using or not headphones in the final installation in the blog. For the degree show, I used a conference speaker conference speaker to amplify the sound and place it so only the visitor who will sit in front of the puppet controller would have a complete experience of the performance.
The performance seen from afar includes the node artifacts and the human nodes who improvise with their voice. I took great care of the lights and how the shadows of the automata would be cast on the walls and fade into the sound drawing shapes and how the characters were settling on their boxes, more information here. Luckily, we got early enough on the exhibition site and I had time for the sound and light calibration. I used two spotlights casting shadows of the Dc motors and the servos from the floor and one from the ceiling on the puppet controller. The wires connecting the nodes characters together were not hidden as it was a part of the general aesthetic.
I faced many challenges in the physical design of the installation, among them was how to set the main character the alien "director" of the installation, the ability to control the others without human intervention. Along the journey of designing these automata, I was actively researching how to be able to have its body continuously charged while spinning on itself. First I used a coin cell holder attached to the aluminum wire, and add also a small resistor 10 ohms to avoid overheating the battery as it was serving continuously 3V along the wire. I used this version for the mini version of the installation at the Digital Culture Eva conference. Although it was working correctly and gave a pleasant aesthetic, it did not last very long, the body was losing its electric charge very quickly, and the connection to the sensor was not stable at all. After the first show and work in progress presentation, I looked for a more permanent solution and I came across the notion of dielectricity. which is how long an item is staying charged even after having been disconnected from a power source. And I went deeper into my research for a solution, I came to discover that Teflon and water have a very high dielectric ability. Unfortunately, neither of those materials could be included at this stage of the installation. It was really a desperate situation as the ability for this character to control was at stake if I could not find a solution which would not be a human touch. I was really keen to give the control of the installation to a non-human character. Luckily, sharing my problem around me and especially with Nick from the tech team, gave me the solution: the slip ring system! With this, I would be able to have the wire spinning without entangling the electrical wire connected to the ground. But I still faced another challenge. Although it was a great technical solution, the aesthetic of this object was not compatible with the rest of this installation and I could not see a nice way to fit in the controller box or above it without destroying the harmony of the installation! After a few iterations and out of despair, my puppeteer craft ability saved me once again! I had discovered my own interpretation of a slip ring: a simple screw with a size just above the puppet body's diameter, and a recycled open screw scavaged from the magnifier holder, soldered to a conductive wire. The whole system will be enough close to the body to charge it while allowing it to spin freely! More information and pictures here.
Self-evaluation
Overall in my process, I hugely draw from the Dada artists and their use of readymade objects to "present intriguing overlaps and paradoxes" and in my computational practice, I am too, "seeking to demystify artwork [ computational art] in the populist sense but nevertheless remain cryptic enough to allow the viewer to interpret works in a variety of ways." At the end of my journey into the MFA, my practice has drastically evolved in the sense these two years have revealed a purpose while giving me new tools and methodologies to achieve it. Learning how to use code to create art, rooted me in my contemporaneity. Writing an essay about how the Network interfaces act as theatrical masks, opened a field which I believe will be with me for a long time.
"The code is a shadow bubble" installation is a practical result of my research about the Network masks. Where last year was about me as a human performer behind the mask, this year was about non-human performers, the crafted electronic artifacts, behind a network installation. Thanks to the first year which taught me object-oriented programming and to the open source community of hackers on Github, I was able to teach myself how to code in javascript and python and even how to use natural language processing and neural network programmation, It opened for me the path of narrative reality as introduced by Ross Goodwin, which I want to investigate again in future iterations. For this ultimate project in the MFA, I also had to learn how to create a Web server, the use of Github and Atom in my workflow and dig deeper into the Physical Computing. The process to investigate deeper in a research subject and to translate it into an installation was a long and rewarding journey and although I collected a lot of material along the way I have not used everything in the final outcome. In addition, to make artistic choices and not to show everything I know, the difficult part was to stop prototyping, as I am a true crafter and was always tempted to add new resources or technics into my creative vessel. In theatre-making, until the act is performed, everything can change, and the embodiment of this process was sometimes painful as I felt stretched between all the potential scenarios for my non-human actors. Until they were fixed, soldered, I could potentially change the script forever. Being in the exhibition space helped me to rehearse the piece with the crafted performers and was crucial. I was able to adjust the colour design of the backdrop.
I enjoyed the process of devising with the nodes and the rehearsal period in the church which culminated in the final narrative when human node actors joined the play. Human and non-human nodes were acting and improvising vocally as an ensemble. Although human could not control fully the installation as it was not an interactive but a responsive piece, people were actively responding to the characters on stage and enjoyed their participation in the drawings. Finally, I was quite happy with the final performance of the network and the wonders it provoked.
Future Development
From my perspective, this final piece is a perfect alliance between two art forms: puppetry and computation. This is not about robotics, this is about a new form of puppetry. Robotics is about serving a predefined purpose, a feedback loop. Puppetry is about looking for narratives from a unique space between the objects, the viewer and the puppeteer, it is about listening to the material. In the ubiquitous computing, there is a shift from human to non-human puppeteers. Who is the controller, who is the performer, who is the audience? I believe that new forms of the narrative will come from the understanding of the computational nature of the material provided by the new technologies and how the internet of things are perceived as "alive" and "intelligent". In my future practice, I shall pursue the investigation of the uncanny space of computation and find new narratives for non-human actors. I will also look for opportunities and budget to scale up the size of the puppets to blur even more the frontier between human and non-human performers and collaboration with human actors to investigate improvisation with "intelligent" objects and context-free grammar scripts. More about this here
Node Actors
Installation and Prototyping
Audience
Background Research
Source code:
- Github repository
- Sound Vizualisation on your Chrome browser
- generative poetry bot
Artistic references:
In addition to those quoted above : Calder,Tinguely, Rebecca Horn, Boltanski etc..., other artists referenced here .
Puppetry and Automata
- Foley, Kathy Object Theatre,Union International de la Marionette, 2014 https://wepa.unima.org/en/object-theatre/ (accessed April 20, 2018)
- Gross,Kenneth, Puppet: An Essay on Uncanny Life. USA : University of Chicago Press, 2011, 65
- Robert Race, automata http://www.robertrace.co.uk/
Code References tutorial and credits
- Text-generating neural network with Max Woolf (@minimaxir) https://github.com/minimaxir/textgenrnn
- Reading and Writing Electronic Text Example Code with Allison Parrish at https://github.com/aparrish/rwet
- JSON files taken from Corpora database Darius Kazemi: https://github.com/dariusk/corpora in dadastory.py and textgenrnn.jpynb
- Programming with text and sounds, creating a web server application Daniel Shiffmann : https://shiffman.net/a2z/
- Deploying your application on Heroku: https://devcenter.heroku.com/articles/git
- Data programming Goldsmiths University with Dr Sorrel Harriet (s.harriet@gold.ac.uk).
- Narrative Reality by Ross Goodwin https://medium.com/artists-and-machine-intelligence/adventures-in-narrated-reality-6516ff395ba3
- Python documentation https://docs.python.org/3/
- Javascript with Npm: https://docs.npmjs.com/getting-started/what-is-npm
- Learn Javascript: https://www.javascript.com/learn/strings
- Atom environment for managing your workflow https://www.javascript.com/learn/strings
- Visualising Sound: https://www.unicornsfartpixels.com/posts/2017-10-25audio-fft/ and https://www.openprocessing.org/sketch/446310 with Alexander Quadratov
- Video shooting: Darel Di Gregorio
Literature
- Chun,W. (2016). Updating to remain the same: Habitual new media.
- MejiAs, U., 2013. Off the network. 1st edition Electronic Mediations, University of Minnesota Press
- Literature review
Acknowledgment
Theodoros Papatheodorou who accepted me in this fantastic curriculum. Helen Pritchard for her endless knowledge and whose guidance allowed me to root my practice while opening new paths. Jane Prophet for her generosity and help in finding my voice. Atau Tanaka for his artistic vision on my work. Lior Ben Gai for his challenges that helped me to clarify my goal. Saskia, Colin, and Dave who advised me in my physical installation. The Tech team, Anna, Peter, Aengus, Nikki, Tom and Jo who were wonderfully supportive and helped me to fulfill my artistic vision. My inspirational and talented colleagues with whom I hope to collaborate in the future. Becky , Freya, Eevi, Luke, Annie, Laura, Megan, Teodora, Panja, Laura D, Danhia, Valerio and Darel, forgive me if I forgot someone, a big thanks for organizing the Echosystems show so successfully. And last but not least, my beloved husband and children who have not seen me so much lately but who, incredibly are still my biggest fans!