The Yellow Wallpaper
Production design for a distributed reading of ‘The Yellow Wallpaper’, a short story by Charlotte Perkins Gilman. The reader’s voice, and intention, interacts with the pattern of the titular paper. This project uses speech recognition via Wit.ai, audio input from the reader’s laptop and osc messaging from a touch screen mobile device.
Katie Tindle
Introduction.
The Yellow Wallpaper is a short story told from the perspective of a woman who is convalescing following an unspecified illness and the birth of her son. As she grows more isolated, she begins to notice increasingly sinister patterns appearing in the wallpaper of her room. This work by American writer Charlotte Perkins Gilman, was first published in January 1892, and is now in the public domain. The video above is an excerpt of the longer text, which runs to about 20 minutes when read aloud. In order to exhibit the range of interaction with the wallpaper, I have truncated what would be quite a slow burning set of visual cues. All visuals and transitions were written in c++ using OpenFrameworks, with speech recognition via Wit.ai enabled with python scripting.
Technical.
Wit.ai
The readers voice interacts with the ‘wallpaper’ code in a number of ways. The most subtle interaction, and the most technically challenging, involved transitioning from one set of images and colour palette to another, based on the content of the readers speech. This involves taking a short recording of the reader’s voice and this as an http POST request to Wit.ai. Wit.ai then return a transcript of the text, along with any ‘entities’ it’s trained to recognise. In this case it was trained to recognise sentiment, and returns value indicating whether the recorded speech was ‘positive’, ‘negative’ or ‘neutral’.
After struggling significantly with posting audio streams in C++, I ended up writing a script in Python. The script records and POSTs the http request detailed above. The response is then converted to JSON, parsed and saved as a text file. My openFrameworks programme then reads this file, extracts the response and uses it to trigger the change in colour palette.
The image below illustrates the 'Positive' and 'Negative' color palettes.
This image shows the Python script running and writing the response to file. The openFrameworks console shows the the sentiment of the statement, in this case 'neutral', which is stored as a variable and printed. It then tries to delete the file - returning an error if the file has already been consumed.
Audio in
Other graphics are manipulated using the audio inputted through the reader’s laptop microphone. The volume of speech is extracted, and used to change the parameters of certain shapes in real time, as the image below illustrates.
OSC messaging
Although my intention was to have all scene transitions be dictated by voice, speech recognition was too unpredictable to rely on fully. In order to have more nuanced control of transitions during performance, I also used OSC messaging to allow me to make finer changes to the scene.
Reflections.
This work was originally conceived as a live performance, to be seen in a physical space. Following the restrictions imposed on movement earlier in 2020, I made the decision to try and adapt my production design for web. I think was ambitious and somewhat successful, however although I had intended to create an experience which was specifically designed for web, I do recognise that as a performance the reading may have been more engaging in person as originally envisioned. I also struggled with the Python scripting (I had never used Python prior to this project) so long that I wish I had spent more time on making the work more varied over time.
References.
Text
The Project Gutenberg eBook of The Yellow Wallpaper, by Charlotte Perkins Gilman (no date). Available at: https://www.gutenberg.org/files/1952/1952-h/1952-h.htm (Accessed: 7 May 2020).
Code
Audio input example, openFrameworks example sketch
indianpythonista (2017) ‘Speech Recognition using Wit.ai’, Indian Pythonista, 10 April. Available at: https://indianpythonista.wordpress.com/2017/04/10/speech-recognition-using-wit-ai/ (Accessed: 7 May 2020).
The Ultimate Guide To Speech Recognition With Python – Real Python (no date). Available at: https://realpython.com/python-speech-recognition/ (Accessed: 7 May 2020).
C++ addons
ofxDelaunay
ofxOpenCv
ofxOsc
Textures
Concrete 2, and Soft Wallpaper by Atle Mo via https://www.transparenttextures.com/
Font
Redruth's Basement Software, Fountain Pen Frenzy, (c) 2001 RBS, Freeware