Jukebox
Jukebox is an audio-visual interactive performance reimagining the famous pop song “Me Against the Music” by Britney Spears featuring Madonna, released in 2003.
produced by: Sai Rapoports
Introduction
For this term’s final assignment, I have chosen to bring together some of my major interests into one artistic creation and challenge them through the incorporation of computation. Dance and performance art, pop culture and the occasional night out partying at the club all take a turn with this new constellation of elements made possible with the use of computer algorithms.
Concept and background research
Since I applied for the program, I have had many thoughts on how to explore the interface between the dancing/performing body and computational arts. This past year has challenged this exploration immensely, not merely because we were studying online but also due to the fact that everything was closed and my own body, my movements, had become restricted and numb. Luckily enough, this frustrating alienation from my physical presence has ultimately manifested into “Jukebox”, which has quite a sweet and sour aftertaste to it with this conflict folded into the piece.
Music is made for us and is there to improve the way we experience life. We are all easily affected by sound, and music in particular. We use it to support our current feelings or shift to a different mood, create an atmosphere, or make ourselves and others dance, to name just a few. Often music that has a repetitive structured rhythm (probably most of the music in the world) makes us move to some extent, react to it with our body almost automatically. This is because the music is always perfectly packaged and finite. If we take parties for example, we can say that it is a place you go to in order to lose control and let the DJ set you free, with you only moving to the beat and letting go.
Having this realization in mind, I decided to look further into how I can conceptually incorporate the longing for freeing the body through music like I am used to, with its unfortunate suppression presented by the restrictions of the pandemic. When Britney and Madonna’s song “Me Against The Music” started playing while I was listening to one of my playlists on Spotify, I knew this is going to be the missing link; the artwork reimagined. I decided that the aforementioned conflict is to be represented in the disruption of the traditional relationship between humans and music.
There were two main inspirations for “Jukebox”. One is the lyrics of the song, which stand in a certain opposition to its engaging upbeat rhythm. The following examples foreshadow the potential shift I made to our regular interaction with music:
The second inspiration for the piece was a performance by choreographer Jérôme Bel from 2001: “The Show Must Go On”. Bel explores how pop music as an agent and byproduct of globalization activates us humans and easily extracts sounds (sing along) and dancing from our body while powerfully “casting a spell” and putting us in an emotional state. With these two anchor points, the next stage was to articulate how to work with the power of music and at the same time detain the satisfaction.
In “Jukebox”, instead of the music directing the movements of the dancer, the dancer is the director of the music. Moreover, the movements control the heart of what makes music so affective- its speed. Without the structure, we find it hard to let go into the sound and move to the beat meditatively. We cannot follow the line in the lyrics that says “get on the floor, baby lose control”. We could imagine that controlling music in such way would correlate with our psychological strive for control in general. However, in this case it only provides part of the experience allowing some playfulness with the system. With the unbreakable knot “Jukebox” ties between our body and the music (“we’re the music”), we could never get the experience we have in a party, in our room in front of the mirror or even when we listen to music while commuting. As Britney and Madonna say: “we’re almost there”, but we will never be there. In addition, it is also interesting to feel how it is to try and react to the music at a certain moment, just to realize it is practically impossible, since once we react, we generate a whole new sound all over again. Another tempting tryout is to keep a steady movement so that the song will play in its original speed (which is largely in the middle movement speed range). Yet this too turns out to be quite a tedious task, with short glimpses of success.
Before I move to describe the system itself, I wish to briefly discuss the graphics. As John Whitney’s sine waves fascinated me since I saw them in the last term, I decided I want to create similar graphics in “Jukebox” and enrich the performance with some visual outputs as well. It has less a conceptual underpinning and more of an experimental character, while it gives me the opportunity to play some more with translating the inputs extracted from the body.
Technical research and structure
In “Jukebox” the spectator/performer can choose a song from a folder and place it into the buffer of a Max MSP patch (in the current documentation I chose to use “Me Against The Music”). Then he/she can dance to the music in the designated space in front of the camera. While they react to the sounds and play with them to get an idea of how they control and are being controlled by the audio output, some graphic visualizations are presented on a television screen positioned outside the camera’s scope. The visuals consist of two sine waves connected by lines and a flickering background. The movement speed of the performer affects the shapes of the waves and number of connective lines between them, while the color of the background is dependent on the amplitude of the music coming from the Max patch (the color is a grey scale lerp between black and dark grey which creates the affect of a party strobe light). In order to draw the background I used an FBO.
A major part of the program is written in C++ in OpenFrameworks using VS 2017 IDE. For the sake of clarity, the processes effecting different elements are done in separate classes, with the ofApp handling all of them together and wrapping it up. Here is a breakdown of the classes:
- Flow class is using the ofxFlowTools and ofxPS3EyeGrabber addons in order to compute the speed of movement of the performer detected by the camera
- Graphics class is in charge of drawing the visuals, i.e. the sine waves and the flickering background.
- MaxCom uses the ofxOsc addon to communicate with the Max patch and sends and receives information to and from it correspondingly to both classes mentioned above. It does so by using variables which are defined as global so that there could be access to them from all parts of the program.
In addition to OpenFrameworks “Jukebox” also incorporates a Max MSP patch which is in charge of getting the calculated movement speed and feeding it into a float object. This object is then fed into a groove object reading the audio buffer playing the song. Furthermore, the patch also includes a timestrech object so that the pitch will not change with the speed, and a recording object which allowes a clear sound in the documentation of the performance.
The process of achieving a relatively smooth interaction with the CV addon was quite challenging. At first, I was planning to use the frame differencing traditional technique but figured it might not operate robustly enough to fit what I imagine. Then I decided to create a little experiment with color tracking to control the speed and succeeded, only to find out later that the addon of skeletal tracking (OpenNI) I was planning to use in the same manner, is too complex to get into within the project’s timeframe, not to mention I have received some peer feedbacks on it being broken and not working properly in VS. The next stage was to try the ofxCV flow-keypoints example. This one worked quite well, except the longer the program was running the more it seemed like the algorithm is fading to a complete halt. With no errors and nothing in the code pointing at a possible reason for the problem, I had to let go of this solution as well and search for another method. Finally, I managed to extract what I want from the ofxFlowTools addon. Since it is extremely rich and has many features to it, I had no choice other than to “dig in” and find what I needed.
Future development
There are several ways in which I would like to develop this idea. Firstly, I would love to control other elements of the music with different inputs generated by the body. It could be the pitch, or even determining whether the song plays forward or backward. This could be done by numerous techniques: blob detection or skeletal tracking which would point out the movement of different parts of the body, or maybe optical flow that would make directions of movement matter. It could also be done by using a machine learning process with training according to sounds or words the spectator produces.
I would probably also rethink the graphics in the piece, decide if I want one screen or two, and what kind of visuals I want to show. Ideally, I might even choose to give up visuals and only have the camera, speakers, and a dancing space in the gallery for spectators to come a try the system. Another option is to replace the graphics with the control of lighting in the space, although at the current moment I have no experience in physical computing.
Overall I think this piece could go in many different directions and this is exactly what drives me to keep working on it in the future.
Self evaluation
I feel quite pleased with what I have achieved. As I started working on the project quite early on, I had time to develop an artistic concept that is appealing to me and will motivate me to go though the potential hurdles I will encounter along the way.
For the first time I experienced a monitored process of breaking down the problems and ideas I have to smaller pieces. I enjoyed conducting little prototype experiments which gave me an idea of what is within my reach and what should be put aside. It has also helped me to develop a sense of capability and strengthen my belief in my own skills. Looking back at the creation of “Jukebox”, it looks like an intricate patchwork.
I am a bit unsatisfied with the graphics and I think I could have done something better if I gave it more time and thought. In addition I wish to improve in searching for more resources and documentations online and to experiment more with incorporating different addons and other peoples code into my projects.
References
Inspiration resources:
John Whitney’s sine waves https://www.youtube.com/watch?v=jIv-EcX9tUs
Excerpt from Jérôme Bel’s “The Show Must Go On” https://www.youtube.com/watch?v=3v69hrtufDw
“Me Against The Music” official clip, performed by Britney Spears feat. Madonna https://www.youtube.com/watch?v=clwLKJ294u4
Addons:
ofxFlowTools by Matthias Oostrik https://github.com/moostrik/ofxFlowTools
Books:
ofBook- sound basics https://openframeworks.cc/ofBook/chapters/sound.html
Special thanks to Armando Gonzalez Sosto for guiding me throughout the process and to Faina Feigin and Nadin Hadasy for taking part in the video documentation process.