Interplay through Wave
It is an audio-visual installation and it aims to co-create the rhythm of the wave by humans and multispecies such as goose and ducks living in the lake. Particularly, the light of wave penetrates through the multi-layered fabric as both spatial and sensory medium. The audience joins to regenerate and interact on an audio-visual wave with their movements through an immersive experience. It dances together in a similar way that multispecies live through the water in lake and water waves do.
produced by: Yuhyeon Jun, Agnes
There are 2 versions of video in below. Both are good to hear with 2 channel speakers or headphone.
1st version: Special & Better version with video recording & spatial sound as well. It also includes 'How it works based on Computer Vision - Kinect'.
2nd version: Previously made - a short version with whole scenarios and demo. It includes the fabrication process of installation as well.
1st - Better & Special version
2nd - Previous version
Introduction
The inspiration journey came from regular field recording the multispecies in lake such as Brokewell park, Burgess park and Hampstead Heath in London. Hence its generated as computational art with the audio-visual communication in waves of resonating light. It is to highlight how audience could see & hear their lives and recreate balanced rhythm together.
As a relationship on medium of light and multi-layer, it let wave of light to penetrate through multi-layer and create spatial & rhythmic experience to viewer. Shadow follows light, which means not only viewer becoming seeing the light but also they can create shadow and see them through their eyes or seen through other audience. Regarding with this, in my installation, audience could join with “Shadow Play” as well. There’s backside space where audience can join to create shadow and play with it. They can’t see the shadow through their eyes, but it can be seen through other viewer or reflected in the wall, which is used as 4th layer of screen. Hence, there is interplaying momentum when viewer becomes to join as spectacle and another viewer look around them.
A website regarding concept and storytelling exploring some other aspects, fieldtrip and sounds introduction about spatialization were previous research step as well. It can be looked around here: https://www.joyfulmediaartistagnes.com/work/wave-scape
Concept
I’ve thought of telling stories of multi-species living in a lake and communicating through waves, which could be both seen and unseen material. To highlight and better communicate with the audience on how they could really see/hear/feel in an immersive way to think beyond “protecting multispecies lives in the lake, which is becoming smaller recently refer to London city council, Brokewell Park and burgess park.
The aim of the storytelling was to co-create the rhythm among us as humans, multispecies especially in the lake through my experience, and nature. Since we are living in the world with a rapidly changing environment and pollutants. It was important to explore and think beyond ‘How could we as audience and artist help find balance among humans, nature, and animals’.
Hence through the interactive installation ‘Interplay through wave’, I’ve generated the space and sensory experience for us to feel the story through the form as a wave, the medium through the light in spatially fabricated multi-layered fabric screens in dark space.
According to BBC ‘What is ray of light made of ?' it proposed that when the light passes through the thin slits, it behaves in the same way that water waves do when they pass through a narrow opening. They diffract and spread out in the form of hemispherical ripples.
My aim and progress during fabricating the piece was always together with thinking on ‘How light could intermingle together with wave?’ and ‘How could sound and visual with wave involve more through audience?’. Through diverse exploration of computer vision-based input such as Kinect, I’ve figured out how to combine the human movement with footage with audiovisual art. The core metaphor was distance and movements between humans and nature/multispecies. Since we’ve experienced distancing society, which we’ve gone through so far, it is telling our well-balanced rhythm and distance between humans and nature such as the lake as well. If we go too close, the lake is being polluted by our activities which make multispecies being harmed. I’ve pulled out that story through my installation to explore and let the audience imagine the co-rhythm.
Background research
As imagining through immersive space with installation, I've inspired by thoughts on viewer's representation with digital media. According to Smith(2007), 'the space' between art and participants are in the state of being 'flexible'. And he especially states that the entire space begins to have 'interactive dialogue' among object, viewer, and the other spectacles. The term 'spectacle' is defined as another object when the participants join the performance and see another audience. The term ‘spectacle’ is defined as another object when the participants join the performance and see another audience, Smith stated. It is also a space where the audience could begin to enjoy and rebuild with self-selected ‘flow experience’. It brings engagement for them to get a sense of ownership for their exploration and their own imaginary world.
In research of Olafur Elliason’s project ‘Beauty’ (1993), the splashing water and a spotlight are used to be shown like a rainbow with the viewer's eye in the darkroom. Elliason(2008) was particularly excited by the 'angle of the viewer's eye' that drop the lamp to make 'rainbows', in which an experience of producing a unique & individual rainbow is made by the viewer. it brings the joy of ‘multi-dimensional space’ for the audience to imagine and co-create their unique thoughts and experience.
In another research of 'Your uncertain shadow(color)'(2010) of Elliason, the room appears to be empty, but once the audience goes there, it creates a shadow on the wall, projected from his shape. It is the participants’ ‘uncertain shadow’ and visitors in the room who are actually the ‘producer of content’ as they take control of narratives of the art, according to Elliason (2010).
Hence throughout the research, I intended to create the momentum when the media art performs to be activated, play ‘rhythm of fluid interaction’ with the audience and re-create with them as co-producer.
Technical Aspects: Bodily Interaction / Movement of footage
It Embodies a computer vision-based interaction both visible and hearable experience. It uses sensing technology as Kinect to detect audience movements. The projected wave reacts to the audience’s movements and position in space and generates audio-visual waves through their bodily interaction. Particularly, sound resonates with their footage movement. Since the Kinect is located on the floor, it detects footage of the audience. When they move around and come closer to a certain distance, the sound resonates and circulates through space with 2 speakers using ‘reverb’ and ‘spat’ for sound spatialization in Max/MSP.
There are 2 versions of audiovisual in Max/MSP.
- Audiovisual version 1: Co-creating wave with audience
- Audiovisual version 2: Co-creating realistic wave collage
In version1, when the audience position detected by Kinect is bigger than the number stated below(depending upon the size of the room. I stated it as 200), it resonates with reverbing & spatial sound and visuals. This max/MSP patch is created for an audio-visual installation. It aims to co-create the rhythm of waves by human and multispecies living in the lake. Particularly, the light of wave penetrates through the multi-layered fabric as both spatial and sensory medium. The audience joins to regenerate and interact on the audio-visual waves with their movements through an immersive experience.
In version 2, collage visual with realistic wave formed by human, multispecies, and nature represent how they intermingle together with rhythm. This max/MSP patch is created for an audio-visual installation. It aims to co-create the rhythm of the wave by human, multispecies living in lake, and nature in intermingled collage.
Hearable Interaction: based on Computer Vision
The main interaction of the installation was movement of footage through Kinect Sensor. I've intentionally installed Kinect Sensor in the floor to detect position and distance of footage of audience. Even if the room was dark, it worked well with help of lights coming from the projector.
The distance measured by Kinect Sensor is generating 'Resonating' sound from Max/MSP. The mix of wave sounds from field-recording, not only resonates but also moves around the left to right and vice versa using the 'Spatialization' & 'Reverb' algorithm. It aims to deliver that when the audience gets too closer to a certain distance, it makes reverbing sounds from multispecies. It is the way that multispecies communicate through not only visual but also audio. It talks about co-rhythm and well-balanced distance among human, multispecies and nature.
Response for better interaction
As a better response to the CoV situation, I've changed my interaction from touch-based sensors to no-touch related input for the audience to safely experience my installation. Before realizing it, I had put some amount of time experimenting with a touch-based sensor such as capacity button/painting with light and ambisonic sound with working prototypes in progress. But I've changed to using the Kinect1 sensor on the floor to detect the audience's movement of footage in the space.
In order to cope with the allocated space for physical installation, I had changed my original plan and scaled-down with screen size, number of screens, and the number of speakers into half of the original idea. However, I was more than satisfied with the changed plan and wish to do the original plan for the future.
The Installation and Fabrication Process
To generate the spatial & immersive journey with the audience through installation, I've planned and implemented the whole process for fabrication in advance.
I began creating and also fabricating the installation in the whole step as work in progress. Beginning from the prototype of diverse materials of fabric sample, exploring with light with Kinect, painting(primer > undercoat>topcoat), attaching each woods, stapling the fabric with wood, and finally fabricating woods with 3 screens of the multi-layered structure.
As seeing the interactive installation with the audience, I was very satisfied with it and wish to develop it into a bigger scale in the future with a quadraphonic speaker and surrounding screens with the audience.
Future development
There is more space and room to develop for my installation in screen size, number of screens and the number of speakers for a more immersive experience with the audience in dark room. That was my very first original plan of this project as well. I am satisfied with current installation, and also wish to explore more with natural light as medium and natural architecture as space to embed into media art. In order to do that, I would explore the relationship and interaction among light (natural sun light, LED, and projector), space (created by screens, existing as original architecture) and audience. It aims to create a unique and spatial ways to deliver stories to the audience and give them the possibility of energy in their perception to produce the story to the world as well.
Self-evaluation
I have been very satisfied with the project with all the plans and processes that I had all together with embracing current circumstances. The happiest journey was communicating with the audience through my audiovisual installation in both physical exhibitions and online broadcasting. My main interest explored through light as medium and form as the wave through the project. The aim of delivering the story with 'generating co-rhythm' among humans, multispecies, and nature' through immersive media and space was well done during the exhibition. I learned a lot and was more than happy during the whole process, work in progress, preparing for installation, rehearsal, and degree festival.
Special Thanks
Thanks to Rob Hall for kind helping hands.
Thanks to Jonny and Rupert Earl for great video recording.
References
Sound
1. Experimental Music Programming : Reverb Module
https://ct6exmus.wordpress.com/2015/02/26/reverb-module/ [Accessed 1/07/20]
2. Max/MSP: Spat Tutorial 1 - Introduction to Spat!
https://www.youtube.com/watch?v=3lOI-1_ixDg&list=PLbu4hkj80wD0d3z3ZXyIZNVUwfFSUxYW5&index=5&t=64s [Accessed 1/06/20]
3. Max/MSP: Spat Tutorial 2 - Meet Spat~ and Oper!
https://www.youtube.com/watch?v=856JuxdajAs&list=PLbu4hkj80wD0d3z3ZXyIZNVUwfFSUxYW5&index=10&t=13s [Accessed 1/06/20]
4. Max. Designing your own Reverb (with gen~) (Full Lesson) [Accessed 1/06/20]
Media / Visual
Roberts, M. 0(rphan)d(rift>) archive. http://www.orphandriftarchive.com/ [Accessed 1/08/20]
BBC. Light is a tool / medium. http://www.bbc.co.uk/earth/story/20150731-what-is-a-ray-of-light-made-of [Accessed 01/07/20]
Turrell, J. (2018) Ganzfeld “Aural” [Media Artwork]
Elliason, O. (1993) Beauty. https://olafureliasson.net/archive/artwork/WEK101824/beauty#slideshow. [Accessed 19/05/20].
Elliason, O. (2010) Your uncertain shadow. https://olafureliasson.net/archive/artwork/WEK100100/your-uncertain-shadow-colour#slideshow [Accessed 19/05/20].
Smith, B. (2007) ‘Performing Interactivity’ Digital Performance. Cambridge : The MIT Press, pp. 559-562.
Bio-Diversity: Nature and Lakeside
London Natural History Society. Get inVOLEd by Elliot Newton. http://lnhs.org.uk/index.php/activities/virtualtalks [Accessed 05/07/20]
WWT Slimbridge Wetland Centre. Wetland Conservation. https://www.wwt.org.uk/news/2019/04/15/a-guide-to-pond-dipping-tips-for-a-successful-trip/16722[Accessed 05/07/20]
Krause, B. (2019) ‘The Great Animal Orchestra’ The Musician of Nature by Gilles Boeuf. Paris : Fondation Cartier pour l’art contemporain, pp.11-17.
Computational Arts Aspect
Algorithmic Thinking from Theodoros Papatheodorou, Goldsmiths University
Visual processing (Jitter) and Max organization (Sound reverb, spatialization) from Atau lecture, Goldsmiths University