The Immersive Experience
Amit Segall
Modern technology has a major role in shaping modern culture. Different art practices embraced the new technologies in creative new ways. Among the different practices one idiom stood out suggesting something new - The Immersive Experience. As an artist I wanted to understand whether it is a cultural trend [1] or an artistic practice. In this paper I will try to define what is an immersive experience and how it relates to audiovisual artworks. I will suggest a framework to analyze audiovisual artworks and question their relation to immersive technologies [2]. The lack of tools to asses these works causing problems defining these types of artefacts, including often confused or misused terminologies. The ability to break down the key factors in different artworks will allow artists, scholars and critics to understand both the technological and the cultural aspects. This will also serve as a guideline for artists as myself to conceptualize new immersive artefacts in the future.
The Arts & Humanities Research in UK generalize immersive experiences stating: “Ubiquitous high-speed telecoms networks, extremely capable mobile devices and new generations of display technology will together enable the creation of new immersive experiences [3]”. This states that by combining new technologies together, an immersive experience emerges. Projects such as The Immersive Pipeline are trying to differentiate the technologies from the immersive experience itself. The project explores the experience in relation to space and body, as opposed to using dedicated hardware “While VR headsets are able to provide the simulation of total immersion by delivering high resolution stereoscopic image unfettered by environmental conditions, it is those very environmental aspects that are lacking – a sense of physical space, a sense of embodiment, and a sense of the social. In this sense, VR in the classical sense is an isolated experience for the user [4]”.
As I was exploring the usage of immersive technologies in audiovisual artworks, I realized that many definitions refer to digital media and immersive experiences interchangeably. While immersion could be understood as being absorbed in matter or alternatively being surrounded by matter, it seems that almost everything could count as an immersive experience. The emergence of new technologies such as 3D image and audio, AR and VR allowed artists to mutate cultural practices and became a crucial part in designing immersive experiences. It created codependent relationship that nurtured one another. Nowadays artists push boundaries and experiment with these new technologies that enable creators to produce artefacts that were impossible in the past. “Artists no longer sit on the sidelines eventually to become grateful users of borrowed tools but have become active in development, creating a disturbance in the field with new contingences [5]". This led to exploration of new formats and platforms for art and design such as: audio-visual spaces, Virtual Reality, interactive experiences and more. Dedicated headsets and computers currently required to develop and explore immersion in virtual or augmented reality. Other technological advancements allowed artists and researchers to develop entire environments based on perceptual audiovisual tools such as interactive lighting, projection mapping and spatial sound [6,7].
As part of my process to formulate a method to explore this subject, I will examine key features of an audiovisual immersive experiences such as engagement in three mediums: visually, sonically and physically. I will also examine the environments that format an audiovisual landscape, where a participator is able to immerse in the experience. Based upon different terminologies I have encountered in the different fields of immersion [8,9,10] I will define immersive experience as follows:
Immersive experience is a subjective impression towards a site specific, experimental interactive audio-visual space. It holds a fine line between submission and engagement. It is not only surround you but also affects you emotionally and makes you a part of it. It carries the promise of a unique multi-sensory experience that strive to be cutting edge and dynamic.
Using Actor-Network-Theory [11] as a basis for a method to analyze immersive artefacts, I will examine the relationship of artists with technological developments as a major actor. Similar to ANT, I will review experiences through a set of three main principles I defined: Environment, Body Position and Relationships. Each principle will explore different aspect evaluating its technical and cultural relations and the artist as a connecting agency. Environments will reflect on the space the experience is situated in, the location that facilitate it and the new space that emerged using the technology. Body Position will explain the different viewpoints that a participator goes through in immersion and how the experience is being reflected by immersed bodies. Relationship offers to explain the conflicting or intersecting viewpoints that emerge culturally and technologically while creating the immersive experience by exploring the artist’s view.
I will put my method to the test and study the different aspects of two audiovisual works I have visited in the past 6 months. The works are: test pattern [N°12] by Ryoji Ikeda (London, Oct’ 2017) and SKALAR by Christopher Bauder and Kangding Ray (Berlin, Feb’ 2018). Both artifacts described as audiovisual site-specific installations and performances. My analysis is based on information found over the artists and curators websites (statements and technical details) and based on my personal experience [12,13,14].
test pattern [N°12] was exhibited in 180 The Strand in central London, a unique space that hosts exhibitions and different creative companies. The studio that held the installation was a dark room transformed into what was described as a highly-charged kinetic environment. The space was 17.6m long and 5.5m wide. Barcode like patterns displayed as a floor using two projectors and six speakers surrounded the room with sound. Each iteration of the installation (started in 2008) responds to the specific context of the space and the visual floor reacted to the generative sounds. Unlike other iterations of this work ([times square] or [100m version]) the size of the environment that emerged or the transformation of the space were not obvious actors in the work in my opinion. Before entering space, the spectators were asked to take off their shoes. By doing so I was able to walk over the projected surface and explore the space freely. Some of the spectators, in order to immerse in the moment, lay on the screen, looking at the flickering projectors while others set or walked around the space. The visual aspect was constantly engaging with your vision, causing disorientation. The sound that immerged was incoherent and worked well with the visuals. The inconvenience I felt in my body reflected both the audio and visuals. The scale of space, in relation to the body, allowed the audience to submerge to the experience and explore it from different locations. In this edition test pattern [N°12] described by the artist as a sequence of tests for machines and humans. The visual patterns converted and generated from sound waveforms in real–time using a dedicated computer software. The intensity of the work attempted to bridge the worlds of art and music by challenging fundamental preconceptions about esthetics. There is a direct relationship between the experience itself the artist’s statement. As I was being immersed in the space experiencing the audiovisual patterns I was mesmerized by an esthetic I never encountered before. Because of the disorienting environment, my endurance was indeed put to the test both physically and mentally. This type of experiment is only possible through the usage of immersive technologies that allowed the artist to successfully created a new space to examine the visitor’s reaction.
SKALAR was exhibited in Kraftwerk, Berlin, a former power plant that was built in the 1960s. The building was abandoned for years and later transformed into a nightclub and today sets as an event hall. In the piece, light was treated as a solid material that can be sculpted and shaped to architectural dimensions (45m x 20m x 10m) and by thus completely transformed the interior of the space. Both the exterior and interior of the space enriched the overall experience because of the unique architectural form of the building. The exploration of light in the surrounding shed new life to the old environment. Before entering inside the space, the spectator could already hear ambient sounds. After arriving to the main hall, a dynamic light sculpture constantly moved, and spectators could hear the music that was being played. While I moved in space I was able to detect different light structures from different locations. Many participators were laying on the floor or sitting down exploring the installation. The sound was constantly changing, working in perfect harmony with the lights, together they told the story of the piece. SKALAR also included motorized mirrors accompanied by LED circles that produced kinetic movements, enhancing the fluidity of light in the space. I felt that as a body in the environment I engaged mostly with my eyes. The scale of the work allowed me to stay in one place while allowing the kinetic movements to lead the experience. The work aimed to explores light and darkness as endless cycles of day and night which define our perception of time and influence our emotions. The kinetic movement of the mirrors enhanced the cycling between the different lighting environments. I felt that the sonic environment did enrich the experience but there was no strong relationship with the artists that attempted to define our perception. The project was a collaboration of the artists with leading technology companies that provided custom made solutions to the project. The artists relayed on the overall experience to highlight their work and used the technologies as their tool of trade. This immersive experience not only dependent on these specific tools but is also dependent on the space to recreate the overall experience.
After breaking down each component in both installations I can conclude that both experiences are immersive. These works bring multiple actors including the participators, artists and technology into a shared experience. The usage of my framework highlights the different threads of relationships between the actors. Both works relay on new technologies to transform different spaces to new environments where the body is an agency for the experience*. The technology was a leading actor in creating the immersion and sometimes undermine what the artists were trying to voice. test pattern [N°12] tried to engage with the audience by creating an endurance test, proving that machines outgrow our abilities. The artist changed the way he explored our endurance from the original version to [N°12] that I have explored. I think the addition of immersive technologies to the work empowered the artist’s voice and still maintained a good balance between the different actors. In SKALAR, by examining the different perspectives, I was able to restore the balance between the actors and shift the focus back to the artist and the participator. The choice of Kinetic Motors to reproduce cycles in being justified when examining the artists statements about the work. I felt that the questions that were asked were not reflecting in the audience experience, and mostly produced a meditative immersive space using the kinetic movements. In my opinion the sonic atmosphere did not examine our perception of time or had a relationship with darkness and light.
*Both works had a performance, where the artists were presenting their work as a live performance. In both experiences the space and the body will transform differently, in a more exaggerated form that enhanced the overall immersion as opposed to the standard installation.
After understanding the backbone of audiovisual immersive experiences, I will use my method to conceptualize a small scale immersive experience. It will be called The Vacant Box and will explore the psychoacoustic environment as a tool to enhance communication with digital AR agencies in a new emerged virtual space. The work is also scalable and transferable, and I’ll be able to develop it in different environments for different needs and explorations.
My experience will be constructed from a black box booth (2m x 2m x 2.5m) from either cardboard or thin plywood. The interior of the box will be colored black and an entrance to the box will be with through a black curtain (one of the 4 walls). Inside the black box there will be two small speakers (one on each side of the box) covered / sunk in the wall (hidden), and 3D sound will immerse the participator. In different locations of the box LEDs will lit small AR markers and participators will be able to engage with them using a tablet with dedicated software. The usage of sound and small LEDs will lead the participators where to look in the box and explore the new emerged space. As I was conceptualizing The Vacant Box I reflected my method examining the Environment, Body Position and Relationships. This might consider my work as an immersive experience.
The environment that emerges has no dependencies of space that facilitate it. It creates a new space that is easy to transform and translate that maintains the same immersive experience. The Body Position and the movement in the space are crucial part of the experience, viewers have to engage their senses, curiosity and be active. The psychoacoustic audio that enhance the visuals enforce your body to position itself and helps defining the space. There are direct relationships that establish my concept and the usage of immersive technologies. My exploration of the environment and our communication with AR agencies in the new space was impossible without immersive technologies. It relays on the usage of immersive technologies such as 3D sound and AR. The immersive experience that arise from this installation could be experienced alone or in a group that also interact with one another. In the past, immersive experiences were trying to mimic real environments, new technologies enabled artists to completely transform spaces and create new surroundings for viewers to explore [15]. What started as a research behind a buzz term [16] revealed many problems defining this trend and analyzing it. The concept of experience economy [17] has been around for almost two decades and it is more obvious than ever that the modern culture is being pushed towards the immersive experiences. “As businesses sought to adapt to an emerging neoliberal paradigm, economic production had to contend with a new kind of consumer and a new personalized and experiential forms of consumption in an expanding ‘experience economy’ [18]”. My research led me to believe that immersive experiences is a term being used indiscriminately and understood differently by a range of audiences and practitioners. While at first, I thought that these experiences are an evolution of an artistic practice that combines multidisciplinary approaches, now I think, that they are an inevitable future caused by social changes. Some of these experiences do not channel a voice in new art medium but a voice in commercialism. As Douglas MacLeod wrote in his book Immersed in Technology: Art and Virtual Environments in 1994: “most worrisome of all, a new approach to cultural initiatives suggest that all explorations must have a commercial or revenue-generating potential. While the effects of these pressures remain to be seen, it is very possible that this book documents a project that could never happen again [5]”. I have created my own definition to the immersive experience that bonds a connection between audiovisual practices and the usage of immersive technologies. My definition allowed me to suggest a method, similar to The Walkthrough Method [19], that relays on Actor-Network Theory to examine human relationship with technology. This allowed me to breakdown audiovisual works in the eyes of the immersive experience. Since audiovisual artwork practice is diverse, as a framework to analyzing existing works using my method might fail to produce coherent conclusions between different works. The absence of proper terminology in both fields of immersive technologies and immersive experiences causes often confusion. I think my method proves itself as a guideline to make new audiovisual works that interact and engage with audience by taking into account its three main principles. I could use this in my practice while producing artefacts that strive to be immersive experiences. I hope that future cultural researches will be made around this emerging field, defining proper terminology and studying these technological and cultural connections better.
References:
1. Vankin, D (2016, Dec 22). Los Angeles Times The arts buzzword of 2016: 'immersive' http://www.latimes.com/entertainment/arts/la-ca-cm-immersive-arts-20161225-story.html
2. Kovács, Péter Tamás, Niall Murray, Gregor Rozinaj, Yevgeniya Sulema, and Renata Rybárová. "Application of immersive technologies for education: State of the art." In Interactive Mobile Communication Technologies and Learning (IMCL), 2015 International Conference on, pp. 283-288. IEEE, 2015.
3. AHRC (2017, July) Research and Partnership Development call for the Next Generation of Immersive Experiences. http://www.ahrc.ac.uk/documents/calls/immersive-call-guidance/
4. SONICS Research Special Interest Group (2017) Immersive Pipeline. http://sonics.goldsmithsdigital.com/immersive-pipeline/
5. Moser, Mary Anne., Douglas. MacLeod, and Banff Centre for the Arts. Immersed in Technology : Art and Virtual Environments. Cambridge, Mass: MIT Press, 1995. 6. Cruz-Neira, Carolina, Daniel Sandin, Thomas Defanti, Robert Kenyon, and John Hart. "The CAVE: Audio Visual Experience Automatic Virtual Environment." Communications of the ACM 35, no. 6 (1992): 64-72.
7. Weissig, Christian, Oliver Schreer, Peter Eisert, and Peter Kauff. "The ultimate immersive experience: panoramic 3D video acquisition." In International Conference on Multimedia Modeling, pp. 671-681. Springer, Berlin, Heidelberg, 2012.
8. Machon, Josephine. Immersive Theatres Intimacy and Immediacy in Contemporary Performance. 2013.
9. Dede, Chris. "Immersive interfaces for engagement and learning." science 323, no.5910 (2009): 66-69.9
10. Cummings, James J., and Jeremy N. Bailenson. "How immersive is enough? A metaanalysis of the effect of immersive technology on user presence." Media Psychology19, no. 2 (2016): 272-309.
11. Tatnall, Arthur, and Anthony Gilding. "Actor-Network Theory in Information Systems Research." (2005): 42-46.
12. CTM festival, SKALAR Installation & Performance by Christopher Bauder & Kangding Ray (2018). from http://www.ctm-festival.de/festival-2018/specials/skalar/
13. Ryoji ikeda, test pattern (2008). http://www.ryojiikeda.com/project/testpattern/
14. Spice, A (2017, Sept 10). The Vinyl Factory Ryoji Ikeda premieres mind-bending new A/V artwork test pattern [N°12] at The Store X. https://thevinylfactory.com/news/ryojiikeda-new-test-pattern-n12-store-studios/
15. Mitchell, Bonnie. "The immersive artistic experience and the exploitation of space." In Proceedings of the 1st international conference on Ideas before their time: connecting the past and present in computer art, pp. 98-107. BCS Learning & Development Ltd.,2010.
16. Petronzo, M (2013, Apr 25) Mashable UK Immersing Consumers in ‘Immersive Experiences’.https://mashable.com/2013/04/25/immersiveexperience/#nVVYt00JSEq3
17. Pine, B. Joseph, and James H. Gilmore. "Welcome to the experience economy." Harvard business review 76 (1998): 97-105.
18. Alston, Adam. Beyond Immersive Theatre. 1st Ed. 2016 ed. Palgrave, 2016.
19. Light, Ben, Jean Burgess, and Stefanie Duguay. "The walkthrough method: An approach to the study of apps." New Media & Society (2016): 1461444816675438.