The integration of multimedia art and performance
How computer art can be combined with actors' performance in stage design so that computer art can also become the main body of stage display. However, within the scope of my exploration, the machine can't perform entirely autonomously. It needs to capture the movements of the characters and make calculations, and finally give unpredictable results. I did two works with P5.js and ML5.js to show the impact of changes in emotion and cognition on life and history. The first Danudice is a generated computer vision display based on the dance. The second Foxudice is an interaction with the audience, which has some scientific popularization and educational significance.
produced by: Haotong Qu
What are the changes in the application of multimedia technology on the stage?
Art and technology combine to create works. Art is still the core, and technology is a tool to realize the imagination. In the past, many ideas would have been shelved due to technical constraints. Today, the development of various technologies can meet the realization of more ideas. Technology has expanded the boundaries of natural space and even created new spatial dimensions. Art leads technology, and technology drives art.
In stage creation, the introduction of multimedia technologies such as 3D Mapping and real-time interaction and the coordination and connection with special-shaped projection, ball screen, and various stage devices have changed the previous creation process of plane images.
The application of virtual reality (VR) and virtual, augmented reality (AR) technology expands the stage's space in vision, endowing more possibilities for visual creation. At the same time, it puts forward higher requirements for multimedia designers, whose content not only involves image creation. It also covers interactive programming, artificial intelligence, virtual/augmented reality, drones, robotics and broader multimedia technologies.
The application of multimedia technology in the stage has produced some new changes and trends.
Analysis of the works of artist Tobias Gremmler
Tobias Gremmler is a new media, motion graphics, and interface designer visual artist based in Munich, Germany. At the core of Tobias' work is motion capture technology, which uses tracking devices at critical parts of moving objects to identify movements through changes in the position and Angle of the tracking device. Several cameras capture the movements of real actors and then restore these movements and render them onto the corresponding virtual characters. Through digital art, the works are endowed with distinct visual styles.
Tobias Gremmler's works integrate new disciplines, new technologies and new trends into the design of drama. Technology mainly serves artistic expression. He believes that each medium has different degrees of intimacy, which determines the freedom of individual interpretation. When you read a book, all the sounds and images come to your mind. They are created through the imagination of the reader.
Therefore, the element because encourages the recipient to use their imagination to complete it.
1) KONGXUELAOFENG
The work tells a story rather than painting a too specific picture. Visual effects should capture the viewer's imagination, not be consumed. As a result, the visuals are often incomplete or hide something that needs to be imagined. For example, dragons are invisible and are covered in smoke. But when its motion is expressed through the turbulence of smoke, one can imagine the action behind it. Smoke becomes a medium, abstracting a scene and spreading it out into a more expansive space. Similar methods are used on bamboo and stones. These elements act as a medium for transforming the underlying scenario. That is a multimedia performance integrating video, costume, dance, drama and installation.
The "emotions, philosophies and memories" drawn by the "wind" on the stage lead the audience to a spiritual journey of Orientalism aesthetics. The performance tells the journey of the invisible wind in the visible world. Nine dancers use dance to interpret the people from the prehistoric world and the natural and artificial disasters in the chaotic world. The two Spaces between people and digital design merge into each other to form the whole of the performance. Computer art forms a part of the stage in this project. It is alive and can be transformed.
(2)Anna Monteverdi
Anna Monteverdi works about facial visuals, in which Tobias Gremmler's face is mapped onto geometric shapes. To some extent, the technique is similar to projection mapping but not in real-time. He wanted to extend the visual effect beyond the shape of the head, which was impossible when projecting onto the face. But the combination of man and machine produced a Seborg-style art that allowed visual effects to change autonomously through machine learning.
Project:
(1)Danudice
I want to show a project of interaction between projection and dance in which people's relationship is separated due to emotional changes. The whole dance wants to express the changes of people's emotions when they get along with others in their lives and shows the ups and downs of emotions following the music. Continuous triangles appear from the junctions and gradually disappear after they leave. The colours also begin to change, indicating that the relationship and distance between the people around us and us will vary with the change of our emotions and may eventually be forgotten by us. I used the webcam and Postnet in ML5.js to dynamically capture several bone points of the human body to control and use Delaunay to paint.
(2)Foxudice
The meaning of an object is defined by society, which may entirely reverse this definition with time. However, the defined object itself has not changed much, and it just carries people's life intention. These definitions may lead to biased cognition and judgment of people or things by Preconceived notions in life. I hope to do an interactive projection project to show that the definition of society or others should not influence us to judge things themselves and eliminate prejudice.I designed this project with the changing image of the fox in Chinese culture as the theme to show viewer this opinion.
I designed to control the position change of the fox with the movement of the viewer's face. When the mode is friendly, humanoid particles will gather at the fox's location; when the mode is evil, the image of the fox will change to express the change of the image of the fox in people's mind. At this time, humanoid particles will move away from the fox.
In this process, the visitor represents the fox. The change of the model is controlled by the sound capture of other people's voices in the environment. The participant does nothing but the sound in the environment causes the change of his image and And the relationship with particles. Concerning the position movement, PoseNet in ML5.js is used to capture the position of the nose through the webcam and change the mode through voice recognition.
Reference list:
[1] Diezmann, T. and Gremmler, T., 2003. Grids for the dynamic image. Crans-près-Céligny [Suisse]: AVA.
[2] DiMarco, J., 2010. Digital design for print and web an introduction to theory, principles, and techniques, Hoboken, N.J.: Wiley.
[3] Gremmler, T., n.d.2008.CyberBionic. Spektrum Akademischer Verlag,.
[4] Gremmler, T., 2014. Creative education and dynamic media. Hong Kong: City University of Hong Kong Press.
[5] GitHub. 2021. darkskyapp/delaunay-fast. [online] Available at:
[6] Lecun, Y., Bengio, Y. & Hinton, G., 2015. Deep learning. Nature, 521(7553), pp.436–444.
[7] Muhamad Syariffuddien Zuhrie et al., 2017. Design of an Artificial Intelligence Robot as a Tool for Teaching Media Based On Contextual Teaching and Learning. Jurnal pendidikan teknologi dan kejuruan (Yogyakarta), 23(4), pp.369–373.
[8] Openprocessing.org. 2021. Browse Sketches - OpenProcessing. [online] Available at:
[9] Romero, J. et al., 2020. Artificial Intelligence in Music, Sound, Art and Design 9th International Conference, EvoMUSART 2020, Held as Part of EvoStar 2020, Seville, Spain, April 15-17, 2020, Proceedings 1st ed. 2020..,
[10] P5js.org. 2021. reference | p5.js. [online] Available at:
[11] Shan ben tu shu, editor, publisher, 2015. Spectacular! : stage design.
[12] Vimeo. 2021. Chinese Theater and Media Scenography. [online] Available at:
[13] Vimeo. 2021. Concert Visuals for Hacken Lee. [online] Available at: