Five Elemental Mythologies
by Helena Wee
Introduction
Five stories about the development of clean technologies are presented from an elemental point of view. I analyse them from the perspective of a classifier, looking at them as if they are composed of one or more of the five Chinese elements: wood, fire, earth, metal, water. The stories explore the internet, search engines and mobile phones; lead acid batteries and electric vehicles; rare earth elements; Bitcoin; and hydroelectric dams. The work utilises interaction, performance, machine learning and generative graphics to ask questions about how clean and everyday technologies affect the environment and other entities. Throughout the work are references to shan shui painting, tai chi and Taoism exploring relationships between us, technology and nature. The work takes two forms: a performance, and an installation.
Concept and background research
I created a classification system for technology using the Chinese five elements. This was in order to describe, through the elements, particular characteristics of different technologies. In Chinese folklore and Taoism the five elements were said to be the constituents of everything in the universe, and so it follows from this that technical objects would also be made up of different proportions of the five elements. Below is a diagram I made showing the classification of technologies using the Chinese five element system. Each technology is made of one main element plus other secondary elements.
From out of chaos came yin and yang. Balancing these two forces created stable conditions, and their interactions gave rise to the five elements: wood, fire, earth, metal and water. These elements went on to form all things in the Universe, including technical objects, each of which embody the characteristics of one or more of the five elements. It is the balance of the five elements that is of concern in these mythologies. In each case one or more of the elements is out of balance. Perhaps it is only through balancing the elements that we can attain harmony with our environment. This idea has its roots in Chinese folklore and Taoism where problems in people’s lives were diagnosed as being due to an imbalance of elements. It was only through re-balancing the elements that problems could be overcome1.
Wood. Spring
The wood or Spring story tells of ARPANET and the beginning of the internet, search engines such as Google and Baidu, mobile phones and electronic waste mountains. Wood represents beginnings, organic life, growth, femininity and creativity. The branching structure of the internet mimics that of wood and other organic forms. The voices are hybrid multiphonic entities which are part human, part computational, and sometimes part other too. The other could refer to animals, plants or the earth itself. Shan shui refers to the ancient Chinese art of landscape painting, but in these stories also represents the landscape or environment. SinaBai is a rival network to the one Google is from, but also has many similarities to modern day China.
I wanted to look at how many clean technologies are made, and the hidden impact that they have on the environment and those living near their production facilities. The conglomerates referred to are big companies and Capitalism in general. They sell this idea that they can solve the climate crisis, but perhaps this is only true for a privileged few. Those that create these products, so that we can continue our conglomerate-sponsored environmentally damaging lifestyle, are harmed in other ways through the production process or through the excessive electronic waste produced. The wood story tells of the toxic electronic waste mountains growing in rural areas of SinaBai. It also sets the scene for the political background to this story, one of Capitalist competition between different networks and conglomerates.
Fire. Summer.
Fire or Summer tells the story of how lead acid batteries are produced, and their use in the manufacturing of electric vehicles and solar panels. It is the lead extraction and processing cycle for these batteries that causes the most leakage of lead and carbon dioxide into the environment. This in turn accelerates climate change and causes health issues in those living nearby, by getting into the food chain and contaminating the environment. Fire is associated with energy, intelligence and ideas. It can exist in many forms including electricity and lightning, and engenders excessive exuberant energy.
Earth. Midsummer.
Earth or Midsummer talks about rare earth elements. These are found in great quantities in China, who have essentially cornered the market, as has SinaBai in the earth story. Rare earth elements are metals that are used in many clean technologies, such as solar panels, wind turbines and magnets in motors and computer hard disk drives. These metals are called rare not because they are in short supply, but because of their disparate distribution and difficult refining process. They are commonly co-extracted with radioactive metals whose leakage into the land, water and sky can cause cancer in those living nearby. Earth or Midsummer is a period of stability and fruitfulness. Energy is leveled out, and technologies associated with earth could be said to share these characteristics. Earth also relates to metals which are extracted from the earth, such as rare earth elements.
Metal. Autumn.
Metal is associated with Autumn and relates to business, competitiveness, masculinity and material aspects of metal such as conductivity and magnetism. The metal story is about Bitcoin and its mining. There are many Bitcoin mining farms in SinaBai due to their cheap coal-based electricity. This is useful as Bitcoin mining requires a lot of energy. This vast consumption of cheap electricity can be significant enough to release a lot of climate changing gases into the atmosphere as a result. But Bitcoin is useful because it opposes and critiques our broken financial system. A move by Bitcoin mining farms to hydroelectric power could decrease its climate changing effects, but at the expense of using energy that could be used more usefully by others currently using fossil fuel based electricity.
Water. Winter.
Water is associated with Winter. It is a component of anything that flows, and is integral to transport and communication. It can create life but in excess can also extinguish it. Winter is a period of stillness, retreat, quiet and cold. The Three Gorges Dam was built to prevent flooding of the Yangtze River. The build process has uprooted farmers due to their land being within areas that would be flooded to form reservoirs after the build. People often went to work in conglomerates in other areas as their previous farming livelihoods were no longer possible. It has endangered local species such as carp, sturgeon, river dolphins and woody shrubs and plants.
The full text of the stories can be found here3.
Artistic Influences
Influences include shan shui painters such as Ma Yuan4 who painted many water scenes, land art such as Robert Smithson’s “Spiral Jetty”5, Tomas Saraceno’s “Cloud Cities”6, Chinese poetry, and theorists such as TJ Demos who wrote “Against the Anthropocene”. Land artists such as Robert Smithson and Tomas Saraceno’s work “Cloud Cites” led to the idea of creating geometric forms within these elemental landscapes. I chose landscapes which reflected the different elements or seasons. Chinese poetry about nature and the seasons, such as poems by Tang dynasty poet Li Bai, influenced the writing style of the stories and formed the basis of the training data for the lstm model.
Tai chi informed my movements in the performance, but I did not want my movements to be actual tai chi stances, instead choosing to perform in a style only reminiscent of it. I made it a performance which people could drop in and see a section of, then leave to see other exhibits. Sometimes moving the elements was easy and fluid, and at other times it was more challenging. The performance and movements of the elements on screen reflected this, and conceptually this referred to the difficulties balancing the elements as told in the stories. Seeing the whole performance was unnecessary. All I needed was for people to sense a connection between my movements and the movements of the elements on screen. Below is an extract from the performance that I did a few times each day throughout the exhibition.
Technical
Three.js
I had previously used p5.js to create a prototype of this idea for the work in progress show. This time I chose to use three.js7 as I thought the prototype’s backgrounds looked a little flat, and I wanted to create something that was more 3D. Three.js is another JavaScript library which is better at 3D than p5.js. It facilitates the use of WebGL in the browser and allows the use of shaders to alter things drawn within your programme. As I was using the Web Speech API to read the text out loud, I needed to be able to create 3D imagery in the browser, and three.js was the ideal library to use for this purpose.
I had not used this library before, but learnt to use it over the summer months. It had a different structure to p5.js and a more sophisticated 3D camera system and capabilities. As such initially it was a steep learning curve. One initial problem was dealing with cross-origin-resource-sharing (CORS) in the Chrome browser (Chrome had the best support for the Web Speech API). Any images or files that I needed to access in my programme could not be linked to locally using JavaScript and would not show up in the page leading to an error. There were two possible solutions to this. Either start a local server on your computer using python or NodeJS, or serve your files from a host that allows CORS headers to be communicated. As I had access to other servers I chose the latter option and changed the .htaccess file of the server to enable CORS headers8. This allowed me to serve image and other files and use them in my programme.
Web Speech API
I used Google’s free Web Speech API9 to create the computer voice reading the stories. It was relatively easy to implement, but had a few teething problems. It started reading with the default male voice initially instead of the female voice I had chosen. The browser required enough time to load all the different voices before it would read text in the requested voice. I created a hack to allow time for the voices to load. Before the scenes began I would press a key, and this would silently say something and at the same time allow time for loading of voices. So when the scene was loaded the correct voice would read the story.
Another problem was that it would sometimes start before a previous story’s scene had finished. To prevent this from happening I added a
Posenet via ml5.js.
I used the ml5.js10 implementation of Posenet and an ordinary webcam to detect poses. During testing I detected the nose, but in performance or installation mode I switched this to the right wrist. The elemental particles I created in three.js for each scene, would follow the movements of the hand as it swept them across the landscape.
I created a pose history array to store previous poses, so that I could create a 3D Catmull Rom curve from these points for the element particles to follow. This worked well mostly but due to Posenet’s inherent instability in certain light conditions occasionally the right wrist of poses were missed or not recognised. To compensate for this I took an average across multiple entries of the pose history array and created points from both real and averaged right wrist positions. This led to smoother movements of the elements. For each scene I created similar movements for the elements, but with slight differences. For the leaves in the wood scene I made them turn according to the gradient of the curve so it would look like they were turning with the wind. In contrast for the flames of fire and the waves of water I kept the elemental particles pointed upwards or aligned with the horizontal. This was to make the movement look more typical of their respective elements. Each set of elemental particles was given a random spread across an area on instantiation, so they would look like a floating group of leaves, waves or flames etc. For fire I varied the opacity of the flames to make it look like they were flickering.
When in installation mode I noticed that although Posenet was set to recognise only single skeletons, sometimes it would have problems tracking them if more than one person was in the frame. So I had to create a basic method for tracking people so as to correctly create a pose history for users interacting with the piece. The nose was generally a more stable feature to track. I distinguished people using the distance of their noses from its last tracked position, then used this skeleton’s index as the one from which to obtain the right wrist position. This meant that when multiple people were in the frame, the one whose right wrist had been tracked the first generally ended up being in control of the elements until they went close to the other person or left the webcam frame.
LSTM and knn classifier
I created a sentence level LSTM in python using keras11. I trained it on many public domain translated Chinese poetry texts, having first created a program to clean the texts into a format which could be used for training. For this I used some natural language processing to remove punctuation and unnecessary spaces and numbers, word embeddings to make the model smarter, and created some seed texts from the data in the style of different seasons to generate new text. I slowly improved the model using more iterations across the data and more layers. The results were sometimes very good, but sometimes more nonsensical, and I curated a selection of the best texts to use within the artwork.
I programmed a knn classifier using ml5.js, and trained it to recognise certain poses. The model had to be of the same resolution as the output animation, so that recognition of poses could be more accurate. The poses I trained it on were of both my arms outstretched to the left, or to the right. When these poses were detected by the json model, a few lines of machine created poetry from a json of curated best lines would appear at the top left corner of the screen.
Animated graphics in three.js
In the wood story blossoms (or voices) slowly appeared through the use of buffergeometries and opacity settings. I created a trefoil group class for the flying radioactivity symbols. These would grow in size as they slowly moved and got closer to the camera. I also deformed some imported red blood cell models using a 3D Perlin noise shader12 with the noise code obtained from a standard library13. I used three.js implementations of the sun and sky14 shaders in all scenes, and a lightning15 strike ray in the Summer scene.
When the elemental particles in each scene go over background textures their uv coordinate mapping is disrupted making the images warp. This is determined only where there is a pose. The textures had been mapped to roughly semi-spherical meshes to create a 3D diorama or panorama effect which looked more like a landscape stretching out into the distance.
Setup of installation
I ended up using 2 computers, simply because I did not have the right adaptor to change vga to hdmi. One computer was for the projection mapped Chinese lantern in the window. I used a long throw projector on a shelf attached to the far wall opposite the window for this. The other computer was my laptop, which I decided looked better for the performance and would perform better if it was only attached to one projector. I sent an hdmi cable from my laptop to the ultra short throw projector attached to the wall. I attached some speakers to the ultra short throw projector’s sound output (as it was getting sound via the hdmi cable) and placed them on the floor just in front of the left radiator. The webcam for the installation was on top of the ultra short throw projector. The webcam for performance mode I attached to the top of my laptop, and I would only use one webcam per mode, detaching the other from my laptop’s usb port when not in use.
Making the screen and testing it was quite challenging due to the fragility of the tracing paper, which tore on one occasion. I then had to make another smaller screen with a predefined width and height calculated using the ultra short throw projector’s throw distance as specified in its manual.
Future development
I would like to create more graphics for each scene that unfold as the story unfolds. It is fun thinking of ideas for graphics, such as the flying radioactivity symbols. More of these graphics would have made the scenes more interesting.
I think it would be good in the future to create better seed texts for the lstm by assigning individual lines to particular seasons, instead of just assigning a verse to a particular season. I would also like the computer to create Chinese style poetry live and on the fly instead of just using a curated selection from a database.
Making the 3D aspect of the scenes more visible by creating some parallax movement within the scene would emphasise this more. Perhaps the movement might not occur on gesture, but increment very slowly as the scene progresses.
In terms of the knn classifier it might be good to use a different gesture for each element. Following on from this I would like to develop the performative language more, to bring more meaning to the piece.
One aim that I did not manage to do this time would be to create an installation version that allows for multiple users and multiple element streams on the screen. This is very possible but might need altering of the elemental particle stream effect to be more visible in the current setup.
Self evaluation
The interaction was mostly good, but in low light conditions when other works were not on, it was a little jittery. I would like to look at this further and see if there is more that I could do in terms of averaging pose history positions to smooth the movement out more. When two or more people were in front of the screen in installation mode, although it settled on one person most of the time, it would sometimes take a bit of time for it to do this, and perhaps the tracking of people could be improved further by including eyes in the pose history positions taken into account.
Occasionally the graphics would not show up due to a loss of context event due to the graphics card being occupied with other duties. In retrospect better long play testing could have alerted me to this problem before the exhibition. I would guard against this in the future by re-initialising all the graphics within the programme or refactoring the programme so that the init() function could be re-run easily and cleanly. A better speaker might have made the sound better, and this is something I would use if I exhibit it in the future.
On the plus side my installation was stable and worked well most of the time in both performance mode and installation mode. Also there were no errors in the code. I think it put across its message well through the spoken story and the elements slowly distorting the landscape. It highlighted the hidden cost of current so called clean technologies well through the story and the destruction it created in the landscapes on interacting with it. It also created the sense of a harmonious interaction with the landscape (through tai chi like movements) not being as peaceful as it initially seemed.
References
1. https://helenawee.com/elements/fiveelementclassification.html
2. Hamilton, Patricia L.; "Feng Shui, Astrology, and the Five Elements: Traditional Chinese Belief in Amy Tan's The Joy Luck Club.", MELUS, Volume 24, Issue 2, pgs 125-145, June 1999.
3. https://www.helenawee.com/elements/fiveelementalmythologies.html and theory references for the stories are here: https://www.helenawee.com/elements/references.html
4. https://www.faena.com/aleph/articles/a-study-on-the-properties-of-water-by-ma-yuan/
5. https://www.diaart.org/visit/visit/robert-smithson-spiral-jetty
6. https://www.metalocus.es/en/news/tomas-saraceno-cloud-cities
9. https://w3c.github.io/speech-api/
12. https://thebookofshaders.com/
13. https://github.com/ashima/webgl-noise/blob/master/src/classicnoise3D.glsl