machinesMemory
Produced by: Dongyuan Liu
Introduction
‘The ancient Greeks made a distinction between two kinds of poesis, i.e. two kinds of ways in which things come into being. The first, labelled techne, refers to that which is made by humans; the second, labelled physis, refers to that which is made by nature’.
—Cited by Morten in ‘Poesis of Human–Computer Interaction’
‘machinesMemory’ is a multi-screen art installation that presents the idea of a machine’s memory. Neural network algorithms bring with them new possibilities for artistic practices. Artists consider them to be independent entities that generate artworks beyond the realm of human control. The quote above presents the idea of the interconnectivity between objects created by humans and nature, and it led me to consider the connections between nature and humans along with nature and artificial intelligence-based machines. Building on this inspiration, this experiment comprises a preliminary exploration of the aesthetic values of machine learning algorithm constructions that includes, but is not limited to, the ontology of AI-based machines and the beautiful forms or patterns taken by machine-generated data. I want to emphasize that this project does not focus on the methodology of constructing a machine learning algorithm, such as in natural language processing, to generate a poem. Instead, I will focus on using machine-generated data as material from which to create something new. The end of this paper considers using screens as structures with which to build a system that gives virtual machine-generated data a ‘body’ to simulate the machine’s memory.
Keywords: Multi-screens installation, intelligent agent, machie generated data, AI culture, image process, machine learning algorithms
Theoretical Background
The independence of AI machines
Both the terms ‘autonomous agent’ (Davidsson) and ‘intelligent agent’ (Klicek) indicate that AI agents should possess the ability to sense the environment via external sensors, to make decisions based on internal mechanisms, and to respond to the environment around them. Additionally, Blackwell presents four attributes of an entity’s autonomy, whether it be human or machine. He especially focuses on the property of autonomy, which suggests that autonomous agents should be uncontrollable, unpredictable, and nonpredetermined (Blackwell).
The effects made by data-visualization
Data visualisation can reflect the information contained in a massive dataset by generating a graphic. However, from the viewpoint of artistic practice, it should not be used as it would for a business report. For example, a business report can reflect how many products were sold over the past several months and whether sales are up or down. Artists, however, would prefer to use meaningful data to produce a beautiful pattern or image that has a sense of order. For them, it is about how they understand the data (Evers et al.).
Screens function as structures
In Ebsen’s dissertation, he argued that screens as materials possess their own materiality, which he defined as window, depth, access, disembodiment, immobility, and dematerialisation. From substances to forms, he considered screens as entities, giving them a larger range of semiotically expressive possibility in the artistic context. As reviewed above, there is a transition from seeing screens as media (i.e. the information carried by them is the main focus of artists and audiences) to seeing screens as components of artefacts from an ontological perspective, meaning that the creative uses of screens have been considered by many people.
Working progress 1
To emphasise the independence of machines, I collected 1,000 photos from the photo albums of my friends’ mobile phones that represent the idea of memory in an abstract way. I used this dataset to train the styleGAN deep learning model. After nine hours and the 1,000-step training, an abstract consequence resulted (the second image in album). Features from the original training set can be found in some machine generated images, but other parts of these generated images are rather abstract. Most of the images in the training set were landscape photos, hence some are recognisable (e.g. Italian landscapes); machines can certainly not possess such memories at the conscious level. However, from the perspective of ontology, we can imagine that these ‘memories’ do exist in some way.
Working progress 2
The first method I considered was to use the colour values to construct a virtual space (showed bottom) in which audiences could ‘walk around’ to experience how the data appears visually. However, a problem with this method was that the visual information did not mean anything. Manovich described machine-generated data as a ‘black box’ (Manovich, 2019. P.12); we do not know what the meanings of the numbers behind the images are. For example, in the data visualisation projects we have seen, artists collected data about, for example, body movements, weather information, and bio information to create visual patterns. In such a case, the artist knows which number represents the left hand, and which number the right. However, as for machine-generated data, we only know that there is a 500 x 500 number matrix.
Although these data cannot be understood actively, I was thinking if there is way that can help us understand them. Figure 4 represents my second attempt, where I mapped the machine-generated data to a real landscape picture by calculating the similarity values between the machine-generated images and the cropped real images. I did this because humans and machines are often compared in the field of artificial intelligence research. Therefore, I selected a landscape image from our real world and cropped it randomly. After comparing 3,000 machine-generated images with the cropped real image, I selected the most similar one and arranged it with the same position and size as the cropped one. Four hours of calculations finally yielded the final collage image. The consequent image does look interesting, but I did not achieve the expected effect. I was thinking if I can get more clear meaningful data directly in a machine’s way, like image process algorithms, which allow machines deal with tasks by themselves.
The image with red points shows the results of the feature tracking image processing algorithm. The red dots were extracted by an algorithm in OpenCV called goodFeatureToTrack. It seems like this yielded more meaningful data. The featured points can be thought of as analogous to feature information in human memories. For example, people tend to capture feature data when they remember a photo instead of remembering every pixel. After obtaining these feature points, I arranged them based on their brightness and hue values. I then simply combined the feature data. The resulting massive visual effect, however, was incomprehensible, which means that machine-generated images cannot be meaningfully differentiated by human viewers. Finally, I spilt the feature points extracted from machine-generated images into different groups and assigned them an index under every ‘sample’. I refer to this as ‘a machine’s way’ because humans are unable to extract these feature points actively; they were extracted by machines themselves, but we know what those data represented.
Working in progress 3
After obtaining a satisfactory result, I contemplated how to connect the various parts to present the idea of ‘memory’, which I intended to be nonlinear and complex. According to the method of using screens I described in section three, I used three screens to function as the different areas representing nonlinear memory. Every machine contained thousand different generated images that resulted from training based on different values for the parameters of the machine learning model. The bottom one received the extracted colour information by open sound control and arranged these data according to the aforementioned rules. Furthermore, the cables were used to connect the first and second parts. Behind this projects, I still did a lot of tests and researches which can be found in my study blogs during the whole year.
Self-evaluation
During the second year, I paid much attentions on the thoery setudy which can help me condiser my working process deeply. Trying to understand the machine generated data maybe still difficult. From the technical side, I had tried my best to understand how the machine learning algorithms help computers study. From the artistic perspective, I followed the art theory to build my entire project step by step, and I am satisfied with the final projects outcomes. I know how can I make a stronger connection between artistic concept and practice.
Future Research Areas
In this project, I did not focus on what types of techniques I can use, such as machine learning algorithms. It is painful to me who does not have comupter science background. However, machine learning or AI as one kind of media, exploring the media malleabiliity of AI seems like an interesting research for computational artist. For example, I always emephasize the independence of AI machines and considering machine generated data as materials in this project. As an artist, I should not only care about what artificial intelligence can do; I still need to understand what it is.
Code:
- Image similarity: https://zhuanlan.zhihu.com/p/68215900
- Running openframeworks on raspberry pi: https://openframeworks.cc/setup/raspberrypi/
- pix2pix machine learning algorithm: https://www.tensorflow.org/tutorials/generative/pix2pix/
- Tensorflow Tutorial: https://www.tensorflow.org/
Reference：
Blackwell, Tim et al. ‘Live Algorithms: Towards Autonomous Computer Improvisers’. Computers and Creativity, edited by Jon McCormack and Mark d’Inverno, Springer Berlin Heidelberg, 2012, pp. 147–74. DOI.org (Crossref), doi:10.1007/978-3-642-31727-9_6.
Davidsson, Paul. Concepts and Autonomous Agents. p. 137.
Klicek, Bozidar. ‘Intelligent agents. International Encyclopedia of Hospitality Management, edited by Abraham Pizam, Routledge, 2nd edition, 2010. Credo Reference, https://search-credoreference-com.gold.idm.oclc.org/content/entry/esthospitality/intelligent_agents/0?institutionId=1872. Accessed 07 Jun. 2020.
Ebsen, T. (2013). Material Screen (Doctoral dissertation, PhD Dissertation).
Evers, Lucas, and Frank Nack. ‘Data Aesthetics: The Ethics and Aesthetics of Big Data Gathering Seen from the Artists Eye’. Proceedings of the 2016 ACM on Multimedia Conference - MM ’16, ACM Press, 2016, pp. 779–80. DOI.org (Crossref), doi:10.1145/2964284.2993205.