Systematic Spontaneity

Systematic Spontaneity is a computational process art wall painting created by applying Systems art approaches developed in a fine art context to computation. 

By: Laura Traver 

Introduction

Systematic Spontaneity uses blob detection to assess the proximity of programme-generated predictions to my attempt to hit the prediction by throwing paint at predicted targets. The predictions are represented by black circles in the painting and are also referred to in my programme and throughout as the grid or grid targets. In line with my fine art Systems work, the essential movement of the grid points, and, therefore, my throws, is from the left to the right across the wall.

The fundamental process of work is that the programme sets a prediction target which is projected onto the wall. Using image transfer medium, I then transfer a printed black circle onto the wall where the computer has predicted the next target should be. I throw paint at the target and then select the most recent paint throw and most recent predicted target in the programme and these are stored in a grid array and a paint array respectively. The centers are compared between the most recent grid target and the paint throw in the array, and the programme categorises how close the paint throw was to the target. In addition to this, other calculations may be carried out based on my average accuracy, previous accuracy or total throws. The predictions tend to move towards where I am more likely to hit the target. If I have hit the target previously, the next predicted point is more likely to be a regular distance away from the previous target. If I have badly missed the target, then the next prediction is more likely to be close to the previous paint throw, as this is where I was presumably more comfortable throwing paint. In addition to this, the size of the targets also change. The worse I perform, the larger the target and the better I perform, the smaller the target.

The next target point is predicted (in black) based on the calculations undertaken by the programme and drawn to the screen along with an average point (in red), the paint throw center (pink), and the previous target point (blue). Two examples are set out in the gallery below. The prediction is projected onto the wall and the next target point transferred to the wall over the projection, and the serial process continues. The clip here of my first throw summarises the process. The piece has no set finish point and I had wanted to continue it until it began to go onto the floor or other walls; future works will do this. The fundamental premise of the work is not to achieve accuracy, score myself or to play a game, it is to explore creativity through Systems by involving both myself and the computer programme in a system. 

Concept and background research

The essential concept behind Systematic Spontaneity is my conclusion that the basic premise of Systems art, that art can be constructed (not created) without the artist (or the expression of the artist) to result in discovery or surprise, is partially unachieveable and fundamentally false. This conclusion has been reached through my work in practice which I have set out as part of my research project and is also set out on my portfolio website. In my previous fine art Systems practice, I seriously sought to reduce the artist through the construction of systems of rules built from my deconstructed drawing practice. For each piece, the rules were built from the material I used, the gesture I made, the position I was in, and whether time was a duration of time to end the piece or a fragment of time to confine the action. Through my systems, I sought to reduce the intuitive and aesthetic interference of the artist as much as possible and reduce the artist down to one decision at the beginning of the work.

To my knowlege unusually for fine art Systems art, I treated myself as a body (not an artist) within the system and used the restrictions of my body as part of the rules of the system to construct my work. Long before I had considered programming, I was programming myself. The drawings set out in the gallery below, for example, are 8 feet high and the drawings were made with ink, my finger and a simple movement from as high to as low as I could reach standing in one position. This position was marked on the floor or a stool with prints of my feet that also marked my presence and my absence. Within the limiting system, the reach of my arms and my height became part of the limiting rules, and Systematic Spontaneity is directly drawing upon this idea of myself as a physical presence in the system in a fixed position marked on the floor by my footprints while I am carrying the system out in a computational context. I am not programming myself, however, but working with a computer. I am not inspired by the polarised spectrum of debate around whether the computer is a tool or a partner. I am working with the computer within a system that it is in as well to make art. The computer runs the system that I made, and, in the case of Systematic Spontaneity, I carry it out. In another instance perhaps the computer will be carrying it out physically too, or, perhaps, carrying it out without me.

I am interested in the idea of presence and absence within the system, rather than tool versus partner. In Systematic Spontaneity, and in my future work, both myself and the programme are present in the system while it is being made. The evidence of my physical presence is there in my footprints marked on the floor and in the clothes I had been wearing and paintbrushes I was using that I dropped on the floor when I was done. And the work could not have been made without the computer directing where I should next aim. Additionally, if someone was so inclined, they could work backwards and work my piece out a little and follow my marks to disentangle some of what I had been doing, and how I had been moving or where I had been standing. They could even stand in my place. 

Systematic Spontaneity is the direct extension of the division of my Systems fine art practice into traditional conceptual systems and what I termed visual systems which are represented by my Board Works from 2009-2012, some examples of which are in the gallery below. These works used targets in the same way the grid points act as targets in Systematic Spontaneity. In the Board Works, I threw paint and would stop the work when I had either hit the target or targets, or completed a certain number of throws.

The Board Works really began to articulate, or define, my problem with Systems art as it was and it has taken me until this year to really solve that problem. Despite the fact I pursued erasing the artist, I found in my practice, instead, that everytime I touched paper or threw paint something much more interesting happened than erasing the artist. I made the ocean from just my finger and ink moving on paper, or raindrops, or blobs I had not meant to, or clouds of marks that altered and changed over the paper. All because I touched the paper and interacted with the system. Every interaction was a point of possibility rather than confinement, every interaction ruined my ambition to erase the artist because it actually highlighted the artist. In addition, the unavoidable and simple truth was that I made the system. It was fundamentally flawed to pursue the eradication of myself as the artist in the system, because I set the system up in the first place. With the Board Works, this became painfully obvious because I could physically see my hand drawing the target, or the visual system. 

Systematic Spontaneity has directly arisen through a process of reconsidering Systems art from these observations. It has also been inspired very directly by this work that was made between 2012-2015 which took me until this course and the addition of computing to structure. The computer now defines the targets though a chain of reactions to my interaction with the art, and the system is responsive. Systematic Spontaneity has been made because of the conclusion I have reached that Systems are conceptual constructions that enable creativity (the artist) and can be disrupted and broken through contact with the artist. Every point of contact is a point of possibility, randomness, infinity. Because I now regard systems for what they are, constructions to enable creativity at which any point of interaction is an entry point for the artist, Systematic Spontaneity demonstrates some breaking and interferencce with the system towards the end of the work as I began to group throws and predictions in threes to explore the reaction, some images of this are set out below.

The work has also developed over the last year as I have explored how to work with computers. This research is set out extensively in my comptuationalresearch.art blog and Systematic Spontaneity represents one possible version of how to work with a thing that is so easily accurate and regular and nonphysical when I am the exact opposite. By projecting the prediction targets and transferring them to the wall I have avoided working directly with projector light, which I learned in Imitation of Life I am not comfortable working with because it is watery, indirect and intangible. In addition, the fact that I used printed, black circles to represent the computer was both practical and conceptual. The use of black separated the grid array from the paint array and avoided any confusion when comparing the two. It also represented my work around the idea of my hand versus the computer 'hand' that I both discovered and explored in Imitation of Life. Through these sketches I realised how accurate and regular the computer was compared to me and understood that working with it to make art would not be straight forward from a visual art viewpoint. Systematic Spontaneity does not try to make the computer present itself in any way other than the most simple for it to do, a perfectly drawn circle, such as I could not easily make, and this lies in visual contrast to my throws in all their complexity, variety and simplicity, that, in turn, the computer could not easily make.

The visually contrasting black circles and colourful, irregular paint throws also relate to my interest in abstraction and represent my conclusions on the nature of Systems art itself which consists of dualities that arise from the structure and the spaces between the structure of a system i.e. control and spontaneity, freedom and limitations, the known and the unknown (or, perhaps, random) etc. I have explored this in my writing about my fine art practice here. The constrasting aesthetic of the work also simply visually explores the aesthetics of simple, computational graphic shapes with human gestural strokes. 

An aspect of the process of making this work which has taken a secondary position in this instance, is my intention to capture the secondary results of my actions when painting. This is something I have been interested in for years and my practice has frequently sought to capture the knock on effects of my actions in terms of splashes of paint on the floor or attempted to predict splashes to create work, some examples of such work is set out in the gallery below. Systematic Spontaneity continues this aim in that the walls, floor and my clothes were covered in paint, however, it also began to explore other ways to intentionally capture the results of my actions, in this instance, in images. As such, I had a frame differencing programme running on a separate computer which captured an image of the wall when there had not been movement for ten seconds. It was intended that this programme would be angled at the floor where I was throwing paint to catch the marks on the floor with the idea I could use this for a future project or even machine learning. Instead, I decided to simply set it to capture the painting as it changed over time to begin with. 

Technical

I used Processing for the blob detection code and also the secondary frame differencing code. I had explored OpenMV and Python in the months prior to this, however, I concluded that Processing was better documented and I wanted to understand the basics of how blob detection and frame differencing work practically before I progress onto openFrameworks or Python etc in the coming months in my studio. I also explored OpenCV and simply found it easier to adapt and understand the programme from scratch in Processing. A key aspect of my decision was that I needed to select colours not brightness and it was more straightforward to do this in Processing. In terms of other technology used for this work, the programmes simply ran from my Mac laptop because the camera was of good enough quality and I used a projector.

There were many practical, technical challenges to plan for and overcome in this project. Initially, I carried out a lot of research into possible options regarding the identification of changes in a painting. I considered background subtraction and different ways to do motion detection. I had explored frame differencing for my work in progress piece in June and I continued to explore this for the final project and developed the frame differencing code in parallel alongside the blob detection code. This code use frame differencing to take photos when there was no activity, and it was intended that it would translate the image to black and white, and then count black and white pixels in new images compared to the previous image to determine how the white to black balance of the painting had changed in fractions based on pixels. This would then create an accuracy factor. This became my backup code and was ultimately used to take the frame differencing pictures of the changing work on the wall, some photos taken by the programme can be see here. It may still be developed in the future. 

I settled on blob detection because I wanted to learn something new and because it was the best fit for the work I wanted to do in identifying very specific changes in a painting. I carried out extensive testing to identify specific problems as much as possible even before I began testing on the wall. My first consideration in planning the work was that it was important that when I walked in front of the camera to paint, I did not get detected as a blob. Therefore, I began to individually select blobs by a mouse click to add them to the grid or paint array, and draw the stored blobs to the screen so that the blob location was fixed, rather than having a certain colour be tracked wherever it appeared on the screen. In addition, to ensure that there was a clear differentiation between the grid blobs and the paint blobs, I decided early on that the grid blobs needed to be one tone and the paint blobs needed to be a range of colours. By using multicolours for the paint blob, I hoped to avoid the duplication of paint blobs, for example, the counting of a new blue blob and the recounting of a previous blue blob. During the making of the final piece, however, I relaxed this rule. To avoid duplication I began to use white paper to cover certain blobs up and I also used bright pink paper, which I had not used anywhere in the painting, to identify paint blobs in circumstances where the colour had previously been used and also where the paint blob was below the size at which a blob could be detected by the programme. Ultimately, this use of paper then faciliated my breaking of the system as I started to cover up past blobs and select new blobs in threes, rather than one by one.

Due to the large difference in the type of mark that was being selected, I needed to ensure that paint blob detection was set to the correct sensitivity for blob detection. This required the tuning of size thresholds and, again, paper became very useful in covering up paint splatters that were large enough to be detected. In addition, grid blobs were all one colour and, to avoid the duplication of previously selected blobs when a new blob was added, duplication functions (Daniel Shiffman, ref 3 and 4) were included to ensure that when a blob had previously been detected it would not be redected with each addition. This took quite a lot of experimentation to refine. 

In addition, from my testing work, I knew that shadows would be an issue with the grid array because they were in the black range. To counteract this, I began testing on the wall as soon as possible so that I could set the colour thresholds to be as narrow as possible to avoid the picking up of similar tones of grey or black. I also tested which tonal range of grey or black would best avoid the detection of shadows. As such, I tested a range of black and grey marks on the wall. In addition, this also served to explore the type of distance range between the wall and the camera I had to work within, and the minimum and maximum size of the blobs I could reliably detect as well as the minimum distance between grid blobs. I also introduced functions to eliminate either the whole of the grid or paint array, or the most recent grid or paint blob in the array. This was to ensure if there was a mis-selected blob, that it could be erased without impacting the programme. As the painting became more complex and certain blobs, particularly grid blobs, got closer together the programme did try to merge blobs together on occasion. I used paper to redefine the blobs (and here from another angle) which worked well, some examples of my use of paper are included in the gallery below. 

Another technical area of work and planning was my awareness of the liklihood that the work would be created over several days, and, because I am also interested in exploring the additional impacts of my actions and introducing machine learning in the future, I wanted to gather as much data as possible from the blobs. As such, I explored storing the blobs as data in CSV tables to perhaps make other work from or to practically allow me to restart from where I had left off the day before if I needed to. This was also done because, prior to actually setting up to work on the wall, I did not know the precise scale of the work and the projection on the wall. I considered that I might have to do the work in two or four parts and even start on new section of the wall. This meant that I wanted to be able to store the data from previous throws to continue the drawing on a new part of the wall. This was not needed in the end for this project, but I plan to use it for future work. It could even be interesting to break the work up and have different sections reacting differently, and yet cumulatively, on different walls. 

I also realised as I prepared the programme for Systematic Spontaneity, that the prediction might easily go off screen because, until I actually ran the code in real life I could not know how it might turn out. As such, I made a restriction function to bounce the prediction target back onto the screen or bounce it back to the start of a new line if it had reached the far right hand side of its fundamental right to left trajectory. 

Future development

I have many ideas for future development of systems work, not least because I have ten years' worth of ideas already behind it. The most straight forward development I will start shortly is to explore greater complexity in terms of how the programme analyses what I have done and how it predicts what I will do next. I could make the variations more extreme, or make the functions more dense and complex, I could simply reverse the premise of the code in that the code could make targets more difficult to hit the more accurate I have been rather than becoming more regular. I also want to explore how to make the grid marks more dense and the blob detection more sensitive so that I can detect blobs even on top of each other, and I want to include the ability to add only one colour blob to the array at a time, wherever I press the mouse, no matter if there are similar colours already in the painting. Being able to differentiate between close colour tones or black tones would mean that I could begin to make very suble walls of colour or tone. Once I am able to do that I will be able to everything I want to with this programme prior to the addition of machine learning. I will also simply explore using openFrameworks and Python, or whatever else emerges as a good option for this work beyond Processing. 

Machine learning is something I will be adding in to the project imminently. I have been looking at it over the past few months and the project is basically made for machine learning. This will mean that the computer can become something quite interesting within the system and more akin to myself. I do not want what I observe to be the liquid aesthetics of some machine learning to enter into the work, only the functions of it. I also want to refine the use of the projector and find a camera that I can set up in galleries so I can do the work in spaces with less throw away from the walls or even do it live like I'm in my studio. In terms of other related variations to this project, I want to explore and expand upon the work the frame differencing programme was doing and begin to direct it at the floor or at other areas of the painting so that I can begin to explore predicting marks on the floor and applying machine learning to them or some other system of prediction. 

The use of paper in the work became quite pronounced as the painting grew, as the gallery above demonstrates, and it was also essential in the testing stage. One future direction is to play with the idea of detecting blobs that are not there, by using pink pieces of paper for example as I was doing for this piece to detect colours that were too small, or duplicates of other colours. Another option is to make a giant wall collage layering paint and paper, or even peeling paper off the wall as I did so often to reverse work or break the system towards the end.

The development of the other forms of interaction with the computer in a system is another direction I will be taking. Making a pen plotter capable of covering a large canvas, Jackson Pollock style, remains one of my goals. This will mean that a computer can be watching and physically responding to what I am doing on a canvas in a very slow painting. I also have the idea of playing with screens and adapting them so that they become some kind of hybrid screen-canvas. A hybrid surface would be another surface on which my hand and a computer could interact to make art. I have had this idea for quite a while because I continue to be quite interested in the differences and similarities between the screen and the canvas and what pixels actually are and what a pixel could be e.g. do they have to be small, bright, light, could a canvas have a pixel? How? I will also be continuing to explore how to make art with a computer in a system in other ways. I will be working on having a computer talking to me in my ear, articulating the system and acting like intuition, telling me what to do. I have experimented with this already using keypresses and very simple instructions in Draw Command. 

Unpainting my piece was (not surprisingly) quite upsetting. I was going to include the unpainting of it in my video showing my work at the top of this blog post, however, I did not like to look at it and I did not want it associated with my piece, even though it is fundamental to the piece. Erasing the piece did give me quite a lot of ideas, however. I have set out some of my ideas in the gallery below. Firstly, presumably due to some reaction or incompatibility between the wall paint and the acrylic paint, I found that marks started peeling off the wall. The idea of reversing marks, or perhaps using background subtraction to paint in reverse is one idea that arose from this. In addition, the out of context marks on the floor versus the increasingly bare wall was something interesting that I observed as I painted over the wall. Given that the context of where I have been standing and how I have thrown is essential to the work, playing with the context of the marks is a direction of practice worth exploring further. Finally, the subtlety of the increasingly ghostly marks and the idea that they are hidden behind paint but still there on the wall is something that I quite like and, again, raises the idea of reversing the system in some way or focusing on the space between the marks in some way rather than the marks themselves. 

Self evaluation

Overall, the project worked well. I detected the center of grid and multiple-coloured paint blobs and compared their centers, drew the prediction and continued the serial cycle across the wall. One thing I found, however, was that the grid blobs could become unstable once the array on the wall reach around 15 because of the distance setting between grid blobs and an interplay with shadows on the wall. This interfered with the grid array at times because of the function included to check if a grid blob was already present at a certain location because I think the programme simply became a bit confused by dark patches and began counting them in as new blobs, and merging old ones. This was not a significant problem, however, it meant I had to rewind through the array or at least make sure the most recent grid and paint throws were also the most recent grid and paint blobs in the array to ensure that they were compared rather than other grid and paint blobs. The paint array functions worked very well at selecting multiple colours, and the paint array was very stable. So I will definitely be removing or otherwise stabilising the duplication function in the grid array perhaps by borrowing from the paint array. 

I did also find that when grid blobs were close to each other there could be some issues differentiating between two different grid blobs. However, this was not a big issue and I used the function to delete the most recent blobs when this arose or I physically separated the blobs using paper. This maintained their center, which is what I was using for the calculations, so it made no difference to the process of the system.

I seriously considered using OpenCV or openFrameworks to condense my code, but I am still happy I used Processing because it worked perfectly for this project and, because I am used to it, I could spend more time actually writing the code rather than working out how OpenCV works or where I got an indent wrong in Python. The use of paper to assist in the selection of paint blobs also became essential as I began to repeat colours and where a paint mark was below the identification threshold for the programme. The nature of the paint array code meant that I could simply place pink over the colour I was identifying and the blob would remain there permanently. I quite liked how I had to adapt to the circumstances and operation of the code itself, rather than everything going according to plan (which I had not assumed it would do anyway). It emphasised the spontaneous side of Systematic Spontaneity for me and also gave me ideas for collaging work rather than painting it in future versions of this piece. 

I did find that I could have made the prediction functions more extreme, or perhaps more extreme at certain points. Sometimes moving in tens of pixels did not make much difference. However, it did work perfectly to respond to my interaction with the targets, so I am happy with it. In the future, I will look at whether and where switch statements would be useful in my programme as it becomes more complex.  

I set out to achieve exactly what I intended to achieve with this project. I wanted to bring myself and a computer together in a system and make art in the new kind of system I had been thinking of, and I did it. I solved a ten year old problem about creativity and the artist and systems that had stunted my practice and made me surprisingly miserable in the studio for the past few years, and I definitely solved it. And I made art I like so much it still makes me sad I painted over it. So I would not do anything differently for this project, but I will be developing it further and expanding upon it and Systems art in computation going forwards. 

References

1) Daniel Shiffman, Motion Sensor, https://github.com/shiffman/LearningProcessing/blob/master/chp16_video/example_16_14_MotionSensor/example_16_14_MotionSensor.pde
2) Daniel Shiffman, Motion Detection, https://youtu.be/QLHMtE5XsMs  
3) Daniel Shiffman, Computer Vision: Adding Lifespan to Blobs, https://youtu.be/o1Ob28sF0N8
4) Daniel Shiffman, Computer Vision: Blob Tracking - Lifespan, https://github.com/CodingTrain/website/tree/master/Tutorials/Processing/11_video