This Creature Does Not Exist
This project explores how art and digital technologies can alternate realities, making the invisible visible. In order to do that, through the research, I have integrated insights derived from existing art-science collaboration as well as a model for new artistic engagement with the science and life-form.
produced by: Ngoc Vo
INTRODUCTION
Nudibranchs
Sea slugs, or nudibranchs, are a group of soft-bodied, marine gastropod molluscs which shed their shells after their larval stage. They are known for their extraordinary colours and striking forms. There were about 3000 valid species of nudibranchs known in 2017 (Ocean Portal, 2017), and more that have yet to be identified.
Figure1: Cratena Lineata, an aeolid nudibranch, a marine gastropod mollusc in the family Facelinidae. Image credit: Clay Bryce
Nudibranchs are important indicators of declining ocean conditions. They have a very rapid turnover, which implies that they respond very quickly to changes in the quality of the environment including pollution and loss of habitat. Since nudibranchs have such specialised and varied diets, an area with many different species indicates a variety of prey—which means that coral reef ecosystem is likely thriving. Coral is the ocean’s life system and without it, we will soon also have an ocean without life. Documenting nudibranchs’ diversity and monitoring their regional presence helps us understand how ecosystems are faring (Science Daily, 2018). Nudibranchs’ patterns play a vital role for indicating changes to scientists. Through their captivating patterns and colours that are so visible, recreational divers and scientists who come across them can see their movement. Scientists in Australia are using these intriguing creatures to document the effect of climate change. They believe that as climate change is occurring, the species is reacting to the temperature change and heading south (Tropical sea slugs migrating south as ocean waters warm, 2017). Deep sea animals are increasingly under threat from destructive fishing practices, resource exploitation, impacts from marine debris, and effects from climate change (Roberts 2002; Schlining et al. 2013; Boetius & Haeckel 2018). Despite the fact that the current climate conditions are great for sea slugs to bloom beautifully, we can not ignore that warming seas mean less food for sea birds, and adverse impacts for all marine ecosystems.
Reliable identification of species is a key step to assess ecosystems as well as biodiversity. Many of the issues facing our ocean today originate with us not knowing that much about it, in spite of it making up over three-quarter of our planet. It is interesting that we know much more about the topography of Mars and the Moon than our own ocean. Life underwater is not always visible, it has largely stayed out of sigh and out of our cultural mindshare. Most studies examining organismal responses to climate change (it has to be about identification) have focused on commercially valuable species of fish, crustaceans, and bivalves (Gledhill et al. 2015; Lesser, 2016), while the majority of non-commercial marine species, such as sea slugs, have been underrepresented. For fishes, it is now possible to gather point data from national history museums, worldwide, in real time, and in a common format (Vieglais et al., 2000). Museum data are high quality because voucher specimens can be examined if identification is questionable. However, like all point data, museum specimens provide only a limited view of the actual species’ range (Krohn, 1996), hence the need for predictive approaches. Predictive approaches, if applied for nudibranchs, will help define the undiscovered species, which could be a great asset for marine conservation and biodiversity.
Predictive Approach with Machine Learning
Machine Learning (ML) is a discipline of the computer science that develops dynamic algorithms capable to produce data-driven decisions (Thessen, 2016). ML has proven useful for a very large number of applications in many parts of the Earth system (land, ocean, and atmosphere) and beyond, from retrieval algorithms, crop disease detection, new product creation, bias correction and code acceleration (Yi and Prybutok, 1996). Common applications of ML in oceanography includes oceanic climate prediction and forecasting, species identification, detection of ocean pollution, marine and coastal water monitoring, coastal morphological and morphodynamic modelling, habitat modelling and species distribution (Ahmad, 2019).
Large amount of data which is collected by scientific instruments then separated into train set and test set. Therefore ML algorithms are trained by this data then build model with high accuracy and its parameters are optimised based on sample data during the learning step. During prediction, the model parameters are used to infer results on the previously unseen data.
Figure 2: A simple Machine Learning approach
Generative adversarial networks (GAN) architecture is a ML approach to training a generator model, typically used for generating images. A discriminator model is trained to classify images as real (from the dataset) or fake (generated), and the generator is trained to fool the discriminator model. Using GAN, we can learn how to create realistic-looking fake versions of almost anything. “This X Does Not Exist” (https://thisxdoesnotexist.com) is a website that collects multiple sites based on this premise.
Figure 3 (from left to right): This X Does Not Exist site, a fake real person generated by GAN
Pix2Pix belongs to one such type called conditional GAN or cGAN, where they have some conditional settings and they learn the image-to-image mapping under this condition. This approach was presented by Phillip Isola, et al. in their 2016 paper title “Image-to-Image Translation with Conditional Adversarial Networks” (Phillip Isola et at., 2016). The Pix2Pix GAN was demonstrated on a range of interesting image-to-image translation tasks, including: sketch to photo, edge to photo, day-to-night photographs, thermal to colour photo,... Understanding the habitat and distribution of marine species are important tasks for management and conservation of oceanography. An algorithm can be trained using a large data set matching environmental variables to taxon abundance or presence/absence data. If the algorithm tests well, it can be given a suit of environmental variables from a different location to make predictions of what taxa are present. This technique has been used to identify current suitable habitat for specific taxa, model future species distribution including predicting invasive and rare species presence, and predict biodiversity of an area (Thessen, 2016).
Here I explore the possibility of whether a cGAN can be used to predict an undiscovered type of nudibranchs from a sketch input. I implement a network architecture and train it on nudibranchs images I collected. I then evaluate the similarity of my prediction by comparing it to any newly discovered existing creature from the deep ocean.
EXPERIMENTING & LEARNING
For this experiment, I trained a Pix2Pix model called Edge2seaslug. To begin with, I scrapped 3000 photos of nudibranchs on 3 different Instagram accounts: @naotoetoh, @fuwa._fuwa._umiushi and @seaslugs_n_stargazers. Then I chose 533 pictures and removed the background and resized them to 256x256px. I detected the edges of images in batch and attached input images and target images together.
Figure 4: A is input image and B is target image
I adapted my code on a Pix2Pix pre-trained model by Christopher Hesse (Hesse, 2017) for my dataset. With my limitation in knowledge of Python, I trained 3 times, each time with regardingly 50, 161, 328 epochs (the number of passes of the entire training dataset the machine learning algorithm has completed). The model is deployed on a website, which can be found here (performs best on Google Chrome).
Peinaleopolynoe is a genus belonging to the family Polynoidae (scale worms). In 2020, scientists in California discovered phylogenetic of Peinaleopolynoe (Polynoidae, Annelida), with four new species (Hatch, Liew, Hourdez and Rouse, 2020). They were described as 'glitter worms’ due to their shiny and colourful bristles and plates. Figure 5 displays a comparison between a sketch input’s result with Peinaleopolynoe orphanae, which we can see a significant link in colours. Figure 6 compares a more in detail of them that share the same trend of pattern: gradient blobs made of dainty circles. Figure 7 shows the comparison between different checkpoints of training of a sea slug and different glitter worms. There are a strong colleration between output with 50 epochs with Peinaleopolynoe mineoi, 161 epochs with Peinaleopolynoe elvisi, 328 epochs with Peinaleopolynoe goffrediae).
Figure 5: Comparison between a sketch input with Peinaleopolynoe orphanae
Figure 6: Comparison between pattern of a sketch input with Peinaleopolynoe orphanae
Figure 7: Comparison between different epochs from training process of a sea slug and 3 new scale worms ( from left → right: Peinaleopolynoe mineoi, Peinaleopolynoe elvisi, Peinaleopolynoe goffrediae)
REFLECTION
The objective of this research is to investigate to what extent a cGAN could be used to predict unexposed nudibranchs. Even though the training was limited by my technical skills, the correlation of the results from Edge2seaslug and newly discovered scale worms has shown a potential of making invisible things seen. There is room for development for accuracy, but less than perfect accuracy maybe sufficient. In my opinion, the sampling variance is large, a small bias is acceptable if a larger number of observation can be exploited.
Art and science have much more in common than one might think. Each of these pursuits is, after all, an attempt by human to describe and understand the world, and even the universe around us. Technology is changing the way we experience and create art and technology as a medium is a connector to the natural environment. We live in a time where virtual screens, virtual devices are replacing a lot of elements that we would normally experience. In a way, what we experience is imagination, an interpretation, a projection of something that is here, in a natural environment, that you are now experiencing transform. This research/experiment has helped me see things that otherwise I will not be able to see, and feel a very deep connection with nature.
At the intersection of art, technology and science, we find a creative solution for real-world problems. The beauty in exploring new forms and making things visible is a powerful narrative because dealing with loss and disappearing nature today has become alien to people because of technology. Saving images seeds the idea of they are eternally preserved that we can always come back to experience something from the past. By archiving things that will be and are being lost (collecting data), we are given the access to experience to future past. Nature, in the case, marine nature, through immersive technology pleads humanity to think about loss and consciousness.
?ARTEFACT?
BIBLIOGRAPHY
Abc.net.au. 2017. Tropical sea slugs migrating south as ocean waters warm. [online] Available at:
Ahmad, H., 2019. MACHINE LEARNING APPLICATIONS IN OCEANOGRAPHY. Aquatic Research, pp.161-169.
Boetius, A. and Haeckel, M., 2018. Mind the seafloor. Science, 359(6371), pp.34-36.
Gledhill, D., White, M., Salisbury, J., Thomas, H., Misna, I., Liebman, M., Mook, B., Grear, J., Candelmo, A., Chambers, R., Gobler, C., Hunt, C., King, A., Price, N., Signorini, S., Stancioff, E., Stymiest, C., Wahle, R., Waller, J., Rebuck, N., Wang, Z., Capson, T., Morrison, J., Cooley, S. and Doney, S., 2015. Ocean and Coastal Acidification off New England and Nova Scotia. Oceanography, 25(2), pp.182-197.
Hatch, A., Liew, H., Hourdez, S. and Rouse, G., 2020. Hungry scale worms: Phylogenetics of Peinaleopolynoe (Polynoidae, Annelida), with four new species. ZooKeys, 932, pp.27-74.
Hesse, C., 2017. Image-to-Image Demo - Affine Layer. [online] Affinelayer.com. Available at:
Isola, P., Zhu, J., Zhou, T. and Efros, A., 2017. Image-to-image translation with conditional adversarial networks. [online] Openaccess.thecvf.com. Available at:
Krohn, W.B., 1996: Predicted vertebrate distributions from Gap analysis: considerations in the designs of statewide accuracy assessments. In: Gap Analysis: A Landscape Approach to Biodiversity Planning. J.M. Scott, T.H. Tear and F.W. Davis, eds. American Society for Photogrammetry and R
Lesser, M., 2016. Climate change stressors cause metabolic depression in the blue mussel, Mytilus edulis, from the Gulf of Maine. Limnology and Oceanography, 61(5).
Moen E., Handegard N. O., Allken V., Albert O. T., Harbitz A., Malde K. 2018. Automatic interpretation of otoliths using deep learning. PLoS One, 13: e0204713.
Montavon G., Samek W., Müller K.-R. 2018. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73: 1–15.
Ocean Portal. 2017. A Collage of Nudibranch Colors. [online] Available at:
Roberts, C., 2002. Deep impact: the rising toll of fishing in the deep sea. Trends in Ecology & Evolution, 17(5), pp.242-245.
ScienceDaily. 2018. Scientists describe 17 new species of sea slugs: Team constructs new family tree for group of nudibranchs; discovers color mimics. [online] Available at:
Schlining, K., von Thun, S., Kuhnz, L., Schlining, B., Lundsten, L., Jacobsen Stout, N., Chaney, L. and Connor, J., 2013. Debris in the deep: Using a 22-year video annotation database to survey marine litter in Monterey Canyon, central California, USA. Deep Sea Research Part I: Oceanographic Research Papers, 79, pp.96-105.
Thessen, A., 2016. Adoption of Machine Learning Techniques in Ecology and Earth Science. One Ecosystem, 1, p.e8621.
Uusitalo, L., Fernandes, J., Bachiller, E., Tasala, S. and Lehtiniemi, M., 2016. Semi- automated classification method addressing marine strategy framework directive (MSFD) zooplankton indicators. Ecological Indicators, 71, pp.398-405.
Vieglais, D., Wiley, E., Robins, R. and Peterson, T., 2000. Harnessing Museum Resources for the Census of Marine Life: The FISHNET Project. Oceanography, 13(3), pp.10-13.
Yi, J. and Prybutok, V., 1996. A neural network model forecasting for prediction of daily maximum ozone concentration in an industrialized urban area. Environmental Pollution, 92(3), pp.349-357.