The search for neural correlates of consciousness
Imagine you are in bed about to sleep, but then you hear someone talking. Scared, you want to open your eyes and lean forward to see who is there, but your body is not moving. You try to call for help but cannot voice any sound. You are helplessly stuck with your physical body with no way of communicating to the outside world. Days, months then years past, you hear your loved ones discussing the possibility to let you go. You are stuck with yourself and cannot care for your own needs.
This terrifying scenario illustrates the contrast between consciousness and responsiveness or in other words: the potential of a conscious mind to be trapped inside an unresponsive body. For patients in a disorder of consciousness, this scenario might be reality. A “disorder of consciousness” is the general term to describe patients in a coma, vegetative state, or minimally conscious state which are mostly caused by traumatic brain injury. While most of these patients’ vital functions often recover, the communication between the brain and the body might be irreparable - resulting in a conscious mind trapped inside a body that is unable to move. The most relevant question for the next of kin - whether or not the patient is consciously aware, perceiving, feeling and thinking - is one of the hardest questions in recent neuroscience research. How to determine whether someone is conscious if they are not able to communicate? This project will engage the theories of consciousness and how to visualise brain states taking inspiration from the unicellular organism that is slime mould.
This raises the question of, how do we perceive present time? Put simply, it all depends on the speed at which the neurons are firing. Sacks provide the metaphor of the frame rate, where if your neurons are firing at 60 frames per second, your perception is standard. Now, if you jump at say 120 per second, you can process a lot more information and act more quickly. On the other hand, if you are at 1 per second you will notice some serious gaps. The case of Hester is an example of how a person perceives speeding up:
The neuroscientist Adrian Owen first demonstrated that it is possible to communicate with patients who suffer from consciousness disorders and appears to be physically unresponsive. The idea is simple: if a patient’s body is not able to answer your question, you would need to learn to read a patient’s thoughts. Patients were asked to imagine playing tennis or walking around their house, and the scientists recorded their brain activity using an fMRI scanner. Based on the knowledge of which specific parts of the brain become active when the patient imagines playing tennis (or any other kind of arm movement) the scientists were able to record the patient's answer to simple yes or no questions. This research demonstrated for the first time the possibility to communicate with physically unresponsive patients. It raises many ethical concerns as a result and proves that the scenario illustrated at the beginning of this article is more than just a thought experiment – it could be the actual reality for many patients with such conditions.
In 2006, Owen's work opened the floodgate to research on consciousness. Still, consciousness remains the biggest mysteries in today’s science and continue to challenge us to go beyond a mechanistic understanding of the brain and consider the fundamental ethical and philosophical questions on the identity of humankind.
The Theories of Consciousness:
David Chalmers narrowed down the questions encompassing the consciousness research to two problems - the easy one and the hard one. The former implies computational or neural mechanisms explanation for behaviors, as for the hard question of why all that is associated with subjective experience (“Hard Problem of Consciousness — David Chalmers”). As it is, the problem is open and a lot of great minds are debating it, the main theories defended are dualism, materialism, and panpsychism.
Dualism:
People who embrace dualism try to prove that consciousness is an entity entirely separate from the body. They attempt to address the hard problem of mind and body relations with quantum mechanics.
Materialism:
On the complete opposite is the materialist point of view which holds the belief that what you see is what you get, and that everything can be explained in terms of data. As for consciousness, it is made of higher-level brain activity caused by lower-level processes. “Materialists have the theoretical obligation to explain how subjective qualities could be accounted for in terms of objective quantities” (Goff p171). In this context, the project data is based on EEG readings that are electrical signals extracted from the brain and could be said to represent consciousness.
Panpsychism:
And in the middle, there is panpsychism, which implies that all things are conscious, to some level, and the combination of many particles or cells makes for a higher state of consciousness, where they have to solve the combination problem. An example of a possible conscious state at the most basic level is slime mould which is seemingly unconscious but can learn (Moskvitch).
Consciousness Our Research:
Despite many schools of thought, opinions, and theories about consciousness, most of the recent neuroscience community agrees on one fact: the inexistence of “the consciousness area”. Scientists tried for many years to locate one point in the brain that is responsible for consciousness, similar to how they locate the area for vision or area for auditory perception. However, it turned out to be much harder to locate consciousness. Instead of an accumulation of cells or a particular place in the brain, consciousness is nowadays assumed to emerge from highly complex tempo-spatial computation patterns within the whole brain. However, the high dimensionality of such patterns underscores the limitation of classic neuroscientific methods. This is where the artistic approach could come into play.
Early data visualization is based on the Boid algorithm by Ben Eater. Using the Cartesian x and y plane, each dot represents a brain area. The color pallet indicates two experimental states, with blue referring to the anesthetized and orange to the conscious brain. As for the data, it is represented by the items moving in space and interacting with each other, giving an early sketch of a small data set with 16,430 data points.
The project offers an alternative way of looking at brain activity through the methods of art. Using AI (artificial intelligence) to reduce the dimensionality of the data and create visualizations of Spatio-temporal activity patterns in the human brain. By doing so, it puts special emphasis on the crucial role of time and dynamics. Thus this project takes a new perspective on the brain’s dynamic computation by finding ways inspired by the to visualize high dimensional dynamic patterns emerging from neural activity.
Artistic Research:
The aim is to capture the patterns of the brain’s dynamic principles in an artistic representation. The project visually characterizes a healthy brain’s dynamic pattern, then compares it to brain patterns in altered states of consciousness, such as anesthesia or disorders of consciousness. Thus, challenge the question of whether these patterns can really be the physical principle of human consciousness.
Here we have the first experimentation to see the limits of data agents I can have on-screen. This is due to every agent communicating their position and velocity to each other to calculate coherence, separation, alignment from their visual range. To improve the performance we could include spatial acceleration structures.
The main problem is the high dimensionality of the data which contradicts most intuitive visualization approaches. Every single one of the 105 electrodes is able to record the brain activity 250 times per second. Over a recording period of five minutes, this results in about 7,875,000 data points. To stay as far away from any conventional data visualization, the project draws inspiration from nature. A first exploration was inspired by Theodoros Papatheodorou and Refik Anadol, to represent each data point as an agent moving in space where their speed is changing depending on the EEG reading. This approach resulted in a really interesting fluid bird-flock like behaviour, representing the complexity and dynamics of brain activity in a rather unconventional way. But this model quickly showed its limitations as we cannot see the temporal change or the connections between different areas. In the next steps, we increasingly focused on the brain’s functional connectivities (i.e. the communication between brain areas) as this approach has shown a lot of promise in recent research of consciousness. It especially focuses on evolution through time. As a result, we found inspirations in trees, leaves, thunder and slime mould as they all share similar patterns and behaviours.
Slime Mould:
Early slime mould model exploration, without data.
To visualise the data, a slime mould or Physarum Polycephalum model was created as a visual inspiration, as it presents some fascinating behaviour, renowned for its ability to network, to learn, to survive adversity, and to navigate complex systems in an efficient and equitable way. It has been used to accurately map routes to major cities as well as agglomeration of dark matter, thus, it was used on a smaller scale to map the connectivity between the different brain areas.
To better reproduce an accurate model of the slime mould, a sample was obtained for close observation and deep understanding its behavior. This further led to exploration ants like activity or stigmergy , as the pheromone trail they leave behind in order to find their way to food and back to the nest.
First we can observe that it spreads by attempting to cover most of the surface with fractal paths in search for food. Those paths are then connected to a larger one for nutrients, very much like a tree or a neuron. What get interesting is when the paths make contact with each other they start to merge, since it is a single cell organism. If it spreads in a direction for too long without finding food (or new information), that section is slowly cut off from nutrients and disappears.
Looking closely at active slime mould, directional transfer of liquid happens to promote growth and expansion. If they find food, the shortest paths connecting the two pieces of food will expand in diameter due to the amount of nutrients being passed through. For data visualization, those conducts can be used. With brain areas for food, the fluid as the direction of communication, and the size of the paths for the intensity of connection giving an indication of time, the temporal brain activity can be mapped.
Code:
Here, in the terminal, we have an example of the data pipeline problem I am still fixing. It affects the visuals where particles drastically change speed.
The code is based on the stigmergy model by Jeff Jones (“Characteristics of pattern formation and evolution in approximations of physarum transport networks”) and presented in my class with Andy Lomas. To gather the data and use it, a pipeline of multiple programming languages is utilized, starting by collecting the data from the EEG, it is then cleaned and tidied using Python as the electrodes are agglomerated to the area they scan (from 100 electrodes to 10 areas) and their connectivity is extracted. Then, using R, every area gets a dataset composed of their connectivity, this is the “easy” technique found to deploy the connectivity by area in the Opneframeworks sketch. Where each area has a dataset with its connectivity with all the other areas and the particles of each have their own area they are attracted to depending on the level of connectivity between their starting area and target area.
Improvement:
Talking with Josef Luis Plez, a Master’s student in Mathematics making beautiful creative code, and Andy Lomas we agreed that the project needs to be using the GPU instead of the CPU. Making the transition means that more data can be visualized since the CPU works like a single pipe where all the agents are waiting to pass one by one through it, the GPU operates as multiple small pipes where the information is all computed at the same time. This technique enables Josef to have up to 10,000,00 particles in real-time and at once without experiencing frame drops. Another approach to take is to convert the work into a voxel, thus making an efficient three-dimensional rendering. Overall, great improvement can come from using the GPU for the visual graphics while at the same time increasing the amount of data that can be accessed. This could make it possible to represent two brain states at the same time and offering an alternative and informative way to explore brain data.
Conclusion:
Comparison of brain states baseline, in orange, and anesthesia, in blue of a healthy patient. There is obviously more activity under anesthesia but it is too soon to say for sure due to the data pipeline problem, which is further demonstrated in the left video as, after 30 second the connectivity change and displays intense movement which should not happen since the data range from 0 to 1.
This project tackles the challenging problem of rendering consciousness and big data, it looked at the materialistic approach to gather data as a base for consciousness and panpsychism with the learning unicellular slime mould for a visual interpretation of the data. I made renders that compare brain states and some ways to visualize consciousness. After this exploration, the aim is to find the best setting and then expand the model by adding more data using shaders and the GPU and translating the project in 4D with voxels. Hopefully, as the project advances the more data can be used, the more information can be extracted and the more capable the system will be.