Opposing Self
Opposing self is an interactive performance and installation which presents as a two-sided mirror for the audience to face and reflect upon the good and evil sides living inside them.
produced by: Yue Chen
Introduction
For the final project in Workshops in Creative Coding Two, the brief was given to produce a piece of work reimagining a previous artwork. The artwork of my choice is a movie series started back in 2015 called Detective Chinatown directed by Chen Sicheng. It is a Chinese comedy-mystery buddy film series where the protagonist Qin Feng goes to visit his uncle Tang Ren in Thailand after being rejected from a police academy in China and later goes on to solve a murder crime together. The theme of the movie series revolves around the investigation of the opposition between good and evil. Sicheng tries to express the idea of opposition in a few ways including the protagonist duo between Qin and Tang as one being a genius and the other being a jumpy “failure”. When I was watching this film, I was deeply impressed and inspired by the theme and began to think about ways that I can transform the same ideas into an interactive experience and performance to provoke some thinking around humanity and the relationship between good and evil.
Concept and background research
“Good and evil are incompatible. When there is evil, there has to be good.” – Qin Feng
Drawing inspiration from Detective Chinatown, my ideas for the final project and its structure mostly come from a scene near the end of the first movie of the series. In the end, Qin finds out the truth about Snow planning Sompat’s death and secretly guides his pervert foster father to commit the crime so that she can get rid of them both. Before Qin leaves, Snow says to him:
“While we are individual lives, the summation of good and evil is invariant. Everyone plays his role from birth. Some of them are good and some of them are evil.” – Snow
In response to that, Qin once again takes a piece of paper and folds it in half so that the paper can stand on its own. This is an act that has appeared once earlier in the movie with Qin not knowing about what Snow has done and simply does it to answer Snow’s question at school. However, folding the paper at this point implies that Qin is trying to say human nature is like a folded piece of paper as no human being is perfect by nature. According to ancient Chinese philosophy, Yin and Yang cannot exist alone and thus, one is interdependent on the other. The same relation applies to good and evil here as if there is no good, there will be no evil. A human is human only if he or she has both good and evil sides.
So I decided to use the folding paper analogy in the movie as a starting point and combines it with the medium of “mirror”. The final installation is therefore an interactive mirror experience where a user can see himself/herself on both sides of a mirror reflecting its inner selves. Linking back to the philosophy of Yin and Yang, it suggests that despite how opposite and disconnected two things may seem, contrary forces may actually be complementary and interconnected to each other. With that in mind, I chose to work with the elements of Water and Fire from the Wuxing conceptual scheme (Five Phases) which relate to the idea of Yin and Yang to convey the notion of “being opposite but unified” as well as reveal the calmness and anger hidden inside.
Technical
The project is comprised of several stages. First, it needs to track human users in front of a camera and then converts what the camera sees into the graphics I programmed with openFrameworks. In terms of user detection, I used the Kinect v2 along with ofxKinectV2 addon by ofTheo. With the help of Kinect v2, I was able to retrieve the depth image source from my Kinect and store it in an instance of ofxCvGrayscaleImage and work from there. Then, I used ofxCv to apply median filtering with cv::medianBlur method on the image to remove noises and get a smooth texture. After that, I did some thresholding on the grayscale image so that the Kinect v2 would only see things within the specified range. However, to make sure I can use this depth texture as a mask later in the project, I looped through every single pixel of that grayscale image to ensure that each black pixel is also set to have a alpha value of 0, which is transparent with an if statement. Finally, I draw this image to an ofFbo object called maskFbo.
Then I switched to work on the graphics. I decided to experiment with shaders immediately as I reckoned it would be a good opportunity to do so but also knew that creating effects such as water and fire would actually be quite handy with shaders. Thanks to The Book of Shaders and some other GLSL tutorials, I was able to write some shader effects that look like water and fire. The shader effects were drawn into two separate ofFbo objects.
To enable the audience to interact with the graphics, I used ofxOpenCv and the optical flow controller example from Week 13 to detect the motion of the user and extract the movement on the x, y axes. Those values, named as avgX and avgY were mapped to other ranges that were sent as uniform values to the shaders to change the behaviors of the water and fire graphics. Thus, the user can change the speed, color, and movements of the shader effects with their body movements.
Here I needed to use the processed grayscale image of the user as a mask on top of the shader effects. The way I ended up approaching to this problem was calling the setAlphaMask() method on the shader FBOs and pass in the mask FBO which is the grayscale user image as an argument. The final output looks like the picture below:
Lastly, I drew all the textures and images on the screen for debugging purposes. Then I used the function called loadScreenData() to load the fire and water outputs into separate ofTexture objects. Then, I used ofxSyphon to stream both of them as individual textures to MadMapper so that I can adjust the mapping there with more ease.
Future development
Due to the limitations, I was only able to set up the installation in the corner of my bedroom. This proves to be a big challenge to me and has greatly affected the final presentation of the project. If I had more time, I would improve the setup of the installation so that the projection would work better and more accurately on each “mirror”. Also, instead of using projectors, I would consider using use high-res displays (if accessible) as it would remove the use of projectors and the user can see themselves more clearly with shaders. Also, since this is my first experience with using shaders, I feel like there is a huge amount of possibilities that I can do further to make this project more exciting in terms of its aesthetics, interactivity and performances. That said, I want to create a scenario where the silhouette of the user is more blended and smoothed into the shader effects rather than a solid and sharp shape from the alpha mask. Such an effect would also help me reveal the entanglement between good and evil symbolized by water and fire. Lastly, I would attempt to make the movement of the user generate some sound effects related to the graphics as well.
Self evaluation
Overall, I am quite pleased with what I have created for this project. Technically, I have learnt a lot of things such as shaders, Kinect, and image processing with OpenCv during the entire working process. However, I wish I had managed my time better as I spent a lot of time trying to understand shaders, write them from sketch and get them to work with imagery coming from Kinect. Also, I wish that I had been a little bit more thoughtful about the setup to save more time. Nevertheless, these things are all planned for future development to make it better!
References
Shader tutorials and code references:
GLSL Noise functions – Link: https://gist.github.com/patriciogonzalezvivo/670c22f3966e662d2f83
Domain warping – Link: https://www.iquilezles.org/www/articles/warp/warp.htm
ofBook – Image Processing and Computer Vision – Link: https://openframeworks.cc/ofBook/chapters/image_processing_computer_vision.html
ofBook – Introducing Shaders – Link: https://openframeworks.cc/ofBook/chapters/shaders.html
Fire shader tutorial - Link: https://www.febucci.com/2019/05/fire-shader/
The Book of Shaders – Link: https://thebookofshaders.com/11/
Forum post on Kinect v2 – Link: https://forum.openframeworks.cc/t/kinectv2-with-ofxopencv-and-ofxkinectv2/37227
Forum post on texture masking – Link: https://forum.openframeworks.cc/t/using-texture-as-alpha-mask-to-apply-transparency/26888/14