Digital Ghoul
by Max Jala
Introduction
For my final project I decided to re-imagine of Frieder Nake's Walk-through-Raster series. When I was first getting into creative coding and learning about the history of computer art Nake's work stood out to me immediately. His use of simple rectangular geometry to create intricate designs appealed to me immediately, and is something I have tried to adopted in my own work to date. In my re-imagination I have attempted to add a modern twist through the use of live-video and modern computer graphic techniques. Digital Ghoul is an interactive mirror that tracks its user incessantly to put the them into the centre focus.
Frieder Nake's Walk-through-Raster series
Zach Lieberman's Your Face Is Not Your Own
Concept and Background Research
Though the Walk-through-Raster series are all printed stills, I always pictured them as animated with therapeutic sinewave movements and this is what I implemented in my re-imagination. The squares in Nake's piece have always reminded me of windows and this informed the use of the Mac windows that capture different sections of the user's face. Upon further research, I stumbled upon Zach Lieberman's work for a New York Times article and it inspired the noise-altered facial composition of the piece. With Digital Ghoul I wanted to create something that was both playful and unnerving - the user's face is stalked and their fractured reflection is constantly in the centre of the piece. The user can also trigger specific animations by smiling or opening their mouth (as if saying 'ooo...'), covering or hiding their face scatters the windows around the scene to disrupt and conceal the webcam's image. The window's cast a dark shadow that give a ghostly impression befitting of the title. Aesthetically I wanted to complement the already distorted image of the users face and this informed the dark shadows behind the window that give off a ghostly quality, the subject's face also has a severe amount of contrast and specific colour blending to further enhance the impression of a ghoul or ghost.
Technical
This project uses ofxFaceTracker2 addon to handle all face tracking features. This includes: detecting whether a face is present, face position, smile value and 'ooo' value. One useful attribute obtained from the library was the face centroid, by translating to that position and recalibrating all face image points to be relative to the face centroid point I could then shift the whole face image to always be centred in the screen. One technical challenge was smoothing values for face landmark points that are constantly moving and hyper-sensitive, by storing previous positions I could apply a smoothing amount to change positions in a controlled way. This was important to complement the sine animated sizing of the windows. Another challenge was attaining good framerate, with smile and 'ooo' detection already being heavy processes I had to be very wary of unnecessary processes and inefficient methods - most importantly reducing the grabber size was key to alleviating a lot of the computing load. One technical goal I had when approaching this piece was to achieve high production-value graphics within openFrameworks itself. Usually I use syphon to broadcast my openFramework sketches to VDMX to apply post effects that enable me to achieve a higher level of visual fidelity, but I wanted to make this piece as accessible as possible to as many people possible. I hope to potentially distribute this app online so it was necessary that all graphics functions be handled within the app itself. To help me achieve the visual effect I desired, I used various addons: ofxFboBlur, ofxPostProcessing and ofxPSBlend. ofxFboBlur enabled me to create the blurry shadow behind the windows, while ofxPSBlend allowed me to create the metallic, ghostly colouring of the users face, finally ofxPostProcessing controlled the final image's overall contrast and brightness for further tweaking. To get the most from these addons I used various FBO's to separate different parts of the composition (foreground, background, whole image) for specific adjustments.
Future Development
There are still a few areas I wish to improve about the piece, namely the CPU usage which can be very high when running the app. Attempts to decimate the ofVideoGrabber feed as an input for face tracking, while keeping the original for drawing slowed operations down significantly so in the end I only used half of the video grabber's full resolution for both tracking and drawing.
I also want to implement different effects for the piece as a whole that transition to each other in a smooth and natural way. For further customisability I also intend to learn more about shaders to grant myself more manual control over the effects I deploy.
Lastly, I believe this piece would be very well-suited as an Instagram filter that a wide audience can enjoy. Of course, this won't be built in openFrameworks but hopefully the existing project and code will serve as a useful foundation.
Self Evaluation
Overall, I am pleased with what I have accomplished with this piece. One landmark for me was conducting graphics modification in openframeworks itself in a way that is familiar to my usual workflow. Other than that, I think the user interactions are fun, simple and accessible.