Invisible Form
Invisible until you interact with it, this is an interactive projection piece that textures the wall with subtle form and motion.
produced by: Rebecca Aston
Introduction
This is a piece that takes the projector light, the wall and the viewer’s shadow as inspiration and medium. The projector is seemingly just projecting white light onto the wall until someone walks into the space, the wall then shifts and moves under the moving simulated light as motion is detected in the physical space. My intention was to boil things down to the elements of light, shadow and interaction.
Concept and background research
My concept was to create a subtle, unnerving interactive projection piece that made the viewer think about their body in relation to the space. I set out with the task of replicating the light from the projector in a simulated 3D space and playing with the viewer’s shadow.
I take inspiration from a number of different artists who use light as medium, Olafur Eliasson and James Turrell to name a few. The use of shadow in interactive projection pieces is an additional inspiration; I researched and looked at a number of different works that used the user’s shadow as part of the interaction, such as Golan Levin’s work with shadows. The technical interest I had was in GLSL shaders for 3D meshes. I did a large amount of reading and tutorials about shaders. This work arose from somewhere between these different interests and goals.
Technical
This is an Openframeworks project. The interaction was achieved using Optical Flow, a machine vision technique, with a basic PsEye web-cam. I took the average direction of the motion and used it to update the position of the light in the code.
The effect was achieved using a GLSL shader on a 3D mesh. It was a flat plane which I transformed in the z-axis using a vertex shader with a Voronoi equation. The light that was being updated by the Optical Flow directional vector was an xyz position that was fed into the shader. I used it to calculate basic diffuse shading of the transformed Voronoi mesh and used that calculated light value in the fragment shader to set the color.
Future development
I’d like to explore creating a vertex shader that is more reactive to the viewer’s interaction. After a few failed experiments, I decided to experiment with Voronoi, which, although interesting and almost creepy, simply consistently animated over time. As I began to develop this piece I started to imagine creating a shader that was a record of viewer interaction, to keep a tracing of the light trails, almost like a drawing tool that perhaps grew textures on the wall. You’d use the light to expose the graffiti growth on the wall. Having reflected on it, I think keeping a buffer of previous light positions and using that to gradually create a fading line drawing in an FBO to feed in as another texture to the shader as a mask for an animated effect such as Voronoi or even Noise is now within technical reach for me.
Self evaluation
I spent a lot of time bashing my head against shaders. I have wanted to learn how to write shaders for some time now, taking them on for a final assignment was perhaps not the best plan I have ever had. I would have hoped to have gotten to a more advanced place in writing my own shaders, however that being said I feel that I got a good grasp of basic lighting in shaders (with one light point that is) in addition to the work flow of shaders, as well as the conceptual possibilities of vertex shaders versus fragment shaders.
My experiments with duplicating the viewers shadow into the 3D simulated space were less successful than I’d hoped. When creating an outline of the viewer (using blob detection or frame differencing), I could never get a clean enough edge to make it believable, and also this lead to some strange, interesting, but unintended feedback loops with the camera looking at it’s own output.
Ultimately for my final submission, I focused on delivering an interaction that felt good and felt connected to the content; picking up the webcam and moving it felt very much like using a torch to shine at the wall.
I would have liked the displacement texture, in this iteration the Voronoi, to be more reactive to the viewer's interaction in addition to the light, as covered in the "Future Development” section above.
References
- OpenGL Shading Language, 2nd Edition: A Simple Shading Example in OpenGL Shading Language here.
- The Book of Shaders: Voronoi shader taken and adapted from here.
- ofxCV example: Flow Distortion here.
- Inspiration that lead me to understand I had bitten off more than I could chew in this time-frame and point in my knowledge, here and here.