Sea Calm
Keep the waves low and the boat afloat. A game controlled by sounds made by the player, trained via machine learning.
produced by: Hazel Ryan
Concept and inspiration
I’ve always loved calm, meditative games and I’m particularly interested in experiences which move away from combat and focus on other primary play mechanisms. In this interactive game-like piece, you don’t control a character, but instead control the waves of the sea. The waves, of varying heights to begin with, swell as they travel towards the little boat, and if a wave is too tall at the point it reaches the boat then the boat will sink. The boat has no agency and the interaction style is impersonal.
I hoped to blur the lines of what could be classed as a game or instead as a different sort of interactive experience. Here there are no lives or levels, there is no real character identification, and you directly control the ‘adversaries’. Once your boat sinks. I avoided using the word ‘game’ within the program itself to maintain an ambiguity and allow the audience to decide where the piece sits for them.
The graphics style here is influenced by my keenness to code everything directly with 2D geometric primitives in openFrameworks. I wanted to nod to classic 8-bit games by embracing a simple and punchy approach. I did also integrate Perlin noise for the body of the ocean which I hope gives an engaging sense of movement to complement the movement of the clouds. The size of the boat ‘character’ is quite small so that there is a sense of being adrift and overwhelmed by the environment.
Process and technical
I began by attempting an endless runner game in isometric perspective, but after getting very caught up in calculating the projections of all my shapes and how they would change over time, I realised that this was a challenge beyond the scope of this project.
I changed to another type of infinite scrolling game format, this time a 2D side scroller, which made sense for me as I knew I wanted the character to be fixed without controllable movement.
The boat interacts with the ocean when it performs a sequence of movements to ‘surf’ over a suitably low wave. This relies on treating the boat as part of an ‘Ocean’ class in order to make it perform actions depending on the position and height of the closest wave. The waves are generated with (constrained) random height and grow in size as they move towards the player unless the player intervenes by making a sound. If the wave is too high when it reaches the boat, the boat will sink.
Audio input is recorded and analysed in terms of its decomposition into 13 MFCCs, a process which identifies the distinguishing features of a sound. The resulting vector of values for each particular sound specifies a different ‘gesture’ which can then be recognised by the program when it is performed again.
I invite the player to use the ‘gestures’ of a “shh” sound and a clap, but really they could be any two distinctly different and easily-repeated short sounds. As displayed on the title screen, the game is set up to train itself hands-free - it begins and ends recording for the classification process based on the elapsed time, and the graphics change according to the same cycle, to give visual information on the fact the training is taking place. If the training cycle is missed by the player it simply begins again.
Evaluation and development
The MFCC algorithm I have used is computationally lightweight and conveniently wrapped up in an addon. However it does not consistently, reliably learn to recognise the two distinct sounds on the first try, so running the program is a bit of a circular process. Using patterns of automation to control the game state noticeably degraded the performance of the program, and this is something I would like to make more stable.
In this project I am proud of the interwoven logic of my various classes. There are a lot of nested ‘if’ statements based on geometric calculations, which was quite labour-intensive. Some of the mechanics are a bit clunky - you can see that the boat’s surfing motion is inelegant, but I think the behaviour is satisfying and the visual details are feasible to improve upon in successive builds.
I am very keen to make a 2D game-like experience with an isometric perspective, which can take logic from this project but would need a clever approach to coordinate calculation. I’m excited to look further into non-standard game mechanics in openFrameworks and I look forward to integrating the power that C++ code provides with some more graphics-intensive software approaches.
References
The custom addon ofxVoiceController was created by Rachel Max and I as a lab exercise. This is based on ofxRapidLib by mzed and ofxMaxim by micknoise.
I studied Phoenix Perry and Jane Friedhoff’s Experimental Game Design chapter from the ofBook, and watched some of Daniel Shiffman’s videos on game design for Processing, so my code is certainly inspired by these sources. However nothing is lifted or directly derived - all code is my own other than the joint work and dependencies for the addon used, although of course I have referred to course material and Mastering Openframeworks: Creative Coding Demystified by Denis Perevalov.