Fleeting Signals

This project was created for the Sound, Space and Interaction course of the Creative Intelligence and Technology programme at Leiden Univeristy.

Using PlugData, the visual programming tool for sound design, my teammate and I created a program that recognises hand gestures and generates recursive ambient sounds in response to these gestures. This project was inspired by the series Jujutsu Kaisen, in which characters form specific hand gestures to invoke powers typically connected to natural elements.

The project consisted of two parts. The first part involved using Python and a gesture-recognition library to detect hand gestures. Using the laptop camera and providing numerous samples of specific gestures, we trained the system to recognise when a particular gesture was shown. Depending on the gesture detected by the camera, a corresponding signal was sent to PlugData, triggering specific ambient sounds. For example, a particular hand gesture was linked to the element of fire, resulting in the ambient sound being enriched with fire-like sounds.

All the sounds were generated within PlugData without the use of external samples. Elements such as wind and fire were created using fundamental synthesis functions like sine waves and resonance filters, showcasing the capabilities of PlugData for procedural sound design.

The system was designed to produce a unique outcome every time it was used. Since it is impossible for a person to replicate the exact duration, precision, and transitions between gestures each time, every user would experience a different balance of elements in the soundscape (e.g. one person might hear more fire, another might hear more wind). In contrast, others might never trigger certain elements, such as electricity, depending on their gesture patterns.

See here to read the entire process.

Media