Fleeting Signals

Fleeting Signals is an interactive sound installation where your hands shape what you hear. Each of four gestures corresponds to an element: water, wind, fire, and thunder. The longer you hold a gesture, the more dominant that element becomes in the soundscape. Let it go, and it slowly fades back into the background.

The system was built in two parts. I handled the hand gesture recognition using Python and MediaPipe, training the model by photographing my own hand in each position so it could reliably detect gestures regardless of environment or hand differences. That data was sent via OSC to PlugData, where my collaborator engineered all the elemental sounds and the underlying logic that blended them together.

There's also a suspense layer: background tension builds over time, nudging the user to switch gestures. Ignore it long enough and the whole soundscape cuts out, then restarts.

The gesture recognition was trained on photos of my own hand in each position. MediaPipe maps the joints of the hand into data points, discarding the images entirely — which means it works reliably across different environments and hand shapes.

That data is sent in real time via OSC to PlugData, which handles all the sound engineering. The system tracks how long and how many times each gesture has been used and uses that to determine the volume and prominence of each elemental layer. Hold wind long enough and it starts to dominate; switch to fire and the wind gradually retreats.

The setup needs nothing more than a laptop with a camera and speakers.