As we move in physical space, we leave echoes behind. Invisible, undetectable traces of intermingled actions that shape our reality without revealing themselves. What if we could capture these fleeting ripples we make in space as we move? And all that with the dexterity and speed that is so typical for human movement? Our inspiration drew from multiple sources, such as the trails of light in long-exposure photography, the mathematical beauty of motion trajectories in physics and robotics, and the ancient human practice of choreography and storytelling. TRACES emerged from this profound curiosity about the ephemeral nature of human movement.

TRACES was born as a cross-disciplinary experiment to explore these questions. The project is placed between science, technology, and choreography, where we created a physical space in which motion can leave marks behind, all visible, proud, and seen. Traces of light, flashes, swirls, colors, and patterns aligned with physical motion. By learning and exploiting action and reaction within the system, anyone can create their own embodied language resulting in dynamic art on the ground. A living, breathing canvas where the body guides the creation of the surrounding visual landscape, truly in real time. By that we join virtual and physical motion in a mixed realm in between ours and that of virtual creations. With our experiment, we want to show how interactive visual art can be augmented with state-of-the-art technology to see what was only part of our imagination so far.

Photo credits: Alexander Krivitskiy

We are an independent, interdisciplinary team collaborating on the border of visual arts, choreography, cinematography, engineering, and scientific exploration. Our combined expertise covers real-time computing, projection mapping and tracking technologies, experimental choreography, film documentation, and rapid prototyping. The goal of our collaboration within this project was to create a link between visual and motion arts using cutting-edge technology and creative solutions. The project involved intensive prototyping phases where, although speaking multiple professional languages, we needed to find the common voice to guide our project in the right direction. The result is a truly cross-disciplinary experiment where science and technology complement motion and visuals.

Creating TRACES required months of intensive collaboration between disciplines. We developed a unique methodology where movement phrases were created in response to visual possibilities, and visual algorithms were refined based on choreographic needs. This iterative process led to the development of what we call an "embodied language", in which movement qualities, that dancers can consciously manipulate, translate directly to specific visual effects. Over time, learning this language means developing an intuitive understanding of the system's responses, learning to "play" the technology like an instrument. This combination of technical precision and artistic intuition makes each performance of TRACES both technically unique and emotionally resonant.

Photo credits: Google DeepMind

Mixed reality (MR) is a type of reality extension (XR) technology similar to VR goggles where virtual animations and real-world imagery are combined. In contrast to VR, however, MR does this in the physical world (using projectors) without the need of specialized goggles. We use four projectors attached to a 2 m tall metal frame around a 4 m by 4 m arena. The overlapping projections allow us to create an immersive experience with little to no shadows. We update the projected images so fast that the experience becomes immersive, with effectively no visible delay between movement and visualizations. This fast reaction time is what makes our system feel like it naturally extends human motion with visuals.

We update the visuals according to human motion. But how do we know how a body moves exactly in 3D space with such high precision? We use cutting-edge tracking solutions based on infrared light (developed by OptiTrack) and detect the 3D position of human body parts with a mindblowing speed, capturing it 300 times every second. We use these captured coordinates to calculate posture, movement direction, velocity, and many more, and feed these into our visualization pipeline. Our visualizations are built on the work of established digital artists and are written with WebGL to allow a fully real-time experience. We developed three distinct visual worlds based on these WebGL shaders with different colors, styles, and feels to allow artists to develop unique choreographies within each.