About This Project

AI allows humans to study the conversations of other species. It forages in bio-acoustic datasets for patterns in non-human communications previously hidden from our ears. As the inner structures of how other species communicate are being mapped. How can researchers and the public navigate, discuss and experience these high dimension latent acoustic spaces? Our research is a public outreach project looking at how AI models of beluga vocalisations can be visualised into physical format.

Ask the Scientists

What is the context of this research?

Living alongside the work developed by ESP with bioacousticians Valeria Vergara and Jaclyn Aubin on beluga whales communications, we propose an immersive visualisation exploring the architecture of whale communication as modelled with AI.

We will be working with AI model AVES, the foundation model created by ESP already trained on beluga data, using the model output as raw material for creating a series of digital sound sculptures, enabling users to explore latent descriptions of beluga calls through hearing, sight, touch and intuition.

Our proposition is a public outreach experience about the future of audio visualisations for interspecies communication. Its success will be measured via assessment of public engagement and feedback through social media.

What is the significance of this project?

Traditional sound visualisations represent audio properties that matter to the ear (level, frequency). Although machine listening incorporates these parameters, the AVES model operates on internal representations of up to 1024 dimensions, that are not accounted for in waveforms or spectrograms.

What human words fail to depict, what our brain fails to grasp, art can help us engage with the unreachable. So as to navigate latent spaces, creative strategies such as experience design, immersive practices and data storytelling can help us design sensorial, collective and intuitive interfaces.

Our objective is for our visualisation to become a platform for scientists to explore the notion of "artificial listening" in discussion with a diversity of audiences.

What are the goals of the project?

Our prototype will be developed by slicing individual beluga vocalisations in time, apply dimensionality (e.g. PCA, t-SNE, UMAP) reduction to slices, to feed data into a creative coding environnement (e.g. Blender, VVVV, TouchDesigner) and translate beluga vocalisations into shape, texture, light and more. Different dimensionality reductions and data visualisation approaches will be explored.

Once our visualisation process and aesthetics are established, the same translation key will be applied to other whale vocalisations such maintaining consistency across an entire dataset.

Our realisation will be evaluated through audience engagement measurements and feedback through social media platforms, using a short film presentation of our latent beluga whale visualisations.