How MediaFutures’ supported artists impact the media field?

After the first year of the MediaFutures project and the first round of the support programme, we would like to reflect back on the impact of artists working with data and startups in the media field, somehow pointing out the added value of the MediaFutures support programme. We classified the approaches into 4 categories.

Improve the understanding of data

Through their artworks and artistic approach, the artists help the general audience have a clearer understanding of how data is collected and used and what the numbers really mean. In order to do so, they use data visualisation and sonification. The artworks are based on data sets, processed by the teams. 

This is what Antoni Rayzhekov did in collaboration with the SME JECT-AI, when creating the artwork Fragile Perspectives. This is a multi-sensory experience of news landscapes, information distortion and the fragility of perspectives formed by unbalanced news consumption. By analyzing a news index of over 20 million stories in six European languages targeting the keywords “pandemic”, “populism”, “censorship”, “climate crisis” and “future” the artist generated 3-dimensional news landscapes that depict the development of these themes over the past 3 years. In order to produce these landscapes, a machine learning algorithm was employed to reduce the multi-dimensional information into 3D landscapes. The specific news landscapes representing these keywords were then modulated into tangible pieces: five glass objects were produced, each representing one of these themes. A soundscape surrounds the installation. It is produced by a software sonifying the news landscapes maps, the sound changing depending on the landscape scanned. Experiencing the tension of the glass pieces on vibrating metal makes the audience aware of the fragility of the perspectives that are generated by the news consumed. Through this project, the team helps the audience understand that only limited news articles can be processed by a single person. As the variety provided is tremendous, we need to learn to navigate the news landscape.

Some documentations complete the artwork, helping the audience understand how the data was processed, and what the landscapes represent for each topic. The audience can go further with the tool JECT-AI which was used to analyse news and was partly developed by the SME during the project.

By creating data visualisation artists bring new, multimodal and interactive representations of data. Another example is HyperViz, where Frederik de Wilde in collaboration with the SME Scanworld created 3D visualisation of hyperspectral satellite data from a wide variety of sectors: environmental management, agriculture, pandemics… Only one hyperspectral satellite is currently in orbit, the PRISMA mission from the Italian Space Agency. Yet, the game-changer CHIME will be launched before the end of the decade. As a result, more and more hyperspectral data will be available. With HyperViz, the teams provide a data visualisation solution for the scientists but also for the general public. This immersive prototype turns hyperspectral data into a digital experience that can be experienced by the general public in a way that empirical data cannot. In the future version of the work, Frederik de Wilde would like to create an interactive experience, where the audience can navigate inside the hyperspectral data, and choose where to go and what to showcase inside augmented reality devices. 

Thanks to data visualisation, citizens can discover what is hidden behind numbers. Such artworks help them decrypt the information in a more accessible and poetic way.

Draw attention to hidden data and behaviours 

Art can be a space for peaceful debates and eye-opener discoveries. Focusing on the media sector, the MediaFutures’ artists shed light on hidden information, particularly on social media. This second category of outcomes gathers data-based artworks. Those works are based on research relying on data sets. The artworks present the results of the study.

In 730 hours of violence, the artists collective Domestic Data Streamers studies the new paradigms of violence. This exhibition is composed of 9 artworks, each about one area of violence in the 21st century. Each piece is based on specific data sets. One of them is about online hate speech and cyberbullying. Through their work, they explain that online hate speech, as well as cyberbullying, are forms of discursive violence that take place on online platforms that offer the benefit of anonymity and a wide reach. Studies show that on social media, hate speech travels the fastest, being shared and spread more rapidly than any other form of digital content. The artists studied the number of offensive tweets worldwide containing the three most used words on cyberbullying and online hate speech today: an extreme racist term, a pejorative word for homosexual people and a pejorative word for sex workers. On the exhibition wall, the visitor can discover three hammers, each of them knocking on the wall at the frequency at which those 3 words are used on Twitter. It goes from 120 beats per hour for the first word to 1 800 beats per hour for the last. Adding to the speed rhythms, hammers hitting the wall create an eerie sound, thus helping the audience acknowledge this phenomenon.

On its side, Critical Climate Machine, an artwork by Gaëtan Robillard quantifies and reveals the mechanisms of misinformation on global warming found on Twitter. It is composed of two data sets: one from the paper “computer-assisted detection and classification of misinformation about climate change” (CARDs) by Cook, Boussalis and Coan and the living collection of tweets from 44 Twitter accounts linked to conservative think tanks and contrarian blogs. The architecture of the project consists of a database, a claim monitor and a machine learning model. The claim monitor collects tweets from climate sceptic accounts and writes them in the database. Over time, the database’s entries are classified by the machine learning model, which is itself multiplied in a network of 32 nanocomputers. Results are both displayed on the sculpture and added to the database. Each fake news item is categorised (following CARDs study) and the number of the category is displayed on the artwork. In the exhibition, a paper establishes a correspondence between values and categories of detected false arguments. Confronted with a landscape of numbers, the visitors are invited to evaluate the quantities of each type of false arguments that they can see on the artwork, reflecting the activity on Twitter. 

MediaFutures’ data-based artworks bring the discussion around important topics for our society through artwork-based displays. It shows how datasets are used in research projects and what are the technologies processing them. This knowledge can be transposed to everyday life, when navigating on the internet, for example, recognizing the datasets and technologies used to produce content. 

Reinforce social links 

The third category of approaches within MediaFutures is the one gathering network-based artworks. Those artworks aim at reinforcing social links. The teams try to counteract the polarization, fragmentation and lack of fruitful debate on the media sphere and create solutions to increase participation, co-creation and dialogue with citizens.

Social Sandwich, by Fast Familiar, offers encounters with anonymous strangers. The users are invited to collaborate with one another to determine the trustworthiness of the news that appears online. During a 15 minute message-based conversation, they see the world from another perspective and discover how to keep exchanging when they don’t agree with someone. Apart from an experiment, it is also a playful glimpse into another person’s life, where they exchange pictures, jokes and songs. If they want, they can share their social media accounts at the end of the chat. Apart from fighting misinformation by forcing the audience to analyse news content, this experiment also creates links between strangers that would not have met otherwise. 

PONTE is also based on an online tool (Sensemaker) which allows discussions starting with an illustration. The artwork-based discussion launches the participants into a creative narrative mode. The metaphors and abstractions at the centre of arts allow people to contrast perspectives without devaluing an opinion or attributing blame for being wrong. The inputs from those discussions with people around the world are then presented back in illustrations to the participants in an exciting way. Apart from being used online, it aims to become a tool in museums and exhibition spaces to foster speeches and enhance user experience in cultural places. It can also be used in schools, universities and sports clubs. 

With Biblio-graph, Mariana Lanari and Remco Van Bladel created a participatory tool to link data from cultural libraries. It offers a menu of functions for the mapping, browsing and visualization of annotated and recorded data in different levels of granularity within the context of a specific collection of data. Apart from the tool creating links and improving collective knowledge, the artwork can be presented as an installation performance where the visitors of the library are invited to collectively use the tool. Strangers can share their views on books, authors and concepts. Moreover, during the artwork and tool creation, the artist worked on Yiddish culture and calls on Yiddish speakers to record fractions of books. This group, composed mostly of elderly people, met frequently to read and help Mariana. 

Through their artworks, artists develop techniques to bring people together (physically or virtually), meet and discuss. Being in an art context helps to break down barriers, and let people connect with strangers they would not have talked to otherwise. These connections can also happen during the artwork creation process.

Increase citizens’ technological awareness and media literacy 

 Finally, some of the MediaFutures’ artworks are experimental support for technological awareness. They help raise questions about the limits and potential harms of new technologies.

In Soft Evidence, Ania Catherine and Dejha Ti created a series of synthetic scenes. They don’t call it deep fake, as they are not created with the intention to harm or lie. The artists’ intention is to make the most believable synthetic media possible. This way they warn the audience about how sophisticated synthetic media can be, and the difficulty to recognise what is true or fake. In the exhibition, they mix synthetic scenes with real scenes to invite the audience to guess which scenes are modified. This interactive guessing encourages the audience to question the images they see, who created them and for what purpose. 

In order to create those scenes, they developed their own data sets, concerned about unethical extraction methods of some artificial intelligence APIs. This way they can be fully transparent on how they created those sets. 

Obvious Collective also worked on synthetic media with the artwork The Evil Magic Mirror. In this experiment, the artists create a deep fake in real-time with the face of the user. Whereas in Soft Evidence the artists wanted to warn about the sophistication of synthetic media, here they demonstrate how quickly they can be created. They show how in a few seconds a software can capture your face and body gestures, and turn your image into a deep fake saying words that you would never have said. At the end of the experiment, the data are deleted, and the artists express the potential harms of deep fakes and artificial intelligence.

Those artworks use new technologies and data sets in order to warn the audience about how they could be misused and become harmful. 

From visualising data to creating synthetic media, MediaFutures artists played their parts in improving media literacy. And we surely will discover new methods in the next two rounds!