How artistic research can help make history palpable – an artist’s perspective.
Blogpost concluding artistic research fellowship with the Visual Methodologies Collective’s programme ‘Climate Imaginaries at Sea’ during the Fall semester of 22/23.
By Kim Spierenburg.
Summary
In this fellowship, I used artistic research to explore Dutch identity and its connection to water. Being an audiovisual artist, I used artistic methods to connect and engage with the topic of climate change. After thematically reflecting on historical video footage, I discovered a general theme and sentiment: resilience surfacing over disaster. Then, I artistically translated these emerging associations and sensations into sonic and visual interpretations and created ‘Water Nation’. The animated visual interpretations in ‘Water Nation’ were created using AI-image generation technology. ‘Water Nation’ is an audiovisual artwork inspired by Dutch archival material of flooding in the 20th century from the archives of the Institute of Sound & Vision. The archival video footage shows Dutch villages underwater and the human effort to control the force of water. There is a contradiction between the vast natural violence of water and the industrial machine ingenuity and hand labour of the Dutch. I found that affectivity connected to nature is at the heart of my artistic practice and research. When the emotional power of music comes together with familiar imagery, audiovisual art can touch on how we feel about water, technology and history. By doing so and forming novel, yet recognisable narratives and associations, I found artistic research to be useful to help make history palpable in an intuitive way.
Introduction
‘Nooit meer 1953’
A typical Dutch farmhouse under-water, its inhabitants on the roof waiting to be rescued. A deserted animal in what seems to be miles and miles of water, a church flooded, people sleeping on army cots, fleeing the water. These are some of the black-and-white images of the flood disaster of 1953, de Watersnoodramp, every Dutch person grew up with. At school, with special projects, exhibitions, books and survivors’ stories. We learnt about dykes, delta works, technology and the collective ingenuity and effort to survive. The phrase ‘Nooit meer 1953’ (Never again 1953) stuck in our heads. The images are etched in my memory, but the possibility of an actual flood was far from my reality when I was growing up in Amsterdam. My parents had no dyke guard duty as was common in Zeeland during a storm and I was never afraid of losing my house to the water. Still, I had my own experience with water, almost drowning as a kid. Although it is compulsory for every child to learn to swim in the Netherlands, I faced the unyielding power of water from an early age. Yet I did not get scared of water, instead I learned to respect the laws of water, its power, tide, cold and depth. That and all the other stories and imagery about water shaped the way I perceived water. Now, facing sea-level rise, I wonder if my experience is unique or collectively felt, shaping subconscious thought processes and a national sentiment around water.
Although it is impossible to write about the rapidly developing field of artificial intelligence (AI) and not be behind already, I will try to state my experience as an artist using AI. Although AI cannot be compared to graphic design software or Digital Audio Workstations (DAW) because of its complexity, I see it as a tool. I used the technology as a digital creative tool instead of the black box spitting out art as some make it out to be. In creation, I wish to understand what is happening and have influence over the outcome. That’s what I tried to achieve. There have been a lot of complaints from artists about AI text-to-image generation models being trained on their art for free. Since the technology can form new images in the same style as the artist’s, the fear is that the artist will become redundant. Citing media theorist Anna Notaro: “Art has a significant role to play in engaging with and exploring new technologies and in contributing to interdisciplinary conversations. Artists have often been pioneers in reflecting upon social and technological transformations by creating work that makes explicit the dangers, but also the exciting possibilities ushered in by innovation” (2020, p. 322). While I believe it is important to protect artists with copyright, I also believe there is a role for artists in ensuring that the movement of technological progress is in a direction that is beneficial for the development of their art and artistic development as a whole.
At the beginning of this artistic research fellowship, I wondered how I relate to AI technology as an artist. Since I am trained as a composer, electronic music producer, vocalist and violinist I was interested in music generation models. During this research project, I looked into different generative AI models. The refinement of the creative outputs in recent years has been of such quality and accessibility that AI art has now entered the mainstream (Bonadio et al., 2022). Interest in automatic music composition is increasing and main players, both in the AI and music industries (Google, OpenAI, Amazon, Sony, Spotify, etc.) are involved (Civit et al., 2022). If visual art had been as protected by copyright as music, the text-to-image models would probably not have existed in their current form (yet). That is why the language models that have been around for a long time have not yet been developed into music generation models of the same calibre. After doing several experiments with AI text-to-image models I found that the repeatability of the method would become my main focus as an artistic researcher. It is necessary to further explore and develop ways of using the technique to get a result that makes sense. That means achieving a palpable result: it does something to you when you look at it on an emotional level. For me, that’s what makes it art. Digital (AI) artist Giusy Amoroso sees nature as programming and refrains from futuristic vision: “With all my work, I try my best to focus on the now and to interconnect to who I am today”. The connection with oneself, the environment and ‘more-than-human world’ can also be seen in generative artist Feileacan McCormick and neural artist Sofia Crespo with their work ‘Entangled Ways of being’. I think it is very important to develop how to use the technology, rather than just developing the technology itself. A human working with AI is still more than just AI. AI is not (yet) able to emulate the feelings, associations, historical value and sensations we feel with something. Visual tech director of the Berlin based collective Transmoderna, Timur Novikov compares AI to a “chaotic child that needs to learn so much”. Therefore, it is not enough to chase every new AI technique and model. Something I saw a lot on Discord forums, where a small community of developers and digital art enthusiasts formed. Much of the development went into creating new python-run models. The models gradually became more shielded with tokens to develop towards monetizing the technology. On the ‘share your AI art’ Discord and Reddit forums, where tips are exchanged on prompt design (what text to use to get a precise result) and creations are shared, I saw a lot of the same kind of results. Much future-oriented, steampunk style with lots of hyperrealism and attractive warrior women who seem to have walked out of the future with big breasts and giant shiny metal swords, spaceships and megacities. What I made was so different from the rest of what was posted that sometimes I was removed for lack of ‘future’. Other times, I experienced the forums as motivating and stimulating. One member called my work ‘ahead of the curb’ and others encouraged me to keep going. In which direction can AI art develop when artists continue to be involved in its future development? I experienced developers requesting for testing of their technology. It seems developers need artists to create new outcomes right now, but will they keep the technology open for artists or shield it off and move towards mass use and monetization?
As a disabled artist, I gravitate towards being a technology solutionist. In my life and artistic practice, I rely on technology to do the heavy lifting I cannot physically endure. As long as I interact with the machine and have control, I see the added value of technology. That means I am not averse to leaving repetitive strenuous work to a machine. Concerning technology deployed for climate adaptation, there seems to be a division between trust and mistrust in new technologies. Quoting the climate fiction novel Furore by Christiaan Weijts: “Sometimes it seems like we only became so intelligent to repair the damage that the same intelligence has done to our planet for centuries”. Looking at the archival footage, I see technological solutions to a natural issue. What is the expressive quality and symbolic value of involving technology in depicting nature in artistic work? I can’t stop but see a parallel between me, trying to convey the feeling of overwhelming water with not just handcrafted natural techniques but digital sonic methods and AI technology. Why is it we trust a human-made violin but not a human-coded python model? Is it actually humans we don’t trust? Or is it the power gained from having power over that machine, we fear? With ‘Water Nation’ I study the perceptions technology elicits.
How can art “talk about” climate change?
The climate ultimately benefits from “talking about it”, according to both climate scientist Katharine Hayhoe and author Amitav Ghosh. How can art “talk about” climate change? Researcher Jeroen Dera says poets offer an alternative view of humans and society with climate. By doing so, poetry can let the reader reflect on the situation without a direct call to action. Dera sees contemporary poetry as a model of how things can be done differently. In a poem, you can experiment with other views of reality. The same goes for the increasing number of Dutch literature exploring future disastrous flooding scenarios in the Netherlands. The so-called genre of climate fiction offers insights into the language available to interpret the relationship between humans and nature, as well as how humans are connected to the future. In this way, it provides a framework for meaningful action and explores the cultural possibilities for change (Van Herten & Duyvendak, 2023). At the same time, Amitav Ghosh questions the novel as a suitable art form because of its limits in terms of complexity, time and scale, for example, that do not fit the depiction of climate change. However, the value of art is that well-executed artistry makes connections between emotionally distant areas. In doing so, it touches on what else can be connected and touches on hopes and possibilities. It touches on imagination.
In their study Affectieve crisis, literair herstel, De romans van de millenialgeneratie (Affective crisis, literary recovery, The novels of the millennial generation) (2021), Hans Demeyer and Sven Vitse found the postmodern development not only as a movement towards engagement but of affectivity. The millennial generation asks questions of an affective nature such as: “How can I feel and experience reality (myself, the other, the past)?”. At the heart of the programme Climate Imaginaries at Sea of the Visual Methodologies Collective, there is the question of how to critically engage people around the topic of climate change. Not only within the collective but as an artist I wonder how my artistic practice and research can lead to engagement around this topic. I interpret the affective crisis as mentioned above not as an either/or situation between emotional involvement or engagement but as a consequence. Limited engagement is possible without emotional connection with the topic. Therefore, the priority of the art I create is to make that connection.
Methods
Artistic research
What defines artistic research? The methods established science relies on do not necessarily differ substantially from those used by artists (Sandqvist, 2017, p.183). Artists develop methods and techniques. They review material, subjects, references and theories. It is fair to say that every artist does research as she works (Borgdorff, 2010). If doing research is inherently linked to working as an artist, what is the need for further establishing artistic research? From my artistic perspective, the added value of research for artistic practices lies in a reflection of what the artistic methods, choices and reasoning exactly are. It is a way of combing through references and understanding the foundations of what that artistic practice is built on. As philosopher Gilles Deleuze said: “To think is to create” (Deleuze, 1994, p. 147). I see value in artistic thinking but what that value is remains unknown if it is undocumented. Artists generate techniques and practices for a processual and relational approach to knowledge-making (Cull Ó Maoilearca, 2021, p. 23). In my approach to artistic research, there is a certain freedom of methods. An open ending by less working towards a certain end product. By not limiting the scope of the connections that can be made, I think artistic research is a particularly good fit for new knowledge production. Testing and retesting that found knowledge could be a subsequent step in the development of artistic research. Maybe the biggest difference between academic and artistic research is that there are fewer written reports of it.
Artistic methods
How to create a narrative without words, using only sound, movement and images? During this fellowship, I created four audiovisual artworks, developing an artistic method built on the principle of an adequate interplay between sound and vision. Without words, the storytelling capabilities of the method have to be on an intuitive level to still have an emotional impact. An adequate interplay helps our minds think in a certain direction or make new connections. Seeing something familiar such as a Dutch house evokes an association with a feeling or a memory. But what kind of feeling does it evoke? What happens to us when we see an archival image? And what does that mean for the sense of Dutch identity? For the Climate Imaginaries at Sea exhibition during the Warming Up Festival, Tolhuistuin, I created a water soundtrack for the movie that was shown. After reflecting on its musical narrative, I realised that more sonic build-up and depth was needed to realise the desired emotional effect with ‘Water Nation’s soundtrack. To develop that thematic depth, I analysed associations between feelings, history, technology and water. Afterwards, translating these emotional responses into sonic results.
The Institute of Sound & Vision created a selection of videos aimed at artists called Revive: struggle & emerge. The selection consisted of polygon journals from the 20th century. Below, a collection of unmodified, original historical stills from the newsreels used to create ‘Water Nation’ from the archival Revive pack.
AI text-to-image models enable synthesis of images from a given text prompt. For example: [“seascape, flood, light shines upon it, oil on canvas, heavenly rays by beeple”]. The generative AI tool that I used is a diffusion model (https://colab.research.google.com/github/kostarion/guided-diffusion/blob/main/Disco_Diffusion_v5_2_%5Bw_VR_Mode%5D_%2B_symmetry.ipynb). Diffusion models are inspired by non-equilibrium thermodynamics, a subcategory of probability-based models. They slowly add random noise to data by defining a Markov chain of diffusion steps. The next step is a learning process that reverses the diffusion process and constructs the desired data samples from the noise (Weng, 2021). The models mentioned above consist of a combination of CLIP and ImageNet models (Rudenko & Bilokin, 2022). As an artist using this tool I found that the unpredictability of the text-to-image process makes it an intuitive method. Often, I found the visual outcome ambiguous, surprising and allowing the viewer to form their image and feeling. I think expressing a sentiment with this dreamlike animated world is well-suited for evoking emotions or subconscious processes and feelings.
Translating associations and feelings into sound
What happens if we were to directly translate associations such as haunting, violent water and sirens into a sound effect? It is more complex than that. There’s a biological attractiveness of tone combinations in tonal music. The stronger the similarity between a tone combination and the harmonic sequence characteristic of a voiced sound, the greater its attractiveness to a listener and thus the relative biological importance of a sound signal to a listener. (Purves, 2017). Different scales can evoke sensory contributions of appreciation or disgust (Cooke, 2001). I treated the new work as if it was a silent movie. By not using the newsreel text or audio I could create a context. I went back to the question: “What do you feel when you think of water?” Is it danger, joy or possibilities and what sounds do you think of when you think of water? The murmur of the sea? Or roaring water, metre-high waves? There is a large variety of water noises. I have explored the roughness or lack thereof of these sounds. Some are calm and rippling. Those storms and big waves have lots of high frequencies and noise in them. There is a difference with more muffled underwater noise. I modelled synthetic noises to natural water noises to cover the whole spectrum of lows and highs while remaining in the same sonic thematic area.
Another inspiration was found in the rhythm and voicing of the primal force of nature as heard in Nordic music. Then I turned to the industrial associations as mentioned above. Inspired by the iron foundry of Modest Mussorgsky’s pictures at an exhibition. I explored spatiality, resonance and formant. Being far away or close, alien or known, more voice-like or less, transforming my violin and voice with electronic methods. Classical and electronic compositional tools were used. Using both orchestral richness and synthetic roughness. For example, making waves sonically heard with crescendi. Passing on musical themes between instruments, supporting a wash of waves with sudden pulses of energy. Waves retreading, coming back in a slow buildup of rising water. Droplets as a metaphor for what they could transform into.
Results
Associations processed into the artistic work ‘Water Nation’
After reflecting on the questions raised above and the archival footage, there is a general theme I see emerging. The theme is resilience surfacing over disaster. At first glance, it is desolate, with lost people, bicycles, animals and houses without outer walls. Above all, there is water that has penetrated everything. An aerial video shows the levee breach, with water rushing through with tremendous force. But then there is machinery being rigged. And people helping in the reclamation, wheelbarrow by wheelbarrow, feet in the sand. Small efforts of people on gigantic plains. Against siltation, so that there was farmland so that there was food. To survive. The urge to survive spoke from the images I saw. There is a contrast between the enormous indomitable force of the water and the attempts to tame that force by the Dutch with machines, engineering and brains. Not just industrial force, but a community, putting their shoulders on the wheel when necessary. A survival instinct that I see reflected in how the Dutch are and how I am myself. A toughness formed by centuries of fighting for existence. But it is not just a fight. Water has always been a source of life. At the same time giving and destructive. A force to be reckoned with, which cannot be underestimated. To know the water through and through, that is the Dutch way. Living with the water, not against it. The prevailing sentiment nowadays might be either: no big deal, we just raise the walls, the dykes and the dunes, or disaster scenarios of losing parts of the country. But a certain ease is also a way of dealing with a serious opponent, not shivering with fear. Maybe we have settled in because things have been going well for so long or there’s still that sense of community and resilience shown in the footage from the past.
There is a cultural determination around feelings that arise when thinking of water. With this artistic research, I focused on historical events that are imprinted in the national memory of the Netherlands. By reinterpreting those events in an intuitive way using audiovisual artistic methods I tried to evoke an emotional response around the topic of flood and the Dutch response to water. The result is the audiovisual artwork called ‘Water Nation’. Additionally, by doing this research I was reminded of the importance of the geographical location of the Netherlands. The Netherlands is a delta, terminus and outflow point to the sea with water coming from two directions (rivers and sea). Its coastline spans more than 400 kilometres and about a third of the country is below mean sea-level (van Koningsveld, 2008). Water belongs to no one. No country owns water because it flows through different countries and does not stay in one place. But the Netherlands is a country of water.
Conclusion & Discussion
To engage people around the topic of climate change it is necessary to use affective methods, connecting them with how to feel and experience reality, themselves and the past. Artistic research has shown artistic methods to make history palpable in an intuitive way. Audiovisual methods can bring subconscious associations to the surface. The interplay of the emotional power of music and familiar imagery lends itself to the formation of new narratives. Staying out of the dichotomy between blindly trusting technology or shunning it completely. Two worlds that are instinctively far apart. Research into generative technology has shown it to be a useful tool for developing new artistic methods. By doing so, technology can form a bridge between traditional methods and a more inclusive artistic future. The influence of this research on my artistic practice has been one of development. Linking my process to a social theme such as sea-level rise in a concrete way and verbalising my artistic research. Artistic research is well fit for new knowledge production. Testing and retesting found knowledge through making could be a next step in the development of my artistic research.
References
Bonadio, E., Egeland, S. (2022). Street and Graffiti Art between Augmented Reality and Artificial Intelligence: A Copyright Perspective. University of St. Thomas Law Journal, Vol. 18, No. 3, 2022, Available at SSRN: https://ssrn.com/abstract=4243751
Borgdorff, H. (2010). The Production of Knowledge in Artistic Research. Routledge.
Civit, M., Civit-Masto, J., Cuadrado, F., Escalona, M. J. (2022) A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends. Expert Systems with Applications: An International JournalVolume 209 Issue C. https://doi.org/10.1016/j.eswa.2022.118190
Cook, P.R., red. (2001). Music, Cognition, and Computerised Sound: An Introduction to Psychoacoustics. Cambridge, MA: MIT Press.
Cull Ó Maoilearca, L. (2021). Utopian? Perhaps. Impossible? I would not bet against it. Hinterlands, ARIAS, p.23.
Deleuze, G. (1994). Difference and repetition. trans. Paul Patton, New York: Columbia University Press.
Demeyer, H., Vitse, S. (2021). Affectieve crisis, literair herstel, De romans van de millenialgeneratie. Amsterdam University Press.
Dera, J. (2021). Poëzie als alternatief. Amsterdam, Wereldbibliotheek.
Hayhoe, K. (2018). The most important thing you can do to fight climate change: talk about it. TEDx Women.
McCormick, F., Crespo, C. (2022). Entangled Ways of Being – interview with Feileacan McCormick and Sofia Crespo https://verse.works/journal/entangled-ways-of-being.
Notaro, A. (2020). State-of-the-art: AI through the (artificial) Artist’s Eye. In Proceedings of EVA London 2020: Electronic Visualisation and the Arts (pp. 322-328) https://doi.org/10.14236/ewic/EVA2020.58
Novikov, T., Amoroso, G. (2021). Dixon’s AI-based digital ecosystem comes to south London’s nightclub Printworks. https://www.itsnicethat.com/articles/dixon-transmoderna-terraforming-digital-221021.
Purves, D. (2017). Music as Biology: the tones we like and why. Cambridge, Massachusetts: Harvard University Press.
Rudenko, O., Bilokin, O. (2022). Use of generative diffusion models of machine learning in image creation. Control, navigation and communication systems. Collection of scientific papers. – Poltava: PNTU, 2022. – Vol. 4 (70). – P. 114-116. https://doi.org/10.26906/SUNZ.2022.4.114
Sandqvist, G. (2017). New Knowledge from the Artist’s Perspective. Artistic Research in Music: Discipline and Resistance, p. 183, 184. Leuven University Press.
Van Herten, M., Duyvendak, L. (2022). “het geringe nut van fictie in tijden van nood” Nederlandse schrijvers over klimaat. Interférences litteraires/literaire interferenties, Vol. 27 No. 2: Narratives and Climate Change.
Van Koningsveld, M., Mulder, J. P. M., Stive, M. J. F., Van Der Valk, L., & Van Der Weck, A. W. (2008). Living with sea-level rise and climate change: a case study of the netherlands. Journal of Coastal Research, 242, 367–379. https://doi.org/10.2112/07A-0010.1
Van Raaij, B. (2019). ‘We leven in de waan van de moderniteit’, interview met Amitav Ghosh. de Volkskrant.
Verheggen, B. (2020). Wat iedereen zou moeten weten over klimaatverandering. Amsterdam, Prometheus.
Weijts, C. (2020). Furore. De Arbeiderspers.
Weng, L. (2021). What are diffusion models? Lil’Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.