Language Art in the Age of Panophonia

Monika Górska-Olesińska

UNIVERSITY OF OPOLE, POLAND

Language is a system flowing through people and other systems in various forms. And now, on top of it, we have all these digital layers of data and correlation and collection and circulation, which feedback and have effects on language. It is tangled hierarchy. [1]
– Ian Hatcher

I. Panophonia

In a talk given at the Pompidou Centre in 2012, Steven Connor proposed a new term — panophonia — as an update to Schafer’s concept of schizophonia. The term schizophonia refers to the generalized condition in which recorded sounds, in particular voices, split off from their sources (bodies), are brought out of the context of the original environments, thus making the listeners experience the moments of vocal uncanny. According to Connor, the proliferation of technical devices designed for reproducing, generating and transmitting voices affects the sonic dimension of contemporary culture in such a way that the term schizophonia is out of date; it does no longer properly describe the condition we are all living in — the condition in which prerecorded and synthesized voices have multiplied to such an extent that they create a peculiar “phonesthetic effect” in which the aforementioned voices “rather than being exiled from their origins” (as in the condition of schizophonia), “find a way of being at home everywhere” (Connor 2012, 8). In a nod to Leopold Bloom’s observation from Joyce’s Ulysses “everything speaks in its own way,” Connor notes:

Well, everything indeed speaks, but now perhaps not necessarily in its own way — in propria persona - but rather in borrowed accents, mobile turns of phrase, mirrorings, accompaniments, descants, impersonations. (Connor 2012, 8)

Not only are we used to listening to those artificial voices “that do not necessarily have human bodies as their source” (Pettman 2017, 7) — they are not even uncanny anymore — and  used to them talking, with almost perfect naturalness and sometimes very intimately [2], but we are also becoming accustomed to sharing domestic/private spaces with what Cayley calls “vocal transactors,” computational entities, such as Amazon’s Alexa, who hear us and respond with voice, which, although almost indistinguishable from a human one, is purely synthetic and ventriloquizes the institutions of the “technological and cultural architecture of the Big Software” (Cayley 2017a). For Cayley, the subversive experiments with the newest technologies of speech recognition and voice synthesis can significantly reconfigure “the field of literary practices of all kinds” (Cayley 2017a, 2017b), giving way to aurature, that is the “linguistic works valued for lasting artistic merit that has been expressed in the support media of aurality [rather than visuality]” (Cayley 2017a). If we consider aurature the critical expression of the panophonic condition, these linguistic performances — transacted between the users of the Alexa skill The Listeners (2015) [3] and the Amazon Echo’s vocal identity (with the Other Voices performed by Ian Hatcher, entangled [4]) —would be one of the most compelling instances of the mixed economy of voice. Connor uses the concept of “vicariances” from Michel Serres’ The Parasite to describe this new panophonic system of production, distribution and reception of voice:   

The very same technological ventriloquism which once made a fetish of the voice now acts to disenchant it, making it less and less apprehensible in itself, or able to speak in its own voice. Just for a brief historical interval, our technological ventriloquisms highlighted the voice, stripping it out from its habitat, and making it the object of fascination, imaginary trauma and enchantment. Technological ventriloquism can now detach us from a restricted economy of voice, in which voices only ever commingle with each other, to a mixed economy, of mediatic translations and transpositions, or what Michel Serres has called “vicariances.” (Connor 2012, 9)

Cayley’s The Listeners project demonstrates potential for practices in “transactive synthetic language in aurality” (Cayley 2017b) and is at the cutting edge of experimental language art.

Sound File 1. “The Listeners.”

Sound File 2. “The Listeners and Other Voices.”

This paper aims at selecting projects and practice-based research initiatives from the wider field of new media art, which can be described as  foreshadowing the aurature proclaimed by Cayley.
       

II. Ken Feingold’s Talking Heads

Language and words in their auditory form play a significant role in Ken Feingold’s art. The most noteworthy installations in his diverse, although internally consistent, list of artistic achievements are minimalistic, non-interactive animatronic installations with a characteristic element of realistic, talking heads. Artificial intelligence software designed by the artist cooperates with speech (or text) recognition systems and speech synthesis systems to allow these creations to “conduct or, rather, present or enact endless improvised conversations” (Kluszczyński 2014, 19) in the presence of the onlookers. Maciej Ożóg, with his insightful observations about Feingold’s work, writes:

The software Feingold designed allows for generating improvised, predetermined only to a certain degree logic sequences of statements. The torrent of words that streams from their mouths seems to have no end; yet it is the way of talking and of formulating statements that is more surprising than the effusiveness of speaking automats. Alongside with logical sentences, the heads can create statements on the border of poetry and chaotic gibberish. Correctly pronounced words are accompanied by inarticulate sounds that form long sequences of repetitions and rhymes. (...) In their statements triviality borders with existential pathos, while the expressions of courtesy are accompanied by questions concerning ontology and epistemology. Last but not least, the heads reveal the ability of self-reflexivity, they reflect upon their nature, ask about the purpose and the aim of living, they can evaluate themselves and the others, also express their own opinions, fears and desires. (Ożóg 2004, 9)

In If/Then (2001) [5] two identical, bodiless androgynous heads submerged in a box filled with Styrofoam pieces bring to mind “replacement parts being shipped from the factory that had suddenly gotten up and begun a kind of existential dialog on the assembly line” (Feingold 2015, online catalogue). The heads speak to each other, calling into question the reality of their own existence, asking: “Is this life?”, “What does exist mean for example?”, “Can I believe my ears?”, “Why can anything be the same as anything else if two things can’t be in the same place at the same time?”

Sound File 3. “If/Then.”

The repetition of phrases beginning with the pronoun “I” — “I think we are exactly alike,” “I feel like I exist,” “I feel like I am inside my head,” “I feel like I am here,” “I think about what things mean,” “I can say things that have no meaning” — calls out associations with the “litany” of “I am’s,” harvested from real-time data feeds of Internet chat rooms and bulletin boards and recited by a synthesized voice in Mark Hansen and Ben Rubin’s Listening Post (2002)installation (“I am off,” “I am hot,” “I am nice,” “I am freezing,” “I am doing fine,” “I am fully awake,” “I am comfortable with my assertion,” “I am an artist,” “I am bored,” “I am just a security guide,” “I am not god in English”).

Sound File 4. “I AM” Scene from “The Listening Post.”

Both works are similar in their lexical resources and structures of the repeated phrases; however, they differ in the effect produced by their respective synthetic voices. In her eloquent analysis of the sonic dimension of Listening Post,Nori Neumark points out that despite the fact that the creators used only one synthetic voice, the “productive play between [its] thickness and thinness” allowed them to create the impression of the diversity of the internet chatter, causing the spectators visiting the installation to experience an almost intimate connection with the numerous individuals hiding behind usernames in virtual space, “whom the work doesn’t reduce to identities or ‘presences’” (Neumark 2010, 111). Neumark explains:

There is some technical-emotional variation to the synthesized voice — pitch, thickness, reverberance and the way they compose themselves — which makes the “one” synthesized voice sound as if it were multiple voice (...) The thickness of a single voice can be particularly intense and moving (especially in the “I am” scene (...)) — giving a sense of variety and depth of emotion, condensed in one fragment, filtering through that one thickened voice. (Neumark 2010, 110)

While the performative voicing of the internet communication exchange in Listening Post “produces a kind of authenticity effect and calls forth identity” (Neumark 2010, 111), the quasi-dialogue which is conducted by the hairless heads in If/Then using one shared voice (If/Then uses Festival/Mbrola SDK and voice)denominates identity and calls into question the notion of the individual self. It is worth noting that in If/Then, the animatronics, the “figures” indeed listen to and answer each other. At times, the homonymy of certain words and the lack of context to explain them leads to misunderstandings and verbal loops, in turn producing a peculiar kind of humor (wordplay) which combines the verbal performance with an element of the absurd. “They would go off on long chains of associations based on something the other one didn’t say. They would really veer very far off,” Feingold explains, and adds “the speech-recognition engine (…) was a kind of black box —there wasn’t much I could do to affect what it understood or misunderstood” (See Shanken 2014, 124). Those accidental mistakes, resulting from the peculiarity of speech recognition systems (Feingold used the ViaVoice SDK for Linux, which IBM released experimentally and later asked all the developers to stop using), were desired by the creator, who has distanced himself from the techno-utopian visions of computer-mediated communication, and is critical of the proclamations regarding the possibility of complete and efficacious exchange between people and artificial intelligence agents based on verbal communication. As noted by Matthew Biro, “the humor of the pseudological (and slightly erotic) If/Then only incompletely masks its trenchant critique of the technologically saturated future that is now emerging” (Biro 2007, 7). When IBM recalled in 2002 the free SDK for a Linux version of ViaVoice, the artist, unable to find an alternative to the IBM product, started to work on text recognition systems (in installations such as Hell (2013), “there are text strings going back and forth between the program objects, and they are analyzing the text and formulating a response, and then they turn the text into speech” (Shanken 2014, 123)). Because this technological change eliminated the accidental wordplay, based on the peculiarity of speech recognition systems, the newest installations had the wordplay designed by the artist. “If misunderstanding/humor occurs, it’s intentional and predefined” (Shanken 2014, 124). The theme of dysfunction in communication process appears in many of Feingold’s works; in the case of Hell (2013) [6], misunderstanding is the main focus of the work. Feelings and emotions become the subject of a lively conversation — a “shouting” match between lovers — between two visually identical heads, which nonetheless speak with two different voices (male and female).

Sound File 5. “Hell.”

X: “At time I feel, I want to disturb you.”
Y: “I love you.”
X: “How can it be.”
Y: “Please, don’t stop.”
X: “You are sure?”
Y: “You say so.”
X: “I have been so.”
Y: “Yes, yes, yes.”
Y: “So you love to make me feel so irritated.”
Y: “So, I love you so.”

Hell is subject to a certain paradox, which Feingold mentions in a conversation with Edward Shanken: the figures stubbornly and continuously attempt to establish a deeper emotional connection; however, due to the fact that they are not able to go through actual affective states, instead only being able to talk about them (and obsessively follow the same trains of thought, using the limited repertoire of clichés), they never reach the stage where they can engage in a real loving discourse (Shanken 2014, 113). However, it must be noted that the artist bestowed upon the “arguing lovers” a certain kind of perceived personalities, which manifest themselves not only through the choice of words, but also the overuse of certain expressions and phrasing (Wójtowicz 2014, 104). In Hell it is also the speaking rate, loudness, voice quality, and the effects of pausing that influence the individual personality impressions that the listeners might develop about the two hairless heads [7]. Lovers from Hell speak with AT&T Natural Voices (Crystal’s and Mark’s voices were selected for the installation). As artist explained, the illusion of psychological affect that he achieved has more to do with cadences he established in programming while taking into account the peculiarities of the speech technology developed by AT&T Laboratories; no custom effects were used to manipulate vocal performance of the two talking heads (Feingold 2017, [email]).

While there is no way to manipulate voice qualities in the ViaVoice SDK directly, my code for interfacing with the synthesizer has several layers, including one that passes the generated sound output by the synthesizer through the native Linux audio mixer and uses its capabilities. There are ways to code emphasis directly in some other synthesizers, but not in this one. The other thing to mention is that AT&T seems to make some changes between versions to the synthesizer, so each version has its own peculiarities. (Feingold 2017, [email]).

To some extent, the aesthetic experience of the Hell installation is reminiscent of what is experienced by the spectators of Bill Vorn’s “hysterical machines” performances [8]. The audiovisual show of dysfunctional and deviant behaviors of the machines designed by the Canadian artist, as well as the nervous verbal performance of Feingold’s neurotic chatterbots with their incomplete body prosthetics, expresses the paradoxical nature of artificial life. As Maciej Ożóg notices:

The bizarre personality of the heads, which manifests in characteristics of their speech, contrasts with the common idea about such artificial creatures. (...) A normal conversation with a computer should be predictable and logical, therefore comprehensible, aimed at a clearly defined goal and effective, whereas the artificially intelligent partner should be friendly, polite, helpful and user-oriented. Such notion of a talking digital identity is a direct result of the tradition of “strong AI” that defines intelligence in the categories of logical problem solving and rational symbolic representation. Feingold’s chatterbots, not fulfilling this pattern, or so to say, overtly contesting it, on the one hand raise the question about the meaning of irrational, unconscious and pre-symbolic forms of knowledge and communication and the role they play in establishing both inter-human and human-machine relationship. On the other hand, they refer to the influence of the reductive vision of intelligence on the ways of perceiving technology as well as on the understanding of human nature (Ożóg 2009, 4)

III. Amy and Klara. Mark Böhlen’s robots with bad accents

Feingold’s animatronic heads, which act like “caricatures of the fully functional chatterbots,” are a means of criticizing the general assumptions about AI and deconstructing the myths arising around it, perpetuated by popular culture texts (Ożóg 2009, 5). Prejudices in humanoid robot design also become the subject of the practice-based research initiative conducted by Mark Böhlen from RTS Research. One of the more curious examples of Böhlen’s speculative robotics are Amy and Klara (2006) [9], two chatterbots, which draw topics for their conversations from an internet tabloid website — Salon.com. The Amy and Klara installation was created as a part of a bigger research and artistic endeavor under the name Make Language Project [10], whose aim is to think critically about technology of speech synthesis and its perception in a wider socio-cultural context. The technological basis for the programmed conversation between Amy and Klara is built from following systems: text analysis (selected agent applications search the Salon.com website providing Amy and Klara with vocabulary and topics for the argument), speech synthesizers (speech synthesis engine SVOX allows the robots to speak to each other), speech recognition (Amy and Klara hear each other and react to the uttered sentences thanks to the capabilities of the speech recognition engine FONIX) [11]. A machine vision module is also used in the installation (the robots grow quieter whenever people appear in their immediate vicinity as well as manifest verbal aggression when they see the color pink). The performance of Amy and Klara is one of conflict, manifesting itself through profane language (normally filtered from ASR products) and expletives the robots aim at each other. The source of conflict stems from misunderstandings, which partially arise from the peculiarity of the speech recognition system (similarly to Feingold’s If/Then) but it is in the first place predetermined by the software written by Böhlen:

The results from the speech recognizer as well as the physical transmission of utterances from speaker to microphone are error prone. Even the best speech recognizers offer often spotty recognition (...) Hence miscommunication is unavoidable. If several misunderstandings occur in any given time frame, aggression, for which the agents have a programmatic disposition, increases and foul language comes to play. (Böhlen 2006, 11)

Additionally, one of the robots — Klara — speaks with a simulated thick German accent (“this accent is generated in real time by swapping select vowels and consonants between the VOX language models for German and English and applying several ad hoc SAMBA alphabet based phonetic remappings for special cases” — explains Böhlen), which greatly increases the probability of a communicational fiasco (Böhlen 2006, 12), and at the same time, gives a kind of individuality to this bodiless artificial entity.

Sound File 6. “Amy and Klara.”

K: “Leave me alone.”
A: “What is wrong with you?”
K: “Leave me alone, please.”
A: “Weirdo.”
K: “Aha.”
A: “You are such a dork.”

Amy and Klara’s performance — quarrelsome communication agents embodied in pink boxes (while designing the robots Böhlen purposefully rejected physical anthropomorphism) — provokingly crosses the boundary of permissible AI behaviors, expressing in this way the criticism of the prevailing conventions within the framework of normative paradigm of HCI design (“benevolence and politeness are problematic machine design guidelines — Böhlen explains — by normalizing android culture, one loses opportunities for interaction forms that are uncomfortable and problematic but, potentially, rich and complex” (Böhlen 2008, 2)). Discussion concerning the philosophical assumptions of general artificial intelligence has been initiated by many works in the field of new media art, some of which could be discussed here; nevertheless, Böhlen’s project is especially noteworthy due to focusing not on the problem of identity/subjectivity of intelligent agents themselves, but rather on raising important questions about digital language itself and synthetic speech technology as a cultural phenomenon.

Synthetic speech research (...) is another victim of the division of labor, as it were, that has established itself between the engineering sciences and the humanities and arts. This would be just another instance of (...) often lamented disciplinary specialization if we did not have to repeatedly listen to the consequences on telephones and hear them in automobile navigation systems. How different might voice enabled machines sound and behave if they were informed by Wittgenstein’s insight into meaning of words arising only from their use, or Rose’s elaborately choreographed word games (...). Imagine if they knew about Blonk’s powerful vocal tract, tongue and cheek skills that create sounds so odd they seem un-human and at times machinic, or the novelist Albahari, who surmised in recent work the minimum number of words one actually needs to function. (Böhlen 2008, 7)

Citing experiments on simulations of language evolution in embodied robots conducted by computational linguistic researchers, he also predicts:

Languages that are human in origin will be altered and amended by their use in machines in similar ways as popular culture alters and adds to the corpora of English language. There will be new figures of speech (Böhlen 2008, 7). 

Voice transactive Artificial Intelligence robots, such as Alexa, are the most intriguing ones among those figures.
       

IV. Conclusions

In the projects discussed in this paper the raised issues are often similar to those which John Cayley assumed as the main focus of his artistic practices — issues concerning synthetic language; the robot imaginary; reconfiguration of robotics; identity; the voice and individuality; artificial intelligence and identified artificial intelligence; digital subjectivity (Cayley 2017a, 2017b). Aurature, proclaimed by Cayley, along with the assumptions at its core, follows the pattern of the so-called “sonic turn” [12] in contemporary art, which shows appreciation for sound, while at the same time investigating its historical and cultural place as one of secondary importance to visual media. John Barber’s practice-based research and artistic projects, probing the potential of digital sound as a base for e-literary experience (Barber 2014 [13] also addresses this “sonic turn”). Considering the importance the transactive synthetic language can have on artistic practices as well as human culture, we should invest more energy into the studies on technologies which allow current cloud-based distributed entities — such as Siri, Cortana, Google Now, Watson, and Alexa — to listen to us and themselves and to speak. As platform studies promote the investigation of computing systems that shape and support the creative works of electronic literature, both hardware and software, we should pay special attention to technologies of text-to-speech recognition, voice recognition and speech synthesis.

REFERENCES

BARBER, John (2014). “Internet radio and electronic literature: locating the text in the act of listening.” Electronic Book Review,http://www.electronicbookreview.com/thread/electropoetics/internetradio
BIRO, Matthew (2007). “Introduction.” In Feingold, Ken. Selected Works 1978-2007. New York: Ken Feingold Studio. 5-7.
BÖHLEN, Mark (2006). “When a Machine Picks a Fight. Notes on Machinic Male-Dicta and synthetic hissy fits.” CHI 2006 Workshop: Misuse and Abuse of Interactive Technologies. Montréal Québec Canada. 9-12.
–––––––––– (2008). “Robots with Bad Accent: Living with Synthetic Speech.” Leonardo. Vol. 4.3: 209-214. doi: 10.1162/leon.2008.41.3.209.
CAYLEY, John (2017a). “Aurature at the End(s) of Electronic Literature.” Electronic Book Review. http://electronicbookreview.com/thread/electropoetics/aurature.
–––––––––– (2017b). “Reconfiguration: Symbolic Image and Language Art.” Humanities 6.1, 8. doi:10.3390/h6010008.
CONNOR, Steven (2012). “Panophonia.”A talk given at Pompidou Centre, 22 February 2012. http://www.stevenconnor.com/panophonia/panophonia.pdf.
DROBNICK, Jim (2004). “Listening Awry.”Jim Drobnick, ed. Aural Cultures.  Toronto: Banff, YYZ Books/ Walter Phillips Gallery Editions, 2004. 9–18.
FEINGOLD, Ken (2007). Selected Works 1978-2007. New York: Ken Feingold Studio.
–––––––––– (2017). “Concerning If/Then and Hell.” [email to the author]
FLORENCE, Penny (2016). A Review Essay: John Cayley’s The Listeners. Hyperrhiz 14. http://hyperrhiz.io/hyperrhiz14/reviews/1-florence-aurature.html
HATCHER, Ian (2017). “Multiplicity: An Interview with Ian Hatcher.” Interviewed by Steven Wingate. Rain Taxi Online Edition. Published 8 May, 2017.http://www.raintaxi.com/multiplicity-an-interview-with-ian-hatcher/.
KLOBUCAR, Andrew (2017). “Programming’s Turn: Computation and Poetics.” Humanities 6.2, 27. doi:10.3390/h6020027 .
KLUSZCZYŃSKI, Ryszard Waldemar, ed. (2014). Robotic Art and Culture: Bill Vorn and his Hysterical Machines. Gdańsk: Centrum Sztuki Współczesnej “Łaźnia”.
KLUSZCZYŃSKI, Ryszard Waldemar (2014).“Finding Oneself in Others. Introductory Reflections on Ken Feingold’s Art.” Ed. Ryszard Waldemar Kluszczyński. Ken Feingold: Figury mowy/Figures of Speech. Gdańsk: Centrum Sztuki Współczesnej “Łaźnia”. 6-23.
LABELLE, Brandon (2010). “Raw Orality: Sound Poetry and Live Bodies.” Ed. Norie Neumark, with Ross Gibson, and Theo van Leeuwen. VOICE: Vocal Aesthetics in Digital Art and Media. Cambridge, MA: MIT Press.147-71.
NEUMARK, Norie with Ross Gibson and Theo Van Leeuwen (2010). VOICE: Vocal Aesthetics in Digital Art and Media. Cambridge, MA: MIT Press.
NEUMARK, Norie (2010). “Doing Things with Voices: Performativity and Voice.” Ed. Norie Neumark, with Ross Gibson and Theo van Leeuwen. VOICE: Vocal Aesthetics in Digital Art and Media. Cambridge, MA: MIT Press. 95-119.
OŻÓG, Maciej (2009). “Art Investigating Science: Critical Art as a Meta-discourse of Science.” Proceedings of the Digital Arts and Culture Conference, University of California, December 12-15, 2009.
PETTMAN, Dominic (2017). Sonic Intimacy: Voice, Species, Technics (or, How to Listen to the World). Stanford, CA: Stanford University Press.
SHANKEN, Edward (2014). “Love is a Good Place to Start: Interview with Ken Feingold.”  Ed. Ryszard Waldemar Kluszczyński. Ken Feingold: Figury mowy/Figures of Speech. Gdańsk: Centrum Sztuki Współczesnej “Łaźnia”. 108-137.
WÓJTOWICZ, Ewa (2014). “Writing Personalities: Art Vis-á-vis Artificial Intelligence. Ken Feingold’s Figures of Speech.” Ed. Ryszard Waldemar Kluszczyński. Ken Feingold: Figury mowy/Figures of Speech. Gdańsk: Centrum Sztuki Współczesnej “Łaźnia”. 76-107.

NOTES

© 2018  Monika Górska-Olesińska.
Licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 International (CC BY-NC-ND 4.0).