Leaning Haiku — AR tech, generative text & body gestures.

— Praveen Sinha

“How can I bridge the digital virtual layer and the physical reality through the means of text ?”

This question led to the ‘Leaning Haiku’ project during the call for Co(de)-Po(etry)-Jam , 2020 by dra.ft.

Since 2019, I have been exploring Augmented/Extended Reality (AR/ XR) technology to understand the affordance it can provide to extend mundane activities and not-so mundane fantastical interactions. How can looking at a simple soda can engage a user while one waits to take the next sip ? How can we feel like entering a secure area while crossing a simple door ?
Or on the other end — can we tell the story of fairies hiding under our dining tables ? Being a design professional, this practice based research on emerging tech keeps up the excitement to keep working in this field. From extending a post-it note to an act of walking through our house gates, AR does allow one to weave a virtual layer even with the present limited hardware technology. And to conceive an experience that does not just extend rather binds both physical and virtual together was an interesting challenge after these explorations. Further inside this space of extended reality design, unlocking such experiences can initiate further conversations about the expressions of the metaverse,

‘Leaning Haiku’ became my first public extended reality concept work where the act of body gesture and the generative algorithms come together to create a ‘Haiku’ in the virtual AR realm.

The idea to bring ‘Haiku’ as the vehicle of text kindled while playing with head gestures in an existing Instagram AR filter game, where you move your head up-down for yes & left-right for no. And as a habit, I did my usual indian headnod in that filter & watched it glitch humorously. The Indian head nod is something I consider to be poetry in itself — it can be used for anything — yes, no, don’t know, All good, Not good, etc. Pairing it with my liking for Haikus, the poetic head nod and the japanese poetic form amalgamated into this new AR concept.

Writing ‘Haikus’ which can connect with strangers on the internet was the next part of the puzzle. I indulge in my amateurish form of haikus while on my random walks around cities. I click photos when something catches my eye or when i stand aloof on the roadside. Then this picture anchors the haiku I write — I call these #cityhaikus on social platforms. And while working I put haikus at the end of my presentations — a form of #officehaikus. However, writing something where a person can reflect & connect with their own lived experience was difficult at my skill level. And to use standard 5–7–5 syllable structure rules like the the New York Times Haiku project[2] was not suited for the connection I was looking for. Nor a human choice based selective haiku like in the game of Ghost of Tsushima[3], would have impacted the users. I recall using this app called ‘HaikuJAM’ where strangers used to prompt & complete eachothers haiku. It helped understand the structure that can be used to write the algorithm to generate the haikus that connects with the reader. I chose a simple structure to keep the haiku relatable with day to day experiences of people —
1 line : Observation — from Nature or environment.
2 line : Evocation — from human emotions.

3rd line : Truth (like Kireji) — a philosophical truth.

With the concept of the poem in place, I started putting them together using the sparkAR IDE to create an Instagram face filter. I decided to use generative algorithms to write ‘haiku’s’. This form of computer-generated poetry is ‘mere generation’ [1] through a template of a haiku. Where the method is based on the human harvested meaningful sentences[1].The agency of generativity came from both a digital & a physical control. First random factor came from the user’s act of leaning their head left or right. And the second randomness came from the usage of random javascript function in the code for selecting lines from the 6 different lists.

While writing the algorithm I realised that the platform does not support any advanced API calls to external sources. This sort of limited the variety of lines we could use to create the poem. So to bring a further nuance to the experience I added a level of detail to it through the usage of negative & positive flavour to the composition. I wrote 6 different lists of sentences 2 flavours for each of the three lines of a haiku. This was then bonded to the left & right head lean as an easter egg for the users to discover. The structure was very simple in code too, check below for reference.

[//

//Structure of the generative haiku algorithm.

list 1 = [ Collection of Positive Observations ]

list 2 = [ Collection of Negative Observations ]

list 3 = [ Collection of Positive Evocations ]

list 4 = [ Collection of Negative Evocations ]

list 5 = [ Collection of Positive Truths ]

list 6 = [ Collection of Negative Truths ]

count =0

Until ( count == 3) {

If ( count == 0)
Observation = {
if( lean == right) get random line from list 1
if( lean == left) get random line from list 2
}

If ( count ==1)

Evocation = {
if( lean == right) get random line from list 3
if( lean == left ) get random line from list 4
}

If ( count == 2)
Truth = {

if( lean == right ) get random line from list 5
if( lean == left ) get random line from list 6
}

Haiku =

Observation,
Evocation,
Truth

print Haiku

count = count + 1

}

//]

After a few days of ironing out the aesthetics of the filter, it was published to the public on my Instagram profile.

The best part was — the fused single experience of the trio — AR, generative text & head nod — multiplied into more layered experiences. It was exciting and fulfilling to see the many ways people started using the AR filter — none of which I expected or imagined to start with.

The ‘Leaning Haiku’ was used on dancing actors from motion pictures, on still passport photographs, on hand-scribbled face sketches, on recorded haiku outcomes and many more. The AR filter led to an assemblage of poetic usage in itself. This gave the poetic nature of work a participative engagement as observed within the public art exhibits. The emotional acknowledgement to the verses were found to be cryptic, whimsical and at times bizzare to people’s own lives.( link to a few of such engagements )

Added to it was an introspective quip — that we always do the indian nods as left — right — left or right — left — right as if doing a march past with our heads. Even when the algorithm affords a fully positive or fully negative flavour to the generated haikus — the act of leaning head thrice in one direction is completely unnatural to us [ unless it’s in a bollywood item number songs ].

The ‘Leaning Haiku’ project exists as an instagram filter open for people to share. Link to Filter.

It has been curated for the Electronic Literature Collection Volume 4 by the Electronic Literature Organization. And all the files related to it are present here in public domain for access on the Electronic Literature Knowledge Base. For any inquiries on the access feel free to contact me in case of a broken link. As a conversation starter — on top of my head is to create an understanding towards computational poetry on social media as part of the e-literature movement[4].

In the future, I am awaiting the release of API access on the SparkAR platform to use existing NPL ML scripts to generate haikus or tests based on users camera feed. Even though it’s possible to do this already using the web AR kit, regardless it lacks the vibrance & reach of user engagement that a social media platform provides. Moreover, creating this extended reality pattern of merging physicality with augmentation & generative texts — has potential to be formed into an experience framework for an immersive design guide/toolkit. I plan to compile all these experiments into a set of starting guidelines to work in the XR space. It will act as a base to initiate further conversations with other designers, thinkers, innovators to build better, engaging & fulfilling experiences of the extended reality in future.

To sum up the outcome of the project, allow me to put forth my haiku skills :

Dancing heads in yellow,

Augmented letters of human mind,

Design the randomness.

Works Cited:

[1] Carolyn Lamb, Daniel G. Brown, Charles L.A. Clarke. A Taxonomy of Generative Poetry Techniques. In Bridges Finland Conference Proceedings, pages 195–202, 2016.

[2] Jacob Harris. Times haiku: Serendipitous poetry from the New York Times. http://haiku.nytimes.com/, accessed November 9, 2021.

[3] Productions SP. Ghost of Tsushima. Sony Interactive Entertainment. 2020.

[4] Berens, K. E-Lit’s #1 Hit: Is Instagram Poetry E-literature? Electronic Book Review. 2019

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

This article was commissioned by Samya Brata Roy.

Authors Bio :

Praveen Sinha has worn multiple hats in his different roles as a creative technologist, digital product designer, design faculty, entrepreneur and new media artist. At present, his interest lies in the critical dialogue and speculative discourse in the Indian design space. Working with technology as a material and humans as instigators of the medium he explores the fringe design spaces as part of a4.achaar collective. Some of his work has been part of PCD Delhi + India(2020), London Biennale(2020),Now On- ProHelvetia(2020) and DigitalFutures at KHOJ(2021).