by Alison Humphrey
”Naturalism is a good word for a bad idea.
Art is to do with transformation”
– Ariane Mnouchkine
In our first and latest posts, we explored how motion capture and real-time animation works. But we haven’t really talked about why one would want to use it in live performance, or what stories it tells best.
These are key questions. Live animation takes a lot of extra work and costs a bomb. It makes it hard to describe the show to theatre audiences (other parts make it hard to explain to game designers and 3D animators). And it affects the story in a fundamental way. Or at least, it should. Otherwise you’re just sprinkling digital pixie dust on top of a play and hoping no one will notice the story would be better told in television or videogame or non-mocap-theatre form.
Throughout the script development process we’ve asked again and again: why are we telling this story with this technology?
Performance capture has traditionally been used for movies with supernatural or fantastical characters: Gollum in Lord of the Rings, the Na’vi in Avatar, Caesar in Rise of the Planet of the Apes, Davy Jones in Pirates of the Caribbean, and the Hulk in The Avengers:
But for a naturalistic human character, filmmakers still usually prefer to shoot a real human actor. This is partly because it’s cheaper, and partly because of the peril of the uncanny valley, wherein the closer a computer-generated model gets to photorealism, the more disturbing it looks:
Human-looking digital doubles are more common in videogames, where the nature of interactive narrative makes it unfeasible to shoot every possible branching story variation:
So how does motion capture fit into theatre? The short answer is, not easily. Modern theatre tends to stick to certain kinds of stories. Kitchen-sink realism has owned the modern stage for generations. I’m not sure why. Maybe because it’s cheaper, maybe because it’s more grown-up and respectable.
But it wasn’t always thus. Before celluloid split drama into two solitudes, stage and screen, theatre was teeming with the supernatural, the fantastical, the mythological, the magical.
Ancient Greek theatre had its satyrs in the satyr plays. The Erinys (the original “Avengers”) in The Eumenides. The god Dionysus in The Bacchae.
Shakespeare put fairies and an animal-headed man into A Midsummer Night’s Dream; witches into Macbeth and Henry VI parts 1 & 2; ghosts into Hamlet, Richard III and Julius Caesar; and spirits into The Tempest.
South and east Asian traditions have kabuki ghosts, the Monkey King, and the Balinese witch Rangda battling the lion-spirit Barong.
In fact, The Lion King director Julie Taymor drew on her early experiences in Bali, and her fascination with Japanese bunraku theatre, when creating the stage version of Disney’s big-cat Hamlet. Her staging feeds the audience’s joy at watching a puppeteer and a puppet at the same time, a phenomenon she calls the “double event”.
Handspring Puppet Company similarly designed its War Horse to reveal the puppeteers within. For my money, that made the stage version infinitely more fun than the Spielberg movie. We know it’s not a real horse, but like Fox Mulder, we want to believe. The same team has created human puppets for plays like Tooth and Nail and Or You Could Kiss Me, but like their animals, these are still stylized rather than naturalistic.
Motion capture has been used on stage by Disney theme parks (Stitch Live!) and Dreamworks musicals (Shrek the Musical’s Magic Mirror), but both of these seek to reproduce characters from animated movies in a live performance setting.
Dance companies have been far more inventive with mocap technology. Two of the earliest experiments were Bill T. Jones’s Ghostcatching (1999), and Merce Cunningham’s Loops (2000), a hands-only dance that brings to mind Samuel Beckett’s waist-up drama Happy Days, neck-up Play, and disembodied mouth monologue Not I.
Faster than Night is similar to these Beckett body-parts in that the real-time animation shows only the head of astronaut Caleb Smith, as he banters with his spaceship’s artificial intelligence and his Earth audience from inside his hibernation pod. But we hope it shares even more with Krapp’s Last Tape – a story that is inextricably enmeshed with the technology used to tell it.
Beckett wrote that play in 1958 after seeing his first reel-to-reel tape recorder at the BBC. He became fascinated, like Atom Egoyan, by “human interaction with technology… the contrast between memory and recorded memory.”
We hope Faster than Night also tells a story about the human interaction with technology. About art and transformation. About escaping the gravity of realism.
What story is that?
Please join us in the theatre on May 3rd to find out… then tell us whether you think the what fit the how. And why.
Melee Hutton (left, in Toronto rehearsal room) with Pascal Langdale (on laptop, as animated Caleb Smith, Skyping in from Stuttgart)
by Francisco-Fernando Granados
With only 3 days until our public presentation, things are busy, but coming together. The interactive script is live online as a Google Form that can be filled out by the public. There’s still time to participate.
Live on stage, the lights are set as rehearsals go into their third day. Here are some images from the Studio Theatre with my collaborators Manolo Lugo & Maryam Taghavi:
Photo of #legacy by Greg Wong
by Rob Kempson
#legacy opened and closed on April 12th. To say that it was a moving experience would be the understatement of the year. The opportunity to work with these incredibly brave and talented women in such a supportive atmosphere is a process that I will cherish for a long time. If you want to see more about how the week-long residency looked, check out my Storify post. As a follow-up post, I’ve decided to share the three letters that closed the piece–one from each of the women. They are powerful, poignant, and reflective of a legacy that transcends Twitter. This week, I’m sharing Judith’s.
Do you remember the story I told you about my childhood teddy bear, John? I took him everywhere with me.
It was war time when I was born and my father, who was in the RAF, was away from home. He was given special leave to come home for a few days with the family. He brought John with him. John was my first gift. I loved my bear and nothing in the world would have made me part with him at any age. He helped me with my homework, came on family picnics, went to bed with me and even helped me through my nursing training.
During our removal to Canada, he was packed in one of our big shipping crates. Alas it was John’s crate that went missing during the long Atlantic sea crossing. I had to fashion my life as an immigrant in a strange country without him.
When I told you the story of John, my old teddy bear, I never dreamed you would be paying so much attention to it, because you were only five years old. You asked me to draw a coloured picture of John so that you could find me a new bear that looked like him. I was then presented with John II. Now ten years later, I am involved in a project that has made me remember all of this.
For me, John ll is as good, or better, than John l, because it was given as a legacy of love.
Love from Grandma
by Alison Humphrey
As far as we’re aware, Faster than Night is one of the first handful of theatre productions in the world to use facial performance capture live on stage. But while from one perspective it is cutting-edge technology, it is also just the latest mutation of an artform that has been used in theatre for millennia: the mask.
A mask is simply non-living material sculpted into the shape of a face. It can allow a human performer to transform into a different human, or an animal, or a supernatural being.
“Ko-jo” (old man) Noh theatre mask (Children’s Museum of Indianapolis)
It can allow a young person to play an old person, or a man to play a woman.
It can define a character by a single facial expression (0r if the mask-maker is very skilled, several expressions depending on the viewing angle).
In the Balinese tradition of topeng pajegan, a single dancer portrays a succession of masked characters with different personalities: the old man, the king, the messenger, the warrior, the villager. A whole epic story can be told by one skilled performer, simply by switching masks and physicalities.
Just as animation is a series of still drawings, or film is a series of still photos projected fast enough to create the illusion of movement, real-time facial capture is fundamentally a process of switching and morphing between dozens of digital “masks” – at rates of up to 120 frames per second.
But how does real-time facial capture actually work?
We’re working with the Dynamixyz Performer suite of software, which takes live video from a head-mounted camera, and analyzes the video frames of the actor’s face. In the image below, you can see it tracking elements such as eyes, eyebrows and lips.
For each frame of video, the software finds the closest match between the expression on the live actor’s face, and a keyframe in a pre-recorded library of that actor’s expressions, called a “range of motion”. That library keyframe of the actor corresponds to another keyframe (also called a blendshape) of the 3D CGI character making the same expression. By analyzing these similarities, the software can “retarget” the performance frame by frame from live actor to virtual character, morphing between blendshapes in seamless motion. This process is called morph target animation.
Here are a few rough first-draft keyframes for Faster than Night. On the right is a head-cam video frame of Pascal Langdale, and on the left is an animation keyframe by Lino Stephen of Centaur Digital:
There are dozens more expressions in the “range of motion” library for this particular actor / character pair. Some of them are “fundamental expressions”, drawing on the Facial Action Coding System developed by behavioural psychologists Paul Ekman and Wallace V. Friesen in 1978:
Tim Roth played a character inspired by Paul Ekman in the 2009 TV series Lie To Me
Other facial “poses” convey more subtle or secondary expressions…
…while still other expressions represent phonemes, the building blocks of lip-sync, as found in traditional animation:
This image of Aardman Animation’s stop-motion character Morph gives a tangible metaphor for what’s going on inside the computer during the real-time animation process:
With each frame, a new head is taken out of its box and put on the character, just like the topeng performer switching masks.
The cumulative effect creates the illusion of speech, of motion, of emotion… of life:
A longer post is coming soon with details on the 3D facial model work being done by Centaur Digital and Dynamixyz, but to tide you over, here are a few more elements from our talented creative team.
First up, the design of the starship Envoy by Mike Nesbitt and Caroline Stephenson of Capture Scratch:
Next, “Like Clockwork” by composer and audio designer Will Mountain, via Vapor Music:
And our latest piece of concept art was developed by Clementine Konarzewski, who is playing the voice of astronaut Caleb Smith’s daughter, Katy, age 6:
by Francisco-Fernando Granados
If the difference between performance and any other medium is the folding together of the time of the making of the work with that of its showing, the difference between theatre and performance art might be a similar folding together of the space that separates the public and the performers. In The Ballad of ____ B, the stage is a world shared by artists and audience. The seating area of the Harbourfront Studio Theatre will be closed, the curtains will be drawn on the stage, and the public will inhabit the space of the action as they experience it. Two rows of seats will flank the performance space.
The audience may sit or stand, or walk around as the piece takes place. The images on this post come from a series of Photoshop studies attempting to figure out different spatial configurations for the piece. While the history of drama likely has many precedents for this kind of rearrangement of theatrical space, for me, from the perspective of visual art, this approach comes from a desire to import conventions of action art as a way to try to think about the theatre as a specific site: performance in the expanded field.
In action art, audiences are conventionally invited to approach the performance in the same way they would approach any other work of visual art: the piece is structured through a conceptual process or a succession of actions that build up to a tableau rather than through narrative. This allows the audience to wander around the piece, to weave in and out of it as they walk around the gallery. In the type of durational practices that I’m most familiar with, time itself becomes the tread, and the experience of the performance is often the contemplative witnessing of the making of piece. For some people, this experience might be just a couple of minutes, and for others, it may be as long as the work itself.
Toronto has a legendary history of durational work. Paul Couillard, whose own work as an international artist has extended over the course of days and months, curated a series of long-form live works called TIME TIME TIME for FADO Performance Art Centre in 1999. One of the artists in the series, Tanya Mars, has a trajectory that has ranged from early cabaret-based and multimedia theatrical pieces to the monumental tableaus of her oeuvre over the last 15 years. Indeed, FADO just recently focused its Emerging Artist Series on the ways duration is being used by a generation of younger artists.
As part of that generation, I’m interested in thinking about performance in the expanded field as a practice where the hierarchy between theatre and performance is reframed into a palette with the broadest range of possibilities, where the combination of bodies, time, and space can be applied in ways that respond and propose specifically in terms of the situation at hand.
by Melissa D’Agostino
Hello out there! Thanks for following along on my journey developing BroadFish for HatchTO.
Creating a new theatrical work is a crazy, roller coaster ride of an experience.
When I began working on BroadFish, many months ago I thought I was making a modern-day folktale or fairy tale. Most of my self-generated work has involved broad (no pun intended) characters based in clown, bouffon and physical theatre forms. I thought I was going to that place again, and that’s how I was approaching the project.
As I began doing research online via Twitter using the hashtag #BridesNeed2Know:
A question I posed to the Twitter-verse about Say Yes to the Dress.
The response I got (in record time) from the Production Company that makes Say Yes to the Dress
And through the incredible response I got to my blog posts on praxistheatre.com, the piece started shifting.
New works are shifty. They take on a life of their own.
In the midst of my research I came across Anita Chakraburtty – a woman in Australia who was planning her wedding without a groom in site.
I became obsessed with her. In a way that was surprising to me. And everyone around me, I think.
I wanted to understand her and why she was making these choices. I wanted to know everything about her. At first, I wasn’t clear on why this was so important to me, and now I believe I might.
And so, BroadFish has actually become about my obsession with Anita, and by extension our obsession with Fairy Tales, and magic, and weddings.
It’s also about how often we mock or attack other people’s decisions to project our fears, avoid our own vulnerability, or justify our own decisions — AND how the internet can facilitate this distancing we do from one another.
In making the show I’ve used almost every social media tool I could think of: Twitter, Facebook, WordPress, Pinterest, YouTube, Instagram, Voxer, Storify, Google+, Skype, FaceTime… I think I only really avoided Reddit. Because… well… Reddit.
I thank all of you who read these blog posts, responded on Twitter or Facebook or here on WordPress. You have played an integral part of the development process, and will continue to do so as I work on this piece during our residency week, and beyond HATCH.
It would thrill me if you would join us on Saturday, April 19th at 8pm for our one (and only) showing of this stage of the BroadFish project.
And let me know your thoughts on Anita @MelissaDags #BroadFish #HatchTO.
hand hatch for an interview with Harbourfront Centre
if letters were lovers
Catherine of Aragon and Anne Boleyn portrait comparison
except from a collaboration with Nathalie Lozano
diagram from Rosalind Krauss’ Sculpture in the Expanded Field, translated in Spanish
Sunday Scene @ The Power Plant, September 1, 2013
Callas as Anne Boleyn
bits of Spivak
study for The Ballad of ______ B (outtake), 2013; photograph by Manolo Lugo
Rob Kempson, hard at work
By Samantha Serles, Dramaturg
I am the dramaturg for #legacy by Rob Kempson, in residence this week at Harbourfront Centre as part of HATCH 2014! I think this project is fascinating because it combines so many disparate elements: integrating older adults who don’t identify as artists, technology, and social media into a new play development process. Part of my work has been to facilitate the creation of a “Twitter script” – a series of tweets that will be sent from the performer’s twitter accounts during both performances on Saturday, April 12.
Audience members with smartphones (and Twitter) will be able to follow the performer’s tweets from designated “Tweet seats”. For those who can’t follow on Twitter (or do not want to) the Twitter script will also be projected on stage at specific points through out the play. Some of the tweets provide subtext or context to what is being on said on stage. Sometimes they share links to songs, articles about issues mentioned in the play, even a funny cat video. For audience members who choose to follow along, the Twitter script will function a bit like Pop Up Video, the popular 90’s TV show that shared “info nuggets” during music videos.
To help facilitate the creation of the Twitter script I gave the women a list of questions that related to the themes or ideas that they discuss in the play. I asked them to respond to each question in the form of a tweet. We then went through the script and arranged the tweets so that they would emphasis what was being said on stage, or sometimes provide a contradictory point of view.
People in the “tweet seats” will be encouraged to respond via twitter while they watch the play. In one scene, the tweets from the audience will be projected on stage and the women will improvise in response to what they read. This is where the experimental nature of HATCH becomes apparent, as well as the courage of the three women we are working with. Not only are they writing and performing — though they don’t identify as writers or performers — they are willing to come in direct contact with audience reactions to their show while on stage. I am more and more in awe of them every day!
The making of “twitter script”
It’s interesting to me that our twitter script and the audience’s responses will also become a record of the play. I don’t know if Joan, Judith, and Donna will continue to tweet after April 12th. Judith has already stated that her best tweet will be at midnight on April 12th and it will read: “Thank god I never have to tweet again!” But even if their accounts lay abandoned, the legacy of this creation process will remain in the twitterverse. It serves as an example of how the immense amount of content many of us generate and share every day becomes part of our own online legacy.
Now that’s something to think about before you post another funny cat video, or a picture of what you made for lunch!
The public presentations of #legacy are at 2pm & 8pm this coming Saturday, April 12th. Click here for more information & the link to buy tickets.
by Pascal Langdale
In the UK, I’d be called a “jobbing actor”. That means that I work across all sorts of media, picking up a wide variety of acting work that provides a steady income. I have worked in corporates and commercials, stage, film and TV, and interactive games. Going a little further back, I have published poetry, written a radio play, co-produced movement-based theatre, headed a flamenco company, and once danced with Cyd Charisse. In my early thirties I studied nonverbal behavior and re-appraised my acting techniques. When I say I’m a RADA graduate, I sometimes think people expect a more traditional actor – a style or a working process which I can also embrace when necessary.
Every form I’ve worked in has certain established rules and conventions, developed over years (or centuries) to serve the best interests of storytelling within that medium. Each has its own artifice, relies on a shared experience, and requires adaptation of the core craft of acting.
So it was with a jobbing actor’s irreligious approach that I made friends with the newcomer on the storytelling scene: performance capture. In this medium, the rules and conventions are still being established, found wanting, re-established, changed, superseded and hotly debated.
In Faster than Night, my character Caleb Smith, a cross between Tony Stark (Iron Man) and Chris Hadfield (THE man), relies on social media to help decide his fate in a life-or-death situation. Animated in real time.
With this show, we’re exploring whether performance capture can play nice with traditional theatre to create a new form of interactive storytelling for live (or live-streamed) entertainment, and a new role for an engaged audience.
Now, no individual element of our production is entirely new. Faster than Night‘s constituent parts have a broad history. Our live-animation technology is cutting edge, the result of an explosion of development in the field of facial capture and analysis over the past twenty years. In the past three, the range of capture systems available has expanded, offering greater options for quality at differing price-points. Facial capture tech is becoming democratised, and this key component of human-driven animation is finally reaching the hands of a new generation of artists and producers.
As soon as facial capture became advanced enough to animate in real time with some quality, the potential to fuse video-game and film technology with theatrical storytelling became inevitable. Theatre has always grabbed whatever innovative tech could help tell the story – from Greek masks that allowed a character to be heard and seen from the back of a large open-air amphitheatre, to painted backdrops and gaslight.
Why should we stop with digital capture? In recent years video games have created a desire for direct influence over a narrative, and social media has provided a platform to share your feelings about it. Where earlier artists would have needed a show of hands to decide a voted ending, Twitter allows us to canvass the opinions of countless viewers. Tweets have been used as source material in theatre productions (#legacy, one of our fellow HATCH productions, being the most recent – so recent it hasn’t even opened yet!). The viewer-poll competition format, used most famously on American Idol, is spreading to shows like Opposite Worlds or even the scripted TV drama Continuum.
Interactive story as created by video-game developers, and writers of choose-your-own-adventure books, must fix its narratives in stone. However beautifully executed, they can only give a finely-crafted illusion of unlimited freedom. An ancestor of the interactive game can be found in theatre, which has a longer heritage of improvisation and audience participation. The British have a long history of music hall and panto, where a rougher but no-less-organised form of audience participation is part of the entertainment. (Did I mention my first job at the age of 17 was in a pantomime?)
The first vote-based multiple-ending play was written by Ayn Rand in 1935, a courtroom drama in which the jury was drawn from the audience. In the 1970’s, Augusto Boal anticipated the internet-enabled art of flash mobs with Invisible Theatre, “in which an event is planned and scripted but does not allow the spectators to know that the event is happening. Actors perform out-of-the-ordinary roles which invite spectators to join in or sit back and watch.”
Despite the use of Twitter interaction or facial capture animation, our core goal remains primal: to tell a good story. If we fail at that, all the cutting-edge tools we might use become a mere distraction.
Yet wherever linear narrative is challenged, sharing a satisfying story becomes notoriously difficult. I will be learning forty pages of a script that occasionally leaps into improvisation with the audience through their ambassador, @ISMEEtheAI, voiced by Melee Hutton. Learning a script is a challenge, but it’s one I at least know the measure of. Playing an interactively-led character presents a number of far less familiar challenges, which (even more than the performance capture) is why this show is particularly experimental. The interactive aspects demand our greatest attention, and our boldest moves.
Here’s an example – a scan of my own re-typed script from Heavy Rain, an interactive game or movie with multiple narrative paths that led to one of twenty or so differing endings.
As an actor, and as a person, understanding behaviour relies on a certain level of causality. For example, a mood: “I’m in a bad mood, so I snap at my partner.” Or a learnt pre-condition: “I had a violent father, so I struggle with authority.” Or a hardwired precondition: “I am genetically predisposed to bouts of euphoria.” All these are examples of what can cause behavior.
In the script above, the player had three choices for how my character Ethan Mars could interact with an unknown “helper”, Madison. Going from left to right, Ethan (1) seeks basic info, (2) wonders at her selflessness, and (3) suspects her motives. These are quite different (although there are more examples of more extreme differences elsewhere), and demand that the acting choice in the moment before the player choice be appropriate for all three options. Moreover, each acting choice must also finish off in a way that is consistent with Madison’s response.
Ethan Mars and Madison Paige, Heavy Rain
The lack of pre-decision required in this situation is not as foreign to an actor as one might think. Many actors strive to be “in the moment,” to imitate life itself. Not knowing a character’s full behavioral palette is also not uncommon. Playing the character Karl Marsten in Bitten, I did not receive scripts for all thirteen episodes in advance. This is par for the course for a TV series, but even though this one was based on a series of novels I could pick up and read anytime, some characters’ fates departed radically from those in the books, in order to better serve the unique needs of television storytelling.
When a pre-scripted, pre-recorded game story with multiple endings is developed, the creative team try to set up a balance among the player’s possible choices, a “neutral”, making sure that each of them is equally plausible and possible. The nature of live theatre means we don’t need an astronomical budget to shoot every possible outcome. This lets us open up to more variety of audience input, more freedom, more chaos.
Live theatre also means the audience’s final choices can no longer have guaranteed neutral preconditions, because they may have been biased by an unexpected experience that night, something that didn’t happen any other night. A comment, a look, a pause, even a cellphone ring, could pull the audience’s attention away from a vital piece of balancing information, or push them towards a particular relationship with Caleb, ISMEE, Xiao or Dmitri. Every show changes, because every audience changes.
So we need you. Yes, we need people with smartphones who, if not already familiar with Twitter, are willing to give it a try. But more than that we need an audience willing to engage. Become an active participant, and if the experiment is successful, you’ll come away feeling emotions that are harder to come by with passive entertainment: guilt, endorsement, responsibility, vindication, shame, or triumph.
If you’re up for that, start following @ISMEEtheAI on Twitter, and bring your phone along to Harbourfront on May 3rd, ready to participate in your own unique experience of our show.
Pascal Langdale is an actor, producer and writer on Faster than Night.