by Alison Humphrey
As far as we’re aware, Faster than Night is one of the first handful of theatre productions in the world to use facial performance capture live on stage. But while from one perspective it is cutting-edge technology, it is also just the latest mutation of an artform that has been used in theatre for millennia: the mask.
A mask is simply non-living material sculpted into the shape of a face. It can allow a human performer to transform into a different human, or an animal, or a supernatural being.
“Ko-jo” (old man) Noh theatre mask (Children’s Museum of Indianapolis)
It can allow a young person to play an old person, or a man to play a woman.
It can define a character by a single facial expression (0r if the mask-maker is very skilled, several expressions depending on the viewing angle).
In the Balinese tradition of topeng pajegan, a single dancer portrays a succession of masked characters with different personalities: the old man, the king, the messenger, the warrior, the villager. A whole epic story can be told by one skilled performer, simply by switching masks and physicalities.
Just as animation is a series of still drawings, or film is a series of still photos projected fast enough to create the illusion of movement, real-time facial capture is fundamentally a process of switching and morphing between dozens of digital “masks” – at rates of up to 120 frames per second.
But how does real-time facial capture actually work?
We’re working with the Dynamixyz Performer suite of software, which takes live video from a head-mounted camera, and analyzes the video frames of the actor’s face. In the image below, you can see it tracking elements such as eyes, eyebrows and lips.
For each frame of video, the software finds the closest match between the expression on the live actor’s face, and a keyframe in a pre-recorded library of that actor’s expressions, called a “range of motion”. That library keyframe of the actor corresponds to another keyframe (also called a blendshape) of the 3D CGI character making the same expression. By analyzing these similarities, the software can “retarget” the performance frame by frame from live actor to virtual character, morphing between blendshapes in seamless motion. This process is called morph target animation.
Here are a few rough first-draft keyframes for Faster than Night. On the right is a head-cam video frame of Pascal Langdale, and on the left is an animation keyframe by Lino Stephen of Centaur Digital:
There are dozens more expressions in the “range of motion” library for this particular actor / character pair. Some of them are “fundamental expressions”, drawing on the Facial Action Coding System developed by behavioural psychologists Paul Ekman and Wallace V. Friesen in 1978:
Tim Roth played a character inspired by Paul Ekman in the 2009 TV series Lie To Me
Other facial “poses” convey more subtle or secondary expressions…
…while still other expressions represent phonemes, the building blocks of lip-sync, as found in traditional animation:
This image of Aardman Animation’s stop-motion character Morph gives a tangible metaphor for what’s going on inside the computer during the real-time animation process:
With each frame, a new head is taken out of its box and put on the character, just like the topeng performer switching masks.
The cumulative effect creates the illusion of speech, of motion, of emotion… of life:
A longer post is coming soon with details on the 3D facial model work being done by Centaur Digital and Dynamixyz, but to tide you over, here are a few more elements from our talented creative team.
First up, the design of the starship Envoy by Mike Nesbitt and Caroline Stephenson of Capture Scratch:
Next, “Like Clockwork” by composer and audio designer Will Mountain, via Vapor Music:
And our latest piece of concept art was developed by Clementine Konarzewski, who is playing the voice of astronaut Caleb Smith’s daughter, Katy, age 6:
by Francisco-Fernando Granados
If the difference between performance and any other medium is the folding together of the time of the making of the work with that of its showing, the difference between theatre and performance art might be a similar folding together of the space that separates the public and the performers. In The Ballad of ____ B, the stage is a world shared by artists and audience. The seating area of the Harbourfront Studio Theatre will be closed, the curtains will be drawn on the stage, and the public will inhabit the space of the action as they experience it. Two rows of seats will flank the performance space.
The audience may sit or stand, or walk around as the piece takes place. The images on this post come from a series of Photoshop studies attempting to figure out different spatial configurations for the piece. While the history of drama likely has many precedents for this kind of rearrangement of theatrical space, for me, from the perspective of visual art, this approach comes from a desire to import conventions of action art as a way to try to think about the theatre as a specific site: performance in the expanded field.
In action art, audiences are conventionally invited to approach the performance in the same way they would approach any other work of visual art: the piece is structured through a conceptual process or a succession of actions that build up to a tableau rather than through narrative. This allows the audience to wander around the piece, to weave in and out of it as they walk around the gallery. In the type of durational practices that I’m most familiar with, time itself becomes the tread, and the experience of the performance is often the contemplative witnessing of the making of piece. For some people, this experience might be just a couple of minutes, and for others, it may be as long as the work itself.
Toronto has a legendary history of durational work. Paul Couillard, whose own work as an international artist has extended over the course of days and months, curated a series of long-form live works called TIME TIME TIME for FADO Performance Art Centre in 1999. One of the artists in the series, Tanya Mars, has a trajectory that has ranged from early cabaret-based and multimedia theatrical pieces to the monumental tableaus of her oeuvre over the last 15 years. Indeed, FADO just recently focused its Emerging Artist Series on the ways duration is being used by a generation of younger artists.
As part of that generation, I’m interested in thinking about performance in the expanded field as a practice where the hierarchy between theatre and performance is reframed into a palette with the broadest range of possibilities, where the combination of bodies, time, and space can be applied in ways that respond and propose specifically in terms of the situation at hand.
by Melissa D’Agostino
Hello out there! Thanks for following along on my journey developing BroadFish for HatchTO.
Creating a new theatrical work is a crazy, roller coaster ride of an experience.
When I began working on BroadFish, many months ago I thought I was making a modern-day folktale or fairy tale. Most of my self-generated work has involved broad (no pun intended) characters based in clown, bouffon and physical theatre forms. I thought I was going to that place again, and that’s how I was approaching the project.
As I began doing research online via Twitter using the hashtag #BridesNeed2Know:
A question I posed to the Twitter-verse about Say Yes to the Dress.
The response I got (in record time) from the Production Company that makes Say Yes to the Dress
And through the incredible response I got to my blog posts on praxistheatre.com, the piece started shifting.
New works are shifty. They take on a life of their own.
In the midst of my research I came across Anita Chakraburtty – a woman in Australia who was planning her wedding without a groom in site.
I became obsessed with her. In a way that was surprising to me. And everyone around me, I think.
I wanted to understand her and why she was making these choices. I wanted to know everything about her. At first, I wasn’t clear on why this was so important to me, and now I believe I might.
And so, BroadFish has actually become about my obsession with Anita, and by extension our obsession with Fairy Tales, and magic, and weddings.
It’s also about how often we mock or attack other people’s decisions to project our fears, avoid our own vulnerability, or justify our own decisions — AND how the internet can facilitate this distancing we do from one another.
In making the show I’ve used almost every social media tool I could think of: Twitter, Facebook, WordPress, Pinterest, YouTube, Instagram, Voxer, Storify, Google+, Skype, FaceTime… I think I only really avoided Reddit. Because… well… Reddit.
I thank all of you who read these blog posts, responded on Twitter or Facebook or here on WordPress. You have played an integral part of the development process, and will continue to do so as I work on this piece during our residency week, and beyond HATCH.
It would thrill me if you would join us on Saturday, April 19th at 8pm for our one (and only) showing of this stage of the BroadFish project.
And let me know your thoughts on Anita @MelissaDags #BroadFish #HatchTO.
hand hatch for an interview with Harbourfront Centre
if letters were lovers
Catherine of Aragon and Anne Boleyn portrait comparison
except from a collaboration with Nathalie Lozano
diagram from Rosalind Krauss’ Sculpture in the Expanded Field, translated in Spanish
Sunday Scene @ The Power Plant, September 1, 2013
Callas as Anne Boleyn
bits of Spivak
study for The Ballad of ______ B (outtake), 2013; photograph by Manolo Lugo
Rob Kempson, hard at work
By Samantha Serles, Dramaturg
I am the dramaturg for #legacy by Rob Kempson, in residence this week at Harbourfront Centre as part of HATCH 2014! I think this project is fascinating because it combines so many disparate elements: integrating older adults who don’t identify as artists, technology, and social media into a new play development process. Part of my work has been to facilitate the creation of a “Twitter script” – a series of tweets that will be sent from the performer’s twitter accounts during both performances on Saturday, April 12.
Audience members with smartphones (and Twitter) will be able to follow the performer’s tweets from designated “Tweet seats”. For those who can’t follow on Twitter (or do not want to) the Twitter script will also be projected on stage at specific points through out the play. Some of the tweets provide subtext or context to what is being on said on stage. Sometimes they share links to songs, articles about issues mentioned in the play, even a funny cat video. For audience members who choose to follow along, the Twitter script will function a bit like Pop Up Video, the popular 90’s TV show that shared “info nuggets” during music videos.
To help facilitate the creation of the Twitter script I gave the women a list of questions that related to the themes or ideas that they discuss in the play. I asked them to respond to each question in the form of a tweet. We then went through the script and arranged the tweets so that they would emphasis what was being said on stage, or sometimes provide a contradictory point of view.
People in the “tweet seats” will be encouraged to respond via twitter while they watch the play. In one scene, the tweets from the audience will be projected on stage and the women will improvise in response to what they read. This is where the experimental nature of HATCH becomes apparent, as well as the courage of the three women we are working with. Not only are they writing and performing — though they don’t identify as writers or performers — they are willing to come in direct contact with audience reactions to their show while on stage. I am more and more in awe of them every day!
The making of “twitter script”
It’s interesting to me that our twitter script and the audience’s responses will also become a record of the play. I don’t know if Joan, Judith, and Donna will continue to tweet after April 12th. Judith has already stated that her best tweet will be at midnight on April 12th and it will read: “Thank god I never have to tweet again!” But even if their accounts lay abandoned, the legacy of this creation process will remain in the twitterverse. It serves as an example of how the immense amount of content many of us generate and share every day becomes part of our own online legacy.
Now that’s something to think about before you post another funny cat video, or a picture of what you made for lunch!
The public presentations of #legacy are at 2pm & 8pm this coming Saturday, April 12th. Click here for more information & the link to buy tickets.
by Pascal Langdale
In the UK, I’d be called a “jobbing actor”. That means that I work across all sorts of media, picking up a wide variety of acting work that provides a steady income. I have worked in corporates and commercials, stage, film and TV, and interactive games. Going a little further back, I have published poetry, written a radio play, co-produced movement-based theatre, headed a flamenco company, and once danced with Cyd Charisse. In my early thirties I studied nonverbal behavior and re-appraised my acting techniques. When I say I’m a RADA graduate, I sometimes think people expect a more traditional actor – a style or a working process which I can also embrace when necessary.
Every form I’ve worked in has certain established rules and conventions, developed over years (or centuries) to serve the best interests of storytelling within that medium. Each has its own artifice, relies on a shared experience, and requires adaptation of the core craft of acting.
So it was with a jobbing actor’s irreligious approach that I made friends with the newcomer on the storytelling scene: performance capture. In this medium, the rules and conventions are still being established, found wanting, re-established, changed, superseded and hotly debated.
In Faster than Night, my character Caleb Smith, a cross between Tony Stark (Iron Man) and Chris Hadfield (THE man), relies on social media to help decide his fate in a life-or-death situation. Animated in real time.
With this show, we’re exploring whether performance capture can play nice with traditional theatre to create a new form of interactive storytelling for live (or live-streamed) entertainment, and a new role for an engaged audience.
Now, no individual element of our production is entirely new. Faster than Night‘s constituent parts have a broad history. Our live-animation technology is cutting edge, the result of an explosion of development in the field of facial capture and analysis over the past twenty years. In the past three, the range of capture systems available has expanded, offering greater options for quality at differing price-points. Facial capture tech is becoming democratised, and this key component of human-driven animation is finally reaching the hands of a new generation of artists and producers.
As soon as facial capture became advanced enough to animate in real time with some quality, the potential to fuse video-game and film technology with theatrical storytelling became inevitable. Theatre has always grabbed whatever innovative tech could help tell the story – from Greek masks that allowed a character to be heard and seen from the back of a large open-air amphitheatre, to painted backdrops and gaslight.
Why should we stop with digital capture? In recent years video games have created a desire for direct influence over a narrative, and social media has provided a platform to share your feelings about it. Where earlier artists would have needed a show of hands to decide a voted ending, Twitter allows us to canvass the opinions of countless viewers. Tweets have been used as source material in theatre productions (#legacy, one of our fellow HATCH productions, being the most recent – so recent it hasn’t even opened yet!). The viewer-poll competition format, used most famously on American Idol, is spreading to shows like Opposite Worlds or even the scripted TV drama Continuum.
Interactive story as created by video-game developers, and writers of choose-your-own-adventure books, must fix its narratives in stone. However beautifully executed, they can only give a finely-crafted illusion of unlimited freedom. An ancestor of the interactive game can be found in theatre, which has a longer heritage of improvisation and audience participation. The British have a long history of music hall and panto, where a rougher but no-less-organised form of audience participation is part of the entertainment. (Did I mention my first job at the age of 17 was in a pantomime?)
The first vote-based multiple-ending play was written by Ayn Rand in 1935, a courtroom drama in which the jury was drawn from the audience. In the 1970’s, Augusto Boal anticipated the internet-enabled art of flash mobs with Invisible Theatre, “in which an event is planned and scripted but does not allow the spectators to know that the event is happening. Actors perform out-of-the-ordinary roles which invite spectators to join in or sit back and watch.”
Despite the use of Twitter interaction or facial capture animation, our core goal remains primal: to tell a good story. If we fail at that, all the cutting-edge tools we might use become a mere distraction.
Yet wherever linear narrative is challenged, sharing a satisfying story becomes notoriously difficult. I will be learning forty pages of a script that occasionally leaps into improvisation with the audience through their ambassador, @ISMEEtheAI, voiced by Melee Hutton. Learning a script is a challenge, but it’s one I at least know the measure of. Playing an interactively-led character presents a number of far less familiar challenges, which (even more than the performance capture) is why this show is particularly experimental. The interactive aspects demand our greatest attention, and our boldest moves.
Here’s an example – a scan of my own re-typed script from Heavy Rain, an interactive game or movie with multiple narrative paths that led to one of twenty or so differing endings.
As an actor, and as a person, understanding behaviour relies on a certain level of causality. For example, a mood: “I’m in a bad mood, so I snap at my partner.” Or a learnt pre-condition: “I had a violent father, so I struggle with authority.” Or a hardwired precondition: “I am genetically predisposed to bouts of euphoria.” All these are examples of what can cause behavior.
In the script above, the player had three choices for how my character Ethan Mars could interact with an unknown “helper”, Madison. Going from left to right, Ethan (1) seeks basic info, (2) wonders at her selflessness, and (3) suspects her motives. These are quite different (although there are more examples of more extreme differences elsewhere), and demand that the acting choice in the moment before the player choice be appropriate for all three options. Moreover, each acting choice must also finish off in a way that is consistent with Madison’s response.
Ethan Mars and Madison Paige, Heavy Rain
The lack of pre-decision required in this situation is not as foreign to an actor as one might think. Many actors strive to be “in the moment,” to imitate life itself. Not knowing a character’s full behavioral palette is also not uncommon. Playing the character Karl Marsten in Bitten, I did not receive scripts for all thirteen episodes in advance. This is par for the course for a TV series, but even though this one was based on a series of novels I could pick up and read anytime, some characters’ fates departed radically from those in the books, in order to better serve the unique needs of television storytelling.
When a pre-scripted, pre-recorded game story with multiple endings is developed, the creative team try to set up a balance among the player’s possible choices, a “neutral”, making sure that each of them is equally plausible and possible. The nature of live theatre means we don’t need an astronomical budget to shoot every possible outcome. This lets us open up to more variety of audience input, more freedom, more chaos.
Live theatre also means the audience’s final choices can no longer have guaranteed neutral preconditions, because they may have been biased by an unexpected experience that night, something that didn’t happen any other night. A comment, a look, a pause, even a cellphone ring, could pull the audience’s attention away from a vital piece of balancing information, or push them towards a particular relationship with Caleb, ISMEE, Xiao or Dmitri. Every show changes, because every audience changes.
So we need you. Yes, we need people with smartphones who, if not already familiar with Twitter, are willing to give it a try. But more than that we need an audience willing to engage. Become an active participant, and if the experiment is successful, you’ll come away feeling emotions that are harder to come by with passive entertainment: guilt, endorsement, responsibility, vindication, shame, or triumph.
If you’re up for that, start following @ISMEEtheAI on Twitter, and bring your phone along to Harbourfront on May 3rd, ready to participate in your own unique experience of our show.
Pascal Langdale is an actor, producer and writer on Faster than Night.
by Alison Humphrey
“Interactivity” is one of those words.
What does it mean in theatre?
What does it mean in storytelling?
What does it mean to you?
As we develop Faster than Night, which was conceived with the slippery and baffling ambition to involve the audience in the story, I’ve been thinking a lot about what it means to me.
When I was sixteen, there was a comic convention in downtown Toronto. This was way back before comics were cool, a good decade before the invention of the web let geeks find each other en masse, and longer before the San Diego Comic-Con went media-tastic and started attracting 130,000 paying attendees. We’re talking the basement of the Hilton, a bunch of folding tables with dealers selling back-issues from boxes, and a special guest or three.
I was a huge fan of Marvel Comics’s Uncanny X-Men, which was ahead of the curve in introducing strong female superheroes. (One of those heroines, the teenage Kitty Pryde, was the original protagonist of one of my favorite X-Men storylines, ”Days of Future Past“. Yet when its blockbuster movie adaptation is released next month, the very male and very box office-friendly character Wolverine will be taking her role in the narrative. That’s a different blog post…)
I’d heard that legendary comics inker Terry Austin was going to be at the convention, and decided I would make a Fimo figurine of one of his characters as a gift.
He had left the X-Men a couple of years prior, so I sculpted a character from his new gig, Alien Legion. I wasn’t as much of a fan of that book, but Sarigar was a half-snake alien – an awesome challenge that involved a coat-hanger armature and a lot of finger-crossing in the firing process.
I snapped some photos before I took it downtown and gave it to one of my real-life heroes. He was gracious and appreciative, giving me some original art in return.
I started thinking about that formative early moment last June, when the Royal Shakespeare Company partnered with Google+ on an ambitious, interactive theatre/social media hybrid called Midsummer Night’s Dreaming (a followup to 2010′s Such Tweet Sorrow, remixing Romeo and Juliet via Twitter). I’d heard about the concept from the RSC’s forward-thinking digital producer Sarah Ellis, and was instantly fascinated.
Google’s Tom Uglow (co-director with the RSC’s Geraldine Collinge) wrote eloquently beforehand about Why we’re doing it:
We are inviting everyone on the internet to take part. We’d rather like 10,000 contributors extending the RSC across the world, commenting, captioning, or penning a lonely heart column for Helena. Maybe people will invent their own characters. Or make fairy cupcakes; share photos of their dearest darlings as changelings; send schoolboy marginalia about “wooing with your sword”; compose florid poetry to Lysander’s sister; or debate with Mrs Quince on declamation. Or just watch online….
I followed the project as it unfolded online over the course of midsummer weekend, enjoying the original material commissioned from professional writers and artists (“2000 pieces of material for 30 new characters to be shared online non-stop for 72 hours”), and even Photoshopping a few memes and lolcats of my own.
It was fun, and it certainly taught me a lot about the mechanics of Google+ as a social media platform. There were lots of thoughtful analyses after it was over, but for me the most interesting aspect was the strange mix of emotions I was feeling around having participated.
How does the audience feel crossing the line that usually separates them from the professionals providing their entertainment? Passive observation is safe. Active creativity is not. As kids, we all start out as unselfconscious artists, writers, musicians and dancers. But in adolescence, when social pressures and fears kick in, most of us transition into observers and “consumers” of culture made by other people.
For me, the Midsummer Night’s Dreaming project provoked the complicated mix of reactions such invitations always do – the thrill of inclusion or transgression, yes, but also the fear that my contribution won’t be “good enough” or that too much enthusiasm will make me uncool.
More than anything, it brought me back to that kid sitting in her room, creating something to give to an artist whose work she admired. Yes, part of that was the fan hoping to garner the attention of the idol, however briefly. And part was the desire to immerse in the fictional world. But it was also just wanting to step into the creative sandbox and play.
Many people think of interactive story in terms of the classic 80′s Choose Your Own Adventure books. I certainly flipped around their pages and beat my head against the Infocom computer game adaptation of Douglas Adams’s Hitchhiker’s Guide to the Galaxy long enough to pick up the basic mechanics of branching narrative.
Adams took his next step with interactive fiction in 1998 with the Myst-like Starship Titanic, for which I had the dream job of producing an interlocking set of fictional websites to promote the CD-ROM game in the year leading up to its release. These websites were an early version of what would later be dubbed “alternate reality games”, and a hidden forum on one of them unexpectedly became home to a community that was still active in 2011.
Alternate reality games have blossomed as the web has evolved tools that encourage more participatory culture (two insightful analyses: The Art of Immersion and Spreadable Media). Videogames have evolved too. Some of them (like Heavy Rain) provide almost cinematic interactive stories that appear to offer freedom from the author’s control, but actually run on narrative “rails” that branch with the player’s choice, then eventually re-join. At the other end of the spectrum, “sandbox” or “open-world” games give players far more freedom, sometimes at the expense of emotional engagement or satisfying dramatic structure.
Theatre itself has had an uneasy relationship with audience participation as long as anyone can remember. In recent years, “immersive theatre” companies like Punchdrunk and Secret Cinema have grabbed the spotlight by setting audiences loose to literally roam through their fictional worlds. But decades before them, forum theatre creator Augusto Boal engaged the audience in a very different spirit, with techniques like invisible theatre, “a play (not a mere improvisation) that is played in a public space without informing anyone that it is a piece of theatre”.
And last year, in real life, Commander Chris Hadfield engaged with his audience across a fourth wall up to 250km thick.
In Faster than Night, we’re inviting our own audience to interact with fictional astronaut Caleb Smith via his artificial intelligence ISMEE. You can find her on Twitter as @ISMEEtheAI. In the weeks leading up to the performance, as people tweet questions about his mission to travel faster than light, Caleb will answer. And during the show itself, he will need the audience’s help to make the ultimate choice.
But first a question for you.
What does interactive story look like?
Is it this?
Or is it this?
Or does it mean something entirely different to you?
Interact with us in the comments!
Alison Humphrey is directing and co-writing Faster than Night. She has a master’s in interactive multimedia, but that doesn’t mean she’s figured out the damn thing yet.
by Francisco-Fernando Granados
In performance art, the time and the place of the creation of the work is always the same as that of the presentation of the work. This characteristic is perhaps what separates performance from any other aesthetic medium: in painting, sculpture, photography, and film, making and showing tend to be spaced by an irreducible difference that puts the making before the showing. No matter how much an artist working on a live piece plans and pre-produces, and as important as that may be, the performance always becomes something, and then becomes something else as it unfolds. Esther Ferrer says it transforms in situ.
A big part of the thrill of working on The Ballad of ______ B for HATCH has been the challenge of creating a work specifically for the stage. One of the goals is to make a work that uses all of the material and conceptual elements of theatre (stage, script, performers, lights) with an attitude of critical respect for the medium that yields something that is decidedly not-a-play. This negativity has to do with an interest in a creating a productive tension between what I hope will be the site-specific approach of the project, and entrenched conventions of the stage as a social (or indeed anti-social) space. Incorporating an interactive element based on the online contributions of the public will be a way to bring together the previously discussed tactics of spacing and redistributing of the elements of theatre.
It is here where the curatorial premise of HATCH, with its focus on social media as a tool for the creation of new performance comes into play. In the recent history of conceptual text-based performance work, my approach owes something to the Fluxus sensibility of Alison Knowles, particularly in her computer-generated poem A House of Dust. The poem was created in 1967 as a collaboration between Knowles, the California-based composer James Tenney, and the Siemens 4004 computer. A House of Dust creates “stanzas by working through iterations of lines with changing words from a finite vocabulary list.” The capacity of computer technology at the time limited the possibilities of electronic interventions into poetic processes to closed systems reminiscent of Modernism. In The Ballad of _____ B, this approach is updated to incorporate a relational element.
A House of Dust, Alison Knowles and James Tenney (1967)
Social networking platforms will act as randomizing agents, creating interventions in the spaces opened in the text. On April 18th, a significant section of the script will be made available online as a Google Form through a variety of sites including Facebook, Twitter, and the Praxis Theatre site. The fill-in-the blanks approach to the script recalls the aesthetic of vocabulary lessons in ESL language learning. In this case, the didactic form does not follow a function. Audience members are encouraged to fill in the blanks with responses that may range from the obvious to the non-sensical. The accumulation of these results will be used during the week of the residency to re-shape the script.
A prototype for the form is available now as a test. Click here to access section of the script and fill in the blanks.