In one part of the documentary, “Welcome to Chechnya,” which debuted at Sundance earlier this year, a man identified as “Grisha” shares his harrowing experience of being arrested and tortured by Chechen authorities because of his sexual identification. The emotion conveyed on screen — fear, anger, sadness — is palpable. It is also not, in the strictest sense, real. Using a groundbreaking editing technique, the documentarians behind the film were able to digitally “swap” the face of the real Grisha with that of a volunteer actor, thereby protecting his identity without losing the essential human connection that comes with being able to put a face to a story.
To make the digital face editing possible, the documentary team consulted Dartmouth psychology and brain sciences professor Thalia Wheatley, whose Social Systems Laboratory studies human perception and interpersonal connection. Wheatley said the documentary team reached out to her lab to determine which form of editing would resonate most with audience members.
“They wanted us to test them to see which one would be the least unsettling to watch, the most human,” she said.
Wheatley said the main concern with “Welcome to Chechnya” was that the face swapping might fall into the “uncanny valley”— a term that describes the visual tendency of things that are close to, but not quite, human to come off as unsettling. The goal of the filmmakers was to make their altered subjects human-looking enough to avoid this problem.
“They were trying to create a film in which you don't disrupt the impact of the story and you’re just watching this person talking about their experience,” she said. “The danger of course is that if you get the technology wrong, you get even less connection than you would by blurring out the face.”
To study audience perception of the editing techniques, Wheatley polled 100 Dartmouth students who each viewed four clips and were asked to rate their level of comfort and connection. One clip showed the subject completely masked by the volunteer’s face, another composited the volunteer’s face with the eyes of the real subject and a third replaced the subject’s face with a stylized cartoon image that did not try to mimic reality at all. The fourth clip showed the unadulterated footage of the subject without editing. To preserve the identity of the subjects, some of whom are still under direct threat of violence from their families or the Chechen government, the lab made the clips accessible only on a secured site.
Chris Welker, a graduate student working in the Social Systems Laboratory, took the lead on the face swapping research project. He said the structure of the poll allowed students to rate each individual clip according to the degree to which the subjects felt the clip conveyed the positive trait “animacy” or the negative trait “eeriness.” The viewers were then asked to compare the clips against each other in terms of its fostering of emotional connection.
According to Wheatley, the lab’s initial hypothesis was that the composite image retaining the original eyes would be the most compelling. The results, however, revealed a clear alternative winner— among a majority of those polled, the complete face swap was the most visually appealing option. For most, the composite still fell into the uncanny valley.
“Our guess is that by having part of the face from one person and the eyes from another person, what was created was a kind of perceptual mismatch or perceptual Frankenstein,” she said, explaining that small lighting differences between the original interview and the volunteer footage could factor into the audience’s inability to resolve a coherent face.
She explained that, because humans are so visually attuned to reading detail in faces, contradicting lighting cues might cause primal uneasiness: “This is the currency of horror movies — things that are not quite right,” she said. “It’s the subliminal area between the living and the undead.”
The complete face swap option, on the other hand, worked well in both convincing the viewer that the faces were animate (or human) and in allowing them to become invested in the emotions displayed. Wheatley said this achieved the film’s goal of having persecuted LGBTQ people in Chechnya be both heard and seen.
“It’s showing that it can be done, and that a story can be told without obscuring people’s faces or suggesting that they shouldn't be seen,” she said.
The technology that allowed the documentary filmmakers to transpose faces is similar to that used in “deep fake” videos, in which a machine learning algorithm mimics human expressions on the swapped face by combining and stitching together pre-filmed expressions. For “Welcome to Chechnya,” volunteers were asked to record in lighting as close as possible to the original, reciting a poem and moving their eyes to create “a kind of alphabet for what this face can do,” according to Wheatley.
The result is a film without blurred or barred faces that tackles serious, and potentially dangerous, subject matter. Wheatley said she thought it likely that the technology would be used by journalists and documentarians moving forward to protect identities while allowing the full impact of the story to be felt. Though the editing technique is a sort of departure from objective, documented reality, she said that face swapping could become a common choice among journalist filmmakers, as long as the actual appearances of the subjects are not of importance.
“If the doc was about how someone actually looks, and you’ve manipulated that, it would seem problematic to me,” she said. “But here, it’s not really about whether he originally had blue eyes or brown eyes, or if his jaw was slightly thicker in real life versus slightly thinner. It’s still his words, it's still his story.”
While she acknowledges the darker side of the technology, i.e. its use in creating fake videos that steal people’s likenesses, she sees the face swapping in the documentary as representing the lighter side.
“We tend to think about technology in relation to undesirable or nefarious consequences … but technology is just a tool,” she said. “This is going to be part of our world going forward, and we’ll just have to figure out a way to deal with it.”
“Welcome to Chechnya” received critical acclaim at Sundance, winning the festival’s U.S. Documentary Special Jury Award for Editing. The documentary was picked up by HBO in June. Wheatley’s and Welker’s research is now available in a preprint version for free access online, and is also currently undergoing review for the academic journal Perception.
