How outdoor motion capture brought Dawn of the Planet of the Apes to a new level

Five films are nominated for an Academy Award in the “Visual Effects” category this year, and they each offer a nice look at the amazing tricks filmmakers and their effects teams can pull off on the big screen. In recognition of these five films and one of our favorite Oscar categories, we’re putting the spotlight on one “Visual Effects” nominee each day leading up to Sunday’s broadcast and taking a closer look at what made them stand out.

Previously, we looked at the face-mapping effects that made Falcon fly and Peggy Carter look her age in Captain America: The Winter Soldier. This time, we look at the (literally) groundbreaking effects that brought the Planet of the Apes franchise back into the Oscar picture with Weta Digital’s work on last year’s Dawn of the Planet of the Apes.

Rise of the Planet of the Apes was one of the surprise hits of 2011, managing to find success at both the box office and with critics and audiences alike. The film’s impressive performance made a sequel inevitable, and last year’s Dawn of the Planet of the Apes picked up the story a decade after the events of its predecessor – and upped the ante in more ways than one.

It wasn’t enough for Dawn of the Planet of the Apes director Matt Reeves and the visual-effects team at Weta Digital to simply add more apes to the mix in last year’s sequel. This time around, the already impressive VFX techniques that turned motion-capture actor Andy Serkis into the intelligent ape Caesar had to bring an entire colony of apes to life – numbering well into the thousands – and do so while they were in their new, natural habitat: the rainy forests surrounding San Francisco.

“They find a way to take the shapes and put them into the face of an Ape who doesn’t share the same anatomy.”

As if that wasn’t a tall enough task, Weta also had to take the next logical, narrative step beyond Rise of the Planet of the Apes and give the film’s simian stars a level of sentience well beyond what was seen in its Oscar-nominated predecessor.

“In the previous film, Caesar mimed his way through a lot of it,” explained Oscar-winning VFX supervisor Joe Letteri in an interview with Variety. “He had not yet developed any kind of sophisticated communication, but gradually you saw him taking on this mantle of leadership. In this film, there’s clearly dialogue where he needs to communicate his thoughts and intentions with the humans and there’s reasoned thought behind the action.”

The need to blur the line even further between the film’s human actors and their digital alter egos necessitated a level of detail beyond anything the Weta team had done in the past. From the way the apes’ hair moved in the wind and the way water runs down their bodies to the way the sinister ape Koba (played by Toby Kebbell) moved his lips while wagging his tongue at a pair of naïve humans, every detail was studied, compared, and painstakingly translated from human actors to apes.

“We actually wrote new algorithms for doing light transport through hair, and we wrote new algorithms for doing the dynamics of hair,” Letteri told when asked about some of the new additions to Weta’s VFX arsenal that the film necessitated “So the physics are also very important to get right, underneath everything we do. That’s the level of realism for which we apply the final rendering.”

For that particular scene with Koba, Wired reports that the Weta team studied high-speed footage of human actors blowing “raspberries” with their tongues, lips, and cheeks in order to determine exactly what each part of the face is doing during such an action, then generated various digital models of Koba’s face doing the same. The most appropriate models were then mapped to Kebbell’s physical performance.

Most importantly, the nuances of the apes’ actions – when the apes themselves are “acting” in order to fool the humans, for example – must make a successful transition from motion-capture performance to digital recreation in order for audiences follow along. The subtle expressions that tell the audience Koba is tricking the humans, for example, need to carry over from Kebbell’s performance.

For Reeves, much of the challenge involved finding a way to get as much of critically acclaimed performance-capture artist Andy Serkis’ brilliantly layered performance to the screen as possible.

“We want to do whatever we can to make it work.”

“What you want to get through [to the screen] is what Andy’s done,” said Reeves in a 2014 interview. “We go through an emotional search. He does something – discovers something – in the moment and it feels right. We put that in the editing. The challenge for Weta becomes how to take that and translate it to an anatomy that is not the same as Andy’s. So that’s the genius of what they do. They find a way to take the shapes – which are all purely Andy’s – and put them into the face of an Ape who doesn’t share the same anatomy.”

Achieving a greater level of detail with the apes and reducing the dissonance between motion-capture performance and on-screen ape activity wasn’t the only way Weta’s artists raised their game for Dawn of the Planet of the Apes. The team also had to find a way to bring the techniques they perfected in a controlled studio environment out into the great outdoors.

“The technology that we really needed to create for this film was the ability to take our performance-capture out on location,” said Letteri. “That was paramount to getting the drama and the action and the realism. This film didn’t want to be studio-bound.”

“Matt [Reeves] really wanted that feeling of being out with the chimps,” he added. “He wanted to take the gear into this wet, rainy forest. So for the first time we really had to make this gear mobile. We could to get it out on location, and still capture the immediacy and all the detail we needed, even though we might have 10 apes being captured in a shot.”

In order to bring their equipment out into the wild, Weta developed wireless versions of its performance-capture gear that would allow them to record and react with more flexibility to the wide-open stage that the forest sets gave the actors. Weta also had to re-engineer the hardware used to track the actors’ performances, making the physical markers on their motion-capture suits waterproof and significantly more rugged, in order to withstand the occasional roll in the dirt.

“We never want the technology to inhibit the filmmaking process – it’s the other way around,” Letteri told Collider last year. “We want to do whatever we can to make it work. The story called for being out in the forest in the rain, so we wanted to make sure we designed whatever we needed to take with us so that it would work, [and] it would integrate well, and everything would be transparent to Matt, to the other departments, and to the actors.”

Said Letteri of his team’s mission with Dawn of the Planet of the Apes and any other projects they take on: “The whole point is to make the movie you want to make, and we’ll figure out how to put it on the screen.”

Movies & TV

The best movies on Netflix in March, from Buster Scruggs to Roma

Save yourself from hours wasted scrolling through Netflix's massive library by checking out our picks for the streamer's best movies available right now, whether you're into explosive action, witty humor, or anything else.

The best zombie games of all time, from Resident Evil to The Last of Us

Zombies are a longtime media staple, from movies to literature to TV to video games. Plenty of zombie games are bland and boring, but a good number offer tangible reasons to care about the undead. These are the best zombie games.

The best LG G7 ThinQ cases will keep your phone looking new

The LG G7 ThinQ comes with a powerful processor, versatile cameras, and amazing sound. But a powerful phone still needs protection and you might want to change the style. Here are the best LG G7 ThinQ cases.

You've spent a grand, now don't skimp on a screen protector for your iPhone X

Wondering how to protect your new iPhone X against scratches and drops? Look no further than our list of the best iPhone X screen protectors, which includes screen protectors designed to reinforce your phone's front panel.
Movies & TV

Marvel releases more footage from Avengers: Endgame in latest TV ad

The events of Avengers: Infinity War changed the Marvel Cinematic Universe in some big ways and left fans wondering how its heroes can possibly recover. Here's everything we know about Avengers: Endgame, the sequel to Infinity War.
Home Theater

From the Roku Ultra to the Fire TV Cube, these are the best streaming devices

There are more options for media streamers than ever, so it’s difficult to pick the best option. But that’s why we're here. Our curated list of the best streaming devices will get you online in no time.

Stream UFC Fight Night St. Petersburg: Overeem vs. Oleynik with ESPN Plus

This Saturday, veteran UFC heavyweights Alistair Overeem and Alexey Oleynik are going toe to toe in the octagon for UFC Fight Night 149. If you’re looking for a way to watch it online, then ESPN Plus is where you’ll find it.

Avengers aftermath: From The Eternals to The Boys, things are about to get weird

On this week's show, we'll take a deep dive into what's coming after Avengers: Endgame. Sure, it's probably going to be the biggest movie ever, but what about Marvel/Disney's Phase 4? We'll tell you why The Eternals may be a big part of it.
Movies & TV

Clip from John Wick: Chapter 3 confirms the dog is totally fine

John Wick: Chapter 3 — Parabellum, the third installment of the wildly successful action series that stars Keanu Reeves as a deadly assassin forced out of retirement, hits theaters in May 2019. Here's everything we know about it so far.
Movies & TV

The best shows on Netflix right now (April 2019)

Looking for a new show to binge? Lucky for you, we've curated a list of the best shows on Netflix, whether you're a fan of outlandish anime, dramatic period pieces, or shows that leave you questioning what lies beyond.
Movies & TV

Netflix vs. Hulu: Which streaming service is right for you?

Netflix and Hulu are two of the biggest names in streaming entertainment. Our guide will help you decide which streaming service has the content you want and the best value for your limited budget.

No cable? No problem: Here’s how you can watch UFC fights online

Cord cutting is a great way to save money, but watching sports online can be tricky due to restrictions. If you want to stream UFC, the world’s biggest MMA promoter has joined with ESPN to bring the action online. Here's how to watch.

How to watch the Top Rank Boxing: Terence Crawford vs. Amir Khan PPV with ESPN Plus

This Saturday, welterweight champion Terence Crawford will defend his title against former junior welterweight champion Amir Khan at Madison Square Garden. Here's how you can watch this highly anticipated pay-per-view event.
Movies & TV

Best new podcasts: The Ballad of Billy Balls, Decomposed, and more

Feel like you’re drowning in podcasts? In this weekly series, we’ll help you pick out the best of the new and returning shows. This week’s picks include a punk love story, disappearing referees, gun court, and intriguing tales from…