How The Crown used deepfake technology

TVBEurope talks to the team at Framestore about their work on season four of The Crown.

By Jenny Priestley

Published: November 16, 2020

TVBEurope talks to the team at Framestore about their work on season four of The Crown.

target=_blank title=Share on LinkedIn class=share-linkedin> TVBEurope talks to the team at Framestore about their work on season four of The Crown.

https://www.tvbeurope.com/production-post/how-the-crown-used-deepfake-technology title=Share via Email class=share-email>

The brand new season of The Crown has employed deepfake technology for the first time on a sequence involving Erin Doherty's Princess Anne.

The sequence involved replacing the face of an equestrian athlete who stood in for Doherty. For several shots in which Anne is taking part in a show jumping competition, the team at Framstore, who worked on the sequence, employed a deep fake technique to achieve the replacement.

Framestore's deputy head of 2D and the project's 2D Supervisor Ollie Bersey tells TVBEurope they opted to use deepfake because it was going to do the best job in the in the shortest time.

He adds, The deep fake isn't suitable for everything, but it seemed to fit the bill. We had lots of images that we could feed into the machine learning computer. There was no shortage of reference of the faces we were replacing and the faces we wanted to regenerate, so it seemed the best approach. It's quite an exciting new approach that we wanted to test.

Bersey adds that he thinks deepfake technology will continue to develop for film and TV productions. When we first started looking at this, there were a number of limitations in resolution available, and these were kind of improving as we went along, he explains. There is no reason why this isn't going to become a much bigger thing. We're using it on on other projects, we've used it in commercials. It's quick, and it's successful. It's going to be used and it's an important part of the future. In The Crown, we did actually have to get in there with good old fashioned 2D comping and make further adjustments to the face because the performance of the stunt double wasn't quite going to achieve the correct output.

Framestore once again worked on multiple shots and sequences for season four providing VFX work on 230 shots, from creating digital environments and crowd work to a photoreal CG stag. Back in 2006, the company created the stag featured in The Queen. However, for The Crown it was a whole new piece of work. Internally we didn't have a stag asset that we could kind of lean into or pick up and work up from there, says Ahmed Gharraph, Framestore's head of CG.

I think partly what helped win us the job is that last year we delivered a commercial with a photoreal reindeer in it. Reindeers and stags are roughly of a similar ilk, antlers and furry and whatnot. I guess that project kind of gave the client faith that we can make these photo real creatures, as well as our history of creature work in projects such as His Dark Materials. Creating photoreal creatures is in our blood. When this sequence came in we were particularly excited about it, because The Crown is not a visual effects heavy show. It's embedded in history and embedded in the real world so there was no room for error. That made it a really exciting challenge because ultimately we were aiming to create something that should hopefully fool the audience into thinking what they're looking at is real footage.

In order to create the stag, some of the Framestore team visited Richmond Park in London to hang out and observe the animals. Really it was endless amounts of reference, just constantly finding pictures and videos. It's not something that you just study at the start and you go, right, I know what the stag looks like, now I'm going to go off and make it work, adds Gharraph. As the shots and the asset develops, we seek out new references and find little subtle nuances of how that anatomy is put together, how the stag might step in a certain way. It's an iterative and subtle process.

If you look at it in terms of the workflows, it's storyboarded, and then there's a reference pass of that, and then there's an animation block of that where you develop animation cycles, and you put them in the shot, adds Ross Burgess, head of animation. Then you speak to the client to see if that's what they were thinking for pacing and timing. It's like a piece of sculpture, you just chipping away at the blocks until you finally get there. On this project it was actually a very organic process, there was lots of iterations, but it only got better as we chipped away at it.

There's a particular shot that kept me awake at night, says Burgess. It's when he gets shot. We were battling with the idea that what usually happens when stags get shot is that they go down, they collapse and never get back up again. Obviously, that would just ruin the story of what we're trying to do because our stag gets up and runs off. So we were asking ourselves how we were going to take the realism of what a stag would actually do, and giving it a sense that it could get back up again. What we had to do in animation to kind of force that was a real challenge, because it's not something that there was any reference for. We had to traditionally keyframe it and it was very difficult.

Other sequences the Framestore team worked on that they found challenging include turning 21st century Manchester into 1980's New York. New York was quite a challenge, says Standish Millennas, Framestore's VFX producer. There was a lot going on in our new New York scene. We had to make the streets of Manchester look like New York. The set dressing was very good, up to a certain level, but we'

LINK:	https://www.tvbeurope.com/production-post/how-the-crown-used-deepfake-...
	See more stories from tvb

Most recent headlines