Live From PyeongChang: NBC Olympics Digital Workflow Group Finds Its Groove Pretesting, preconfiguration smooth the way for massive infrastructure By Ken Kerschbaumer, Editorial Director Wednesday, February 21, 2018 - 5:45 pm
Print This Story
Darryl Jefferson, VP, postproduction and digital workflow, NBC Sports, his team, and more than 600 people at NBC Olympics split between PyeongChang and Stamford, CT, find themselves in a good spot at the 2018 Olympics. A wide range of testing and preconfiguration allowed a massive digital infrastructure to be up and running in the days prior to the Games without many of the difficulties experienced during Rio 2016.
For me, the story of our group has been all the work we did beforehand, says Jefferson. We have been sprinting since last spring, and that goes for [the work we did on] every subsystem.
Those subsystems include a storage- and asset-management system used by NBC staffers at the venues, in the IBC, in Stamford, and even beyond. Last spring, the subsystems that would be used at the IBC in PyeongChang were set up in the basement level of NBC Sports' Broadcast Center in Stamford, and tests were run to optimize the performance under real-world conditions.
From left: Jim Miles, Matthew Green, Darryl Jefferson, and Kamal Bhangle lead the NBC Sports digital-workflow efforts.
Avid built legions of testing algorithms to hammer on the system and make sure we did not have failures [when hundreds of users access the system], he explains. Every system and subsystem was stood up during the tests and preconfigured, so we were a lot more prepared than we were in the previous Games.
Matthew Green, senior digital workflow engineer, NBC Sports and NBC Olympics, notes that there were five phases to the tests.
The initial round of tests was to find the failure point and then identify the hardware that needed to be replaced and the software optimizations that Avid needed to make, he explains. Then, we would run another round, identify new places to improve, and test again. At the end, it was less about stability and more about usability and making sure search and playback was faster and better. We wanted to end up with a much smoother experience for everyone.
According to Jim Miles, director of digital workflow systems, NBC Sports and Olympics, tests in Stamford raised the baseline for readiness once the equipment was up and running in the IBC. That rock-solid baseline allowed the digital-workflow team to focus more on smaller issues and final tweaks.
The Digital Workflow
The digital system put together for the Olympics is used by hundreds of staffers not only to deliver the live content to viewers but also to edit highlight packages, feature stories, and graphics; create social-media content; and, ultimately, get that content ready for delivery via NBC networks, apps, social-media platforms, and connected-TV devices.
At a basic level, the system begins with content coming in from the venues as feeds or, if cut up and assembled at a venue or elsewhere on EVS replay servers or Avid editing systems, as files. Depending on what those files will be used for, they will land at one or more of 100 targets, such as storage systems at the IBC, Stamford, or even a production team at a venue.
If it is for fast turnaround, says Miles, it will be right next to you on the EVS server. If it was a host feed, it will be here at the IBC, and, if it was an off-air signal, it would be recorded in Stamford.
The SAN at the IBC is just under a petabyte of MediaGrid, about 72 TB of Isilon, and more than 700 TB of Avid storage. The total number of people who can touch the content in the asset-management system is close to 300 at the IBC and 300 in Stamford, and the system is built to handle upwards of 500 concurrent users. There usually will not be more than 150 users at one time, and that potential for a heavy load is one of the reasons the tests were crucial.
One of the key goals prior to the Games, Green says, was to reduce the choices facing users as they went about their tasks.
We wanted to reduce the ways you could [find and move media] because the users would wonder which way was best to do the work they needed to do, he explains. The inability for the user to easily find the most efficient way to do something [like send a file from a venue to Stamford or search for content] also meant that the support team had to be ready to solve more types of problems. By streamlining operations, the users can work faster, and support can more quickly respond to any issues.
Jefferson says the team also worked with Avid on the behavior of the player used to find content in the system. The player has been improved so that video-search results start up more quickly and the set of results is easier to manage. Having it return every gold medal when you just care about the last one means you have too much content to sort through.
Harmonic is playing a key role in storage and ingest alongside Avid editing and storage systems used for both asset management and editing. EVS systems are at every venue and play an important part in the replay operation at the venues and in the control rooms.
We have great long-running relationships with those companies, says Jefferson. It was nice we didn't have to reinvent the wheel in such a short turnaround.
One important part of the process is a File Intake Room at the IBC, which is overseen by Kamal Bhangle, manager, remote engineering and technology, NBC Sports and Olympics.
It is the only room where you can bring content in from the outside and get it into the system, she says. Everything is scanned on a separate machine before it is plugged into the main system. Yes, it takes longer, and people have to wait, but everyone knows how serious [virus] protection is now.
Bhangle notes that, although there we
