"What we wanted to do was to build a file system that would support any resolution at any number of streams"Author:
ContributorPublish date:
Aug 16, 2018
The M&E industry is one of the fastest evolving sectors. The rapid progression of frame rates and video revolution has a considerable impact on production and post-production teams, and their storage and capacity requirements. Throughout the production process, a film increases from approximately 10 TB to roughly 350 TB and the number of workstreams with users working on the same dataset grows rapidly.
To solve this data storage challenge, Technicolor created a scalable and future-proof storage infrastructure using Excelero NVMesh to provide workstations with access to high bandwidth and low latency performance that accommodates streaming, editing and reel presentation at any time.
Future-proofing post-production storage
Like many data centre and infrastructure teams at production and post-production houses, Technicolors IT department along with engineering director Amir Bemanian, were struggling to keep up with the rapid evolution from one resolution to the other.
4K is becoming the default resolution and 8K is on the horizon, and frame rates are increasing from 24 and 30 frames per second to 48fps and even 60fps. The additional storage capacity and thus also bandwidth that is needed for higher frame rates increases linearly: doubling the frame rate from 30fps to 60fps also doubles the storage needs. And increasing the resolution from 2K to 4K quadruples the storage requirements. When a studio switches from 2K 30fps to 4K 60fps, the post-production house also needs to be prepared to store and stream eight times the volume of data, which is a substantial increase.
On top of this, there are also possible investments in extra cameras, additional visual effects, High Dynamic Range (HDR), colour depth, and more. Yet another challenge for post-production workflows is avoiding dropped frames, caused by insufficient performance and high latencies, which impacts the technical quality of an asset and results in on-screen motion.
A common problem is that many companies in the M&E industry such as Technicolor, retain infrastructures which are typically based on a Fibre Channel network with storage arrays serving tens of workstations. This means that the speed to ingest data to the workstations from the Fibre Channel storage arrays is simply too slow and does not scale. At Technicolor, workstreams had to be duplicated in order for multiple people to work on the dataset at the same time.
In order to be better prepared for current and future storage and streaming needs, Amir Bemanian and his team at Technicolor began exploring new "future-proof" storage and network architectures. "What we wanted to do was to build a file system that would support any resolution at any number of streams," explains Bemanian. "And that was essentially impossible until 2017 where we saw an alignment of technologies: the networking, the PCIE slot, the NVMe drives and also the software - from Excelero - that glued it all together. This combination of technologies enabled us to do storage in a very different way to traditional methods.
The IT infrastructure required to feed dozens of workstations of 4K files at 24 fps is mind-boggling and that doesn't even consider what storage demands we'll face with 8K or even 16K formats, Bemanian continues. It's imperative that we can scale to future film standards today. Now, with innovations like the shared NVMe storage that Excelero provides, Technicolor can enjoy a hardware-agnostic approach, enabling flexibility for tomorrow while not sacrificing performance.
Excelero's NVMesh for Any-K workflows
NVMesh software-defined block storage features Elastic NVMe, a distributed block layer that allows any application to utilise pooled NVMe storage devices across a network at local speeds and latencies. Distributed NVMe storage resources are pooled with the ability to create arbitrary, dynamic block volumes that can be utilised by any host running the NVMesh block client. These virtual volumes can be striped, mirrored or both while enjoying centralised management, monitoring and administration. In short, applications can enjoy the latency, throughput and IOPS of a local NVMe device while at the same time getting the benefits of centralised, redundant storage. NVMesh is deployed as a virtual, distributed non-volatile array and supports both converged and disaggregated architectures, giving customers full freedom in their architectural design.
By deploying Excelero NVMesh and six standard NVMe servers powered by Intel 3D NAND flash, Technicolor achieved 4K streaming at 60fps to over 50 workstations concurrently. Delivering 99.8 per cent of local NVMe storage server performance across the network, the solution offers twice the performance of competing solutions at 50 per cent of the cost and requires a much lower data centre footprint.
After an initial proof of concept, the storage solution was brought into production within four weeks for use with a major Hollywood feature film. So far, three projects have been run successfully on the software-defined system.
A major benefit is that the architecture has created an optimal environment where multiple members of the post-production team can simultaneously view and work on the same data set. Bemanian concludes: "It has generated a positive and productive end user experience.
