S3 Object Storage
21/03/2019
At Object Matrix, we provide hybrid storage for media workflows, with the great benefit of making the solution manageable from a single pane of glass . Customers can therefore easily manage their data and where that data moves, whether that data is on-premise, or on the cloud.
A big part of our cloud integration is working with AWS's (Amazon's) S3 Object Storage which has led us to understand at a deeper technical level just what you can and can't do.
This of course is a changing environment, for instance AWS has recently unified different storage classes under the same umbrella: S3 Object Storage.
In this blog, I will share some of our experiences dealing with S3 Object Storage classes and their APIs.
Data or Object Management
The unification of the different storage classes has brought several advantages to our applications and services. The most important to us are:
Facilitates the movement of data between tiers
The new APIs allow direct access to the data to and from the different tiers
However, one of the things we noticed is that what moves seamlessly between tiers is the data, not the metadata. Apparently, Glacier classes do not support any additional metadata for the archives except for an optional archive description.
This means that the metadata has to be kept either on the other AWS tiers or on the client side.
We would guess that there must be an intrinsic limitation in managing metadata by Glacier due to the backend technology but that means this is an important aspect to consider if you want to manage the full lifecycle of an object: Where to keep the metadata safely? And, how to keep the metadata associated with the content, e.g., even if the content is moved to another technology?
At Object Matrix, we consider objects in all scenarios and fundamentally believe that the metadata should always be kept with an object regardless of where the object is moved to. We've seen solutions that, e.g., keep the metadata in a database. That disables the transportability of the object (separately from the database) and can result in a form of vendor lock-in.
Metadata
Some AWS S3 storage APIs support user and application metadata. However, there are some limitations around metadata in AWS S3 that are very relevant in media workflows:
It's not possible to manipulate metadata entries independently
S3 objects metadata is limited roughly to 2KB in size
(reference: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html)
These limitations may not be important for generic applications but in media workflows, metadata can be large and it's as important as the data or more.
We have found, maybe due to those limitations, that many companies use S3 for data-only movements, no metadata is stored with the objects. Basically, they use an object API to access just the data.
At Object Matrix, we are strong users of S3 APIs at both the client and backend sides. However, the way that the S3 metadata storage in that area works means that we see as advantageous to keep metadata in a linked sidecar file rather than in the object itself. This has the added benefit that the same sidecar file can be stored on just about any technology: LTO, another cloud provider with its on metadata storage idiosyncrasies, etc. Also, the sidecar file can hold an almost limitless amount of metadata.
Metadata Specification
Another aspect that came to our attention whilst using AWS S3 APIs between our products and third parties is that having an API only is only half of the story when it comes to building rich integrations and media workflows.
Basically, it would be really helpful for all of the tools if there existed a media file metadata specification for object storage. The specification could define fields around projects, relative paths, media types, etc. There are specifications within, e.g., IMF that could be drawn upon. But how to bring those up to a higher level and in a format that aids applications to collaborate at an object storage level?
So in summary, we've had to go about this ourselves and have defined a high-level metadata specification, or, as we term it internally: a Metadata API. All our applications use that specification, be it via filesystem access, S3 access, FTP, DropSpot or Vision. It allows our applications to collaborate seamlessly. And where we integrate with 3rd parties, we pull their metadata into our standardised format. We'd love to see a specification body (hint hint DPP / SMPTE) look at whether a wider specification would be useful.
Conclusion
Amazon S3 Object Storage is evolving quickly to make simple to use cloud storage. With our tools, we combine different tiers. But whilst it has object metadata commands in its API we find that for transportability, avoiding metadata limitations within an API (either in AWS or any other platform), non-proprietary access and no vendor lock-in keeping metadata in a sidecar file is key.
And, with object APIs, keeping metadata is only half the story for application interoperability. The other half of the story is keeping the metadata in a format that can be easily understood and shared. We need defining metadata specifications if we want to achieve portable integrations for our customers.
About Object Matrix
Object Matrix is the award winning software company that pioneered object storage and the modernisation of media archives. Our on-prem and hybrid cloud storage solutions bring operational and financial benefits to our customers by securely managing content at every stage of its lifecycle; from ingest and nearline to archive and distribution. Deployed where you need it, our technology is non-proprietary, integrates into existing workflows, enabling you to work locally and share globally.
Our flexible approach, coupled with our focus on the media industry, me
Most recent headlines
04/08/2024
Dalet Appoints Santiago Solanas as CEO to Lead Next Era of Growth and Innovation
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet and Veritone Reach Agreement to Distribute, Transact and Monetize Media Archives
Dalet, a leading technology and service provider for media-rich organizations, a...
03/05/2024
Fubo Loses Subs in Q1 But Exceeds Guidance and Reduces Losses
NEW YORK FuboTV reported that it lost 110,000 subs in Q1 2024 but it exceeded its guidance for the quarter with double digit year-over-year increases across key...
03/05/2024
Only 3 Weeks Left To Enter The Creative Industries Most Award
Only 3 Weeks Left To Enter The Creative Industries Most Award Brie Clayton May 3, 2024 0 Comments Creativepool is giving a last call to the creative i...
03/05/2024
WePlay Studios Transforms the Future of Live Event Storytelling with AJA Gear
WePlay Studios Transforms the Future of Live Event Storytelling with AJA Gear Brie Clayton May 3, 2024 0 Comments By 2032, the esports market is expec...
03/05/2024
The Business of TV News: Shannon Bream Singles Out Key Issue in 2024 Election
WASHINGTON Shannon Bream, Fox News Sunday anchor and chief legal correspondent, sat for a keynote chat at The Business of TV News, with Tom Umstead, senior co...
03/05/2024
ABC News Anchor Martha Raddatz Says Local News Is Everything'
WASHINGTON Asked how important is local news, Martha Raddatz, who has covered foreign conflicts for decades and moderated presidential debates for ABC News, s...
03/05/2024
Paramount Stock Jumps on Report of $26 Billion Bid From Sony Pictures and Apollo Global
Sony Pictures and Apollo Global Management have made a $26 billion cash offer fo...
03/05/2024
CBS Shares 2024-2025 Schedule
CBS announced its 2024-2025 primetime schedule, which features the new dramas NCIS: Origins, Matlock and Watson, and new comedies Poppa's House and Georgie ...
03/05/2024
Fubo Continues To Reduce Red Ink Despite Losing 110,000 Subs in Q1
Sports-focused streaming service Fubo said it reduced its red ink in the first quarter, despite losing 110,000 subscribers since the end of the year....
03/05/2024
Walmart Ready to Release 4K Hybrid Streaming/Smart Speaker
Walmart is reportedly about to release a new 4K streaming box/smart speaker hybrid device for Google TV that could retail for as low as $50, according to a repo...
03/05/2024
Limecraft updates content delivery platform
Enhancements to its Delivery Workspace solution provide increased content delivery robustness and security By Matthew Corrigan Published: May 3, 2024 Enha...
03/05/2024
Sony and Apollo submit $26 billion bid for Paramount Global
According to reports, if the bid is successful, Sony would be the significant majority shareholder of the new private company By Jenny Priestley Published: M...
03/05/2024
Global Academy launches Manchester media school
Located in Manchesters Spinningfields district, the course aims to equip students with the skills, knowledge and industry insight for a career in the creative s...
03/05/2024
New Lightbridge C-Move Core
Lightbridge has created a new Cine Reflect Lighting System kit, dedicated to enabling artistic lighting with consideration of today's productivity practical...
03/05/2024
KPBS Modernizes Infrastructure With Major AES67 Control R...
Broadcast media systems integrator BeckTV and Lawo worked together on a large-scale AES67 build of radio control rooms at KPBS, San Diego's NPR and PBS memb...
03/05/2024
SmallHD Free PageOS 6 Update Adds Powerful New Capabiliti...
Raleigh, NC, April 12, 2024 SmallHD announces PageOS 6, a powerful software update free for owners of SmallHD monitors, featuring increased functionality, exp...
03/05/2024
SmallHD Launches Quantum 32 Quantum Dot OLED HDR Referenc...
SmallHD announces the Quantum 32 a new 31.5 OLED monitor designed to provide reference HDR/SDR images for post-production final color grading, as well as cri...
03/05/2024
How WePlay Studios is Transforming the Future of Live Eve...
By 2032, the esports market is expected to grow to $9.29 billion, bolstered by a global player count and fan following that both continue to climb. The 2023 Lea...
03/05/2024
Amagis Latest FAST Report Unveils the Rise of a Diverse G...
Amagi, the global leader in cloud-based SaaS technology for broadcast and connected TV (CTV), today announced that the 11th edition of the Amagi Global FAST Rep...
03/05/2024
WDR Filmhaus to Implement Riedel Infrastructure
Riedel Communications today announced that Westdeutscher Rundfunk (WDR) is relying on a media and intercom network based on Riedel technology for the renewal of...
03/05/2024
TF1 Chooses Broadpeak to Power Targeted Advertising for N...
Broadpeak , a leading provider of content delivery network (CDN) and video streaming solutions for content providers and pay-TV operators worldwide, announced t...
03/05/2024
Limecraft Announces New Platform Update and Preview of MI...
Media technology innovator Limecraft will exhibit the latest updates to its online platform on stand G79 at the 15-16 May 2024 Media Production & Technology Sho...
03/05/2024
Prism Sound Announces Partners For 8 Hours At Rockfield 2...
Audio interface specialist Prism Sound is partnering with console manufacturer Solid State Logic, professional monitor manufacturer PMC and acoustic panel and d...
03/05/2024
Intinor brings excellence in contribution streaming to MP...
Intinor, Sweden's leading developer of products and solutions for high-quality video over the internet, is set to join forces with its UK partner Zest Techn...
03/05/2024
NAKIVO Reports 10 Percent Revenue Growth in EMEA Region
NAKIVO Inc., a fast-growing software company dedicated to protecting physical, virtual, cloud, and SaaS environments, announced its Q1 2024 financial results to...
03/05/2024
Hiltron Introduces Field-Upgradable Motorisation Kit for...
Hiltron Communications announces an addition to its range of satellite communication products and systems: a field-upgradable motorisation kit specifically desi...
03/05/2024
LiveU Showcases Story-Centric Workflows and the LiveU Eco...
Visit LiveU at Stand D30 With this year an exceptional one for major sporting events alongside multiple, key elections, the power and flexibility of LiveU'...
03/05/2024
Agile Content appoints Koldo Unanue as new CEO to boost t...
Agile Content, the leader in the development of technology and solutions for the provision of television services over the Internet, announces Koldo Unanue as i...
03/05/2024
Fubo, Dish, DirecTV & Others Urge Congress to Probe New Sports Streaming JV
Eight co-signers representing streaming services, pay TV operators and public interest groups have sent a letter to Congressional leaders urging their Committee...
03/05/2024
Brightcove Integrates AWS-Powered Generative AI Solution
BOSTON Brightcove has announced that it has implemented Amazon Q Business, a new generative AI assistant on Amazon Web Services (AWS)....
03/05/2024
CBS Claims 16th Straight Prime Time Viewing Title
NEW YORK CBS has declared that it will finish the 2023-24 season as Americas Most-Watched Network in primetime for the 16th consecutive season....
03/05/2024
Survey: U.S. Business and Consumers Plan to Increase Tech Spending
A new survey indicates that U.S. businesses and consumers had an overall positive attitude towards their technology spending plans in the first quarter of 2024,...
03/05/2024
Sony, Apollo Make $26B Bid for Paramount
Sony and Apollo Global Management have sent an offer letter to Paramount Global's board of directors making an $26 billion all cash offer for Paramount Glob...
03/05/2024
EMG / Gravity Media Grow Executive Team
EMG / Gravity Media has appointed Charlie Cubbon chief operating officer (COO) and Jamie Hindhaugh regional CEO for the U.K., U.S., Australia and the Middle Eas...
03/05/2024
WRAL-TV Brain Game Producer Randy Mews Wraps Season Taping Before Retirement
Final 2024 Season Games to Air Saturdays on WRAL at 10:30am in May, Championship on June 1 Taping for the final episodes of the 2024 season of WRAL-TV's Br...
03/05/2024
Estrella MediaCo Launches FAST Channels With Curiosity (NewFronts)
Estrella MediaCo said it is expanding its connected TV portfolio by launching three new channels with Curiosity that will first appear on Samsung TV Plus....
03/05/2024
SAG-AFTRA Will Use Nielsen Streaming Data To Enforce New Contract Terms
Nielsen said that actors' union SAG-AFTRA has selected Nielsen as its third-party provider of streaming content measurement....
03/05/2024
Cineverse Moves AI Content Search Tool cineSearch Into Public Beta
Cineverse said it has released a public beta version of cineSearch, its artificial intelligence-based system designed to help users find content to watch....
03/05/2024
Court TV To Debut Season 2 Of Accomplice To Murder May 5
Court TV said that its series Accomplice to Murder with Vinnie Politan is returning for a second season beginning May 5....
03/05/2024
Paramount Advertising Taps Mastercard, EDO for Attribution
Paramount Advertising said it is working with Mastercard and research and analytics company EDO to measure the impact of advertising campaigns and improve their...
03/05/2024
The Business of TV News: How To Grow Local Programming Without Burning Out Staffers, Viewers
WASHINGTON The panel The Growth of Local Programming happened at The Busines...
03/05/2024
Katz Media Group Names Craig Broitman President
Katz Media Group said that Craig Broitman will become president of the company, succeeding Leo MacCourtney, who announced that he is retiring effective July 1....
03/05/2024
MeTV Toons To Debut This Summer
MeTV Toons, the latest digital broadcast network from Weigel Broadcasting, will debut June 25, Weigel vice chairman Neal Sabin said at a virtual press conferenc...
03/05/2024
2024-05-02
CUPERTINO, CALIFORNIA Apple today announced financial results for its fiscal 2024 second quarter ended March 30, 2024. The Company posted quarterly revenue of $...
03/05/2024
Prison Journalism: A glimpse into South African prisons
Prison Journalism: A glimpse into South African prisonsDavid van Nek was incarcerated in Pollsmoor Prison from 2013 until 2015 and participated in the restorati...
03/05/2024
Prison Journalism: When your time in prison gets hard
Prison Journalism: When your time in prison gets hardFawaaz Adendorff was incarcerated in Pollsmoor Correctional Centre and participated in a Restorative Justic...
03/05/2024
Baby Reindeer: What's all the fuss about Netflix's hit series?
Baby Reindeer: What's all the fuss about Netflix's hit series?Netflix's new miniseries Baby Reindeer is trending on Netflix and it's the kind ...
03/05/2024
Tonight on House of Zwide: Makho, the hitman, turns out to be an old friend of Isaac's
Tonight on House of Zwide: Makho, the hitman, turns out to be an old friend of I...
03/05/2024
Tshepo Matlala discusses his class clown role in Youngins'
Tshepo Matlala discusses his class clown role in Youngins'Breakout star Tshepo Matlala portrays fan-favourite character Tshepo Masemola, who is the class c...