S3 Object Storage
21/03/2019
At Object Matrix, we provide hybrid storage for media workflows, with the great benefit of making the solution manageable from a single pane of glass . Customers can therefore easily manage their data and where that data moves, whether that data is on-premise, or on the cloud.
A big part of our cloud integration is working with AWS's (Amazon's) S3 Object Storage which has led us to understand at a deeper technical level just what you can and can't do.
This of course is a changing environment, for instance AWS has recently unified different storage classes under the same umbrella: S3 Object Storage.
In this blog, I will share some of our experiences dealing with S3 Object Storage classes and their APIs.
Data or Object Management
The unification of the different storage classes has brought several advantages to our applications and services. The most important to us are:
Facilitates the movement of data between tiers
The new APIs allow direct access to the data to and from the different tiers
However, one of the things we noticed is that what moves seamlessly between tiers is the data, not the metadata. Apparently, Glacier classes do not support any additional metadata for the archives except for an optional archive description.
This means that the metadata has to be kept either on the other AWS tiers or on the client side.
We would guess that there must be an intrinsic limitation in managing metadata by Glacier due to the backend technology but that means this is an important aspect to consider if you want to manage the full lifecycle of an object: Where to keep the metadata safely? And, how to keep the metadata associated with the content, e.g., even if the content is moved to another technology?
At Object Matrix, we consider objects in all scenarios and fundamentally believe that the metadata should always be kept with an object regardless of where the object is moved to. We've seen solutions that, e.g., keep the metadata in a database. That disables the transportability of the object (separately from the database) and can result in a form of vendor lock-in.
Metadata
Some AWS S3 storage APIs support user and application metadata. However, there are some limitations around metadata in AWS S3 that are very relevant in media workflows:
It's not possible to manipulate metadata entries independently
S3 objects metadata is limited roughly to 2KB in size
(reference: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html)
These limitations may not be important for generic applications but in media workflows, metadata can be large and it's as important as the data or more.
We have found, maybe due to those limitations, that many companies use S3 for data-only movements, no metadata is stored with the objects. Basically, they use an object API to access just the data.
At Object Matrix, we are strong users of S3 APIs at both the client and backend sides. However, the way that the S3 metadata storage in that area works means that we see as advantageous to keep metadata in a linked sidecar file rather than in the object itself. This has the added benefit that the same sidecar file can be stored on just about any technology: LTO, another cloud provider with its on metadata storage idiosyncrasies, etc. Also, the sidecar file can hold an almost limitless amount of metadata.
Metadata Specification
Another aspect that came to our attention whilst using AWS S3 APIs between our products and third parties is that having an API only is only half of the story when it comes to building rich integrations and media workflows.
Basically, it would be really helpful for all of the tools if there existed a media file metadata specification for object storage. The specification could define fields around projects, relative paths, media types, etc. There are specifications within, e.g., IMF that could be drawn upon. But how to bring those up to a higher level and in a format that aids applications to collaborate at an object storage level?
So in summary, we've had to go about this ourselves and have defined a high-level metadata specification, or, as we term it internally: a Metadata API. All our applications use that specification, be it via filesystem access, S3 access, FTP, DropSpot or Vision. It allows our applications to collaborate seamlessly. And where we integrate with 3rd parties, we pull their metadata into our standardised format. We'd love to see a specification body (hint hint DPP / SMPTE) look at whether a wider specification would be useful.
Conclusion
Amazon S3 Object Storage is evolving quickly to make simple to use cloud storage. With our tools, we combine different tiers. But whilst it has object metadata commands in its API we find that for transportability, avoiding metadata limitations within an API (either in AWS or any other platform), non-proprietary access and no vendor lock-in keeping metadata in a sidecar file is key.
And, with object APIs, keeping metadata is only half the story for application interoperability. The other half of the story is keeping the metadata in a format that can be easily understood and shared. We need defining metadata specifications if we want to achieve portable integrations for our customers.
About Object Matrix
Object Matrix is the award winning software company that pioneered object storage and the modernisation of media archives. Our on-prem and hybrid cloud storage solutions bring operational and financial benefits to our customers by securely managing content at every stage of its lifecycle; from ingest and nearline to archive and distribution. Deployed where you need it, our technology is non-proprietary, integrates into existing workflows, enabling you to work locally and share globally.
Our flexible approach, coupled with our focus on the media industry, me
Europe Stories
04/08/2024
Dalet Appoints Santiago Solanas as CEO to Lead Next Era of Growth and Innovation
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet and Veritone Reach Agreement to Distribute, Transact and Monetize Media Archives
Dalet, a leading technology and service provider for media-rich organizations, a...
28/04/2024
Mediahaus delivers the first SRT live-streaming sports production over 5G with URSA Broadcast G2
Mediahaus delivers the first SRT live-streaming sports production over 5G with U...
25/04/2024
Opinion: charting a cloud driven future
M2A Medias Matt Hughes asks, is it time for news broadcasting to shift to the cloud? By Contributor Published: April 25, 2024 M2A Media's Matt Hughes ...
25/04/2024
Czech mates
The writer and director of a hit Czech series tell Robert Shepherd about their approach to portraying real-life Pakistani kidnappings on the small screen, while...
25/04/2024
Molinaire Creative Group opens new Liverpool facility
Notorious DIT will work with the Liverpool Film Office to increase opportunities for regional talent By Matthew Corrigan Published: April 25, 2024 Notorio...
25/04/2024
IBM Transforms the Storage Ownership Experience with IBM Storage Assurance
LONDON, UK, April 25, 2024 IBM (NYSE: IBM) today announced new storage capabilities that give clients choice and control in the data center to maximise perfor...
25/04/2024
Socio-ecological impact through reuse of IT hardware
Social and Ecological Impact through Reuse of IT Hardware Social & green IT: Arvato Systems and AfB Securing three jobs for people with disabilities Savings...
25/04/2024
AMATEUR DRAMA TAKES CENTRE STAGE AS RT ALL IRELAND DRAMA FESTIVAL LAUNCHES IN THE ABBEY THEATRE
Amateur drama took centre stage as the All Ireland Drama Festival was officially...
25/04/2024
AI-assisted optronics: an unprecedented European project to increase combat perception capabilities
Facebook Twitter LinkedIn The European Commission signed the Grant Agreeme...
24/04/2024
UKTV's pay TV channel Alibi joins forces with S4C and All3Media International for new legal crime drama series Mudtown
Cast set to star include Erin Richards and Tom Cullen Wednesday 24 April 2024 ...
24/04/2024
K-Hip-Hop's Biggest Names Take Center Stage on KrOWN X HAN 2024'
This month marked the return of Spotify's HAN, our collaborative program that brings the unique sounds and culture of Korean hip-hop to the world through ex...
24/04/2024
RADAR Italy Returns With 6 Artists Transforming the Country's Music Landscape
As Spotify's program to spotlight rising talent from around the world, RADAR...
24/04/2024
Spotify Partners With Les Flammes Music Awards To Celebrate French Rap
Spotify is once again teaming up with Les Flammes, the only music awards dedicated to rap, R&B, Afrobeats, and popular culture in France. Spotify helped create...
24/04/2024
Taylor Swift's THE TORTURED POETS DEPARTMENT' Becomes Spotify's Most-Streamed Album in One Week
Records smashed. History made. It's been just five days since Taylor Swift u...
24/04/2024
Authentic Perspectives
The first feature film from writer-director Shaun Seneviratne, Ben and Suzanne: A Reunion in 4 Parts depicts the struggle between love and duty as Ben (Sathya S...
24/04/2024
Melanie Georgieva Joins Panalux as Long Form Sales Director
LONDON, England (24 April 2024) Panalux, a leading rental provider of lighting and power solutions for the motion-picture industry and part of Panavision'...
24/04/2024
Deutsche Telekom Powers New MagentaTV Service With Broadpeak's Cloud DVR Solution
April 24, 2024 -- Deutsche Telekom Powers New MagentaTV Service With Broadpea...
24/04/2024
VEON Announces 2024 AGM and Board Nominees
24 Apr 2024 VEON Announces 2024 AGM and Board Nominees Amsterdam, 24 April 2024 - VEON Ltd. (NASDAQ: VEON, Euronext Amsterdam: VEON), a global digital operator...
24/04/2024
Rohde & Schwarz and IPG Automotive unveil a complete Hardware-in-the-Loop automotive radar test solution
Rohde & Schwarz and IPG Automotive unveil a complete Hardware-in-the-Loop automo...
24/04/2024
Nurturing a new look for Huntapac
Posted on 24 Apr 2024 Creating a contemporary brand update that reflects a company's strong heritage can often pose a challenge, yet its a task ICG embrace...
24/04/2024
2024-04-24
Beginning in May, a special Today at Apple series titled Made for Business will offer small business owners and entrepreneurs free opportunities to learn how ...
24/04/2024
Print beats other media channels to deliver largest ROI
% of profit volume % of advertising investment % of full payback % of sustained parback % of short-term payback % of immediate payback Full payback ROI S...
23/04/2024
Tackling lies online: the answer may be all in the mind
Psychology can play an important role in helping us understand what audiences respond to (Dr Jon Roozenbeek, Cambridge Social Decision-Making Lab) For newsroom...
23/04/2024
Spotify Reports First Quarter 2024 Earnings
Today, we announced our first quarter 2024 earnings. In a solid start to the year, Spotify reported Monthly Active Users grew 19% Y/Y to 615 million. Subscri...
23/04/2024
Spotify rapporterar intkter fr frsta kvartalet i 2024
Idag publicerar Spotify sina finansiella resultat f r f rsta kvartalet 2024. ret har inletts stabilt: M natliga aktiva anv ndare kade med 19 % j mf rt med f...
23/04/2024
Discover Your K-Pop Persona With Spotify's New Interactive Experience
In the last decade, as K-Pop cemented its status as a global phenomenon, Spotify has helped fans around the world keep their fingers on the pulse at every step ...
23/04/2024
Creative UK launches manifesto calling for more investment in creative R&D; revised approach to trade and exports
Entitled Our Creative Future, the manifesto calls for radical new action' t...
23/04/2024
Audiotonix secures landmark' investment
The company has secured investment from private equity firm PAI Partners, which also holds a stake in outside broadcasting company EMG By Jenny Priestley Pub...
23/04/2024
James Cameron backs Marlow Film Studios proposal
Cameron also said he recognised Marlow as a potential UK and European base for his new company, LIGHTSTORM 3D By Matthew Corrigan Published: April 23, 2024 ...
23/04/2024
IBC launches 2024 Innovation Awards with new categories
The five categories cover Content Creation, Content Distribution, Content Everywhere, Social Impact and Environment & Sustainability By Matthew Corrigan Publ...
23/04/2024
Pascal Fratellia leaves Olympics role at EBU
He played a central role in securing key broadcast rights, including the FIFA World Cup 2018 and 2022 By Matthew Corrigan Published: April 23, 2024 He pla...
23/04/2024
Discover what is new: Mistika Boutique and Ultima enhance conforming, rendering and project collaboration
Published On: April 23rd, 2024 data-orig-src=https://www.sgo.es/wp-content/upl...
23/04/2024
Rohde & Schwarz introduces new R&S SMB100B microwave signal generator for analog signal generation up to 40 GHz
Rohde & Schwarz introduces new R&S SMB100B microwave signal generator for analog...
23/04/2024
FilmLight Colour Awards welcomes 2024 entries and introduces new Emerging Talent' award
Entries open from 1 May 31 July to colourists on any grading platform The Fil...
23/04/2024
Niels Steenstrup appointed CEO of Thales InFlyt Experience
Facebook Twitter LinkedIn We thank TK for his leadership and dedication to our customers' success. We wish him well on his retirement, said Yannick ...
23/04/2024
Thales opens a new System Testing Centre for German F126 project
Facebook Twitter LinkedIn On April 23rd 2024, Thales, the German Navy and German military procurement agency, BAAINBw (Federal Office of Bundeswehr Equipm...
22/04/2024
UKTV Ventures partners with Elder, in airtime-for-equity deal
22 April 2024 - Award-winning broadcaster UKTV has today announced it has invested in the award-winning online care marketplace, Elder, using its multi-million ...
22/04/2024
Taylor Swift's THE TORTURED POETS DEPARTMENT' Breaks Multiple Spotify Records in Less Than 24 Hours
It's a beautiful day to be a Swiftie. On April 19, Taylor Swift dropped her ...
22/04/2024
Adapting to Landscapes With LEE Filters
In this video, LEE Filters ambassador Jordan Banks ventures into Snowdonia National Park in Wales. Faced with less-than-ideal weather conditions, he adapts his ...
22/04/2024
Ross Video expands graphics portfolio with Bannister Lake Software acquisition
The Chameleon solution provides access to real-time data from a range of sources, enabling users to construct tickers, L-bars, snipes, web widgets and full fram...
22/04/2024
What happened in Vegas: 2024 edition
TVBEuropes editor Jenny Priestley highlights some of the main talking points at this years NAB Show By Jenny Priestley Published: April 22, 2024 TVBEurope...
22/04/2024
Rohde & Schwarz expands footprint in India by opening new state-of-the-art facility in Bengaluru
Rohde & Schwarz expands footprint in India by opening new state-of-the-art facil...
22/04/2024
Comscore at the 19th Annual Needham Technology, Media & Consumer Conference
Comscore at the 19th Annual Needham Technology, Media & Consumer Conference RESTON, Va., April 22, 2024 Comscore (Nasdaq: SCOR), a trusted partner for plannin...
22/04/2024
CODA Audio system is Crowning Glory for Brooklyn Cultural Hub
Crown Hill Theatre in Brooklyn, New York, officially opened its doors with a red-carpet event in early December 2023. The brainchild of entrepreneur and musicia...
22/04/2024
Cian Cobhin's An Taobh Tuathail marks 25 years on air
An Taobh Tuathail (ATT), RT Raidi na Gaeltachta's flagship alternative radio, presented by Cian C obh in show turns 25 in May 2024. The acclaimed music ...
20/04/2024
Taylor Swift's THE TORTURED POETS DEPARTMENT Breaks Multiple Spotify Records in Less Than 24 Hours
It's a beautiful day to be a Swiftie. On April 19, Taylor Swift dropped her ...
19/04/2024
Thank You for visiting us at NAB 2024!
Thank You for visiting us at NAB 2024! Thank you for joining us at the ToolsOnAir booth during NAB Show 2024! We were delighted to showcase our latest inno...
19/04/2024
How Spotify Approaches Safety With Sarah Hoyle, Global Head of Trust and Safety
Spotify's mission is to unlock the potential of human creativity by giving a million artists the opportunity to live off their art and billions of fans the ...
19/04/2024
Spotify and Taylor Swift Create a One-of-a-Kind Library To Celebrate THE TORTURED POETS DEPARTMENT'
It's a beautiful day to be a Swiftie. Today, Taylor Swift dropped her 11th s...