S3: The API Nobody Loves but Everybody Uses
15/05/2018
At Object Matrix, we are big users of Amazon S3 API and with so much talk about organisations offering S3' interfaces I thought I would share our experiences.
We have been using S3 API for quite some time in two areas:
Development of applications accessing S3 buckets either at Amazon or at systems providing S3 interface.
We developed an S3 interface to our product MatrixStore to facilitate integrations with partners and existing applications using S3.
In each area, we obtained a better knowledge of what it really means dealing with S3.
Client Applications
The first thing we noticed about S3 is that it does not provide great granularity of commands in certain areas but, we understand why since it must be tied to the storage behind the API and the type of service Amazon offers.
Another thing we noticed is that the implementation behind S3 is not very strict with the API definition. That is great for developers, i.e. you can make a few mistakes calling the API with wrong parameters and the system still works. However, that makes it difficult if you are trying to develop a backend that simulates the behaviour of an S3 server, as I describe later.
I must admit Amazon did very well launching an API and infrastructure so easy to use that it attracted developers and made the API a de-facto standard. These days, everybody talks about S3.
However, as an external company working with APIs and partners, we are kind of forced to use S3 even knowing it's not ideal. It does not answer everybody's needs and it didn't come from any standardisation body.
It was interesting to see how the API evolved. It started as a REST API defined by Amazon but it has evolved to have multiple wrappers in different programming languages. That made S3 even easier to use but at the same time, more difficult to support as a backend.
Of course, being a REST API has its own benefits and issues, e.g., the speed of a REST API is never going to be great compared to optimised network protocols something that is important when transferring large media files.
S3 Backend
Implementing an S3 backend has been a challenge for several reasons. Many vendors claim to be S3 compliant (there is no such thing) but in reality we are all S3ish.
Since Amazon's implementation is very relaxed with the API definition, we found that several applications were using the API in an inconsistent and sometimes erroneous way, but they worked well with Amazon's S3 implementation.
Therefore, we had to adapt our implementation of the interface to accept those errors' since Amazon is the reference so how to explain to a developer he is using S3 in the wrong way if it works with Amazon!
The different programming language wrappers around S3 made it easier for developers to use but even more difficult for the implementation of S3 since some of those wrappers call S3 in not by-the-book ways which are hidden to developers.
Another aspect of S3 that complicates its implementation is that there are no official tests to certify the implementation. We found some frameworks but due to the nature of Amazon's implementation, even those frameworks have many failures using Amazon so it's very difficult to use them as a reference although they definitely help and we thank the people making those tests available.
Conclusion
Having S3 so easy to program against and to use has helped many developers and companies to use it in multiple areas; it has become a de-facto standard. That is one of the reasons for the success of Amazon Web Services.
We are and will continue using S3 as an API for application development and backend implementation but it wasn't as straightforward to simulate the backend as we had originally hoped for.
We think it would be for good if Amazon leads S3 standardisation for official test frameworks to appear. Everybody could share a common basis for development, products will increase their quality and after all, the more people using S3, the better for Amazon.
About Object Matrix
Object Matrix is an award winning UK based software company that pioneered Digital Content Governance (DCG), object storage and the modernisation of digital video workflows. Our media focused private and hybrid cloud solutions are tightly integrated into file based and IP workflows and bring economic and operational benefit to all of our customers. Our flagship product, MatrixStore, is used by the world's largest organisations that create and distribute video content, including NBC, TV Globo, MSG-N, the BBC & BT.
www.object-matrix.com
LINK: | http://object-matrix.com/s3-the-api-nobody-loves-but-everybody-uses/... |
See more stories from matrixstore |
Most recent headlines
04/08/2024
Dalet Appoints Santiago Solanas as CEO to Lead Next Era of Growth and Innovation
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet and Veritone Reach Agreement to Distribute, Transact and Monetize Media Archives
Dalet, a leading technology and service provider for media-rich organizations, a...
04/05/2024
Nielsen: Asian American Audiences Spend as Much Time on Mobile as TV
NEW YORK As the Asian Pacific American Heritage Month kicks off in May, Nielsen has released an extensive new report diving into their media habits with data sh...
04/05/2024
Comcast Makes $1M Commitment to Military-Serving Nonprofits
PHILADELPHIA As National Military Appreciation Month gets underway, Comcast has announced several new initiatives to help veterans, service members, and their f...
04/05/2024
FCC Unveils Agenda for May Open Meeting
WASHINGTON, D.C. Federal Communications Commission chairwoman Jessica Rosenworcel has announced a tentative agenda for the May Open Commission Meeting scheduled...
04/05/2024
Enhancements Made to Key 5G Convergence Standards
The Broadband Forum has announced major enhancements to key 5G convergence standards that the group said will help advance next-generation applications, improve...
04/05/2024
Spotify Wrapped Campaign Spot Graded With DaVinci Resolve Studio
Spotify Wrapped Campaign Spot Graded With DaVinci Resolve Studio Brie Clayton May 3, 2024 0 Comments Blackmagic Design today announced that Colorist M...
04/05/2024
Three Boston Conservatory at Berklee Alums Nominated for Tony Awards
Three Boston Conservatory at Berklee Alums Nominated for Tony Awards An additional eight alums and two current students performed in nominated productions. ...
03/05/2024
Sudan: New WhatsApp Course Equips Journalists to Report on Conflict Related Sexual Violence
An innovative learning tool to help media and civil society better understand ho...
03/05/2024
Slow Explores an Unusual Relationship With Sincere Romance
PARK CITY, UTAH - JANUARY 21: (L-R) Actors K stutis Cic nas and Greta Grinevi i t , Director Marija Kavtaradze and Producer Marija Razgute attend the 2023 Sunda...
03/05/2024
Be Honest With Your Art': Danny Ocean Reflects on Sudden Stardom and His Deeply Personal New Album
If you aren't familiar with Danny Ocean's music, it's only a matter ...
03/05/2024
Spotify and ELLE Collaborate in Celebration of Emerging Women in Music
Spotify has an unwavering commitment to supporting emerging artists across all genres, to helping them launch and thrive in their careers, and to connecting the...
03/05/2024
La Msica Mexicana no es solamente un fenmeno en Mxico y Estados Unidos, tambin en toda Amrica Latina
Como uno de los sonidos de mayor expansi n en el mundo, la M sica Mexicana va m ...
03/05/2024
Msica Mexicana Isn't Just a Phenomenon in Mexico and the U.S.-It's Taking Over Latin America.
As one of the fastest-growing sounds worldwide, M sica Mexicana is more than jus...
03/05/2024
Fubo Loses Subs in Q1 But Exceeds Guidance and Reduces Losses
NEW YORK FuboTV reported that it lost 110,000 subs in Q1 2024 but it exceeded its guidance for the quarter with double digit year-over-year increases across key...
03/05/2024
Only 3 Weeks Left To Enter The Creative Industries Most Award
Only 3 Weeks Left To Enter The Creative Industries Most Award Brie Clayton May 3, 2024 0 Comments Creativepool is giving a last call to the creative i...
03/05/2024
WePlay Studios Transforms the Future of Live Event Storytelling with AJA Gear
WePlay Studios Transforms the Future of Live Event Storytelling with AJA Gear Brie Clayton May 3, 2024 0 Comments By 2032, the esports market is expec...
03/05/2024
The Business of TV News: Shannon Bream Singles Out Key Issue in 2024 Election
WASHINGTON Shannon Bream, Fox News Sunday anchor and chief legal correspondent, sat for a keynote chat at The Business of TV News, with Tom Umstead, senior co...
03/05/2024
ABC News Anchor Martha Raddatz Says Local News Is Everything'
WASHINGTON Asked how important is local news, Martha Raddatz, who has covered foreign conflicts for decades and moderated presidential debates for ABC News, s...
03/05/2024
Paramount Stock Jumps on Report of $26 Billion Bid From Sony Pictures and Apollo Global
Sony Pictures and Apollo Global Management have made a $26 billion cash offer fo...
03/05/2024
CBS Shares 2024-2025 Schedule
CBS announced its 2024-2025 primetime schedule, which features the new dramas NCIS: Origins, Matlock and Watson, and new comedies Poppa's House and Georgie ...
03/05/2024
Fubo Continues To Reduce Red Ink Despite Losing 110,000 Subs in Q1
Sports-focused streaming service Fubo said it reduced its red ink in the first quarter, despite losing 110,000 subscribers since the end of the year....
03/05/2024
Walmart Ready to Release 4K Hybrid Streaming/Smart Speaker
Walmart is reportedly about to release a new 4K streaming box/smart speaker hybrid device for Google TV that could retail for as low as $50, according to a repo...
03/05/2024
Limecraft updates content delivery platform
Enhancements to its Delivery Workspace solution provide increased content delivery robustness and security By Matthew Corrigan Published: May 3, 2024 Enha...
03/05/2024
Sony and Apollo submit $26 billion bid for Paramount Global
According to reports, if the bid is successful, Sony would be the significant majority shareholder of the new private company By Jenny Priestley Published: M...
03/05/2024
Global Academy launches Manchester media school
Located in Manchesters Spinningfields district, the course aims to equip students with the skills, knowledge and industry insight for a career in the creative s...
03/05/2024
New Lightbridge C-Move Core
Lightbridge has created a new Cine Reflect Lighting System kit, dedicated to enabling artistic lighting with consideration of today's productivity practical...
03/05/2024
KPBS Modernizes Infrastructure With Major AES67 Control R...
Broadcast media systems integrator BeckTV and Lawo worked together on a large-scale AES67 build of radio control rooms at KPBS, San Diego's NPR and PBS memb...
03/05/2024
SmallHD Free PageOS 6 Update Adds Powerful New Capabiliti...
Raleigh, NC, April 12, 2024 SmallHD announces PageOS 6, a powerful software update free for owners of SmallHD monitors, featuring increased functionality, exp...
03/05/2024
SmallHD Launches Quantum 32 Quantum Dot OLED HDR Referenc...
SmallHD announces the Quantum 32 a new 31.5 OLED monitor designed to provide reference HDR/SDR images for post-production final color grading, as well as cri...
03/05/2024
How WePlay Studios is Transforming the Future of Live Eve...
By 2032, the esports market is expected to grow to $9.29 billion, bolstered by a global player count and fan following that both continue to climb. The 2023 Lea...
03/05/2024
Amagis Latest FAST Report Unveils the Rise of a Diverse G...
Amagi, the global leader in cloud-based SaaS technology for broadcast and connected TV (CTV), today announced that the 11th edition of the Amagi Global FAST Rep...
03/05/2024
WDR Filmhaus to Implement Riedel Infrastructure
Riedel Communications today announced that Westdeutscher Rundfunk (WDR) is relying on a media and intercom network based on Riedel technology for the renewal of...
03/05/2024
TF1 Chooses Broadpeak to Power Targeted Advertising for N...
Broadpeak , a leading provider of content delivery network (CDN) and video streaming solutions for content providers and pay-TV operators worldwide, announced t...
03/05/2024
Limecraft Announces New Platform Update and Preview of MI...
Media technology innovator Limecraft will exhibit the latest updates to its online platform on stand G79 at the 15-16 May 2024 Media Production & Technology Sho...
03/05/2024
Prism Sound Announces Partners For 8 Hours At Rockfield 2...
Audio interface specialist Prism Sound is partnering with console manufacturer Solid State Logic, professional monitor manufacturer PMC and acoustic panel and d...
03/05/2024
Intinor brings excellence in contribution streaming to MP...
Intinor, Sweden's leading developer of products and solutions for high-quality video over the internet, is set to join forces with its UK partner Zest Techn...
03/05/2024
NAKIVO Reports 10 Percent Revenue Growth in EMEA Region
NAKIVO Inc., a fast-growing software company dedicated to protecting physical, virtual, cloud, and SaaS environments, announced its Q1 2024 financial results to...
03/05/2024
Hiltron Introduces Field-Upgradable Motorisation Kit for...
Hiltron Communications announces an addition to its range of satellite communication products and systems: a field-upgradable motorisation kit specifically desi...
03/05/2024
LiveU Showcases Story-Centric Workflows and the LiveU Eco...
Visit LiveU at Stand D30 With this year an exceptional one for major sporting events alongside multiple, key elections, the power and flexibility of LiveU'...
03/05/2024
Agile Content appoints Koldo Unanue as new CEO to boost t...
Agile Content, the leader in the development of technology and solutions for the provision of television services over the Internet, announces Koldo Unanue as i...
03/05/2024
Fubo, Dish, DirecTV & Others Urge Congress to Probe New Sports Streaming JV
Eight co-signers representing streaming services, pay TV operators and public interest groups have sent a letter to Congressional leaders urging their Committee...
03/05/2024
Brightcove Integrates AWS-Powered Generative AI Solution
BOSTON Brightcove has announced that it has implemented Amazon Q Business, a new generative AI assistant on Amazon Web Services (AWS)....
03/05/2024
CBS Claims 16th Straight Prime Time Viewing Title
NEW YORK CBS has declared that it will finish the 2023-24 season as Americas Most-Watched Network in primetime for the 16th consecutive season....
03/05/2024
Survey: U.S. Business and Consumers Plan to Increase Tech Spending
A new survey indicates that U.S. businesses and consumers had an overall positive attitude towards their technology spending plans in the first quarter of 2024,...
03/05/2024
Sony, Apollo Make $26B Bid for Paramount
Sony and Apollo Global Management have sent an offer letter to Paramount Global's board of directors making an $26 billion all cash offer for Paramount Glob...
03/05/2024
EMG / Gravity Media Grow Executive Team
EMG / Gravity Media has appointed Charlie Cubbon chief operating officer (COO) and Jamie Hindhaugh regional CEO for the U.K., U.S., Australia and the Middle Eas...
03/05/2024
WRAL-TV Brain Game Producer Randy Mews Wraps Season Taping Before Retirement
Final 2024 Season Games to Air Saturdays on WRAL at 10:30am in May, Championship on June 1 Taping for the final episodes of the 2024 season of WRAL-TV's Br...
03/05/2024
Estrella MediaCo Launches FAST Channels With Curiosity (NewFronts)
Estrella MediaCo said it is expanding its connected TV portfolio by launching three new channels with Curiosity that will first appear on Samsung TV Plus....