S3: The API Nobody Loves but Everybody Uses

-- S3: The API Nobody Loves but Everybody Uses

At Object Matrix, we are big users of Amazon S3 API and with so much talk about organisations offering S3' interfaces I thought I would share our experiences.

We have been using S3 API for quite some time in two areas:

Development of applications accessing S3 buckets either at Amazon or at systems providing S3 interface.

We developed an S3 interface to our product MatrixStore to facilitate integrations with partners and existing applications using S3.

In each area, we obtained a better knowledge of what it really means dealing with S3.

Client Applications

The first thing we noticed about S3 is that it does not provide great granularity of commands in certain areas but, we understand why since it must be tied to the storage behind the API and the type of service Amazon offers.

Another thing we noticed is that the implementation behind S3 is not very strict with the API definition. That is great for developers, i.e. you can make a few mistakes calling the API with wrong parameters and the system still works. However, that makes it difficult if you are trying to develop a backend that simulates the behaviour of an S3 server, as I describe later.

I must admit Amazon did very well launching an API and infrastructure so easy to use that it attracted developers and made the API a de-facto standard. These days, everybody talks about S3.

However, as an external company working with APIs and partners, we are kind of forced to use S3 even knowing it's not ideal. It does not answer everybody's needs and it didn't come from any standardisation body.

It was interesting to see how the API evolved. It started as a REST API defined by Amazon but it has evolved to have multiple wrappers in different programming languages. That made S3 even easier to use but at the same time, more difficult to support as a backend.

Of course, being a REST API has its own benefits and issues, e.g., the speed of a REST API is never going to be great compared to optimised network protocols something that is important when transferring large media files.

S3 Backend

Implementing an S3 backend has been a challenge for several reasons. Many vendors claim to be S3 compliant (there is no such thing) but in reality we are all S3ish.

Since Amazon's implementation is very relaxed with the API definition, we found that several applications were using the API in an inconsistent and sometimes erroneous way, but they worked well with Amazon's S3 implementation.

Therefore, we had to adapt our implementation of the interface to accept those errors' since Amazon is the reference so how to explain to a developer he is using S3 in the wrong way if it works with Amazon!

The different programming language wrappers around S3 made it easier for developers to use but even more difficult for the implementation of S3 since some of those wrappers call S3 in not by-the-book ways which are hidden to developers.

Another aspect of S3 that complicates its implementation is that there are no official tests to certify the implementation. We found some frameworks but due to the nature of Amazon's implementation, even those frameworks have many failures using Amazon so it's very difficult to use them as a reference although they definitely help and we thank the people making those tests available.

Conclusion

Having S3 so easy to program against and to use has helped many developers and companies to use it in multiple areas; it has become a de-facto standard. That is one of the reasons for the success of Amazon Web Services.

We are and will continue using S3 as an API for application development and backend implementation but it wasn't as straightforward to simulate the backend as we had originally hoped for.

We think it would be for good if Amazon leads S3 standardisation for official test frameworks to appear. Everybody could share a common basis for development, products will increase their quality and after all, the more people using S3, the better for Amazon.

About Object Matrix

Object Matrix is an award winning UK based software company that pioneered Digital Content Governance (DCG), object storage and the modernisation of digital video workflows. Our media focused private and hybrid cloud solutions are tightly integrated into file based and IP workflows and bring economic and operational benefit to all of our customers. Our flagship product, MatrixStore, is used by the world's largest organisations that create and distribute video content, including NBC, TV Globo, MSG-N, the BBC & BT.

www.object-matrix.com

LINK:	http://object-matrix.com/s3-the-api-nobody-loves-but-everybody-uses/...
	See more stories from matrixstore

Most recent headlines