Jump to content

Amazon S3

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by Leoinspace (talk | contribs) at 05:08, 8 June 2016 (→Notable users: Added Cloud Gateways). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Not to be confused with Galaxy S3
Amazon Simple Storage Service
File:AWS Simple Icons Storage Amazon S3 Bucket with Objects.svg
Amazon S3 Bucket with Objects
Type of site
Cloud storage
Available inEnglish
OwnerAmazon.com
URLaws.amazon.com/s3/
IPv6 supportNo
CommercialYes
RegistrationRequired

Amazon S3 (Simple Storage Service) is an online file storage web service offered by Amazon Web Services. Amazon S3 provides storage through web services interfaces (REST, SOAP, and BitTorrent).[1] Amazon launched S3, its first publicly available web service, in the United States in March 2006[2] and in Europe in November 2007.[3]

At its inception, Amazon charged end users US$0.15 per gigabyte-month, with additional charges for bandwidth used in sending and receiving data, and a per-request (get or put) charge.[4] On November 1, 2008, pricing moved to tiers where end users storing more than 50 terabytes receive discounted pricing.[5] Amazon says that S3 uses the same scalable storage infrastructure that Amazon.com uses to run its own global e-commerce network.[6]

Amazon S3 is reported to store more than 2 trillion objects as of April 2013.[7] This is up from 102 billion objects as of March 2010,[8] 64 billion objects in August 2009,[9] 52 billion in March 2009,[10] 29 billion in October 2008,[5] 14 billion in January 2008, and 10 billion in October 2007.[11] S3 uses include web hosting, image hosting, and storage for backup systems. S3 guarantees 99.9% monthly uptime service-level agreement (SLA),[12] that is, not more than 43 minutes of downtime per month.[13]

Design

At AWS Summit 2013 NYC, CTO Werner Vogels announces 2 trillion objects stored in S3.

Details of S3's design are not made public by Amazon, though it clearly manages data with an object storage architecture. According to Amazon, S3's design aims to provide scalability, high availability, and low latency at commodity costs.

S3 is designed to provide 99.999999999% durability and 99.99% availability of objects over a given year,[14] though there is no service-level agreement for durability.

S3 stores arbitrary objects (computer files) up to 5 terabytes in size, each accompanied by up to 2 kilobytes of metadata. Objects are organized into buckets (each owned by an Amazon Web Services account), and identified within each bucket by a unique, user-assigned key. Amazon Machine Images (AMIs) which are used in the Elastic Compute Cloud (EC2) can be exported to S3 as bundles.[15]

Buckets and objects can be created, listed, and retrieved using either a REST-style HTTP interface or a SOAP interface. Additionally, objects can be downloaded using the HTTP GET interface and the BitTorrent protocol.

Requests are authorized using an access control list associated with each bucket and object.

Bucket names and keys are chosen so that objects are addressable using HTTP URLs:

  • http://s3.amazonaws.com/bucket/key
  • http://bucket.s3.amazonaws.com/key
  • http://bucket/key (where bucket is a DNS CNAME record pointing to bucket.s3.amazonaws.com)

Because objects are accessible by unmodified HTTP clients, S3 can be used to replace significant existing (static) web hosting infrastructure.[16] The Amazon AWS Authentication mechanism allows the bucket owner to create an authenticated URL with time-bounded validity. That is, someone can construct a URL that can be handed off to a third-party for access for a period such as the next 30 minutes, or the next 24 hours.

Every item in a bucket can also be served up as a BitTorrent feed. The S3 store can act as a seed host for a torrent and any BitTorrent client can retrieve the file. This drastically reduces the bandwidth costs for the download of popular objects. While the use of BitTorrent does reduce bandwidth, AWS does not provide native bandwidth limiting and as such users have no access to automated cost control. This can lead to users on the 'free-tier' S3 or small hobby users amassing dramatic bills. AWS representatives have previously stated that such a feature was on the design table from 2006 to 2010[17] but have recently stated the feature is no longer in development.[18]

A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in later data mining operations.[19]

Hosting entire websites

Amazon S3 provides options to host static websites with Index document support and error document support.[20] This support was added as a result of user requests dating at least to 2006.[21] For example, suppose that Amazon S3 was configured with CNAME records to host http://subdomain.example.com/. In the past, a visitor to this URL would find only an XML-formatted list of objects instead of a general landing page (e.g., index.html) to accommodate casual visitors. Now, however, websites hosted on S3 may designate a default page to display, and another page to display in the event of a partially invalid URL.

Notable users

Photo hosting service SmugMug has used S3 since April 2006. They experienced a number of initial outages and slowdowns, but after one year they described it as being "considerably more reliable than our own internal storage" and claimed to have saved almost $1 million in storage costs.[22]

There are various User Mode File System (FUSE)-based file systems for Unix-like operating systems (Linux, etc.) that can be used to mount an S3 bucket as a file system. Note that as the semantics of the S3 file system are not that of a Posix file system, the file system may not behave entirely as expected.[23]

Apache Hadoop file systems can be hosted on S3, as its requirements of a file system are partially met by S3.[24] As a result, Hadoop can be used to run MapReduce algorithms on EC2 servers, reading data and writing results back to S3.

Netflix Use Amazon Web Services for their storage and compute operations with S3 being their System of record. To address the consistency limitations of S3, Netflix implemented a tool, S3mper.[25] This stores the fileystem metadata: filenames, directory structure and permissions in Amazon DynamoDB.

reddit is hosted on S3.[26]

Dropbox,[27] Bitcasa,[28] and Tahoe-LAFS-on-S3,[29] among others, use S3 for online backup and synchronization services.

Mojang hosts Minecraft game updates and player skins on S3.[30]

Tumblr, Formspring, and Pinterest host images on S3.

Swiftype's CEO has mentioned that the company uses S3.[31]

S3 was used in the past by some enterprises as a long term archiving solution, until Amazon Glacier was released.[citation needed]

The API has become a popular method for object storage.[32] As a result, more and more applications have been built to natively support the S3 API.[33] This includes applications that write data to AWS S3, as well as to S3-compatible object stores:

Type Company Name Product
Client Backup Haystack Software LLC Arq backup[34]
Client Backup CloudBerry Lab CloudBerry Backup[35]
Client Backup open-source Duplicati[36]
File Browser odrive odrive[37]
MySQL Backup Oracle MySQL Enterprise Backup
Oracle Database Backup Oracle Oracle Secure Backup Cloud Manager[38]
Server Backup Commvault Simpana[39]
Server Backup Veritas NetBackup[40]
Server Backup Asigra Asigra Cloud Backup[41]
Server Backup Rubrik Rubrik[42]
Cloud Storage Gateway CTERA Networks C00 Series[43]
Cloud Storage Gateway Avere FXT Series[44]
Cloud Storage Gateway EMC CloudArray[45]
Cloud Storage Gateway Microsoft StorSimple[46]
Cloud Storage Gateway Nasuni NF Series[47]
Cloud Storage Gateway NetApp Altavault[48]
Sync & Share Storage Made Easy SME

Amazon S3 logs

Amazon S3 allows users to enable or disable logging. If enabled, the logs are stored on Amazon S3 buckets which can then be analyzed. These logs contain useful information like,

  • Date / time of access to your content
  • Protocol used etc.
  • HTTP Status
  • Turn around time

These logs can be analyzed and managed by using third-party tools such as S3Stat, Cloudlytics, Qloudstat, or AWS Stats.

S3 API and competing services

The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 API. These services use the standard programming interface; however, they are differentiated by their underlying technologies and supporting business models.[49] A cloud storage standard (like electrical and networking standards) enables competing service providers to design their services and clients using different piece parts in different ways yet still communicate and provide the following benefits:[50]

  1. Increase competition by providing a set of rules and a level playing field, encouraging market entry by smaller companies which might otherwise be precluded.
  2. Encourage innovation by cloud storage & tool vendors, and developers because they can focus on improving their own products and services instead of focusing on compatibility.
  3. Allow economies of scale in implementation (i.e., if a service provider encounters an outage or as clients outgrow their tools and need faster operating systems or tools, they can easily swap out solutions).
  4. Provide timely solutions for delivering functionality in response to demands of the marketplace (i.e., as business growth in new locations increases demand, clients can easily change or add service providers simply by subscribing to the new service).

Examples of competing S3 compliant storage implementations include:

Amazon S3 tools

Amazon S3 provides an API (Application programming interface) for third-party developers. It describes various API operations, related request and response structures, and error codes.[57] Web services interface can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers. Today there are different kinds of file managers for Amazon S3. An effective solution for Amazon provides a user interface to Amazon S3 accounts, files and buckets, allowing you to browse, create and delete files and buckets.[58]

See List of applications that can use S3 API for more tools and applications.

Notes

  1. ^ Amazon S3, Cloud Computing Storage for Files, Images, Videos. Aws.amazon.com (2006-03-01). Retrieved on 2013-08-09.
  2. ^ "Amazon Web Services Launches "Amazon S3"" (Press release). Amazon.com. 2006-03-14. Retrieved 2015-09-22.
  3. ^ "Amazon Web Services Offers European Storage for Amazon S3" (Press release). Amazon.com. 2007-11-06. Retrieved 2015-09-22.
  4. ^ "Amazon S3 Pricing". Amazon.com. 2009-02-05. Retrieved 2014-05-02.
  5. ^ a b "Amazon S3 - Busier Than Ever". Amazon.com. 2008-10-08.
  6. ^ The same data storage infrastructure that Amazon uses to run its own global network of web sites
  7. ^ - Amazon S3 - Two Trillion Objects, 1.1 Million Requests / Second
  8. ^ Brian Lillie of Equinix said that Amazon now is hosting 102 billion objects in S3
  9. ^ S3 (Amazon's Simple Storage Service) alone has over 64 billion objects in it.
  10. ^ Just a year ago, there were 18 billion objects in S3. As of today there are 52 billion
  11. ^ Vogels, Werner (2008-03-19). "Happy Birthday, Amazon S3!". All Things Distributed.
  12. ^ Amazon S3 SLA
  13. ^ 60 min/hour * 24 hours in a day * 30 days * 0.1% = 43.2
  14. ^ Amazon S3 Protecting Your Data
  15. ^ Starting Websphere in Cloud and saving the data in S3
  16. ^ How to use Amazon S3 for Web Hosting
  17. ^ AWS Developer Forums: Limit my own bandwidth?. Forums.aws.amazon.com. Retrieved on 2013-08-09.
  18. ^ AWS Developer Forums: What is the status on the bill capping. Forums.aws.amazon.com. Retrieved on 2013-08-09.
  19. ^ http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html Server Access Logging
  20. ^ Amazon Simple Storage Service. Docs.amazonwebservices.com. Retrieved on 2013-08-09.
  21. ^ Garnaat, Mitch, (19 Nov 2009). "Re: default key or 'default document' - is it possible to specify in S3?". Retrieved 21 Sep 2010.{{cite web}}: CS1 maint: extra punctuation (link) CS1 maint: multiple names: authors list (link)
  22. ^ "Amazon S3: Show Me the Money". SmugMug Blog. SmugMug. November 10, 2006.
  23. ^ "Comparison of S3QL and other S3 file systems". Retrieved 2012-06-29.
  24. ^ "Hadoop Filesystem Specification".
  25. ^ "S3mper: Consistency in the Cloud".
  26. ^ "AWS Case Study: reddit". aws.amazon.com. 2015. Retrieved March 18, 2015.
  27. ^ "Where are my files stored?". November 28, 2010.
  28. ^ "What is Tahoe-LAFS-on-S3?". August 21, 2012.
  29. ^ "Minecraft Beta 1.2_02". January 21, 2010.
  30. ^ "Swiftype Explains Their Cloud Stack". July 1, 2013.
  31. ^ Lelii, Sonia (23 September 2013). "Amazon S3 API for cloud storage leads pack, for now". TechTarget.com. Retrieved 31 May 2016.
  32. ^ Evans, Chris (12 January 2016). "Has S3 Become the De-Facto API Standard?". Architecting.it. Retrieved 31 May 2016.
  33. ^ Sadun, Erica (6 November 2012). "Arq cloud backup adds low-cost Amazon Glacier support". www.engadget.com. Retrieved 31 May 2016.
  34. ^ Moran, Joe (1 December 2015). "Data Backup Software Review: CloudBerry Lab Backup 4.5". www.smallbusinesscomputing.com. Retrieved 31 May 2016.
  35. ^ Sanders, James (4 August 2014). "Securely back up personal files with Duplicati: Q&A with the open source client's creators". www.TechRepublic.com. Retrieved 31 May 2016.
  36. ^ Lohnash, Mike (19 June 2015). "Odrive Review: One Folder for All Your Clouds". www.BackupReview.com. Retrieved 31 May 2016.
  37. ^ "Oracle Database Backup To Cloud: Amazon Simple Storage Service (S3)" (PDF). Oracle.com. Retrieved 31 May 2016.
  38. ^ "Cloud Storage Support". Commvault.com. Retrieved 31 May 2016.
  39. ^ "Veritas launches NetBackup 7.7 with emphasis on cloud backup". SearchDataBackup. Retrieved 2016-05-31.
  40. ^ "Asigra, Veeam remain top users' choice for backup applications". SearchDataBackup. Retrieved 2016-05-31.
  41. ^ "Startup Rubrik Aiming to Erase Backup, Recovery Software". www.eweek.com. Retrieved 2016-05-31.
  42. ^ Ibm; Emc; Netapp; Seagate; Hp; Hill, Seagate rolls storage kit for manufacturers down Dot; Spectralogic CTO talks up hybrid flash-tape cartridge. Welcome, tape robot overlords; backup, CTERA Networks offers up in-cloud server. "CTERA Networks offers up in-cloud server backup". Retrieved 2016-05-31.
  43. ^ Mellor, Chris (7 October 2015). "Like a wedding cake: Avere unveils three-tier AWS cloud NAS". TheRegister.com. Retrieved 8 June 2016.
  44. ^ Armstrong, Adam (28 January 2015). "EMC CloudArray 5.0 Launched". StorageReview.com. Retrieved 8 June 2016.
  45. ^ Mackie, Kurt (1 June 2015). "Microsoft StorSimple Extends Cloud Support to AWS, OpenStack". RedmondChannelPartner.com. Retrieved 8 June 2016.
  46. ^ Mellor, Chris (14 May 2015). "Azure gives AWS the blues again in Nasuni cloud storage poll". TheRegister.com. Retrieved 8 June 2016.
  47. ^ Ramel, David (28 May 2015). "NetApp Introduces AltaVault for Cloud Backup". AWSInsider.net. Retrieved 8 June 2016.
  48. ^ Watters, Audrey. "Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)". SAY Media, Inc. Retrieved 19 December 2012.
  49. ^ Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council (1990). Crossroads of Information Technology Standards. Washington, DC:: The National Academies Press, 1990. pp. 36–37.{{cite book}}: CS1 maint: extra punctuation (link) CS1 maint: multiple names: authors list (link)
  50. ^ "Connectria Cloud Storage - Amazon S3® Compatible Cloud Storage Service". Connectria. Connectria. Retrieved 19 December 2012.
  51. ^ "Connectria Hosting Launches Cloud Storage Solution". Connectria Hosting. Retrieved 19 December 2012.
  52. ^ Ross, Rose (February 22, 2011). "Connectria selects Scality to launch a public cloud storage service". RealWire. Retrieved 19 December 2012.
  53. ^ "Cloud Storage Providers". Twinstrata. Twinstrata. Retrieved 19 December 2012.
  54. ^ Riak CS. Basho (2013-01-18). Retrieved on 2013-08-09.
  55. ^ Ceph Object Gateway S3 API — Ceph Documentation. Ceph.com. Retrieved on 2013-08-09.
  56. ^ Amazon Simple Storage Service Documentation
  57. ^ Tools for Amazon S3: Backup and Storage Management Software

References

Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=Amazon_S3&oldid=724269794"