Tag Archives: S3

Website with GZip compression

Posted on by

last update : 20 August 2012
Compression is a simple, effective way to save bandwidth and speed up a website. The browser sends a header telling the server it accepts compressed content (Accept-Encoding: gzip ; Accept-Encoding: deflate). The server sends a response if the content is actually compressed: Content-Encoding: gzip. If the server doesn’t want to send back compressed content, the browser has to make do with the heavy regular version.

A webserver can be configured (in the settings or in the .htaccess file) to return zipped content if the browser can handle it, saving bandwidth for everyone. If you use the Amazon AWS S3 service as a webserver, you can upload compressed files in the AWS management console and add a metatag

Content-Encoding: gzip

in the HTTP header of the files.

I use the WinGZip utility (version 1) created by Irnis Haliullin to compress the content uploaded to my Amazon AWS S3 buckets. An online tool GIDZip Test to check compressed websites is available at gidnetwork.com.

Some useful documentation about Gzip compression is listed herafter :

RTMP streaming for Amazon CloudFront

Posted on by

RTMP (Real Time Messaging Protocol) is a web transmission standard which allows a backend server to transfer video, audio and data to the Flash Player. It’s particularly useful for video streaming, since the Flash Player can communicate back to the server about how much video it has consumed, and how much more video the server needs to send. This is a distinct advantage over so-called progressive download video, where a web server simply begins sending bytes out to the video player over standard HTTP until there are no more bytes to send.

In addition to saving bandwidth and allowing better seeking control, RTMP is harder to steal content, allows dynamic bitrate switching and is able to stream live content.

Amazon added RTMP to the CloudFront servers in December 2009 and thus became a real CDN (Content Delivery Network), offering content providers a simple and reliable way to host, format and deliver their video content at high speeds.

CloudFront Streaming is very easy to set up with the JW Player 5.3.

Today a created the first video-distributions on Amazon CloudFront by using the AWS-Console. This process is very straightforward.

The following parameters are used to embed a streaming-videofile in a webpage using a flash-player or the HTML5 <video> tag.

  • videofile = path/filename.ext
  • streamer = rtmp://xxxxxxxxxxxxx.cloudfront.net/cfx/st/
  • provider = rtmp

The suffixes /cfx/st/ at the end of the URL are required to access the streaming-server at the CloudFront server. Videofile extensions can be mp4, flv and webm.

Amazon S3 php source code

Posted on by

The basic php file provided by Amazon AWS in 2006 and updated in 2007 to manage S3 accounts requires the PEAR modules Crypt_HMAC and HTTP_Request.

Today there are several php open source toolkits available which are well documented, easy to set up and offering extended features :

  • undesigned S3 php class :  a standalone Amazon S3 REST implementation for PHP 5.2.x (using CURL), that supports large file uploads and doesn’t require PEAR. The latest version is 0.4.0 published on july 20, 2009. The developer is Donovan Schonknecht, a web/mobile application developer based in Cape Town, South Africa. The sourcecode is available at the Google Code website.
  • Tarzan : a fast, powerful PHP toolkit for building web applications with Amazon Web Services. The latest version is 2.0.5 released on august 14, 2009. The developer is Ryan Parman. The sourcecode is available at the Google Code website. User Metadata is managed if the HTTP header has the prefix x-amz-meta- (example: header : x-amz-meta-city value: Luxembourg).
  • CloudFusion : a rebranded version of Tarzan to better reflect the beyond-Amazon evolution of the product. Tarzan 2.5 will be CloudFusion 2.5 and will come with a refreshed website, documentation, examples, and other goodies. The sourcecode is available at the Google Code website.
  • Amazon GS3 : a stream wrapper to get and send files to Amazon S3, developed by Cesar D. Rodas from Paraguay.

A list of some Amazon S3 php code examples is shown hereafter :

Amazon S3 software clients

Posted on by

Last update : December 10, 2012

To organize/manage/store files on Amazon S3, you need a special software tool.

The following list shows a few selected free 3S clients :

  • S3Fox Organizer : this plugin for the Firefox browser is easy to install. Developed by Suchi Software Solutions, the most recent version is 0.6.
  • S3Browser : this Windows only free S3 tool supports multiple Amazon S3 accounts, S3 metadata editing and bucket sharing on top of other features. The latest version is 3.6.7 and has been released on November 20, 2012. S3 Browser is free for personal use only. Users who run the application in commercial, business, government, or military institutions, or for any other profit activity must purchase a Pro license. The Pro license provides some additional features, e.g. a working user-metadata edition.
  • Cloudberry S3 Explorer : this tool makes managing files in Amazon S3 storage easy. By providing a user interface to Amazon S3 accounts, files, and buckets, CloudBerry lets you manage your files on cloud just as you would on your own local computer. The current version is 3.7 released November 14, 2012. A CloudBerry Explorer PRO version is also available which offers some advanced features over the Freeware version. It makes managing files Amazon S3 easier, more secure and cost efficient.
  • Cyberduck : this open source client is an FTP, SFTP, WebDAV, Cloud Files and Amazon S3 browser for the Mac. It features an easy to use interface with quickly accessible bookmarks. The outline view of the browser allows to browse large folder structures efficiently and you can quickly preview files with Quick Look. (version 4.21 relaesd December 12, 2011)
  • S3fm : this client is the first 100% Ajax file manager application for Amazon S3. It runs directly in your browser as a standalone Javascript application and loads from and communicates directly with Amazon S3 without having to share your access credentials with a 3rd party Web site.
  • Amazon S3 for WordPress : this WordPress plugin (version 0.4) developed by Joe Tan allows you to use Amazon’s Simple Storage Service to host your media for your WordPress powered blog. The plugin was not updated during the last two years.
  • WordPress plugin for Amazon S3 : Donovan Schonknecht created a very simple WordPress plugin for Amazon S3 which uses his Amazon S3 PHP class.

The latest versions of S3Browser and Cloudberry support the new Amazon AWS archive solution Glacier.

There are also some commercial S3 tools available, e.g. JungleDisk, Gladinet Cloud Desktop,  Bucket Explorer, SMEStorage and CrossFTP.

Manage your Amazon S3 Account with Ease

Posted on by

Amazon S3 is a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. Several software clients are available to manage an S3 account with ease, some are listed below:

  • S3 Browser : freeware (version 1.9.7 ; 1.9.8 beta) and pro licence available from NetSDK Software
  • Cyberduck : FTP, SFTP, WebDAV, Cloud Files & Amazon S3 Browser for Mac OS X
  • BucketExplorer : different licenses starting at 49.99$ available for Windows, MAC and Linux
  • FireFox S3 : free plugin for FireFox browser
  • CloudBerry : freeware Windows client
  • CloudBuddy : CloudBuddy Personal is a free tool brought to you by CSS Lab

Limit my bandwith on Amazon S3

Posted on by

In 2006, a developer raised the question in the Amazon Discussion Forum whether there is a risk that the bandwidth cost grow beyond a level he was not willing to pay for. AWS answered that such a feature is in the work. The plan was to enable users to cap how much they are charged each month.

Three and a half year later, this is still a plan. The latest message from AWS was : later this year (2009) or early next year (2010).

Amazon CloudFront

Posted on by

update : October 18th, 2011
Amazon CloudFront is a web service for content delivery. It integrates with other Amazon Web Services (mainly Amazon S3) to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments.

Amazon CloudFront delivers the content using a global network of edge locations. Requests for  objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3) which durably stores the original, definitive versions of the files.

In Amazon CloudFront, objects are organized into distributions. A distribution specifies the location of the original version of the objects. A distribution has a unique CloudFront.net domain name  that  can be used to reference an objects through the network of edge locations. It’s also possible to map an own domain name to a distribution.

Amazon CloudFront is

  • fast
  • simple
  • cost-effective
  • elastic
  • reliable
  • global
  • designed for use with other Amazon Web Services

The price depends on the edge location and the volume transferred. The mean price per GB for low volumes is about 0,2$, for high volumes about 0,1$. A simple monthly AWS bill calculator is provided by Amazon. Normal fees will apply for Amazon S3 usage, including “origin fetches” – data transferred from Amazon S3 to edge locations.

The edge locations in Europe are:

  • Amsterdam
  • Dublin
  • Frankfurt
  • London

Amazon CloudFront is designed for delivery of objects that are frequently accessed – “popular” objects. Objects that aren’t accessed frequently are less likely to remain in CloudFront’s edge locations’ caches. Thus, for less popular objects, delivery out of Amazon S3 (rather than from CloudFront) is the better choice. Amazon S3 will provide strong distribution performance for these objects, and serving them directly from Amazon S3 saves the cost of continually copying less popular objects from Amazon S3 to the edge locations in CloudFront.

I activated my Cloudfront account on November 1st, 2010.

A recent tutorial how to install Amazon CloudFront has been posted by Michael Tieso on the website “Art of Travel Blogging”.

File storage and video hosting on Amazon S3

Posted on by

Amazon S3 (Simple Storage Service) provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. Each object is stored in a bucket and retrieved via a unique, developer-assigned key. It’s possible to write, read, and delete objects containing from 1 byte to 5 gigabytes of data each. The number of objects you can store is unlimited.

The pricing plan for Amazon S3 is based on “Pay only for what you use”. There is no minimum fee. The prices vary across geographic regions and are based on the location of the bucket. For Europe the pricing effective november 2008 is as follows :

Storage :

  • $0.180 per GB – first 50 TB / month of storage used
  • $0.170 per GB – next 50 TB / month of storage used
  • $0.160 per GB – next 400 TB / month of storage used
  • $0.150 per GB – storage used / month over 500 TB

Data Transfer :

  • $0.100 per GB – all data transfer in
  • $0.170 per GB – first 10 TB / month data transfer out
  • $0.130 per GB – next 40 TB / month data transfer out
  • $0.110 per GB – next 100 TB / month data transfer out
  • $0.100 per GB – data transfer out / month over 150 TB

Requests :

  • $0.012 per 1,000 PUT, POST, or LIST requests
  • $0.012 per 10,000 GET and all other requests

The Amazon S3 service is part of the Amazon Web Services, an infrastructure web services platform in the cloud. The direct link to the S3 login page is aws.amazon.com/s3/. I created my account on Amazon S3 on october 13th, 2008.

Some test files to verify the correct configuration are accessible at the following links :

The sender of a request to an AWS web service is authorised by Access Identifiers. Access identifiers are also used to authenticate requests to AWS. For services that require authenticated requests, where you need to prove that you are authorized to make the request, you must sign the request by including a “signature” in the request. A request signature is calculated using the pair of public / private Access Identifiers. AWS supports two types of Request Identifiers: AWS Access Key Identifiers or X.509 Certificates.

The Access Key ID is used as the value of the AWSAccessKeyId parameter in requests you send to Amazon Web Services and identifies you as the party responsible for the request. Since the Access Key ID is not encrypted in requests to AWS, it could be discovered and used by anyone. Services that are not free require you to provide additional information, a request signature, to verify that a request containing your unique Access Key ID could only have come from you. You use your Secret Access Key to calculate a signature to include in requests to web services that require authenticated requests.

Uploading and managing files on the Amazon S3 distributed system is not straight forward. Amazon offers documentation, release notes, sample code & libraries, developer tools, articles & tutorials and a community forum as resources to developers.

A comfortable user interface to upload and manage files on the 3S servers called “Bucket Explorer” (version 2008.06.30.01) is offered by Chambal.com.Inc for Windows, Linux and Mac (website www.bucketexplorer.com). The price for one licence is USD 49.95, a 30 day free trial version is available. A free plugin for the Firefox browser called “S3Fox Organizer” (version 0.4.1) has been developped by Rjonna of Suchi Software Solutions from India.

Another Firefox plugin (Gspace) developped by the same author and distributed by Fon Labs allows you to use your Gmail Space (2700 MB) as a remote machine. You can transfer files between your hard drive and gmail. Your gmail account looks like a FTP host and you can upload and download your files. After you install, you get an option called “GSpace” in your “tools” menu clicking on which opens the window for transfer. You have to be aware that this extension is not endorsed by Google and that this usage is not compliant with the Google Terms of Service.