RTMP streaming for Amazon CloudFront

RTMP (Real Time Messaging Protocol) is a web transmission standard which allows a backend server to transfer video, audio and data to the Flash Player. It’s particularly useful for video streaming, since the Flash Player can communicate back to the server about how much video it has consumed, and how much more video the server needs to send. This is a distinct advantage over so-called progressive download video, where a web server simply begins sending bytes out to the video player over standard HTTP until there are no more bytes to send.

In addition to saving bandwidth and allowing better seeking control, RTMP is harder to steal content, allows dynamic bitrate switching and is able to stream live content.

Amazon added RTMP to the CloudFront servers in December 2009 and thus became a real CDN (Content Delivery Network), offering content providers a simple and reliable way to host, format and deliver their video content at high speeds.

CloudFront Streaming is very easy to set up with the JW Player 5.3.

Today a created the first video-distributions on Amazon CloudFront by using the AWS-Console. This process is very straightforward.

The following parameters are used to embed a streaming-videofile in a webpage using a flash-player or the HTML5 <video> tag.

  • videofile = path/filename.ext
  • streamer = rtmp://xxxxxxxxxxxxx.cloudfront.net/cfx/st/
  • provider = rtmp

The suffixes /cfx/st/ at the end of the URL are required to access the streaming-server at the CloudFront server. Videofile extensions can be mp4, flv and webm.

Progressive video download, pseudo streaming and realtime streaming

Last update : January 30, 2013
In the past, audio and video on the Web was primarily a download-and-play technology. You had to first download an entire media file before it could play. Today, streaming technologies allow watching audio and video files almost immediately, while the data is being sent, without having to wait for the whole file to download.

There are three methods of delivering streaming audio and video content over the Web.

The first method uses a standard HTTP server to deliver the audio and video data to a media player. Unlike the download-and-play client, a special streaming client embedded in the webpage starts playing the audio or video while it is downloading, after only a few seconds wait for buffering, the process of collecting the first part of a media file before playing. This streaming method is called progressive media download.

The second method is called pseudostreaming. Pseudostreaming is a protocol that can be installed on regular HTTP servers. It uses a server side script for Flash-to-server communication. The player sends a HTTP request to the server with a start time parameter in the request URL’s query string and the server script responds with the video stream so that its start position corresponds to the requested parameter. This start time parameter is usually named simply start. The video viewer skips the nondownloaded parts of the videos.

Both FLV and MP4 video can be played back with  pseudostreaming. The following scripts or tools are available :

  • The H264 streaming module for Apache, Lighttpd, IIS and NginX.
  • The mod_flv_streaming module for Lighttpd.
  • PHP/ASP scripts such as XmooV PHP.
  • Content delivery networks such as Bitgravity, Edgecast or Limelight.

There is one major advantage to streaming with a Web server rather than with a streaming media server—utilizing existing infrastructure.

The third method uses a separate streaming media server specialized to the audio/video streaming task. A streaming server offers the following advantages :

  • More efficient use of the network bandwidth
  • Better audio and video quality to the user
  • Advanced features like detailed reporting and multi-stream multimedia content
  • Supports large number of users
  • Multiple delivery options
  • Content copyright protection

The following protocols are commonly used by streaming servers :

  • UDP – this protocol provides the most efficient network throughput. The only downside to UDP is that many network administrators close their firewalls to UDP traffic, limiting the potential audience of UDP-based streams
  • TCP – this protocol provides an adequate, though not necessarily efficient, protocol for delivering streaming media content to flow through the firewalls
  • HTTP + TCP – this combination has the benefit of working with all firewalls that let Web traffic through (port 80) and provides much more control (fast forward, rewind, etc) than a standard Web server, but also adds some overhead to the raw TCP stream that decreases scalability.
  • Multicast – this protocol enables hundreds or thousands of users to play a single stream, but will only work on networks with Multicast-enabled routers. Multicast is becoming prevalent on corporate networks, but is still very rare on the Internet

Useful informations and tutorials about streaming are available at the streamingmedia.com website.

In 2009, Amazon CloudFront, the easy-to-use content delivery service, introduced the ability to stream audio and video files. Streaming with Amazon CloudFront is exceptionally easy: with only a few clicks on the AWS Management Console or a simple API call, you’ll be able to stream your content using a world-wide network of edge locations running Adobe’s Flash® Media Server. Like all AWS services, Amazon CloudFront streaming requires no up-front commitments or long-term contracts. There are no additional charges for streaming with Amazon CloudFront; you simply pay normal rates for the data that you transfer using the service.

Amazon S3 php source code

The basic php file provided by Amazon AWS in 2006 and updated in 2007 to manage S3 accounts requires the PEAR modules Crypt_HMAC and HTTP_Request.

Today there are several php open source toolkits available which are well documented, easy to set up and offering extended features :

  • undesigned S3 php class :  a standalone Amazon S3 REST implementation for PHP 5.2.x (using CURL), that supports large file uploads and doesn’t require PEAR. The latest version is 0.4.0 published on july 20, 2009. The developer is Donovan Schonknecht, a web/mobile application developer based in Cape Town, South Africa. The sourcecode is available at the Google Code website.
  • Tarzan : a fast, powerful PHP toolkit for building web applications with Amazon Web Services. The latest version is 2.0.5 released on august 14, 2009. The developer is Ryan Parman. The sourcecode is available at the Google Code website. User Metadata is managed if the HTTP header has the prefix x-amz-meta- (example: header : x-amz-meta-city value: Luxembourg).
  • CloudFusion : a rebranded version of Tarzan to better reflect the beyond-Amazon evolution of the product. Tarzan 2.5 will be CloudFusion 2.5 and will come with a refreshed website, documentation, examples, and other goodies. The sourcecode is available at the Google Code website.
  • Amazon GS3 : a stream wrapper to get and send files to Amazon S3, developed by Cesar D. Rodas from Paraguay.

A list of some Amazon S3 php code examples is shown hereafter :

Amazon S3 software clients

Last update : December 10, 2012

To organize/manage/store files on Amazon S3, you need a special software tool.

The following list shows a few selected free 3S clients :

  • S3Fox Organizer : this plugin for the Firefox browser is easy to install. Developed by Suchi Software Solutions, the most recent version is 0.6.
  • S3Browser : this Windows only free S3 tool supports multiple Amazon S3 accounts, S3 metadata editing and bucket sharing on top of other features. The latest version is 3.6.7 and has been released on November 20, 2012. S3 Browser is free for personal use only. Users who run the application in commercial, business, government, or military institutions, or for any other profit activity must purchase a Pro license. The Pro license provides some additional features, e.g. a working user-metadata edition.
  • Cloudberry S3 Explorer : this tool makes managing files in Amazon S3 storage easy. By providing a user interface to Amazon S3 accounts, files, and buckets, CloudBerry lets you manage your files on cloud just as you would on your own local computer. The current version is 3.7 released November 14, 2012. A CloudBerry Explorer PRO version is also available which offers some advanced features over the Freeware version. It makes managing files Amazon S3 easier, more secure and cost efficient.
  • Cyberduck : this open source client is an FTP, SFTP, WebDAV, Cloud Files and Amazon S3 browser for the Mac. It features an easy to use interface with quickly accessible bookmarks. The outline view of the browser allows to browse large folder structures efficiently and you can quickly preview files with Quick Look. (version 4.21 relaesd December 12, 2011)
  • S3fm : this client is the first 100% Ajax file manager application for Amazon S3. It runs directly in your browser as a standalone Javascript application and loads from and communicates directly with Amazon S3 without having to share your access credentials with a 3rd party Web site.
  • Amazon S3 for WordPress : this WordPress plugin (version 0.4) developed by Joe Tan allows you to use Amazon’s Simple Storage Service to host your media for your WordPress powered blog. The plugin was not updated during the last two years.
  • WordPress plugin for Amazon S3 : Donovan Schonknecht created a very simple WordPress plugin for Amazon S3 which uses his Amazon S3 PHP class.

The latest versions of S3Browser and Cloudberry support the new Amazon AWS archive solution Glacier.

There are also some commercial S3 tools available, e.g. JungleDisk, Gladinet Cloud DesktopBucket Explorer, SMEStorage and CrossFTP.

Manage your Amazon S3 Account with Ease

Amazon S3 is a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. Several software clients are available to manage an S3 account with ease, some are listed below:

  • S3 Browser : freeware (version 1.9.7 ; 1.9.8 beta) and pro licence available from NetSDK Software
  • Cyberduck : FTP, SFTP, WebDAV, Cloud Files & Amazon S3 Browser for Mac OS X
  • BucketExplorer : different licenses starting at 49.99$ available for Windows, MAC and Linux
  • FireFox S3 : free plugin for FireFox browser
  • CloudBerry : freeware Windows client
  • CloudBuddy : CloudBuddy Personal is a free tool brought to you by CSS Lab

Limit my bandwith on Amazon S3

In 2006, a developer raised the question in the Amazon Discussion Forum whether there is a risk that the bandwidth cost grow beyond a level he was not willing to pay for. AWS answered that such a feature is in the work. The plan was to enable users to cap how much they are charged each month.

Three and a half year later, this is still a plan. The latest message from AWS was : later this year (2009) or early next year (2010).

Simple DB : Amazon database

Today I installed a simple database on the Amazon Webservices. Amazon SimpleDB is a web service providing the core database functions of data indexing and querying in the cloud. This service is available in Europe since a few weeks. This allows to achieve lower latency, operate closer to other resources like Amazon EC2, Amazon S3, and Amazon SQS in the EU Region, and help meet EU data storage requirements when applicable.

Simple DB is simple to use, low touch, scalable, highly available, fast, flexible, inexpensive and designed for use with other Amazon Web Services.

The prices are:

  • First 25 Amazon SimpleDB Machine Hours consumed per month are free
  • $0.154 per Amazon SimpleDB Machine Hour consumed thereafter
  • First 1 GB of data transferred in per month is free
  • $0.100 per GB – all data transfer in thereafter
  • First 1 GB of data transferred out per month is free; thereafter:
  • $0.170 per GB – first 10 TB / month data transfer out
  • $0.130 per GB – next 40 TB / month data transfer out
  • $0.110 per GB – next 100 TB / month data transfer out
  • $0.100 per GB – data transfer out / month over 150 TB

When using Amazon SimpleDB, you organize your structured data in domains within which you can put data, get data, or run queries. Domains consist of items which are described by attributename-value pairs. The spreadsheet model shown in the following image explains the structure:

Amazon Simple DB Data Model

Amazon Simple DB Data Model

Amazon CloudFront

update : October 18th, 2011
Amazon CloudFront is a web service for content delivery. It integrates with other Amazon Web Services (mainly Amazon S3) to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments.

Amazon CloudFront delivers the content using a global network of edge locations. Requests for  objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3) which durably stores the original, definitive versions of the files.

In Amazon CloudFront, objects are organized into distributions. A distribution specifies the location of the original version of the objects. A distribution has a unique CloudFront.net domain name  that  can be used to reference an objects through the network of edge locations. It’s also possible to map an own domain name to a distribution.

Amazon CloudFront is

  • fast
  • simple
  • cost-effective
  • elastic
  • reliable
  • global
  • designed for use with other Amazon Web Services

The price depends on the edge location and the volume transferred. The mean price per GB for low volumes is about 0,2$, for high volumes about 0,1$. A simple monthly AWS bill calculator is provided by Amazon. Normal fees will apply for Amazon S3 usage, including “origin fetches” – data transferred from Amazon S3 to edge locations.

The edge locations in Europe are:

  • Amsterdam
  • Dublin
  • Frankfurt
  • London

Amazon CloudFront is designed for delivery of objects that are frequently accessed – “popular” objects. Objects that aren’t accessed frequently are less likely to remain in CloudFront’s edge locations’ caches. Thus, for less popular objects, delivery out of Amazon S3 (rather than from CloudFront) is the better choice. Amazon S3 will provide strong distribution performance for these objects, and serving them directly from Amazon S3 saves the cost of continually copying less popular objects from Amazon S3 to the edge locations in CloudFront.

I activated my Cloudfront account on November 1st, 2010.

A recent tutorial how to install Amazon CloudFront has been posted by Michael Tieso on the website “Art of Travel Blogging”.

Amazon® Product Advertising API with signed requests

By August 15, 2009, all calls to the Amazon® Product Advertising API must be signed to authenticate the request. Ulrich Mierendorff developped a simple function in PHP that lets you make authenticated requests with only a few lines of code. He is also the author of “Antialiased filled Arcs/Ellipses” for PHP and of the rounded css boxes.

Other PHP example for Amazon signed requests have been posted by Mark on his blog “Every Good Path” and by Horacix on his blog “Inside Things“.

Amazon’s guidelines for signing requests are available on the AWS website.

The following error messages are sent:

  • the timestamp is old : Request has expired

EC2 : Amazon Elastic Compute Cloud

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.

Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing to quickly scale capacity, both up and down, as the computing requirements change. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios.

Amazon EC2 presents a true virtual computing environment, allowing you to use web service interfaces to launch instances with a variety of operating systems, load them with your custom application environment, manage your network’s access permissions, and run your image using as many or few systems as you desire.

To use Amazon EC2, an Amazon Machine Image (AMI) containing the applications, libraries, data and associated configuration settings is created or a pre-configured and templated image is used to get up and running immediately. The AMI is loaded into Amazon S3.

Amazon EC2 is elastic (scalable), flexible, completely controlled, designed for use with other Amazon Web Services, reliable, featured for Building Failure Resilient Applications (Amazon Elastic Block Store, Multiple Locations, Elastic IP Addresses), secure, inexpensive.

The default instance is a 32-bit platform with 1.7 GB of memory, 1 compute unit and 160 GB of instance storage (small instance). Large, extra-large and high-CPU instances are available for compute-intensive applications.Price:

The price for using a small instance is $0.10 per instance hour for Linux and $0.125 per instance hour for Windows. Amazon EC2 uses a variety of measures to provide each instance with a consistent and predictable amount of CPU capacity. Understanding the EC2 price model is not easy and several questions in the AWS forum refer to this issue. Here are some answers :

If you start an instance once every 24 hours and run it for less than an hour, and do this every day for a 30-day month, you will be charged for 30 instance hours.  As long as your instance is running, it is billable at the rate for that instance. A website would not be publicly available if the instance is not running. To run a website available 24/7 in a month, it will cost about $90 for windows instance-hours + storage charge + data in/out charge + others.

One customer confirmed the price calculation and stated that he compared AWS pricing with that of other companies. Although there were several companies that offered hosting for cheaper, in the end he decided to start using EC2 for the following reasons :

Although those sites were cheaper, none of them gave him as much flexibility as he wanted. He had to use their preset hosting configurations, which wasn’t very condusive towards hiswebsite-model. Similar cloud offerings, such as the ones offered by RackSpace, actually ended up being more expensive in the end. But most importantly, he could take advantage of Amazon’s other web services as well.  The one he was most excited about is Amazon Cloudfront to get a full-fledged content delivery service in one easy-to-use package.  Thats one offering that those “$9.99 a month!” companies can’t offer, and it’s something he needs.

There are other providers offering elastic computing power :