Petabytes.

Crazy.

Anyways, starting July 1, new pricing for bandwidth goes into effect:

New data transfer price for US-Standard, US-West and Europe regions (effective July 1, 2011)

• $0.000 – first 1 GB / month data transfer out

• $0.120 per GB – up to 10 TB / month data transfer out (10 TB total)
• $0.090 per GB – next 40 TB / month data transfer out (50TB total)
• $0.070 per GB – next 100 TB / month data transfer out (150 TB total)
• $0.050 per GB – next 350 TB / month data transfer out (500 TB total)
• Contact us – next 524 TB / month data transfer out (1PB total)
• Contact us – next 4 PB / month data transfer out (5PB total)
• Contact us – data transfer out / month over 5 PB

According to Amazon, if you were moving 10TB in and 10TB out a month, your bill just went down 50%. Not bad.

CloudFront prices are also going down. For more specifics, visit the AWS Data Transfer Pricing Update detail page.

One part of their servers on demand product had issues – specifically their Elastic Block Storage product in one of their availability zones. Many servers use it for persistent storage, something the EC2 product doesn’t offer by default. With these volumes being flaky, throwing errors or being office, many sites were in trouble.

The services that we use the most here at John Carroll, the Simple Storage Service (S3) and the Cloudfront content delivery network were not affected, thankfully, so I could enjoy the holiday weekend. I would have liked to play some online games on my PS3, but as you’ll see below, that too was off-line.

So what are some takeaways I see coming out of this outage?

First, don’t put all your eggs in one basket. SmugMug CEO Don MacAskill posted a very detailed blog post about the Amazon outage and how and why his company’s servers there weren’t affected. He says:

All of our services in AWS are spread across multiple Availability Zones (AZs). We’d use 4 if we could, but one of our AZs is capacity constrained, so we’re mostly spread across three. (I say “one of our” because your “us-east-1b” is likely different from my “us-east-1b” – every customer is assigned to different AZs and the names don’t match up). When one AZ has a hiccup, we simple use the other AZs. Often this is a graceful, but there can be hiccups – there are certainly tradeoffs.

Second, if you are going to leverage the cloud for services, and you should, you must have a backup plan or set of protocols for what to do if it hits the fan.

For example, if S3 did go down, our WordPress CMS would be affected, as we store user-uploaded assets in S3. To remedy that, we keep a local copy on our server, so our assets stay available to our site visitors. If S3 goes down, we can make a change to a plugin configuration and our assets will still be available. When S3 comes back online, we’d flip the switch and go back to serving things from the cloud.

Third, have a communication plan ready and keep users updated during the day.

The only spot I was finding out official information on the outage was on the AWS Service Health Dashboard, which is fine, that’s where it should be. In addition, many sites put up their own pages (Quora, Reddit come to mind) saying their were being affected by the outage.

If you have a blog, use it. Same goes for Twitter and Facebook. Amazon, even though the info was hidden, was good with updating exactly what was going on and where they were in the process of getting services back online. For example:

Apr 24, 5:05 AM PDT: As detailed in previous updates, the vast majority of affected EBS volumes have been restored by this point, and we are working through a more time-consuming recovery process for remaining volumes. We have made steady progress on this front over the past few hours. If your volume is among those recently recovered, it should be accessible and usable without additional action.

Good information that’s being updated often is important to help keep customers in the loop. Compare that to Sony, who’s Playstation network has been offline since last Wednesday. Their updates have been nebulous, at best. On April 21, they posted on their official blog:

While we are investigating the cause of the Network outage, we wanted to alert you that it may be a full day or two before we’re able to get the service completely back up and running.

The last update given by the company, on April 23, said this:

We sincerely regret that PlayStation Network and Qriocity services have been suspended, and we are working around the clock to bring them both back online. Our efforts to resolve this matter involve re-building our system to further strengthen our network infrastructure. Though this task is time-consuming, we decided it was worth the time necessary to provide the system with additional security.

We thank you for your patience to date and ask for a little more while we move towards completion of this project. We will continue to give you updates as they become available.

And then, silence. It’s now Monday morning in the US and the service is not online and the current status/ETA for being online hasn’t been updated since Saturday. IGN has more on Sony’s PR response to this outage.

That type of communication wouldn’t work on our campuses. Part of your planning must be a communications plan for who is responsible for keeping a certain audience up to date on the status of services.

My colleagues at Allegheny are doing it right this morning. They had a power outage over the weekend and took to their intranet to update the campus community, on a Sunday.

Am I going to stop using Amazon’s cloud services over this outage? No, definitely not. Is this going to make Amazon improve the service? Yes. Is this a sucky way to do it? Of course.

I’ll be updating this post with feedback from other higher ed web and marketing folks. Andrew Careaga has some interesting thoughts on the outage looking at it through a lens of education.

There are free services, such as uStream and Justin.tv, but the quality they offer can be erratic and, if you are using one of the free tiers of service, your content will have pre-roll and/or pop-up ads. This is annoying.

In a perfect world, we’d all have our own Flash media streaming setups we could push a button and start using. I don’t know about you, but I don’t have the time to manage all that stuff.

Enter Amazon, naturally.

They’ve announced today a new service and framework to get up and running doing live Flash media streaming. They’ve combined several of their services, including DNS, servers on demand and their content delivery network to offer an interesting on-demand Flash streaming rig.

The actual setup looks like this, but don’t be scared by all the pieces.

With their CloudFormation service, much of the work of setup only needs to be done once. Instances can be created from that template as needed.

Let’s look at costs for this type of setup. According to Amazon, they say this about costs:

In addition to the $5.00 monthly subscription fee for Flash Media Server on Amazon EC2, you pay for only for the AWS resources you consume.

Let’s examine those costs for a moment. After the $5 monthly charge, you’ll pay $0.44 USD for a server in Virginia that can support 100 simultaneous connections. Prices go up from there. Streaming to 1,000 users would run you $1.30 USD per hour. Prices are higher in Asia and Japan.

The big unknown here is bandwidth usage. I’m having a hard time trying to estimate the amount of bandwidth needed for an event, such as graduation. 20GB? 50GB? 100GB? If you have a number you’ve seen in the past, let me know so I can correct the numbers.

Let’s use 100 as our basis here – that bandwidth would cost you $15.00 USD.

So let’s say you are streaming two hours of Commencement. That would be:

• Flash: $5
• 1000 streams at $1.30 per hour: $2.50
• 100GB of Bandwidth: $15.00

Under $25 for a platform you have full control over? That’s not too shabby.

Let’s compare that to some other services out there:

uStream offers ad-free streams, and you can get 100 hours for $99 per month. 4,000 ad-free hours per month will run you $500 a month. LiveStream.com offers 3,000 ad-free hours and HD quality for $350 per month.

Looks like an interesting offering from Amazon. As we start to plan our graduation streaming, it will definitely be in the mix. If you’d like to read a tutorial from Amazon on live Flash streaming, you can check it out here.

I also host some small websites there. Technically, you could always host static HTML web pages at Amazon, but you were pretty limited in your options. The pages had to be static HTML (no PHP, ASP, etc) and could only be accessed with the direct full URL. For example:

http://web.highedwebtech.com/pages/index.html

If you tried to enter just:

http://web.highedwebtech.com/pages

The user would get an error message that isn’t very user friendly:

A few months ago, Amazon added a feature for hosting sites that would let users see a list of files in a directory, but if something went wrong, you’d still get that nasty error message.

Just this week, Amazon has released the ability for you to define both an index file for your S3 hosted site as well as an error document.

It looks like this:

That makes it much easier to fully host a site at S3. Why would you want to host your site at S3?

For one, you can leverage Amazon’s cheap bandwidth and built-in scalability. If you’re launching a project on your campus, or doing a large scale outreach project for prospective students or alumni, there can often be a large spike of users, especially if there’s a big marketing push. By having the site living at Amazon, you wouldn’t have to worry about Amazon being able to handle the load.

The downside of this type of service is that it just accepts static HTML files. If I upload a PHP document (index.php for example) it just downloads the file, it doesn’t display in the browser. That means if you’re using any type of CMS, you can’t host your site in S3, you’d have to build a server in EC2.

But, it sure would be nice if you could export static files, such as what WP-Super-Cache creates out of a system like WordPress and serve them from S3. That’d be super fast, for reals.

Sort of.

They’ve launched a new service called Amazon Simple Email Service, or SES for short. This service will allow you to send emails through their cloud infrastructure at a greatly reduced price. On the AWS blog, they say:

You’ll be able to send email without having to worry about the undifferentiated heavy lifting of infrastructure management, configuring your hosts for optimal sending, and the like. Amazon SES also provides you with access to a number of metrics that will provide you with the feedback needed to tune your email strategy to maximize deliverability.

The big difference is that you’ll send your emails through an API instead of a nice, friendly environment like say Constant Contact. I’m sure that very soon, someone will write a wrapper for the API that takes a lot of the heavy lifting out of the process. You do, however, get analytics data such as deliverability stats.

Pricing seems pretty reasonable as well. Email messages are charged at $0.10 per thousand messages. If you are using an EC2 server to send your messages, you get 2,000 messages a day free.

To send 1,000 emails at say Campaign Monitor would cost you $15. Yes, that’s a bit of comparing apples to oranges, but if there’s a very nice easy to use system built on top of SES, you could see some pretty decent savings.

This is interesting and worth keeping an eye on going forward.

Faster loading pages are always good. They make users happy, who don’t click off our sites as they wait for giant pages to load. They also make Google happy, who has begun to penalize slow loading sites.

Running YSlow locally is good, but the tests it performs are only run from your local machine. If you’re on the same network as your web server, as many of us are, your results will be skewed towards your very fast connection. Not every user at home has the network speeds we have.

So how do you get an idea of how your site performs around the world? Look to the cloud.

Yottaa was a finalist in this year’s Amazon Web Services start-up challenge, and basically, they offer a cloud-based YSlow using Amazon servers around the world. This is very helpful to find bottlenecks around the world or make sure you’re using a CDN that has nodes around the world, such as Akamai or CloudFront. From their site:

Every site in our database is monitored using a real browser in an effort to measure user experience while visiting your site. We are able to experience your site the exact same way that your users will including every image, javascript, flash file and every other dynamic content asset that makes your site awesome.

The reports Yottaa gives are very nice and and easy to understand. Here’s a global look at this blog’s speeds. Click for a larger version.

They also give you regular YSlow results:

The only nag I had is that while I use a CDN for this site, YSlow and Yottaa don’t recognize it because I use a CNAME. Basically, media.highedwebtech.com points to my file bucket at Amazon instead of using a long Amazon Web Services address. I’d very much like to tell Yottaa that media.highedwebtech.com is really a CDN and to stop penalizing my scores.

The really cool thing about Yottaa is that you can create a login and create what they call Benchmarks, and track several sites on one page and monitor their progress over time. What a great tool if you’ve got sites on your campus on different servers, platforms, etc. This could be a very valuable tool.

I created this one, which is monitoring 22 institutions I could think of off the top of my head. I know I’m missing a ton, DM me or leave a comment and I’ll add yours in. I think it’s an interesting way to see how you stack up, technology-wise, to other institutions. This time, it’s about technical speed and good web practices instead of content or a pretty design.

You can sign up at the site to do more regular tracking of results over time. It’s a nice web-based, easy-to-use tool. I will be using it quite a bit going forward.

Beginning November 1, new Amazon Web Services customers will receive an unprecedented amount of services for free to introduce you to their services and how you can implement these into your web workflow.

Here’s what you get.

• 750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*
• 750 hours of an Elastic Load Balancer plus 15 GB data processing*
• 10 GB of Amazon Elastic Block Storage, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests*
• 5 GB of Amazon S3 storage, 20,000 Get Requests, and 2,000 Put Requests*
• 30 GB per of internet data transfer (15 GB of data transfer “in” and 15 GB of data transfer “out” across all services except Amazon CloudFront)*
• 25 Amazon SimpleDB Machine Hours and 1 GB of Storage**
• 100,000 Requests of Amazon Simple Queue Service**
• 100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service**

Seriously – you get all this. I’m stunned. That’s basically a free server for a year from Amazon, storage, load balancing and more.

So how can you integrate these into your web workflow?

1. Backups and content delivery

With 5 GB of free storage, use it to backup your blog or website. There are automated plugins for many CMS and blog systems, especially WordPress.

If you run WordPress, use the TanTan S3 plugin to have media that you or your content creators upload go right to S3 and be served from there. Why? Bandwidth and storage space mostly.

2. Try out a new plugin, code framework, blogging tool, CMS, etc.

With the micro server, you can fire up whatever you want and try it out – especially if you’ve always wanted to run, say, Ruby on Rails on a CentOS server, this is your opportunity to try it out.

3. Get out of your comfort zone.

I’m a pretty heavy AWS user, and I’ve never used their SimpleDB or Simple Notification Services before. I’m going to use this free tier (on a new account, naturally) to put them through their paces and see if they are things might make my job easier on any given day. I’m especially interested in the notification service.

If you read this blog often or have seen me talk recently, you may have heard some of this before, but if you haven’t, I think its worth a few minutes to see how quickly and easy it is to use the cloud for testing, backups and more. My mom read it and thinks I’m pretty smart.

Here are a few key takeaways from the piece, courtesy of Educause:

• Cloud services can help colleges and universities meet sudden surges or long-term increases in demand for processing, bandwidth, and storage from the campus community.
• The cloud can also support low-risk, low-cost testing of large technology systems before the institution purchases them.
• Hosting a large fundraising video on the cloud solved a bandwidth problem for Allegheny College, while testing content management systems on cloud servers reduced risk for John Carroll University.
• Balancing the pros and cons of cloud services, including cost and project length, helps in determining whether to look to the cloud to solve IT problems.

The story also included a video I did showing just how easy it is to launch a server at Amazon. I’ll embed it here, but if you want to really see the detail of what I’m doing, go watch it at YouTube in a larger size.

For the majority of work I do in S3, I use Panic’s excellent FTP-and-then-some client, Transmit. It treats S3′s buckets and objects model more as regular FTP, where you can create folders and drag and drop folders. S3Fox is a Firefox plugin that makes managing your S3 folders a snap and it’s cross-platform. On the Windows side, I’m a fan of Cloudberry S3 Explorer, a free app. I reviewed Cloudberry Explorer last year, you can read it here.

While those clients are nice, you may find yourself in a situation when you need to access S3 and don’t have the tools I mentioned above. Such a sitation happened to me last week. I stopped by the Cleveland Clinic to talk with John Sharp‘s bioinformatics group. Talk about an intimidating group – these were some seriously smart programmers, doctors and database professionals. They were all very nice, to be sure, but, wow. After all, the Cleveland Clinic was recently voted one of the top 4 hospitals in the country.

As I was prepping to talk with that group about the cloud and some of the neat services out there, I wanted to post a few graphs and wanted to add them to my S3 account so they could grab them after the talk. Since I was on the public Cleveland Clinic wireless, I was pretty much limited to web-based services. I didn’t have S3Fox installed on my Macbook, so I logged into Amazon’s new S3 Management Console.

Much like you can manage your servers in Amazon’s console, they’ve added S3 as a service you can manage from the web, making uploading, deleting and sharing files very easy. Since it all happens in the browser, you can access the service from any web connection and quickly do the work you need to do. I posted the photos and an updated version of my Powerpoint presentation (based on the one I gave at EduComm) in S3 and the rest, as they say, was history.

Here are some screenshots from Amazon’s S3 Console.

New prices for everything but CloudFront look like this now:

While it doesn’t sound like a lot, if you are serving out a ton of files from S3, you should see some pretty decent savings. Let’s say you serve out 250GB in a month. Your bill just for outgoing bandwidth would drop from $42.50 to $37.50. Not bad, and the more you serve the more you would save.

You can learn more about the new pricing and more specific CloudFront details over the Amazon Web Services blog.