Petabytes.

Crazy.

Anyways, starting July 1, new pricing for bandwidth goes into effect:

New data transfer price for US-Standard, US-West and Europe regions (effective July 1, 2011)

• $0.000 – first 1 GB / month data transfer out

• $0.120 per GB – up to 10 TB / month data transfer out (10 TB total)
• $0.090 per GB – next 40 TB / month data transfer out (50TB total)
• $0.070 per GB – next 100 TB / month data transfer out (150 TB total)
• $0.050 per GB – next 350 TB / month data transfer out (500 TB total)
• Contact us – next 524 TB / month data transfer out (1PB total)
• Contact us – next 4 PB / month data transfer out (5PB total)
• Contact us – data transfer out / month over 5 PB

According to Amazon, if you were moving 10TB in and 10TB out a month, your bill just went down 50%. Not bad.

CloudFront prices are also going down. For more specifics, visit the AWS Data Transfer Pricing Update detail page.

One part of their servers on demand product had issues – specifically their Elastic Block Storage product in one of their availability zones. Many servers use it for persistent storage, something the EC2 product doesn’t offer by default. With these volumes being flaky, throwing errors or being office, many sites were in trouble.

The services that we use the most here at John Carroll, the Simple Storage Service (S3) and the Cloudfront content delivery network were not affected, thankfully, so I could enjoy the holiday weekend. I would have liked to play some online games on my PS3, but as you’ll see below, that too was off-line.

So what are some takeaways I see coming out of this outage?

First, don’t put all your eggs in one basket. SmugMug CEO Don MacAskill posted a very detailed blog post about the Amazon outage and how and why his company’s servers there weren’t affected. He says:

All of our services in AWS are spread across multiple Availability Zones (AZs). We’d use 4 if we could, but one of our AZs is capacity constrained, so we’re mostly spread across three. (I say “one of our” because your “us-east-1b” is likely different from my “us-east-1b” – every customer is assigned to different AZs and the names don’t match up). When one AZ has a hiccup, we simple use the other AZs. Often this is a graceful, but there can be hiccups – there are certainly tradeoffs.

Second, if you are going to leverage the cloud for services, and you should, you must have a backup plan or set of protocols for what to do if it hits the fan.

For example, if S3 did go down, our WordPress CMS would be affected, as we store user-uploaded assets in S3. To remedy that, we keep a local copy on our server, so our assets stay available to our site visitors. If S3 goes down, we can make a change to a plugin configuration and our assets will still be available. When S3 comes back online, we’d flip the switch and go back to serving things from the cloud.

Third, have a communication plan ready and keep users updated during the day.

The only spot I was finding out official information on the outage was on the AWS Service Health Dashboard, which is fine, that’s where it should be. In addition, many sites put up their own pages (Quora, Reddit come to mind) saying their were being affected by the outage.

If you have a blog, use it. Same goes for Twitter and Facebook. Amazon, even though the info was hidden, was good with updating exactly what was going on and where they were in the process of getting services back online. For example:

Apr 24, 5:05 AM PDT: As detailed in previous updates, the vast majority of affected EBS volumes have been restored by this point, and we are working through a more time-consuming recovery process for remaining volumes. We have made steady progress on this front over the past few hours. If your volume is among those recently recovered, it should be accessible and usable without additional action.

Good information that’s being updated often is important to help keep customers in the loop. Compare that to Sony, who’s Playstation network has been offline since last Wednesday. Their updates have been nebulous, at best. On April 21, they posted on their official blog:

While we are investigating the cause of the Network outage, we wanted to alert you that it may be a full day or two before we’re able to get the service completely back up and running.

The last update given by the company, on April 23, said this:

We sincerely regret that PlayStation Network and Qriocity services have been suspended, and we are working around the clock to bring them both back online. Our efforts to resolve this matter involve re-building our system to further strengthen our network infrastructure. Though this task is time-consuming, we decided it was worth the time necessary to provide the system with additional security.

We thank you for your patience to date and ask for a little more while we move towards completion of this project. We will continue to give you updates as they become available.

And then, silence. It’s now Monday morning in the US and the service is not online and the current status/ETA for being online hasn’t been updated since Saturday. IGN has more on Sony’s PR response to this outage.

That type of communication wouldn’t work on our campuses. Part of your planning must be a communications plan for who is responsible for keeping a certain audience up to date on the status of services.

My colleagues at Allegheny are doing it right this morning. They had a power outage over the weekend and took to their intranet to update the campus community, on a Sunday.

Am I going to stop using Amazon’s cloud services over this outage? No, definitely not. Is this going to make Amazon improve the service? Yes. Is this a sucky way to do it? Of course.

I’ll be updating this post with feedback from other higher ed web and marketing folks. Andrew Careaga has some interesting thoughts on the outage looking at it through a lens of education.

I also host some small websites there. Technically, you could always host static HTML web pages at Amazon, but you were pretty limited in your options. The pages had to be static HTML (no PHP, ASP, etc) and could only be accessed with the direct full URL. For example:

http://web.highedwebtech.com/pages/index.html

If you tried to enter just:

http://web.highedwebtech.com/pages

The user would get an error message that isn’t very user friendly:

A few months ago, Amazon added a feature for hosting sites that would let users see a list of files in a directory, but if something went wrong, you’d still get that nasty error message.

Just this week, Amazon has released the ability for you to define both an index file for your S3 hosted site as well as an error document.

It looks like this:

That makes it much easier to fully host a site at S3. Why would you want to host your site at S3?

For one, you can leverage Amazon’s cheap bandwidth and built-in scalability. If you’re launching a project on your campus, or doing a large scale outreach project for prospective students or alumni, there can often be a large spike of users, especially if there’s a big marketing push. By having the site living at Amazon, you wouldn’t have to worry about Amazon being able to handle the load.

The downside of this type of service is that it just accepts static HTML files. If I upload a PHP document (index.php for example) it just downloads the file, it doesn’t display in the browser. That means if you’re using any type of CMS, you can’t host your site in S3, you’d have to build a server in EC2.

But, it sure would be nice if you could export static files, such as what WP-Super-Cache creates out of a system like WordPress and serve them from S3. That’d be super fast, for reals.

It’s made the job on syncing files across my desktop machines, laptop, iPhone and iPad drop-dead simple. Thanks to sharing a few links with people and doing the recent Dropbox scavenger hunt, I’ve got 4.75GB worth of storage.

I love too that you can share files with other Dropbox users. This has proven to be very valuable, especially in the office (and our senseless 100MB email quota.)

With a new web developer starting to work for me today and a second person working with us from the UK, I immediately thought of Dropbox as a way for us all to collaborate and share files easily and quickly.

Armed with that knowledge, I set up an account for my office (separate from my personal account) and paid $10 for 50GB of storage. It’s probably a bit more than we needed, but better to be safe than sorry.

Once the account was created, I set up a “JCU WEB” folder on my work account, and shared it with my personal Dropbox account. Thinking I had access to that 50GB of space, I put up about 5GB of files, archives and other stuff the team would need.

It turns out that Dropbox counts all that storage against my personal account space, not the 50GB quota of the paid account. There’s not a way that my team can use that 50GB that we’re paying for.

Lame, dudes.

In reading through the Dropbox forums, many users share this frustration. Dropbox has answer, but for many small teams, may be overkill.

They’ve released a product called Dropbox Teams, which for $795 a year, gives 5 users a shared 350GB of file space. They say:

Storage quotas are shared by the team rather than bound to individual accounts. Now you and your team can share one large pool of storage instead of having to manage the storage limitations of individual accounts. Shared folders only take up your team’s storage quota rather than space in each individual account.

That’s exactly what we need, but we don’t need 350GB. We need just the 50GB, or even let us connect to our own Amazon S3 account (that’s where Dropbox keeps everything anyway.)

I think if Dropbox could solve this problem – they’d find a lot of users willing to pay $10-20 a month for a shared pool of storage. I definitely would.

For us, it’s back to the drawing board on how to share project files. Amazon S3? Basecamp?

Then, the unthinkable happened. While my iMac was fine – the external drive died and I lost everything. It sucked.

No more, I promised. It’s online backup for me. So I began my research.

The first company I used was Mozy, who for $5 a month per computer promised unlimited backup. They had a Mac client, and after an initial upload of about two weeks (seriously, I just let it run at night,) I was backed up. Any files I added would automatically get backed up.

Unfortunately, the software was buggy on the Mac when I was a customer, and it seemed they were throttling my uploads. I began my search for a new backup company.

After some research, I settled on BackBlaze (not an affiliate link). I’ve never looked back.

For $5 a month (£4 for my UK friends), you get unlimited backup of 1 computer – be it Mac or PC.

After about a week and a half for my initial backup, I’ve been good ever since. I don’t back up everything – I don’t need all the log files, plists and more that my Mac generates. I could, if I wanted to, just backup my mp3 files. I can also put in my own encryption key so that the data I upload to Backblaze is viewable only by me. That’s nice if you’re worried about security.

In fact, I’ve also installed it on my work Mac so it too is backed up. They’ve got a very nice native Mac app that is controlled from the Control Panel. That’s nice. If you’re a corporate user, they also offer business plans at $50 per computer per year. More here.

Restores are super simple as well – you can restore files at a secure website and download them as needed. If you need a larger amount of data, they’ll burn you a DVD or send you an external hard drive.

I dig BackBlaze because they’re geeky. Sure, they’re easy enough that my Mom can restore files on her Macbook, but I like that they share about their technology and how they accomplish things. They’ve even open-sourced their storage pod enclosure designs, which is pretty neat.

Basically, I’m a big fan, and I bring this up today because I’ve been seeing a lot of blog posts about Mozy and the fact that they’re the first of the all you can eat cloud backup providers (Carbonite, CrashPlan, etc.) to change their pricing structure – making users pay by how much they backup. Whenever your current plan updates (monthly, yearly, 2 years), you’ll be switched to the new pricing plan.

Under the new plan, you can backup 50GB on 1 computer for $6 a month. For $10 a month, you can backup up three computers up to 125GB. You can add $2 per month for an additional 20GB.

I was surprised when I checked Backblaze this morning and saw that I’m backing up 149GB. In the Mozy model, I’d be paying $16 a month, or about $120 more a year.

I get that there are some power users backing up a ton, but there also people like my parents paying $5 each to BackBlaze and backing up very little (a few GB of music and photos.) I would think the average home user doesn’t use 150GB like I do and it would even out.

Would I pay more for BackBlaze? Yes, but only if the tiers were clearly defined. For example – users backing up less than 5Gb would pay one price, and over 5GB would pay a certain amount. I’m sure Backblaze has the data showing a breakdown among users and what percentage of users are “power” users backing up over a certain amount.

If you’d like to try out Backblaze for a few months, here’s a code to get your starts. Enter coupon code ‘byemozy’ for a 10% discount

Faster loading pages are always good. They make users happy, who don’t click off our sites as they wait for giant pages to load. They also make Google happy, who has begun to penalize slow loading sites.

Running YSlow locally is good, but the tests it performs are only run from your local machine. If you’re on the same network as your web server, as many of us are, your results will be skewed towards your very fast connection. Not every user at home has the network speeds we have.

So how do you get an idea of how your site performs around the world? Look to the cloud.

Yottaa was a finalist in this year’s Amazon Web Services start-up challenge, and basically, they offer a cloud-based YSlow using Amazon servers around the world. This is very helpful to find bottlenecks around the world or make sure you’re using a CDN that has nodes around the world, such as Akamai or CloudFront. From their site:

Every site in our database is monitored using a real browser in an effort to measure user experience while visiting your site. We are able to experience your site the exact same way that your users will including every image, javascript, flash file and every other dynamic content asset that makes your site awesome.

The reports Yottaa gives are very nice and and easy to understand. Here’s a global look at this blog’s speeds. Click for a larger version.

They also give you regular YSlow results:

The only nag I had is that while I use a CDN for this site, YSlow and Yottaa don’t recognize it because I use a CNAME. Basically, media.highedwebtech.com points to my file bucket at Amazon instead of using a long Amazon Web Services address. I’d very much like to tell Yottaa that media.highedwebtech.com is really a CDN and to stop penalizing my scores.

The really cool thing about Yottaa is that you can create a login and create what they call Benchmarks, and track several sites on one page and monitor their progress over time. What a great tool if you’ve got sites on your campus on different servers, platforms, etc. This could be a very valuable tool.

I created this one, which is monitoring 22 institutions I could think of off the top of my head. I know I’m missing a ton, DM me or leave a comment and I’ll add yours in. I think it’s an interesting way to see how you stack up, technology-wise, to other institutions. This time, it’s about technical speed and good web practices instead of content or a pretty design.

You can sign up at the site to do more regular tracking of results over time. It’s a nice web-based, easy-to-use tool. I will be using it quite a bit going forward.

Judging by Twitter, lots of people were peeved today that the service is going to be shut down by Yahoo. No one’s exactly sure when, but it’s coming soon. Other people don’t care, which is fine, but if you are one of the ones who do have links and tags in there, you can get them out.

I found this code snippet from a commenter on TechCrunch, and it worked like a charm. Here’s what you do. From a command line (or your Terminal if you’re on a Mac,) run this command.

curl https://username:password@api.del.icio.us/v1/posts/all > bookmarks.xml

You should have a file called bookmarks.xml that looks like this:

That will get all your bookmarks, tags and other info into an XML file that you can use to import into whatever you want. Send it off to pinboard.in, or save it in a big Excel file. Once it’s in there, you can still search and find the info you need.

Want an HTML that’s a little easier to read? You may want to try Delicious’ own tool, available here. You’ll get a long list of links to your pages. Tags are included, but they’re in HTML code.

Beginning November 1, new Amazon Web Services customers will receive an unprecedented amount of services for free to introduce you to their services and how you can implement these into your web workflow.

Here’s what you get.

• 750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*
• 750 hours of an Elastic Load Balancer plus 15 GB data processing*
• 10 GB of Amazon Elastic Block Storage, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests*
• 5 GB of Amazon S3 storage, 20,000 Get Requests, and 2,000 Put Requests*
• 30 GB per of internet data transfer (15 GB of data transfer “in” and 15 GB of data transfer “out” across all services except Amazon CloudFront)*
• 25 Amazon SimpleDB Machine Hours and 1 GB of Storage**
• 100,000 Requests of Amazon Simple Queue Service**
• 100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service**

Seriously – you get all this. I’m stunned. That’s basically a free server for a year from Amazon, storage, load balancing and more.

So how can you integrate these into your web workflow?

1. Backups and content delivery

With 5 GB of free storage, use it to backup your blog or website. There are automated plugins for many CMS and blog systems, especially WordPress.

If you run WordPress, use the TanTan S3 plugin to have media that you or your content creators upload go right to S3 and be served from there. Why? Bandwidth and storage space mostly.

2. Try out a new plugin, code framework, blogging tool, CMS, etc.

With the micro server, you can fire up whatever you want and try it out – especially if you’ve always wanted to run, say, Ruby on Rails on a CentOS server, this is your opportunity to try it out.

3. Get out of your comfort zone.

I’m a pretty heavy AWS user, and I’ve never used their SimpleDB or Simple Notification Services before. I’m going to use this free tier (on a new account, naturally) to put them through their paces and see if they are things might make my job easier on any given day. I’m especially interested in the notification service.

These very small servers are good for things that don’t need a ton of throughput, such as load balancers, CRON-based tasks and proxies. Amazon gives some tech specs for these new machines:

Micro instances provide 613 MB of memory and support 32-bit and 64-bit platforms on both Linux and Windows. Micro instance pricing for On-Demand instances starts at $0.02 per hour for Linux and $0.03 per hour for Windows.

If you ran a micro server all the time, it’d run you roughly $15 USD, which isn’t bad for a server that you can set up to do quick tasks and (auto)scale as needed using Amazon’s CloudWatch service.

One of the things to know about these servers is that there’s no local storage offered as part of them, which is probably a big reason the cost per hour is so low. Instead, you set up an Elastic Block Store, which is basically a disk image you setup that lives in your S3 storage account. The nice thing about this is that the block doesn’t go away when you turn off the instance. With Amazon Elastic Block Store, you only pay for what you use. Volume storage is charged by the amount you allocate until you release it, and is priced at a rate of $0.10 per allocated GB per month Amazon EBS also charges $0.10 per 1 million I/O requests you make to your volume. So be aware there may be some additional costs when using EBS with the micro server.

All in all, a pretty interesting offering by Amazon. The commodity price of these servers is falling fast – $0.02 an hour at Amazon, $0.015 an hour at Rackspace. Pretty amazing.

If you read this blog often or have seen me talk recently, you may have heard some of this before, but if you haven’t, I think its worth a few minutes to see how quickly and easy it is to use the cloud for testing, backups and more. My mom read it and thinks I’m pretty smart.

Here are a few key takeaways from the piece, courtesy of Educause:

• Cloud services can help colleges and universities meet sudden surges or long-term increases in demand for processing, bandwidth, and storage from the campus community.
• The cloud can also support low-risk, low-cost testing of large technology systems before the institution purchases them.
• Hosting a large fundraising video on the cloud solved a bandwidth problem for Allegheny College, while testing content management systems on cloud servers reduced risk for John Carroll University.
• Balancing the pros and cons of cloud services, including cost and project length, helps in determining whether to look to the cloud to solve IT problems.

The story also included a video I did showing just how easy it is to launch a server at Amazon. I’ll embed it here, but if you want to really see the detail of what I’m doing, go watch it at YouTube in a larger size.