One part of their servers on demand product had issues – specifically their Elastic Block Storage product in one of their availability zones. Many servers use it for persistent storage, something the EC2 product doesn’t offer by default. With these volumes being flaky, throwing errors or being office, many sites were in trouble.

The services that we use the most here at John Carroll, the Simple Storage Service (S3) and the Cloudfront content delivery network were not affected, thankfully, so I could enjoy the holiday weekend. I would have liked to play some online games on my PS3, but as you’ll see below, that too was off-line.

So what are some takeaways I see coming out of this outage?

First, don’t put all your eggs in one basket. SmugMug CEO Don MacAskill posted a very detailed blog post about the Amazon outage and how and why his company’s servers there weren’t affected. He says:

All of our services in AWS are spread across multiple Availability Zones (AZs). We’d use 4 if we could, but one of our AZs is capacity constrained, so we’re mostly spread across three. (I say “one of our” because your “us-east-1b” is likely different from my “us-east-1b” – every customer is assigned to different AZs and the names don’t match up). When one AZ has a hiccup, we simple use the other AZs. Often this is a graceful, but there can be hiccups – there are certainly tradeoffs.

Second, if you are going to leverage the cloud for services, and you should, you must have a backup plan or set of protocols for what to do if it hits the fan.

For example, if S3 did go down, our WordPress CMS would be affected, as we store user-uploaded assets in S3. To remedy that, we keep a local copy on our server, so our assets stay available to our site visitors. If S3 goes down, we can make a change to a plugin configuration and our assets will still be available. When S3 comes back online, we’d flip the switch and go back to serving things from the cloud.

Third, have a communication plan ready and keep users updated during the day.

The only spot I was finding out official information on the outage was on the AWS Service Health Dashboard, which is fine, that’s where it should be. In addition, many sites put up their own pages (Quora, Reddit come to mind) saying their were being affected by the outage.

If you have a blog, use it. Same goes for Twitter and Facebook. Amazon, even though the info was hidden, was good with updating exactly what was going on and where they were in the process of getting services back online. For example:

Apr 24, 5:05 AM PDT: As detailed in previous updates, the vast majority of affected EBS volumes have been restored by this point, and we are working through a more time-consuming recovery process for remaining volumes. We have made steady progress on this front over the past few hours. If your volume is among those recently recovered, it should be accessible and usable without additional action.

Good information that’s being updated often is important to help keep customers in the loop. Compare that to Sony, who’s Playstation network has been offline since last Wednesday. Their updates have been nebulous, at best. On April 21, they posted on their official blog:

While we are investigating the cause of the Network outage, we wanted to alert you that it may be a full day or two before we’re able to get the service completely back up and running.

The last update given by the company, on April 23, said this:

We sincerely regret that PlayStation Network and Qriocity services have been suspended, and we are working around the clock to bring them both back online. Our efforts to resolve this matter involve re-building our system to further strengthen our network infrastructure. Though this task is time-consuming, we decided it was worth the time necessary to provide the system with additional security.

We thank you for your patience to date and ask for a little more while we move towards completion of this project. We will continue to give you updates as they become available.

And then, silence. It’s now Monday morning in the US and the service is not online and the current status/ETA for being online hasn’t been updated since Saturday. IGN has more on Sony’s PR response to this outage.

That type of communication wouldn’t work on our campuses. Part of your planning must be a communications plan for who is responsible for keeping a certain audience up to date on the status of services.

My colleagues at Allegheny are doing it right this morning. They had a power outage over the weekend and took to their intranet to update the campus community, on a Sunday.

Am I going to stop using Amazon’s cloud services over this outage? No, definitely not. Is this going to make Amazon improve the service? Yes. Is this a sucky way to do it? Of course.

I’ll be updating this post with feedback from other higher ed web and marketing folks. Andrew Careaga has some interesting thoughts on the outage looking at it through a lens of education.

There was an interesting post in yesteday’s Wired Campus about the University of Alabama at Huntsville‘s emergency alert system.

Last week, there was a shooting on their campus and sadly, three faculty members were killed, two other faculty members and one staff assistant was injured in the shooting. While their response after the fact has been very good, people on their campus are upset that an emergency text alert was not sent until an hour after the shooting was reported. Officials at the school report the shooter was apprehended minutes after a 911 call. The school posted this time line:

University, local, state and federal law enforcement officials arrived on the scene at 4:01 and the alleged shooter was apprehended without incident at 4:10, according to university police. Residence halls on the campus were locked down at 4:10 as a precaution. The campus was closed at 4:42 p.m. University and local law enforcement officials swept and secured the building by 5:45, confirming that no other victims were found

Despite that and the fact the event was contained to one building, an alert should have been immediately sent out, notifying people on campus to stay put and lock doors and windows until further notice. It’s always better to err on the side of caution in situations as of these, as I’m sure word had already spread about the event on Facebook and Twitter. The UAH police chief added this about the alert system:

The U-Alert was triggered late because the people involved in activating that system were involved in responding to the shooting.

At two institutions I have worked for, there were multiple people tasked with the ability to send out alerts, ranging from the Dean of Students, Security Chief and all the way down to me as web guy. That way, all it takes a phone call and the message can go out immediately.

Since the Virginia Tech tragedy, pretty much all of us have instituted text alert systems and have reviewed and streamlined emergency preparation plans across campus. Now would be a good time to review your role in emergency planning and response as a web person on your campus and if there isn’t a specific time line or set of ready text messages standing by, you may want to speak up and have those things added in to your plan.

Photo by Digital:Slurp

This week, I started a list of stuff to do this summer. It’s a great time to tackle some larger projects, and of the largest I’m looking at is deciding where to host my school’s website. The service agreements run out on my on-campus hosted hardware this month, so I’ve got to decide if I shell out big bucks for new servers or move the website off-site.

I’ve been thinking of a few reasons to move the site off-campus. In a future post, I’ll also detail five reasons to leave your web server on-campus.

So, why move your hosting off-campus?

1. 24/7/365 Support
We’re a small school with a limited team to manage our server farm. Sometimes I need to call someone at 3 in the morning and have them at the machine ready to fix the issue. I’m not diminishing the knowledge or dedication of our IT staff, they do amazing stuff, but off-hours support is a challenge.

2. Emergency Response and Availability
Having your site prepared for an emergency is a big topic right now. If an emergency happens on your campus, having your site off-site may be beneficial for a few reasons. First, you are taking the crush of traffic off your campus and onto your web host. Second, if your network is affected by a weather emergency, the website can still be used as an information resource, especially if you have a setup like one that uses a third-party tool, like Jott.

3. Reduction in bandwidth usage on campus
Our site serves a lot of traffic. I’ve been slowing migrating content to S3, but there’s still a lot of traffic going out over our pipes. Moving the site would reduce the load a bit, though I’ll talk about a potential downside to this in the next post.

4. Dolla Dolla Bill Y’all
Hosting off-site can also make economic sense. If you are buying, say, two servers and associated maintenance plans, this can easily cost over $10,000, and that’s before you start adding in software costs, backup costs and more. By hosting off-site, you are letting your hosting company worry about the infrastructure and you are spreading the cost of hosting over several years.

5. The kids grow so up so fast
For some institutions, adding server power is as easy as turning on a new server, tweaking the load balancer and you’re good to go. For a small institution like mine, this is much harder to do. Hosting off site allows you to grow and add capacity at lower cost. Doing so on-campus would incur additional cost and time. There are several hosts now, like Mosso, that will scale your site on the fly to handle the load during emergencies, Diggs, etc.

What’s Jott? Jott is, at its core, a speech to text transcription service. You call into Jott and leave yourself, or anyone you set up as a contact, a message. It’s really that simple. I called in several times today and it transcribed everything I threw at it.

At first glance, it’s a great organization app that will keep your schedule in order and allow you to remind yourself about things you need to do, especially after meetings. I’m always forgetting stuff right after a meeting that I need to do.

To me, what makes Jott a potential killer app is its “link” service. Through Jott, you can connect to other services and push your updates through Jott.

Here’s an example. It’s very easy to link up your Twitter account to Jott. You can call Jott, and say you want your message sent to Twitter. You talk, it types and moments later your twit is posted.

It’s not just Twitter – Jott can post to Google Calendar, Jaiku, your WordPress blog, and so on.

So let’s look at the higher ed uses. You can call Jott and report the happenings of a baseball game and send them to Twitter. You can keep your events calendar up to date on the fly. You could send a quick note to the Tumblr page you set up for your campus group. It really makes things simple.

The use that really gets me excited is the use of Jott in an emergency situation. God forbid, let’s assume something bad happens on your campus and your network is down. Let’s also assume, you still have cell coverage or a landline. You can have Jott set up to send updates to an off-site emergency blog hosted at WordPress.com or Blogger.

You dial into Jott, post a status update or further instructions, and it immediately gets posted on your blog. That update triggers your RSS feed update, which users can receive by email or feed reader. If you use a service like Twitterfeed, your Twitter feed will also be updated automatically, keeping even more people in the loop.

It’s also great for non-technical people who may need to update your community about a situation. We use e2campus as an emergency notification tool, and it works great, but for a novice user getting a message out can be a bit daunting. Imagine the ease of having that person call into Jott and say the update and let the technology handle the rest.

I’m not saying put all your eggs in the Jott basket in case of emergency, but it could definitely have a place in your emergency procedures.