Everything Typepad

About Pages for Sharing Info

This post is the fourth in a series that introduces some great TypePad features and offers tips on finding help with TypePad in general.

Kymberlie on About Pages for Sharing Info.

Q: What can you tell us about yourself?

I'm Kymberlie and I've been working for Six Apart in Technical Services for about a year now. I've been using Movable Type to run my main blog, Neurotic Fishbowl since January 2002. There I talk about all kinds of random things from poker to skydiving to nights out on the town with friends. Blogging has been a great way for me to meet other people and it seems like these days, almost all of my friends are bloggers.

Adding Sidebar Content with TypeLists

This post is the third in a series that introduces some great TypePad features and offers tips on finding help with TypePad in general.

Melissa on Adding Sidebar Content with TypeLists.

Q: What can you tell us about yourself?

Not much, due to the restrictions of the Witness Protection Program. (Kidding!) My name is Melissa, I live in Detroit and I've been blogging since June 2002. My main blog is powered by Movable Type and available here: anything but ordinary. I also have a TypePad blog and a LiveJournal. :D

Compensation for TypePad Outage

As some of you might know, last week on Thursday night and Friday morning TypePad suffered a severe hardware problem, causing blogs to be inaccessible for two hours and posting to be inaccessible for 16 hours. This was the longest outage in TypePad’s history, and given our performance problems in October I understand why at this point an apology might sound hollow. But the truth is, we are very sorry. The outage, of course, is inexcusable, and for us at Six Apart it was mortifying.

We know that our customers rely on us and expect to receive superior service and performance. Friday last week we didn’t provide that kind of experience, and as a result we will be crediting all active TypePad members with one free week of service. We will apply this to your account by extending your next payment due date by seven days. Customers billed monthly will receive this extension in January; customers billed on an annual basis will have the extension applied to their next billing date.

Over the past ten weeks we had been making great strides towards creating a faster and more reliable TypePad. While not yet perfect, November was better than October, and December was better than November…until last Friday.

While there is always a chance of outages on any web service, the fact is that when you are upgrading systems and moving data you create additional risk. The move to our new data center is paying off, but the frustrating reality is that these changes for long term stability have been painful. In fact this outage was the result of a failure in a device we were installing to provide extra redundancy in our disk storage system. I know it is not a comfort to those of you who were inconvenienced last week, but I’m sure you will appreciate the improved speed and reliability of TypePad.

We’ve also heard from some customers that it can be difficult to find information and announcements from us about TypePad maintenance activities and problems. We are working to improve that, and while we do please bookmark both this blog and status.sixapart.com, which are our two key communication channels.

I want to thank you all for the patience and loyalty so many of you have expressed to me. And I want to make it clear that we understand and share the frustration and anger some of you have been feeling. We are doing everything in our power to avoid ever having a problem on this scale again. That said, problems sometime will happen, but I promise;

1. We will continue to work hard to lower the risk of you being affected by any problems that may happen behind the scenes.

2. If you are affected by a problem that we were not able to protect you from, we will clearly and proactively communicate the problem and our estimated time to resolution.

3. We will work after the problem to analyze its cause, and whenever possible, eliminate the weakness that allowed it to affect you.

Finally, I always appreciate hearing from our customers, especially in times like these. Feel free to email me at [email protected]. I read all these mails and reply to those that need a reply.

This is a great time of the year and we know that next year will bring improved reliability and speed for TypePad. To all of our friends and customers from all of us at Six Apart, we wish you a wonderful holiday season and a happy New Year.

Network Maintenance

On our status site we have announced a maintenance window for this evening (Wednesday, December 21st) between 10:00 pm PST and 11:00 pm PST, during which time the application and a subset of customer's blogs will be inaccessible for one period of approximately 10-15 minutes.

This maintenance window is the last step in our physical move from our old data center to our new data center. While over the past several weeks we've shifted most of our network traffic (for both the application and published blogs) to our new facility, tonight we'll be removing the final physical network connection at our old facility. While there are always risks in maintenance windows, the risk of this activity is very low, and we can fall back to our original network configuration very quickly if problems arise.

As always, we'll be posting updates to status.sixapart.com at the beginning and end of the maintenance window. We appreciate your patience as we complete the move to our new data center.

Update, 11:00 pm PST: The network maintenance was completed successfully.

Recap of Friday's TypePad Outage

Now that we've restored user's photos and files, and the application has been reliably back online for three days, we want to provide more details about our outage on Friday and the steps we took to recover from it. We will post more information in the coming days about our plans to avoid future outages and compensate customers for Friday's downtime.

Before we get into the details, we want to reiterate just how sorry we are for the inconvenience this has caused. We know our customers rely on us to provide superior service and performance, and that on Friday we let you down. The fact that Friday's outage came on the heels of our performance issues in October is obviously frustrating, both for you and for us.

What happened

As you may know from our previous posts to Everything TypePad, we have been migrating our operations to a new state-of-the-art data center. Over the past two months we've nearly completed that move, and as part of it made upgrades to our networking equipment, our bandwidth capacity, our application hardware and our storage systems. The maintenance window we took on Thursday night was designed to accomplish two things: configuring and testing high-performance and redundant network traffic load balancers, and configuring a recently added redundant "head" in our new, high capacity disk storage system.

The first task on Thursday night went off without a hitch. The second task, however, was where the problem started. As part of the configuration of the new storage system, we needed to reboot the storage device. When we attempted to bring the device back online at 10:50 pm PST, a hardware failure occurred and damaged the index of the file system. Essentially, the system couldn't mount the drive, even though the data was still there.

This disk failure led to both published blogs and the TypePad application being down. After diagnosing the problem in order to understand the severity of the issue, we decided to serve published blogs from a snapshot that was between 2-6 days old. This was why some TypePad blogs were out of date.

Recovering TypePad and restoring your data

Through the night and into Friday, our operations engineers worked to diagnose the problem with our storage vendor, and to bring the application back online.

We knew that when we brought TypePad back up user's posts, comments and TrackBacks, TypeLists and photo albums would be current in the application's database, but out of sync with their published blog. When we restored the application at approximately 3:00 pm on Friday, we encouraged users who were logging in to republish their weblogs to bring them up to date, and at the same time began a process to proactively republish user's content on their site. We advised users that republishing would resolve some issues, but not those related to photos or files that they had uploaded to TypePad in the past several days.

On Saturday morning we completed our process of republishing user's weblogs. Also on Saturday morning, we worked with our storage vendor to bring the data with the damaged file index back online in a new unit that they rush delivered to our data center. Once we were able to bring that machine up, we began the process of restoring user's missing photos and files to their weblogs and photo albums. Additionally, over the course of Saturday night and Sunday, we did more republishing of weblogs, photo albums and TypeLists to restore data and to address particular customer issues.

As of Sunday night at approximately 10:30 pm, we completed the data restoration process. We encourage any of our customers who are having any unresolved or related issues to file a help ticket with our support team, and they'll work hard to get your problems resolved. You can file a help ticket by visiting the Help tab under the Control Panel inside TypePad.

We hope that this post provides a bit more detail about what happened on Friday and over the weekend. Again, we are very sorry for the inconvenience, and appreciate your continued patience. We will post more information in the coming days about our plans to avoid future outages and compensate customers for Friday's downtime.

TypePad Update: Data Restored

As we reach the end of the weekend, we want to follow up with more information about the activities over the past two days, and the current state of TypePad.

• Over the weekend we were able to recover the missing photos and files from the outage on Friday, and they have been successfully restored.
• On Saturday morning, we completed republishing user's weblogs, which brought blog posts, comments and TrackBacks up to date. Over the remainder of the weekend we proactively republished TypeLists, specific blogs, and photo albums as we restored files, and to resolve any individual customer issues.
• Since Friday afternoon, the TypePad application has been performing well according to internal metrics and Keynote, our independent monitoring service.

As always, our support team is standing by to help any of our customers who have any unresolved or related issues. Customers can file a help ticket by visiting the Help tab under the Control Panel inside TypePad.

We appreciate your patience as we restored your data. We will continue to use status.sixapart.com for more frequent updates regarding application availability and maintenance activities. Please also note that We will be posting more in-depth information about Friday's outage to Everything TypePad in the coming day.

TypePad is Back Up

The TypePad application is back up, and users can now log in. We've begun the process of republishing user's blogs. As we noted in our last post, if your blog is out of date and needs to be republished, you can do that yourself inside TypePad by visiting the Design tab for your weblog, and clicking the "Republish Weblog" button.

We want you to know that your blog data is safe. We have no reason to believe that any of your posts, comments, TrackBacks, photos or files have been lost. Over the weekend we will be restoring any photos or files you uploaded over the past several days. We will be providing updates throughout the weekend to status.sixapart.com, and will post any major updates to Everything TypePad.

Update on TypePad Issues

Here is an update on the current issues we're experiencing with TypePad.

• Most importantly, we want you to know that your blog data is safe. We have no reason to believe that any of your posts, comments, TrackBacks, photos or files have been lost.
• We are currently working to restore access to TypePad. We expect users to be able to log in to the application and post to their blogs later this afternoon Pacific time.
• When TypePad returns, your published weblog will be in a state from a recent snapshot, taken a few days ago.

Over the next few days we will be taking two steps to bring your weblog fully up to date.

• First, we will be proactively republishing weblogs so that they contain the most recent posts, comments and TrackBacks. However, when you log in to TypePad, you should feel free to update your blog yourself, if our process has yet to reach your site. You can do that by visiting the Design tab for your weblog, and clicking the "Republish Weblog" button.
• Second, we will be restoring photos and files that you've uploaded to TypePad this week. To reiterate, we have no reason to believe that any of your photos or files have been lost, and expect to have them restored to your blog no later than the end of the weekend.

We appreciate your patience as we work through these issues. We will be posting updates to status.sixapart.com, and will post major news to Everything TypePad.

Current Issues with TypePad

During routine maintenance of our network and storage systems last night, we experienced an issue with our primary disk system where data from published blogs are stored. We are currently running diagnostics on the device, and working to restore your data as soon as possible. Verifying data can be a slow process and will take time.

In the meantime we are currently deploying backup copies of your weblogs from approximately 2 days ago. This is what will be displayed for your blog. The TypePad application is currently unavailable, which means that users will not be able to log in, and visitors to weblogs will not be able to post comments. We are working to bring TypePad back online as soon as possible.

We appreciate your patience as we work through these issues. We will be posting updates to status.sixapart.com throughout the day, and will post to Everything TypePad with major news.

Wide Variety of Themes available!

This post is the second in a series that introduces some great TypePad features and offers tips on finding help with TypePad in general.

Carla on Theme Selection.

Q: What can you tell us about yourself?

My name is Carla and I have been working for Six Apart in the Technical Services area for a few months. I've been a blogger for 5 years now and a user of Six Apart products for most of that time so I'm very excited to be working with TypePad now. I can be found at carla.typepad.com blogging about movies, music and everyday life and at Product Girl blogging about my favorite beauty products.