To our customers
October 28, 2005
I wanted to add some detail to Mena and Ben's post yesterday and to give you additional insight into what we are doing to fix the current problems and how we got here.
How did we get to this place?
For over a year we have been growing the TypePad service and adding capacity as needed. This growth has been extraordinary, and we are now serving ten times the traffic we were serving a year ago. About 3 months ago we learned that the data center we were in had no more available power (not space, but power) with no option to obtain more. After some evaluation of options we decided that we needed to go through the process of physically moving out of that data center. And while we would need to grow capacity, we had to stay within the limits of the power available in our current facility.
At this point a race began to procure a new data center, buy more equipment and move data and then servers to the new facility.
We did a search for a facility that had the capacity for us to avoid getting into this situation again. We then leased space, numerous new servers and other equipment, added ten times the bandwidth of the old facility, got a dedicated data connection between the old and new facility and began to build out new cabinets to house the servers. Two weeks ago we began the move. Due to even more rapid growth than expected in the use of TypePad, we needed the capacity a week earlier than that. So basically we lost the race to move and the excess load on the servers in the old data center started to cause failures in systems that have created poor service for you and, unfortunately, further slowed the move.
The engineering and operations team have been working around the clock to fix problems as they arise, while continuing the move to the new data center and get the new equipment in place and your data over to the new machines.
So where are we now?
Over the last two days we have made significant progress in relieving the stress on the servers in the old data center and completing the move. Below is a chart presenting specific improvements scheduled over the next few days and over the next few weeks. Many of these items were already mentioned in the post yesterday but I want to be sure you have all the data you need to understand the process.
| Over the Next Few Days | Over the Next Few Weeks | |
|---|---|---|
| Storage | Add 2 Terabytes of additional data storage | Add new high-performance network storage system with over 10TB of capacity |
| Database | Implement code improvements making the database more efficient on its current hardware | Move to new high end database servers |
| Bandwidth | 5X more available than last week | Double the bandwidth again (10X over today!) |
| Application | Introduce less database intensive logic to calculate stats | Continue to make the application faster and more efficient |
| Power | Enough power for the next 6 months of growth | Power for the next year |
| Space | Space to accommodate TypePad for the next year | Options on another year's worth of space |
| Load balancing | Ten times the bandwidth of today | Double again |
| Backups | 2 tier backups today | 3 tier backup solution |
As you can see a great deal of activity will be happening in the next few days. This Saturday night between the hours of 9PM – midnight PDT, we have scheduled downtime to complete some of these activities. Specific details are posted on our status page.
By the end of next week, we hope to have all this behind us and to be back to providing you the superior service you have come to expect from TypePad. We ask for a little bit more patience until that time. We all can’t wait to get back to the situation where our customers are delighted with our service and happy to recommend it to their friends. Until that time please accept our sincere apologies and thanks for your patience.
Barak Berkowitz
Chairman and CEO Six Apart
The providers of TypePad, Movable Type and LiveJournal
