Just thought I'd post a letter I just got from the CEO of host my site about what happened. As expected it sounds like even though it was no fault of theirs they will be taking some measures to make sure it doesnt happen again.
Dear Valued Customers and Friends: I am writing today to apologize for the service outage you experienced yesterday. As many of you know, our first datacenter facility (DC1) was impacted by a severe fiber cut. The fiber cut was caused by subcontractors of Comcast cutting our lines with a backhoe while installing new service into our office park. The cut of this fiber, which has been trouble free for the 5 years we have used it, resulted in loss of all connectivity for our DC1 site, including one Gigabit link and two OC3's. Our staff has made some pictures of the incident available at the following URL: http://hostmysite.com/emergency/ We are still working with all involved parties, including Cavalier Communications (the fiber provider) to make sure that those responsible for this mistake are held fully accountable. I was personally on the scene for the entire repair process, even going so far as to pick up a shovel and help the workers dig a trench for the new fiber being installed. I know it was a long process, and although we did everything we could to move as fast as possible, the problem was compounded by two issues. First, the fiber we utilize in our business park was originally installed by the power company and is directly in line with dangerous power cables feeding the park. This made digging out the old cable and splicing in a new patch very difficult. Second, the cut occurred very close to the sidewalk where the fiber then runs under the parking lot. This left the splicers without enough slack to easily splice in a new piece of cable (they were hoping for 30 feet of slack and got less than 5). Throughout the course of the day, our staff really pulled together to keep our customers informed and to execute on our emergency plans. If there is any silver lining on something like this, I felt fortunate to witness the amazing efforts of our staff. We had 5 year veterans and employees fresh out of training on their first day in the call center putting in fantastic effort. I would also like to apologize for those customers that were unable to get through to our support center or experienced long wait times. We are committed to providing live phone service without hold-time, and although we kept up with call volume for most of the day, our phones were overwhelmed at the outset of the problem. After an incident like we experienced yesterday, I thought it was important for me to reach out to our customers and reassure you of our commitment to providing the best web hosting service available. Even though this incident was caused by the action of a third party, we do not discount our full responsibility to our end customers to provide a network that is 100% reliable and available. I realize that you trust us with your website so you don't have to worry about incidents like this. We are taking a number of steps to prevent this sort of problem in the future. Our fiber provider, Cavalier, is working non-stop to make sure that all fiber in our park is retraced and clearly marked. They are also working with us to implement additional diversity measures in the short term to limit risk. Our Datacenter 2 (DC2) facility, which is a state-of-the-art facility with multiple diverse fiber conduits, is also being upgraded to accommodate all servers from DC1. While we had been planning to slowly make the transition over the next 12 months, we will do our best to accelerate the process while minimizing impact to our customers. I understand the magnitude of yesterday's incident, but I don't want our customers to lose confidence in the ability of our DC1 site to provide services. This site has a great 5 year track record and we are leaning heavily on our providers to make the necessary changes in the short term to secure additional diversity. Finally, I would like to let you know how much I personally appreciate your business. I treat your website and your web business as if it were my own. You can trust everyone at HostMySite.com to care for your web presence, and when things do go wrong, you can trust that we are making a Herculean effort to get things right. Every staff member is a caring, dedicated professional that doesn't hesitate to pitch in wherever needed. You can also count on the fact that I will be right there with them every step of the way, along with a talented management team that has been with our company for a very long time. My personal contact information is available to you as part of this email should you wish to follow up with me directly. Regards, Lou Honick, CEO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:250228 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
