Unplanned Downtime: Managed/Shared Database Failed Just 2 Days Before our Move

by Lance Walley


This is definitely not what we wanted or expected just 2 days before we move to our own database machines.

Here’s the summary:

1. Our data center runs a robust, redundant PCI-compliant managed/shared database system. It’s been a reasonable option for 2 years, but we’ve been outgrowing it – slowly at first, then quickly at the end of 2011.

2. In Q4 2012, we decided to move to our own dedicated DB system. This entails having the data center set up equipment, as well as bringing on a sys admin in-house as our general needs increase. Our new sys admin started earlier in January (thanks @sebastianstadil for the personal recommendation).

3. The move away from the managed/shared DB to our own DB system was scheduled for this coming Friday night (just 2 days from now), and we sent out a notice about it yesterday.

4. Our data center’s DB went down early this morning, and they have been unable to bring it back up, at least without losing some data.

5. We had data replication running 2 nights ago, and given the data center’s inability to restore their database quickly, we decided to expedite the work on our new DB… to move to our system now instead of Friday night. That will allow us to get back up around the same time, but with little or no data loss, and without another (albeit short) cutover downtime Friday night.

Sorry about this. We know our merchants depend on us to keep their businesses running!

We also have a solid recommendation for a new PCI data center, recommended to us by a respected ecommerce site that’s made the move through several data centers and seems to have found one that keeps them happy long-term.

—- Lance Walley, co-founder/CEO
—- Chargify
—- lwalley@chargify.com
—- 415-244-0349 cell

Turn Business Blockers Into Business Accelerators

Revenue management for SaaS allows businesses to respond quicker to market needs, build long-lasting relationships and derive meanigful insights from the noise.