We try really hard to keep our systems running as close to 100% uptime as possible.
We’re hosted in a great data center that’s one of the best in the USA for systems that need to be PCI compliant. We share the place with banks, payment gateways, etc.
Our database is managed by the data center staff. This is good for a number of reasons, including keeping the DB separate from our app and being managed by others who are typically better at DB management.
We were informed recently that the DB we’re on is growing too big and is going to run out of disk space very soon. A number of attempts have been made to remedy this without downtime. The data center DB staff has tried various things with help from their DB vendor, but with no success.
Thus, we have been told today that they have to take the database offline tonight to install new disks.
They estimate that it will take 2 hours to complete this process, but having been involved in many such things for years, I’m afraid there’s a good chance it’s going to take longer.
They’re going to start this at 10pm Pacific tonight (Wednesday) and they hope to have it done within 2 hours.
Chargify hosted signup pages and API access will be offline during this time. Our merchants who rely on those pages or API calls to sign up and manage their customers will not be able to do so.
This is extremely painful! And we know it’s more painful for our merchants.
This database downtime (at least without much more warning) is shaking our confidence in our data center.
If you know me, you know I’m a founder of Engine Yard, which Chargify was hosted on before we needed PCI Level 1 compliance. Very frustrating, but PCI adds all sorts of benefits (and requirements) that are nearly impossible to replicate elsewhere.
We apologize for this – this is not what we want our merchants to experience as we grow with them, so we are working on a lot of infrastructure improvements over the coming months.