Hosted databases sound great. Pay a flat monthly fee for a certain amount of service. Typically you will see a storage limit but some also limit the number of connections and offer nicer features for higher plans like replication or automatic failover. Heroku heavily pushes these types of services through their heroku add-ons platform.
When using one of these “add-ons” through Heroku it makes things even nicer. The connection url is automatically added to your application’s environment config, billing is lumped together with your hosting bill, and changing plans (up or down) is seamless on most providers.
Now here’s why you should avoid them, specifically my experiences with MyRedis run by Miles Smith and formerly Jacob Jervey.
MyRedis originally entered private beta with Heroku in October 2012. We started using them for a client on April 4th 2013.
Trouble began when on May 1st we exceeded our connection limit (100 connections) due to a usage spike. MyRedis has no notion of soft limits, only hard ones. The only way to fix this was to provision another instance and use that one. No upgrade or migration plan was in place to deal with these contingencies. A side effect of maxing out the number of connections was that the MyRedis dashboard was completely blank, showing ??/100 connections and zero memory usage. The dashboard itself uses a connection to poll the Redis server for information about stats, and having a hard connection limit in the configuration prevented the dashboard from being responsive when it was most needed.
I choose not to move away from MyRedis at this point, and in hindsight this was a bad idea. Had I known that only one person was continuing to run a platform that many people relied upon I wouldn’t have continued to trust it to manage my data. Part of the reason for choosing an externally hosted database is because it takes an operations load off of our team. We don’t need to go upgrade the servers, check configurations. Easy justification of the cost.
On July 9th I received an email from MyRedis saying that they were shutting down the entire service down at the end of the month. I’m skipping the extremely confusing emails that said everything would be shut down after three days.
Is 21 days notice acceptable for a service that I hinge a business off of? I can’t find a terms of service anywhere on the MyRedis site, is that my fault for not properly vetting them?
To top all that off, today my Redis instance died completely and knocked several websites offline. I had to provision another instance on the fly and link that back up. Thankfully I’m not in Kenya or anything like that.
Oh, remember where I said before that MyRedis was formerly run by Jacob Jervey? You might find this and this interesting reads. Apparently Mr. Jervey went AWOL a few weeks back and this contributed to the shut down.
I need services to be incredibly reliable. 99.999% uptime is an absolute must, with solid fall back plans in place for problems. MyRedis has proven to me that in order to reap the benefits of an externally hosted database I need to do a significant amount of research into that company.
Let’s do a quick checklist of why a hosted database solution was appealing:
- Integrated billing — still great.
- No support or maintenance time — terrible. One major outage, a spontaneous service shutdown notice, and an instance dying with no plans to recover it.
- Automatic backups and failover — utter bullshit. Not only have I never seen a backup of my Redis instance, but there are no options to restore or manage those backups. Dead / over connection instances were not failed over to make them available again.
- Helpful support and no surprises — I think this speaks for itself. On the original max connections discussion I offered a lot of help regarding how to better manage this problem in the future.
I’m done with hosted database solutions. If you’re prototyping an application, or really don’t care about the data, go for it. It’s cheap and avoids having sysops experience which is great for frontend developers. For our team, making sure we can deliver exceptional uptime and service is a priority. Unless other “DaaS” providers can match that, we can’t use any of them.
For full disclosure here’s what we’ll be migrating away from:
- Heroku PostgreSQL is frankly the best DaaS we’ve used and I’m sorry to let them go.
- Redis-To-Go temporary replacement for MyRedis.
- MyRedis is shutting down, so duh, moving.
- MongoHQ offers a neat admin panel, but you have to pay through the nose (starting at $1,345) to get replica sets.
Here’s what we’ll be migrating to:
- Cloud Servers hosted by Rackspace.
- PostgreSQL cluster configured at a database level for each client.
- MongoDB replica set for each client.
- Redis as a central caching server.
As I put together these systems I’ll be writing some basic documentation and posting it here to help other people get their feet wet from a sysops / dba perspective. The upfront time cost on this won’t be cheap, but in the long run avoiding problems like the above will pay off.