Dan's Drivelings » MySQL

Database bullet dodging and lessons learned

Dan Skaggs — Mon, 13 Apr 2009 03:56:08 +0000

I dodged a very large (metaphorical) bullet yesterday. I’m not in the habit of trying to find bullets to dodge, but this one found me because it tuns out I didn’t plan as well as I thought I had.

It all started yesterday morning when a client asked me to make some batch updates to the data in his MySQL database. So, I wrote a script to make the required updates. Before running the script, I made a backup of the database in case something went wrong. Everything was good so far. I ran the script, checked the database to see if anything looked amiss and called it a night (oh yeah, this was like 11:00pm last night). This morning, I get an urgent text message from the client saying that the data didn’t look quite right. No problem I think, I made a backup copy of the database last night, I can fix this. But nooooo, that database backup wasn’t around any more. Why, you ask? Oh yeah, that. I saved it to the same location that the regular, nightly backup saves to. So by the time I needed the backup this morning, it was already gone. Luckily, the eventual fix was very simple once I figured out where my script had gone wrong. It could have been really ugly though.

So, here’s to lessons learned:

Don’t do production database work late at night. The root of the problem was that I misread the client’s original email and wrote my script incorrectly because of it. Part of that failing was the late hour and long day that preceded it.
Plan, plan, plan and re-plan. Thinking my one backup was sufficient was my second mistake. When working with production data, you can absolutely not be “too backed up”
Test your scripts on non-production systems first. This client doesn’t have a dedicated development system, but I do have a few-weeks-old copy of the database on my development machine. Another mistake I made was being lazy and not testing these changes on my local copy before applying them to production
Have a plan in place to verify the database changes as soon as they’re made. My cursory inspection of the data didn’t reveal the actual problems that occurred. If I had done this when the client was available to test the changes, we’d have been able to immediately restore the database and assess what went wrong.
Nothing is ever as simple as it seems. These updates seemed like a walk in the park and lulled me into not taking the precautions I knew I should have

So, in the end, my client has his data back where it belongs and I have a few things I’ve learned I’ll never do again. My biggest embarrassment in this whole thing is that I consider myself a senior developer. I’ve been writing web apps for almost 10 years across Oracle, MS SQL, MySQL and PostGres databases and I should have known better than to take shortcuts. My only excuse is that I’m human and I can only try to learn from my mistakes.

Start/Stop MySQL from a terminal session on OS X

Dan Skaggs — Wed, 30 Jul 2008 03:01:20 +0000

This is a quick post as much to remind myself the next time I need it as anything about how to start and stop MySQL from a terminal window on OS X.

Start:

sudo /usr/local/mysql/support-files/mysql.server start

Stop:

sudo /usr/local/mysql/support-files/mysql.server stop

Server migration complete

Dan Skaggs — Thu, 21 Jun 2007 22:12:03 +0000

Just a quick note to mention that my blog is now running off my virtual private server at

Stuck in D.C.

Dan Skaggs — Sat, 17 Mar 2007 17:12:29 +0000

Thanks to a late afternoon winter storm yesterday, I am spending the weekend in Chantilly, VA. I was due to go home yesterday (and really looking forward to it too!) but the storm cancelled most of the flights out of Dulles and I was told they couldn’t get me back to Nashville until Sunday. Since I had to be back up here Monday, it didn’t make sense to do that, so I’m just not going home this weekend.

Verity woes this week

Dan Skaggs — Fri, 12 Jan 2007 13:22:01 +0000

I’ve been in Charlotte, NC this week for my “day” job and have been working on a side project from the hotel that involves converting a large amount of data from a, let’s just say, “non-standard” database program to MySQL. One of the tables stores information about 14,000+ legal documents that we have in PDF format on the filesystem.

Eclipse/Ant/MySQL goodness

Dan Skaggs — Tue, 02 Jan 2007 02:40:15 +0000

I’ve spent several hours this past weekend working on a project for a client that uses a MySQL database backend. It involves converting thousands of records from XML files exported from the legacy data system. One of the things that’s been bugging me is the process that I have to go to every time I make changes to the the database or import more data into my local development copy in order to get those changes/data up to the staging server.