It happens to every web developer at one point or another. You do something completely innocuous to a website and the whole thing goes kaput. It happened to us this week. I made a simple change that crashed a site we were planning on launching. We spent 5 hours trying to fix it and another 6 completely rebuilding it. The process was grueling, but we learned some nice lessons. We did well with one, not so well with one and horribly on the other. Here's what happened:
The Story
A while ago we gained a new client who had all of his ducks in a row. He had a plan. He even had a point person. We were pretty excited about this. We followed our usual course of action. I handled the technical and the big picture while Kristen handled the detail work. Things were going fine until we ran into just a few problems. Though the owner knew his stuff, the point person had a completely different vision for the website and no plan on how to get there. So, things started getting confusing. Kristen put in all of the content only to have the (similarly named) products changed and then moved and then recategorized and then given new pictures and then new PDF descriptions and then moved back. The entire thing was convoluted, and she verbally kicked herself the entire time for not being more organized.
The site was done, though. We had a few quick corrections to make on Monday before the site launched on Tuesday. While I was making these simple changes, I noticed that the Content Management System needed updating. I have updated CMS's a countless number of times and never had a problem. It is a perfectly safe thing to do. But, for some reason, the entire thing crashed. I mean: white-screen-of-death crash. There was no warning, no error message, just death. I did not panic at all. I thought to myself, good thing I have a backup system in place.
For reasons that are undoubtedly my fault, the back up system wasn't working. And, by that I mean it wasn't on; I hadn't turned it on. I panicked. I tried the rest of the day to restore a site from a backup that simply did not exist. Finally I had to come to grips with reality. It was time to rebuild. Kristen and I came back to the office that night (after I put my kids to bed) and worked until 2am.
It was pretty bad, but we got it up and running before the client ever knew it was down. And, oh, the lessons we learned
The Moral
Get Organized
(We really did poorly here.) Yes, the information you are given may be presented in a confusing manner and doubled over itself many times, but that's no reason to add to the confusion with disarray. If you you are given a bag full coins of various amounts and asked to tally their value it is unlikely that you would begin adding things up as you pulled the coins out at random. Odds are you would survey all of the contents and organize them by category. By doing this you can complete the task more quickly, accurately and flexibly. If coins are added or removed halfway through the process it would be no big deal. You would just go to your categories. The confusing process can be simple and consistent when it's organized.
Back it Up
There's an old mantra in the tech world. "Save early. Save often." This much I thought I was doing. However, there is a wisdom that overshadows this. "Inspect what you expect." This is true for both employees and for systems. A backup system is supposed to be your safety net, your shield, your life support. It is what is there if the worst happens. Who in his right mind would just assume that his alarm system works without testing it out? Who would rest easy just assuming his kids are safe without actually finding out? Who would believe that a backup system is running properly without ever checking on it? People who are busy and distracted, that's who. A backup system can save time and effort when time and effort are spent making sure it works.
Have a system
(This is where we actually did well.) I've got one more saying for you. "Good help is hard to find." I guess that's generally true, but I ran into Kristen in a parking lot. She is a great asset to this business, and I don't thank or pay her enough. She and I have a wonderful working relationship because she likes to do (and does well) all of the stuff that I don't like to do. Likewise, I really enjoy the parts of the job that intimidate her. So, we've got a great system in place. We know what to do; we like what we do, and we do it symbiotically. A good system is the cheat code to a hard game; it helps you win, and winning is fun. A good system can increase productivity and morale even in the most challenging of circumstances.
So, before trouble strikes be prepared. Get organized; establish a good system, and back that junk up!