I have always thought that it is never wise to get too comfortable, to think that everything is going your way, that everything is perfect. Why? Experience. Life is very much a series of episodes, large and small, of waiting for the other shoe to drop. At best, there is a long, tranquil window of time, between shoes number one and two. Sometimes, the second shoe falls so quickly that it simply echoes the sound of the first.
And, bad things happen in threes, no? Isn't that the conventional wisdom?
I don't believe in much of that mumbo jumbo good and bad luck stuff. I simply know it is always a good idea to be prepared, to have your guard up. Carry that umbrella, you never know when it might rain.
What are the chances of a catastrophic data loss on a network server, and then on each of the son/father/grandfather backup tapes, all failing to yield their precious information for a system restore? I don't know what the odds are, but I do know that anything that can go wrong will, eventually. They went wrong last Friday, and we are still trying to recover. Add up the cost of people working around the clock, data retrieval experts trying to salvage information from backup tapes, many frayed nerves and too much coffee, and you will get an idea of the present atmosphere here in my office.
It's not that we didn't have proper systems in place. We have built-in redundancy with mirrored drives on our server, several layers of backup, virus protection at both server and client levels, and firewalls against unwelcome intrusion. Yes, the systems are in place, but they aren't a damn bit of good if someone in the chain of responsibility doesn't follow the proper protocols.
Oh well. Sh*t happens, and all that. I'm not someone to whine, though. We will get through this in good humour and with all the aplomb we can muster. When something like this happens, in any organization, the star employees shine their brightest. And I have a bunch of those. They are working at breakneck speed to get us back up. Data retrieval experts are trying to see what information they can extract from fried tape media. And while everyone else is working on solving the problem, I am writing a blog post. After all, everyone knows that bosses can't actually do anything useful, right? I do know that when there are people around who know better than I how to do something, it is usually a good idea to stay out of the way and let them do their jobs.
Things like this are a pain in the butt, but it is not the end of the world. In a few days, things will be more-or-less normal again, and life will go on. I guess I will have to add yet another level of redundancy to our backup systems, and appoint someone to monitor the persons who were supposed to do things to keep this sort of thing from happening. Too much protection is better than not enough, right?
Yes, it is definitely a good thing that shoes come in twos, not threes.