There is something I always found attractive about working in high-rise office buildings before the whole “working from home” thing was a functional reality.
It had a sense of importance to it, almost a movie quality that was less Office Space and more Mr. Anderson in The Matrix. The work day wasn’t a world on fire with people that were there to simply watch it all burn down, since that was the most entertaining thing to do, and you could actually get paid for it. Instead, every day was a tactical adventure with a real IT purpose and mission to be accomplished. You got to meet people you would never otherwise see. Pretty girls that would not otherwise give you the time of day would treat you like a celebrity and have an honest conversation with you.
These were the waning days of Token Ring, and the advent of Ethernet, when in a networking sense we were still banging stones together to see if they’d spark.
I’d just finished a job with a telecom company that had been purchased after I’d set up their intranet, introduced them to RedHat, and developed a website for them.
That’s a different story.
The new position was one that I was very excited about. A rather large insurance company had bought five other insurance companies over the previous years and made the move into a high-rise in a lovely wooded area just north of a major city. Needless to say, this had made them even bigger.
The issue I was hired to help resolve was to upgrade all of the antiquated technology where it could be found and connect those systems into our new network. I was a newly-minted CNE (Certified NetWare Engineer), and they had older servers to upgrade, and now five networks to splice together in one building, under one umbrella.
It was right up my alley.
For months, everything went according to plan, some early mornings, some late nights, a few overnights from time to time. One of the transfers landed me a mini-VAX that they were going to trash, so they put it next to the dumpster, and off I drove with it.
I had been partnered with a fellow we’ll call Harry, who had actually been working for the company, and in the process of things I imagine had been let go and brought back for more money as a contractor. He was much more versed in their Windows NT setup than I was and could have been, so we joined forces. With that, I brought the gift of IntraNetWare 4.11.
IntraNetWare 4.11 had a nice feature of combining all of your login connections together. Previously, one had to login to NT first, then make the connection to NetWare, and I was snotty enough to think the whole darned thing was aesthetically horrid. 4.11 was cleaner, prettier, and did everything more easily for us on the backend.
It was these technologies that led Harry and I in our quest for more firepower, and of course, we got the approval in four memory chips, two for each of our domain controllers that would almost triple the amount of memory on those machines.
We had everything scheduled, and the morning of the upgrade had arrived.
It was a blissful morning. I had my Starbucks quad-shot Venti Mocha in my custom cup, four pristine SIMM chips, and a willingness to succeed. I was going to get a jump on the day and knock this puppy out for the good of the company. My intentions were as pure as the driven snow.
Once in the data center, wrapped sufficiently in my fleece hoodie, I located the primary and secondary domain controller boxes, pulled them off the rack and moved them to our workbench.
The original plan was that Harry would meet me in the datacenter, we would upgrade them both and put them back in operation in record time. We thought we had it all planned down to the heroic second. But once I got to my desk, an hour or so early, I might add, I had a voice message that Harry would be a little late coming in. A willingness to give him less to do, and a desire to have speed from the higher-powered servers was what propelled me to the position at the workbench.
I opened the two machines, inserted the memory chips, and powered them back on again. I waited for the familiar Windows NT bootup screen, but instead was looking at a black console screen asking me to enter a code.
It never looked like that before. I was confuddled. Harry passed by the window on the way to his desk, looking like he’d been caught in a whirlwind, trying to fix his hair. He wasn’t awake yet, and gave a half-hearted wave as he ambled towards his cubicle.
I tried a few more things I thought might be codes, but none of them worked. I called Harry, who did not feel like talking through what was likely a hangover from Hellraiser. I quickly explained the situation that he was sleep-listening to.
“You have the SIMMs, yeah?” he asked.
“They’re in the machines already,” I replied.
“Well, ask the HP Rep what’s going on. He knows more than I do.”
“Who?”
“The HP guy. Isn’t he next to you?”
“Uh, no. I don’t know who that is. Why would I have a guy from HP here?”
Harry sucked in his breath sharply over the phone and then moaned, “Oh, shit. No, no, no…”
Harry was now fully awake.
“What?”
“We forgot to call them for service yesterday.”
“Explain. I’m not understanding,” I responded, confused.
“You cracked both boxes open at the same time, didn’t you?”
“Well, yes. I know how to replace memory.”
“You do them one at a time, not at the same time! Those are different, they’re under contract. I shoulda told you about that, but I assumed you knew. You have to have an HP Technician on site to enter a code if the server is physically opened. Otherwise, it won’t boot. That’s probably the error you’re seeing. This is not good. We’re locked out. We can’t bring those servers back up. I gotta call them right now!”
He hung up, and I began to mentally calculate what I thought was going to happen. The box I had been working on was the Primary Domain Controller, and the machine next to it, in identical state, was the Secondary Domain Controller.
If you know, you know what level of Hell we were on at this moment.
And why I was an idiot.
Because that is precisely the point I realized what I had actually done.
With the Primary and the Secondary domain controllers down, there was no network access. No one could get to any of their applications, email, or anything else because I was looking at the two dead machines that handled all of that.
I, friends, was the single point of failure. For a national company that had offices in all 50 US states.
For a company that in a few short hours had planned a meeting with multi-million dollar potential clients who had flown in from out of town, headed by a vice president that could not get to her presentation, her notes, her email. Or anything else.
At that moment, I was transported at lightning-fast speed from the consummate expert and tech god I had previously been treated as to a specimen that was mentally, very possibly, only a degree or two off of plant life.
It was a stupid mistake, and the more painful part was that I knew better. I knew that the two should never come down at the same time, but was not thinking about it at the time. Over 25 years later, this still remains the dumbest mistake I ever made on duty. Thank goodness I never fell for sudo rm -rf /.
9 AM came and went, and the calls hit like a party line. It was explained that yes, there was a hardware issue, and yes, HP was coming to fix it. He appeared around 11 AM, waltzed in with a smirk and said, “This is a helluva way to get hazed,” tapped the code into each machine and left. The whole process happened in less than ten minutes.
Harry had an impromptu meeting after lunch. I wasn’t invited, but that wasn’t unusual. His silence afterward was.
Any math I had going on in my head was solved the next morning when I saw the cardboard boxes in my cubicle and a cart nearby. A security guard I was friends with stood by with an irritated look on his face as the boss from my contracting company that I had only seen twice informed me that my services were no longer required. He handed me my final check.
I understood.
I wasn’t upset. I might have been in most circumstances, but I knew something that they didn’t know yet. The security guard escorted me out, and the irritation turned out to be against them, not me. “This is just stupid,” he said, shaking his head. “It just seems wrong to me. I don’t think they know what they’re doing.”
“It’s okay,” I reassured him with a smile, “I have an ace up my sleeve.”
He cocked his head and with a grin asked, “And what’s that?”
“I have an interview with their Unix datacenter next week. In California.”
“You sneaky little bastard!” he snorted happily, “Good luck and good move. I hope you get it!”
The Aftermath
I hadn’t lied.
I had met a girl on the other side of the country, and in that process, the job listing just came up. At the same time, I had learned enough from my short experience to know that hedging your bets is always the best idea, so I gave my resume to a recruiter who help land me two other interviews.
The datacenter didn’t pan out.
I also heard that the vice president with the delayed meeting was so angry she threw a whiteboard eraser at Harry. He later wouldn’t confirm or deny it. I have no idea what happened to any of them. Except she was not only the one that brought an abrupt end to my contract, but also the one that put the kibosh on the datacenter interview.
My second interview snapped me up so fast my head spun, and two weeks later, on Labor Day, I was living in California.
I married the girl. We divorced seven years later. No erasers were thrown.