From the blog.

Managing Digital Racket
The more I tune out, the less I miss it. But that has presented me with some complex choices for a nuanced approach to curb
Complexity – My Friend, My Enemy
Over my years of network engineering, I've learned that the fewer features you can implement while still achieving a business goal, the better. Why? Fewer

Unwelcome to the Zed-Zone

1,234 Words. Plan about 8 minute(s) to read this.

Zed sat alone on the floor in the dark, the data center having just gone quiet. This was not good. He let his UTP cable crimper drop to the floor and raised up his six foot five frame, bracing himself against cabinets on either side of the hot aisle. He listened. Surely *something* in this hundred thousand square foot, multi-tiered, concrete encased, single tenant monsterplex was still running.

Silence.

Panic was not Zed’s way. Anger was always a possibility; a little mayhem always kept hardware in its place. Panic was for other people, though. Still listening for any sort of sound, he bent over to scrounge through the padded pockets of his laptop bag, coming up with the little vendor LED light he’d obtained at some nameless trade show long forgotten. He shot the beam up to the signage that told him what he already knew: he was at ZZ-99 – the “Zed-zone”. In all his data center designs, the Zed-zone was where he put the cabinets with the sniffers, misleadingly documented cross-connects, and terabyte arrays storing data not allowed to be at rest for very good reasons.

The emergency lighting finally kicked on in the still silent data center. Zed felt a little better about that, but made a mental note to review the failover lighting circuits. The trouble with subbing out in a ground up build was that testing was less rigorous when deadlines loomed. That particular contractor was off Zed’s list.  Zed sighed, tugged his vendor tee down to a more modest length, and headed towards the main egress point, a whole lot of tiles away. While panic wasn’t his style, it was the style of everyone else in this place. There would be explaining to do, right after the data center came back up.

Before Zed made it to the door, the main lighting came back on, and the network core in row Z began to spin up. A few more steps, and he heard the network security tiers start to spin. Then the network access tiers. He paused for a moment and watched a Bacon Networks Strip Stack he happened to be walking by kick over. He smirked at the familiar boot sequence. Pink turned to red, red to green, and the LED then read “~SizlOS~ Ready!” Yeah. Zettabyte transport over Cat5E. The Consortium said it couldn’t be done over UTP, glass, or anything else, because the laws of physics are what they are. Whatever. Suck it San Jose. Zed had just installed it – had architected this world-class data center around it as a matter of fact. Zed was the man. And he knew it.

Rows A-G were spinning up all their rust when the panic met Zed. “ZED!! What just happened in here? You know how much this place cost the taxpayers? How could we have a full power-off scenario? I thought that couldn’t happen! What is going on? Is everything coming back up? I have a room full of operators staring at blank video walls. How much longer are we going to be out? Do you know how bad this is? Seriously, do you?”

Zed hated this pushy little man. At five foot two with a bald pate and abundant moustache, he came about to Zed’s chest. Zed has fantasized more than once about pounding him on his pointy little head until his spine telescoped. The temptation to do exactly that was extraordinarily poignant as the uniformed gnome spastically proclaimed his displeasure. Zed kept his anger in check long enough to emit a reasoned response.

“Colonel Smith, which I know isn’t your real last name but whatever since you guys are all hush-hush about what you do here. Have I reminded you today that I have a top secret clearance? Anyway Colonel, tell me what you were doing exactly 4 minutes before the lights went out.”

Rows H-M fired up, power supplies and exhaust fans belching their pent up heat. Craning his neck, Zed tiptoed and could see a brief shimmer of heat rising from that part of the center. He sighed in frustration and entered Colonel “Smith’s” personal zone, looking straight down at him. “Colonel? I’m waiting.”

Smith backed up a step. “Log review,” he said sheepishly, looking at his suspiciously high-heeled shoes. “Bull,” Zed said quietly, clipping off the Colonel’s last syllable. “I know exactly what you were doing, because your keystrokes all echo to my phone. You truly are a poser. Shall I read what you were typing, or can we agree that you disobeyed the rules of my data center? You went onto a server which I told you to never go to except in an absolute emergency, you started reading through expect scripts you definitely don’t understand, and you accidentally on purpose ran one. Am I close?”

Smith continued his shoe examination. Zed was surprised by the determination in Smith’s voice as he stated, calmly at first, “We can agree that despite your considerable talents, you are too cocky for your own good, Mr. Zilnik. We can agree that of necessity you know more about this facility than you ought. We can agree that ultimately, *I* am responsible for what goes on here, not you. We can agree that I will run any scripts I like, that you are contractually obligated to document your scripts truthfully and completely. We can further agree that my curiousity got the better of me when I read the ONE comment I could find in the last script I ran stating that ‘Without Bacon, Everything Goes Dark.’ Things went a bit darker than I anticipated. Perhaps in the interest of fulfilling your contract, you could re-document that to say ‘This script takes this entire facility hard down. Use only in the most dire of circumstances.’ A thought?”

Smith was no longer examining his shoes. He was looking up at Zed, face having turned red, a vein on his temple visibly pulsing in time to his angrily pounding heart.

Zed did not respond. The rest of the single-letter rows had spun up, and the double-letter rows were now kicking over in sequence. Zed turned from the angry, pointless little man, and walked back towards ZZ-99, almost two tiles at a stride. “Zed, stop by my office on your way out, which needs to be quite shortly,” the Colonel called to the disappearing form. Zed sighed. Paperwork. That nitwit had taken down a data center by going places he ought not, running scripts he ought not, in the context of a superuser he should not have been acting as, but somehow, Zed was to blame.

Arrival back at the Zed-zone took a couple of minutes. His racks, filled with equipment cleverly hidden from accounting and snuck through on sundry purchase orders, was all back up. He scanned the racks up and down, seeing “~SizlOS~ Ready!” on the appropriate screens, power supply lights glowing green. No reds. No ambers. Virtual cluster running fine. Internet 10G line up. All was well. No – wait. Zed scanned the racks again. Something wasn’t right, and he’d missed it because he wasn’t really looking for a problem during the first scan. He was looking to stroke his ego with just how cool it was to be in the Zed-zone. A more purposeful scan revealed the problem.

All of Zed’s secret old-school terabytes of rust were gone. The disk array was empty.