It’s the 10^th annual System Administrator Appreciation Day. My company is taking us to lunch tomorrow at Famous Dave’s, who is celebrating their 15^th year of Smokin’ Hot Ribs & Ice Cold Beer. My company is nice like that.

Unfortunately, this being a lunch and all, the Smokin’ Hot Ribs will have to suffice for tomorrow’s event. My company is also pretty savvy; with our current crew, they realize the cost savings of ribs over beer.

Still, I’m looking forward to an afternoon of in-cubicle loginess tomorrow afternoon. Pork Fat Rules!

-k-

One of my co-workers¹ is evidently on his annual two-week sabbatical to North Carolina’s Outer Banks. I say this, based on a careful analysis of my server access logs and my Sitemeter reports², which reveal a sharp downturn in accesses from his home 20, and an elevation in hits from around the OBX.

And I hope he has a blast. My only regret was that I didn’t get in my order for him to mule me back some Pigmans Bar-B-Que sauce. MLB just checked the larder, and we’re well-stocked. While the Pigman’s online store still consists of a call us for orders page, we can still keep our sauce supply intact.

So everything is right with the world. When my boss arrives back one week hence, I need to put on my Chief Morale Officer hat, and welcome him back in the proper manner.

-k-

No matter how big of a goat screw it is at work, and it’s approaching the three ring variety, there’s one thing they can’t take away.

Today, I passed the stinking on-call pager off to the next poor sap.

I’m enjoying the extra spring in my step already.

-k-

I just got back from the nearby Safeway. They have a branch bank on-premise; I went to get some cash from the ATM, and to deposit a paper check. Their cashier windows are open on Sunday afternoon, a convenience of modern times. I’m old school enough that I don’t trust ATMs to make such deposits; no sir, endorse the check, fill out a deposit slip, and hand it to the human behind the counter. That’s the ticket.

I clipped the stinking on-call pager to the waistband of my unbelted chinos before I left the house. As I was walking from the car to the store, I passed over a large, grate-covered storm drain. The grate openings appeared to be large enough that if the stinking on-call pager were to become dislodged from my waistband and take a judicious trajectory, that it could easily be at the bottom of that drain; with rain in the forecast, it could well be washed away to the bottom of a river, or better yet, out into the Atlantic, never to be heard from again. I’m not exactly skinny; part of my waistline spare tire could easily dislodge the electronic jailkeeper,and send it to the fate it so richly deserves. Alas, it wasn’t to happen today.

We have a member of our group who has lost the stinking on-call pager; he was running errands, got home, and found that the so-cp was nowhere to be found. He retraced his steps, even calling the infernal device from his cell phone at appropriate times. No soap. This guy is an excellent sysadmin; if you have any cluster software or Veritas filesystems in need of tweakage, he’s your man. If you’ve guzzled the Oracle RAQ kool-aid, he can help you out there. So, in spite of all his techincal acumen, this guy has never stood taller in my mind than on the day he lost the stinking on-call pager. We had a replacement so-cp by lunchtime; so the victory was short-lived. On a visceral level, though, this was a victory for the common man. Nothing like putting the so-cp into the Atlantic would have been, but a victory nonetheless.

-k-

As I near the end of a fairly quiet week with the stinking on-call pager, I just got paged on an incident that’s not my problem. Our NOC is there 24×7, staring into an array of consoles, dutifully ready to page someone if something goes awry. There are all kinds of groups in the pager barrel; MonitoringGroupOne, MonitoringGroupTwo, DatabaseSupport, WebSupport, LANSupport, NetworkSupport, UnixSupport; the list goes on and on. For every “Support” group, there’s an “Engineering” group that feels the pain from time to time as well.

The NOC’s primary function is to get some group, any group to respond to a page, so they can update the incident notes, and go back to staring into their consoles. And so it just went, after I answered the last page:

NOC: MegaIronDatabaseMachine001 is down.

MegaIronDatabaseMachine001 is a biggie; I didn’t know I had so many sphincters that could tighten simultaneously.

Me: It’s down?

NOC: No, the database is down.

Observation: The database services being provided by MegaIronDatabaseMachine001 are its only reason for existence; this is still a huge problem.

Me: Is MegaIronDatabaseMachine001 pingable?

NOC: I don’t know; the ForeignBornDBA0023 just called us. You want me to conference him in?

Me: OK, might as well.

ForeignBornDBA0023: MonitoringSystem001 is not running. We need to have the process started.

Me: Tell me how to restart it, and I’ll log on and fire it up.

ForeignBornDBA0023: MonitoringSystem001 is operated by MonitoringGroupOne.

Me: I’d call MonitoringGroupOne,then.

NOC: We already paged MonitoringGroupOne; they didn’t call back.

I wanted to ask whom they would have paged if I hadn’t called back, WindowsSupport? Such snark seemed out of place.

Me: If that process is managed by MonitoringGroupOne, I’d raise someone from there.

NOC: OK, we’ll try them again. We’ll let you know how it turns out.

Just another day aboard the Full Employment Express. On the bright side, tomorrow I hand off the stinking on-call pager to the next poor sod in the rotation.

-k-