As I near the end of a fairly quiet week with the stinking on-call pager, I just got paged on an incident that’s not my problem. Our NOC is there 24×7, staring into an array of consoles, dutifully ready to page someone if something goes awry. There are all kinds of groups in the pager barrel; MonitoringGroupOne, MonitoringGroupTwo, DatabaseSupport, WebSupport, LANSupport, NetworkSupport, UnixSupport; the list goes on and on. For every “Support” group, there’s an “Engineering” group that feels the pain from time to time as well.

The NOC’s primary function is to get some group, any group to respond to a page, so they can update the incident notes, and go back to staring into their consoles. And so it just went, after I answered the last page:

NOC: MegaIronDatabaseMachine001 is down.

MegaIronDatabaseMachine001 is a biggie; I didn’t know I had so many sphincters that could tighten simultaneously.

Me: It’s down?

NOC: No, the database is down.

Observation: The database services being provided by MegaIronDatabaseMachine001 are its only reason for existence; this is still a huge problem.

Me: Is MegaIronDatabaseMachine001 pingable?

NOC: I don’t know; the ForeignBornDBA0023 just called us. You want me to conference him in?

Me: OK, might as well.

ForeignBornDBA0023: MonitoringSystem001 is not running. We need to have the process started.

Me: Tell me how to restart it, and I’ll log on and fire it up.

ForeignBornDBA0023: MonitoringSystem001 is operated by MonitoringGroupOne.

Me: I’d call MonitoringGroupOne,then.

NOC: We already paged MonitoringGroupOne; they didn’t call back.

I wanted to ask whom they would have paged if I hadn’t called back, WindowsSupport? Such snark seemed out of place.

Me: If that process is managed by MonitoringGroupOne, I’d raise someone from there.

NOC: OK, we’ll try them again. We’ll let you know how it turns out.

Just another day aboard the Full Employment Express. On the bright side, tomorrow I hand off the stinking on-call pager to the next poor sod in the rotation.

-k-