It's 1am, I'm still at work, I'm trapped in New York by a fucking blizzard, but the backups are damn well running again. Fuck you, symantec. Fuck you right in the open ports.
I would guess that the reason he was stuck in New York until 1AM and subsequently got trapped by a blizzard would have something to do with some difficulty encountered in Symantec's shite software. Of course, I have certain advantages of prognostication in this regard.
This week, for the second time in my career (at two different banks), I got to watch the emergency comms flying around while people try to frantically stop a tape robot from attacking a trapped Sun engineer.
The chat room log (from memory) was just unbelievable:
i need someone to disable the tape robot asap who knows the login to the robot console? call oscar no, he's at the dentist anyone? PLEASE JUST PULL THE PLUG oh my god pulling the plug won't work, it's on redundant UPSes right under the raised floor YOU HAVE GOT TO BE KIDDING ME call our sun account manager, we need another engineer right away okay What about a circuit breaker? i can hear him screaming from here, is there nothing else we can do the account manager wants to know if we called the desk for a ticket number, what was it from this morning, maybe we can just escalate the current ticket Tell the account manager we want two engineers incase this one dies oh okay he's sending someone now is there financial impact, or is this sev3?
I feel obligated to mention that I was in a different country.
I'm trying to imagine how this works, and no matter what scenario I concoct in my head, it's hilarious.
What was the tape robot doing, trying to insert a new backup tape into the tech? Or, um, trying to extract a used one?
How did the tech get trapped? I'm alternating between imagining him cornered by R2D2 with a spindly robot arm wielding a cartridge menacingly, and stuck head-and-shoulders in a server rack while a more realistic tape system keeps jabbing him in the ass.
Symantec Endpoint Protection has a particularly shitty way of going bad that I have to deal with at least once a week. Basically, srtspl.sys/srtspl64.sys shits itself and causes BSODs on boot. To kill it you have to boot to safe mode and disable SEP in services, then boot back to regular Windows to uninstall it because it fucking uses the Windows installer service, which won't run in safe mode.
This was BackupExec, which, somewhere between renaming the service account and changing the password (which I WAS NOT told about and dammit that was annoying) and demoting the secondary DC and removing it, it decided it didn't ever need to talk to the SQL server ever again.
(no subject)
Date: 2011-03-10 08:24 am (UTC)(no subject)
Date: 2011-03-10 09:13 am (UTC)(no subject)
Date: 2011-03-10 03:42 pm (UTC)(no subject)
Date: 2011-03-10 03:50 pm (UTC)(no subject)
Date: 2011-03-10 03:55 pm (UTC)(But other people paying for stuff is okay.)
(no subject)
Date: 2011-03-10 04:12 pm (UTC)(no subject)
Date: 2011-03-10 09:45 am (UTC)Please, tell us what you *really* think. And, remember, don't hold back any.
(no subject)
Date: 2011-03-10 11:47 am (UTC)(no subject)
Date: 2011-03-10 11:56 am (UTC)(no subject)
Date: 2011-03-11 03:58 am (UTC)The chat room log (from memory) was just unbelievable:
i need someone to disable the tape robot asap
who knows the login to the robot console?
call oscar
no, he's at the dentist
anyone?
PLEASE JUST PULL THE PLUG
oh my god
pulling the plug won't work, it's on redundant UPSes right under the raised floor
YOU HAVE GOT TO BE KIDDING ME
call our sun account manager, we need another engineer right away
okay
What about a circuit breaker?
i can hear him screaming from here, is there nothing else we can do
the account manager wants to know if we called the desk for a ticket number, what was it from this morning, maybe we can just escalate the current ticket
Tell the account manager we want two engineers incase this one dies
oh okay he's sending someone now
is there financial impact, or is this sev3?
I feel obligated to mention that I was in a different country.
(no subject)
Date: 2011-03-11 05:48 pm (UTC)What was the tape robot doing, trying to insert a new backup tape into the tech? Or, um, trying to extract a used one?
How did the tech get trapped? I'm alternating between imagining him cornered by R2D2 with a spindly robot arm wielding a cartridge menacingly, and stuck head-and-shoulders in a server rack while a more realistic tape system keeps jabbing him in the ass.
(no subject)
Date: 2011-03-11 05:53 pm (UTC)(no subject)
Date: 2011-03-11 10:10 pm (UTC)(no subject)
Date: 2011-03-11 04:15 am (UTC)Pain in the ass.
(no subject)
Date: 2011-03-11 05:03 am (UTC)(no subject)
Date: 2011-03-11 05:21 am (UTC)