Dead machine count from Saturday up to 6.

[theweaselking]

"Uh, dude? 'Just stick that drive in a different machine and recover the data' may be a LITTLE harder than anticipated."

Comments

[thornae.livejournal.com]

Is this one of those "non-critical machines don't need UPSs, but we're going to leave them powered up over the weekend anyway" deals?
Because if so, my commiserations - I had much the same experience last month. Fortunately, it only fried 2 power supplies - although, one of those was attached to a UPS, which is how we learned that the UPS was faulty too.

I have explained the importance of powering down at hometime, but those extra couple of clicks at the end of the day are effort, y'know...


Also, I deal with a lot of ribbon cables in my job, and the only times I've seen that happen before are when corrosion has rust-welded the pins to the connector. So yeah - impressive.

[theweaselking.livejournal.com]

Of the machines that failed, 4 were powered off the entire time, one was a dumb-as-hell 100M workgroup switch, and one was remotely powered on by an overeager admin[1] when power and internet came back up 1.5 hours through the scheduled 3 hour outage and he didn't think to question *WHY* power was on so early or whether work was actually complete, which meant that machine was *on* when the power went out again. The UPS should have helped, but still, 2 hours later everything came back up and that server had lost a power supply.

(So that machine wasn't actually "dead" since only 50% of it's power failed and it can run on 50%. On the other hand, a cleanly shut down Xen hypervisor is MUCH easier to bring back up than one that had the power yanked again after booting)

Essentially, what should have been a 45 minute Saturday turned into *hours* due to human error, and then there were a really wacky number of unexpected hardware failures in the more elderly kit.

[1]: Who was not me, for the record.

[thornae.livejournal.com]

Ah, one of those days. Perhaps the chaos gods of IT are punishing you for that "Why isn't it Friday yet?" post...

And honestly, who spends their Saturday going "Ooh, I wonder if the workplace scheduled outage finished early? Let me just check."...?

The powered-off failures are interesting. Power surges, you reckon, or just bad luck?
Anyway, I think I'll make sure that if I'm ever in such a situation, the protocol will be "Power off and cords yanked, and I'll come in and plug the critical boxes back in when they're finished."

Valuable lessons through the misfortune of others - there's probably a word for it in German.

[theweaselking.livejournal.com]

No idea if it was power surges or just crap luck. But I think power surges.

(and it wasn't "check if the outage ended early", it was "the UPSen texted us to tell us power was back on, just like they texted us to tell us it was down". I wisely ignored the text as "probably a fluke, not really relevant, I'm sure as hell not doing this early.")