I swear, I didn't touch it! (IT, Engineering, Workplace Automated Disasters)

[k0ala]

Story time! Who else makes a living trying to pick up after other people's bad technical decisions? Pour your grievances out here.

[A. Crow]

I've found that scheduling meetings over lunch to deal with fuckups has cut down on the number of fuckups I've had to deal with. 

[StolenToast]

Story time! Who else makes a living trying to pick up after other people's bad technical decisions? Pour your grievances out here.

My roommate messed up his computer by taking out the hard drive and putting it back in.  He gave it to me to try to fix and it was truly messed up.  I decided to run a chkdsk and it found an error one every single sector of the disk.  I figured this would take a while so I gave it back with a single command:

"Thou Shalt Pluggeth Thine Laptop In and Make Sure it Doth Not Die."

Of course this task was too difficult and I received the laptop again the next morning having gone through about half of a chkdsk.  I attempted to run another, this time keeping it in my room and it completed successfully but the laptop was no better off.  So now I have to do a full re-install which was painful on this old terrible laptop.  Drivers were a bitch too, and in the end I couldn't even get the graphics drivers to work.

that's my little story.

[k0ala]

Last night just before end of shift I got a good one. The guy found out he was in a hole but kept digging anyway. I found half the hard drives had been marked offline by the user, and the other half had been prepped for transport (moving the drives into another enclosure). When I checked the event logs, they had been wiped just 5 minutes prior.

This morning we talked some more. It turns out they'd gotten a filesystem error and decided that "since all the disks were copies of each other in a RAID-5" (no) "we would just force some drives offline" (hell no) "and then when the corrupted files would become readable on the computer" (wat) "then we'd know we had the drive that was uncorrupted and could replace all other drives." (waaaaat)

It worked out for them until the second drive was forced offline and the logical drive fell offline. Then they noticed that there is no corresponding "Force ONline" button for those physical drives. So they decided to turn off the whole RAID box. That didn't fix the pair of red hard drive lights when it was turned back on, so they forced a third drive offline. That didn't do what they wanted, then they hulked out, saw red, and then started clicking buttons randomly. When they were out of options and their RAID box was thoroughly hosed up, then they decided to do the thing they should have done first -- call the technical support staff for which they had already paid.

Filesystem errors are handled by software like Disk Utility, Partition Magic, or Diskwarrior... not by pulling drives on the storage array which lies under the partition.

Now they know, and knowing is half the battle. (The other half is blue lasers and red lasers.)

[FlyingMongoose]

I don't think we should limit this to IT, how about call center horror stories as well? (Back when I worked in one for a cable/satellite company)

Elderly Woman: "Where are my shows?"
(exchange involving pulling up account)
Me: "What shows are you attempting to watch ma'am?"
Elderly Woman: "They're my shows, they're on every night!"
Me: "What stations do you typically watch these shows own?"
She gives me the numbers (luckily I recognize them all as local).
I place her on hold so that I can check her local feeds (and I find severe weather warning broadcasts on all local stations).
I explain to the woman that there's a sever weather advisory for her area and that she should probably take precautions.
Elderly woman: "But I want to watch my shows!"
Me: "Ma'am, there is a severe weather warning for your area, I can not change the feed coming from your local stations as they are required to do this federally and are providing you information from the national weather service."
Elderly woman: "So you mean you can't fix my problem?"
Me: "Ma'am, no, this has to do with severe weather in your area, the national weather service, and your local stations, you're welcome to attempt to contact your local station, but I doubt it will resolve this as they can not change the weather."
*click*

[Eschatos]

Not me, but the worst call I ever witnessed at work was an account password reset that took an hour and forty minutes due to a massive language barrier.

[Goffin]

There's a sucker born every minute and that saying still holds true for anybody working IT.   

[Dtrain323i]

[k0ala]

A couple of my work stories in rage-comic format:

[FlyingMongoose]

Oh this one happens way more often than you'd like to think it happens in the world;

Customer: "I don't have a picture, the lights on the box are on but I don't have a picture"
Us: "Is your TV plugged in and on or possibly even on an outlet that's on a light switch?"
Customer: "Of course it's... oh..." *click*
(Most of the time they would get really pissed at us thinking it was our intent to insult them, when it's not it just happens so damn much).

Company I worked for only had I'd say a few thousand customers (3-5k), possibly even less. Once a day at minimum 1 call like this. We have 7 billion people on the planet, assuming 3k customers we had and lets say 2 on average per day. Assuming every person on the planet had a cable box, 4.67 million of those people forget to turn on their tv's, or that there is no power going to the tv per day.

[Evil Cheese]

Company I worked for only had I'd say a few thousand customers (3-5k), possibly even less. Once a day at minimum 1 call like this. We have 7 billion people on the planet, assuming 3k customers we had and lets say 2 on average per day. Assuming every person on the planet had a cable box, 4.67 million of those people forget to turn on their tv's, or that there is no power going to the tv per day.

That's assuming you're not getting any repeat calls from the same person/household. Troubleshooting rule no. 1: always start at the physical layer.

[k0ala]

[k0ala]

[rumbot]

As someone who's had to coordinate downstream launches between user groups that range between 1k, 25k, and 400k, I can attest that we also have our own rage-worthy moments.

As they say, "Only wet babies like change."

Recently, we fielded an inquiry as to how many power strips would be needed. Obviously, this was a function of their N outlets and their N client machines and available N slots in commodity power strips.

[CaffeinePowered]

http://imgur.com/yClu2Ak