- My Forums
- Tiger Rant
- LSU Recruiting
- SEC Rant
- Saints Talk
- Pelicans Talk
- More Sports Board
- Fantasy Sports
- Golf Board
- Soccer Board
- O-T Lounge
- Tech Board
- Home/Garden Board
- Outdoor Board
- Health/Fitness Board
- Movie/TV Board
- Book Board
- Music Board
- Political Talk
- Money Talk
- Fark Board
- Gaming Board
- Travel Board
- Food/Drink Board
- Ticket Exchange
- TD Help Board
Customize My Forums- View All Forums
- Show Left Links
- Topic Sort Options
- Trending Topics
- Recent Topics
- Active Topics
Started By
Message
What's your biggest "oh...shite" moment at work as an IT Pro?
Posted on 5/28/14 at 8:08 am
Posted on 5/28/14 at 8:08 am
Worst one for me was many, many moons ago. We'd renovated our (really small) datacenter, and when stuff got moved back in the power wasn't split between the two sides of the UPS on all devices. One of those devices was a big, nasty HP SCSI DAS that was attached to the main file server, and contained all of the CAD stuff for the company. A few weeks later, I decided to clean everything up, making sure that every device was split between two PDUs.
Long story short: I found out that the DAS wasn't capable of running on only one power supply. Oops.
After 3 pants shitting hours and, a bunch of BSoDs and errors, I had the server back up, and Windows happy again.
In the grand scheme of things, it wasn't a gigantic deal, and I've given myself plenty of opportunities to break even more stuff since then, but it was still my first real screw up.
Long story short: I found out that the DAS wasn't capable of running on only one power supply. Oops.
After 3 pants shitting hours and, a bunch of BSoDs and errors, I had the server back up, and Windows happy again.
In the grand scheme of things, it wasn't a gigantic deal, and I've given myself plenty of opportunities to break even more stuff since then, but it was still my first real screw up.
11
Posted on 5/28/14 at 8:18 am to Schwartz
Recently - getting nailed by Cryptolocker. I was able to recover most everything, but there were some obscure things we did lose.
Ever - one time I discovered that disks were beginning to fail on an older Exchange server. There was a waiting period for a replacement server to come in, and every minute of every day my butthole was puckering, even though I was taking nightly backups just in case. Once it came in, I did the install and moved all the mailboxes between 5:00 PM and about 3:00 AM.
Ever - one time I discovered that disks were beginning to fail on an older Exchange server. There was a waiting period for a replacement server to come in, and every minute of every day my butthole was puckering, even though I was taking nightly backups just in case. Once it came in, I did the install and moved all the mailboxes between 5:00 PM and about 3:00 AM.
Posted on 5/28/14 at 8:28 am to jdd48
Not in IT but I feel bad for the IT crew at my job. So many dumb questions go their way.
Posted on 5/28/14 at 8:31 am to goldennugget
When I deleted a code repository for a web development company by accident and found the owners had no backups. rm -rf can be a nasty little beast if used improperly.
Posted on 5/28/14 at 8:34 am to Schwartz
I don't recall one personally but where I used to work there were a couple of famous ones. A EE asked for a diagram for the power grid on the main system floor one day and after a week of asking and nobody producing it. He took it upon himself to go to the power box and cycle on and off the breakers. So there were multiple HPC systems going up and down.. nice.
Same site. Had a cleaning lady open the window on a data storage silo to clean the inside of the glass, This brought the robot to a such a complete stop that it threw a disc across the silo.
Same site. Had a cleaning lady open the window on a data storage silo to clean the inside of the glass, This brought the robot to a such a complete stop that it threw a disc across the silo.
This post was edited on 5/28/14 at 8:36 am
Posted on 5/28/14 at 8:39 am to Schwartz
I'm a digital hoarder so I haven't had any major data loss except for maybe deleting contacts from some VIPs iPhone by syncing it to Outlook.
My biggest 'oops', would be this one time when I took down my whole network after hours while rearranging my server room. I worked through the evening and night trying to fix a connectivity issue myself. I ended up having to get someone in to repair a fiber connection that I broke. It was a 24 hour straight work 'day'. I was pretty delirious when it was over.
My biggest 'oops', would be this one time when I took down my whole network after hours while rearranging my server room. I worked through the evening and night trying to fix a connectivity issue myself. I ended up having to get someone in to repair a fiber connection that I broke. It was a 24 hour straight work 'day'. I was pretty delirious when it was over.
Posted on 5/28/14 at 8:41 am to Schwartz
Electricity tripped the breaker on the AC unit for the server room. By the time I got there it was 120 degrees in that room, and half the hard drives in our SAN were dead.
Posted on 5/28/14 at 8:47 am to Schwartz
I was in hurry to get out one day and we had a database (with no backup) that was a physical server. It was to be upgraded by our DB guy that night, so in order "backup" I had to pop out a couple drives so they didn't mirrored over, in case it needed to be rolled back. Well....I popped the wrong mirrored set and shut that bad boy right down. After dicking around for an hour trying to recover, I call HP and asked what my next course of action was. He basically said I was screwed. I figured if the server had to be rebuilt anyway, I'd give the old smartstart recovery disc a try. Fortunately, it worked and it was like nothing ever happened. I don't do hot swaps anymore and make sure to shut the server down first 
Posted on 5/28/14 at 8:54 am to Schwartz
I am pretty locked down but in my early years I dropped a production SQL table. Took a while to locate the backup and restore so the client was down for about 4 hours. Capital One was the client
Posted on 5/28/14 at 9:39 am to Schwartz
Accidentally caused the front door firewalls in a 900,000 sq/ft data center to go active/active(they were only designed for active/standby). I may have literally shite my pants a little bit.
Posted on 5/28/14 at 9:58 am to LSUDropout
I haven't really had one of those moments yet, or else I've blocked the memory (14 years in IT, I am betting it's the latter)
Our Sys Admin sent out an agency-wide email about something (can't remember what) a few years ago. This email went to everyone in the state (about 1,000 folks). There was some mistake in it and our resident funny-guy/smartass decided to respond with "DEEEERRRRRRRRRRRRRRRRRRRRRRRPPPPPPPPPPPPPP!!!!!!!" in huge font. He replied to all. I was just getting up to go across the hall to make fun of him when I heard him do the Darth Vader scream.
We had a student worker reimage someone's computer without backing it up. They were fairly high up the food chain so our entire shop looked bad for a while.
(14 years in IT, I am betting it's the latter) Our Sys Admin sent out an agency-wide email about something (can't remember what) a few years ago. This email went to everyone in the state (about 1,000 folks). There was some mistake in it and our resident funny-guy/smartass decided to respond with "DEEEERRRRRRRRRRRRRRRRRRRRRRRPPPPPPPPPPPPPP!!!!!!!" in huge font. He replied to all. I was just getting up to go across the hall to make fun of him when I heard him do the Darth Vader scream.
We had a student worker reimage someone's computer without backing it up. They were fairly high up the food chain so our entire shop looked bad for a while.
Posted on 5/28/14 at 10:04 am to Bard
I loled at the derp one. Good stuff.
Posted on 5/28/14 at 11:54 am to Schwartz
Update statement without a where clause
Posted on 5/28/14 at 12:05 pm to Schwartz
I didn't do this, but we had a consultant installing and configuring Tivoli Identity Management. He ended up running something that started deleting everybody's windows accounts. Needless to say a ton of people were locked out of their computers for several hours.
Posted on 5/28/14 at 12:08 pm to jefforize
quote:
Update statement without a where clause
Been there.
Posted on 5/28/14 at 12:34 pm to foshizzle
with 1 mouse click I moved about 200+ production servers to a different VLAN and they all went down. That was fun.
Posted on 5/28/14 at 12:48 pm to jefforize
quote:
Update statement without a where clause
347000 row(s) affected
Posted on 5/28/14 at 12:49 pm to Schwartz
updating a customer's database
DELETE * FROM Items
I forgot my WHERE statement and just hit f5 to run like a moron instead of testing it like I did 99% of the time. I freaked the frick out.
DELETE * FROM Items
I forgot my WHERE statement and just hit f5 to run like a moron instead of testing it like I did 99% of the time. I freaked the frick out.
Posted on 5/28/14 at 1:07 pm to Schwartz
rm -rf *
Page 1 of 2
Popular
Back to top
Follow TigerDroppings for LSU Football News