On 5/13/2011 6:54 PM, Tom Perrine wrote:
> So, what's your most memorable command line typo, "think-o",
> "brain-fart", or "#$$@#$*&@#$" moment?

That would be the one that almost cost me my job.

Seven months or so after being hired, I had a customer who wanted a 
particular software package installed on his Windows server.  I 
diligently went through and installed the package only to find out that 
the DLL wasn't being read by IIS even after registration.  Somehow, I 
tracked down that the issue had something to do with the IIS identity. 
So, following directions in the documentation, I changed the password 
for IUSR -- bringing down IIS.  I then tried to change the password used 
by IIS but that didn't work.  Sick to my stomach, I panicked.  I found 
that if I changed the password for each individual site, it would work 
so I started that.  There were a lot of sites on this server and the 
customer called wondering why their sites weren't working correctly... 
I tried the global password change again and then everything worked.

Only the fact that I had documentation saying to do this saved me from 
the unemployment line.

After that point, I was forbidden to install software on that server again.


Close runners up are:

* I once had to fix a Linux system where / had been chmod'd to 0 because 
a script didn't properly check its inputs.  root could do everything but 
nothing else would work.  Fixing it was easy.  Finding the cause of the 
problem was hard.

* I was remotely working on a Cisco router.  I was in config mode inside 
the external interface and wanted to see the current running config.

In that version of IOS, "sh run" when inside the interface means 
"shutdown the interface".  One phone call to the data center techs later...

* I once inherited a system left behind by a co-worker who quit.  I was 
told he had backup and RAID monitoring set up on this system.  Since I 
trust my co-workers, I took him at his word and never checked myself.

Several years later, the system dies.  The first drive in the RAID 1 
array has a lot of bad sectors and isn't bootable anymore.  The second 
drive has apparently been long dead.  So much for monitoring.

I mount the first drive in another system and look around.  The 
partition for backups is empty.  So much for backups.

Since this was a production system, I then had the... pleasure of using 
dd to pull the data off of this drive, carefully working around all of 
the bad sectors.  By some miracle, this actually worked and I was able 
to put the system back online in a few days.

I hope to never do that again.


--CAE
_______________________________________________
Discuss mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Reply via email to