Fri - Feb 10, 2006 : 09:49 pm
Oh... My... Goodness...
I can't think of where to begin, so I'll just start typing.
Two days ago we decided it was time to cut the umbilical for the other guys who were the old I.T. team for the company I work for. We did so by changing all passwords (or so I thought), and that's about it. We thought we had em'.... So.... We all went home and went to bed.
Boy were we in for a surprise.
I monitored the logs until about 1:00am yesterday morning, I left the logs going and when I woke up, I found that the server had been rebooted, and the password for root had been changed.
This was..... BAD.
To tell ya the obvious, I was scared and I didn't really want to tell my boss, but I knew I had to - so I did. We all ended up at work scratching our heads wondering what to do.
Well... This is about the same time all the experience I had with Gentoo started to kick in. I remembered the chrooting process that is required to get into the new environment. I started wondering if there was a way to do that here....
To make a long story short, I found out that there was a rescue system for the server we were renting in New York. With a little help from their "2nd level techs" and many prayers said in a short period of time, we succeeded in chrooting into the system, changing the root password and logging back in.
How do I spell relief? K-E-E-P-I-N-G M-Y J-O-B.
heh.... I don't think they would have fired me, but the alternative was *much* more difficult - which was to transfer *everything* to our in-house servers. Talk about working through weekends. I don't even wanna think about it.
Anyway... We got the server back, all the passwords changed (they had got us through an ssh account they had which cloned the root user... I'm not sure how they created it, but I'm going to find out soon... *tee hee*).
The best part of all of this was the fact that the guy who rooted us was the head of the old I.T. team. When he found out he was being "let go", he basically gave us the finger and told us he had "complete control" of the systems. At around the time he was saying this, we were changing the last of the passwords, setting up a firewall which blocked out the entire continent of the old I.T. team (guess where that was), and a couple of other ones we know he was using.
I can only imagine the look on his face when he left saying he had "complete control" and arriving home to find out he had absolutely no way to get back into the box.
We monitored the logs all night last night. My watch was from 4:00am to 8:00am. No activity at all. I guess the firewall worked.
Today I set up a monitoring system for the logs which calls a cell phone everytime someone successfully logs in to the box. I'm glad I don't have the phone because I log in a lot. ;-)
Anyway.... What a week, I tell ya. Nutso.
I'm actually having a grand ol' time at this new job. I feel extremely productive, and very needed. It felt good to be a key player in getting through a potentially *very* sticky situation.
Linux came through again... and I'm seriously considering to push the fact that I really want Gentoo on our Linux boxes in the office in the next meeting we have. I feel *much* more comfortable using Gentoo than anything else. We'll see how it goes.
Talk to ya'll later.