Hot Diggety! Tom Perrine was rumored to have written:
> On Tue, Jan 24, 2012 at 12:08 PM, Dan Schlitt <[email protected]> wrote:
> >
> > Thanks for all the suggestions. Reinstalling didn't seem to change
> > anything, To take care of the file I just pointed it to /devnull.
>
> Did you do a full system re-install, or just the ssh package?
>
> If you haven't done the full system re-install, you really need to go
> that route.
Hey Dan,
Tom speaks the absolute truth. From what you've described so
far, you've been compromised badly.
There are other innocuous possibilities -- failing CPU, failing
CPU fan, failing RAM, etc... but from what you've written, it has
'serious compromise' painted all over it.
From experience, first order of business:
1) Remove the system from the network -- minimally, at least
unplug the network cable(s). This preserves the running
environment -- better odds of catching trojaned tools.
2) If you are in a position to do so, run security diagnosis
scans (all kinds exists, often from CDs)
3) Regardless of how #2 goes, remove at least one HD with a copy
of data and image it to a disk file, so you can examine it in
depth later with compromise data relatively untouched.
4) In the meantime, to get back up, once at least #3 has been
done, do a total from-scratch OS install and be very, very
careful about what you restore from backups. Only restore
_data_, not binaries or libraries!
With such a situation, running usual tools (ps, etc.) will often mask
out the actual attacking tool due to having been replaced. Or the attack
tool will be a standard daemon (e.g. sshd) that was modified yet shows
up innocuously in a ps listing.
There are special security tools and techniques available to investigate
this kind of situation.
Right now, you know you have a serious issue. You also know you've been
compromised, but you don't yet know *how* -- or the full extent of the
compromise. Thus, merely reinstalling ssh alone at this point is a dead
end.
Nor do you know the _full_ extent of the compromise...
You just know you're sick, but until you know how you got sick, you
won't ever get better because you won't know how to target the actual
root cause.
My first SA job, we avoided doing a full machine reinstall since it was
a lot of work -- but we spent 6 months chasing the attacker since he'd
helpfully deposited so many backdoors after the initial compromise.
Stupid idea to wait for so long in hindsight, but I honestly didn't know
any better back then. Subsequently, at another place, use of tripwire
and other special tools quickly helped us zero on things.
So... the first job... 6 months later, we finally agreed to reinstall
the machine from scratch. Boy, was that fun... NOT! But it stopped the
attacker cold, and we were much more diligent in blocking various attack
vectors (including staying much more up to date on patching).
Merely replacing binaries the attacker swapped out was just a game of
playing Whack-A-Mole. Ultimately fruitless and a total waste of time.
The real importance was to figure out *HOW* he originally got in.
What would also have had helped us was with installation of tripwire (or
equivalent tool -- Samhain, etc) at OS installation time for a baseline
comparison. So between being vigilant and some intelligent forethought
in planning ahead, you're talking about a much smoother life _when_
attacks resulting in compromise do occur -- and not if. When.
-Some other Dan
P.S. I eventually figured out the real life identity of the attacker. He
had the gall to apply at my second job (which also had my boss from the
first job and remembered the nasty situation).
He didn't get hired. (No, I wasn't consulted in the hiring decision, but
my boss remembered the whole thing and, well... his philosophy was to
hire people with a track record of contributing rather than destructing.)
_______________________________________________
Tech mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
