Re: [Full-disclosure] one of my servers has been compromized

[Dan Ballance]

I'm no expert, but here's something to get you started while you wait for
more experienced replies. Check for root kits:

sudo apt-get install rkhunter
sudo rkhunter --update
sudo rkhunter --check

On 5 December 2011 10:44, Lucio Crusca  wrote:

> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I
> never
> believed it could happen to me until it actually happened: they compromized
> one of my servers. It's a Ubuntu 10.04 server with all security patches
> regularly applied. I'm inclined to believe they used some hole in the web
> application, which is a old customized Virtuemart version (1.1.3), which is
> not upgradable because of the invasive code customizations (I'm not the
> author of that code, so I have no clue about what had been changed back
> then).
>
> Now the problem for me is to track down the security hole. Here is the
> email
> my provider received and forwarded to me:
>
> > Subject: ISP Report; botnet activity on irc.undernet.org
> > [...]
> >
> > Hello, I am an operator on the irc chat network,
> > irc.undernet.org and i would like you to investigate the
> > owner of the Ip addresses that are listed at the foot of this
> > email.
> >
> > This/These host(s) have likely been compromised, and had an
> > altered/rogue process installed on it, and was part of a botnet
> > that was found on our network.
> >
> > The exploit or compromise running on this system is likely
> > to be an irc bot. Can you please alert the person who is
> > responsible, for its security to patch/upgrade, remove the
> > irc process and secure their system.
> >
> > = Unix System owners =
> > A favourite place for hiding the bot(s) is in tmp
> > and in /var/tmp/ or /dev/shm/ or in a users home directory
> > sometimes it may be hidden like /tmp/".  ."/ or similar.
> >
> > The bot files can usually be found by running these one line
> > commands as the root user.
> >
> > find / -exec grep -l "undernet" {} +
> > find / -exec grep -l "sybnc" {} +
> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> >
> > netstat -tanp
> > lsof -i tcp:
> >
> > *netstat looking for connections to remote port 6667 or the
> > range of ports between 6660-7000 once you find the port you
> > can use the command, lsof -i tcp:portnumber to determine
> > which process/user it is running under, and terminate it.
> >
> > = Windows System Owners =
> > most windows bots are mIRC scripted bots and generally
> > need a file called mirc.ini to run, you should search for
> > this file. Run a good antivirus scanner and firewall.
> >
> > This Ip/host may be removed from our Irc network due to the
> > risks it presents to our users.
> >
> > Should you need any help with removing the files or bot
> > process, feel free to contact me by mail or on our network,
> > which you connect to using any irc client and issuing
> > /server irc.undernet.org
> >
> > I look forward to your reply
> > Scot
> >
> > * Affected host/IPs, capture time is GMT+1: United kingdom
> > and servers they were connected to.
> >
> > Please note: when resolving server names to IP Addresses
> > that all our servers end with .undernet.org (for example)
> > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
> >
> > Important: If you reply to this mail needing further
> > information, please leave this mail intact, or supply us
> > with the IP Address(es) in question, as we reference these
> > mails by the unique IP Address
> >
> > Time of Capture: DECEMBER 3, 2011 10:03:48 PM
> >
> > List of IP address(es) and server it connected to:
> > my.server.ip.address (CHICAGO.IL.US
> >
> > BUDAPEST.HU.EU
> >
> > MONTREAL.QC.CA.undernet.org)
> >
>
> I've run the "find" commands and found a number of file with the first
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl
> processes that where trying to connect to port 6959 (only trying because
> I've now set up iptables so that they actually can't), but those processes
> kept spawning. Checked crontab of www-data, found the launcher, removed it.
>
> Now the problem is: how do I pervent further abuse? What should I search in
> the logs (if anything) to spot the security hole?
>
> TIA
> Lucio.
>
>
>
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

Dan Ballance wrote:

> I'm no expert, but here's something to get you started while you wait for
> more experienced replies. Check for root kits:
> 
> sudo apt-get install rkhunter
> sudo rkhunter --update
> sudo rkhunter --check

Thanks for the tip, I've ran those but the check found nothing. I keep 
waiting for the experts...

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

If it was a rootkit then trying to run the outdated rkhunter would be a
moot point. Whatever seizes the kernel first wins, hands down.

Fortunately for him, since the bot was so easy to find in the first place
and such a simple way of maintaining it, the box was clearly seized by
someone who didn't give a rats ass about it. Probably a skiddie or an
automated attack to begin with.

As for plugging any security holes, check your httpd error logs. If you
noted down the time of the bot files creation date you would look around
the same time for suspicious log entries. If they were as careless in
scrubbing the logs as they were holding the box it would give you a look
into how it may have been compromised. If you're getting things like
../.../../../../etc/passwd then some sort of lfi vuln was likely exploited,
start grepping your php files for stuff like include(), or if you're
getting something like into outfile then check your mysql user permissions
and don't let it have file perms, and then start grepping down for sql
vulns.

If it comes down to being too much of a hassle to get all the obvious vulns
at least then go to your boss, admit there is an issue and that time needs
to be taken to remove such legacy code as this could have been a far worse
incident if it had been more targetted and the end goal wasn't a botnet.
On Dec 5, 2011 3:02 AM, "Dan Ballance"  wrote:

> I'm no expert, but here's something to get you started while you wait for
> more experienced replies. Check for root kits:
>
> sudo apt-get install rkhunter
> sudo rkhunter --update
> sudo rkhunter --check
>
> On 5 December 2011 10:44, Lucio Crusca  wrote:
>
>> Hello *,
>>
>> I'm not new here, but I've mostly lurked all the time through gmane. I
>> never
>> believed it could happen to me until it actually happened: they
>> compromized
>> one of my servers. It's a Ubuntu 10.04 server with all security patches
>> regularly applied. I'm inclined to believe they used some hole in the web
>> application, which is a old customized Virtuemart version (1.1.3), which
>> is
>> not upgradable because of the invasive code customizations (I'm not the
>> author of that code, so I have no clue about what had been changed back
>> then).
>>
>> Now the problem for me is to track down the security hole. Here is the
>> email
>> my provider received and forwarded to me:
>>
>> > Subject: ISP Report; botnet activity on irc.undernet.org
>> > [...]
>> >
>> > Hello, I am an operator on the irc chat network,
>> > irc.undernet.org and i would like you to investigate the
>> > owner of the Ip addresses that are listed at the foot of this
>> > email.
>> >
>> > This/These host(s) have likely been compromised, and had an
>> > altered/rogue process installed on it, and was part of a botnet
>> > that was found on our network.
>> >
>> > The exploit or compromise running on this system is likely
>> > to be an irc bot. Can you please alert the person who is
>> > responsible, for its security to patch/upgrade, remove the
>> > irc process and secure their system.
>> >
>> > = Unix System owners =
>> > A favourite place for hiding the bot(s) is in tmp
>> > and in /var/tmp/ or /dev/shm/ or in a users home directory
>> > sometimes it may be hidden like /tmp/".  ."/ or similar.
>> >
>> > The bot files can usually be found by running these one line
>> > commands as the root user.
>> >
>> > find / -exec grep -l "undernet" {} +
>> > find / -exec grep -l "sybnc" {} +
>> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>> >
>> > netstat -tanp
>> > lsof -i tcp:
>> >
>> > *netstat looking for connections to remote port 6667 or the
>> > range of ports between 6660-7000 once you find the port you
>> > can use the command, lsof -i tcp:portnumber to determine
>> > which process/user it is running under, and terminate it.
>> >
>> > = Windows System Owners =
>> > most windows bots are mIRC scripted bots and generally
>> > need a file called mirc.ini to run, you should search for
>> > this file. Run a good antivirus scanner and firewall.
>> >
>> > This Ip/host may be removed from our Irc network due to the
>> > risks it presents to our users.
>> >
>> > Should you need any help with removing the files or bot
>> > process, feel free to contact me by mail or on our network,
>> > which you connect to using any irc client and issuing
>> > /server irc.undernet.org
>> >
>> > I look forward to your reply
>> > Scot
>> >
>> > * Affected host/IPs, capture time is GMT+1: United kingdom
>> > and servers they were connected to.
>> >
>> > Please note: when resolving server names to IP Addresses
>> > that all our servers end with .undernet.org (for example)
>> > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
>> >
>> > Important: If you reply to this mail needing further
>> > information, please leave this mail intact, or supply us
>> > with the IP Address(es) in question, as we reference these
>> > mails by the unique 

Re: [Full-disclosure] one of my servers has been compromized

[Ferenc Kovacs]

On Mon, Dec 5, 2011 at 11:44 AM, Lucio Crusca  wrote:

> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I
> never
> believed it could happen to me until it actually happened: they compromized
> one of my servers. It's a Ubuntu 10.04 server with all security patches
> regularly applied. I'm inclined to believe they used some hole in the web
> application, which is a old customized Virtuemart version (1.1.3), which is
> not upgradable because of the invasive code customizations (I'm not the
> author of that code, so I have no clue about what had been changed back
> then).
>
> Now the problem for me is to track down the security hole. Here is the
> email
> my provider received and forwarded to me:
>
> > Subject: ISP Report; botnet activity on irc.undernet.org
> > [...]
> >
> > Hello, I am an operator on the irc chat network,
> > irc.undernet.org and i would like you to investigate the
> > owner of the Ip addresses that are listed at the foot of this
> > email.
> >
> > This/These host(s) have likely been compromised, and had an
> > altered/rogue process installed on it, and was part of a botnet
> > that was found on our network.
> >
> > The exploit or compromise running on this system is likely
> > to be an irc bot. Can you please alert the person who is
> > responsible, for its security to patch/upgrade, remove the
> > irc process and secure their system.
> >
> > = Unix System owners =
> > A favourite place for hiding the bot(s) is in tmp
> > and in /var/tmp/ or /dev/shm/ or in a users home directory
> > sometimes it may be hidden like /tmp/".  ."/ or similar.
> >
> > The bot files can usually be found by running these one line
> > commands as the root user.
> >
> > find / -exec grep -l "undernet" {} +
> > find / -exec grep -l "sybnc" {} +
> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> >
> > netstat -tanp
> > lsof -i tcp:
> >
> > *netstat looking for connections to remote port 6667 or the
> > range of ports between 6660-7000 once you find the port you
> > can use the command, lsof -i tcp:portnumber to determine
> > which process/user it is running under, and terminate it.
> >
> > = Windows System Owners =
> > most windows bots are mIRC scripted bots and generally
> > need a file called mirc.ini to run, you should search for
> > this file. Run a good antivirus scanner and firewall.
> >
> > This Ip/host may be removed from our Irc network due to the
> > risks it presents to our users.
> >
> > Should you need any help with removing the files or bot
> > process, feel free to contact me by mail or on our network,
> > which you connect to using any irc client and issuing
> > /server irc.undernet.org
> >
> > I look forward to your reply
> > Scot
> >
> > * Affected host/IPs, capture time is GMT+1: United kingdom
> > and servers they were connected to.
> >
> > Please note: when resolving server names to IP Addresses
> > that all our servers end with .undernet.org (for example)
> > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
> >
> > Important: If you reply to this mail needing further
> > information, please leave this mail intact, or supply us
> > with the IP Address(es) in question, as we reference these
> > mails by the unique IP Address
> >
> > Time of Capture: DECEMBER 3, 2011 10:03:48 PM
> >
> > List of IP address(es) and server it connected to:
> > my.server.ip.address (CHICAGO.IL.US
> >
> > BUDAPEST.HU.EU
> >
> > MONTREAL.QC.CA.undernet.org)
> >
>
> I've run the "find" commands and found a number of file with the first
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl
> processes that where trying to connect to port 6959 (only trying because
> I've now set up iptables so that they actually can't), but those processes
> kept spawning. Checked crontab of www-data, found the launcher, removed it.
>
> Now the problem is: how do I pervent further abuse? What should I search in
> the logs (if anything) to spot the security hole?
>
> TIA
> Lucio.
>
>
>
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>

If you take security seriously, you should remove that box from the
network(or take a snapshot and wipe everything and reinstall from scratch),
and start the investigation according to your (security) incident response
plan.
In the meantime you can start restoring the services on a clean server, but
you should consider the compromised server as fully compromised, so you
shouldn't restore data from that server, until you can't guarantee without
a proof that the data is intact/genuine.
http://en.wikipedia.org/wiki/Computer_security_incident_management
Based on your area of business, you can be obligated to report the breach
to some kind of authority and co-operate with the

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

Ferenc Kovacs wrote:

> ps: "I neverbelieved it could happen to me until it actually happened:
> they compromizedone of my servers." this is a really bad attitude.

No, it's just common saying. I apply patches, change password regularly, 
move ssh to nonstandard ports, disable remote root access and do all the 
rest I've learnt about security in years of running linux servers, also if I 
couldn't believe they would hack my server. I only overlooked a piece of 
unknown-third-party php code. It's just experience that makes you stronger.



___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Chris M]

You could ch-root your apache process/webserver going forward. This would
effectively stop the malicious process when/if your machine is compromised
via web based vulnerabilities to spread to entire machine.. meaning your
area of investigation is more isolated.

I'd expect if its automatically spread to your box the vuln would be some
sort of exec via PHP or similar -- do you have php/many client sites with
outdated software/forums/message boards/image uploaders/ anything like that
on there? .. or just badly coded bespoke dynamic/cgi scripts generally..

Ps. Did you take a copy of the bot code before you deleted it? :) would
like to see it.

On Mon, Dec 5, 2011 at 12:07 PM, Lucio Crusca  wrote:

> Ferenc Kovacs wrote:
>
> > ps: "I neverbelieved it could happen to me until it actually happened:
> > they compromizedone of my servers." this is a really bad attitude.
>
> No, it's just common saying. I apply patches, change password regularly,
> move ssh to nonstandard ports, disable remote root access and do all the
> rest I've learnt about security in years of running linux servers, also if
> I
> couldn't believe they would hack my server. I only overlooked a piece of
> unknown-third-party php code. It's just experience that makes you stronger.
>
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>



-- 
 I’m a hot-wired, heat seeking, warm-hearted cool customer, voice activated
and bio-degradable. I interface with my database, my database is in
cyberspace, so I’m interactive, I’m hyperactive and from time to time I’m
radioactive.
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Christophe Garault]

On 12/O5/2011 13:07, Lucio Crusca wrote :
> Ferenc Kovacs wrote:
>
> No, it's just common saying. I apply patches, change password regularly,
> move ssh to nonstandard ports, disable remote root access and do all the
> rest I've learnt about security in years of running linux servers, also if I
> couldn't believe they would hack my server. I only overlooked a piece of
> unknown-third-party php code. It's just experience that makes you stronger.
>
Having your /tmp partition with noexec,nosuid is also considered a good 
practice.

-- 
Toff

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[mitchell]

Hi,

Here is what you generally need to do in such cases.
1. Suspend the webapp until you investigate.
2. Check the web server logs for unusual entries and identify the entry
point. You should have noticed the timestamp of the Perl script in the /tmp
directory. Try looking for entries around this timestamp. Usually, the
timestamp would be your starting point. The files created in the /tmp/.m
directory were most probably put there via an HTTP request.
3. Find all locations writable by www-data.
4. Touch a file with a timestamp = the date of the incident.
5. Find all files -newer than the file you `touch`-ed in the locations
writable by www-data.
6. Identify any malicious files in the returned listing.
7. Stat the malicious files and log the data.
8. Quarantine / remove the malicious file(s).
 9. *Patch* the Web application.
10. Check the application code for other vulnerabilities.
11. Allow access to the Webapp.
12. Check for updates for the application regularly and apply fixes for any
security issues if full upgrade is not possible.

Unless you patch the application, the issue will most certainly re-occur.

Regards,

--
# m

On Mon, Dec 5, 2011 at 12:44, Lucio Crusca  wrote:

> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I
> never
> believed it could happen to me until it actually happened: they compromized
> one of my servers. It's a Ubuntu 10.04 server with all security patches
> regularly applied. I'm inclined to believe they used some hole in the web
> application, which is a old customized Virtuemart version (1.1.3), which is
> not upgradable because of the invasive code customizations (I'm not the
> author of that code, so I have no clue about what had been changed back
> then).
>
> Now the problem for me is to track down the security hole. Here is the
> email
> my provider received and forwarded to me:
>
> > Subject: ISP Report; botnet activity on irc.undernet.org
> > [...]
> >
> > Hello, I am an operator on the irc chat network,
> > irc.undernet.org and i would like you to investigate the
> > owner of the Ip addresses that are listed at the foot of this
> > email.
> >
> > This/These host(s) have likely been compromised, and had an
> > altered/rogue process installed on it, and was part of a botnet
> > that was found on our network.
> >
> > The exploit or compromise running on this system is likely
> > to be an irc bot. Can you please alert the person who is
> > responsible, for its security to patch/upgrade, remove the
> > irc process and secure their system.
> >
> > = Unix System owners =
> > A favourite place for hiding the bot(s) is in tmp
> > and in /var/tmp/ or /dev/shm/ or in a users home directory
> > sometimes it may be hidden like /tmp/".  ."/ or similar.
> >
> > The bot files can usually be found by running these one line
> > commands as the root user.
> >
> > find / -exec grep -l "undernet" {} +
> > find / -exec grep -l "sybnc" {} +
> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> >
> > netstat -tanp
> > lsof -i tcp:
> >
> > *netstat looking for connections to remote port 6667 or the
> > range of ports between 6660-7000 once you find the port you
> > can use the command, lsof -i tcp:portnumber to determine
> > which process/user it is running under, and terminate it.
> >
> > = Windows System Owners =
> > most windows bots are mIRC scripted bots and generally
> > need a file called mirc.ini to run, you should search for
> > this file. Run a good antivirus scanner and firewall.
> >
> > This Ip/host may be removed from our Irc network due to the
> > risks it presents to our users.
> >
> > Should you need any help with removing the files or bot
> > process, feel free to contact me by mail or on our network,
> > which you connect to using any irc client and issuing
> > /server irc.undernet.org
> >
> > I look forward to your reply
> > Scot
> >
> > * Affected host/IPs, capture time is GMT+1: United kingdom
> > and servers they were connected to.
> >
> > Please note: when resolving server names to IP Addresses
> > that all our servers end with .undernet.org (for example)
> > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
> >
> > Important: If you reply to this mail needing further
> > information, please leave this mail intact, or supply us
> > with the IP Address(es) in question, as we reference these
> > mails by the unique IP Address
> >
> > Time of Capture: DECEMBER 3, 2011 10:03:48 PM
> >
> > List of IP address(es) and server it connected to:
> > my.server.ip.address (CHICAGO.IL.US
> >
> > BUDAPEST.HU.EU
> >
> > MONTREAL.QC.CA.undernet.org)
> >
>
> I've run the "find" commands and found a number of file with the first
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl
> processes that where trying to connect to port 6959 (only trying because
> I've now set up iptables so that they actually can't), but tho

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

Gage Bystrom wrote:

> Fortunately for him, since the bot was so easy to find in the first place
> and such a simple way of maintaining it, the box was clearly seized by
> someone who didn't give a rats ass about it. Probably a skiddie or an
> automated attack to begin with.

That makes me think the problem is more likely in VirtueMart/other common 
app original code as opposed to the custom pieces of code. Skiddies or 
automated attacks are not very likely to succeed with custom code, right? If 
so, I may speed up the process of finding vulnerable code by starting from 
the common apps changelogs.

> 
> As for plugging any security holes, check your httpd error logs. If you
> noted down the time of the bot files creation date

I did and then I've found this highly suspicious log entry:

217.170.53.79 - - [03/Dec/2011:19:48:13 +0100] "POST 
/phpmyadmin/index.php?session_to_unset=123&token=42..._SESSION[!bla]=%7Cx...

which led me to this blog:

http://ha.xxor.se/2011/07/phpmyadmin-3x-multiple-remote-code.html

Now my phpmyadmin is version 3.3.2deb1 as packaged with Ubuntu 10.04.1. 

# cat /etc/issue
Ubuntu 10.04.1 LTS \n \l

# apt-cache policy phpmyadmin
phpmyadmin:
  Installed: 4:3.3.2-1
  Candidate: 4:3.3.2-1
  Version table:
 *** 4:3.3.2-1 0
990 http://archive.ubuntu.com/ubuntu/ lucid/universe Packages
100 /var/lib/dpkg/status

Ubuntu 10.04.1 was released way before july 2011 (the date of the blog) and, 
since the package does not come from security repo, I can assume it's not 
been patched.

Two questions for experts:
1. (noob question) why the hell apt-cron did not update my system to 
10.04.3? (ok, don't waste time replying that, I'll check myself)
2. Do you think said phpmyadmin vulns are reasonable attack vectors in my 
case?

> If it comes down to being too much of a hassle to get all the obvious
> vulns at least then go to your boss, admit there is an issue 

Yes obviously that's already done (btw, I'm self employed and the php code 
came directly from my customer, no problem to admit).

> and that time
> needs to be taken to remove such legacy code as this could have been a far
> worse incident if it had been more targetted and the end goal wasn't a
> botnet. 

We plan to dismiss the current code in a few days (let the shop sell some 
christmas items before dismissing it).

For the time being, I configured crontab so that only root can execute it, 
protected phpmyadmin with http authentication and configured iptables so 
that the server cannot make outgoing connections. I'm applying other 
measures in the next few hours.



___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

> 2. Do you think said phpmyadmin vulns are reasonable attack vectors in my
> case?

I do, I believe this is to be the initial infection vector.  Scanning for 
PHPMyAdmin is often and frequent and since it's likely that it was present in 
it's default (or one of the default) URIs discovery is likely.  There are a 
plethora of scanners out there which look for PHPMyAdmin specifically and add 
to the Internet noise-floor.

You are taking the correct steps with the egress firewall policy.

Forward-going, I think it may be valuable to consider:

1) Leveraging AppArmor and creating an enforcing profile for Apache; one that 
controls by extension or path, what the HTTPd can write to or access.  Be 
strict but sane.
2) Consider chrooting Apache via the 'chroot' directive for Apache (no more 
mod_chroot required).
3) Consider a strict ingress and egress firewall which would have prevent the 
egress connection to the IRCd.
4) Remain up to date; perhaps cron 'apt-get clean all; apt-get update; apt-get 
-t lucid-security -y dist-upgrade' (I believe the security channel is correct)
5) Consider sane php.ini values and leverage Suhosin (plugin) as well 
(http://www.hardened-php.net/suhosin/index.html); disallow url_fopen and 
url_include.  Disallow the exec(), system(), passthru(), etc commands if 
possible.  url_fopen() will thwart RFI.  LFI should be thwarted by a sane 
AppArmor profile.
6) Restrict access to PHPMyAdmin based on authentication or remove it's access 
entirely.
7) Consider leveraging something like Fail2ban against Apache's error and 
access logs looking for excessive high-frequency HTTP 404, 403, or 500 errors 
as these are indicative of scanning.  This is a great tool to stop Web-app 
scanning.
8) As you've already done with SSH, move it from TCP 22, PermitRootLogin no, 
and disable password authentication using key-based authentication.
9) Using OSSEC-HIDS (http://www.ossec.net/) with inotify() to watch changes to 
your system and Apache directories including those that are HTTP writable.
10) Mount /tmp noexec,nosuid,nodev as others have recommended.
11) Optionally use mod_security with a tuned ruleset or another WAF.

I find #7 to be extremely helpful.  Feel free to hit me up for additional 
clarification if needed.  I wish you the best, remember that defense-in-depth 
is the best approach here.

This is a good list-discussion as it is likely to yield many valuable ways to 
correctly secure web applications.  Potentially any one of the suggestiosn in 
#1, #2, #3, #4, #5, #6, #7, and #10 would have saved your box.

I hope this helped,
John
  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Michael Wood]

Awesome tips guys...
On Dec 5, 2011 11:01 AM, "John Jacobs"  wrote:

>
> > 2. Do you think said phpmyadmin vulns are reasonable attack vectors in my
> > case?
>
> I do, I believe this is to be the initial infection vector.  Scanning for
> PHPMyAdmin is often and frequent and since it's likely that it was present
> in it's default (or one of the default) URIs discovery is likely.  There
> are a plethora of scanners out there which look for PHPMyAdmin specifically
> and add to the Internet noise-floor.
>
> You are taking the correct steps with the egress firewall policy.
>
> Forward-going, I think it may be valuable to consider:
>
> 1) Leveraging AppArmor and creating an enforcing profile for Apache; one
> that controls by extension or path, what the HTTPd can write to or access.
> Be strict but sane.
> 2) Consider chrooting Apache via the 'chroot' directive for Apache (no
> more mod_chroot required).
> 3) Consider a strict ingress and egress firewall which would have prevent
> the egress connection to the IRCd.
> 4) Remain up to date; perhaps cron 'apt-get clean all; apt-get update;
> apt-get -t lucid-security -y dist-upgrade' (I believe the security channel
> is correct)
> 5) Consider sane php.ini values and leverage Suhosin (plugin) as well (
> http://www.hardened-php.net/suhosin/index.html); disallow url_fopen and
> url_include.  Disallow the exec(), system(), passthru(), etc commands if
> possible.  url_fopen() will thwart RFI.  LFI should be thwarted by a sane
> AppArmor profile.
> 6) Restrict access to PHPMyAdmin based on authentication or remove it's
> access entirely.
> 7) Consider leveraging something like Fail2ban against Apache's error and
> access logs looking for excessive high-frequency HTTP 404, 403, or 500
> errors as these are indicative of scanning.  This is a great tool to stop
> Web-app scanning.
> 8) As you've already done with SSH, move it from TCP 22, PermitRootLogin
> no, and disable password authentication using key-based authentication.
> 9) Using OSSEC-HIDS (http://www.ossec.net/) with inotify() to watch
> changes to your system and Apache directories including those that are HTTP
> writable.
> 10) Mount /tmp noexec,nosuid,nodev as others have recommended.
> 11) Optionally use mod_security with a tuned ruleset or another WAF.
>
> I find #7 to be extremely helpful.  Feel free to hit me up for additional
> clarification if needed.  I wish you the best, remember that
> defense-in-depth is the best approach here.
>
> This is a good list-discussion as it is likely to yield many valuable ways
> to correctly secure web applications.  Potentially any one of the
> suggestiosn in #1, #2, #3, #4, #5, #6, #7, and #10 would have saved your
> box.
>
> I hope this helped,
> John
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Tim]

For future reference, and for the benefit of people searching for
solutions to similar problems: You've made the most common rookie
mistake.  You have already trashed potentially critical information
about the attack by trying to clean up the server first.  Don't do
that.


> I've run the "find" commands and found a number of file with the first 
> "find", under /tmp/.m
> 
> Deleted them, looked up remote connections with netstat, killed perl 
> processes that where trying to connect to port 6959 (only trying because 
> I've now set up iptables so that they actually can't), but those processes 
> kept spawning. Checked crontab of www-data, found the launcher, removed it.


Instead, your first step should be trying to preserve the evidence.
You should obtain forensic images of the disk and if possible,
physical memory of the host before taking any corrective action on the
host itself.  

If you don't have budget to bring in a professional to do the
investigation, then capturing memory is probably not practical (it is
easy to do it wrong and trash useful information on disk).  So in that
case, shut the server down and use a boot disk to obtain a DD image of
your drive(s).  If your server has any useful interactions with other
hosts, such as relying on an LDAP server for authentication or sending
logs to a syslog server, then capture all of the logs you can from
those hosts as well.  (Simple copying of those files is sufficient,
unless you have reason to believe they are compromised as well.)

If you really need the server back in production ASAP, you can build a
new copy of it in parallel with these preservation activities.  I
definitely recommend against trying to "clean" a compromised server.
You might say: "gosh, then I'll need more hardware to do that, and
hardware is expensive!"  No, hardware is cheap.  Allowing an attacker
continued access to your infrastructure is much more expensive.

Ok, that's the "what".  Now the "why":

By doing what you did (deleting "evil" files, running a find across
the filesystem), you may have lost the back door files that tell you
exactly how the malware works and also altered
created/modified/accessed dates of the malicious files themselves.
Those dates, while not always reliable, can be very useful in
determining the timeframe of the attack.  In addition, each time you
do anything on the host, more data is written to disk (logs,
.bash_history, you name it) and that new information could be
overwriting previously deleted files that could be useful to the
investigation.  

Suffice it to say, you've probably already trashed a wealth of
information that would have helped identify how the attacker got in.
Enough information may still be available to determine what the
vulnerability is that they exploited, but you've certainly made it a
lot harder to isolate the event.

tim

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

> For future reference, and for the benefit of people searching for
> solutions to similar problems: You've made the most common rookie
> mistake. You have already trashed potentially critical information
> about the attack by trying to clean up the server first. Don't do
> that.

Tim, while I do believe there is some truth in what you are saying here, I 
respectfully disagree in that this tends to be a run-of-the-mill IRC bot as 
evidenced by the Undernet advisory.  This looks like a skiddie-de-jour attack 
against PHPMyAdmin and nothing to be concerned with regarding cloning disk 
images and full forensics.  I do respect your input and thoughts though for a 
more targeted attack; not an IRC bot in /tmp.

That being said, I strongly believe in preserving bash_history as well as vital 
log data.  It's best/wise to ship this off to a separate Syslog server.  If 
you're paranoid turn up stunnel between the devices.  For example and as 
evidenced by many of the documented attacks here purging of bash_history is 
common ala 'history -c' after fun.  To thwart this I like the idea of logging 
to syslog often, ensure permissions are strict for the syslog messages, and 
shipping the syslog data off to a separate box.  I like to:

1) Generate an E-Mail alert when someone logs in, by adjusting /etc/bash.bashrc 
(or similar based on distribution) to:

#Email alert for login
echo -e "Subject: Login from $(/usr/bin/whoami) on $(/bin/hostname) at 
$(/bin/date)\n\n$(/usr/bin/last -ian 10)\n"|/usr/sbin/sendmail 
recipi...@example.com

2) Preserve, via Syslog, commands executed at the prompt, by adjusting 
/etc/profile.  Adjust /etc/syslog.conf or /etc/rsyslog.conf to forward these 
syslog messages off-box to another asset.  If you're paranoid use stunnel.

export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo "$$ $USER 
$(history 1)"|/usr/bin/logger -p user.alert -t bash_history'
readonly PROMPT_COMMAND

3) Preserve bash_history by adjusting /etc/profile:

#Secure the Bash History
export HISTSIZE=1500
export HISTCONTROL=''
export HISTIGNORE=''
export HISTTIMEFORMAT='%F %T '
readonly HISTFILE
readonly HISTFILESIZE
readonly HISTSIZE
readonly HISTCONTROL
readonly HISTIGNORE
readonly HISTTIMEFORMAT

4) Optionally use chattr to set ~/.bash_history to append-only:

  #Secure .bash_history (poke fun of the while subshell if you wish)
  /usr/bin/find / -maxdepth 3|/bin/grep -i bash_history|while read line; do 
/usr/bin/chattr +a "$line"; done

5) Use of an IP Recorder, something like daemonlogger, in ring-buffer mode, as 
a way to record all ingress/egress traffic using a percentage of the disk.  See 
http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html

I am eager to hear any additional thoughts or methods for security information 
such as this.

Thanks,
John





  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Dave]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/12/2011 10:44, Lucio Crusca wrote:
> Hello *,
> 
> I'm not new here, but I've mostly lurked all the time through gmane. I never 
> believed it could happen to me until it actually happened: they compromized 
> one of my servers. It's a Ubuntu 10.04 server with all security patches 
> regularly applied. I'm inclined to believe they used some hole in the web 
> application, which is a old customized Virtuemart version (1.1.3), which is 
> not upgradable because of the invasive code customizations (I'm not the 
> author of that code, so I have no clue about what had been changed back 
> then).
> 
> Now the problem for me is to track down the security hole. Here is the email 
> my provider received and forwarded to me:
> 
>> Subject: ISP Report; botnet activity on irc.undernet.org
>> [...]
>>  
>> Hello, I am an operator on the irc chat network, 
>> irc.undernet.org and i would like you to investigate the 
>> owner of the Ip addresses that are listed at the foot of this 
>> email.
>>  
>> This/These host(s) have likely been compromised, and had an
>> altered/rogue process installed on it, and was part of a botnet
>> that was found on our network. 
>>  
>> The exploit or compromise running on this system is likely 
>> to be an irc bot. Can you please alert the person who is 
>> responsible, for its security to patch/upgrade, remove the 
>> irc process and secure their system.
>>  
>> = Unix System owners = 
>> A favourite place for hiding the bot(s) is in tmp 
>> and in /var/tmp/ or /dev/shm/ or in a users home directory
>> sometimes it may be hidden like /tmp/".  ."/ or similar.
>>  
>> The bot files can usually be found by running these one line 
>> commands as the root user.
>>  
>> find / -exec grep -l "undernet" {} +
>> find / -exec grep -l "sybnc" {} +
>> find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>> find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>>  
>> netstat -tanp
>> lsof -i tcp:
>>  
>> *netstat looking for connections to remote port 6667 or the 
>> range of ports between 6660-7000 once you find the port you 
>> can use the command, lsof -i tcp:portnumber to determine
>> which process/user it is running under, and terminate it. 
>>  
>> = Windows System Owners = 
>> most windows bots are mIRC scripted bots and generally 
>> need a file called mirc.ini to run, you should search for 
>> this file. Run a good antivirus scanner and firewall.
>>  
>> This Ip/host may be removed from our Irc network due to the
>> risks it presents to our users.
>>  
>> Should you need any help with removing the files or bot
>> process, feel free to contact me by mail or on our network,
>> which you connect to using any irc client and issuing
>> /server irc.undernet.org
>>  
>> I look forward to your reply
>> Scot
>>  
>> * Affected host/IPs, capture time is GMT+1: United kingdom
>> and servers they were connected to.
>>  
>> Please note: when resolving server names to IP Addresses
>> that all our servers end with .undernet.org (for example)
>> Tampa.FL.US. is actually  Tampa.FL.US.undernet.org 
>>  
>> Important: If you reply to this mail needing further
>> information, please leave this mail intact, or supply us 
>> with the IP Address(es) in question, as we reference these 
>> mails by the unique IP Address
>>  
>> Time of Capture: DECEMBER 3, 2011 10:03:48 PM
>>  
>> List of IP address(es) and server it connected to:
>> my.server.ip.address (CHICAGO.IL.US
>>  
>> BUDAPEST.HU.EU
>>  
>> MONTREAL.QC.CA.undernet.org)
>>  
> 
> I've run the "find" commands and found a number of file with the first 
> "find", under /tmp/.m
> 
> Deleted them, looked up remote connections with netstat, killed perl 
> processes that where trying to connect to port 6959 (only trying because 
> I've now set up iptables so that they actually can't), but those processes 
> kept spawning. Checked crontab of www-data, found the launcher, removed it.
> 
> Now the problem is: how do I pervent further abuse? What should I search in 
> the logs (if anything) to spot the security hole?
> 
> TIA
> Lucio.

Much of what can be done to secure your server has been mentioned in the 
replies.

I would just like to add this:
You may want to consider running a rootkit check as a daily cron job and having 
the results emailed to you.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBTtz/uLIvn8UFHWSmAQL1GAgA4JWPXX1klCE6Tbi3rYuYzHnr7nU739V8
/5+1dKq/30Sq3MeKOxRWbrJRMJkEdf9Dg+61wZUq1YMvNFjX+ISmGBQzY8oay2y2
iRhzgE14YWFisZH4xEMliioC7kjRDl64+sOKahCImOmnJNChdpoQWwsheNn/gqa9
sR6g8A13ERN0Ngm9TT+9C4t4OukZGU01B9mI+9XMOY4cryKyb8jseqOrvyxYcU2d
vBdKQeUuMSo6d+069bb2y7nmojdBcdgj/wr1pJuo2wxe2QS9hqHBNOnYmy6Qiios
N4nd3wHiWIaUlwVSuC8BsT+d+G/rGGy/dh1vI3RIOoVvvJUXs5/JyQ==
=nqLC
-END PGP SIGNATURE-

___
Full-Disclosure - We believe in it.
Charter: http:/

Re: [Full-disclosure] one of my servers has been compromized

[James Condron]

John,

All good thoughts but can we show the server was rooted?

In otherwords; instead of an attacker getting root and then adding this to a 
botnet this way is it not more likely that the original attack added the server 
in one step to avoid the need to do this?

Attackers, from my experience, don't need to worry about rooting when they can 
find a vuln, exploit and add to a botnet- they don't need to be able to SSH in, 
nor do they need a shell; the IRC channel takes care of all they need.

Additionally suppose we're looking at your old fashioned shellcode payload- 
probably  a fair assumption. What happens if this is not using bash, perhaps sh 
isn't pointing to bash, suppose even csh or zsh.

Lets  also not forget without a proper disassembly of the server pma is only a 
likely vector.

In this case I tend to suggest that OP takes a stab at the time of compromise 
to get a backup that can be trusted, takes a new hard drive, restore from 
backup and upgrade everything that can be possibly upgraded (and so on- the 
same repair stuff we all do in our sleep; nothing new) and uses the old data to 
go through the logs, go through the dumped data from memory, assuming fmem and 
understanding of /proc/iomem



On 5 Dec 2011, at 17:20, John Jacobs wrote:

> 
>> For future reference, and for the benefit of people searching for
>> solutions to similar problems: You've made the most common rookie
>> mistake. You have already trashed potentially critical information
>> about the attack by trying to clean up the server first. Don't do
>> that.
> 
> Tim, while I do believe there is some truth in what you are saying here, I 
> respectfully disagree in that this tends to be a run-of-the-mill IRC bot as 
> evidenced by the Undernet advisory.  This looks like a skiddie-de-jour attack 
> against PHPMyAdmin and nothing to be concerned with regarding cloning disk 
> images and full forensics.  I do respect your input and thoughts though for a 
> more targeted attack; not an IRC bot in /tmp.
> 
> That being said, I strongly believe in preserving bash_history as well as 
> vital log data.  It's best/wise to ship this off to a separate Syslog server. 
>  If you're paranoid turn up stunnel between the devices.  For example and as 
> evidenced by many of the documented attacks here purging of bash_history is 
> common ala 'history -c' after fun.  To thwart this I like the idea of logging 
> to syslog often, ensure permissions are strict for the syslog messages, and 
> shipping the syslog data off to a separate box.  I like to:
> 
> 1) Generate an E-Mail alert when someone logs in, by adjusting 
> /etc/bash.bashrc (or similar based on distribution) to:
> 
> #Email alert for login
> echo -e "Subject: Login from $(/usr/bin/whoami) on $(/bin/hostname) at 
> $(/bin/date)\n\n$(/usr/bin/last -ian 10)\n"|/usr/sbin/sendmail 
> recipi...@example.com
> 
> 2) Preserve, via Syslog, commands executed at the prompt, by adjusting 
> /etc/profile.  Adjust /etc/syslog.conf or /etc/rsyslog.conf to forward these 
> syslog messages off-box to another asset.  If you're paranoid use stunnel.
> 
> export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo "$$ $USER 
> $(history 1)"|/usr/bin/logger -p user.alert -t bash_history'
> readonly PROMPT_COMMAND
> 
> 3) Preserve bash_history by adjusting /etc/profile:
> 
> #Secure the Bash History
> export HISTSIZE=1500
> export HISTCONTROL=''
> export HISTIGNORE=''
> export HISTTIMEFORMAT='%F %T '
> readonly HISTFILE
> readonly HISTFILESIZE
> readonly HISTSIZE
> readonly HISTCONTROL
> readonly HISTIGNORE
> readonly HISTTIMEFORMAT
> 
> 4) Optionally use chattr to set ~/.bash_history to append-only:
> 
>   #Secure .bash_history (poke fun of the while subshell if you wish)
>   /usr/bin/find / -maxdepth 3|/bin/grep -i bash_history|while read line; do 
> /usr/bin/chattr +a "$line"; done
> 
> 5) Use of an IP Recorder, something like daemonlogger, in ring-buffer mode, 
> as a way to record all ingress/egress traffic using a percentage of the disk. 
>  See http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
> 
> I am eager to hear any additional thoughts or methods for security 
> information such as this.
> 
> Thanks,
> John
> 
> 
> 
> 
> 
> 
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

Tim wrote:

> 
> For future reference, and for the benefit of people searching for
> solutions to similar problems: You've made the most common rookie
> mistake.

Well, I actually made 2 mistakes and the second compensated the harm the 
first did...

My second mistake I did not mention before was to overlook various other 
folders in /tmp that were older than /tmp/.m and contained lots of other 
crap (so they are even more useful in finding the original attack vector, 
being older).

However I did save the original launcher found in /tmp and all that older 
stuff too for analisys.

> If you don't have budget to bring in a professional to do the
> investigation, then capturing memory is probably not practical (it is
> easy to do it wrong and trash useful information on disk).  

Using dd on /dev/mem and piping results through netcat it's not that 
difficult, and a bit of google explains how to do it the right way, but in 
my case there are two other problems:

1. The attack took place several days ago and it's likely the system ram has 
been overwritten several time since then

2. My server runs in a OpenVZ container (it's a hosted vps)... /dev/mem 
exists but it's obviously not accessible.

However I understand your point. 





___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Tim]

> > For future reference, and for the benefit of people searching for
> > solutions to similar problems: You've made the most common rookie
> > mistake. You have already trashed potentially critical information
> > about the attack by trying to clean up the server first. Don't do
> > that.
> 
> Tim, while I do believe there is some truth in what you are saying here, I 
> respectfully disagree in that this tends to be a run-of-the-mill IRC bot as 
> evidenced by the Undernet advisory.  This looks like a skiddie-de-jour attack 
> against PHPMyAdmin and nothing to be concerned with regarding cloning disk 
> images and full forensics.  I do respect your input and thoughts though for a 
> more targeted attack; not an IRC bot in /tmp.

Why take the risk?  You don't know what the attacker actually did
until you do some analysis.  If you do analysis before capturing a
disk image, you're destroying evidence.

Rebuilding a server is not hard.  It has a known quantity of effort
involved and reliably prevents further intrusion which leverages the
access previously gained.

On the other hand, conducting an investigation to the point where you
are reasonably sure an attacker can't continue to leverage that access
costs a lot of time and money.

tim

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

> Subject: Re: [Full-disclosure] one of my servers has been compromized
> From: ja...@zero-internet.org.uk
> Date: Mon, 5 Dec 2011 17:36:53 +
> CC: tim-secur...@sentinelchicken.org; lu...@sulweb.org; 
> full-disclosure@lists.grok.org.uk
> To: flamdu...@hotmail.com
>
> John,
>
> All good thoughts but can we show the server was rooted?
>
> In otherwords; instead of an attacker getting root and then adding this to a 
> botnet this way is it not more likely that the original attack added the 
> server in one step to avoid the need to do this?

James, thank you for your response.  I agree with you here, I was speaking more 
along the lines of defense-in-depth and security generalities.  In this case I 
believe a vulnerability in an unpatched/unprotected version of PHPMyAdmin 
permitted the Apache process to write to /tmp and spawn a process.

In my previous experience, including a few vulnerabilities in Roundcube, I've 
seen a programmatic scan, exploitation, and then wget/dropping of a Perl IRC 
bot.  Of course this is all speculation based on previous experience, I am not 
looking at lucio's box.  The Apache access/error logs should have the offending 
log entries and/or any output from the system()/exec() etc PHP functions.  
Lucio -- what user was the IRC bot running under?

It's evident that 10.04.1 LTS is certainly out of date and there are a 
multitude of vulnerabilities that could be leverage for privilege escalation.

Lucio, I would also recommend subscribing to the Ubuntu Security-Announce 
mailing lists which issue the USNs so you are aware of which packages have been 
patched and those which require a reboot to effect the changes, such as Kernel 
update.

Thanks again for the dialog Tim and James, this exchange is very beneficial and 
appreciated.

James you are correct regarding the shell; I would assume /bin/sh would be 
pointed to /bin/dash or /bin/bash on a Ubuntu system.  I would assume the 
user's login shell would be /bin/bash.  Again, these are all assumptions made 
on my part and can certainly be incorrect, however, I would hope if someone is 
using KSH that they would see past the BASHisms of the previous message.

Thanks,
John
  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Paul Schmehl]

--On December 5, 2011 1:48:24 PM +0100 Christophe Garault 
 wrote:

>>
> Having your /tmp partition with noexec,nosuid is also considered a good
> practice.

That's not a bad generic suggestion, but it won't do a thing for this hack. 
They deposit perl scripts in /tmp/.m and then run them by calling perl, 
which is not in tmp.  This is a very common hack of a poorly written web 
application.  I doubt seriously that the "box" has been hacked - only the 
webserver is affected - especially based on his description of how he found 
and got rid of its elements.  (IOW, they didn't get root on the box - they 
only compromised the web application and then ran shells in perl off of 
that.)

-- 
Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
***
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Tim]

> Well, I actually made 2 mistakes and the second compensated the harm the 
> first did...
> 
> My second mistake I did not mention before was to overlook various other 
> folders in /tmp that were older than /tmp/.m and contained lots of other 
> crap (so they are even more useful in finding the original attack vector, 
> being older).
> 
> However I did save the original launcher found in /tmp and all that older 
> stuff too for analisys.

As soon as you started reading the files in /tmp, you would have
overwritten access times.  Maybe those wouldn't have been useful,
maybe they would have been, who knows.  Basically, as soon as you're
reasonably sure a compromise happened, acquiring forensic images is in
order.


> > If you don't have budget to bring in a professional to do the
> > investigation, then capturing memory is probably not practical (it is
> > easy to do it wrong and trash useful information on disk).  
> 
> Using dd on /dev/mem and piping results through netcat it's not that 
> difficult, and a bit of google explains how to do it the right way, but in 
> my case there are two other problems:

Sending memory image data over the network, such as with netcat is
very important, so it is good you realize that.  Writing to disk (even
an external drive) causes you to lose evidence.


> 1. The attack took place several days ago and it's likely the system ram has 
> been overwritten several time since then

Don't be so sure.  If there are any processes that were running
several days ago and are still running now, they may have unallocated
data in the stack or heap that is related to the attack.  The attacker
may have executed programs that are running but have since been
deleted from disk.  If the system has had a rootkit installed, having
memory available can make analysis a lot easier.


> 2. My server runs in a OpenVZ container (it's a hosted vps)... /dev/mem 
> exists but it's obviously not accessible.

Yes, there are many difficulties with acquiring reliable physical
memory images from Linux hosts these days.  If your system is a VM,
the best way to do it is to simply pause the VM or take a snapshot and
then backup the memory image file that was created on the host.  You
can also just take a copy of the VM's disks and use them as your
forensic image, no DD required.

Outside of that, if you're not familiar with the risks involved with
acquiring a memory image from within a running system though, I would
recommend against it.

tim

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Paul Schmehl]

--On December 5, 2011 11:44:04 AM +0100 Lucio Crusca  
wrote:

>
> Now the problem is: how do I pervent further abuse? What should I search
> in  the logs (if anything) to spot the security hole?
>

Install mod-security and write custom rules to prevent the hack.  Look in 
your webserver logs to see what they did to get in.


-- 
Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
***
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

John Jacobs wrote:

> In my previous experience, including a few vulnerabilities in Roundcube,
> I've seen a programmatic scan, exploitation, and then wget/dropping of a
> Perl IRC bot.  Of course this is all speculation based on previous
> experience, I am not looking at lucio's box.  The Apache access/error logs
> should have the offending log entries and/or any output from the
> system()/exec() etc PHP functions.  Lucio -- what user was the IRC bot
> running under?

www-data

> Lucio, I would also recommend subscribing to the Ubuntu Security-Announce
> mailing lists which issue the USNs so you are aware of which packages have
> been patched and those which require a reboot to effect the changes, such
> as Kernel update.

hosted vps, the kernel updates are up to the provider.

> James you are correct regarding the shell; I would assume /bin/sh would be
> pointed to /bin/dash or /bin/bash on a Ubuntu system.  I would assume the
> user's login shell would be /bin/bash.  

Ubuntu has bash and dash installed by default, and /bin/sh -> dash, but any 
user is defaulted to /bin/bash



___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

> Why take the risk? You don't know what the attacker actually did
> until you do some analysis. If you do analysis before capturing a
> disk image, you're destroying evidence.
>
> Rebuilding a server is not hard. It has a known quantity of effort
> involved and reliably prevents further intrusion which leverages the
> access previously gained.
>
> On the other hand, conducting an investigation to the point where you
> are reasonably sure an attacker can't continue to leverage that access
> costs a lot of time and money.

I'm conflicted here, though I do agree with what you say.  A simple attack 
lacking any sophistication whose goal is simply to drop an IRC bot and nothing 
more may result in undue expense through effort and analysis.  Conversely, 
assuming that former does not confirm the scope of attack or impact...  I'm 
conflicted but I agree with your assessment.

Perhaps in this situation one would be more likely to fall back on the output 
of the defense in depth tools previously recommended.  In this case they are 
not available so perhaps caution is wise as you have suggested.

I think it reasonable to assume a documented and vulnerable out of date 
world-accessible version of PHPMyAdmin is likely the initial attack vector.  
The Apache/PHP logs should prove this to be the case, so rebuilding a box and 
disabling PHPMyAdmin may be a reasonable response to this incident without the 
need for forensic analysis... depending on the organization.  If this box is a 
singular VPS I think it reasonable, if it's a part of a larger system which 
could be leveraged as a launching platform to attack other LAN or 
organizational-owned boxes then by all means a comprehensive analysis should be 
undertaken.

Tim thanks for your thoughts and reply.

Thanks,
John
  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

Tripwire is awesome for many reasons. The original use of rootkit detection
is no longer one of them. It was used back when userland rootkits were big,
it has zero effect on kernel rootkits. That being said you can use it to
watch other critical files for improper access. Keep tabs on your cron
files, configs, and your web pages(why crack password hashes when you can
force the login script to hand deliver the plaintext?), etc.

Afaik, rootkit detection has made practically no progress. In part because
of several advancements in rootkits and also in part of the overzealous
trend in reimaging even the slightest compromised box.

That being said a modern rootkit detector would need to be installed first
and watch for suspicious behavior, such as attempting to hook system calls
and watching kernel module loading. Nothing but the kernels mod loader
should have a legitimate reason to change the list of kernel modules.

In OPs case he really shouldn't need to worry about a rootkit. This wasn't
a targeted attack. If it was, or even if the script tried to load a rootkit
then he wouldn't even have seen the questionable files in the first place,
nor the processes, or the loader. He wouldn't have even been able to grep
them. Also deploying a rootkit as part of a serious attack is annoying. You
fuck up one thing and not only have you lost the box, but left a strew of
evidence that you were trying to hide. Not to mention public rootkits never
really caught up with the kernel developments. Most rootkits in place are
still targetting 2.4 kernels because only smart dedicated attackers would
have the skills to develop and deploy a modern rootkit for a modern kernel.
Such an attacker wouldn't make so many rookie/nonchalant mistakes as the
attacker on the ops box did. At most he needs to be concerned if he had
caught all the backdoors or not. Considering he doesn't realistically need
to worry about a rootkit(remember, rootkits are annoying, usually easier
and more practical to stay quiet, get want you want, and leave quietly),
then he could watch for outgoing connections while monitoring any new open
ports that have spawned. I would suggest iptables but the OP stated he
doesn't own the server and has no root access. Sure there are many clever
ways to reserve access but they all start falling apart as long as your
waiting and watching for them to make a peep.
On Dec 5, 2011 5:53 AM, "Dan Ballance"  wrote:

> Thanks for the heads-up on rkhunter Gage.
>
> Is there anything else out there atm that works as a reasonable root kit
> detector or is such a thing considered impossible now? I realise a skilled
> attack will be able to bury itself without a trace, but I'm thinking of
> something that can be used in less skilled breaches such as the one thought
> to have been identified in this thread. Sometimes something imperfect is
> still better than nothing I think.
>
> Also, am I correct to think that using something like tripwire is the best
> way to detect root kits properly, but that it obviously needs installing
> when the box is fresh and before it has been physically connected to a
> network?
>
> thanks to everyone for their valuable contributions here - much
> appreciated!
>
> dan :)
>
> On 5 December 2011 11:13, Gage Bystrom  wrote:
>
>> If it was a rootkit then trying to run the outdated rkhunter would be a
>> moot point. Whatever seizes the kernel first wins, hands down.
>>
>> Fortunately for him, since the bot was so easy to find in the first place
>> and such a simple way of maintaining it, the box was clearly seized by
>> someone who didn't give a rats ass about it. Probably a skiddie or an
>> automated attack to begin with.
>>
>> As for plugging any security holes, check your httpd error logs. If you
>> noted down the time of the bot files creation date you would look around
>> the same time for suspicious log entries. If they were as careless in
>> scrubbing the logs as they were holding the box it would give you a look
>> into how it may have been compromised. If you're getting things like
>> ../.../../../../etc/passwd then some sort of lfi vuln was likely exploited,
>> start grepping your php files for stuff like include(), or if you're
>> getting something like into outfile then check your mysql user permissions
>> and don't let it have file perms, and then start grepping down for sql
>> vulns.
>>
>> If it comes down to being too much of a hassle to get all the obvious
>> vulns at least then go to your boss, admit there is an issue and that time
>> needs to be taken to remove such legacy code as this could have been a far
>> worse incident if it had been more targetted and the end goal wasn't a
>> botnet.
>>  On Dec 5, 2011 3:02 AM, "Dan Ballance"  wrote:
>>
>>> I'm no expert, but here's something to get you started while you wait
>>> for more experienced replies. Check for root kits:
>>>
>>> sudo apt-get install rkhunter
>>> sudo rkhunter --update
>>> sudo rkhunter --check
>>>
>>> On 5 December 2011 10:44, Lucio C

Re: [Full-disclosure] one of my servers has been compromized

[Javier Bassi]

In addition to the tips given (chroot, disable shell_exec,etc), you
should also use open_basedir with DocumentRoot as path on each
VirtualHost. In case of a compromise via webapp, this will reduce the
compromised zone in the filesystem to the DocumentRoot of one
VirtualHost instead of the whole chroot jail.

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Aris Adamantiadis]

A good start would have been to take a snapshot of your compromized
system and not to delete any file.
Now, good luck to understand what happened.

Aris

Le 5/12/11 11:44, Lucio Crusca a écrit :
> Hello *,
> 
> I'm not new here, but I've mostly lurked all the time through gmane. I never 
> believed it could happen to me until it actually happened: they compromized 
> one of my servers. It's a Ubuntu 10.04 server with all security patches 
> regularly applied. I'm inclined to believe they used some hole in the web 
> application, which is a old customized Virtuemart version (1.1.3), which is 
> not upgradable because of the invasive code customizations (I'm not the 
> author of that code, so I have no clue about what had been changed back 
> then).
> 
> Now the problem for me is to track down the security hole. Here is the email 
> my provider received and forwarded to me:
> 
>> Subject: ISP Report; botnet activity on irc.undernet.org
>> [...]
>>  
>> Hello, I am an operator on the irc chat network, 
>> irc.undernet.org and i would like you to investigate the 
>> owner of the Ip addresses that are listed at the foot of this 
>> email.
>>  
>> This/These host(s) have likely been compromised, and had an
>> altered/rogue process installed on it, and was part of a botnet
>> that was found on our network. 
>>  
>> The exploit or compromise running on this system is likely 
>> to be an irc bot. Can you please alert the person who is 
>> responsible, for its security to patch/upgrade, remove the 
>> irc process and secure their system.
>>  
>> = Unix System owners = 
>> A favourite place for hiding the bot(s) is in tmp 
>> and in /var/tmp/ or /dev/shm/ or in a users home directory
>> sometimes it may be hidden like /tmp/".  ."/ or similar.
>>  
>> The bot files can usually be found by running these one line 
>> commands as the root user.
>>  
>> find / -exec grep -l "undernet" {} +
>> find / -exec grep -l "sybnc" {} +
>> find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>> find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>>  
>> netstat -tanp
>> lsof -i tcp:
>>  
>> *netstat looking for connections to remote port 6667 or the 
>> range of ports between 6660-7000 once you find the port you 
>> can use the command, lsof -i tcp:portnumber to determine
>> which process/user it is running under, and terminate it. 
>>  
>> = Windows System Owners = 
>> most windows bots are mIRC scripted bots and generally 
>> need a file called mirc.ini to run, you should search for 
>> this file. Run a good antivirus scanner and firewall.
>>  
>> This Ip/host may be removed from our Irc network due to the
>> risks it presents to our users.
>>  
>> Should you need any help with removing the files or bot
>> process, feel free to contact me by mail or on our network,
>> which you connect to using any irc client and issuing
>> /server irc.undernet.org
>>  
>> I look forward to your reply
>> Scot
>>  
>> * Affected host/IPs, capture time is GMT+1: United kingdom
>> and servers they were connected to.
>>  
>> Please note: when resolving server names to IP Addresses
>> that all our servers end with .undernet.org (for example)
>> Tampa.FL.US. is actually  Tampa.FL.US.undernet.org 
>>  
>> Important: If you reply to this mail needing further
>> information, please leave this mail intact, or supply us 
>> with the IP Address(es) in question, as we reference these 
>> mails by the unique IP Address
>>  
>> Time of Capture: DECEMBER 3, 2011 10:03:48 PM
>>  
>> List of IP address(es) and server it connected to:
>> my.server.ip.address (CHICAGO.IL.US
>>  
>> BUDAPEST.HU.EU
>>  
>> MONTREAL.QC.CA.undernet.org)
>>  
> 
> I've run the "find" commands and found a number of file with the first 
> "find", under /tmp/.m
> 
> Deleted them, looked up remote connections with netstat, killed perl 
> processes that where trying to connect to port 6959 (only trying because 
> I've now set up iptables so that they actually can't), but those processes 
> kept spawning. Checked crontab of www-data, found the launcher, removed it.
> 
> Now the problem is: how do I pervent further abuse? What should I search in 
> the logs (if anything) to spot the security hole?
> 
> TIA
> Lucio.
> 
> 
> 
> 
> 
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
> 

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Dan Ballance]

Thanks for the heads-up on rkhunter Gage.

Is there anything else out there atm that works as a reasonable root kit
detector or is such a thing considered impossible now? I realise a skilled
attack will be able to bury itself without a trace, but I'm thinking of
something that can be used in less skilled breaches such as the one thought
to have been identified in this thread. Sometimes something imperfect is
still better than nothing I think.

Also, am I correct to think that using something like tripwire is the best
way to detect root kits properly, but that it obviously needs installing
when the box is fresh and before it has been physically connected to a
network?

thanks to everyone for their valuable contributions here - much appreciated!

dan :)

On 5 December 2011 11:13, Gage Bystrom  wrote:

> If it was a rootkit then trying to run the outdated rkhunter would be a
> moot point. Whatever seizes the kernel first wins, hands down.
>
> Fortunately for him, since the bot was so easy to find in the first place
> and such a simple way of maintaining it, the box was clearly seized by
> someone who didn't give a rats ass about it. Probably a skiddie or an
> automated attack to begin with.
>
> As for plugging any security holes, check your httpd error logs. If you
> noted down the time of the bot files creation date you would look around
> the same time for suspicious log entries. If they were as careless in
> scrubbing the logs as they were holding the box it would give you a look
> into how it may have been compromised. If you're getting things like
> ../.../../../../etc/passwd then some sort of lfi vuln was likely exploited,
> start grepping your php files for stuff like include(), or if you're
> getting something like into outfile then check your mysql user permissions
> and don't let it have file perms, and then start grepping down for sql
> vulns.
>
> If it comes down to being too much of a hassle to get all the obvious
> vulns at least then go to your boss, admit there is an issue and that time
> needs to be taken to remove such legacy code as this could have been a far
> worse incident if it had been more targetted and the end goal wasn't a
> botnet.
>  On Dec 5, 2011 3:02 AM, "Dan Ballance"  wrote:
>
>> I'm no expert, but here's something to get you started while you wait for
>> more experienced replies. Check for root kits:
>>
>> sudo apt-get install rkhunter
>> sudo rkhunter --update
>> sudo rkhunter --check
>>
>> On 5 December 2011 10:44, Lucio Crusca  wrote:
>>
>>> Hello *,
>>>
>>> I'm not new here, but I've mostly lurked all the time through gmane. I
>>> never
>>> believed it could happen to me until it actually happened: they
>>> compromized
>>> one of my servers. It's a Ubuntu 10.04 server with all security patches
>>> regularly applied. I'm inclined to believe they used some hole in the web
>>> application, which is a old customized Virtuemart version (1.1.3), which
>>> is
>>> not upgradable because of the invasive code customizations (I'm not the
>>> author of that code, so I have no clue about what had been changed back
>>> then).
>>>
>>> Now the problem for me is to track down the security hole. Here is the
>>> email
>>> my provider received and forwarded to me:
>>>
>>> > Subject: ISP Report; botnet activity on irc.undernet.org
>>> > [...]
>>> >
>>> > Hello, I am an operator on the irc chat network,
>>> > irc.undernet.org and i would like you to investigate the
>>> > owner of the Ip addresses that are listed at the foot of this
>>> > email.
>>> >
>>> > This/These host(s) have likely been compromised, and had an
>>> > altered/rogue process installed on it, and was part of a botnet
>>> > that was found on our network.
>>> >
>>> > The exploit or compromise running on this system is likely
>>> > to be an irc bot. Can you please alert the person who is
>>> > responsible, for its security to patch/upgrade, remove the
>>> > irc process and secure their system.
>>> >
>>> > = Unix System owners =
>>> > A favourite place for hiding the bot(s) is in tmp
>>> > and in /var/tmp/ or /dev/shm/ or in a users home directory
>>> > sometimes it may be hidden like /tmp/".  ."/ or similar.
>>> >
>>> > The bot files can usually be found by running these one line
>>> > commands as the root user.
>>> >
>>> > find / -exec grep -l "undernet" {} +
>>> > find / -exec grep -l "sybnc" {} +
>>> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>>> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>>> >
>>> > netstat -tanp
>>> > lsof -i tcp:
>>> >
>>> > *netstat looking for connections to remote port 6667 or the
>>> > range of ports between 6660-7000 once you find the port you
>>> > can use the command, lsof -i tcp:portnumber to determine
>>> > which process/user it is running under, and terminate it.
>>> >
>>> > = Windows System Owners =
>>> > most windows bots are mIRC scripted bots and generally
>>> > need a file called mirc.ini to run, you should search for
>>> > this file. 

Re: [Full-disclosure] one of my servers has been compromized

[Larry W. Cashdollar]

Hi,I'd say tell your boss your application has been compromised right away.    Tell them you'll need to rebuild the entire system from scratch and they'll need to either devise an upgrade path for virtuemart or find a new ecommerce solution.You can't trust a system once it has been compromised. -- larry C$On Dec 05, 2011, at 05:35 AM, mitchell  wrote:Hi,Here is what you generally need to do in such cases.1. Suspend the webapp until you investigate.2. Check the web server logs for unusual entries and identify the entry point. You should have noticed the timestamp of the Perl script in the /tmp directory. Try looking for entries around this timestamp. Usually, the timestamp would be your starting point. The files created in the /tmp/.m directory were most probably put there via an HTTP request.3. Find all locations writable by www-data.4. Touch a file with a timestamp = the date of the incident.5. Find all files -newer than the file you `touch`-ed in the locations writable by www-data.6. Identify any malicious files in the returned listing.7. Stat the malicious files and log the data.8. Quarantine / remove the malicious file(s).9. *Patch* the Web application.10. Check the application code for other vulnerabilities.11. Allow access to the Webapp.12. Check for updates for the application regularly and apply fixes for any security issues if full upgrade is not possible.Unless you patch the application, the issue will most certainly re-occur. Regards,--# mOn Mon, Dec 5, 2011 at 12:44, Lucio Crusca  wrote:Hello *,  I'm not new here, but I've mostly lurked all the time through gmane. I never believed it could happen to me until it actually happened: they compromized one of my servers. It's a Ubuntu 10.04 server with all security patches regularly applied. I'm inclined to believe they used some hole in the web application, which is a old customized Virtuemart version (1.1.3), which is not upgradable because of the invasive code customizations (I'm not the author of that code, so I have no clue about what had been changed back then).  Now the problem for me is to track down the security hole. Here is the email my provider received and forwarded to me:  > Subject: ISP Report; botnet activity on irc.undernet.org > [...] > > Hello, I am an operator on the irc chat network, > irc.undernet.org and i would like you to investigate the > owner of the Ip addresses that are listed at the foot of this > email. > > This/These host(s) have likely been compromised, and had an > altered/rogue process installed on it, and was part of a botnet > that was found on our network. > > The exploit or compromise running on this system is likely > to be an irc bot. Can you please alert the person who is > responsible, for its security to patch/upgrade, remove the > irc process and secure their system. > > = Unix System owners = > A favourite place for hiding the bot(s) is in tmp > and in /var/tmp/ or /dev/shm/ or in a users home directory > sometimes it may be hidden like /tmp/".  ."/ or similar. > > The bot files can usually be found by running these one line > commands as the root user. > > find / -exec grep -l "undernet" {} + > find / -exec grep -l "sybnc" {} + > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq > > netstat -tanp > lsof -i tcp: > > *netstat looking for connections to remote port 6667 or the > range of ports between 6660-7000 once you find the port you > can use the command, lsof -i tcp:portnumber to determine > which process/user it is running under, and terminate it. > > = Windows System Owners = > most windows bots are mIRC scripted bots and generally > need a file called mirc.ini to run, you should search for > this file. Run a good antivirus scanner and firewall. > > This Ip/host may be removed from our Irc network due to the > risks it presents to our users. > > Should you need any help with removing the files or bot > process, feel free to contact me by mail or on our network, > which you connect to using any irc client and issuing > /server irc.undernet.org > > I look forward to your reply > Scot > > * Affected host/IPs, capture time is GMT+1: United kingdom > and servers they were connected to. > > Please note: when resolving server names to IP Addresses > that all our servers end with .undernet.org (for example) > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org > > Important: If you reply to this mail needing further > information, please leave this mail intact, or supply us > with the IP Address(es) in question, as we reference these > mails by the unique IP Address > > Time of Capture: DECEMBER 3, 2011 10:03:48 PM > > List of IP address(es) and server it connected to: > my.server.ip.address (CHICAGO.IL.US > > BUDAPEST.HU.EU > > MONTREAL.QC.CA.undernet.org) >  I've run the "find" commands and found a number of file with the first "find", under /tmp/.m  Deleted them, looked up remote connections with n

Re: [Full-disclosure] one of my servers has been compromized

[Larry W. Cashdollar]

I'd check these too:http://virtuemart.net/security-bulletinsOn Dec 05, 2011, at 05:35 AM, mitchell  wrote:Hi,Here is what you generally need to do in such cases.1. Suspend the webapp until you investigate.2. Check the web server logs for unusual entries and identify the entry point. You should have noticed the timestamp of the Perl script in the /tmp directory. Try looking for entries around this timestamp. Usually, the timestamp would be your starting point. The files created in the /tmp/.m directory were most probably put there via an HTTP request.3. Find all locations writable by www-data.4. Touch a file with a timestamp = the date of the incident.5. Find all files -newer than the file you `touch`-ed in the locations writable by www-data.6. Identify any malicious files in the returned listing.7. Stat the malicious files and log the data.8. Quarantine / remove the malicious file(s).9. *Patch* the Web application.10. Check the application code for other vulnerabilities.11. Allow access to the Webapp.12. Check for updates for the application regularly and apply fixes for any security issues if full upgrade is not possible.Unless you patch the application, the issue will most certainly re-occur. Regards,--# mOn Mon, Dec 5, 2011 at 12:44, Lucio Crusca  wrote:Hello *,  I'm not new here, but I've mostly lurked all the time through gmane. I never believed it could happen to me until it actually happened: they compromized one of my servers. It's a Ubuntu 10.04 server with all security patches regularly applied. I'm inclined to believe they used some hole in the web application, which is a old customized Virtuemart version (1.1.3), which is not upgradable because of the invasive code customizations (I'm not the author of that code, so I have no clue about what had been changed back then).  Now the problem for me is to track down the security hole. Here is the email my provider received and forwarded to me:  > Subject: ISP Report; botnet activity on irc.undernet.org > [...] > > Hello, I am an operator on the irc chat network, > irc.undernet.org and i would like you to investigate the > owner of the Ip addresses that are listed at the foot of this > email. > > This/These host(s) have likely been compromised, and had an > altered/rogue process installed on it, and was part of a botnet > that was found on our network. > > The exploit or compromise running on this system is likely > to be an irc bot. Can you please alert the person who is > responsible, for its security to patch/upgrade, remove the > irc process and secure their system. > > = Unix System owners = > A favourite place for hiding the bot(s) is in tmp > and in /var/tmp/ or /dev/shm/ or in a users home directory > sometimes it may be hidden like /tmp/".  ."/ or similar. > > The bot files can usually be found by running these one line > commands as the root user. > > find / -exec grep -l "undernet" {} + > find / -exec grep -l "sybnc" {} + > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq > > netstat -tanp > lsof -i tcp: > > *netstat looking for connections to remote port 6667 or the > range of ports between 6660-7000 once you find the port you > can use the command, lsof -i tcp:portnumber to determine > which process/user it is running under, and terminate it. > > = Windows System Owners = > most windows bots are mIRC scripted bots and generally > need a file called mirc.ini to run, you should search for > this file. Run a good antivirus scanner and firewall. > > This Ip/host may be removed from our Irc network due to the > risks it presents to our users. > > Should you need any help with removing the files or bot > process, feel free to contact me by mail or on our network, > which you connect to using any irc client and issuing > /server irc.undernet.org > > I look forward to your reply > Scot > > * Affected host/IPs, capture time is GMT+1: United kingdom > and servers they were connected to. > > Please note: when resolving server names to IP Addresses > that all our servers end with .undernet.org (for example) > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org > > Important: If you reply to this mail needing further > information, please leave this mail intact, or supply us > with the IP Address(es) in question, as we reference these > mails by the unique IP Address > > Time of Capture: DECEMBER 3, 2011 10:03:48 PM > > List of IP address(es) and server it connected to: > my.server.ip.address (CHICAGO.IL.US > > BUDAPEST.HU.EU > > MONTREAL.QC.CA.undernet.org) >  I've run the "find" commands and found a number of file with the first "find", under /tmp/.m  Deleted them, looked up remote connections with netstat, killed perl processes that where trying to connect to port 6959 (only trying because I've now set up iptables so that they actually can't), but those processes kept spawning. Checked crontab of www-data, found the launcher, removed it.  No

Re: [Full-disclosure] one of my servers has been compromized

[Josh Yavor]

Really enjoying following this thread. Any one have thoughts/opinions
on APF as an option to prevent this in the future?

On Mon, Dec 5, 2011 at 11:18 AM, Michael Wood  wrote:
> Awesome tips guys...
>
> On Dec 5, 2011 11:01 AM, "John Jacobs"  wrote:
>>
>>
>> > 2. Do you think said phpmyadmin vulns are reasonable attack vectors in
>> > my
>> > case?
>>
>> I do, I believe this is to be the initial infection vector.  Scanning for
>> PHPMyAdmin is often and frequent and since it's likely that it was present
>> in it's default (or one of the default) URIs discovery is likely.  There are
>> a plethora of scanners out there which look for PHPMyAdmin specifically and
>> add to the Internet noise-floor.
>>
>> You are taking the correct steps with the egress firewall policy.
>>
>> Forward-going, I think it may be valuable to consider:
>>
>> 1) Leveraging AppArmor and creating an enforcing profile for Apache; one
>> that controls by extension or path, what the HTTPd can write to or access.
>> Be strict but sane.
>> 2) Consider chrooting Apache via the 'chroot' directive for Apache (no
>> more mod_chroot required).
>> 3) Consider a strict ingress and egress firewall which would have prevent
>> the egress connection to the IRCd.
>> 4) Remain up to date; perhaps cron 'apt-get clean all; apt-get update;
>> apt-get -t lucid-security -y dist-upgrade' (I believe the security channel
>> is correct)
>> 5) Consider sane php.ini values and leverage Suhosin (plugin) as well
>> (http://www.hardened-php.net/suhosin/index.html); disallow url_fopen and
>> url_include.  Disallow the exec(), system(), passthru(), etc commands if
>> possible.  url_fopen() will thwart RFI.  LFI should be thwarted by a sane
>> AppArmor profile.
>> 6) Restrict access to PHPMyAdmin based on authentication or remove it's
>> access entirely.
>> 7) Consider leveraging something like Fail2ban against Apache's error and
>> access logs looking for excessive high-frequency HTTP 404, 403, or 500
>> errors as these are indicative of scanning.  This is a great tool to stop
>> Web-app scanning.
>> 8) As you've already done with SSH, move it from TCP 22, PermitRootLogin
>> no, and disable password authentication using key-based authentication.
>> 9) Using OSSEC-HIDS (http://www.ossec.net/) with inotify() to watch
>> changes to your system and Apache directories including those that are HTTP
>> writable.
>> 10) Mount /tmp noexec,nosuid,nodev as others have recommended.
>> 11) Optionally use mod_security with a tuned ruleset or another WAF.
>>
>> I find #7 to be extremely helpful.  Feel free to hit me up for additional
>> clarification if needed.  I wish you the best, remember that
>> defense-in-depth is the best approach here.
>>
>> This is a good list-discussion as it is likely to yield many valuable ways
>> to correctly secure web applications.  Potentially any one of the
>> suggestiosn in #1, #2, #3, #4, #5, #6, #7, and #10 would have saved your
>> box.
>>
>> I hope this helped,
>> John
>>
>> ___
>> Full-Disclosure - We believe in it.
>> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
>> Hosted and sponsored by Secunia - http://secunia.com/
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[sam]

Very useful john jacob ... really helpful.
do you maintaine your blog or any other resource you want to share with us.
thanx a ton .

On Mon, Dec 5, 2011 at 8:18 AM, Michael Wood  wrote:
> Awesome tips guys...
>
> On Dec 5, 2011 11:01 AM, "John Jacobs"  wrote:
>>
>>
>> > 2. Do you think said phpmyadmin vulns are reasonable attack vectors in
>> > my
>> > case?
>>
>> I do, I believe this is to be the initial infection vector.  Scanning for
>> PHPMyAdmin is often and frequent and since it's likely that it was present
>> in it's default (or one of the default) URIs discovery is likely.  There are
>> a plethora of scanners out there which look for PHPMyAdmin specifically and
>> add to the Internet noise-floor.
>>
>> You are taking the correct steps with the egress firewall policy.
>>
>> Forward-going, I think it may be valuable to consider:
>>
>> 1) Leveraging AppArmor and creating an enforcing profile for Apache; one
>> that controls by extension or path, what the HTTPd can write to or access.
>> Be strict but sane.
>> 2) Consider chrooting Apache via the 'chroot' directive for Apache (no
>> more mod_chroot required).
>> 3) Consider a strict ingress and egress firewall which would have prevent
>> the egress connection to the IRCd.
>> 4) Remain up to date; perhaps cron 'apt-get clean all; apt-get update;
>> apt-get -t lucid-security -y dist-upgrade' (I believe the security channel
>> is correct)
>> 5) Consider sane php.ini values and leverage Suhosin (plugin) as well
>> (http://www.hardened-php.net/suhosin/index.html); disallow url_fopen and
>> url_include.  Disallow the exec(), system(), passthru(), etc commands if
>> possible.  url_fopen() will thwart RFI.  LFI should be thwarted by a sane
>> AppArmor profile.
>> 6) Restrict access to PHPMyAdmin based on authentication or remove it's
>> access entirely.
>> 7) Consider leveraging something like Fail2ban against Apache's error and
>> access logs looking for excessive high-frequency HTTP 404, 403, or 500
>> errors as these are indicative of scanning.  This is a great tool to stop
>> Web-app scanning.
>> 8) As you've already done with SSH, move it from TCP 22, PermitRootLogin
>> no, and disable password authentication using key-based authentication.
>> 9) Using OSSEC-HIDS (http://www.ossec.net/) with inotify() to watch
>> changes to your system and Apache directories including those that are HTTP
>> writable.
>> 10) Mount /tmp noexec,nosuid,nodev as others have recommended.
>> 11) Optionally use mod_security with a tuned ruleset or another WAF.
>>
>> I find #7 to be extremely helpful.  Feel free to hit me up for additional
>> clarification if needed.  I wish you the best, remember that
>> defense-in-depth is the best approach here.
>>
>> This is a good list-discussion as it is likely to yield many valuable ways
>> to correctly secure web applications.  Potentially any one of the
>> suggestiosn in #1, #2, #3, #4, #5, #6, #7, and #10 would have saved your
>> box.
>>
>> I hope this helped,
>> John
>>
>> ___
>> Full-Disclosure - We believe in it.
>> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
>> Hosted and sponsored by Secunia - http://secunia.com/
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/



-- 
Best Regards,
Suresh Kumar Prajapati
Linux System Admin
E-mail: er.sureshprajap...@gmail.com
Mob No: +91-8800920533

Theory is when you know all and nothing works. Practice is when all
works and nobody knows why. In this case we have put together theory
and practice: nothing works... and nobody knows why!

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Dan Ballance]

Thanks for that. I've learnt a lot from this thread. Peace to you all.

On 5 December 2011 20:11, Gage Bystrom  wrote:

> Tripwire is awesome for many reasons. The original use of rootkit
> detection is no longer one of them. It was used back when userland rootkits
> were big, it has zero effect on kernel rootkits. That being said you can
> use it to watch other critical files for improper access. Keep tabs on your
> cron files, configs, and your web pages(why crack password hashes when you
> can force the login script to hand deliver the plaintext?), etc.
>
> Afaik, rootkit detection has made practically no progress. In part because
> of several advancements in rootkits and also in part of the overzealous
> trend in reimaging even the slightest compromised box.
>
> That being said a modern rootkit detector would need to be installed first
> and watch for suspicious behavior, such as attempting to hook system calls
> and watching kernel module loading. Nothing but the kernels mod loader
> should have a legitimate reason to change the list of kernel modules.
>
> In OPs case he really shouldn't need to worry about a rootkit. This wasn't
> a targeted attack. If it was, or even if the script tried to load a rootkit
> then he wouldn't even have seen the questionable files in the first place,
> nor the processes, or the loader. He wouldn't have even been able to grep
> them. Also deploying a rootkit as part of a serious attack is annoying. You
> fuck up one thing and not only have you lost the box, but left a strew of
> evidence that you were trying to hide. Not to mention public rootkits never
> really caught up with the kernel developments. Most rootkits in place are
> still targetting 2.4 kernels because only smart dedicated attackers would
> have the skills to develop and deploy a modern rootkit for a modern kernel.
> Such an attacker wouldn't make so many rookie/nonchalant mistakes as the
> attacker on the ops box did. At most he needs to be concerned if he had
> caught all the backdoors or not. Considering he doesn't realistically need
> to worry about a rootkit(remember, rootkits are annoying, usually easier
> and more practical to stay quiet, get want you want, and leave quietly),
> then he could watch for outgoing connections while monitoring any new open
> ports that have spawned. I would suggest iptables but the OP stated he
> doesn't own the server and has no root access. Sure there are many clever
> ways to reserve access but they all start falling apart as long as your
> waiting and watching for them to make a peep.
>  On Dec 5, 2011 5:53 AM, "Dan Ballance"  wrote:
>
>> Thanks for the heads-up on rkhunter Gage.
>>
>> Is there anything else out there atm that works as a reasonable root kit
>> detector or is such a thing considered impossible now? I realise a skilled
>> attack will be able to bury itself without a trace, but I'm thinking of
>> something that can be used in less skilled breaches such as the one thought
>> to have been identified in this thread. Sometimes something imperfect is
>> still better than nothing I think.
>>
>> Also, am I correct to think that using something like tripwire is the
>> best way to detect root kits properly, but that it obviously needs
>> installing when the box is fresh and before it has been physically
>> connected to a network?
>>
>> thanks to everyone for their valuable contributions here - much
>> appreciated!
>>
>> dan :)
>>
>> On 5 December 2011 11:13, Gage Bystrom  wrote:
>>
>>> If it was a rootkit then trying to run the outdated rkhunter would be a
>>> moot point. Whatever seizes the kernel first wins, hands down.
>>>
>>> Fortunately for him, since the bot was so easy to find in the first
>>> place and such a simple way of maintaining it, the box was clearly seized
>>> by someone who didn't give a rats ass about it. Probably a skiddie or an
>>> automated attack to begin with.
>>>
>>> As for plugging any security holes, check your httpd error logs. If you
>>> noted down the time of the bot files creation date you would look around
>>> the same time for suspicious log entries. If they were as careless in
>>> scrubbing the logs as they were holding the box it would give you a look
>>> into how it may have been compromised. If you're getting things like
>>> ../.../../../../etc/passwd then some sort of lfi vuln was likely exploited,
>>> start grepping your php files for stuff like include(), or if you're
>>> getting something like into outfile then check your mysql user permissions
>>> and don't let it have file perms, and then start grepping down for sql
>>> vulns.
>>>
>>> If it comes down to being too much of a hassle to get all the obvious
>>> vulns at least then go to your boss, admit there is an issue and that time
>>> needs to be taken to remove such legacy code as this could have been a far
>>> worse incident if it had been more targetted and the end goal wasn't a
>>> botnet.
>>>  On Dec 5, 2011 3:02 AM, "Dan Ballance"  wrote:
>>>
 I'm no expert, b

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

> Very useful john jacob ... really helpful.
> do you maintaine your blog or any other resource you want to share with us.
> thanx a ton .

Thank you for the kind words and I consider it an honor to have been helpful.  
I do not have a blog.  I have enjoyed this thread, sharing what I know, and 
learning from it.  Perhaps one day I will start a blog and it may be a house 
along the road where I can share what I have learned.

Thanks,
John
  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Guillaume Friloux]

On 05/12/2011 18:20, John Jacobs wrote:
> Tim, while I do believe there is some truth in what you are saying here, I 
> respectfully disagree in that this tends to be a run-of-the-mill IRC bot as 
> evidenced by the Undernet advisory.  This looks like a skiddie-de-jour attack 
> against PHPMyAdmin and nothing to be concerned with regarding cloning disk 
> images and full forensics.  I do respect your input and thoughts though for a 
> more targeted attack; not an IRC bot in /tmp.
>
> That being said, I strongly believe in preserving bash_history as well as 
> vital log data.  It's best/wise to ship this off to a separate Syslog server. 
>  If you're paranoid turn up stunnel between the devices.  For example and as 
> evidenced by many of the documented attacks here purging of bash_history is 
> common ala 'history -c' after fun.  To thwart this I like the idea of logging 
> to syslog often, ensure permissions are strict for the syslog messages, and 
> shipping the syslog data off to a separate box.  I like to:
>
> 1) Generate an E-Mail alert when someone logs in, by adjusting 
> /etc/bash.bashrc (or similar based on distribution) to:
>
> #Email alert for login
> echo -e "Subject: Login from $(/usr/bin/whoami) on $(/bin/hostname) at 
> $(/bin/date)\n\n$(/usr/bin/last -ian 10)\n"|/usr/sbin/sendmail 
> recipi...@example.com
>
> 2) Preserve, via Syslog, commands executed at the prompt, by adjusting 
> /etc/profile.  Adjust /etc/syslog.conf or /etc/rsyslog.conf to forward these 
> syslog messages off-box to another asset.  If you're paranoid use stunnel.
>
> export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo "$$ $USER 
> $(history 1)"|/usr/bin/logger -p user.alert -t bash_history'
> readonly PROMPT_COMMAND
>
> 3) Preserve bash_history by adjusting /etc/profile:
>
> #Secure the Bash History
> export HISTSIZE=1500
> export HISTCONTROL=''
> export HISTIGNORE=''
> export HISTTIMEFORMAT='%F %T '
> readonly HISTFILE
> readonly HISTFILESIZE
> readonly HISTSIZE
> readonly HISTCONTROL
> readonly HISTIGNORE
> readonly HISTTIMEFORMAT
>
> 4) Optionally use chattr to set ~/.bash_history to append-only:
>
>   #Secure .bash_history (poke fun of the while subshell if you wish)
>   /usr/bin/find / -maxdepth 3|/bin/grep -i bash_history|while read line; do 
> /usr/bin/chattr +a "$line"; done
>
> 5) Use of an IP Recorder, something like daemonlogger, in ring-buffer mode, 
> as a way to record all ingress/egress traffic using a percentage of the disk. 
>  See http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
>
> I am eager to hear any additional thoughts or methods for security 
> information such as this.
>
> Thanks,
> John
>
Hello,
You have been giving a lot of good practices, and i would like to add some :
Logs can be easilly compromised. Here, we send store logs locally, but
we also send them over the network to another server that store them “as
is” but also store them in a database (a “locked from the outside”
elasticsearch).

We also log every PTS session using scripts, so even if there is more
than one PTS open, we get accurate logs (we choosed to use the 'script'
proggy to record sessions, and be able to replay them later, and we
automaticcally archive them as sessions get closed). For this, you can
do it in bashrc, also monitor from ssh public keys, you can also replace
/bin/sh with a logger generating unique logs indexed in an external DB.

Logs cleaners will not be able to erase that stuff.
<>

signature.asc
Description: OpenPGP digital signature
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

Gage Bystrom wrote:
> I would suggest iptables but the OP stated he doesn't own the
> server and has no root access. 

If I ever stated that, it means I misused my poor english for sure... I DO 
have root access and I DO own the server, where the server means the *guest* 
OpenVZ instance. I DID configure iptables yesterday in order to block 
outgoing connections. What I can't do is upgrading the kernel because OpenVZ 
is a limited "paravirtualization" system where the guest kernel it's more 
like a stub on top of the only shared host kernel. I have no control over 
the host kernel, so I can't upgrade it.

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[BH]

I'm not sure if this has been said in this thread yet, but is it
possible the host O/S was compromised? I have not used OpenVZ but I
assume it's the same as Virtuozzo in the respect that you can just
'vzctl enter ' to get a root shell inside the container with no
password (assuming you have control of the parent O/S). I have come
across a Virtuozzo server before that has been compromised. When I had a
look, as far as I could see the only thing they appeared to do was copy
a Perl script to each container, execute the script then delete it.


On 6/12/2011 5:17 PM, Lucio Crusca wrote:
> Gage Bystrom wrote:
>> I would suggest iptables but the OP stated he doesn't own the
>> server and has no root access. 
> If I ever stated that, it means I misused my poor english for sure... I DO 
> have root access and I DO own the server, where the server means the *guest* 
> OpenVZ instance. I DID configure iptables yesterday in order to block 
> outgoing connections. What I can't do is upgrading the kernel because OpenVZ 
> is a limited "paravirtualization" system where the guest kernel it's more 
> like a stub on top of the only shared host kernel. I have no control over 
> the host kernel, so I can't upgrade it.
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

Ahh I see. Then yeah I would advise using iptables to deny as much outgoing
traffic as possible and set up the chain so that all attempted traffic
statistics get logged. Back that up with denying as much incoming traffic
as possible. Then monitor for any spawning services with netstat.

Assuming no rootkit was involved(and I explained how unlikely that'd be),
then any incoming connection to seize back the box via a backdoor would
rely on a process spawning a daemon which will be caught. Any connect back
backdoor will not only be stopped immediately but since you can hone in via
the logged statistics you can know what remote port its looking for. Then
you simply watch netstat for outgoing connections to said port and you got
em.

The 'tricky' part is if they have some sort of ssh based access. Contrary
to some previous suggestions locking down bash and logging users is neigh
useless if the attacker has remotely browsed the man pages for the
client(here's a hint, you don't need to 'login' to get a shell as long as
you don't mind not having a tty). Instead the remedy is fairly simple.
Reinstall ssh(preferably from source) and then change every user password.
If the daemon was changed, its now safe. If a password was compromised its
now useless.

No matter how you look at it, if no kernel rootkit was in place then any
backdoor becomes fudged. From there its a simple matter of wiping /tmp of
any code scripts and then dealing with the matter of the vulnerable web
pages.

Cause yes, even unusual side channels relying on icmp or dns queries become
useless. Iptables will still record the unusual jump in stats for those and
they've just handed you and potential authorities either their home box(if
they're morons), or revealed another compromised server. Which means its a
win win even if they tried a 'clever' trick like that.

Set the right options, plug the holes, and relish in the fact they weren't
serious about your box and you will be just find.
On Dec 6, 2011 1:18 AM, "Lucio Crusca"  wrote:

> Gage Bystrom wrote:
> > I would suggest iptables but the OP stated he doesn't own the
> > server and has no root access.
>
> If I ever stated that, it means I misused my poor english for sure... I DO
> have root access and I DO own the server, where the server means the
> *guest*
> OpenVZ instance. I DID configure iptables yesterday in order to block
> outgoing connections. What I can't do is upgrading the kernel because
> OpenVZ
> is a limited "paravirtualization" system where the guest kernel it's more
> like a stub on top of the only shared host kernel. I have no control over
> the host kernel, so I can't upgrade it.
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Lucio Crusca]

BH wrote:

> I'm not sure if this has been said in this thread yet, but is it
> possible the host O/S was compromised? 

Nothing is impossible, security wise. However I'd talk about likelihood 
instead. I own two other OpenVZ containers hosted in the same host OS. They 
haven't been compromised, though they're very similar systems (Debian based 
instead of Ubuntu).
The one that has been compromised is the only one having a online shop and 
greater network traffic.



___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Kerem Erciyes]

I regularly use iftop, netstat and htop to see what is going on on my
servers.
I have found that raw information always helps the best in determining
acitve compromised systems.

Kerem

On Tue, Dec 6, 2011 at 11:55 AM, Lucio Crusca  wrote:

> BH wrote:
>
> > I'm not sure if this has been said in this thread yet, but is it
> > possible the host O/S was compromised?
>
> Nothing is impossible, security wise. However I'd talk about likelihood
> instead. I own two other OpenVZ containers hosted in the same host OS. They
> haven't been compromised, though they're very similar systems (Debian based
> instead of Ubuntu).
> The one that has been compromised is the only one having a online shop and
> greater network traffic.
>
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>



-- 
Kerem Erciyes - Sistem Danismani
http://keremerciyes.com
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Valdis . Kletnieks]

On Mon, 05 Dec 2011 13:53:21 GMT, Dan Ballance said:

> Also, am I correct to think that using something like tripwire is the best
> way to detect root kits properly, but that it obviously needs installing
> when the box is fresh and before it has been physically connected to a
> network?

tripwire needs to be installed on a known-good system.  This is obviously
*easier* before you connect to a network, but you certainly should *not*
say "zomg I connected it to a network for 35 seconds, I'll not be able to
use tripwire ever again".

The bigger hassle with tripwire is patching your system - the recommended
way is to:

1) re-run a tripwire report and verify your system looks OK.
2) patch
3) re-run tripwire to report all changed files
4) Verify that only things changed are files you intended to patch, 
   4a) and that you got the versions you intended
5) re-re-run tripwire to commit the new values to the tripwire database.

Note that 4a is often harder than it looks - even if you have a GPG-signed
RPM, there's often scripts run at install/update time that screw around with
other files (I'm looking at you, every program that integrates itself into
Gnome and scribbles into /etc/gconf ;)


pgpJsmlkveOc1.pgp
Description: PGP signature
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Valdis . Kletnieks]

On Mon, 05 Dec 2011 19:04:02 +0100, Lucio Crusca said:

> Using dd on /dev/mem and piping results through netcat it's not that 
> difficult, and a bit of google explains how to do it the right way, but in 
> my case there are two other problems:

Note that the effectiveness and safety of doing a dd on /dev/mem is highly
dependent on kernel configuration and release.  See CONFIG_STRICT_DEVMEM
and CONFIG_DEVKMEM for details.  Also, some I/O hardware behaves strangely
or badly when you accidentally read from memory-mapped control registers (which
was one of the reasons for STRICT_DEVMEM being added).


pgpDwfrgIzOAj.pgp
Description: PGP signature
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Paul Schmehl]

A "poor man's" root kit detector is to take md5sums of critical system 
binaries (you'd have to redo these after patching), and keep the list on an 
inaccessible media (such as a thumb drive).  If you think the system is 
compromised, run md5sum against those files, and you will quickly know. 
You could even keep statically compiled copies on the thumb drive to use in 
an investigation.

Start with things you use to check for problems; ls, ps, fstat, sockstat, 
netstat, wtmp, nc, sshd, etc.

It would be fairly trivial to create a simple shell script that would 
compare the md5sums of system binaries to the saved copies and flag 
anomalies.

And, of course, if you can take a system offline, there are a number of 
bootable security distros that allow you to do extensive analysis of 
systems.



In general, on Unix systems, look for oddly named directories in odd places 
(like /tmp, /dev, etc. and review logs that have been syslogged elsewhere 
for telltale signs of compromise.

It's surprising how few times the shell history logs get wiped, but there 
are some kits out there that do that for you.  Web apps and improper 
permissions (world writeable) are the two most frequent causes of 
compromises that I've seen on Unix systems.

--On December 5, 2011 1:53:21 PM + Dan Ballance 
 wrote:

> Thanks for the heads-up on rkhunter Gage.
>
>
> Is there anything else out there atm that works as a reasonable root kit
> detector or is such a thing considered impossible now? I realise a
> skilled attack will be able to bury itself without a trace, but I'm
> thinking of something that can be used in less skilled breaches such as
> the one thought to have been identified in this thread. Sometimes
> something imperfect is still better than nothing I think.
>
>
> Also, am I correct to think that using something like tripwire is the
> best way to detect root kits properly, but that it obviously needs
> installing when the box is fresh and before it has been physically
> connected to a network?
>
>
> thanks to everyone for their valuable contributions here - much
> appreciated!
>
>
> dan :)
>
>
>
> On 5 December 2011 11:13, Gage Bystrom  wrote:
>
>
> If it was a rootkit then trying to run the outdated rkhunter would be a
> moot point. Whatever seizes the kernel first wins, hands down.
>
> Fortunately for him, since the bot was so easy to find in the first place
> and such a simple way of maintaining it, the box was clearly seized by
> someone who didn't give a rats ass about it. Probably a skiddie or an
> automated attack to begin with.
>
> As for plugging any security holes, check your httpd error logs. If you
> noted down the time of the bot files creation date you would look around
> the same time for suspicious log entries. If they were as careless in
> scrubbing the logs as they were holding the box it would give you a look
> into how it may have been compromised. If you're getting things like
> ../.../../../../etc/passwd then some sort of lfi vuln was likely
> exploited, start grepping your php files for stuff like include(), or if
> you're getting something like into outfile then check your mysql user
> permissions and don't let it have file perms, and then start grepping
> down for sql vulns.
>
> If it comes down to being too much of a hassle to get all the obvious
> vulns at least then go to your boss, admit there is an issue and that
> time needs to be taken to remove such legacy code as this could have been
> a far worse incident if it had been more targetted and the end goal
> wasn't a botnet.
>
>
> On Dec 5, 2011 3:02 AM, "Dan Ballance"  wrote:
>
> I'm no expert, but here's something to get you started while you wait
> for more experienced replies. Check for root kits:
>
>
> sudo apt-get install rkhunter
> sudo rkhunter --update
> sudo rkhunter --check
>
> On 5 December 2011 10:44, Lucio Crusca  wrote:
>
> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I
> never
> believed it could happen to me until it actually happened: they
> compromized
> one of my servers. It's a Ubuntu 10.04 server with all security patches
> regularly applied. I'm inclined to believe they used some hole in the web
> application, which is a old customized Virtuemart version (1.1.3), which
> is
> not upgradable because of the invasive code customizations (I'm not the
> author of that code, so I have no clue about what had been changed back
> then).
>
> Now the problem for me is to track down the security hole. Here is the
> email
> my provider received and forwarded to me:
>
>> Subject: ISP Report; botnet activity on irc.undernet.org
>> [...]
>>
>> Hello, I am an operator on the irc chat network,
>> irc.undernet.org and i would like you to investigate the
>> owner of the Ip addresses that are listed at the foot of this
>> email.
>>
>> This/These host(s) have likely been compromised, and had an
>> altered/rog

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

But the problem with that is it is a mentality roughly a little more then a
decade old. What you described is a userland rootkit detector. Problem is
no one uses userland rootkits anymore! Sure there was some recent
development in managed code rootkits but it really hasn't home anywhere and
is Windows centric. Not to mention your plan is totally flawed. You assume
md5sum is safe to begin with. Meaning that to be remotely safe with this
you have to run the script for a livecd. Meaning you have to bring down the
server everytime you suspect you MAY have been compromised. Completely
unacceptable for anyone other than a home user. The only way to circumvent
such issues is to recreate tripwire, in which you still have the same
fundemental problems tripwire has always has.

I know ya mean well, but your first block of advice isn't pratical or
effective. The second one the OP already did so alls well for that.

:)
On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:

> A "poor man's" root kit detector is to take md5sums of critical system
> binaries (you'd have to redo these after patching), and keep the list on an
> inaccessible media (such as a thumb drive).  If you think the system is
> compromised, run md5sum against those files, and you will quickly know. You
> could even keep statically compiled copies on the thumb drive to use in an
> investigation.
>
> Start with things you use to check for problems; ls, ps, fstat, sockstat,
> netstat, wtmp, nc, sshd, etc.
>
> It would be fairly trivial to create a simple shell script that would
> compare the md5sums of system binaries to the saved copies and flag
> anomalies.
>
> And, of course, if you can take a system offline, there are a number of
> bootable security distros that allow you to do extensive analysis of
> systems.
>
>  cd-distros-pen-test-forensics-**recovery/
> >
>
> In general, on Unix systems, look for oddly named directories in odd
> places (like /tmp, /dev, etc. and review logs that have been syslogged
> elsewhere for telltale signs of compromise.
>
> It's surprising how few times the shell history logs get wiped, but there
> are some kits out there that do that for you.  Web apps and improper
> permissions (world writeable) are the two most frequent causes of
> compromises that I've seen on Unix systems.
>
> --On December 5, 2011 1:53:21 PM + Dan Ballance <
> tzewang.do...@gmail.com> wrote:
>
>  Thanks for the heads-up on rkhunter Gage.
>>
>>
>> Is there anything else out there atm that works as a reasonable root kit
>> detector or is such a thing considered impossible now? I realise a
>> skilled attack will be able to bury itself without a trace, but I'm
>> thinking of something that can be used in less skilled breaches such as
>> the one thought to have been identified in this thread. Sometimes
>> something imperfect is still better than nothing I think.
>>
>>
>> Also, am I correct to think that using something like tripwire is the
>> best way to detect root kits properly, but that it obviously needs
>> installing when the box is fresh and before it has been physically
>> connected to a network?
>>
>>
>> thanks to everyone for their valuable contributions here - much
>> appreciated!
>>
>>
>> dan :)
>>
>>
>>
>> On 5 December 2011 11:13, Gage Bystrom  wrote:
>>
>>
>> If it was a rootkit then trying to run the outdated rkhunter would be a
>> moot point. Whatever seizes the kernel first wins, hands down.
>>
>> Fortunately for him, since the bot was so easy to find in the first place
>> and such a simple way of maintaining it, the box was clearly seized by
>> someone who didn't give a rats ass about it. Probably a skiddie or an
>> automated attack to begin with.
>>
>> As for plugging any security holes, check your httpd error logs. If you
>> noted down the time of the bot files creation date you would look around
>> the same time for suspicious log entries. If they were as careless in
>> scrubbing the logs as they were holding the box it would give you a look
>> into how it may have been compromised. If you're getting things like
>> ../.../../../../etc/passwd then some sort of lfi vuln was likely
>> exploited, start grepping your php files for stuff like include(), or if
>> you're getting something like into outfile then check your mysql user
>> permissions and don't let it have file perms, and then start grepping
>> down for sql vulns.
>>
>> If it comes down to being too much of a hassle to get all the obvious
>> vulns at least then go to your boss, admit there is an issue and that
>> time needs to be taken to remove such legacy code as this could have been
>> a far worse incident if it had been more targetted and the end goal
>> wasn't a botnet.
>>
>>
>> On Dec 5, 2011 3:02 AM, "Dan Ballance"  wrote:
>>
>> I'm no expert, but here's something to get you started while you wait
>> for more experi

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

My bad, should have said that you can't trust the md5sum tampering(since
you stated to have a static copy on the flash drive) but you couldn't trust
it since you couldn't trust the system calls.

The immediate moment you have to worry about a legit userland rootkit you
have to worry about a kernel rootkit. After all you have to consider the
psychology of the attacker. If you were to compromise a box, and cared
enough to hide a backdoor they cannot detect without static, write proof
media, then you care enough to go the extra step for a kernel rootkit.
Otherwise you would be spending even more time and effort to make your
userland kit work to satisfaction for a far weaker hold on the box. It
would simply be idiotic. And I think we can all agree that an attacker able
to do either of the above is not an idiot.

On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>
> A "poor man's" root kit detector is to take md5sums of critical system
binaries (you'd have to redo these after patching), and keep the list on an
inaccessible media (such as a thumb drive).  If you think the system is
compromised, run md5sum against those files, and you will quickly know. You
could even keep statically compiled copies on the thumb drive to use in an
investigation.
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Paul Schmehl]

Don't be silly.  You can run static binaries off a thumb drive without 
taking the system down.  And that includes md5sum.  You can put everything, 
including the script, on a thumb drive and be perfectly comfortable that 
the results are reliable, because statically compiled binaries don't use 
system libraries.  As a quick check to see if system binaries have been 
altered, it's hard to beat.

And I wasn't responding to the OP.  I already did that the other day, when 
I told him he had a web app that got hacked and the scripts were run with 
perl, so noexec, nosuid for /tmp wouldn't have helped.

I will agree with you that we don't typically see the old style hacks any 
more, but some hackers are still installing trojaned sshd binaries. 
Nowadays it's mostly app hacks and primarily (as it's always been) to 
Windows boxes.

--On December 6, 2011 10:49:14 AM -0800 Gage Bystrom 
 wrote:

>
> But the problem with that is it is a mentality roughly a little more then
> a decade old. What you described is a userland rootkit detector. Problem
> is no one uses userland rootkits anymore! Sure there was some recent
> development in managed code rootkits but it really hasn't home anywhere
> and is Windows centric. Not to mention your plan is totally flawed. You
> assume md5sum is safe to begin with. Meaning that to be remotely safe
> with this you have to run the script for a livecd. Meaning you have to
> bring down the server everytime you suspect you MAY have been
> compromised. Completely unacceptable for anyone other than a home user.
> The only way to circumvent such issues is to recreate tripwire, in which
> you still have the same fundemental problems tripwire has always has.
>
> I know ya mean well, but your first block of advice isn't pratical or
> effective. The second one the OP already did so alls well for that.
>
> :)
> On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>
> A "poor man's" root kit detector is to take md5sums of critical system
> binaries (you'd have to redo these after patching), and keep the list on
> an inaccessible media (such as a thumb drive).  If you think the system
> is compromised, run md5sum against those files, and you will quickly
> know. You could even keep statically compiled copies on the thumb drive
> to use in an investigation.
>
> Start with things you use to check for problems; ls, ps, fstat, sockstat,
> netstat, wtmp, nc, sshd, etc.
>
> It would be fairly trivial to create a simple shell script that would
> compare the md5sums of system binaries to the saved copies and flag
> anomalies.
>
> And, of course, if you can take a system offline, there are a number of
> bootable security distros that allow you to do extensive analysis of
> systems.
>
>  est-forensics-recovery/>
>
> In general, on Unix systems, look for oddly named directories in odd
> places (like /tmp, /dev, etc. and review logs that have been syslogged
> elsewhere for telltale signs of compromise.
>
> It's surprising how few times the shell history logs get wiped, but there
> are some kits out there that do that for you.  Web apps and improper
> permissions (world writeable) are the two most frequent causes of
> compromises that I've seen on Unix systems.
>
> --On December 5, 2011 1:53:21 PM + Dan Ballance
>  wrote:
>
>
> Thanks for the heads-up on rkhunter Gage.
>
>
> Is there anything else out there atm that works as a reasonable root kit
> detector or is such a thing considered impossible now? I realise a
> skilled attack will be able to bury itself without a trace, but I'm
> thinking of something that can be used in less skilled breaches such as
> the one thought to have been identified in this thread. Sometimes
> something imperfect is still better than nothing I think.
>
>
> Also, am I correct to think that using something like tripwire is the
> best way to detect root kits properly, but that it obviously needs
> installing when the box is fresh and before it has been physically
> connected to a network?
>
>
> thanks to everyone for their valuable contributions here - much
> appreciated!
>
>
> dan :)
>
>
>
> On 5 December 2011 11:13, Gage Bystrom  wrote:
>
>
> If it was a rootkit then trying to run the outdated rkhunter would be a
> moot point. Whatever seizes the kernel first wins, hands down.
>
> Fortunately for him, since the bot was so easy to find in the first place
> and such a simple way of maintaining it, the box was clearly seized by
> someone who didn't give a rats ass about it. Probably a skiddie or an
> automated attack to begin with.
>
> As for plugging any security holes, check your httpd error logs. If you
> noted down the time of the bot files creation date you would look around
> the same time for suspicious log entries. If they were as careless in
> scrubbing the logs as they were holding the box it would give you a look
> into how it may have been compromised. If you're getting things like
> ../.../../../.

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

Those considering Tripwire I would ask they take a look at OSSEC-HIDS; the 
filesystem change notification is outstanding and with inotify() support you 
get immediate notification of changes.  The monitoring and alerting of log 
files is also exceptional.  I am not affiliated with OSSEC in any way.  
http://www.ossec.net/main/about

I would recommend from a "rooting" aspect that kernel module loading be 
disabled after boot.  This is accomplished by removing the CAP_SYS_MODULE 
permission using something like lcap on older systems or by using the sysctl 
value of 'kernel.modules_disabled = 1'.  This can save a box by preventing 
automatic or intentional loading of a vulnerable modules or a module-based 
rootkit.

The sysctl value of 'kernel.panic_on_oops = 1' also is a good idea.

Thanks,
John


  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Charles Morris]

Sorry paul, Gage is right here!

Instead of "silly" maybe more like "correct" :(

On Tue, Dec 6, 2011 at 2:42 PM, Paul Schmehl  wrote:
> Don't be silly.  You can run static binaries off a thumb drive without
> taking the system down.  And that includes md5sum.  You can put everything,
> including the script, on a thumb drive and be perfectly comfortable that
> the results are reliable, because statically compiled binaries don't use
> system libraries.  As a quick check to see if system binaries have been
> altered, it's hard to beat.
>
> And I wasn't responding to the OP.  I already did that the other day, when
> I told him he had a web app that got hacked and the scripts were run with
> perl, so noexec, nosuid for /tmp wouldn't have helped.
>
> I will agree with you that we don't typically see the old style hacks any
> more, but some hackers are still installing trojaned sshd binaries.
> Nowadays it's mostly app hacks and primarily (as it's always been) to
> Windows boxes.
>
> --On December 6, 2011 10:49:14 AM -0800 Gage Bystrom
>  wrote:
>
>>
>> But the problem with that is it is a mentality roughly a little more then
>> a decade old. What you described is a userland rootkit detector. Problem
>> is no one uses userland rootkits anymore! Sure there was some recent
>> development in managed code rootkits but it really hasn't home anywhere
>> and is Windows centric. Not to mention your plan is totally flawed. You
>> assume md5sum is safe to begin with. Meaning that to be remotely safe
>> with this you have to run the script for a livecd. Meaning you have to
>> bring down the server everytime you suspect you MAY have been
>> compromised. Completely unacceptable for anyone other than a home user.
>> The only way to circumvent such issues is to recreate tripwire, in which
>> you still have the same fundemental problems tripwire has always has.
>>
>> I know ya mean well, but your first block of advice isn't pratical or
>> effective. The second one the OP already did so alls well for that.
>>
>> :)
>> On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>>
>> A "poor man's" root kit detector is to take md5sums of critical system
>> binaries (you'd have to redo these after patching), and keep the list on
>> an inaccessible media (such as a thumb drive).  If you think the system
>> is compromised, run md5sum against those files, and you will quickly
>> know. You could even keep statically compiled copies on the thumb drive
>> to use in an investigation.
>>
>> Start with things you use to check for problems; ls, ps, fstat, sockstat,
>> netstat, wtmp, nc, sshd, etc.
>>
>> It would be fairly trivial to create a simple shell script that would
>> compare the md5sums of system binaries to the saved copies and flag
>> anomalies.
>>
>> And, of course, if you can take a system offline, there are a number of
>> bootable security distros that allow you to do extensive analysis of
>> systems.
>>
>> > est-forensics-recovery/>
>>
>> In general, on Unix systems, look for oddly named directories in odd
>> places (like /tmp, /dev, etc. and review logs that have been syslogged
>> elsewhere for telltale signs of compromise.
>>
>> It's surprising how few times the shell history logs get wiped, but there
>> are some kits out there that do that for you.  Web apps and improper
>> permissions (world writeable) are the two most frequent causes of
>> compromises that I've seen on Unix systems.
>>
>> --On December 5, 2011 1:53:21 PM + Dan Ballance
>>  wrote:
>>
>>
>> Thanks for the heads-up on rkhunter Gage.
>>
>>
>> Is there anything else out there atm that works as a reasonable root kit
>> detector or is such a thing considered impossible now? I realise a
>> skilled attack will be able to bury itself without a trace, but I'm
>> thinking of something that can be used in less skilled breaches such as
>> the one thought to have been identified in this thread. Sometimes
>> something imperfect is still better than nothing I think.
>>
>>
>> Also, am I correct to think that using something like tripwire is the
>> best way to detect root kits properly, but that it obviously needs
>> installing when the box is fresh and before it has been physically
>> connected to a network?
>>
>>
>> thanks to everyone for their valuable contributions here - much
>> appreciated!
>>
>>
>> dan :)
>>
>>
>>
>> On 5 December 2011 11:13, Gage Bystrom  wrote:
>>
>>
>> If it was a rootkit then trying to run the outdated rkhunter would be a
>> moot point. Whatever seizes the kernel first wins, hands down.
>>
>> Fortunately for him, since the bot was so easy to find in the first place
>> and such a simple way of maintaining it, the box was clearly seized by
>> someone who didn't give a rats ass about it. Probably a skiddie or an
>> automated attack to begin with.
>>
>> As for plugging any security holes, check your httpd error logs. If you
>> noted down the time of the bot files creation date you wou

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

Sounds pretty neat to be honest. But one thing I'm wondering is that if
they have root, what's stopping them from turning that off? After all they
need root to load the modules in the first place, so if they are in a
position to want to do that, then they are in a position to turn that off.
Granted they probably wouldn't be able to load modules till next boot(at
least Id probably cry if that wasn't the case) but even that can be a win
scenario depending on how they want to execute the final step.
Even in the scenario that they can't unneuter root, that's even a worse
situation. Marginal protection will fall in the face of what needs to be
done. Namely taking that semi permanent step to neuter root could be a
serious pain if suddenly you needed unneutered root again. Would likely
have to take the system down to fix it. Who wants to be the guy to explain
that situation to their boss? Ergo Im pretty doubtful that such an option
couldn't be reversed by root and even if you can, its a pretty large risk
to do so if the server is fairly important.

But the hid itself could be a formidible opponent(I'm going off your word
for this one), and the kernel.panic_on_oops is a good idea since at least
then you can blame the shutdown on an attacker that screwed up and likely
left ample evidence behind.

Basically what I've been trying to say(outside of satisfying my curiosity
about several good points) is that people need to pay attention to who
their 'opponent' is at the moment. You remedy the problem presented by the
opponent with the right response. Anything more is a waste, anything less
is disastrous. Maybe that's why people like to over respond to things, to
be on the safe side, but all that means is you are far more easier to
figure out and predict to a skilled attacker. What's worse of you are
applying fixes to things that are a nonissue to an attacker in the first
place then you are in a false sense of security. Not to say that all the
things mentioned have been bad ideas(I'm endorsing several of the good
ones), but people need to make sure they understand what it is they are
/really/ stopping or mitigating and ask themselves what is it that they
should be stopping or mitigating. Good tools for the wrong job still makes
them wrong tools for ones situation.
On Dec 6, 2011 12:40 PM, "John Jacobs"  wrote:

>
> Those considering Tripwire I would ask they take a look at OSSEC-HIDS; the
> filesystem change notification is outstanding and with inotify() support
> you get immediate notification of changes.  The monitoring and alerting of
> log files is also exceptional.  I am not affiliated with OSSEC in any way.
> http://www.ossec.net/main/about
>
> I would recommend from a "rooting" aspect that kernel module loading be
> disabled after boot.  This is accomplished by removing the CAP_SYS_MODULE
> permission using something like lcap on older systems or by using the
> sysctl value of 'kernel.modules_disabled = 1'.  This can save a box by
> preventing automatic or intentional loading of a vulnerable modules or a
> module-based rootkit.
>
> The sysctl value of 'kernel.panic_on_oops = 1' also is a good idea.
>
> Thanks,
> John
>
>
>
> ___
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Valdis . Kletnieks]

On Tue, 06 Dec 2011 13:20:51 PST, Gage Bystrom said:

> serious pain if suddenly you needed unneutered root again. Would likely
> have to take the system down to fix it. Who wants to be the guy to explain
> that situation to their boss?

If the server is critical enough that you can't take it down to fix it, it 
should have
been in an HA configuration in the first place.  Who wants to be the guy to
explain to the boss that you're dead in the water because of a bad system board?


pgpzIw9oFtImx.pgp
Description: PGP signature
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

Maybe I'm misreading what you said, and if so please correct me, but
whether or not the changes described were applied in the first place or not
wouldn't change the issue that if you needed root unneutered again you
would need to bring down the system. Especially if the change doesn't
really solve anything in the first place and assuming that the change can't
be reversed by root itself;that would defeat the whole purpose of even
using that option in a security context.
On Dec 6, 2011 3:05 PM,  wrote:

> On Tue, 06 Dec 2011 13:20:51 PST, Gage Bystrom said:
>
> > serious pain if suddenly you needed unneutered root again. Would likely
> > have to take the system down to fix it. Who wants to be the guy to
> explain
> > that situation to their boss?
>
> If the server is critical enough that you can't take it down to fix it, it
> should have
> been in an HA configuration in the first place.  Who wants to be the guy to
> explain to the boss that you're dead in the water because of a bad system
> board?
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Valdis . Kletnieks]

On Tue, 06 Dec 2011 15:14:29 PST, Gage Bystrom said:

> Maybe I'm misreading what you said, and if so please correct me, but
> whether or not the changes described were applied in the first place or not
> wouldn't change the issue that if you needed root unneutered again you
> would need to bring down the system. 

Right.

> > serious pain if suddenly you needed unneutered root again. Would likely
> > have to take the system down to fix it. Who wants to be the guy to explain
> > that situation to their boss?

"Hey boss, I *told* you a server failure would cause an outage, and you didn't
fund the purchase of redundant HA gear".  All the explain you should be needing
to do.





pgptAfKO8JGi1.pgp
Description: PGP signature
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Charles Morris]

+1. Except instead of MD5 you want to use something that isn't garbage.

On Tue, Dec 6, 2011 at 1:18 PM, Paul Schmehl  wrote:
> A "poor man's" root kit detector is to take md5sums of critical system
> binaries (you'd have to redo these after patching), and keep the list on an
> inaccessible media (such as a thumb drive).  If you think the system is
> compromised, run md5sum against those files, and you will quickly know.
> You could even keep statically compiled copies on the thumb drive to use in
> an investigation.
>

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[John Jacobs]

> Sounds pretty neat to be honest. But one thing I'm wondering is that if  
> they have root, what's stopping them from turning that off? After all  
> they need root to load the modules in the first place, so if they are  
> in a position to want to do that, then they are in a position to turn  
> that off. Granted they probably wouldn't be able to load modules till  
> next boot(at least Id probably cry if that wasn't the case) but even  
> that can be a win scenario depending on how they want to execute the  

Hi Gage, thank you for your reply.  What you are missing is that by disabling 
kernel module loading you are applying a defense-in-depth strategy to prevent a 
*vulnerable* module from automatically loading in the first place resulting in 
root compromise.  I believe you may not be aware that some modules are loaded 
automatically if a corresponding special device is accessed.  Usually the 
userspace modprobe utility is executed though this can be controlled by the 
value of /proc/sys/kernel/modprobe

Preventing module loading has historically be a valuable way to prevent 
privilege escalation or further root compromise.  Such an example would be the 
'ptrace' exploit, see 
http://www.sans.org/security-resources/malwarefaq/Ptrace.php

Historically there have been various kernel modules that are vulnerable that 
could be loaded by userland non-root programs or access.  Ubuntu likes to 
automatically load modules.

Removing CAP_SYS_MODULE or kernel.modules_disabled=1 make good security sense.  
See 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3d43321b7015387cfebbe26436d0e9d299162ea1
 and 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=25354c4fee169710fd9da15f3bb2abaa24dcf933
 and https://wiki.ubuntu.com/Security/Features#block-modules

The goal here is defense in depth.  Revocation of loading the kernel modules 
cannot be undone unless a system reboot is effected which should be highly 
suspicious.

The goal isn't about protecting ones boxens from a theoretical boogie-man it is 
to leverage all available and sane methods for properly securing ones box.  I 
see no point to to use these options.

Thanks,
John


  
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

Well in that case it becomes fairly sane, assuming you've safeguarded
against the one of the worst case scenario like Valdis previously
mentioned. There are a handful of things I can think of however that
could still work, at which point depends on the attackers goals.

But at that point it'd be a complete loss for the defender, and only a
half victory for the attacker. After all the defender only wins if the
attacker fails to accomplish his goals. The minute he changes his
goals into something you've already been forced to concede to him the
minute he concedes the following: "I'm not getting the kernel" and one
of the following: "I'm not modifying critical files" or "The intrusion
has a high chance of being detected".

But meh, at the point it is an unrealistic scenario anyways. An
attacker who can recognize that, while going through with the
decision, while being able to plan ahead, while being skilled enough
to actually prepare for the plan, while actually encountering the
scenario needed for the per-requisites for this to occur is perhaps
the very scenario behind the "everything can be hacked" possibility we
all inherently recognize.

Oh well, anyways this thread has been very interesting to me, and I'm
glad that I'm not the only one who could see how over-responding would
have been completely useless to the OP. That and he likely has more
than he needs to put an end to his current circumstance.

On Tue, Dec 6, 2011 at 5:33 PM, John Jacobs  wrote:
>
>> Sounds pretty neat to be honest. But one thing I'm wondering is that if
>> they have root, what's stopping them from turning that off? After all
>> they need root to load the modules in the first place, so if they are
>> in a position to want to do that, then they are in a position to turn
>> that off. Granted they probably wouldn't be able to load modules till
>> next boot(at least Id probably cry if that wasn't the case) but even
>> that can be a win scenario depending on how they want to execute the
>
> Hi Gage, thank you for your reply.  What you are missing is that by disabling 
> kernel module loading you are applying a defense-in-depth strategy to prevent 
> a *vulnerable* module from automatically loading in the first place resulting 
> in root compromise.  I believe you may not be aware that some modules are 
> loaded automatically if a corresponding special device is accessed.  Usually 
> the userspace modprobe utility is executed though this can be controlled by 
> the value of /proc/sys/kernel/modprobe
>
> Preventing module loading has historically be a valuable way to prevent 
> privilege escalation or further root compromise.  Such an example would be 
> the 'ptrace' exploit, see 
> http://www.sans.org/security-resources/malwarefaq/Ptrace.php
>
> Historically there have been various kernel modules that are vulnerable that 
> could be loaded by userland non-root programs or access.  Ubuntu likes to 
> automatically load modules.
>
> Removing CAP_SYS_MODULE or kernel.modules_disabled=1 make good security 
> sense.  See 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3d43321b7015387cfebbe26436d0e9d299162ea1
>  and 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=25354c4fee169710fd9da15f3bb2abaa24dcf933
>  and https://wiki.ubuntu.com/Security/Features#block-modules
>
> The goal here is defense in depth.  Revocation of loading the kernel modules 
> cannot be undone unless a system reboot is effected which should be highly 
> suspicious.
>
> The goal isn't about protecting ones boxens from a theoretical boogie-man it 
> is to leverage all available and sane methods for properly securing ones 
> box.  I see no point to to use these options.
>
> Thanks,
> John
>
>
>

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Paul Schmehl]

But whether you have a kernel rootkit or not isn't all that important.  In 
either case the system is going to be doing unwanted things, and you detect 
those unwanted things with the usual system utilities.  If a kernel rootkit 
didn't affect userland, what would be its purpose?  Even to transmit data 
offsite you have to invoke network capabilities, file system capabilities, 
etc.

IOW, it's a distinction without difference.

--On December 6, 2011 11:48:02 AM -0800 Gage Bystrom 
 wrote:

>
>
> My bad, should have said that you can't trust the md5sum tampering(since
> you stated to have a static copy on the flash drive) but you couldn't
> trust it since you couldn't trust the system calls.
>
> The immediate moment you have to worry about a legit userland rootkit you
> have to worry about a kernel rootkit. After all you have to consider the
> psychology of the attacker. If you were to compromise a box, and cared
> enough to hide a backdoor they cannot detect without static, write proof
> media, then you care enough to go the extra step for a kernel rootkit.
> Otherwise you would be spending even more time and effort to make your
> userland kit work to satisfaction for a far weaker hold on the box. It
> would simply be idiotic. And I think we can all agree that an attacker
> able to do either of the above is not an idiot.
>
> On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>>
>> A "poor man's" root kit detector is to take md5sums of critical system
>> binaries (you'd have to redo these after patching), and keep the list on
>> an inaccessible media (such as a thumb drive).  If you think the system
>> is compromised, run md5sum against those files, and you will quickly
>> know. You could even keep statically compiled copies on the thumb drive
>> to use in an investigation.



-- 
Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
***
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

Oh it certainly is a distinction, and that very distinction is important
enough to have caused the creation of kernel rootkits in the first place:
the kernel is absolute. There is nothing any software can do without the
kernel.

For instance say you got a guy with a userland rootkit. He wants to hide a
file so ls, and several other binaries were modified. You load up python,
whip up a quick script and boom you can see all the previously hidden
files. Kernel kit and you have to hook a few system calls and monitor the
incoming values. If it would return your file and the 'password' wasn't
given, you can return bogus information and EVERY tool will fall for it.

Also not everything has to be done in userland to get done. The kernel is
fully capable of sending out packets, creating files, etc. Userland in fact
relies on the kernel for all of these. If you get to the kernel you control
all of both worlds. You get the userland and in truth you only control a
portion of the userland.

Mighty difference indeed.
On Dec 7, 2011 7:20 AM, "Paul Schmehl"  wrote:

> But whether you have a kernel rootkit or not isn't all that important.  In
> either case the system is going to be doing unwanted things, and you detect
> those unwanted things with the usual system utilities.  If a kernel rootkit
> didn't affect userland, what would be its purpose?  Even to transmit data
> offsite you have to invoke network capabilities, file system capabilities,
> etc.
>
> IOW, it's a distinction without difference.
>
> --On December 6, 2011 11:48:02 AM -0800 Gage Bystrom <
> themadichi...@gmail.com> wrote:
>
>
>>
>> My bad, should have said that you can't trust the md5sum tampering(since
>> you stated to have a static copy on the flash drive) but you couldn't
>> trust it since you couldn't trust the system calls.
>>
>> The immediate moment you have to worry about a legit userland rootkit you
>> have to worry about a kernel rootkit. After all you have to consider the
>> psychology of the attacker. If you were to compromise a box, and cared
>> enough to hide a backdoor they cannot detect without static, write proof
>> media, then you care enough to go the extra step for a kernel rootkit.
>> Otherwise you would be spending even more time and effort to make your
>> userland kit work to satisfaction for a far weaker hold on the box. It
>> would simply be idiotic. And I think we can all agree that an attacker
>> able to do either of the above is not an idiot.
>>
>> On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>>
>>>
>>> A "poor man's" root kit detector is to take md5sums of critical system
>>> binaries (you'd have to redo these after patching), and keep the list on
>>> an inaccessible media (such as a thumb drive).  If you think the system
>>> is compromised, run md5sum against those files, and you will quickly
>>> know. You could even keep statically compiled copies on the thumb drive
>>> to use in an investigation.
>>>
>>
>
>
> --
> Paul Schmehl, Senior Infosec Analyst
> As if it wasn't already obvious, my opinions
> are my own and not those of my employer.
> *
> "It is as useless to argue with those who have
> renounced the use of reason as to administer
> medication to the dead." Thomas Jefferson
> "There are some ideas so wrong that only a very
> intelligent person could believe in them." George Orwell
>
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Paul Schmehl]

>From a computer science standpoint there's a difference, of course, but not 
from an investigation standpoint.  Say the kernel has a rootkit and is 
creating files.  How do you find those files?  If it's opening network 
connections, how do you find out what those connections are and what 
process is tied to them?

--On December 7, 2011 10:13:42 AM -0800 Gage Bystrom 
 wrote:

>
> Oh it certainly is a distinction, and that very distinction is important
> enough to have caused the creation of kernel rootkits in the first place:
> the kernel is absolute. There is nothing any software can do without the
> kernel.
>
> For instance say you got a guy with a userland rootkit. He wants to hide
> a file so ls, and several other binaries were modified. You load up
> python, whip up a quick script and boom you can see all the previously
> hidden files. Kernel kit and you have to hook a few system calls and
> monitor the incoming values. If it would return your file and the
> 'password' wasn't given, you can return bogus information and EVERY tool
> will fall for it.
>
> Also not everything has to be done in userland to get done. The kernel is
> fully capable of sending out packets, creating files, etc. Userland in
> fact relies on the kernel for all of these. If you get to the kernel you
> control all of both worlds. You get the userland and in truth you only
> control a portion of the userland.
>
> Mighty difference indeed.
> On Dec 7, 2011 7:20 AM, "Paul Schmehl"  wrote:
>
> But whether you have a kernel rootkit or not isn't all that important.
>  In either case the system is going to be doing unwanted things, and you
> detect those unwanted things with the usual system utilities.  If a
> kernel rootkit didn't affect userland, what would be its purpose?  Even
> to transmit data offsite you have to invoke network capabilities, file
> system capabilities, etc.
>
> IOW, it's a distinction without difference.
>
> --On December 6, 2011 11:48:02 AM -0800 Gage Bystrom
>  wrote:
>
>
>
>
> My bad, should have said that you can't trust the md5sum tampering(since
> you stated to have a static copy on the flash drive) but you couldn't
> trust it since you couldn't trust the system calls.
>
> The immediate moment you have to worry about a legit userland rootkit you
> have to worry about a kernel rootkit. After all you have to consider the
> psychology of the attacker. If you were to compromise a box, and cared
> enough to hide a backdoor they cannot detect without static, write proof
> media, then you care enough to go the extra step for a kernel rootkit.
> Otherwise you would be spending even more time and effort to make your
> userland kit work to satisfaction for a far weaker hold on the box. It
> would simply be idiotic. And I think we can all agree that an attacker
> able to do either of the above is not an idiot.
>
> On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>
>
> A "poor man's" root kit detector is to take md5sums of critical system
> binaries (you'd have to redo these after patching), and keep the list on
> an inaccessible media (such as a thumb drive).  If you think the system
> is compromised, run md5sum against those files, and you will quickly
> know. You could even keep statically compiled copies on the thumb drive
> to use in an investigation.



-- 
Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
***
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell

___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Re: [Full-disclosure] one of my servers has been compromized

[Gage Bystrom]

You use everything but the compromised box, right. And that's because of
the proliferation of kernel rootkits in the first place. Userland rootkits
can be defeated quickly, easily, and sometimes by accident. A kernel
rootkit can only realistically be beaten by other machines monitoring the
network, imaging the hard drive, etc. As an attacker you increase the
chances of losing by not using a kernel rootkit. Which is why if you're
going for a rootkit, there's no reason to use a userland over kernel. Not
to mention a kernel rootkit is in a better position to delay or prevent
discovery in the first place barring good mitigations.

Which is where my statement 'if you are worried about a userland kit, you
must worry about a kernel rootkit.
On Dec 7, 2011 1:18 PM, "Paul Schmehl"  wrote:

> From a computer science standpoint there's a difference, of course, but
>> not
>>
> from an investigation standpoint.  Say the kernel has a rootkit and is
> creating files.  How do you find those files?  If it's opening network
> connections, how do you find out what those connections are and what
> process is tied to them?
>
> --On December 7, 2011 10:13:42 AM -0800 Gage Bystrom <
> themadichi...@gmail.com> wrote:
>
>
>> Oh it certainly is a distinction, and that very distinction is important
>> enough to have caused the creation of kernel rootkits in the first place:
>> the kernel is absolute. There is nothing any software can do without the
>> kernel.
>>
>> For instance say you got a guy with a userland rootkit. He wants to hide
>> a file so ls, and several other binaries were modified. You load up
>> python, whip up a quick script and boom you can see all the previously
>> hidden files. Kernel kit and you have to hook a few system calls and
>> monitor the incoming values. If it would return your file and the
>> 'password' wasn't given, you can return bogus information and EVERY tool
>> will fall for it.
>>
>> Also not everything has to be done in userland to get done. The kernel is
>> fully capable of sending out packets, creating files, etc. Userland in
>> fact relies on the kernel for all of these. If you get to the kernel you
>> control all of both worlds. You get the userland and in truth you only
>> control a portion of the userland.
>>
>> Mighty difference indeed.
>> On Dec 7, 2011 7:20 AM, "Paul Schmehl"  wrote:
>>
>> But whether you have a kernel rootkit or not isn't all that important.
>>  In either case the system is going to be doing unwanted things, and you
>> detect those unwanted things with the usual system utilities.  If a
>> kernel rootkit didn't affect userland, what would be its purpose?  Even
>> to transmit data offsite you have to invoke network capabilities, file
>> system capabilities, etc.
>>
>> IOW, it's a distinction without difference.
>>
>> --On December 6, 2011 11:48:02 AM -0800 Gage Bystrom
>>  wrote:
>>
>>
>>
>>
>> My bad, should have said that you can't trust the md5sum tampering(since
>> you stated to have a static copy on the flash drive) but you couldn't
>> trust it since you couldn't trust the system calls.
>>
>> The immediate moment you have to worry about a legit userland rootkit you
>> have to worry about a kernel rootkit. After all you have to consider the
>> psychology of the attacker. If you were to compromise a box, and cared
>> enough to hide a backdoor they cannot detect without static, write proof
>> media, then you care enough to go the extra step for a kernel rootkit.
>> Otherwise you would be spending even more time and effort to make your
>> userland kit work to satisfaction for a far weaker hold on the box. It
>> would simply be idiotic. And I think we can all agree that an attacker
>> able to do either of the above is not an idiot.
>>
>> On Dec 6, 2011 10:19 AM, "Paul Schmehl"  wrote:
>>
>>
>> A "poor man's" root kit detector is to take md5sums of critical system
>> binaries (you'd have to redo these after patching), and keep the list on
>> an inaccessible media (such as a thumb drive).  If you think the system
>> is compromised, run md5sum against those files, and you will quickly
>> know. You could even keep statically compiled copies on the thumb drive
>> to use in an investigation.
>>
>
>
>
> --
> Paul Schmehl, Senior Infosec Analyst
> As if it wasn't already obvious, my opinions
> are my own and not those of my employer.
> *
> "It is as useless to argue with those who have
> renounced the use of reason as to administer
> medication to the dead." Thomas Jefferson
> "There are some ideas so wrong that only a very
> intelligent person could believe in them." George Orwell
>
>
___
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/