Re: [Full-disclosure] one of my servers has been compromized

2011-12-07 Thread Gage Bystrom
You use everything but the compromised box, right. And that's because of the proliferation of kernel rootkits in the first place. Userland rootkits can be defeated quickly, easily, and sometimes by accident. A kernel rootkit can only realistically be beaten by other machines monitoring the network,

Re: [Full-disclosure] one of my servers has been compromized

2011-12-07 Thread Paul Schmehl
>From a computer science standpoint there's a difference, of course, but not from an investigation standpoint. Say the kernel has a rootkit and is creating files. How do you find those files? If it's opening network connections, how do you find out what those connections are and what process

Re: [Full-disclosure] one of my servers has been compromized

2011-12-07 Thread Gage Bystrom
Oh it certainly is a distinction, and that very distinction is important enough to have caused the creation of kernel rootkits in the first place: the kernel is absolute. There is nothing any software can do without the kernel. For instance say you got a guy with a userland rootkit. He wants to hi

Re: [Full-disclosure] one of my servers has been compromized

2011-12-07 Thread Paul Schmehl
But whether you have a kernel rootkit or not isn't all that important. In either case the system is going to be doing unwanted things, and you detect those unwanted things with the usual system utilities. If a kernel rootkit didn't affect userland, what would be its purpose? Even to transmit

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Gage Bystrom
Well in that case it becomes fairly sane, assuming you've safeguarded against the one of the worst case scenario like Valdis previously mentioned. There are a handful of things I can think of however that could still work, at which point depends on the attackers goals. But at that point it'd be a

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread John Jacobs
> Sounds pretty neat to be honest. But one thing I'm wondering is that if > they have root, what's stopping them from turning that off? After all > they need root to load the modules in the first place, so if they are > in a position to want to do that, then they are in a position to turn

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Charles Morris
+1. Except instead of MD5 you want to use something that isn't garbage. On Tue, Dec 6, 2011 at 1:18 PM, Paul Schmehl wrote: > A "poor man's" root kit detector is to take md5sums of critical system > binaries (you'd have to redo these after patching), and keep the list on an > inaccessible media (

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Valdis . Kletnieks
On Tue, 06 Dec 2011 15:14:29 PST, Gage Bystrom said: > Maybe I'm misreading what you said, and if so please correct me, but > whether or not the changes described were applied in the first place or not > wouldn't change the issue that if you needed root unneutered again you > would need to bring d

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Gage Bystrom
Maybe I'm misreading what you said, and if so please correct me, but whether or not the changes described were applied in the first place or not wouldn't change the issue that if you needed root unneutered again you would need to bring down the system. Especially if the change doesn't really solve

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Valdis . Kletnieks
On Tue, 06 Dec 2011 13:20:51 PST, Gage Bystrom said: > serious pain if suddenly you needed unneutered root again. Would likely > have to take the system down to fix it. Who wants to be the guy to explain > that situation to their boss? If the server is critical enough that you can't take it down

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Gage Bystrom
Sounds pretty neat to be honest. But one thing I'm wondering is that if they have root, what's stopping them from turning that off? After all they need root to load the modules in the first place, so if they are in a position to want to do that, then they are in a position to turn that off. Granted

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Charles Morris
Sorry paul, Gage is right here! Instead of "silly" maybe more like "correct" :( On Tue, Dec 6, 2011 at 2:42 PM, Paul Schmehl wrote: > Don't be silly.  You can run static binaries off a thumb drive without > taking the system down.  And that includes md5sum.  You can put everything, > including t

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread John Jacobs
Those considering Tripwire I would ask they take a look at OSSEC-HIDS; the filesystem change notification is outstanding and with inotify() support you get immediate notification of changes.  The monitoring and alerting of log files is also exceptional.  I am not affiliated with OSSEC in any wa

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Paul Schmehl
Don't be silly. You can run static binaries off a thumb drive without taking the system down. And that includes md5sum. You can put everything, including the script, on a thumb drive and be perfectly comfortable that the results are reliable, because statically compiled binaries don't use sy

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Gage Bystrom
My bad, should have said that you can't trust the md5sum tampering(since you stated to have a static copy on the flash drive) but you couldn't trust it since you couldn't trust the system calls. The immediate moment you have to worry about a legit userland rootkit you have to worry about a kernel

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Gage Bystrom
But the problem with that is it is a mentality roughly a little more then a decade old. What you described is a userland rootkit detector. Problem is no one uses userland rootkits anymore! Sure there was some recent development in managed code rootkits but it really hasn't home anywhere and is Wind

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Paul Schmehl
A "poor man's" root kit detector is to take md5sums of critical system binaries (you'd have to redo these after patching), and keep the list on an inaccessible media (such as a thumb drive). If you think the system is compromised, run md5sum against those files, and you will quickly know. You

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Valdis . Kletnieks
On Mon, 05 Dec 2011 19:04:02 +0100, Lucio Crusca said: > Using dd on /dev/mem and piping results through netcat it's not that > difficult, and a bit of google explains how to do it the right way, but in > my case there are two other problems: Note that the effectiveness and safety of doing a dd

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Valdis . Kletnieks
On Mon, 05 Dec 2011 13:53:21 GMT, Dan Ballance said: > Also, am I correct to think that using something like tripwire is the best > way to detect root kits properly, but that it obviously needs installing > when the box is fresh and before it has been physically connected to a > network? tripwire

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Kerem Erciyes
I regularly use iftop, netstat and htop to see what is going on on my servers. I have found that raw information always helps the best in determining acitve compromised systems. Kerem On Tue, Dec 6, 2011 at 11:55 AM, Lucio Crusca wrote: > BH wrote: > > > I'm not sure if this has been said in th

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Lucio Crusca
BH wrote: > I'm not sure if this has been said in this thread yet, but is it > possible the host O/S was compromised? Nothing is impossible, security wise. However I'd talk about likelihood instead. I own two other OpenVZ containers hosted in the same host OS. They haven't been compromised, th

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Gage Bystrom
Ahh I see. Then yeah I would advise using iptables to deny as much outgoing traffic as possible and set up the chain so that all attempted traffic statistics get logged. Back that up with denying as much incoming traffic as possible. Then monitor for any spawning services with netstat. Assuming no

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread BH
I'm not sure if this has been said in this thread yet, but is it possible the host O/S was compromised? I have not used OpenVZ but I assume it's the same as Virtuozzo in the respect that you can just 'vzctl enter ' to get a root shell inside the container with no password (assuming you have control

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Lucio Crusca
Gage Bystrom wrote: > I would suggest iptables but the OP stated he doesn't own the > server and has no root access. If I ever stated that, it means I misused my poor english for sure... I DO have root access and I DO own the server, where the server means the *guest* OpenVZ instance. I DID con

Re: [Full-disclosure] one of my servers has been compromized

2011-12-06 Thread Guillaume Friloux
On 05/12/2011 18:20, John Jacobs wrote: > Tim, while I do believe there is some truth in what you are saying here, I > respectfully disagree in that this tends to be a run-of-the-mill IRC bot as > evidenced by the Undernet advisory. This looks like a skiddie-de-jour attack > against PHPMyAdmin

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread John Jacobs
> Very useful john jacob ... really helpful. > do you maintaine your blog or any other resource you want to share with us. > thanx a ton . Thank you for the kind words and I consider it an honor to have been helpful.  I do not have a blog.  I have enjoyed this thread, sharing what I know, and l

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Dan Ballance
Thanks for that. I've learnt a lot from this thread. Peace to you all. On 5 December 2011 20:11, Gage Bystrom wrote: > Tripwire is awesome for many reasons. The original use of rootkit > detection is no longer one of them. It was used back when userland rootkits > were big, it has zero effect on

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread sam
Very useful john jacob ... really helpful. do you maintaine your blog or any other resource you want to share with us. thanx a ton . On Mon, Dec 5, 2011 at 8:18 AM, Michael Wood wrote: > Awesome tips guys... > > On Dec 5, 2011 11:01 AM, "John Jacobs" wrote: >> >> >> > 2. Do you think said phpmya

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Josh Yavor
Really enjoying following this thread. Any one have thoughts/opinions on APF as an option to prevent this in the future? On Mon, Dec 5, 2011 at 11:18 AM, Michael Wood wrote: > Awesome tips guys... > > On Dec 5, 2011 11:01 AM, "John Jacobs" wrote: >> >> >> > 2. Do you think said phpmyadmin vulns

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Larry W. Cashdollar
I'd check these too:http://virtuemart.net/security-bulletinsOn Dec 05, 2011, at 05:35 AM, mitchell wrote:Hi,Here is what you generally need to do in such cases.1. Suspend the webapp until you investigate.2. Check the web server logs for unusual entries and identify the entry point. You should have

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Larry W. Cashdollar
Hi,I'd say tell your boss your application has been compromised right away.    Tell them you'll need to rebuild the entire system from scratch and they'll need to either devise an upgrade path for virtuemart or find a new ecommerce solution.You can't trust a system once it has been compromised. --

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Dan Ballance
Thanks for the heads-up on rkhunter Gage. Is there anything else out there atm that works as a reasonable root kit detector or is such a thing considered impossible now? I realise a skilled attack will be able to bury itself without a trace, but I'm thinking of something that can be used in less s

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Aris Adamantiadis
A good start would have been to take a snapshot of your compromized system and not to delete any file. Now, good luck to understand what happened. Aris Le 5/12/11 11:44, Lucio Crusca a écrit : > Hello *, > > I'm not new here, but I've mostly lurked all the time through gmane. I never > believed

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Javier Bassi
In addition to the tips given (chroot, disable shell_exec,etc), you should also use open_basedir with DocumentRoot as path on each VirtualHost. In case of a compromise via webapp, this will reduce the compromised zone in the filesystem to the DocumentRoot of one VirtualHost instead of the whole chr

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Gage Bystrom
Tripwire is awesome for many reasons. The original use of rootkit detection is no longer one of them. It was used back when userland rootkits were big, it has zero effect on kernel rootkits. That being said you can use it to watch other critical files for improper access. Keep tabs on your cron fil

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread John Jacobs
> Why take the risk? You don't know what the attacker actually did > until you do some analysis. If you do analysis before capturing a > disk image, you're destroying evidence. > > Rebuilding a server is not hard. It has a known quantity of effort > involved and reliably prevents further intrusion

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Lucio Crusca
John Jacobs wrote: > In my previous experience, including a few vulnerabilities in Roundcube, > I've seen a programmatic scan, exploitation, and then wget/dropping of a > Perl IRC bot. Of course this is all speculation based on previous > experience, I am not looking at lucio's box. The Apache a

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Paul Schmehl
--On December 5, 2011 11:44:04 AM +0100 Lucio Crusca wrote: > > Now the problem is: how do I pervent further abuse? What should I search > in the logs (if anything) to spot the security hole? > Install mod-security and write custom rules to prevent the hack. Look in your webserver logs to se

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Tim
> Well, I actually made 2 mistakes and the second compensated the harm the > first did... > > My second mistake I did not mention before was to overlook various other > folders in /tmp that were older than /tmp/.m and contained lots of other > crap (so they are even more useful in finding the o

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Paul Schmehl
--On December 5, 2011 1:48:24 PM +0100 Christophe Garault wrote: >> > Having your /tmp partition with noexec,nosuid is also considered a good > practice. That's not a bad generic suggestion, but it won't do a thing for this hack. They deposit perl scripts in /tmp/.m and then run them by callin

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread John Jacobs
> Subject: Re: [Full-disclosure] one of my servers has been compromized > From: ja...@zero-internet.org.uk > Date: Mon, 5 Dec 2011 17:36:53 + > CC: tim-secur...@sentinelchicken.org; lu...@sulweb.org; > full-disclosure@lists.grok.org.

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Tim
> > For future reference, and for the benefit of people searching for > > solutions to similar problems: You've made the most common rookie > > mistake. You have already trashed potentially critical information > > about the attack by trying to clean up the server first. Don't do > > that. > > Tim

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Lucio Crusca
Tim wrote: > > For future reference, and for the benefit of people searching for > solutions to similar problems: You've made the most common rookie > mistake. Well, I actually made 2 mistakes and the second compensated the harm the first did... My second mistake I did not mention before was t

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread James Condron
John, All good thoughts but can we show the server was rooted? In otherwords; instead of an attacker getting root and then adding this to a botnet this way is it not more likely that the original attack added the server in one step to avoid the need to do this? Attackers, from my experience, d

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Dave
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 10:44, Lucio Crusca wrote: > Hello *, > > I'm not new here, but I've mostly lurked all the time through gmane. I never > believed it could happen to me until it actually happened: they compromized > one of my servers. It's a Ubuntu 10.

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread John Jacobs
> For future reference, and for the benefit of people searching for > solutions to similar problems: You've made the most common rookie > mistake. You have already trashed potentially critical information > about the attack by trying to clean up the server first. Don't do > that. Tim, while I do

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Tim
For future reference, and for the benefit of people searching for solutions to similar problems: You've made the most common rookie mistake. You have already trashed potentially critical information about the attack by trying to clean up the server first. Don't do that. > I've run the "find" c

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Michael Wood
Awesome tips guys... On Dec 5, 2011 11:01 AM, "John Jacobs" wrote: > > > 2. Do you think said phpmyadmin vulns are reasonable attack vectors in my > > case? > > I do, I believe this is to be the initial infection vector. Scanning for > PHPMyAdmin is often and frequent and since it's likely that

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread John Jacobs
> 2. Do you think said phpmyadmin vulns are reasonable attack vectors in my > case? I do, I believe this is to be the initial infection vector.  Scanning for PHPMyAdmin is often and frequent and since it's likely that it was present in it's default (or one of the default) URIs discovery is like

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Lucio Crusca
Gage Bystrom wrote: > Fortunately for him, since the bot was so easy to find in the first place > and such a simple way of maintaining it, the box was clearly seized by > someone who didn't give a rats ass about it. Probably a skiddie or an > automated attack to begin with. That makes me think th

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread mitchell
Hi, Here is what you generally need to do in such cases. 1. Suspend the webapp until you investigate. 2. Check the web server logs for unusual entries and identify the entry point. You should have noticed the timestamp of the Perl script in the /tmp directory. Try looking for entries around this t

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Christophe Garault
On 12/O5/2011 13:07, Lucio Crusca wrote : > Ferenc Kovacs wrote: > > No, it's just common saying. I apply patches, change password regularly, > move ssh to nonstandard ports, disable remote root access and do all the > rest I've learnt about security in years of running linux servers, also if I

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Chris M
You could ch-root your apache process/webserver going forward. This would effectively stop the malicious process when/if your machine is compromised via web based vulnerabilities to spread to entire machine.. meaning your area of investigation is more isolated. I'd expect if its automatically spre

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Lucio Crusca
Ferenc Kovacs wrote: > ps: "I neverbelieved it could happen to me until it actually happened: > they compromizedone of my servers." this is a really bad attitude. No, it's just common saying. I apply patches, change password regularly, move ssh to nonstandard ports, disable remote root access an

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Ferenc Kovacs
On Mon, Dec 5, 2011 at 11:44 AM, Lucio Crusca wrote: > Hello *, > > I'm not new here, but I've mostly lurked all the time through gmane. I > never > believed it could happen to me until it actually happened: they compromized > one of my servers. It's a Ubuntu 10.04 server with all security patche

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Gage Bystrom
If it was a rootkit then trying to run the outdated rkhunter would be a moot point. Whatever seizes the kernel first wins, hands down. Fortunately for him, since the bot was so easy to find in the first place and such a simple way of maintaining it, the box was clearly seized by someone who didn't

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Lucio Crusca
Dan Ballance wrote: > I'm no expert, but here's something to get you started while you wait for > more experienced replies. Check for root kits: > > sudo apt-get install rkhunter > sudo rkhunter --update > sudo rkhunter --check Thanks for the tip, I've ran those but the check found nothing. I ke

Re: [Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Dan Ballance
I'm no expert, but here's something to get you started while you wait for more experienced replies. Check for root kits: sudo apt-get install rkhunter sudo rkhunter --update sudo rkhunter --check On 5 December 2011 10:44, Lucio Crusca wrote: > Hello *, > > I'm not new here, but I've mostly lurk

[Full-disclosure] one of my servers has been compromized

2011-12-05 Thread Lucio Crusca
Hello *, I'm not new here, but I've mostly lurked all the time through gmane. I never believed it could happen to me until it actually happened: they compromized one of my servers. It's a Ubuntu 10.04 server with all security patches regularly applied. I'm inclined to believe they used some hol