Re: Server hacked - next...?
On Fri, Jul 04, 2003 at 01:09:53PM +0100, Shri Shrikumar wrote: On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. /dev/kmem? You can remove it, sure, but it can just be mknod'd again. - Keegan pgp0.pgp Description: PGP signature
Re: Server hacked - next...?
On Mon, Jul 07, 2003 at 12:05:12PM -0700, Keegan Quinn wrote: On Fri, Jul 04, 2003 at 01:09:53PM +0100, Shri Shrikumar wrote: I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. /dev/kmem? You can remove it, sure, but it can just be mknod'd again. Of course, after I wrote this, I noticed the correction was made later in the thread. Sorry. If you really think getting rid of this device node is critical, you might be able to edit some parts of the kernel source to prevent it from working at all. This might break any number of things in confusing ways. You could also look at patches like SE Linux and GRsecurity, which deal in different ways with restricting different kinds of access. - Keegan pgp0.pgp Description: PGP signature
Re: Server hacked - next...?
On Fri, Jul 04, 2003 at 01:09:53PM +0100, Shri Shrikumar wrote: On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. /dev/kmem? You can remove it, sure, but it can just be mknod'd again. - Keegan pgp7W0nEbVCfQ.pgp Description: PGP signature
Re: Server hacked - next...?
On Mon, Jul 07, 2003 at 12:05:12PM -0700, Keegan Quinn wrote: On Fri, Jul 04, 2003 at 01:09:53PM +0100, Shri Shrikumar wrote: I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. /dev/kmem? You can remove it, sure, but it can just be mknod'd again. Of course, after I wrote this, I noticed the correction was made later in the thread. Sorry. If you really think getting rid of this device node is critical, you might be able to edit some parts of the kernel source to prevent it from working at all. This might break any number of things in confusing ways. You could also look at patches like SE Linux and GRsecurity, which deal in different ways with restricting different kinds of access. - Keegan pgpesur0n51Za.pgp Description: PGP signature
Re: Server hacked - next...?
On Fri, 2003-07-04 at 15:44, Thomas Lamy wrote: Shri Shrikumar: On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. I'm sure he meant /dev/kmem Ok, I have that file. Can anyone point me in the direction of something I can do to make it more difficult to exploit this. Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
I'm sure he meant /dev/kmem Yes I meant /dev/kmem, sorry for that one :) Ok, I have that file. Can anyone point me in the direction of something I can do to make it more difficult to exploit this. There are several projects that try to make Linux a secure operating system, I have not tested them too much, but I this kind of things can help secure your system. But you have to be advised that administering a system with LIDS it's a pain in the a**, you just cant do pretty nothing with the LIDS activated. http://www.lids.org/ Mario. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Fri, 2003-07-04 at 15:44, Thomas Lamy wrote: Shri Shrikumar: On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. I'm sure he meant /dev/kmem Ok, I have that file. Can anyone point me in the direction of something I can do to make it more difficult to exploit this. Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
I'm sure he meant /dev/kmem Yes I meant /dev/kmem, sorry for that one :) Ok, I have that file. Can anyone point me in the direction of something I can do to make it more difficult to exploit this. There are several projects that try to make Linux a secure operating system, I have not tested them too much, but I this kind of things can help secure your system. But you have to be advised that administering a system with LIDS it's a pain in the a**, you just cant do pretty nothing with the LIDS activated. http://www.lids.org/ Mario.
Re: Server hacked - next...?
On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
Shri Shrikumar: On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. I'm sure he meant /dev/kmem -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Shri Shrikumar: On Thu, 2003-07-03 at 22:30, Mario Lopez wrote: In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. I dont have module support and I dont have /proc/kmem. Am I missing something ? Running 2.4.20. I'm sure he meant /dev/kmem
Re: Server hacked - next...?
Hello Jason, Am 21:47 2003-06-29 +0800 hat Jason Lim geschrieben: Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ I am using a 4 and a 8 Channel 3 Ware too. All with Raid-5. On my 4-Channel I have three IBM's with 120 GB and 4 Hotswap Racks (one for reserve). I have tried to reinstall a Raid 5 Array while removing one of the 3 HD'S and then deactivating this channel permanently. Then I have installed two new HD's on the two free Hotswap-Racks and reinstalled a new system. Copied the /home data to it and rebooted the System put out the infected HD's while the Computer is in BIOS and the it booted... It takes arround 50 seconds to start... Maybe it will help you. Michelle
Re: Server hacked - next...?
Am 09:56 2003-06-29 +0200 hat Ralf G. R. Bergs geschrieben: On Sun, 29 Jun 2003 13:00:57 +0800, Jason Lim wrote: Is there a document somewhere, or procedure, to recover after this? It's as simple as reinstalling. There's no other way, you can't get around this... ...but install a newer Kernel like 2.4.19|20 Michelle
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 01:00:57PM +0800, Jason Lim wrote: One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Some things like: www-data 17451 0.0 0.0 2164 928 ?S02:31 0:00 /bin/sh www-data 21550 0.0 0.0 1232 236 ?S05:02 0:00 ./x www-data 21551 0.0 0.0 00 ?Z05:02 0:00 [x defunct] root 21552 0.0 0.0 00 ?Z05:02 0:00 [modprobe defunc root 21554 0.0 0.0 2148 912 ?S05:02 0:00 /bin/sh root 21755 0.0 0.0 2164 948 ?S05:02 0:00 /bin/sh root 21801 0.0 0.0 2180 964 ?S05:03 0:00 /bin/bash ./troja root 22010 0.0 0.0 1244 204 ?S05:03 0:00 ./siz ifconfigx / root 12267 0.0 0.0 00 ?Z07:15 0:00 [date defunct] root 12266 0.0 0.0 1264 252 ?T07:15 0:00 date +%d Hi! I'm no expert in this at all... Here are some basic try-to-solve-it hints. In most cases its not possible to reinstall the whole system, as in this case. I mean. A home-server/workstation is no problem to reinstall but a high SLA 60k -user cluster is quite boring and time consuming. I'd do it like this. First. We need some fresh clean tools; kill, killall, ps, more, netstat, ls, dpkg, apt-tools, chattr, lsattr, bash (or whatever shell you prefer). Replace your shell with the clean one (the /etc/passwd -race). Killing the procs right off is almost impossible unless you find the master process (often protected and hidden in a patched ps or proctable and chattr'ed away on your filesystem). since you're using the 2.4.20 kernels modprobe-bug exists, so get rid of that bug first. echo blah /proc/sys/kernel/modprobe then I'd; lsattr -a /*|more to see weither we have some hidden and/or write protected files that we dont know. I bet you'll get some interesting output here. chattr this files and move them to some secret place so you can check them out later... Dont forget to check .history-files, logs etc. Most hacks are done in a rush and there are always pices of information left every here and there. After the filesystem looks nice and clean I would try find and kill the processes. As stated abowe it's quite hard to kill processes that are not meant to be killed. netstat -anp is a good tool here aswell as kill and ps. I've seen cases when the mastersprocess is hidden within sshd, init, various daemons such as ftp, telnet, ldap, gpm etc. Kill all processes you dont need. Look for respawning ones. A reboot might help, but dont reboot until you've checked the startup rc-files, stuff needed to boot etc... then i'd apt-getted the base system and then all debian-packages. Now try find out how he did it and try fix it before it happens again. Hope this helps... -- __ Yours sincerely, Christofer Algotsson - [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi!, First. We need some fresh clean tools; kill, killall, ps, more, netstat, ls, dpkg, apt-tools, chattr, lsattr, bash (or whatever shell you prefer). Replace your shell with the clean one (the /etc/passwd -race). Be aware that sometimes the rootkits also apply to the libc or even kernel modules, so just uploading new dinamicaly linked versions of the above programs will not help you, try compiling some os those tools statically (-static in gcc) in some other host you trust, specially ps command. In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. Mario Lopez. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Mario Lopez wrote:
Hi!,
First. We need some fresh clean tools;
kill, killall, ps, more, netstat, ls, dpkg, apt-tools,
chattr, lsattr, bash (or whatever shell you prefer).
Replace your shell with the clean one (the /etc/passwd -race).
Better, boot off a clean medium (I prefer KNOPPIX for that, as all needed
tools are there, and it runs completely from CD). Only _this_ way you can
make pretty sure that neither some nasty kernel module, nor some hacked
shared library is loaded while cleaning the compromised system.
When you have Knoppix up and running, the other posts apply (re-install base
debs, clean up /etc/{init.d,rc?d,passwd,shadow,modules(!)), move away that
nasty root kit (via chkrootkit, may be included on the knoppix cd, but I'm
not sure).
But - as others already stated - it's highly recommended to set up from
scratch. Boot from CD, take a full backup via rsync or scp or whatever you
prefer, set up a blank system, and restore the user parts thorougly.
Thomas
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 01:00:57PM +0800, Jason Lim wrote: One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Some things like: www-data 17451 0.0 0.0 2164 928 ?S02:31 0:00 /bin/sh www-data 21550 0.0 0.0 1232 236 ?S05:02 0:00 ./x www-data 21551 0.0 0.0 00 ?Z05:02 0:00 [x defunct] root 21552 0.0 0.0 00 ?Z05:02 0:00 [modprobe defunc root 21554 0.0 0.0 2148 912 ?S05:02 0:00 /bin/sh root 21755 0.0 0.0 2164 948 ?S05:02 0:00 /bin/sh root 21801 0.0 0.0 2180 964 ?S05:03 0:00 /bin/bash ./troja root 22010 0.0 0.0 1244 204 ?S05:03 0:00 ./siz ifconfigx / root 12267 0.0 0.0 00 ?Z07:15 0:00 [date defunct] root 12266 0.0 0.0 1264 252 ?T07:15 0:00 date +%d Hi! I'm no expert in this at all... Here are some basic try-to-solve-it hints. In most cases its not possible to reinstall the whole system, as in this case. I mean. A home-server/workstation is no problem to reinstall but a high SLA 60k -user cluster is quite boring and time consuming. I'd do it like this. First. We need some fresh clean tools; kill, killall, ps, more, netstat, ls, dpkg, apt-tools, chattr, lsattr, bash (or whatever shell you prefer). Replace your shell with the clean one (the /etc/passwd -race). Killing the procs right off is almost impossible unless you find the master process (often protected and hidden in a patched ps or proctable and chattr'ed away on your filesystem). since you're using the 2.4.20 kernels modprobe-bug exists, so get rid of that bug first. echo blah /proc/sys/kernel/modprobe then I'd; lsattr -a /*|more to see weither we have some hidden and/or write protected files that we dont know. I bet you'll get some interesting output here. chattr this files and move them to some secret place so you can check them out later... Dont forget to check .history-files, logs etc. Most hacks are done in a rush and there are always pices of information left every here and there. After the filesystem looks nice and clean I would try find and kill the processes. As stated abowe it's quite hard to kill processes that are not meant to be killed. netstat -anp is a good tool here aswell as kill and ps. I've seen cases when the mastersprocess is hidden within sshd, init, various daemons such as ftp, telnet, ldap, gpm etc. Kill all processes you dont need. Look for respawning ones. A reboot might help, but dont reboot until you've checked the startup rc-files, stuff needed to boot etc... then i'd apt-getted the base system and then all debian-packages. Now try find out how he did it and try fix it before it happens again. Hope this helps... -- __ Yours sincerely, Christofer Algotsson - [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi!, First. We need some fresh clean tools; kill, killall, ps, more, netstat, ls, dpkg, apt-tools, chattr, lsattr, bash (or whatever shell you prefer). Replace your shell with the clean one (the /etc/passwd -race). Be aware that sometimes the rootkits also apply to the libc or even kernel modules, so just uploading new dinamicaly linked versions of the above programs will not help you, try compiling some os those tools statically (-static in gcc) in some other host you trust, specially ps command. In any case if you have a lkm rootkit, your done, dosent matter if you upload static, dinamic or whatever, kernel root kits are hard to find, not even lsmod, rmmod can help you because it is quite easy to make a kernel module unloadable or even hiden, some of you may be thinking that they are safe to those kind of attacks because they have disabled kernel module support in theyr kernel, well they are wrong :), there is code, and nice white papers explaining how to insert kernel code through /proc/kmem, if I am not wrong Silvio Cesare developed this technique two or three years ago, although it hasent being exploited too much you must be aware of it's existance. Mario Lopez.
Re: Server hacked - next...?
Mario Lopez wrote:
Hi!,
First. We need some fresh clean tools;
kill, killall, ps, more, netstat, ls, dpkg, apt-tools,
chattr, lsattr, bash (or whatever shell you prefer).
Replace your shell with the clean one (the /etc/passwd -race).
Better, boot off a clean medium (I prefer KNOPPIX for that, as all needed
tools are there, and it runs completely from CD). Only _this_ way you can
make pretty sure that neither some nasty kernel module, nor some hacked
shared library is loaded while cleaning the compromised system.
When you have Knoppix up and running, the other posts apply (re-install base
debs, clean up /etc/{init.d,rc?d,passwd,shadow,modules(!)), move away that
nasty root kit (via chkrootkit, may be included on the knoppix cd, but I'm
not sure).
But - as others already stated - it's highly recommended to set up from
scratch. Boot from CD, take a full backup via rsync or scp or whatever you
prefer, set up a blank system, and restore the user parts thorougly.
Thomas
Re: Server hacked - next...?
On Thu, 3 Jul 2003, Thomas Lamy wrote: nasty root kit (via chkrootkit, may be included on the knoppix cd, but I'm not sure). Yes it is - /usr/sbin/chrootkit -- Martin Wheeler - StarTEXT / AVALONIX - Glastonbury - BA6 9PH - England [EMAIL PROTECTED] http://startext.demon.co.uk/ GPG pub key : 8D6B948B ECC6 D98E 4CC8 60E3 7E32 D594 BB27 3368 8D6B 948B - Share your knowledge. It's a way of achieving immortality. -
Re: Server hacked - next...?
On Tue, Jul 01, 2003 at 10:07:01AM +0800, Jason Lim wrote: As Russell Coker points out, the attaccer probably got in trough -snipp- DOH... I just posted saying that in my previous email before reading his message! Bah... Russell gets credit for it ;-) Not easy to say who said that first, as my previous mail took a day and a half to get troug the list. Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. What would the advantage of mounting /tmp with noexec be?? An attacker can't upload his ptrace or whatever exploit to tmp and execute it. A recent automated attack against Apache (1.3 something) failed because i have /tmp mounted noexec. Another cool, and hopefully effective restraining of apache and others that i implemented in iptables yesterday: # This stops apache from connecting to anything else than imap2 $IPTABLES -P OUTPUT ACCEPT $IPTABLES -A OUTPUT -p tcp --dport 143 -d localhost \ -m owner --uid-owner www-data -m state \ --state NEW -j ACCEPT $IPTABLES -A OUTPUT -m owner --uid-owner www-data -m state \ --state NEW -j REJECT This requires a custom kernel with ipt_owner as module or static. Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail. I run Grsecurity for /proc restrictons, safer chroot, and protection against kernel exploits like the ptrace bug. That didn't take too long to get running. Disclaimer: I have not tested any of this in production, as i am a student. -- Frode Haugsgjerd Norway -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Mon, 2003-06-30 at 23:41, Russell Coker wrote: On Mon, 30 Jun 2003 16:12, Donovan Baarda wrote: In then end it is nearly always easier to re-install than to just clean the system without it, even if the hacker did leave .bash_histories behind that show everything he/she did. Besides, just because a script-kiddie got in first does not mean that a skilled and experienced person didn't follow them. Or maybe a skilled person cracked the machine months or years before the script-kiddie... I almost added that the .bash_history's could have been forged by a more skilled cracker to hide something more subtle... -- Donovan Baardahttp://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Tue, 1 Jul 2003 12:07, Jason Lim wrote: Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. What would the advantage of mounting /tmp with noexec be?? If you have /tmp, /var/tmp, /home, and any other place the user can possibly write to be noexec then it is more difficult for them to increase their access. It won't stop them, but it will make things more difficult. Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail. It's not that difficult. The SE Linux sourceforge project has some docs on installing it. For Debian it's reasonably easy, the only difficult parts are compiling a new kernel with support, and writing any necessary policy. The #selinux IRC channel on irc.debian.org can be used for advice. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Tue, 2003-07-01 at 03:07, Jason Lim wrote: What would the advantage of mounting /tmp with noexec be?? You wont be able to execute anything that is in /tmp. This would of have prevented your attach altogether since the file was being executed from /tmp. Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail. The first thing I did for the server is to remove module support. This renders quite a few of the exploits useless. It might be a stopgap solution until you get the hardened kernels deployed. HTH, Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
On Mon, 2003-06-30 at 23:41, Russell Coker wrote: On Mon, 30 Jun 2003 16:12, Donovan Baarda wrote: In then end it is nearly always easier to re-install than to just clean the system without it, even if the hacker did leave .bash_histories behind that show everything he/she did. Besides, just because a script-kiddie got in first does not mean that a skilled and experienced person didn't follow them. Or maybe a skilled person cracked the machine months or years before the script-kiddie... I almost added that the .bash_history's could have been forged by a more skilled cracker to hide something more subtle... -- Donovan Baardahttp://minkirri.apana.org.au/~abo/
Re: Server hacked - next...?
On Tue, 1 Jul 2003 12:07, Jason Lim wrote: Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. What would the advantage of mounting /tmp with noexec be?? If you have /tmp, /var/tmp, /home, and any other place the user can possibly write to be noexec then it is more difficult for them to increase their access. It won't stop them, but it will make things more difficult. Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail. It's not that difficult. The SE Linux sourceforge project has some docs on installing it. For Debian it's reasonably easy, the only difficult parts are compiling a new kernel with support, and writing any necessary policy. The #selinux IRC channel on irc.debian.org can be used for advice. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Tue, 2003-07-01 at 03:07, Jason Lim wrote: What would the advantage of mounting /tmp with noexec be?? You wont be able to execute anything that is in /tmp. This would of have prevented your attach altogether since the file was being executed from /tmp. Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail. The first thing I did for the server is to remove module support. This renders quite a few of the exploits useless. It might be a stopgap solution until you get the hardened kernels deployed. HTH, Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
On Sun, 29 Jun 2003 13:00:57 +0800, Jason Lim wrote: Is there a document somewhere, or procedure, to recover after this? It's as simple as reinstalling. There's no other way, you can't get around this... -- L I N U X .~. The Choice /V\ of a GNU /( )\ Generation ^^-^^ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 2003-06-29 at 19:02, Donovan Baarda wrote: [...] Once you get compromised, it's pretty darn hard to get clean without starting fresh. Some rootkit compromises do weird stuff like infect every binary file you even 'ls'. One system I saw had been compromised via an ssh vulerability (old ssh) and rootkit'ed... there was a very good security guy doing the (remote) cleanup, and he ended up having to install buisybox just so that he had a clean environment he could work from. Dispite it being damn hard to clean up, it was just the work of a Thought I'd better clarify here... he was using buisybox to get a clean environment so he could remotely shut down the system to a bare minimium and still get critical stuff off. After that the system was wiped and fresh re-installed with new passwords. Just in case anyone was thinking this showed it was worth resurrecting a compromised system without a fresh re-install :-( In then end it is nearly always easier to re-install than to just clean the system without it, even if the hacker did leave .bash_histories behind that show everything he/she did. -- Donovan Baardahttp://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Mon, 30 Jun 2003 16:12, Donovan Baarda wrote: In then end it is nearly always easier to re-install than to just clean the system without it, even if the hacker did leave .bash_histories behind that show everything he/she did. Besides, just because a script-kiddie got in first does not mean that a skilled and experienced person didn't follow them. Or maybe a skilled person cracked the machine months or years before the script-kiddie... -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 11:28:47AM -0400, bda wrote: On Sun, Jun 29, 2003 at 09:47:13PM +0800, Jason Lim wrote: Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? find / -uid 0 -perm 0400 I guess this should have been 04000 Marcin -- Marcin Owsiany [EMAIL PROTECTED] http://marcin.owsiany.pl/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 2003-06-29 at 06:00, Jason Lim wrote: Hi all, Well... bad day for me. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. I know how you feel. I almost got compromised too on the same date funnily enough. They had a program running called bd which opened port 5000 which was luckily blocked by the iptables firewall which proabably helped. There were also some php scripts which gave access to the machine in the web root. The lesson that I learnt. *do not give www-data write access* even in the web root. I shoulda thought of this earlier but the setup kinda required it. I have now reworked the setup and it looks better. I used debsums to check that the binaries themselves are not compromised. I also have tiger, snort and logcheck installed to try and detect something like this. Although, neither snort nor logcheck detected anything and I installed tiger after the attack. Fingers crossed. Hope this helps, Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
chkrootkit is also avaialble through apt-get apt-get install chkrootkit ## On Sun, 29 Jun 2003, Jason Lim wrote: Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux and various others) and migrate the accounts over. Thanks in advance!!! Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: 29 June, 2003 4:02 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Mon, Jun 30, 2003 at 08:03:11PM +0200, Marcin Owsiany wrote: find / -uid 0 -perm 0400 I guess this should have been 04000 Actually, it should be find / -uid 0 -perm +4000 Sorry about that.. -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi Daniel, Yeap, I follow Bugtraq... too bad Debian came out with the official kernel ages after it was revealed. Actually running 2.4.21 now... but it certainly is possible that during the gap between when we were running 2.4.17/18 to when we upraded to 2.4.21, someone got in. Actually, from what I can tell now, someone APPEARS to have got in through a user's vulnerable script... i think phpBB or one of those, and then obtained remote access that way, then maybe run those rootkits and stuff I am thinking it might have been related to the ptrace bug because i saw a weird modprobe entry there, but then again many rootkits fool around with modprobe??? Sincerely, Jas - Original Message - From: Daniel K. Gebhart [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, 29 June, 2003 6:29 PM Subject: Re: Server hacked - next...? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
As Russell Coker points out, the attaccer probably got in trough apache and a vulnerable CGI script. When you reinstall, be sure you dont run any insecure CGI's. There is probably a bunch of other improvements jou can do. DOH... I just posted saying that in my previous email before reading his message! Bah... Russell gets credit for it ;-) Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. What would the advantage of mounting /tmp with noexec be?? Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi Shri, Good idea... installing tiger too now! Sincerely, Jas - Original Message - From: Shri Shrikumar [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, 01 July, 2003 2:15 AM Subject: Re: Server hacked - next...? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 04:16:47PM +0200, Thomas Lamy wrote: Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ IMHO there's only one save way to go after being hacked: reinstall. Jason, if you're really determined not to do a reinstall, you can first construct a list of all installed packages, download them from your nearest mirror, and re-install them with dpkg. that will fix up the debian packages but WILL NOT do anything about non-debian binaries that may be on your system...you'll have to find and fix them by hand. overall, this is not a good idea - your script-kiddie may be logged in while you're fixing the machine, silently unfixing it. that said, i have fixed a few (remotely-located) boxes this way, to avoid the travel time expense of going out to sit at the console to do a complete reinstall. you do need to be very skilled (in security issues, general systems admin, and debian systems admin) before you even consider doing thisdefinitely NOT recommended for newbies. actually, it's not recommended for anyone at all. if you insist on doing this, some important packages binaries to reinstall first are netstat, ls, libc6, and procps - i.e. diagnostic tools that you can run to show what processes are running, what files are in a directory, what network connections are open etc. script-kiddies routinely replace these with compromised versions that try to hide SK activity. Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? Dunno - rpm has the option of checking md5 sums, but the dpkg manpage isn't promising in this regard. my dlocate package has an md5 checking facility but a) not all debian packages have md5sum files, and b) debian md5sum files are not signed, so they're easily compromised by any script-kiddie that cares to bother. in short, it's a nice idea but not terribly useful. e.g. # dlocate -md5check net-tools /usr/sbin/arp OK /usr/share/man/man5/ethers.5.gz OK /usr/share/man/man8/arp.8.gzOK /usr/share/man/man8/ifconfig.8.gz OK /usr/share/man/man8/mii-tool.8.gz OK /usr/share/man/man8/nameif.8.gz OK /usr/share/man/man8/netstat.8.gz OK /usr/share/man/man8/plipconfig.8.gz OK /usr/share/man/man8/rarp.8.gz OK /usr/share/man/man8/route.8.gz OK /usr/share/man/man8/slattach.8.gz OK /usr/share/locale/de/LC_MESSAGES/net-tools.mo OK /usr/share/locale/fr/LC_MESSAGES/net-tools.mo OK /usr/share/locale/pt_BR/LC_MESSAGES/net-tools.mo OK /usr/share/locale/et_EE/LC_MESSAGES/net-tools.mo OK /usr/share/locale/cs/LC_MESSAGES/net-tools.moOK /usr/share/doc/net-tools/README OK /usr/share/doc/net-tools/README.ipv6 OK /usr/share/doc/net-tools/TODOOK /usr/share/doc/net-tools/copyright OK /usr/share/doc/net-tools/changelog.Debian.gz OK /sbin/ifconfig OK /sbin/nameif OK /sbin/plipconfig OK /sbin/rarp OK /sbin/route OK /sbin/slattach OK /sbin/ipmaddrOK /sbin/iptunnel OK /sbin/mii-tool OK /bin/netstat OK craig -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 29 Jun 2003 13:00:57 +0800, Jason Lim wrote: Is there a document somewhere, or procedure, to recover after this? It's as simple as reinstalling. There's no other way, you can't get around this... -- L I N U X .~. The Choice /V\ of a GNU /( )\ Generation ^^-^^
Re: Server hacked - next...?
On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Sun, 2003-06-29 at 19:02, Donovan Baarda wrote: [...] Once you get compromised, it's pretty darn hard to get clean without starting fresh. Some rootkit compromises do weird stuff like infect every binary file you even 'ls'. One system I saw had been compromised via an ssh vulerability (old ssh) and rootkit'ed... there was a very good security guy doing the (remote) cleanup, and he ended up having to install buisybox just so that he had a clean environment he could work from. Dispite it being damn hard to clean up, it was just the work of a Thought I'd better clarify here... he was using buisybox to get a clean environment so he could remotely shut down the system to a bare minimium and still get critical stuff off. After that the system was wiped and fresh re-installed with new passwords. Just in case anyone was thinking this showed it was worth resurrecting a compromised system without a fresh re-install :-( In then end it is nearly always easier to re-install than to just clean the system without it, even if the hacker did leave .bash_histories behind that show everything he/she did. -- Donovan Baardahttp://minkirri.apana.org.au/~abo/
Re: Server hacked - next...?
On Mon, 30 Jun 2003 16:12, Donovan Baarda wrote: In then end it is nearly always easier to re-install than to just clean the system without it, even if the hacker did leave .bash_histories behind that show everything he/she did. Besides, just because a script-kiddie got in first does not mean that a skilled and experienced person didn't follow them. Or maybe a skilled person cracked the machine months or years before the script-kiddie... -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 11:28:47AM -0400, bda wrote: On Sun, Jun 29, 2003 at 09:47:13PM +0800, Jason Lim wrote: Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? find / -uid 0 -perm 0400 I guess this should have been 04000 Marcin -- Marcin Owsiany [EMAIL PROTECTED] http://marcin.owsiany.pl/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216
Re: Server hacked - next...?
On Sun, 2003-06-29 at 06:00, Jason Lim wrote: Hi all, Well... bad day for me. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. I know how you feel. I almost got compromised too on the same date funnily enough. They had a program running called bd which opened port 5000 which was luckily blocked by the iptables firewall which proabably helped. There were also some php scripts which gave access to the machine in the web root. The lesson that I learnt. *do not give www-data write access* even in the web root. I shoulda thought of this earlier but the setup kinda required it. I have now reworked the setup and it looks better. I used debsums to check that the binaries themselves are not compromised. I also have tiger, snort and logcheck installed to try and detect something like this. Although, neither snort nor logcheck detected anything and I installed tiger after the attack. Fingers crossed. Hope this helps, Shri -- Shri Shrikumar U R Byte Solutions Tel: 0845 644 4745 I.T. Consultant Edinburgh, Scotland Mob: 0773 980 3499 Web: www.urbyte.com Email: [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Server hacked - next...?
chkrootkit is also avaialble through apt-get apt-get install chkrootkit ## On Sun, 29 Jun 2003, Jason Lim wrote: Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux and various others) and migrate the accounts over. Thanks in advance!!! Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; debian-isp@lists.debian.org Sent: 29 June, 2003 4:02 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Mon, Jun 30, 2003 at 08:03:11PM +0200, Marcin Owsiany wrote: find / -uid 0 -perm 0400 I guess this should have been 04000 Actually, it should be find / -uid 0 -perm +4000 Sorry about that.. -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org
Re: Server hacked - next...?
Hi Daniel, Yeap, I follow Bugtraq... too bad Debian came out with the official kernel ages after it was revealed. Actually running 2.4.21 now... but it certainly is possible that during the gap between when we were running 2.4.17/18 to when we upraded to 2.4.21, someone got in. Actually, from what I can tell now, someone APPEARS to have got in through a user's vulnerable script... i think phpBB or one of those, and then obtained remote access that way, then maybe run those rootkits and stuff I am thinking it might have been related to the ptrace bug because i saw a weird modprobe entry there, but then again many rootkits fool around with modprobe??? Sincerely, Jas - Original Message - From: Daniel K. Gebhart [EMAIL PROTECTED] To: debian-isp@lists.debian.org Sent: Sunday, 29 June, 2003 6:29 PM Subject: Re: Server hacked - next...?
Re: Server hacked - next...?
As Russell Coker points out, the attaccer probably got in trough apache and a vulnerable CGI script. When you reinstall, be sure you dont run any insecure CGI's. There is probably a bunch of other improvements jou can do. DOH... I just posted saying that in my previous email before reading his message! Bah... Russell gets credit for it ;-) Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. What would the advantage of mounting /tmp with noexec be?? Definitely looking into running a hardend kernel now... especially after all this crap. Only thing that's been holding me back is the amount of work it would entail.
Re: Server hacked - next...?
Hi Shri, Good idea... installing tiger too now! Sincerely, Jas - Original Message - From: Shri Shrikumar [EMAIL PROTECTED] To: debian-isp@lists.debian.org Sent: Tuesday, 01 July, 2003 2:15 AM Subject: Re: Server hacked - next...?
Re: Server hacked - next...?
Top-posting... but please forgive. The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Sunday, 29 June, 2003 1:49 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 15:00, Jason Lim wrote: One of our servers was hacked (woody)... badly, from what I can see. A From the ps output it appears that the hack originated from the web server or a CGI-BIN script it ran. As they ran modprobe I guess they got root. :( The recommended method is to backup configuration files and data and reinstall the machine from scratch. Fighting off a hacker who is already in your machine as root is difficult. Doing it properly is more difficult than preventing them cracking your machine in the first place. Best to reinstall. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( This really, really sucks. - Original Message - From: Dena Whitebirch [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED] Sent: Sunday, 29 June, 2003 2:16 PM Subject: Re: Server hacked - next...? Hi Jason, My condolences! We've been cracked twice, both times on RH boxes, (in 10 years...so it's really not so bad) so we've got a bit of a system for cleaning up. I applaud you for wanting to clean up correctly. I've seen/heard too many horror stories out there where a user on someone else's system writes to me and shows me their cracked site and their host makes no apparent efforts to secure the box correctly. I'd like to offer any assistance I could give you. This is the time of year it always happened to us. School's out and people get bored ;) The first thing you can assume is the cracker probably has all the usernames and passwords on your system. You can also suspect that your logs and everything else on your system *may not be* telling you the truth any longer. The liklihood that you'll need to rebuild from scratch is high. It will probably, however be possible to maintain some/many user services while you do this after securing the box. The first step to this is normally to lock all users out by changing their passwords until they all change them. You'll next want to consider the cracker's motives...there are a few types of crackers. If you can figure that out it will help you decide what they may have done and the extent of the damage. Sometimes they truly want to harm you, and sometimes they want to plant things on your server so they can play with people on IRC, etc. sans.org has a pretty good section on cleanup if I remember correctly. Let me know what else I can do to help. (And don't berate yourself too badly if you're tempted to do so! Most any server can be cracked.) You may never know for sure how you got cracked as there are so many ways. Any system with users, usernames/passwords, clients uploading insecure scripts, etc. will always be somewhat vulnerable. -Dena -=Dena Whitebirch=- @quasar Internet Solutions, Inc. Internet Powered by Experience Register .MART domains and more @quasar! http://quasar.net/ On Sun, 29 Jun 2003, Jason Lim wrote: Hi all, Well... bad day for me. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Some things like: www-data 17451 0.0 0.0 2164 928 ?S02:31 0:00 /bin/sh www-data 21550 0.0 0.0 1232 236 ?S05:02 0:00 ./x www-data 21551 0.0 0.0 00 ?Z05:02 0:00 [x defunct] root 21552 0.0 0.0 00 ?Z05:02 0:00 [modprobe defunc root 21554 0.0 0.0 2148 912 ?S05:02 0:00 /bin/sh root 21755 0.0 0.0 2164 948 ?S05:02 0:00 /bin/sh root 21801 0.0 0.0 2180 964 ?S05:03 0:00 /bin/bash ./troja root 22010 0.0 0.0 1244 204 ?S05:03 0:00 ./siz ifconfigx / root 12267 0.0 0.0 00 ?Z07:15 0:00 [date defunct] root 12266 0.0 0.0 1264 252 ?T07:15 0:00 date +%d Anyone seen anything like this? Could this be the kernel hack ppl were talking about affecting 2.4.17? Guess you guys would know a lot about this stuff... Any help and suggestions greatly appreciated. Sincerely, Jas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux and various others) and migrate the accounts over. Thanks in advance!!! Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: 29 June, 2003 4:02 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 29 Jun 2003 15:00, Jason Lim wrote: One of our servers was hacked (woody)... badly, from what I can see. A From the ps output it appears that the hack originated from the web server or a CGI-BIN script it ran. As they ran modprobe I guess they got root. :( The recommended method is to backup configuration files and data and reinstall the machine from scratch. Fighting off a hacker who is already in your machine as root is difficult. Doing it properly is more difficult than preventing them cracking your machine in the first place. Best to reinstall. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 2003-06-29 at 17:15, Jason Lim wrote: Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( I think you need to find out how they got in. look around for .bash_history files to see what's in them (particularly in /root, but with some compromises they get in with other directories as HOME, so they can be other places, like /). Once you get compromised, it's pretty darn hard to get clean without starting fresh. Some rootkit compromises do weird stuff like infect every binary file you even 'ls'. One system I saw had been compromised via an ssh vulerability (old ssh) and rootkit'ed... there was a very good security guy doing the (remote) cleanup, and he ended up having to install buisybox just so that he had a clean environment he could work from. Dispite it being damn hard to clean up, it was just the work of a script-kiddy because he left .bash_history files behind that showed everything he did. moral of the story; apply security updates ASAP... -- Donovan Baardahttp://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 03:15:05PM +0800, Jason Lim [EMAIL PROTECTED] wrote: I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( Which kernel does your system run? Do you know the ptrace bug? br, daniel -- (___) Daniel K. Gebhart «http://dkg.con-fuse.org/» | «dkg_at_con-fuse.org» ( oo Key fingerprint = 10A6 A760 2635 6184 981A B19E 03AC D8F6 F412 9574 \_ | \O Have you mooed today?... pgp0.pgp Description: PGP signature
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 03:15:05PM +0800, Jason Lim wrote: Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( This really, really sucks. As Russell Coker points out, the attaccer probably got in trough apache and a vulnerable CGI script. When you reinstall, be sure you dont run any insecure CGI's. There is probably a bunch of other improvements jou can do. Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. -- Frode Haugsgjerd Norway -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi Jason, a good programm to check for rootkits can be found here: http://www.chkrootkit.org/ - Achim Am Son, 2003-06-29 um 15.47 schrieb Jason Lim: Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux and various others) and migrate the accounts over. Thanks in advance!!! Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: 29 June, 2003 4:02 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, 29 Jun 2003 23:47, Jason Lim wrote: Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? dpkg --get-selections will give you a list of installed packages. The thing to do is to boot from a CD-ROM to do all the work (otherwise you are using potentially trojaned executables), and resist the temptation to chroot to the hacked FS. You can then backup /etc (make sure you don't preserve any SETUID binaries and check all the security related files for correct contents) and blow away the root fs. Then you can do a Debian install and use dpkg --set-selections to install the right packages. If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Not all packages support this. Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? find allows this. Make sure you change all passwords. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Jason Lim: Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Consider to chroot apache, and keep available binaries to a minimum. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ IMHO there's only one save way to go after being hacked: reinstall. While you are re-installing (on another machine), limit the traffic to this machine to port 80 only, and either do web site updates yourself and/or refuse them totally until you have a replacement up and running. Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? Dunno - rpm has the option of checking md5 sums, but the dpkg manpage isn't promising in this regard. If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Check the packages that get installed in debootstrap (perhaps you could just exactly do that in a sperate tree/partition), and download and install them manually. This should get at least login, libc et al overwritten with proper binaries. If you choose to run debootstrap on a sperate partition (or machine), you may have to write a little script to gather md5sums for the fresh install, and compare to the hosed machine. Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? chkrootkit. Get it from http://www.chkrootkit.org/ Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 09:47:13PM +0800, Jason Lim wrote: The user CGIs run as the user's UID... suexec. suexec doesn't run PHP suid the owner, unless you're using php-cgi. By default, PHP is incredibly insecure. If a user is using an insecure PHP application (or any other insecure CGI application, I use PHP as an example due to the preponderance of administrators who do not realize this), they can run arbitrary commands as that user with relatively little work, and as such gain shell access. Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? find / -uid 0 -perm 0400 You will need to use a known-good copy of the `find' command. Copy it to the machine via sneakernet (by floppy) and run it locally. Even then, there's little gaurantee the command is not being tampered with while running... Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux You have few options now by now but to do a complete reinstall. There is no intermediate step when a machine has been breeched. I mention PHP above because... Briefly looking at your previous posts, it would appear that the machine was taken via the `www-data' user, suggesting either an Apache exploit, or an application running as the http daemon (the www-data user), which was not being run under suexec. From there, considering that you were (if I am recalling correctly) running a 2.4.17 kernel, which has a few known local root exploits (again, if I am recalling this correctly); I don't believe you mention if you are using Debian kernel packages, or vanilla source, or patched source. However, the attack vector seems relatively clear: Web app/server (if you're running stable, that would be 1.3.26. If you are indeed keeping up with security updates, it should be patched against the known Apache remote exploits), to shell, to kernel or suid buffer overflow or something of that nature. From that point, they have root access. And don't bother hiding themselves at all, which is lucky for you, really. Of course, all of the above is simply a hypothesis based on incomplete information, but... I would suggest taking a look at what CGI you or your customers are running, and searching the web for known security issues with them. -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Just backup the datafiles, the password files and rebuild. At least that is how I would do it. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. I am curious as to how your server has gotten hacked. More specifically, I run Debian on about 15 of my servers and none of them have been hacked. Maybe I am just lucky, but I trust Debian stable and Debian Sarge more than any other Distro. To prevent hackers from getting in I follow the Debian Security HOWTO, and I keep them all updated. Additionally, I run the minimum of services. I only run services which are needed. Hopefully these questions I have will help you secure your system a little better in the future. Questions: 1. How many users were able to login in to this box and call a shell ? Most hackers use regular user accounts to gain access to higher privilege parts of the OS. 2. Did you setup any security as defined in the Debian Security HOWTO ? http://www.debian.org/doc/user-manuals#securing A. Did you run IP-tables or ipchains ? B. Did you use TCP wrappers in the /etc/hosts.allow and /etc/hosts.deny Much of which follow comes from the Debian Security Howto. 3. How many services did you have running on the hacked machine ? i.e. sshd daemon apache daemon mysqld Limiting services to only the services needed helps make your systems exposure to potential exploits less. Some people in the RedHat world install everything. This could result in problems if the system does not get updated on daily basis. 4. Was the machine updated on a regular basis ? When was the last update ? Did you use the security updates ? /etc/apt/source.lists #for stable deb http://security.debian.org/ stable/updates main contrib non-free #for sarge deb http://security.debian.org/ sarge/updates main contrib non-free 5. How long was the root password and could anyone su to root ? Simple root passwords can be guessed. Don't type 'root' as the root password or simple dictionary words. 6. Did you allow root logins from a remote location ? I do not. This is defined in the /etc/ssh/sshd_config with this line. PermitRootLogin no This will help limit dictionary attacks on root. 7. Did you allow telnet and FTP on this machine ? If I allow FTP, I do not give them shell access. For example, the PAM ssh file allows you to limit who can login to a shell. Pam limits shell access to with the following line in the /etc/pam.d/ssh file: authrequiredpam_listfile.so item=user sense=allow \ file=/etc/sshusers-allowed onerr=fail In this example, the '/etc/sshusers-allowed' holds the names of people who can use ssh. 8. Do you have any external firewall ? If so what ? -- -\ - /- --([0]-[0])-- +oOOo-(_)-oOOo--+ | Theodore Knab | | Annapolis Linux LUG when not a sysadmin | +---+ |oOOo | | ()oOOo | +\ (( )--+ \_) ) / (-/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Il 29 Jun 2003 alle 21:47, Jason Lim immise in rete: Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? The problem occurred me some months ago. My solution was to reinstall from scratch one machine, then just copyiing all binaries into the compromised one. It worked fine. -- Leonardo Boselli Nucleo Informatico e Telematico del Dipartimento Ingegneria Civile Universita` di Firenze , V. S. Marta 3 - I-50139 Firenze tel +39 0554796431 cell +39 3488605348 fax +39 055495333 http://www.dicea.unifi.it/~leo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
El dom, 29 de 06 de 2003 a las 02:15, Jason Lim escribió: Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( You have to realize this is a normal step in the life of any sysadmin. So stop being worried and learn from it. 1.- Save all thats possible to save (homedirs, emails, homepages) 2.- Yeah, hard to believe an updated, all standard packages woody could be cracked. Its no normal, highschool script kiddie if he pulled that off (probably a college script kiddie though...;)...). Your box as is provides very good information, but you have to realize that, if you didnt take a couple of steps to forsee this, such as having a network flight recorder somewhere to do forensics on your dead box, its going to be hard to determine where and how did he got in. 2-1/2.- Do a list of ANY installed stuff that is not strict debian woody. I mean, web database administrators, counters, extra perl modules got from cpan (as oposed from apt-get isntall libperl...etc.). Its more probable that the first level vulnerability got in there (nevertheless, if you got hacked by a perl script, then the perl package, apache package or similar is borked). 3.- So, mirror your killed hard drive so that you can disect it later, set up the box again with certain limited things, say forbid cgi's and move to mod-perl and php, forbid ppl from having bash cgi's (since there is a good chance this is where they got in). What am i doing? I dunno, there is no checklist that will cover any site, this is what i would do and im not very experienced. But whatever you end up with, you should implement postmortem analysis capabilities to your site (couple of snort/tcpdump boxes and an actual formalization of your security policies will do). So policy is the thing here anyhow, work on that. Think of syslog-ng server, your tcpdump network capture server, snort ID analysys server, log analyzer for the syslog server. Once cracked all one can do is think better for the next time. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Server hacked - next...?
Hi all, Well... bad day for me. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Some things like: www-data 17451 0.0 0.0 2164 928 ?S02:31 0:00 /bin/sh www-data 21550 0.0 0.0 1232 236 ?S05:02 0:00 ./x www-data 21551 0.0 0.0 00 ?Z05:02 0:00 [x defunct] root 21552 0.0 0.0 00 ?Z05:02 0:00 [modprobe defunc root 21554 0.0 0.0 2148 912 ?S05:02 0:00 /bin/sh root 21755 0.0 0.0 2164 948 ?S05:02 0:00 /bin/sh root 21801 0.0 0.0 2180 964 ?S05:03 0:00 /bin/bash ./troja root 22010 0.0 0.0 1244 204 ?S05:03 0:00 ./siz ifconfigx / root 12267 0.0 0.0 00 ?Z07:15 0:00 [date defunct] root 12266 0.0 0.0 1264 252 ?T07:15 0:00 date +%d Anyone seen anything like this? Could this be the kernel hack ppl were talking about affecting 2.4.17? Guess you guys would know a lot about this stuff... Any help and suggestions greatly appreciated. Sincerely, Jas
Re: Server hacked - next...?
Top-posting... but please forgive. The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; debian-isp@lists.debian.org Sent: Sunday, 29 June, 2003 1:49 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 15:00, Jason Lim wrote: One of our servers was hacked (woody)... badly, from what I can see. A From the ps output it appears that the hack originated from the web server or a CGI-BIN script it ran. As they ran modprobe I guess they got root. :( The recommended method is to backup configuration files and data and reinstall the machine from scratch. Fighting off a hacker who is already in your machine as root is difficult. Doing it properly is more difficult than preventing them cracking your machine in the first place. Best to reinstall. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( This really, really sucks. - Original Message - From: Dena Whitebirch [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED] Sent: Sunday, 29 June, 2003 2:16 PM Subject: Re: Server hacked - next...? Hi Jason, My condolences! We've been cracked twice, both times on RH boxes, (in 10 years...so it's really not so bad) so we've got a bit of a system for cleaning up. I applaud you for wanting to clean up correctly. I've seen/heard too many horror stories out there where a user on someone else's system writes to me and shows me their cracked site and their host makes no apparent efforts to secure the box correctly. I'd like to offer any assistance I could give you. This is the time of year it always happened to us. School's out and people get bored ;) The first thing you can assume is the cracker probably has all the usernames and passwords on your system. You can also suspect that your logs and everything else on your system *may not be* telling you the truth any longer. The liklihood that you'll need to rebuild from scratch is high. It will probably, however be possible to maintain some/many user services while you do this after securing the box. The first step to this is normally to lock all users out by changing their passwords until they all change them. You'll next want to consider the cracker's motives...there are a few types of crackers. If you can figure that out it will help you decide what they may have done and the extent of the damage. Sometimes they truly want to harm you, and sometimes they want to plant things on your server so they can play with people on IRC, etc. sans.org has a pretty good section on cleanup if I remember correctly. Let me know what else I can do to help. (And don't berate yourself too badly if you're tempted to do so! Most any server can be cracked.) You may never know for sure how you got cracked as there are so many ways. Any system with users, usernames/passwords, clients uploading insecure scripts, etc. will always be somewhat vulnerable. -Dena -=Dena Whitebirch=- @quasar Internet Solutions, Inc. Internet Powered by Experience Register .MART domains and more @quasar! http://quasar.net/ On Sun, 29 Jun 2003, Jason Lim wrote: Hi all, Well... bad day for me. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Some things like: www-data 17451 0.0 0.0 2164 928 ?S02:31 0:00 /bin/sh www-data 21550 0.0 0.0 1232 236 ?S05:02 0:00 ./x www-data 21551 0.0 0.0 00 ?Z05:02 0:00 [x defunct] root 21552 0.0 0.0 00 ?Z05:02 0:00 [modprobe defunc root 21554 0.0 0.0 2148 912 ?S05:02 0:00 /bin/sh root 21755 0.0 0.0 2164 948 ?S05:02 0:00 /bin/sh root 21801 0.0 0.0 2180 964 ?S05:03 0:00 /bin/bash ./troja root 22010 0.0 0.0 1244 204 ?S05:03 0:00 ./siz ifconfigx / root 12267 0.0 0.0 00 ?Z07:15 0:00 [date defunct] root 12266 0.0 0.0 1264 252 ?T07:15 0:00 date +%d Anyone seen anything like this? Could this be the kernel hack ppl were talking about affecting 2.4.17? Guess you guys would know a lot about this stuff... Any help and suggestions greatly appreciated. Sincerely, Jas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Server hacked - next...?
Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux and various others) and migrate the accounts over. Thanks in advance!!! Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; debian-isp@lists.debian.org Sent: 29 June, 2003 4:02 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Sun, 29 Jun 2003 15:00, Jason Lim wrote: One of our servers was hacked (woody)... badly, from what I can see. A From the ps output it appears that the hack originated from the web server or a CGI-BIN script it ran. As they ran modprobe I guess they got root. :( The recommended method is to backup configuration files and data and reinstall the machine from scratch. Fighting off a hacker who is already in your machine as root is difficult. Doing it properly is more difficult than preventing them cracking your machine in the first place. Best to reinstall. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Sun, 2003-06-29 at 17:15, Jason Lim wrote: Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( I think you need to find out how they got in. look around for .bash_history files to see what's in them (particularly in /root, but with some compromises they get in with other directories as HOME, so they can be other places, like /). Once you get compromised, it's pretty darn hard to get clean without starting fresh. Some rootkit compromises do weird stuff like infect every binary file you even 'ls'. One system I saw had been compromised via an ssh vulerability (old ssh) and rootkit'ed... there was a very good security guy doing the (remote) cleanup, and he ended up having to install buisybox just so that he had a clean environment he could work from. Dispite it being damn hard to clean up, it was just the work of a script-kiddy because he left .bash_history files behind that showed everything he did. moral of the story; apply security updates ASAP... -- Donovan Baardahttp://minkirri.apana.org.au/~abo/
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 03:15:05PM +0800, Jason Lim [EMAIL PROTECTED] wrote: I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( Which kernel does your system run? Do you know the ptrace bug? br, daniel -- (___) Daniel K. Gebhart «http://dkg.con-fuse.org/» | «dkg_at_con-fuse.org» ( oo Key fingerprint = 10A6 A760 2635 6184 981A B19E 03AC D8F6 F412 9574 \_ | \O Have you mooed today?... pgpgAwDRiHLIE.pgp Description: PGP signature
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 03:15:05PM +0800, Jason Lim wrote: Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( This really, really sucks. As Russell Coker points out, the attaccer probably got in trough apache and a vulnerable CGI script. When you reinstall, be sure you dont run any insecure CGI's. There is probably a bunch of other improvements jou can do. Mount /tmp with noexec Run a hardened kernel like NSA or Grsecurity. etc. -- Frode Haugsgjerd Norway
Re: Server hacked - next...?
Hi Jason, a good programm to check for rootkits can be found here: http://www.chkrootkit.org/ - Achim Am Son, 2003-06-29 um 15.47 schrieb Jason Lim: Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux and various others) and migrate the accounts over. Thanks in advance!!! Sincerely, Jason - Original Message - From: Russell Coker [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED]; debian-isp@lists.debian.org Sent: 29 June, 2003 4:02 PM Subject: Re: Server hacked - next...? On Sun, 29 Jun 2003 17:12, Jason Lim wrote: The box is a very recently updated stable box... virtually every other date apt-get is update/upgrade. The box is setup very secure... the usual things were done... like ensuring no unused services are running and things like that. So does that mean stable is actually vulnerable to something we all don't know about??? That could be the case. Or it could be some issue of your configuration. Maybe you have Apache set to run customer cgi-bin scripts under the same UID and a customer uploaded an insecure or hostile cgi-bin script. Have you considered using SE Linux? -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
On Sun, 29 Jun 2003 23:47, Jason Lim wrote: Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? dpkg --get-selections will give you a list of installed packages. The thing to do is to boot from a CD-ROM to do all the work (otherwise you are using potentially trojaned executables), and resist the temptation to chroot to the hacked FS. You can then backup /etc (make sure you don't preserve any SETUID binaries and check all the security related files for correct contents) and blow away the root fs. Then you can do a Debian install and use dpkg --set-selections to install the right packages. If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Not all packages support this. Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? find allows this. Make sure you change all passwords. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Server hacked - next...?
Jason Lim: Hi Russell, Well, SE Linux certainly seems like something that needs to be installed. Most annoying is that all the recent security updates were already done! The user CGIs run as the user's UID... suexec. Consider to chroot apache, and keep available binaries to a minimum. Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ IMHO there's only one save way to go after being hacked: reinstall. While you are re-installing (on another machine), limit the traffic to this machine to port 80 only, and either do web site updates yourself and/or refuse them totally until you have a replacement up and running. Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? Dunno - rpm has the option of checking md5 sums, but the dpkg manpage isn't promising in this regard. If just a list of packages could be shown that do not match what is actually on the disk, those could be re-downloaded and re-installed, so at least the system can start working (right now, just typing gcc produces garbage on the screen, no doubt because some libraries have been replaced). Check the packages that get installed in debootstrap (perhaps you could just exactly do that in a sperate tree/partition), and download and install them manually. This should get at least login, libc et al overwritten with proper binaries. If you choose to run debootstrap on a sperate partition (or machine), you may have to write a little script to gather md5sums for the fresh install, and compare to the hosed machine. Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? chkrootkit. Get it from http://www.chkrootkit.org/ Thomas
Re: Server hacked - next...?
On Sun, Jun 29, 2003 at 09:47:13PM +0800, Jason Lim wrote: The user CGIs run as the user's UID... suexec. suexec doesn't run PHP suid the owner, unless you're using php-cgi. By default, PHP is incredibly insecure. If a user is using an insecure PHP application (or any other insecure CGI application, I use PHP as an example due to the preponderance of administrators who do not realize this), they can run arbitrary commands as that user with relatively little work, and as such gain shell access. Is there any tool that could search the system for root suid scripts (so the hacker can login again and gain root easily)? find / -uid 0 -perm 0400 You will need to use a known-good copy of the `find' command. Copy it to the machine via sneakernet (by floppy) and run it locally. Even then, there's little gaurantee the command is not being tampered with while running... Hope you can shed some light on the above, so at least the system can get back up and running, then we can even setup a new server (with SE Linux You have few options now by now but to do a complete reinstall. There is no intermediate step when a machine has been breeched. I mention PHP above because... Briefly looking at your previous posts, it would appear that the machine was taken via the `www-data' user, suggesting either an Apache exploit, or an application running as the http daemon (the www-data user), which was not being run under suexec. From there, considering that you were (if I am recalling correctly) running a 2.4.17 kernel, which has a few known local root exploits (again, if I am recalling this correctly); I don't believe you mention if you are using Debian kernel packages, or vanilla source, or patched source. However, the attack vector seems relatively clear: Web app/server (if you're running stable, that would be 1.3.26. If you are indeed keeping up with security updates, it should be patched against the known Apache remote exploits), to shell, to kernel or suid buffer overflow or something of that nature. From that point, they have root access. And don't bother hiding themselves at all, which is lucky for you, really. Of course, all of the above is simply a hypothesis based on incomplete information, but... I would suggest taking a look at what CGI you or your customers are running, and searching the web for known security issues with them. -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org
Re: Server hacked - next...?
Il 29 Jun 2003 alle 21:47, Jason Lim immise in rete: Re-installing from scratch would be a real pain... the server runs on a 3ware array, and has hundreds of users, all active :-/ Is there any way to verify the Integrity of the files somehow, and download/re-install any binaries that do not match the checksums or something? Does dpkg or some other Debian tool have this ability? The problem occurred me some months ago. My solution was to reinstall from scratch one machine, then just copyiing all binaries into the compromised one. It worked fine. -- Leonardo Boselli Nucleo Informatico e Telematico del Dipartimento Ingegneria Civile Universita` di Firenze , V. S. Marta 3 - I-50139 Firenze tel +39 0554796431 cell +39 3488605348 fax +39 055495333 http://www.dicea.unifi.it/~leo
Re: Server hacked - next...?
El dom, 29 de 06 de 2003 a las 02:15, Jason Lim escribió: Okay... so supposing the whole system needs to be installed, we can make a backup of the home directory now... but after we restore everything, what is to stop the hacker immediately re-gaining access again? The server is a fully updated stable debian system. In fact, it was updated just yesterday. I'm thinking that even if we do all the trouble of a complete re-installation of the entire system, it won't fix this as it will get re-hacked again, especailly since we can't see what is going on anymore. What do you think? :-( You have to realize this is a normal step in the life of any sysadmin. So stop being worried and learn from it. 1.- Save all thats possible to save (homedirs, emails, homepages) 2.- Yeah, hard to believe an updated, all standard packages woody could be cracked. Its no normal, highschool script kiddie if he pulled that off (probably a college script kiddie though...;)...). Your box as is provides very good information, but you have to realize that, if you didnt take a couple of steps to forsee this, such as having a network flight recorder somewhere to do forensics on your dead box, its going to be hard to determine where and how did he got in. 2-1/2.- Do a list of ANY installed stuff that is not strict debian woody. I mean, web database administrators, counters, extra perl modules got from cpan (as oposed from apt-get isntall libperl...etc.). Its more probable that the first level vulnerability got in there (nevertheless, if you got hacked by a perl script, then the perl package, apache package or similar is borked). 3.- So, mirror your killed hard drive so that you can disect it later, set up the box again with certain limited things, say forbid cgi's and move to mod-perl and php, forbid ppl from having bash cgi's (since there is a good chance this is where they got in). What am i doing? I dunno, there is no checklist that will cover any site, this is what i would do and im not very experienced. But whatever you end up with, you should implement postmortem analysis capabilities to your site (couple of snort/tcpdump boxes and an actual formalization of your security policies will do). So policy is the thing here anyhow, work on that. Think of syslog-ng server, your tcpdump network capture server, snort ID analysys server, log analyzer for the syslog server. Once cracked all one can do is think better for the next time.
Server hacked - next...?
Hi all, Well... bad day for me. One of our servers was hacked (woody)... badly, from what I can see. A whole bunch of binaries have been modified, and strange processes are running on the server. The hack date appears to be jun 6. Is there a document somewhere, or procedure, to recover after this? This is a working and running system, so somehow need to be able to recover from this with minimal impact to end-users. Some things like: www-data 17451 0.0 0.0 2164 928 ?S02:31 0:00 /bin/sh www-data 21550 0.0 0.0 1232 236 ?S05:02 0:00 ./x www-data 21551 0.0 0.0 00 ?Z05:02 0:00 [x defunct] root 21552 0.0 0.0 00 ?Z05:02 0:00 [modprobe defunc root 21554 0.0 0.0 2148 912 ?S05:02 0:00 /bin/sh root 21755 0.0 0.0 2164 948 ?S05:02 0:00 /bin/sh root 21801 0.0 0.0 2180 964 ?S05:03 0:00 /bin/bash ./troja root 22010 0.0 0.0 1244 204 ?S05:03 0:00 ./siz ifconfigx / root 12267 0.0 0.0 00 ?Z07:15 0:00 [date defunct] root 12266 0.0 0.0 1264 252 ?T07:15 0:00 date +%d Anyone seen anything like this? Could this be the kernel hack ppl were talking about affecting 2.4.17? Guess you guys would know a lot about this stuff... Any help and suggestions greatly appreciated. Sincerely, Jas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
