Re: HELP, my Debian Server was hacked!
Il Thu, 24 Apr 2003 07:03:48 -0500 (CDT) David Ehle sì che favelando sibillò: > I use a cronjob. I'll send it to you privatly, if anyone else wants it > let me know. > David. Thanks a lot, i'll be happy to see it. -- sracatus
Re: HELP, my Debian Server was hacked!
On Thu, 24 Apr 2003, Dale Amon wrote: > On Wed, Apr 23, 2003 at 10:44:34AM -0400, James Duncan wrote: > > Obviously steps should be in place to mitigate the damage of these sorts > > of acts. Have steps in place to quickly replace machines that have to be > > removed from production quickly and without warning. Use syslog to log > > locally AND remotely. Have a backup of all your logs. The smart attacker > > will have covered their tracks. > > I'd go further. If you know the machine has been > hacked, pull the ethernet, copy the disks and swap to > CD if you have time... Data protection procedures will vary depending on internal security policy (which every company should have - there should be a clear set of guidelines in place that states how these sorts of situations should be handled). Certainly this is a viable solution, and one I have practiced myself in situations where speed was more important than forensic preservation. > Then just wipe it and re-install. It's a very rare > facility that actually has time for forensics. Places > with deep enough pockets to have a senior person > grepping swap disks and reconstructing activity on > one single machine and taking perhaps days or even > weeks to do it. > > It just doesn't happen very often. Typically this won't be a job for the admins, anyway. Data like this should be looked at by either a dedicated information security officer, whose job it is to track these things, or by law inforcement, should one choose to involve them. Just because you don't have time to look at the data, doesn't mean that you can't take a few minutes to preserve it in the first place. If the attack in question can be shown to have cost your company palpable revenue, then the preservation of this data could be the difference between being rewarded damages, and a case that is thrown out of court.
Re: HELP, my Debian Server was hacked!
On Thu, Apr 24, 2003 at 01:16:49PM +, simon raven wrote: > Le Thu, Apr 24, 2003 at 08:48:27 -0400, Raymond Wood a ?crit: > > On Thu, Apr 24, 2003 at 02:17:48PM +0200, Adam ENDRODI imagined: > > [snip] > > > Details on how to implement this have been discusssed in the > > > list several times and an adequate answer can be found in the > > > Debian Security FAQ. Upon request, I can send the perl script > > > we're using on daily basis. > > > [snip] > > I would be most interested in seeing the perl script. > > > as would i :). > eric > Me too please - Sorry for replying to the list. Lost the original posters email address. Thanks and regards, David. -- .''`. David Ramsden <[EMAIL PROTECTED]> : :' :http://portal.hexstream.eu.org/ `. `'` PGP key ID: 507B379B on wwwkeys.pgp.net `- Debian - when you have better things to do than to fix a system. pgpJ5Wr5e8gm1.pgp Description: PGP signature
Re: HELP, my Debian Server was hacked!
Le Thu, Apr 24, 2003 at 08:48:27 -0400, Raymond Wood a écrit: > On Thu, Apr 24, 2003 at 02:17:48PM +0200, Adam ENDRODI imagined: > > > On Thu, Apr 24, 2003 at 11:43:06AM +0200, I.R. van Dongen wrote: > > > > > > lamorak:~# crontab -l > > > @daily apt-get -q -q -q -q update && apt-get -s -q -q -q -q > > > dist-upgrade > > > Before you deploy such a mechanism, I advise that you set up > > another one between the "update" and "upgrade" which checks > > the authority of the downloaded databases. > > > > Details on how to implement this have been discusssed in the > > list several times and an adequate answer can be found in the > > Debian Security FAQ. Upon request, I can send the perl script > > we're using on daily basis. > > > > bit, > > adam > > I would be most interested in seeing the perl script. > as would i :). eric > TIA, > Raymond > -- > "What difference does it make to the dead, the orphans, and the > homeless, whether the mad destruction is brought under the name > of totalitarianism or the holy name of liberty and democracy?" > (Gandhi) -- UNIX is user friendly, it's just picky about who its friends are. --- ,''`. http://www.debian.org/ | http://www.nuit.ca/ : :' : Debian GNU/Linux| http://simonraven.nuit.ca/ `. `' | PGP key ID: 6169 BE0C 0891 A038 `- |
Re: HELP, my Debian Server was hacked!
On Thu, Apr 24, 2003 at 02:17:48PM +0200, Adam ENDRODI imagined: > On Thu, Apr 24, 2003 at 11:43:06AM +0200, I.R. van Dongen wrote: > > > > lamorak:~# crontab -l > > @daily apt-get -q -q -q -q update && apt-get -s -q -q -q -q > > dist-upgrade > Before you deploy such a mechanism, I advise that you set up > another one between the "update" and "upgrade" which checks > the authority of the downloaded databases. > > Details on how to implement this have been discusssed in the > list several times and an adequate answer can be found in the > Debian Security FAQ. Upon request, I can send the perl script > we're using on daily basis. > > bit, > adam I would be most interested in seeing the perl script. TIA, Raymond -- "What difference does it make to the dead, the orphans, and the homeless, whether the mad destruction is brought under the name of totalitarianism or the holy name of liberty and democracy?" (Gandhi) pgpuYfebIXSrT.pgp Description: PGP signature
Re: HELP, my Debian Server was hacked!
I'm really sorry, I didn't notice that when I wrote my message François TOURDE wrote: Le 12166ième jour après Epoch, Nicolas Sulek écrivait: [SNIP...] Please, please, please... No HTML in text messages... Even if you run NT on your box :)
Re: Re: HELP, my Debian Server was hacked!
On Thu, Apr 24, 2003 at 11:43:06AM +0200, I.R. van Dongen wrote: > > lamorak:~# crontab -l > @daily apt-get -q -q -q -q update && apt-get -s -q -q -q -q > dist-upgrade Before you deploy such a mechanism, I advise that you set up another one between the "update" and "upgrade" which checks the authority of the downloaded databases. Details on how to implement this have been discusssed in the list several times and an adequate answer can be found in the Debian Security FAQ. Upon request, I can send the perl script we're using on daily basis. bit, adam -- 1024D/37B8D989 954B 998A E5F5 BA2A 3622 82DD 54C2 843D 37B8 D989 finger://[EMAIL PROTECTED] | Some days, my soul's confined http://www.keyserver.net | And out of mind Sleep forever
Re: HELP, my Debian Server was hacked!
> Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) > David Ehle sì che favelando > sibillò: > > > nightly apt-get update && apt-get upgrade > > But if it asks human interaction?? How can i do?? I use a cronjob. I'll send it to you privatly, if anyone else wants it let me know. David. > > -- > sracatus > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: HELP, my Debian Server was hacked!
Le 12166ième jour après Epoch, Mathias Gygax écrivait: > On Don, Apr 24, 2003 at 11:19:34 +0200, Mauro Chiarugi wrote: > > Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) > > David Ehle sì che favelando > > sibillò: > > > > > nightly apt-get update && apt-get upgrade > > > > But if it asks human interaction?? How can i do?? > > from the apt-get manual page: > > [...] >-y > >--yes > >--assume-yes > Automatic yes to prompts; assume "yes" as answer to all prompts > and run non-interactively. If an undesirable situation, such as > changing a held package or removing an essential package occurs > then apt-get will abort. Configuration Item: APT::Get::Assume- > Yes. > > [...] > > be sure to also "dpkg-reconfigure debconf" and set it to not ask trivial > questions. > > cron-apt is a package to automate apt-get handling via cron. it could > assist you in setting up automatic security upgrades. But be very careful with this mechanism... If you're not on stable, some evil problems may occurs. Remember the libc6 problem on testing/unstable weeks ago. Mathias, sorry for the question, but are you a parent of the well known Gary? If so, your family is responsible of most of my nightmares :) -- Romeo wasn't bilked in a day. -- Walt Kelly, "Ten Ever-Lovin' Blue-Eyed Years With Pogo" -- François TOURDE - tourde.org - 23 rue Bernard GANTE - 93250 VILLEMOMBLE Tél: 01 49 35 96 69 - Mob: 06 81 01 81 80 eMail: mailto:[EMAIL PROTECTED] - URL: http://francois.tourde.org/
Re: HELP, my Debian Server was hacked!
Le 12166ième jour après Epoch, Nicolas Sulek écrivait: > > [SNIP...] > Please, please, please... No HTML in text messages... Even if you run NT on your box :) -- QOTD: "What I like most about myself is that I'm so understanding when I mess things up." -- François TOURDE - tourde.org - 23 rue Bernard GANTE - 93250 VILLEMOMBLE Tél: 01 49 35 96 69 - Mob: 06 81 01 81 80 eMail: mailto:[EMAIL PROTECTED] - URL: http://francois.tourde.org/
Re: HELP, my Debian Server was hacked!
On Thu, Apr 24, 2003 at 04:02:56AM +0100, Dale Amon wrote: > On Wed, Apr 23, 2003 at 10:44:34AM -0400, James Duncan wrote: > > Obviously steps should be in place to mitigate the damage of these sorts > > of acts. Have steps in place to quickly replace machines that have to be > > removed from production quickly and without warning. Use syslog to log > > locally AND remotely. Have a backup of all your logs. The smart attacker > > will have covered their tracks. > > I'd go further. If you know the machine has been > hacked, pull the ethernet, copy the disks and swap to > CD if you have time... > > Then just wipe it and re-install. It's a very rare > facility that actually has time for forensics. Places > with deep enough pockets to have a senior person > grepping swap disks and reconstructing activity on > one single machine and taking perhaps days or even > weeks to do it. > > It just doesn't happen very often. There are those of us that actually do this kind of stuff for hire, as long as we can get good images of the disk and /proc. dd is best Tim -- >< >> Tim Sailer (at home) >< Coastal Internet, Inc. << >> Network and Systems Operations >< PO Box 671 << >> http://www.buoy.com >< Ridge, NY 11961 << >> [EMAIL PROTECTED]/[EMAIL PROTECTED] >< (631)924-3728 (888) 924-3728 >> << ><
Re: HELP, my Debian Server was hacked!
On Don, Apr 24, 2003 at 11:19:34 +0200, Mauro Chiarugi wrote: > Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) > David Ehle sì che favelando > sibillò: > > > nightly apt-get update && apt-get upgrade > > But if it asks human interaction?? How can i do?? from the apt-get manual page: [...] -y --yes --assume-yes Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively. If an undesirable situation, such as changing a held package or removing an essential package occurs then apt-get will abort. Configuration Item: APT::Get::Assume- Yes. [...] be sure to also "dpkg-reconfigure debconf" and set it to not ask trivial questions. cron-apt is a package to automate apt-get handling via cron. it could assist you in setting up automatic security upgrades. - regards, turrican
Re: HELP, my Debian Server was hacked!
On Thu, Apr 24, 2003 at 11:19:34AM +0200, Mauro Chiarugi wrote: > Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) > David Ehle s? che favelando > sibill?: > > > nightly apt-get update && apt-get upgrade > > But if it asks human interaction?? How can i do?? > apt-get --assume-yes upgrade That'll answer 'yes' to any questions asked (like, asking you if you want to install the package). I'm not a huge fan of doing this, without me actually executing it because it may break something or stop a service etc. etc. I have a shell script that's in /etc/cron.daily that looks like: #!/bin/sh # apt-get update apt-get --assume-yes --simulate upgrade #EOF So when it's executed, it does an apt-get update as normal and then does a simulated upgrade. So it shows you what would happen if there are packages to be upgraded but it doesn't actually apply it. With any output to STDOUT from cron, you get an email about it so this way I can look at these everyday and see which servers have updates availble for them and what will happen if I apply them. HTH, David. -- .''`. David Ramsden <[EMAIL PROTECTED]> : :' :http://portal.hexstream.eu.org/ `. `'` PGP key ID: 507B379B on wwwkeys.pgp.net `- Debian - when you have better things to do than to fix a system. pgpVUzBt1Y9OZ.pgp Description: PGP signature
Re: HELP, my Debian Server was hacked!
you can use cron-apt cron-apt - Automatic update of packages using apt Mauro Chiarugi wrote: Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) David Ehle sì che favelando sibillò: nightly apt-get update && apt-get upgrade But if it asks human interaction?? How can i do?? -- sracatus
Re: Re: HELP, my Debian Server was hacked!
lamorak:~# crontab -l @daily apt-get -q -q -q -q update && apt-get -s -q -q -q -q dist-upgrade make sure the output is mailed to an address you use daily. When an update is available you will be mailed, otherwise you get no mail. Gr, Ivo van Dongen On Thu, 24 Apr 2003 11:19:34 +0200, [EMAIL PROTECTED] wrote: > Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) > David Ehle sì che favelando > sibillò: > > > nightly apt-get update && apt-get upgrade > > But if it asks human interaction?? How can i do?? > > -- > sracatus > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > >
Re: HELP, my Debian Server was hacked!
Il Tue, 22 Apr 2003 17:48:23 -0500 (CDT) David Ehle sì che favelando sibillò: > nightly apt-get update && apt-get upgrade But if it asks human interaction?? How can i do?? -- sracatus
Re: HELP, my Debian Server was hacked!
On Wed, Apr 23, 2003 at 10:44:34AM -0400, James Duncan wrote: > Obviously steps should be in place to mitigate the damage of these sorts > of acts. Have steps in place to quickly replace machines that have to be > removed from production quickly and without warning. Use syslog to log > locally AND remotely. Have a backup of all your logs. The smart attacker > will have covered their tracks. I'd go further. If you know the machine has been hacked, pull the ethernet, copy the disks and swap to CD if you have time... Then just wipe it and re-install. It's a very rare facility that actually has time for forensics. Places with deep enough pockets to have a senior person grepping swap disks and reconstructing activity on one single machine and taking perhaps days or even weeks to do it. It just doesn't happen very often. -- -- IN MY NAME:Dale Amon, CEO/MD No Mushroom clouds over Islandone Society London and New York. www.islandone.org --
RE: HELP, my Debian Server was hacked!
On Wed, 23 Apr 2003, DEFFONTAINES Vincent wrote: > What to do > --- > > The first 3 basic steps to handling a "situation" (roughly taken from > the wonderful Criminalistics, An Introduction to Forensic Science, by > Saferstein (see the "bibliography" file) are: > > o Secure and isolate the scene > o Record the scene > o Conduct a systematic search for evidence Good general guidelines for a static forensic situation... however, some significant differences exist between a static crime scene and a system break-in. The most important of these being, that while a crime scene can be secured, a computer that has been broken into is immediately suspect and is quite probably still being used to conduct criminal activities. > And while speed is of the essence, attempt to stay calm and don't panic. Of course. :) > And do *NOT* touch the keyboard or the computer yet unless you absolutely > have to. > > We repeat. Do *NOT* touch the keyboard or the computer yet. > > Did you hear us? STAY AWAY FROM THE COMPUTER! Anything you do will > destroy evidence, so simply don't touch it for now, or do as little as > possible and don't start looking for damage yet. And what is the correct time to start looking for damage?? I have to disagree here. There are important corporate legal concerns outside of simply collecting evidence, the most primary of these being liability. In some jurisdictions, if you are aware that your machine is under someone else's control, and that machine is currently being used in the commission of an attack upon a 3rd party machine, that 3rd party might be able to hold you liable. IANAL, but be sure to check all the applicable laws for your jurisdiction to be sure. Immediate action is ALWAYS required in a case like this. The moment you discover a break-in, get down to your datacenter, get on a physical console, and begin the process of cleaning the mess up. This should involve steps like the following: - tar up /proc and move to another machine - take an image of all mounted and unmounted filesystems and put on read-only media (becomes redundant later, but redundancy is good) - make a quick examination of open sockets - check netstat, run nmap, etc... but do this quickly. if you want to spend some time at this, firewall off the machine and put a packet sniffer in front of it. - shut down the machine, remove the drives, and mount them in another machine -o ro for closer examination. Obviously steps should be in place to mitigate the damage of these sorts of acts. Have steps in place to quickly replace machines that have to be removed from production quickly and without warning. Use syslog to log locally AND remotely. Have a backup of all your logs. The smart attacker will have covered their tracks. > And while you might get lucky and find all the damage and evidence and > perpetrator immediately, don't get your hopes up too much, this is still > not an exact science, and almost every case has more than its share of > disappointments. Agreed.
RE: HELP, my Debian Server was hacked!
Have a look at the coroner toolkit from Dan Farmer and Wietse Venema. Debian packaged : tct It is advised *not* to turn off your box, maybe you can unplug its network... not sure its a good idea even. http://www.fish.com/tct/help-when-broken-into Chosen extract : What to do --- The first 3 basic steps to handling a "situation" (roughly taken from the wonderful Criminalistics, An Introduction to Forensic Science, by Saferstein (see the "bibliography" file) are: o Secure and isolate the scene o Record the scene o Conduct a systematic search for evidence And while speed is of the essence, attempt to stay calm and don't panic. And do *NOT* touch the keyboard or the computer yet unless you absolutely have to. We repeat. Do *NOT* touch the keyboard or the computer yet. Did you hear us? STAY AWAY FROM THE COMPUTER! Anything you do will destroy evidence, so simply don't touch it for now, or do as little as possible and don't start looking for damage yet. And while you might get lucky and find all the damage and evidence and perpetrator immediately, don't get your hopes up too much, this is still not an exact science, and almost every case has more than its share of disappointments.
Re: HELP, my Debian Server was hacked!
Hi, Boot your machine in single user. Run a md5sum in /sbin/init and compare with a 'secure' machine. Download http://www.chkrootkit.org and run it. It's recommended to run chkrootkit using your own static binaries on another path or CDROM (you can see which binaries is needed on chkrootkit website). chkrootkit provides a 'string' binary. Run it on /sbin/init and look for strange expressions (usually FUCK or something like that). Its recommended to run a nightly apt-get update and apt-get upgrade to keep your machines safe. :-) There are a lot of exploits for openssl, the most used is openssl-too-open and it can exploit an non-updated version of Woody. On Tue, Apr 22, 2003 at 09:00:11PM +0200, Christian Könning wrote: > Hello List, > > I hope this is not of topic: > > My private server has been hacked: > debian woody 2.4.18bf2.4 kernel, apache-ssl, samba, squid. > > now my problem: the intruder used a rootkit, i think, cause he deleted > /var/log, symlinked /root/.bash_history > /dev/null, etc. > Is there any way to recover the evidences, e.g. the /var/log/ directory? > (ext2) > > and there three sh processes running as root? Ptrace exploit? > how can i dump this processes to file, to keep this evidence? > > > Thanks for help > > -- > Christian Koenning > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > -- Christiano Anderson <[EMAIL PROTECTED]> http://people.debian-rs.org/~anderson Porto Alegre/RS
Re: HELP, my Debian Server was hacked!
On Tue, Apr 22, 2003 at 09:00:11PM +0200, Christian K?nning wrote: > /var/log, symlinked /root/.bash_history > /dev/null, etc. > Is there any way to recover the evidences, e.g. the /var/log/ directory? > (ext2) Examine your /dev/swap after following advice in other replies about making sure things are RO. You'll want to do a swapoff to preserve the evidence right away. Best if you pull the ether cable and work off a local console while you do any of this. You'd be amazed what you can find in /dev/swap ;-)
Re: HELP, my Debian Server was hacked!
While the earlier advice is probably the best advice, don't forget to run chkrootkit. I recently had the same thing happen to one of my machines. I've found a kit in /dev/proc/fuckit The total nuking of /log makes this look like a very amature job. If they were hot they would edit the appropriate logs and retouch the dates ect leaving less blatant signs. I can't totally rule out a physical hack as it is an office machine, but it it was network I really want to know what in sarge can be so blatently abused. (nightly apt-get update && apt-get upgrade) David. On Tue, 22 Apr 2003, xbud wrote: > tar up your /proc/ directory > to save a copy of your kcore - it should have useful information unless he > managed to zero out all the memory that was being utilized during the break > in. > > turn the box off but make sure it don't delete crap, watch out for logic bombs > or what not. > > remove the disk and mount it on another box -o ro (read only) and do your > analysis there. > > > On Tuesday 22 April 2003 13:00, Christian Könning wrote: > > Hello List, > > > > I hope this is not of topic: > > > > My private server has been hacked: > > debian woody 2.4.18bf2.4 kernel, apache-ssl, samba, squid. > > > > now my problem: the intruder used a rootkit, i think, cause he deleted > > /var/log, symlinked /root/.bash_history > /dev/null, etc. > > Is there any way to recover the evidences, e.g. the /var/log/ directory? > > (ext2) > > > > and there three sh processes running as root? Ptrace exploit? > > how can i dump this processes to file, to keep this evidence? > > > > > > Thanks for help > > -- > -- > Orlando Padilla > http://www.g0thead.com/xbud.asc > -- > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: HELP, my Debian Server was hacked!
tar up your /proc/ directory to save a copy of your kcore - it should have useful information unless he managed to zero out all the memory that was being utilized during the break in. turn the box off but make sure it don't delete crap, watch out for logic bombs or what not. remove the disk and mount it on another box -o ro (read only) and do your analysis there. On Tuesday 22 April 2003 13:00, Christian Könning wrote: > Hello List, > > I hope this is not of topic: > > My private server has been hacked: > debian woody 2.4.18bf2.4 kernel, apache-ssl, samba, squid. > > now my problem: the intruder used a rootkit, i think, cause he deleted > /var/log, symlinked /root/.bash_history > /dev/null, etc. > Is there any way to recover the evidences, e.g. the /var/log/ directory? > (ext2) > > and there three sh processes running as root? Ptrace exploit? > how can i dump this processes to file, to keep this evidence? > > > Thanks for help -- -- Orlando Padilla http://www.g0thead.com/xbud.asc --
Re: HELP, my Debian Server was hacked!
On Tue, Apr 22, 2003 at 09:00:11PM +0200, Christian Könning wrote: > Hello List, > > I hope this is not of topic: > > My private server has been hacked: > debian woody 2.4.18bf2.4 kernel, apache-ssl, samba, squid. Ouch. Was it up-to-date to security patches? > > now my problem: the intruder used a rootkit, i think, cause he deleted > /var/log, symlinked /root/.bash_history > /dev/null, etc. > Is there any way to recover the evidences, e.g. the /var/log/ directory? > (ext2) Use e2undel (but you should mount read-only) > > and there three sh processes running as root? Ptrace exploit? > how can i dump this processes to file, to keep this evidence? > Go to /proc/# (with # being the process number of these) you will find all the information on running processes there (environment, commandline, filedescriptor, the executable...) You probably need a crash course on forensics in UNIX (me too :-), maybe this helps: http://staff.washington.edu/dittrich/talks/blackhat/blackhat/forensics.html and http://www.dpo.uab.edu/~kalyan/incidentchecklist.html Plenty of reading also at http://www.sans.org/rr/incident/, if you are interested. But I believe you want to get over this as fast as possible, consider using 'tct' (The Coroner Toolkit, packaged for Debian) . Hope that helps Javi pgpoBflRucsHl.pgp Description: PGP signature
