Re: Need urgent help regarding security
Mark Jayson Alvarez [EMAIL PROTECTED] writes: Suggestions are welcome... very much welcome. I just need to collate everything. Start with security(7). In future, keep up with Security Advisories. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
Mark Jayson Alvarez wrote: Now we have a couple of inputs, we just have to figure out which is the proper combination. Here they are: 1. Use private key for ssh logins (should bring the private key always... and if it is stolen.) Private keys can (and should) be passphrase protected. Then the key itself is worthless without the passphrase and it (usually) takes social engineering to get that. There is plenty of security info out there about writing security policies and you cannot forget social engineering. For keys you can't passphrase protect (used for cron jobs for example) the keys can be limited to perform only specified actions. There are plenty of manual pages and HowTo's out there. Don't allow root logins. Limit root users. Enforce good password practices. Investigate sudo, perhaps. 3. Constantly upgrade third party softwares (ssh, ssl, apache, bind) etc.. (too much work.. there are so many of them(postgres, proftp, mysql, php) must be member of various security mailing lists and discussions). If this is too much work then maybe you are in the wrong business. Keeping software up-to-date against security patches is priority number one for any responsible system administrator irrespective of what OS they run. Reading bugtraq takes me maybe 20 minutes a day, and that's only because I choose to read messages that are interesting, even if irrelevant. Portaudit can be run over night and email you output (and does that out-of-the box on 5.4, probably earlier too). Time to check email from even a dozen servers is small. If they are all the same, then you only really *need* to read one message. Also decide if you really *need* all these services. Proftpd pops up as one that, in some circumstances, is easily got rid of and replaced with ssh/sftp -- not always possible, but it's one less difficult-to-configure package to worry about. Is proftpd actually buying you anything over regular ftpd? 4. Constant Os upgrade(or should we shift to OpenBSD like one of our boss recommended(need to familiarize first, it is a *nix no problem... but it is still OpenBSD :)Also, was it really the 4.8 that has been hacked or the old version of BIND running on it? Anyway, its 6.0 now, guess we really have to upgrade now. 5.4 is still supported (as is 4.11 I believe, but I can no longer find the relevant pages on the revamped website). If 6.0 works, then it's a good time to choose it. What OS you run is simply irrelevant if you don't keep up-to-date with security fixes. If you keep up-to-date with security fixes you can run a version as long as it is supported. I am not aware that there are any outstanding security issue in any of 4.11, 5.4 or 6.0. For a production server, an OS version upgrade should not be taken lightly. No project can test a new release against every combination of h/w and s/w and most especially they cannot test it against *your* h/w and s/w. If you really care about stability then you can pick a server, upgrade just it and burn it in. Once it proves stable you can upgrade others like it. You can also plan for OS upgrade at install time. These days, I always leave minimally sized spare partitions specifically for installing a new (especially major) version e.g. going from 5.X to 6.X. If you don't leave that space at install time, you'll never get it once a server is running without adding a new disk. Minor version upgrades usually go just fine using simple src upgrade, but for production systems you should still do one and burn in before committing to doing them all. But what OS you run (FreeBSD 4/5/6, OpenBSD) is pretty much irrelevant if you can't be bothered keeping your software up-to-date with respect to security issues and have your servers and security practices nailed down to start with. OpenBSD will fall just as fast as FreeBSD if you leave an insecure sshd running on it, or give a root password away. Given that your root password was apparently found on the servers, you likely had much bigger problems than any switch of OS or version would solve. Was your root password a simple word (i.e. did a password cracker get it)? Or did you log in with telnet as root so a network monitor caught it? 11. Use ip forwarding so that public servers will never again face the Internet directly( does this require a supers strong machine that will act as firewall? or perhaps an appliance(brand new) can we acquire this right away? It's not clear to me how you think this would actually help. If all your machines are internet-facing (and from your ip forwarding comment, it seems that they are) what good will forcing all the packets through one machine do? If you have a buggy service on a hidden machine, but you just forward packets to it from your firewall, what difference has the firewall made? Maybe I misunderstand. If you are trying to hide mostly internal hosts and forward only a limited
Re: Need urgent help regarding security
Good day again!! This has something to do with my previous email about finding an IRC bouncer installed into one of our freebsd servers(4.9). Someone suggested here to run a rootkit finder... I installed an rkhunter and eventually found an ascii text file inside the /dev/ named saux and to my surprise, it contains all of our username and passwords we used to login to other servers from that machine. Afterwards, we didn't even run the same root kit finder into other machines and just looked for that file(saux) and walla!! all machines have one!! We immediately killed all remote administration daemons and allow only root console access. Now we have a lot of work to do. more than 10 servers have been compromised founded the same file(saux) containing our passwords. Critical servers such as dns, proxy, mail etc. Even two of our cisco routers are 80% possibly compromised as well.. The question is: Now what?? I guess we will be spending 7 days of work starting from this day till we have a properly created policies, not just for user accounts... but I guess for everything, as in everything. And it wouldn't be only for a short period of time...I'm sure though. The bigger question is: Where should we start? Investigate how the cracker got into the system? Why? perhaps we should bring back the server first into their functional state because hundreds of thousands of people are relying to them?? Or should we tell our Director first, in case he might wonder why he is not receiving his emails on Monday morning or cannot telnet into the cisco router? Now we have a couple of inputs, we just have to figure out which is the proper combination. Here they are: 1. Use private key for ssh logins (should bring the private key always... and if it is stolen.) 2. Use kerberos for ssh logins? useful for cisco telnet authentication too. Should we replace the existing radius for the routers? Do we have enough time? can we afford to run a compromised server while setting up these servers? 3. Constantly upgrade third party softwares (ssh, ssl, apache, bind) etc.. (too much work.. there are so many of them(postgres, proftp, mysql, php) must be member of various security mailing lists and discussions). 4. Constant Os upgrade(or should we shift to OpenBSD like one of our boss recommended(need to familiarize first, it is a *nix no problem... but it is still OpenBSD :)Also, was it really the 4.8 that has been hacked or the old version of BIND running on it? Anyway, its 6.0 now, guess we really have to upgrade now. 5. Use nmap versioning etc. constantly check for unknown services (must audit all of the services running on every machine) 6. Always compile into a jail environment 7. Create a standard firewall ruleset template, (if it is a web server... uncomment this etc.) 8. use a livecd... (use for binary trojaning) 9. remote sysloging (I thought -ss flag is recommended?) 10. Implement kernel secure level chflags(undeletable, firewall unchangeable) 11. Use ip forwarding so that public servers will never again face the Internet directly( does this require a supers strong machine that will act as firewall? or perhaps an appliance(brand new) can we acquire this right away? What else?? Do you have anymore idea? Right now I am about to reformat one of our proxy server and install 6.0 on it. Perhaps I should check the squid config throughly... Suggestions are welcome... very much welcome. I just need to collate everything. - Yahoo! FareChase - Search multiple travel sites in one click. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
On Wed, Nov 16, 2005 at 09:51:08PM -0500, Steve Bertrand wrote:
Most *((cr/h)ackers* (and I use that term VERY loosely (aka:
script kiddies)) are interested in rooting a box, and setting up a
storage/sharing area that is free to them. This may not be
the case,
but it's better to 'observe' your foreign presence first.
I understand the rationale behind this advice, but I
disagree. I made my suggestion plain in another part of this
thread, but (in
general) the first priority should be to disrupt the attack.
For some organizations (universities, especially), computing
resources are our number one asset. We have oodles of cycles
and network bandwidth -- a rooted box directly targets our
valuables, even if it's only doing IRC or warez.
I do agree with you. When it happened to me, generally the whole process
of finding out where the origination of the attack (at least the network
it was launched from), what they had done on the box, how they intruded
in the first place etc was 15 minutes. I understand that in a critical
environment where important data can be compromised it has to be taken
offline as quickly as possible.
Moreover, the longer the hole remains open, the greater the
chance that the attacker will extend the breach. In most
every scenario I can imagine, this is unacceptable. Real
forensic investigation can't really even be performed until
the box is offline; looking at /tmp and other likely trouble
spots is excellent advice, but should come later in the process.
Agreed again. However in at least 3 cases I've dealt with, they were
pretty much the same other than some minor differences. I've always had
backups too. However there is always that fear that they could have
infiltrated other boxen on the network, which if you just 'broke' one
aspect of their intrusion suddenly, may provoke them to do something
nastier then they originally intended.
I guess it's a lose-lose situation any way you look at it.
For now, take a snapshot of the network activity (using lsof,
ngrep, tcpdump, etc); I recommended lsof because it will
reveal all open files and network sockets very quickly. Dump
the output to a file and unplug the machine. tcpdump and
friends will work well, too, and give you a more indepth look
at the network activity, but will also require you to keep
the box up for longer than I'd be comfortable.
OP has some asset that is being threatened or diminished by
this attack, be it his bandwith, CPU cycles, host/network
integrity or self confidence. He needs to identify that asset
and work quickly to protect it. In most cases, this will mean
immediately removing the box and preparing to rebuild the
machine; if he's interested in investigating, he can do that
on an image of the disk (since investigations are of little
use if they ruin the evidence).
Allowing the attack to proceed may be moderately
enlightening, but (from the OP's message) it seems like the
basic problem is known.
Crufty machines attract attacks.
--
o--{ Will Maier }--o
| jabber:[EMAIL PROTECTED] | email:[EMAIL PROTECTED] |
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
*--[ BSD Unix: Live Free or Die ]--*
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to
[EMAIL PROTECTED]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
[...] You can easily rebuild a new kernel with: options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT_1000 Then create a script blocking ALL ports exept those what you need. Especially only allowing SSH access to the box from limited IP's. If you need help, just ask. Thanks for the suggestion. I personally have no experience with IPFW (I have played with IPF a little bit on a test box here) so I will have to think on that a little. I am guessing you suggest IPFW as opposed to IPF correct? I read up on IPFW and IPF in the handbook when I was experimenting with firewalls and the rule syntax and things seemed more logical to me with IPF, but I did not look that far in depth. I only recommend IPFW because that is what I am familiar with. I don't want to start a flame war, as I've been told by others that IPF is just as good. If you are experienced with IPF and understand the syntax of it's rules, by all means, go for it. My servers are also remote so I would have to make sure I didn't firewall myself out when enabling any firewall. ;) Yes, that is always a concern. I've been there/done that before on more than one occasion. There are scripts that can 'reset' to a previous config if this does happen though (I learned the hard way ;) Have you checked your daily cron outputs lately? What do they say? All I see is legit cronjobs from a billing system that I run and some from cPanel such as cpumonitor and backups. Sorry, I meant the security run outputs that get sent at around 0300 every day. Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
--On Wednesday, November 16, 2005 20:29:55 -0500 Steve Bertrand [EMAIL PROTECTED] wrote: I think we have a serious problem. One of our old server running FreeBSD 4.9 have been compromised and is now connected to an ircd server.. 195.204.1.132.6667 ESTABLISHED Ran into this recently. Please post the entire output from: # top # w # last # ps -aux # uname -a Just keep in mind that any or all of these could be hacked versions designed to hide everything the attacker is doing. Once a box has been hacked, you can no longer trust any of the binaries unless you can verify their integrity with MD5 sums from the same binaries on a known good box. Paul Schmehl ([EMAIL PROTECTED]) Adjunct Information Security Officer University of Texas at Dallas AVIEN Founding Member http://www.utdallas.edu/ir/security/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
I think we have a serious problem. One of our old server running FreeBSD 4.9 have been compromised and is now connected to an ircd server.. 195.204.1.132.6667 ESTABLISHED Ran into this recently. Please post the entire output from: # top # w # last # ps -aux # uname -a ...after that, depending on the intruders knowledge and depending on what/if they are covering up, we can probably tell what is going on via further troubleshooting. The output from: # ls -la /tmp would probably help too. However, we still haven't brought the server down in an attempt to track the intruder down. Right now we are clueless as to what we need to do.. Most of our servers are running legacy operating systems(old versions mostly freebsd) Also, that particular server is running - ProFTPD Version 1.2.4 which someone have suggested to have a known vulnerability.. I really need all the help I can get as the administration of those servers where just transferred to us by former admins. The server is used for ftp. First...just relax. Do not panic. Just let them do what they are going to do (with hopes you have backups), and the problem can be found and eradicated. Now, answer these: - do you have an external firewall in front of this box - do you have a firewall running on this box - is this box Internet facing - is this machines ONLY purpose FTP Another thing...what is the IP of the box. I can quickly nmap it, give you instructions on how to config IPFW firewall into the mix, tell you what ports are listening/responding and send you a ruleset to block all ports in/out to/from that IP. Don't be concerned about finding out who did what at this point...again, relax. Running IRC usually doesn't appear they are malicious. THey are likely just trying to use your bandwidth/resources. Provide the above, and something can be done. Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
Mark Jayson Alvarez wrote: Good Day! I think we have a serious problem. One of our old server running FreeBSD 4.9 have been compromised and is now connected to an ircd server.. 195.204.1.132.6667 ESTABLISHED I believe I'm having the same issue as you, except on FreeBSD 5.4-RELEASE. I notice a connection to the same IP and port as you posted (which by the way is an Undernet IRC server). I also see a psyBNC server listening on port 7978: server# sockstat -l4 | grep psybnc USER COMMANDPID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS wicked6 psybnc 15819 3 tcp4 *:7978*:* Funny thing is there is no process by wicked6 (or by anyone currently) called psybnc. I can connect to an IP on that server on port 7978 and get a psyBNC though. I've checked for other processes by wicked6, nothing. It's trying to make a connection on 6667 to that IP as I said: server1# netstat -n | grep 6667 tcp4 0 0 xx.xx.xx.xx.64243195.197.175.21.6667SYN_SENT top lists nothing using up much CPU. /tmp doesn't show much except many session files. I found a psybnc.tar.gz file in a user's home directory but cannot find any directories with psybnc config files or binaries. Port 6667 is blocked by my datacenter so this is not actually doing any damage against the target, but I wanted to post here and let you know I'm having the same problem on a different version of FBSD with everything up to date. To Steve: I don't want to post the full outputs of those since this is a client server, but I will say the following points: - top lists nothing significant. 97% idle CPU - w only shows myself and one other legit user logged in who is editing config files with vi - last shows nothing but myself and that one other user - ps -aux doesn't say anything about psyBNC or bnc. everything looks normal as of now - It's a FreeBSD 5.4-RELEASE machine with a generic kernel except with quota support -Mark -- GnuPG Public Key: http://www.mkproductions.org/mk_pubkey.asc Internet Radio: Party107 (Trance/Electronic) - http://www.party107.com Rock 101.9 The Edge (Rock) - http://www.rock1019.net IRC: MIXXnet IRC Network - irc.mixxnet.net (Nick: MIXX941) signature.asc Description: OpenPGP digital signature
RE: Need urgent help regarding security
# ls -la /tmp also /var/tmp Indeed, many people would install with a /var partition, which would put /tmp under /var via symlink, but a good point. if you run awstats or phpBB - upgrade... Agreed, but even phpBB may not be the fault. Many problems with PHP come with the binary, not necissarily the app that uses it. However...like I said before...it's best not to panic, and what you DON'T want, is for the invader to know you are looking. It's best (IMHO), to walk around him/her, until you find their access point and intention, then go from there. Most *((cr/h)ackers* (and I use that term VERY loosely (aka: script kiddies)) are interested in rooting a box, and setting up a storage/sharing area that is free to them. This may not be the case, but it's better to 'observe' your foreign presence first. If it is a real blackhat, you don't want to go pissing all over his work before you have evidence, lest he pisses back on you...as he will. Otherwise, if it's a kiddie, there are simple ways to deal with that, and learn from your vulnerabilities...always with the expectation that the next hack will be from someone who didn't just download a vulnerability from the 'net, and come across you with a point-and-click-type scanner in a GUI interface. Only my .02 Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
also /var/tmp Indeed, many people would install with a /var partition, which would put /tmp under /var via symlink, but a good point. My mistake...symlink was the wrong word to use here, for those who create a /var partition without physically making a /tmp partition. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
- top lists nothing significant. 97% idle CPU Irrelavent, the process is probably idle right now. - w only shows myself and one other legit user logged in who is editing config files with vi Perhaps they aren't currently logged in. - last shows nothing but myself and that one other user What is the last entry that last shows (no pun intended)...ie: what is the date? - ps -aux doesn't say anything about psyBNC or bnc. everything looks normal as of now Ok, here's what to do: # pkg_add -r nmap # rehash # nmap -sS -P0 my.ip.server.com ...then (probably futile): # nmap -sU -P0 my.ip.server.com which will tell you if you are listening on ports you *shouldn't* have open. - It's a FreeBSD 5.4-RELEASE machine with a generic kernel except with quota support You still didn't answer the FTP question. What services should be running on it? You can easily rebuild a new kernel with: options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT_1000 Then create a script blocking ALL ports exept those what you need. Especially only allowing SSH access to the box from limited IP's. If you need help, just ask. This sounds like a brute-forced password hack via remote access, or overflow via a vulnerable software that should not be Internet facing. Don't give me your IP if you don't want, just tell us (or me personally) what should be Internet facing (as far as services), and get you fixed up. Have you checked your daily cron outputs lately? What do they say? nmap is your friend, and so is IPFW. Figure out exactly what you need to face the Internet, and staple the rest closed. Steve -Mark -- GnuPG Public Key: http://www.mkproductions.org/mk_pubkey.asc Internet Radio: Party107 (Trance/Electronic) - http://www.party107.com Rock 101.9 The Edge (Rock) - http://www.rock1019.net IRC: MIXXnet IRC Network - irc.mixxnet.net (Nick: MIXX941) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
First, I want to thank you all for replying. For now what I just did is to just pulled the utp cable from its ethernet port. Now, no one can access it. However I tried once to put it back and then the ircd connection went up silently. It is confirmed that we are running psybnc like what someone who replied has experienced too, installed in a folder(pnybnc) inside etc that is named with a special character... hard to get inside, but we've managed to read some files using find and grep... The chat logs are still there.. seems like it has been turned into a sex chatroom.. also the config of psybnc which contains the username/password the intruder used in connecting... Now what I want to do is to just reinstall the whole operating system and secure it as possible as I can. Like someone told, its just a waste to try to track it down because the intruder might be located somewhere on the other side of the world. To others who replied... I will just answer you all one by one... Thanks again. - Yahoo! FareChase - Search multiple travel sites in one click. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
Now what I want to do is to just reinstall the whole operating system and secure it as possible as I can. Like someone told, its just a waste to try to track it down because the intruder might be located somewhere on the other side of the world. They are always on the other side of the world...this is the Internet. If that is your solution, I would recommend reconfiguring your FTP servers DNS entries, and applying another IP to the box,lest you be affected again. However, that won't even fix it, becuase it will just be found again by someone else. Unplugging the box just informs the attacker that you are aware of them. Moving the IP just makes people re-locate you. The solution is make the box accessible to only those who need it...and only the services they need. .02 Steve To others who replied... I will just answer you all one by one... Thanks again. Yahoo! FareChase - Search multiple travel sites in one click. http://us.lrd.yahoo.com/_ylc=X3oDMTFqODRtdXQ4BF9TAzMyOTc1MDIE X3MDOTY2ODgxNjkEcG9zAzEEc2VjA21haWwtZm9vdGVyBHNsawNmYw--/SIG=110oav78o/* *http% 3a//farechase.yahoo.com/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
On 11/16/05, Mark Kane [EMAIL PROTECTED] wrote: I also see a psyBNC server listening on port 7978: server# sockstat -l4 | grep psybnc USER COMMANDPID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS wicked6 psybnc 15819 3 tcp4 *:7978*:* Funny thing is there is no process by wicked6 (or by anyone currently) called psybnc. I can connect to an IP on that server on port 7978 and get a psyBNC though. I've checked for other processes by wicked6, nothing. It's very common for them to overwrite argv[0], or use setproctitle stuff to hide the real name of the program. Some programs don't read that -- sockstat and top are two that don't read the modified program name. It's trying to make a connection on 6667 to that IP as I said: server1# netstat -n | grep 6667 tcp4 0 0 xx.xx.xx.xx.64243195.197.175.21.6667SYN_SENT netstat -aAn (specifically, the -A) instructs netstat to prepend each line with the memory address of the network connection. If you run that you'll see something like: f0d710c0 tcp4 0 0 xxx.xxx.xxx.xxx.29 211.119.136.240.66 ESTABLISHED (sometimes, the port numbers get truncated, so you may have to grep for the destination IP instead of the port number.) You can take that address and run fstat | grep address: $ fstat | grep f0d710c0 www iroffer191333* internet stream tcp f0d710c0 In this specific case, it's an iroffer program run from some PHP backdoor someone installed on the server (see http://malformed.org/2005/11/15/zend-encoder-bad-for-the-internet/ for a description of the present/near-future of these PHP backdoors). In your case it may be that you're running suexec or suPHP, or it may not have been started from the web at all. If that's the case, you may be able to find out what else is going on by ensuring /proc is mounted and then run: ps -uxwwep pid: ps -uxwwep 19133 USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND www 19133 0.0 0.0 1244 424 ?? S22Oct05 12:52.03 ... DOC_ROOT=/usr/home/user/websites/domain.com ... You may also see SCRIPT_FILENAME or PWD or other environment variables that may give you hints as to where this was started from. There are some other programs that'll do all this for you, I think 'lsof' is one. I dunno. I prefer to use base system utilities. But to each their own. Of course, if the listening process isn't showing up at all, but you can still connect to the port, then you may have some sort of hacked kld loaded or hacked ps, in which case the attacker has root, which is a far more serious situation. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Need urgent help regarding security
Steve Bertrand [EMAIL PROTECTED] wrote: Now what I want to do is to just reinstall the whole operating system and secure it as possible as I can. Like someone told, its just a waste to try to track it down because the intruder might be located somewhere on the other side of the world. They are always on the other side of the world...this is the Internet. If that is your solution, I would recommend reconfiguring your FTP servers DNS entries, and applying another IP to the box,lest you be affected again. However, that won't even fix it, becuase it will just be found again by someone else. Unplugging the box just informs the attacker that you are aware of them. Moving the IP just makes people re-locate you. The solution is make the box accessible to only those who need it...and only the services they need. .02 Steve No, that is not the solution I'm thinking of.. You see right now, that machine contains at least 200 Gb of important files... I'm just paranoid that the intruder might just launch an rm -rf. Right now we don't have a backup of those files yet. I'm really eager to know how the intruder got into our machine, I'm just afraid that he might be reading everything I am typing in the terminal. I am also dissapointed because most of our server configuration files are in my home directory but doing the ls /tmp I found those files. Those files are our proxy configurations containing all of our peer proxies (ipaddress) and also the squid.conf which I'm afraid that the intruder can use to launch an attack to our proxy farm. You see those proxies aren't in a much secure mode yet but they are the MOST critical service in our company because all of our partners are passing through that proxies. Now what I really wan't to do is to just do the right thing but only one by one. I got so many replies, someone even suggested finding out the irc channel and try to have a little chat with the intruders. Someone suggested putting up a firewall before it and try to dump the packets to retrieve relevant informations. I'm really so confused right now as to where to start Right now, the server is currently inaccessible from the network, but it is still running( I just remembered someone suggested not shutting it down because the script the intruder used might get automatically erased). From there... where should I start.? Thank you very much. - Yahoo! FareChase - Search multiple travel sites in one click. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
On Wed, Nov 16, 2005 at 09:51:08PM -0500, Steve Bertrand wrote:
Most *((cr/h)ackers* (and I use that term VERY loosely (aka:
script kiddies)) are interested in rooting a box, and setting up a
storage/sharing area that is free to them. This may not be the
case, but it's better to 'observe' your foreign presence first.
I understand the rationale behind this advice, but I disagree. I
made my suggestion plain in another part of this thread, but (in
general) the first priority should be to disrupt the attack. For
some organizations (universities, especially), computing resources
are our number one asset. We have oodles of cycles and network
bandwidth -- a rooted box directly targets our valuables, even if
it's only doing IRC or warez.
Moreover, the longer the hole remains open, the greater the chance
that the attacker will extend the breach. In most every scenario I
can imagine, this is unacceptable. Real forensic investigation can't
really even be performed until the box is offline; looking at /tmp
and other likely trouble spots is excellent advice, but should come
later in the process.
For now, take a snapshot of the network activity (using lsof, ngrep,
tcpdump, etc); I recommended lsof because it will reveal all open
files and network sockets very quickly. Dump the output to a file
and unplug the machine. tcpdump and friends will work well, too, and
give you a more indepth look at the network activity, but will also
require you to keep the box up for longer than I'd be comfortable.
OP has some asset that is being threatened or diminished by this
attack, be it his bandwith, CPU cycles, host/network integrity or
self confidence. He needs to identify that asset and work quickly to
protect it. In most cases, this will mean immediately removing the
box and preparing to rebuild the machine; if he's interested in
investigating, he can do that on an image of the disk (since
investigations are of little use if they ruin the evidence).
Allowing the attack to proceed may be moderately enlightening, but
(from the OP's message) it seems like the basic problem is known.
Crufty machines attract attacks.
--
o--{ Will Maier }--o
| jabber:[EMAIL PROTECTED] | email:[EMAIL PROTECTED] |
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
*--[ BSD Unix: Live Free or Die ]--*
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
On Wed, Nov 16, 2005 at 05:16:37PM -0800, Mark Jayson Alvarez wrote:
Good Day!
At first I thought I was confused, but then I realized that you had
cross-posted your message to freebsd-security@ and
[EMAIL PROTECTED] Please don't do this, as it fragments the
discussion.
Good luck.
--
o--{ Will Maier }--o
| jabber:[EMAIL PROTECTED] | email:[EMAIL PROTECTED] |
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
*--[ BSD Unix: Live Free or Die ]--*
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
Steve Bertrand wrote: - top lists nothing significant. 97% idle CPU Irrelavent, the process is probably idle right now. I understand, but I was trying to give you the results of the commands that you asked Mark Alvarez to run. - w only shows myself and one other legit user logged in who is editing config files with vi Perhaps they aren't currently logged in. It doesn't look like someone got SSH access, it looks more to me like it's a vulnerable PHP script or something. Not sure, but that would be my guess. - last shows nothing but myself and that one other user What is the last entry that last shows (no pun intended)...ie: what is the date? The dates on last range from Nov 1st to today. All but 2 are from my IP logging in, and the other are users who just edit config files and untar files on the server (I've verified that it's their real legit IP's) - ps -aux doesn't say anything about psyBNC or bnc. everything looks normal as of now Ok, here's what to do: # pkg_add -r nmap # rehash # nmap -sS -P0 my.ip.server.com ...then (probably futile): # nmap -sU -P0 my.ip.server.com which will tell you if you are listening on ports you *shouldn't* have open. I will email you off the list with that info. - It's a FreeBSD 5.4-RELEASE machine with a generic kernel except with quota support You still didn't answer the FTP question. What services should be running on it? Well I am a different Mark than originally posted. I just saw this on the list and found a connection attempt through netstat to the same IP and port as the original Mark that posted. I, unlike Mark Alvarez run more than just an FTP server. I will email you with those services. You can easily rebuild a new kernel with: options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT_1000 Then create a script blocking ALL ports exept those what you need. Especially only allowing SSH access to the box from limited IP's. If you need help, just ask. Thanks for the suggestion. I personally have no experience with IPFW (I have played with IPF a little bit on a test box here) so I will have to think on that a little. I am guessing you suggest IPFW as opposed to IPF correct? I read up on IPFW and IPF in the handbook when I was experimenting with firewalls and the rule syntax and things seemed more logical to me with IPF, but I did not look that far in depth. My servers are also remote so I would have to make sure I didn't firewall myself out when enabling any firewall. ;) This sounds like a brute-forced password hack via remote access, or overflow via a vulnerable software that should not be Internet facing. Don't give me your IP if you don't want, just tell us (or me personally) what should be Internet facing (as far as services), and get you fixed up. I will email you the services that need to be open. Have you checked your daily cron outputs lately? What do they say? All I see is legit cronjobs from a billing system that I run and some from cPanel such as cpumonitor and backups. nmap is your friend, and so is IPFW. Figure out exactly what you need to face the Internet, and staple the rest closed. Steve Thanks again for your help. -Mark Kane -- GnuPG Public Key: http://www.mkproductions.org/mk_pubkey.asc Internet Radio: Party107 (Trance/Electronic) - http://www.party107.com Rock 101.9 The Edge (Rock) - http://www.rock1019.net IRC: MIXXnet IRC Network - irc.mixxnet.net (Nick: MIXX941) signature.asc Description: OpenPGP digital signature
Re: Need urgent help regarding security
On Nov 16, 2005, at 9:38 PM, Will Maier wrote: OP has some asset that is being threatened or diminished by this attack, be it his bandwith, CPU cycles, host/network integrity or self confidence. He needs to identify that asset and work quickly to protect it. In most cases, this will mean immediately removing the box and preparing to rebuild the machine; One thing I have done to minimize the threat of crackers is to have my machines built thus: I install FreeBSD and activate only SSH and only SSH with certificates -- no passwords allowed. I then build a master jail hierarchy but I don't actually run a jail in it. I create file backed md devices for most jails to be their root filesystems. Some jails I don't do this with but most of them I do. I then create one or more jails that use nullfs to READ ONLY mount specific parts of the master hierarchy into the jail. namely /bin /lib /libexex /sbin /usr For example: # df -h | grep myjail /dev/md1410290M 108M171M39%/local/jails/myjail /local/jails/master/bin 66G 28G 33G46%/local/jails/myjail/bin /local/jails/master/lib 66G 28G 33G46%/local/jails/myjail/lib /local/jails/master/libexec 66G 28G 33G46%/local/jails/myjail/libexec /local/jails/master/sbin66G 28G 33G46%/local/jails/myjail/sbin /local/jails/master/usr 66G 28G 33G46%/local/jails/myjail/usr procfs 4.0K 4.0K 0B 100%/local/jails/myjail/proc devfs 1.0K 1.0K 0B 100%/local/jails/myjail/dev /etc /var are native to each jail in their own filesystem and /usr/local is set up so that the master has a symlink that resolves to something inside of each separate jails local filesystem so that they can have a RW /usr/local. Any and all other services run inside of one or more jails. You can set up other md devices if you want separate log / tmp / whatever partitions inside your jail... Unless there is a breach in SSH, it is highly unlikely that the root machine itself will be cracked as the ONLY port open is the SSH one and it is restricted to certificate logins only. You can use your firewalls to only allow logins to SSH from certain IP ranges etc if you have that luxury. If someone cracks one of the jails, it is harder for him to screw up the jail system since most of the important system executables are actually mounted read only and they cannot replace system binaries, for example. And if a jail does become compromised, it is much easier to rebuild a jail inside of a good machine than to rebuild your whole native machine... And if one jail gets compromised, I can easily shut it down, rename the root file that is used for the md device (ie, every jail uses an image for it fs) and save it for forensic study later. best Chad --- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
Marco Wertejuk [EMAIL PROTECTED] wrote: try sockstat | grep 6667 to see which process is connecting to irc and try to see what this process is doing with lsof, but depending on what backdoor or rootkit is used, it's possible to see nothing because intelligent rootkits hide themself Ok done this... and I found something First the output of nestat: 10.10.8.140.2994 195.204.1.132.6667 SYN_SENT 10.10.8.140.2993 195.204.1.132.6667 SYN_SENT Then sockstat root adjkernt 4926 445 tcp4 10.10.8.140:2994 195.204.1.132:6667 So.. is it the adjkernt that has been replaced? What should I do with it? P.S. I just plugged this server into our private network in order to access it from my workstation. - Yahoo! FareChase - Search multiple travel sites in one click. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need urgent help regarding security
David Kirchner wrote: On 11/16/05, Mark Kane [EMAIL PROTECTED] wrote: I also see a psyBNC server listening on port 7978: server# sockstat -l4 | grep psybnc USER COMMANDPID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS wicked6 psybnc 15819 3 tcp4 *:7978*:* Funny thing is there is no process by wicked6 (or by anyone currently) called psybnc. I can connect to an IP on that server on port 7978 and get a psyBNC though. I've checked for other processes by wicked6, nothing. It's very common for them to overwrite argv[0], or use setproctitle stuff to hide the real name of the program. Some programs don't read that -- sockstat and top are two that don't read the modified program name. It's trying to make a connection on 6667 to that IP as I said: server1# netstat -n | grep 6667 tcp4 0 0 xx.xx.xx.xx.64243195.197.175.21.6667SYN_SENT netstat -aAn (specifically, the -A) instructs netstat to prepend each line with the memory address of the network connection. If you run that you'll see something like: f0d710c0 tcp4 0 0 xxx.xxx.xxx.xxx.29 211.119.136.240.66 ESTABLISHED (sometimes, the port numbers get truncated, so you may have to grep for the destination IP instead of the port number.) You can take that address and run fstat | grep address: $ fstat | grep f0d710c0 www iroffer191333* internet stream tcp f0d710c0 In this specific case, it's an iroffer program run from some PHP backdoor someone installed on the server (see http://malformed.org/2005/11/15/zend-encoder-bad-for-the-internet/ for a description of the present/near-future of these PHP backdoors). In your case it may be that you're running suexec or suPHP, or it may not have been started from the web at all. If that's the case, you may be able to find out what else is going on by ensuring /proc is mounted and then run: ps -uxwwep pid: ps -uxwwep 19133 USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND www 19133 0.0 0.0 1244 424 ?? S22Oct05 12:52.03 ... DOC_ROOT=/usr/home/user/websites/domain.com ... You may also see SCRIPT_FILENAME or PWD or other environment variables that may give you hints as to where this was started from. There are some other programs that'll do all this for you, I think 'lsof' is one. I dunno. I prefer to use base system utilities. But to each their own. Of course, if the listening process isn't showing up at all, but you can still connect to the port, then you may have some sort of hacked kld loaded or hacked ps, in which case the attacker has root, which is a far more serious situation. Okay well I looked around some more now and found it. It was in /var/tmp/.packlist.0928456/ and it was showing up as [psybnc] (wasn't there before). A kill -9 got rid of it. I'm now grepping to try to find out what may have created that or launched it. Thanks -Mark -- GnuPG Public Key: http://www.mkproductions.org/mk_pubkey.asc Internet Radio: Party107 (Trance/Electronic) - http://www.party107.com Rock 101.9 The Edge (Rock) - http://www.rock1019.net IRC: MIXXnet IRC Network - irc.mixxnet.net (Nick: MIXX941) signature.asc Description: OpenPGP digital signature
