Re: [asterisk-users] Strange Issue: asterisk deleted
- Original Message - Hi Thank you for your support. The server is actually compromised, I discovered that after making a deep trace using the audit daemon and looking for the kill signal (SIGKILL) that terminates asterisk. I discovered that there is an executable with a random name in the /boot folder that is killing and deleting asterisk !!! This executable is launched by a service in /etc/rc.d/ with the same random name. When I stopped this service, a new service was created with another different random name and it too is killing and deleting asterisk. This was the evidence i needed to be convinced that the server has a virus and is compromised. The good thing is that this is a fresh install and hence there are no sensitive data or a lot of work done on it so i will reinstall the OS and start over. The bad thing is that I spent more than 4 days trying to understand what was going on. Very interesting. Any ideas on how the system was compromised? Are any other daemons being actively replaced, or just Asterisk? I did hear of a similar issue to the one you describe (also on an Asterisk box) via a third party recently, but don't have any real specifics other than it being Asterisk 1.4.x on Debian (5 or 6), running on a local LAN, no outside access. Curious if there are any commonalities to the two compromised systems. --Tim -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
Did you take a look at /var/log/syslog? Am 26.11.2014 21:08, schrieb Antoine Megalla: Hi, I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system. Again any indicators to solve this very weird issue are welcomed. Regards, Antoine Megalla Sent from my iPhone On Nov 26, 2014, at 6:12 PM, Thorsten Göllner t...@ovm-group.com mailto:t...@ovm-group.com wrote: Am 26.11.2014 11:37, schrieb Antoine Megalla: Hi, I am struggling with a very strange issue I have been facing for the past week; I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources. The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I cleaned the source and re-installed asterisk and again the same thing happened again !!! I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a make then make install and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin. I tried to look a the asterisk log files but I cannot find a single error in them. Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ? I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted. If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful. Hi, you write Also if it was really deleted .. - did you looked at it via ls /usr/sbin/asterisk? You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as normal user? You write the process is killed. Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog? -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
Yes I did, and there is nothing about asterisk in the /var/log folder I am starting to think that the server on compromised. Sent from my iPhone On Nov 27, 2014, at 11:09 AM, Thorsten Göllner t...@ovm-group.com wrote: Did you take a look at /var/log/syslog? Am 26.11.2014 21:08, schrieb Antoine Megalla: Hi, I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system. Again any indicators to solve this very weird issue are welcomed. Regards, Antoine Megalla Sent from my iPhone On Nov 26, 2014, at 6:12 PM, Thorsten Göllner t...@ovm-group.com wrote: Am 26.11.2014 11:37, schrieb Antoine Megalla: Hi, I am struggling with a very strange issue I have been facing for the past week; I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources. The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I cleaned the source and re-installed asterisk and again the same thing happened again !!! I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a make then make install and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin. I tried to look a the asterisk log files but I cannot find a single error in them. Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ? I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted. If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful. Hi, you write Also if it was really deleted .. - did you looked at it via ls /usr/sbin/asterisk? You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as normal user? You write the process is killed. Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog? -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
On Wednesday 26 Nov 2014, Antoine Megalla wrote: Hi, I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system. Again any indicators to solve this very weird issue are welcomed. It sounds as though your server might have been compromised. Get another machine of the same bit architecture and perform a fresh install of exactly the same OS as your Asterisk box on that. Install busybox too (it's usually there anyway, as it's required for building the initial RAMdisks used by most distros for booting). Using a USB stick (preferrably one that can be set read-only), copy at least the `ls`, `ps`, `netstat`, `w`, `lsattr`, `md5sum`, `cat`, `diff` and `busybox` binaries over (to somewhere that isn't /usr/bin/). Use both the existing installed and the newly-copied md5sum and diff to check each system binary against the known-good ones. You can use busybox to replicate commands you haven't copied (but note that busybox versions are rather cut-down as compared to the GNU tools you know and love. Come to think of it, they're cut-down as compared to the BSD tools everyone replaces with GNU versions once they have a C compiler up and running). Compare /etc/inittab between the two machines. Many rootkits mess with ext[2-4]fs attributes, presumably to stop you overwriting their overwritten system binaries; so use a known good lsattr to check the attributes of everything in /bin/, /sbin/, /usr/bin/ and /usr/sbin/ -- watch out for anything set immutable. Getting rid of the compromise fortunately is reasonably easy, especially if your /home folder is on its own partition. Just ignore that partition during reinstallation, edit your /etc/fstab afterwards and reboot -- your original /home will be preserved intact. If not, use systemrescuecd or something similar to boot a known-good system. Use mv to rename /home to a new name. Shrink a disk partition and create a new small partition. Use that for your /home during the reinstall. Then again edit /etc/fstab, unmount /home, mv your old /home back to /home and reboot. -- AJS Note: Originating address only accepts e-mail from list! If replying off- list, change address to asterisk1list at earthshod dot co dot uk . -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
Hi Thank you for your support. The server is actually compromised, I discovered that after making a deep trace using the audit daemon and looking for the kill signal (SIGKILL) that terminates asterisk. I discovered that there is an executable with a random name in the /boot folder that is killing and deleting asterisk !!! This executable is launched by a service in /etc/rc.d/ with the same random name. When I stopped this service, a new service was created with another different random name and it too is killing and deleting asterisk. This was the evidence i needed to be convinced that the server has a virus and is compromised. The good thing is that this is a fresh install and hence there are no sensitive data or a lot of work done on it so i will reinstall the OS and start over. The bad thing is that I spent more than 4 days trying to understand what was going on. Again, thank you for your support. Regards, Antoine Megalla Sent from my iPhone On Nov 27, 2014, at 8:00 PM, asterisk-users-requ...@lists.digium.com wrote: Send asterisk-users mailing list submissions to asterisk-users@lists.digium.com To subscribe or unsubscribe via the World Wide Web, visit http://lists.digium.com/mailman/listinfo/asterisk-users or, via email, send a message with subject or body 'help' to asterisk-users-requ...@lists.digium.com You can reach the person managing the list at asterisk-users-ow...@lists.digium.com When replying, please edit your Subject line so it is more specific than Re: Contents of asterisk-users digest... Today's Topics: 1. Re: Strange Issue: asterisk deleted (Antoine Megalla) 2. Re: High resident memory with 11.14.0 ? (James Lamanna) 3. Re: Strange Issue: asterisk deleted (Chad Wallace) 4. Re: Strange Issue: asterisk deleted (Marie Fischer) 5. Re: SIP call drops after 32 seconds,but only when (Marie Fischer) 6. Re: SIP call drops after 32 seconds,but only when (Amit Patkar) 7. Re: Strange Issue: asterisk deleted (Thorsten G?llner) 8. Re: Strange Issue: asterisk deleted (Antoine Megalla) 9. Re: Strange Issue: asterisk deleted (A J Stiles) -- Message: 1 Date: Wed, 26 Nov 2014 22:08:05 +0200 From: Antoine Megalla aa...@rocketmail.com To: Thorsten G?llner t...@ovm-group.com Cc: Asterisk Users Mailing List - Non-Commercial Discussion asterisk-users@lists.digium.com Subject: Re: [asterisk-users] Strange Issue: asterisk deleted Message-ID: 7d5a57fb-657c-439b-9dcb-2790ae9c9...@rocketmail.com Content-Type: text/plain; charset=utf-8 Hi, I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system. Again any indicators to solve this very weird issue are welcomed. Regards, Antoine Megalla Sent from my iPhone On Nov 26, 2014, at 6:12 PM, Thorsten G?llner t...@ovm-group.com wrote: Am 26.11.2014 11:37, schrieb Antoine Megalla: Hi, I am struggling with a very strange issue I have been facing for the past week; I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources. The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I cleaned the source and re-installed asterisk and again the same thing happened again !!! I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a make then make install and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin. I tried to look a the asterisk log files but I cannot find a single error in them. Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ? I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted. If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful. Hi, you write Also if it was really deleted .. - did you looked at it via ls
Re: [asterisk-users] Strange Issue: asterisk deleted
Question remains, how was it compromised? In the original install ? A fresh install perhaps from another source? Best you determine HOW before spending more time going down another rabbit hole! John Novack Antoine Megalla wrote: Hi Thank you for your support. The server is actually compromised, I discovered that after making a deep trace using the audit daemon and looking for the kill signal (SIGKILL) that terminates asterisk. snipped to please the mailing list -- Dog is my Co-pilot -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
Am 26.11.2014 11:37, schrieb Antoine Megalla: Hi, I am struggling with a very strange issue I have been facing for the past week; I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources. The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I cleaned the source and re-installed asterisk and again the same thing happened again !!! I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a make then make install and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin. I tried to look a the asterisk log files but I cannot find a single error in them. Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ? I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted. If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful. Hi, you write Also if it was really deleted .. - did you looked at it via ls /usr/sbin/asterisk? You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as normal user? You write the process is killed. Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog? Best regards -Thorsten- -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
On Wed, Nov 26, 2014 at 10:37:49AM +, Antoine Megalla wrote: Hi, I am struggling with a very strange issue I have been facing for the past week;I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I cleaned the source and re-installed asterisk and again the same thing happened again !!!I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a make then make install and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin. If you suspect that something is being run from the asterisk process or one of its children, run it under 'strace -f' and look for hints (e.g.: 'unlink') in the generated log. -- Tzafrir Cohen icq#16849755 jabber:tzafrir.co...@xorcom.com +972-50-7952406 mailto:tzafrir.co...@xorcom.com http://www.xorcom.com -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
Hi, I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system. Again any indicators to solve this very weird issue are welcomed. Regards, Antoine Megalla Sent from my iPhone On Nov 26, 2014, at 6:12 PM, Thorsten Göllner t...@ovm-group.com wrote: Am 26.11.2014 11:37, schrieb Antoine Megalla: Hi, I am struggling with a very strange issue I have been facing for the past week; I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources. The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I cleaned the source and re-installed asterisk and again the same thing happened again !!! I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a make then make install and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin. I tried to look a the asterisk log files but I cannot find a single error in them. Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ? I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted. If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful. Hi, you write Also if it was really deleted .. - did you looked at it via ls /usr/sbin/asterisk? You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as normal user? You write the process is killed. Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog? Best regards -Thorsten- -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
On Wed, 26 Nov 2014 22:08:05 +0200 Antoine Megalla aa...@rocketmail.com wrote: I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system. I don't know if there's any way to see where the signal comes from. But I think it would have to be another process. Is this a hosted machine? Could it be that your hosting provider doesn't allow asterisk? This would be a good way to enforce that rule. Otherwise, it could be a root kit or a virus. Or it could be that you (or someone else) wanted to make sure asterisk wasn't running at some point and left while true; do killall -9 asterisk; done running in a shell, and forgot about it. You can list all the processes with the command ps -ef And to see if anyone else (or yourself) is logged in, run w. That will show every individual session and where they're connected from. -- C. Chad Wallace, B.Sc. The Lodging Company http://www.lodgingcompany.com/ OpenPGP Public Key ID: 0x262208A0 -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Strange Issue: asterisk deleted
On 26.11.2014, at 22:08, Antoine Megalla aa...@rocketmail.com wrote: The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the Ready line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there. I know that the process is killed because when I start asterisk using the command asterisk -c it starts and then it exits and the word killed is wrote on the console. Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too. Interesting problem, I'm quite curious what the cause is. Are you 100% sure that the asterisk your are running is in /usr/sbin? Try 'which asterisk' to see what your shell is running and/or start asterisk with a full path as /usr/sbin/asterisk -c. You could also try renaming the binary to find out if indeed something kills Asterisk by name. There's a tool called SystemTap which could give you information which process sent the SIGKILL: https://sourceware.org/systemtap/ http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigterm-sigkill-issue/ -- marie -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
