I've been playing with it for a while and can't figure it out. You should probably post an issue to github (https://github.com/ossec/ossec-hids). The devs may pay attention to it there.
On Mon, Dec 15, 2014 at 2:12 PM, <mje...@gmail.com> wrote: > I decided to give gdb a try again. I can get it to segfault pretty > consistently. Here is my output. > > [root@ossectst ossec]# ps -ef | grep ossec > root 5018 4903 0 12:39 pts/2 00:00:00 tail -f logs/ossec.log > root 5675 2407 0 12:50 pts/1 00:00:00 grep ossec > [root@ossectst ossec]# service ossec start > Starting OSSEC: [ OK ] > [root@ossectst ossec]# ps -ef | grep ossec > root 5018 4903 0 12:39 pts/2 00:00:00 tail -f logs/ossec.log > ossecm 5711 1 0 12:50 ? 00:00:00 /var/ossec/bin/ossec-maild > root 5715 1 0 12:50 ? 00:00:00 /var/ossec/bin/ossec-execd > ossec 5719 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-analysisd > root 5723 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-logcollector > ossecr 5728 1 0 12:50 ? 00:00:00 /var/ossec/bin/ossec-remoted > root 5734 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-syscheckd > ossec 5737 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-monitord > root 5747 2407 0 12:50 pts/1 00:00:00 grep ossec > [root@ossectst ossec]# gdb /var/ossec/bin/ossec-analysisd 5719 > GNU gdb (GDB) Amazon Linux (7.6.1-51.24.amzn1) > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-amazon-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from /var/ossec/bin/ossec-analysisd...done. > Attaching to program: /var/ossec/bin/ossec-analysisd, process 5719 > Reading symbols from /lib64/libm.so.6...Reading symbols from > /usr/lib/debug/lib64/libm-2.17.so.debug...done. > done. > Loaded symbols for /lib64/libm.so.6 > Reading symbols from /lib64/libc.so.6...Reading symbols from > /usr/lib/debug/lib64/libc-2.17.so.debug...done. > done. > Loaded symbols for /lib64/libc.so.6 > Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from > /usr/lib/debug/lib64/ld-2.17.so.debug...done. > done. > Loaded symbols for /lib64/ld-linux-x86-64.so.2 > Reading symbols from /lib64/libnss_files.so.2...Reading symbols from > /usr/lib/debug/lib64/libnss_files-2.17.so.debug...done. > done. > Loaded symbols for /lib64/libnss_files.so.2 > 0x00007f9f2b003b53 in __recvfrom_nocancel () at > ../sysdeps/unix/syscall-template.S:81 > 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) > (gdb) bt > #0 0x00007f9f2b003b53 in __recvfrom_nocancel () at > ../sysdeps/unix/syscall-template.S:81 > #1 0x0000000000430569 in OS_RecvUnix (socket=4, sizet=6144, > ret=0x7fffe0f85870 "1:/var/log/maillog") at os_net.c:539 > #2 0x0000000000403612 in OS_ReadMSG (m_queue=4) at analysisd.c:754 > #3 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at > analysisd.c:555 > (gdb) cont > Continuing. > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000000000 in ?? () > (gdb) where > #0 0x0000000000000000 in ?? () > #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x1addcd0) at analysisd.c:1631 > #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x1adcac0) at analysisd.c:1654 > #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x1ad53f0) at analysisd.c:1654 > #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x199c4e0) at analysisd.c:1654 > #5 0x0000000000403a5e in OS_ReadMSG (m_queue=4) at analysisd.c:984 > #6 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at > analysisd.c:555 > (gdb) list > 76 #else > 77 > 78 /* This is a "normal" system call stub: if there is an error, > 79 it returns -1 and sets errno. */ > 80 > 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) > 82 ret > 83 T_PSEUDO_END (SYSCALL_SYMBOL) > 84 > 85 #endif > (gdb) bt full > #0 0x0000000000000000 in ?? () > No symbol table info available. > #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x1addcd0) at analysisd.c:1631 > currently_rule = 0x1add7c0 > #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x1adcac0) at analysisd.c:1654 > child_node = 0x1addcd0 > child_rule = 0x0 > currently_rule = 0x1adc770 > #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x1ad53f0) at analysisd.c:1654 > child_node = 0x1adcac0 > child_rule = 0x0 > currently_rule = 0x1ad5ff0 > #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > curr_node=0x199c4e0) at analysisd.c:1654 > child_node = 0x1ad53f0 > child_rule = 0x0 > currently_rule = 0x199c210 > #5 0x0000000000403a5e in OS_ReadMSG (m_queue=4) at analysisd.c:984 > rulenode_pt = 0x199c4e0 > i = 765 > msg = "1:(itchy) 10.0.1.0->netstat -tan |grep LISTEN |grep -v > 127.0.0.1 | sort\000ossec: output: 'netstat -tan |grep LISTEN |grep -v > 127.0.0.1 | sort':\ntcp 0 0 0.0.0.0:22", ' ' <repeats 18 times>, > "0.0.0.0:*", ' ' <repeats 19 times>... > lf = 0x1b0a490 > ---Type <return> to continue, or q <return> to quit--- > stats_rule = 0x1b08b00 > #6 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at > analysisd.c:555 > c = -2 > m_queue = 4 > test_config = 0 > run_foreground = 0 > debug_level = 0 > dir = 0x444bc0 "/var/ossec" > user = 0x444bcb "ossec" > group = 0x444bcb "ossec" > uid = 503 > gid = 502 > cfg = 0x444bd1 "/var/ossec/etc/ossec.conf" > (gdb) cont > Continuing. > > Program terminated with signal SIGSEGV, Segmentation fault. > The program no longer exists. > (gdb) quit > [root@ossectst ossec]# service ossec stop > Stopping OSSEC: [ OK ] > [root@ossectst ossec]# tail /var/log/messages > ... snip ... > Dec 15 10:45:51 ossectst kernel: [ 2082.001287] ossec-analysisd[3772]: > segfault at 0 ip (null) sp 00007fffa4fa8218 error 14 in > ossec-analysisd[400000+65000] > > > > On Monday, December 15, 2014 8:56:33 AM UTC-6, dan (ddpbsd) wrote: >> >> On Mon, Dec 15, 2014 at 8:47 AM, dan (ddp) <ddp...@gmail.com> wrote: >> > On Fri, Dec 12, 2014 at 5:37 PM, <mje...@gmail.com> wrote: >> >> Ok, I can reproduce the segfault with a bare minimum. Here is what I >> >> did: >> >> >> >> 1) I spun up a brand new server (clone of my current ossec server), >> >> installed a fresh 2.8.1 ossec server. Modified local_rules.xml to look >> >> like >> >> this: >> >> >> >> <!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ >> >> >> >> - Example of local rules for OSSEC. >> >> - >> >> - Copyright (C) 2009 Trend Micro Inc. >> >> - All rights reserved. >> >> - >> >> - This program is a free software; you can redistribute it >> >> - and/or modify it under the terms of the GNU General Public >> >> - License (version 2) as published by the FSF - Free Software >> >> - Foundation. >> >> - >> >> - License details: http://www.ossec.net/en/licensing.html >> >> --> >> >> <!-- Modify it at your will. --> >> >> >> >> <group name="local,syslog,"> >> >> <!-- move from level 7 to level 5 6/25/2013 --> >> >> <rule id="533" level="5" overwrite="yes"> >> >> <if_sid>530</if_sid> >> >> <match>ossec: output: 'netstat -tan</match> >> >> <check_diff /> >> >> <description>Listened ports status (netstat) changed (new port >> >> opened or >> >> closed).</description> >> >> </rule> >> >> </group> <!-- SYSLOG,LOCAL --> >> >> <!-- EOF --> >> >> >> >> I did NOT add any agents, but let that run for about an hour. >> >> >> >> 2) I then stopped the ossec server, ran manage_agents, added one agent >> >> and >> >> restarted the ossec server. NOTE, I did not transfer the agent key, so >> >> the >> >> agent was not trying to communicate yet. I let that run for about an >> >> hour. >> >> >> >> 3) I transferred the key to the agent - this is an existing agent, I >> >> just >> >> added the new key and change the <server-ip> to point to my new test >> >> server >> >> and restarted the agent and server. I begin to see communication >> >> between >> >> server and agent. I let it run it's initial scans - about 10 minutes. >> >> >> >> 4) Then I opened a new listen port to try and trigger the netstat -tan >> >> rule >> >> using nc -l 7777 on the agent. >> >> >> >> 5) ossec-analysisd segfaulted after about 7 minues. >> >> >> >> Let me know if you want any of the output or if you want me to run any >> >> other >> >> tests. >> >> >> > >> > Wow, thanks. I was able to reproduce it with this info. >> > >> >> Sometimes. It apparently doesn't like crashing when running under gdb. >> >> >> >> >> >> >> On Friday, December 12, 2014 11:14:47 AM UTC-6, mje...@gmail.com wrote: >> >>> >> >>> I get the segfault with ONLY the posted rule in my local_rules.xml. >> >>> In >> >>> other words, with a completely vanilla install of 2.8.1, by just >> >>> adding that >> >>> rule to local_rules.xml and restarting. Unfortunately, I can't >> >>> remember if >> >>> I needed to add agents before I got the segfault. I'm happy to spin up >> >>> a new >> >>> instance and test it out though. >> >>> >> >>> On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: >> >>>> >> >>>> On Fri, Dec 12, 2014 at 11:33 AM, <mje...@gmail.com> wrote: >> >>>> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but I >> >>>> > meant v >> >>>> > 2.7 and 2.8. >> >>>> > >> >>>> >> >>>> I'm not able to reproduce this, could you provide your entire >> >>>> local_rules.xml? >> >>>> >> >>>> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) wrote: >> >>>> >> >> >>>> >> On Fri, Dec 12, 2014 at 10:43 AM, <mje...@gmail.com> wrote: >> >>>> >> > I recently migrated our ossec server to new hardware, and at the >> >>>> >> > same >> >>>> >> > time >> >>>> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept >> >>>> >> > segfaulting >> >>>> >> > in the new environment. It's taken me a couple of weeks to >> >>>> >> > narrow it >> >>>> >> > down, >> >>>> >> > and it seems to be consistently caused by the following rule >> >>>> >> > from my >> >>>> >> > local_rules.xml (downgrade of a base rule from level 7 to level >> >>>> >> > 5): >> >>>> >> > >> >>>> >> > <rule id="533" level="5" overwrite="yes"> >> >>>> >> > <if_sid>530</if_sid> >> >>>> >> > <match>ossec: output: 'netstat -tan</match> >> >>>> >> > <check_diff /> >> >>>> >> > <description>Listened ports status (netstat) changed (new >> >>>> >> > port >> >>>> >> > opened or >> >>>> >> > closed).</description> >> >>>> >> > </rule> >> >>>> >> > >> >>>> >> > This rule has been running under v1.7 in our old environment for >> >>>> >> > over a >> >>>> >> > year. For now, I've just commented out this rule and all is >> >>>> >> > good, >> >>>> >> > but I >> >>>> >> > thought it might be helpful to the community if we figure out >> >>>> >> > what's >> >>>> >> > going >> >>>> >> > on - is it a bug, or some boneheaded misconfiguration on my >> >>>> >> > part. >> >>>> >> > I'm >> >>>> >> > happy >> >>>> >> > to provide more detail if needed. >> >>>> >> > >> >>>> >> > Current environment is Amazon AWS VPC. All servers are running >> >>>> >> > Amazon >> >>>> >> > Linux >> >>>> >> > 64-bit AMI's (based on RHEL/CentOS). >> >>>> >> > [root@smithers ossec]# uname -a >> >>>> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 >> >>>> >> > 23:07:48 >> >>>> >> > UTC >> >>>> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >> >>>> >> > >> >>>> >> > Initially I tried to migrate everything that was relevant >> >>>> >> > (client.keys, >> >>>> >> > ossec.conf, local_rules.xml, rids files, etc). But with constant >> >>>> >> > crashes, I >> >>>> >> > backed everything out and used the following basic process to >> >>>> >> > narrow >> >>>> >> > down >> >>>> >> > what was happening: >> >>>> >> > >> >>>> >> > 1) I installed v1.8 server, and ran the default install with no >> >>>> >> > agents >> >>>> >> > for >> >>>> >> > about 24 hours. All good. >> >>>> >> > 2) Added one agent (fresh install, new key, no custom rules), >> >>>> >> > ran >> >>>> >> > for 24 >> >>>> >> > hours. All good. >> >>>> >> > 3) Added remaining 7 agents (fresh install, new key, no custom >> >>>> >> > rules), >> >>>> >> > ran >> >>>> >> > for 24 hours. All good. >> >>>> >> > 4) Added my local_rules.xml, analysisd segfaulted within a few >> >>>> >> > minutes. >> >>>> >> > 5) Removed local_rules.xml, ran for 24 hours with no problems. >> >>>> >> > 6) Added local_rules.xml back, analysisd segfaulted after a few >> >>>> >> > minutes. >> >>>> >> > 7) Removed local_rules.xml. Started adding one rule at a time, >> >>>> >> > and >> >>>> >> > running >> >>>> >> > for 6 - 12 hours with each new rule. >> >>>> >> > 8) The above rule seems to consistently be the problem. HOWEVER, >> >>>> >> > since >> >>>> >> > it >> >>>> >> > can take anywhere from a few minutes to a few hours (up to 5) >> >>>> >> > for >> >>>> >> > the >> >>>> >> > segfault to occur, it is possible the above rule is not the >> >>>> >> > problem. >> >>>> >> > >> >>>> >> > I did try running analysisd with gdb to get some more >> >>>> >> > information, >> >>>> >> > but >> >>>> >> > I'm >> >>>> >> > not confident that I got all the ossec processes started >> >>>> >> > correctly >> >>>> >> > when >> >>>> >> > I >> >>>> >> > did it. I can post all the output from that run if someone wants >> >>>> >> > it. >> >>>> >> > >> >>>> >> > Here is an example of the output in /var/log/messages: >> >>>> >> > >> >>>> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] >> >>>> >> > ossec-analysisd[12455]: >> >>>> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 >> >>>> >> > in >> >>>> >> > ossec-analysisd[400000+65000] >> >>>> >> > >> >>>> >> > Following the segfault, these are the ossec processes running >> >>>> >> > (although >> >>>> >> > sometimes syscheckd and/or monitord is not running): >> >>>> >> > >> >>>> >> > [root@smithers ossec]# ps -ef | grep ossec >> >>>> >> > ossecm 12447 1 0 09:58 ? 00:00:00 >> >>>> >> > /var/ossec/bin/ossec-maild >> >>>> >> > root 12451 1 0 09:58 ? 00:00:00 >> >>>> >> > /var/ossec/bin/ossec-execd >> >>>> >> > root 12470 1 0 09:58 ? 00:00:03 >> >>>> >> > /var/ossec/bin/ossec-syscheckd >> >>>> >> > ossec 12473 1 0 09:58 ? 00:00:00 >> >>>> >> > /var/ossec/bin/ossec-monitord >> >>>> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec >> >>>> >> > >> >>>> >> > Here is the ossec.log from start to finish (this time it took a >> >>>> >> > couple >> >>>> >> > of >> >>>> >> > hours for the segfault to occur): >> >>>> >> > >> >>>> >> > [root@smithers ossec]# tail -f logs/ossec.log >> >>>> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder >> >>>> >> > file. >> >>>> >> [snip] >> >>>> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). >> >>>> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending >> >>>> >> > message >> >>>> >> > to >> >>>> >> > queue. >> >>>> >> > >> >>>> >> > Let me know if anything else would be helpful. >> >>>> >> > >> >>>> >> >> >>>> >> I'm not seeing this issue on post 2.8.1 code, any chance you can >> >>>> >> upgrade to something remotely recent? >> >>>> >> >> >>>> >> > -- >> >>>> >> > >> >>>> >> > --- >> >>>> >> > You received this message because you are subscribed to the >> >>>> >> > Google >> >>>> >> > Groups >> >>>> >> > "ossec-list" group. >> >>>> >> > To unsubscribe from this group and stop receiving emails from >> >>>> >> > it, >> >>>> >> > send >> >>>> >> > an >> >>>> >> > email to ossec-list+...@googlegroups.com. >> >>>> >> > For more options, visit https://groups.google.com/d/optout. >> >>>> > >> >>>> > -- >> >>>> > >> >>>> > --- >> >>>> > You received this message because you are subscribed to the Google >> >>>> > Groups >> >>>> > "ossec-list" group. >> >>>> > To unsubscribe from this group and stop receiving emails from it, >> >>>> > send >> >>>> > an >> >>>> > email to ossec-list+...@googlegroups.com. >> >>>> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> >> >> >> --- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "ossec-list" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> >> an >> >> email to ossec-list+...@googlegroups.com. >> >> For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ossec-list+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
