Got into work late today... I'm glad the info I gave is a help. I got it to crash under gdb once, but as I said, I'm not sure I started everything properly. What I did was start analysisd using gdb, then, if I recall, I ran the normal ossec startup script. Unfortunately it appears I only saved the gdb output, not the ossec logs from that run. It may help you though. I'm not very proficient with gdb, so I only ran some basic stuff, but I'm happy to try it again if you think it will help.
[root@smithers ossec]# gdb /var/ossec/bin/ossec-analysisd GNU gdb (GDB) Amazon Linux (7.6.1-51.24.amzn1) Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-amazon-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /var/ossec/bin/ossec-analysisd...done. (gdb) set follow-fork-mode child (gdb) run -d Starting program: /var/ossec/bin/ossec-analysisd -d 2014/12/03 12:10:36 ossec-analysisd: DEBUG: Starting ... 2014/12/03 12:10:36 ossec-analysisd: DEBUG: Found user/group ... 2014/12/03 12:10:36 ossec-analysisd: DEBUG: Active response initialized ... 2014/12/03 12:10:36 adding rule: rules_config.xml 2014/12/03 12:10:36 adding rule: pam_rules.xml 2014/12/03 12:10:36 adding rule: sshd_rules.xml 2014/12/03 12:10:36 adding rule: telnetd_rules.xml 2014/12/03 12:10:36 adding rule: syslog_rules.xml 2014/12/03 12:10:36 adding rule: arpwatch_rules.xml 2014/12/03 12:10:36 adding rule: symantec-av_rules.xml 2014/12/03 12:10:36 adding rule: symantec-ws_rules.xml 2014/12/03 12:10:36 adding rule: pix_rules.xml 2014/12/03 12:10:36 adding rule: named_rules.xml 2014/12/03 12:10:36 adding rule: smbd_rules.xml 2014/12/03 12:10:36 adding rule: vsftpd_rules.xml 2014/12/03 12:10:36 adding rule: pure-ftpd_rules.xml 2014/12/03 12:10:36 adding rule: proftpd_rules.xml 2014/12/03 12:10:36 adding rule: ms_ftpd_rules.xml 2014/12/03 12:10:36 adding rule: ftpd_rules.xml 2014/12/03 12:10:36 adding rule: hordeimp_rules.xml 2014/12/03 12:10:36 adding rule: roundcube_rules.xml 2014/12/03 12:10:36 adding rule: wordpress_rules.xml 2014/12/03 12:10:36 adding rule: cimserver_rules.xml 2014/12/03 12:10:36 adding rule: vpopmail_rules.xml 2014/12/03 12:10:36 adding rule: vmpop3d_rules.xml 2014/12/03 12:10:36 adding rule: courier_rules.xml 2014/12/03 12:10:36 adding rule: web_rules.xml 2014/12/03 12:10:36 adding rule: web_appsec_rules.xml 2014/12/03 12:10:36 adding rule: apache_rules.xml 2014/12/03 12:10:36 adding rule: nginx_rules.xml 2014/12/03 12:10:36 adding rule: php_rules.xml 2014/12/03 12:10:36 adding rule: mysql_rules.xml 2014/12/03 12:10:36 adding rule: postgresql_rules.xml 2014/12/03 12:10:36 adding rule: ids_rules.xml 2014/12/03 12:10:36 adding rule: squid_rules.xml 2014/12/03 12:10:36 adding rule: firewall_rules.xml 2014/12/03 12:10:36 adding rule: cisco-ios_rules.xml 2014/12/03 12:10:36 adding rule: netscreenfw_rules.xml 2014/12/03 12:10:36 adding rule: sonicwall_rules.xml 2014/12/03 12:10:36 adding rule: postfix_rules.xml 2014/12/03 12:10:36 adding rule: sendmail_rules.xml 2014/12/03 12:10:36 adding rule: imapd_rules.xml 2014/12/03 12:10:36 adding rule: mailscanner_rules.xml 2014/12/03 12:10:36 adding rule: dovecot_rules.xml 2014/12/03 12:10:36 adding rule: ms-exchange_rules.xml 2014/12/03 12:10:36 adding rule: racoon_rules.xml 2014/12/03 12:10:36 adding rule: vpn_concentrator_rules.xml 2014/12/03 12:10:36 adding rule: spamd_rules.xml 2014/12/03 12:10:36 adding rule: msauth_rules.xml 2014/12/03 12:10:36 adding rule: mcafee_av_rules.xml 2014/12/03 12:10:36 adding rule: trend-osce_rules.xml 2014/12/03 12:10:36 adding rule: ms-se_rules.xml 2014/12/03 12:10:36 adding rule: zeus_rules.xml 2014/12/03 12:10:36 adding rule: solaris_bsm_rules.xml 2014/12/03 12:10:36 adding rule: vmware_rules.xml 2014/12/03 12:10:36 adding rule: ms_dhcp_rules.xml 2014/12/03 12:10:36 adding rule: asterisk_rules.xml 2014/12/03 12:10:36 adding rule: ossec_rules.xml 2014/12/03 12:10:36 adding rule: attack_rules.xml 2014/12/03 12:10:36 adding rule: openbsd_rules.xml 2014/12/03 12:10:36 adding rule: clam_av_rules.xml 2014/12/03 12:10:36 adding rule: dropbear_rules.xml 2014/12/03 12:10:36 adding rule: local_rules.xml 2014/12/03 12:10:36 ossec-analysisd: DEBUG: Read configuration ... [New process 14891] [New process 14892] Program received signal SIGSEGV, Segmentation fault. [Switching to process 14892] 0x0000000000000000 in ?? () Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.87.amzn1.x86_64 (gdb) (gdb) where #0 0x0000000000000000 in ?? () #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x7bcd40) at analysisd.c:1631 #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x7bbb30) at analysisd.c:1654 #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x7b5370) at analysisd.c:1654 #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x6939a0) at analysisd.c:1654 #5 0x0000000000403a5e in OS_ReadMSG (m_queue=8) at analysisd.c:984 #6 0x0000000000403262 in main (argc=2, argv=0x7fffffffe4c8) at analysisd.c:555 (gdb) list 129 130 /** int main(int argc, char **argv) 131 */ 132 #ifndef TESTRULE 133 int main(int argc, char **argv) 134 #else 135 int main_analysisd(int argc, char **argv) 136 #endif 137 { 138 int c = 0, m_queue = 0, test_config = 0,run_foreground = 0; (gdb) bt full #0 0x0000000000000000 in ?? () No symbol table info available. #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x7bcd40) at analysisd.c:1631 currently_rule = 0x7bc830 #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x7bbb30) at analysisd.c:1654 child_node = 0x7bcd40 child_rule = 0x0 currently_rule = 0x7bb7e0 #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x7b5370) at analysisd.c:1654 child_node = 0x7bbb30 child_rule = 0x0 currently_rule = 0x7b45c0 #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x7f6820, curr_node=0x6939a0) at analysisd.c:1654 child_node = 0x7b5370 child_rule = 0x0 currently_rule = 0x691350 #5 0x0000000000403a5e in OS_ReadMSG (m_queue=8) at analysisd.c:984 rulenode_pt = 0x6939a0 i = 502 msg = "1:(scratchy) 10.0.1.90->netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort\000ossec: output: 'netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort':\ntcp 0 0 0.0.0.0:22", ' ' <repeats 18 times>, "0.0.0."... lf = 0x7f6820 stats_rule = 0x7eb770 ---Type <return> to continue, or q <return> to quit--- #6 0x0000000000403262 in main (argc=2, argv=0x7fffffffe4c8) at analysisd.c:555 c = -2 m_queue = 8 test_config = 0 run_foreground = 0 debug_level = 1 dir = 0x444bc0 "/var/ossec" user = 0x444bcb "ossec" group = 0x444bcb "ossec" uid = 503 gid = 502 cfg = 0x444bd1 "/var/ossec/etc/ossec.conf" (gdb) On Monday, December 15, 2014 8:56:33 AM UTC-6, dan (ddpbsd) wrote: > > On Mon, Dec 15, 2014 at 8:47 AM, dan (ddp) <ddp...@gmail.com <javascript:>> > wrote: > > On Fri, Dec 12, 2014 at 5:37 PM, <mje...@gmail.com <javascript:>> > wrote: > >> Ok, I can reproduce the segfault with a bare minimum. Here is what I > did: > >> > >> 1) I spun up a brand new server (clone of my current ossec server), > >> installed a fresh 2.8.1 ossec server. Modified local_rules.xml to look > like > >> this: > >> > >> <!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ > >> > >> - Example of local rules for OSSEC. > >> - > >> - Copyright (C) 2009 Trend Micro Inc. > >> - All rights reserved. > >> - > >> - This program is a free software; you can redistribute it > >> - and/or modify it under the terms of the GNU General Public > >> - License (version 2) as published by the FSF - Free Software > >> - Foundation. > >> - > >> - License details: http://www.ossec.net/en/licensing.html > >> --> > >> <!-- Modify it at your will. --> > >> > >> <group name="local,syslog,"> > >> <!-- move from level 7 to level 5 6/25/2013 --> > >> <rule id="533" level="5" overwrite="yes"> > >> <if_sid>530</if_sid> > >> <match>ossec: output: 'netstat -tan</match> > >> <check_diff /> > >> <description>Listened ports status (netstat) changed (new port > opened or > >> closed).</description> > >> </rule> > >> </group> <!-- SYSLOG,LOCAL --> > >> <!-- EOF --> > >> > >> I did NOT add any agents, but let that run for about an hour. > >> > >> 2) I then stopped the ossec server, ran manage_agents, added one agent > and > >> restarted the ossec server. NOTE, I did not transfer the agent key, so > the > >> agent was not trying to communicate yet. I let that run for about an > hour. > >> > >> 3) I transferred the key to the agent - this is an existing agent, I > just > >> added the new key and change the <server-ip> to point to my new test > server > >> and restarted the agent and server. I begin to see communication > between > >> server and agent. I let it run it's initial scans - about 10 minutes. > >> > >> 4) Then I opened a new listen port to try and trigger the netstat -tan > rule > >> using nc -l 7777 on the agent. > >> > >> 5) ossec-analysisd segfaulted after about 7 minues. > >> > >> Let me know if you want any of the output or if you want me to run any > other > >> tests. > >> > > > > Wow, thanks. I was able to reproduce it with this info. > > > > Sometimes. It apparently doesn't like crashing when running under gdb. > > >> > >> > >> On Friday, December 12, 2014 11:14:47 AM UTC-6, mje...@gmail.com > wrote: > >>> > >>> I get the segfault with ONLY the posted rule in my local_rules.xml. > In > >>> other words, with a completely vanilla install of 2.8.1, by just > adding that > >>> rule to local_rules.xml and restarting. Unfortunately, I can't > remember if > >>> I needed to add agents before I got the segfault. I'm happy to spin up > a new > >>> instance and test it out though. > >>> > >>> On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: > >>>> > >>>> On Fri, Dec 12, 2014 at 11:33 AM, <mje...@gmail.com> wrote: > >>>> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but I > >>>> > meant v > >>>> > 2.7 and 2.8. > >>>> > > >>>> > >>>> I'm not able to reproduce this, could you provide your entire > >>>> local_rules.xml? > >>>> > >>>> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) wrote: > >>>> >> > >>>> >> On Fri, Dec 12, 2014 at 10:43 AM, <mje...@gmail.com> wrote: > >>>> >> > I recently migrated our ossec server to new hardware, and at the > >>>> >> > same > >>>> >> > time > >>>> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept > >>>> >> > segfaulting > >>>> >> > in the new environment. It's taken me a couple of weeks to > narrow it > >>>> >> > down, > >>>> >> > and it seems to be consistently caused by the following rule > from my > >>>> >> > local_rules.xml (downgrade of a base rule from level 7 to level > 5): > >>>> >> > > >>>> >> > <rule id="533" level="5" overwrite="yes"> > >>>> >> > <if_sid>530</if_sid> > >>>> >> > <match>ossec: output: 'netstat -tan</match> > >>>> >> > <check_diff /> > >>>> >> > <description>Listened ports status (netstat) changed (new > port > >>>> >> > opened or > >>>> >> > closed).</description> > >>>> >> > </rule> > >>>> >> > > >>>> >> > This rule has been running under v1.7 in our old environment for > >>>> >> > over a > >>>> >> > year. For now, I've just commented out this rule and all is > good, > >>>> >> > but I > >>>> >> > thought it might be helpful to the community if we figure out > what's > >>>> >> > going > >>>> >> > on - is it a bug, or some boneheaded misconfiguration on my > part. > >>>> >> > I'm > >>>> >> > happy > >>>> >> > to provide more detail if needed. > >>>> >> > > >>>> >> > Current environment is Amazon AWS VPC. All servers are running > >>>> >> > Amazon > >>>> >> > Linux > >>>> >> > 64-bit AMI's (based on RHEL/CentOS). > >>>> >> > [root@smithers ossec]# uname -a > >>>> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 > 23:07:48 > >>>> >> > UTC > >>>> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux > >>>> >> > > >>>> >> > Initially I tried to migrate everything that was relevant > >>>> >> > (client.keys, > >>>> >> > ossec.conf, local_rules.xml, rids files, etc). But with constant > >>>> >> > crashes, I > >>>> >> > backed everything out and used the following basic process to > narrow > >>>> >> > down > >>>> >> > what was happening: > >>>> >> > > >>>> >> > 1) I installed v1.8 server, and ran the default install with no > >>>> >> > agents > >>>> >> > for > >>>> >> > about 24 hours. All good. > >>>> >> > 2) Added one agent (fresh install, new key, no custom rules), > ran > >>>> >> > for 24 > >>>> >> > hours. All good. > >>>> >> > 3) Added remaining 7 agents (fresh install, new key, no custom > >>>> >> > rules), > >>>> >> > ran > >>>> >> > for 24 hours. All good. > >>>> >> > 4) Added my local_rules.xml, analysisd segfaulted within a few > >>>> >> > minutes. > >>>> >> > 5) Removed local_rules.xml, ran for 24 hours with no problems. > >>>> >> > 6) Added local_rules.xml back, analysisd segfaulted after a few > >>>> >> > minutes. > >>>> >> > 7) Removed local_rules.xml. Started adding one rule at a time, > and > >>>> >> > running > >>>> >> > for 6 - 12 hours with each new rule. > >>>> >> > 8) The above rule seems to consistently be the problem. HOWEVER, > >>>> >> > since > >>>> >> > it > >>>> >> > can take anywhere from a few minutes to a few hours (up to 5) > for > >>>> >> > the > >>>> >> > segfault to occur, it is possible the above rule is not the > problem. > >>>> >> > > >>>> >> > I did try running analysisd with gdb to get some more > information, > >>>> >> > but > >>>> >> > I'm > >>>> >> > not confident that I got all the ossec processes started > correctly > >>>> >> > when > >>>> >> > I > >>>> >> > did it. I can post all the output from that run if someone wants > it. > >>>> >> > > >>>> >> > Here is an example of the output in /var/log/messages: > >>>> >> > > >>>> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] > >>>> >> > ossec-analysisd[12455]: > >>>> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 > in > >>>> >> > ossec-analysisd[400000+65000] > >>>> >> > > >>>> >> > Following the segfault, these are the ossec processes running > >>>> >> > (although > >>>> >> > sometimes syscheckd and/or monitord is not running): > >>>> >> > > >>>> >> > [root@smithers ossec]# ps -ef | grep ossec > >>>> >> > ossecm 12447 1 0 09:58 ? 00:00:00 > >>>> >> > /var/ossec/bin/ossec-maild > >>>> >> > root 12451 1 0 09:58 ? 00:00:00 > >>>> >> > /var/ossec/bin/ossec-execd > >>>> >> > root 12470 1 0 09:58 ? 00:00:03 > >>>> >> > /var/ossec/bin/ossec-syscheckd > >>>> >> > ossec 12473 1 0 09:58 ? 00:00:00 > >>>> >> > /var/ossec/bin/ossec-monitord > >>>> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec > >>>> >> > > >>>> >> > Here is the ossec.log from start to finish (this time it took a > >>>> >> > couple > >>>> >> > of > >>>> >> > hours for the segfault to occur): > >>>> >> > > >>>> >> > [root@smithers ossec]# tail -f logs/ossec.log > >>>> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder > >>>> >> > file. > >>>> >> [snip] > >>>> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). > >>>> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending > >>>> >> > message > >>>> >> > to > >>>> >> > queue. > >>>> >> > > >>>> >> > Let me know if anything else would be helpful. > >>>> >> > > >>>> >> > >>>> >> I'm not seeing this issue on post 2.8.1 code, any chance you can > >>>> >> upgrade to something remotely recent? > >>>> >> > >>>> >> > -- > >>>> >> > > >>>> >> > --- > >>>> >> > You received this message because you are subscribed to the > Google > >>>> >> > Groups > >>>> >> > "ossec-list" group. > >>>> >> > To unsubscribe from this group and stop receiving emails from > it, > >>>> >> > send > >>>> >> > an > >>>> >> > email to ossec-list+...@googlegroups.com. > >>>> >> > For more options, visit https://groups.google.com/d/optout. > >>>> > > >>>> > -- > >>>> > > >>>> > --- > >>>> > You received this message because you are subscribed to the Google > >>>> > Groups > >>>> > "ossec-list" group. > >>>> > To unsubscribe from this group and stop receiving emails from it, > send > >>>> > an > >>>> > email to ossec-list+...@googlegroups.com. > >>>> > For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> > >> --- > >> You received this message because you are subscribed to the Google > Groups > >> "ossec-list" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to ossec-list+...@googlegroups.com <javascript:>. > >> For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
