On Fri, Dec 12, 2014 at 10:43 AM, <mje...@gmail.com> wrote: > I recently migrated our ossec server to new hardware, and at the same time > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept segfaulting > in the new environment. It's taken me a couple of weeks to narrow it down, > and it seems to be consistently caused by the following rule from my > local_rules.xml (downgrade of a base rule from level 7 to level 5): > > <rule id="533" level="5" overwrite="yes"> > <if_sid>530</if_sid> > <match>ossec: output: 'netstat -tan</match> > <check_diff /> > <description>Listened ports status (netstat) changed (new port opened or > closed).</description> > </rule> > > This rule has been running under v1.7 in our old environment for over a > year. For now, I've just commented out this rule and all is good, but I > thought it might be helpful to the community if we figure out what's going > on - is it a bug, or some boneheaded misconfiguration on my part. I'm happy > to provide more detail if needed. > > Current environment is Amazon AWS VPC. All servers are running Amazon Linux > 64-bit AMI's (based on RHEL/CentOS). > [root@smithers ossec]# uname -a > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 UTC > 2014 x86_64 x86_64 x86_64 GNU/Linux > > Initially I tried to migrate everything that was relevant (client.keys, > ossec.conf, local_rules.xml, rids files, etc). But with constant crashes, I > backed everything out and used the following basic process to narrow down > what was happening: > > 1) I installed v1.8 server, and ran the default install with no agents for > about 24 hours. All good. > 2) Added one agent (fresh install, new key, no custom rules), ran for 24 > hours. All good. > 3) Added remaining 7 agents (fresh install, new key, no custom rules), ran > for 24 hours. All good. > 4) Added my local_rules.xml, analysisd segfaulted within a few minutes. > 5) Removed local_rules.xml, ran for 24 hours with no problems. > 6) Added local_rules.xml back, analysisd segfaulted after a few minutes. > 7) Removed local_rules.xml. Started adding one rule at a time, and running > for 6 - 12 hours with each new rule. > 8) The above rule seems to consistently be the problem. HOWEVER, since it > can take anywhere from a few minutes to a few hours (up to 5) for the > segfault to occur, it is possible the above rule is not the problem. > > I did try running analysisd with gdb to get some more information, but I'm > not confident that I got all the ossec processes started correctly when I > did it. I can post all the output from that run if someone wants it. > > Here is an example of the output in /var/log/messages: > > Dec 9 12:02:03 smithers kernel: [513311.628392] ossec-analysisd[12455]: > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 in > ossec-analysisd[400000+65000] > > Following the segfault, these are the ossec processes running (although > sometimes syscheckd and/or monitord is not running): > > [root@smithers ossec]# ps -ef | grep ossec > ossecm 12447 1 0 09:58 ? 00:00:00 /var/ossec/bin/ossec-maild > root 12451 1 0 09:58 ? 00:00:00 /var/ossec/bin/ossec-execd > root 12470 1 0 09:58 ? 00:00:03 > /var/ossec/bin/ossec-syscheckd > ossec 12473 1 0 09:58 ? 00:00:00 > /var/ossec/bin/ossec-monitord > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec > > Here is the ossec.log from start to finish (this time it took a couple of > hours for the segfault to occur): > > [root@smithers ossec]# tail -f logs/ossec.log > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder file. [snip] > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending message to > queue. > > Let me know if anything else would be helpful. >
I'm not seeing this issue on post 2.8.1 code, any chance you can upgrade to something remotely recent? > -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ossec-list+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
