hi Leonard,
when sec is connected to a syslog server over a pipe, there is always a
theoretical chance of data loss, since syslog servers usually write to
pipes in a non-blocking way. If bytes are written to pipe in a faster rate
than the reader is able to fetch them, the pipe will eventually become
full, and the following writes into pipe will fail with data loss. This
problem was more frequent on older platforms where pipes could only
accommodate 4KB. However, on more recent Linux (and other) platforms pipes
can take significantly more bytes (like 256KB) and occasional data transfer
peaks can be seamlessly handled.
You mentioned an event rate of 100 messages per second -- is this the rate
of messages for the rule you have included in your post, or an overall
message rate for all rules? If it's the overall rate, sec should be able to
handle this easily, but a lot depends on the actual configuration. Just out
of curiosity, what is the total number of rules in your ruleset, how many
matches they typically produce per minute (or hour), and do the rules run
expensive actions (like calling computationally expensive Perl functions
with 'lcall' or 'eval' actions)? Also, you mentioned 15 hours of reporting
data within 15 minutes -- does this mean that a past data for 15 hour time
frame are submitted for processing to sec within 15 minutes?
Also, may I offer you a small advise on starting sec from syslog-ng -- I'd
recommend to include the --notail option in sec command line, since this
will ensure that sec exits when syslog-ng closes its end of the data
transfer pipe. Otherwise, orphaned instances may stay around in the process
table which consume system resources. There is also a FAQ entry which
attempts to provide a small syslog-ng config example and an explanation of
the issue: http://simple-evcorr.sourceforge.net/FAQ.html#3
kind regards,
risto
2015-03-24 22:59 GMT+02:00 Leonard Lawton <[email protected]>:
> I'm using syslog-ng to pipe events to SEC 2.7.4. In some times in what
> seems to be higher volumes of syslog traffic(Maybe 100 log
> messages/sec), I don't see SEC taking action on some rules(it's not
> making into the SEC logfile). I do not have any rate limiting setup for
> said rules.
>
> Here's an example of a rule that seems to processed intermittently:
>
> type=Single
> continue=DontCont
> ptype=RegExp
> pattern=\S+\s+\S+\s+\S+\s+(\S+).domain.com clamscan: Time: (\S+) sec .*
> - <user.notice>
> desc=$0
> action=shellcmd /usr/local/zabbix/
>
> The above rule might have about 15 hours reporting data within a 15
> minute period. Additionally, there are no other rules that would match
> this(trying to rule out a window)
>
> Syslog-ng config:
>
> destination log_watch {
> program("/usr/local/sbin/sec.pl -input=\"-\" -conf
> /etc/sec.conf -debug=5 -log=/var/log/sec.log -dump=/tmp/sec.dump"
> template(t_fp));
> };
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Simple-evcorr-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
