|
Hi Risto,
On 03/24/2015 03:37 PM, Risto Vaarandi wrote:
hi Leonard,
when sec is connected to a syslog server over a pipe, there
is always a theoretical chance of data loss, since syslog
servers usually write to pipes in a non-blocking way. If bytes
are written to pipe in a faster rate than the reader is able
to fetch them, the pipe will eventually become full, and the
following writes into pipe will fail with data loss. This
problem was more frequent on older platforms where pipes could
only accommodate 4KB. However, on more recent Linux (and
other) platforms pipes can take significantly more bytes (like
256KB) and occasional data transfer peaks can be seamlessly
handled.
This is a RHEL 5.10 machine with 3GB of RAM, it's a VM running on a
lightly loaded R620(pretty beefy hardware). I do notice that CPU
usage of SEC(not syslog-ng) is pegged at 99% fairly often.
You mentioned an event rate of 100 messages per second --
is this the rate of messages for the rule you have included in
your post, or an overall message rate for all rules? If it's
the overall rate, sec should be able to handle this easily,
but a lot depends on the actual configuration. Just out of
curiosity, what is the total number of rules in your ruleset,
how many matches they typically produce per minute (or hour),
and do the rules run expensive actions (like calling
computationally expensive Perl functions with 'lcall' or
'eval' actions)? Also, you mentioned 15 hours of reporting
data within 15 minutes -- does this mean that a past data for
15 hour time frame are submitted for processing to sec within
15 minutes?
The rate is the amount of events coming to syslog-ng, which should
in turn go to SEC; so for all rules. I have 433 rules. In regards to
the 15 hours, that's correct(I should have worded it better). For
example, at 7PM exactly I'd expect to get 10 syslog messages
destined for 10 different clamav rules, but from 7PM-7:15PM, I'd
expect a total of 200 messages for the same clamav rules. During
this timeframe I would easily get several thousand non-clamav
related messages for processing.
Also, may I offer you a small advise on starting sec from
syslog-ng -- I'd recommend to include the --notail option in
sec command line, since this will ensure that sec exits when
syslog-ng closes its end of the data transfer pipe. Otherwise,
orphaned instances may stay around in the process table which
consume system resources. There is also a FAQ entry which
attempts to provide a small syslog-ng config example and an
explanation of the issue: http://simple-evcorr.sourceforge.net/FAQ.html#3
I have not seen this happen before, but I'll look into this.
|
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users