2013/5/5 David Lang <[email protected]>
> On Fri, 3 May 2013, John Zhang wrote:
>
> Hi everyone,
>>
>> I am researching the big data security log management, such as Kibana +
>> ElasticSearch + Logstash for my security log management, I need event
>> correlation on this platform, i know SEC(
>> http://simple-evcorr.**sourceforge.net/<http://simple-evcorr.sourceforge.net/>)
>> can do event correlation.
>>
>> Do you have any idea of SEC on such big data security log platform? Any
>> experience, any reference?
>>
>> Any comment, advise will be highly appreciated!
>>
>
> my presentation on logging from LISA is at https://www.usenix.org/**
> conference/lisa12/building-**100k-logsec-logging-**infrastructure<https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure>
>
> I used rsyslog as my transport, but if logstash can handle your load,
> there's nothing wrong with using it.
>
> What sort of data volume are you expecting?
>
> SEC works wonderfully well for this sort of thing, but the one issue that
> you will run into is the fact that SEC is single-threaded, so it's limited
> to how much data a single process can handle (how much depends on the
> number and complexity of your ruleset).
>
> The work-around for this is to split your logs into seperate feeds and
> have a seperate copy of SEC processing each feed. If you can split the
> feeds so that your correlation rules don't need to be across feeds, it's
> really easy to do this. Also note that you don't need to run all the feeds
> on one system.
>
> If you do need to correlate across feeds, take a look at this paper from a
> few years ago http://static.usenix.org/**events/lisa10/tech/full_**
> papers/Krizak.pdf<http://static.usenix.org/events/lisa10/tech/full_papers/Krizak.pdf>they
> used syslog-ng and their own analysis engine, but the approach of
> having small instances that look at a particular type of log and then when
> one instance finds something, instead of alerting directly, it generates a
> new log message and there is a instance of SEC that only watches these
> 'internal' messages and that instance generates all the alerts, and deals
> with correlation rules that need to take into account the different feeds.
>
> David Lang
>
>
>
Being single-threaded is indeed one limitations of SEC, but on the other
hand, it would be quite hard to turn it into a multi-threaded application,
since its design allows for sharing and modifying event correlation state
across all inputs/rulesets. I was briefly thinking about creating a
configuration file directive for indicating that some input-ruleset
combinations can be processed independently from others, which would
naturally allow for running them as separate threads. Nevertheless, in most
cases it is easier to run another SEC process for this purpose. Of course,
it could be argued that having several processes complicates startup
scripts and makes system administration more complex. However, the 'spawn'
action allows for running a custom subprocess with its standard output
being captured and turned into synthetic events. Thus, one could run
several SEC instances with 'spawn' from one master process, which can
receive events from child processes and terminate them at its own graceful
shutdown. In principle, other process creation actions like 'shellcmd' can
be used for the same purpose, except that they can't issue synthetic events
back to the master process.
kind regards,
risto
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
