hi John,
if you plan to use Logstash for feeding Elasticsearch database, SEC can be
quite easily connected to it, since Logstash supports receiving data
through wide variety of inputs. Depending on your system and log data
volumes, you could have just one SEC instance which correlates all your
events and then sends them to Logstash --> Elasticsearch, but you could
also have a number of instances, each handling a part of the event volume
and accomplishing some more specific task. Since Elasticsearch is about
storing large volumes of log data, it is likely that you want to send a
significant amount of log messages directly to Elasticearch, and use SEC
for adding additional events to stored data. (My own setup looks like this,
but you could also have very different requirements in your environment.)
If you are looking for references and published materials about using SEC,
then I recollect a recent paper written by David Lang (I think it was
published in a recent USENIX LISA conference).
Also, if you want to consider fast alternatives to Logstash, then rsyslog
has a builtin support for Elasticsearch, and since it's written in C, it
can feed Elasticsearch much more efficiently. Here is a reference about
creating a basic configuration:
http://wiki.rsyslog.com/index.php/HOWTO:_rsyslog_%2B_elasticsearch
It is also fairly easy to configure rsyslog to store log data into
Elasticsearch in the way Kibana expects it to see (you have to set up the
same index name for rsyslog and Kibana, and also include couple of
mandatory fields in each log message).
kind regards,
risto
2013/5/2 John Zhang <[email protected]>
> Hi everyone,
>
> I am researching the big data security log management, such as Kibana +
> ElasticSearch + Logstash for my security log management, I need event
> correlation on this platform, i know SEC(
> http://simple-evcorr.sourceforge.net/) can do event correlation.
>
> Do you have any idea of SEC on such big data security log platform? Any
> experience, any reference?
>
> Any comment, advise will be highly appreciated!
>
> Thanks!
>
> John
>
>
> ------------------------------------------------------------------------------
> Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET
> Get 100% visibility into your production application - at no cost.
> Code-level diagnostics for performance bottlenecks with <2% overhead
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap1
> _______________________________________________
> Simple-evcorr-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
