Clayton,
We did not need RabbitMQ for helping with the quantity of inserts. I only
provided the number show how much we were pushing through it in case anyone
wanted a reference. We opted to add MQ for the ability to control and monitor
the data via its queues instead of having other parts of the system deal with
it.
One case we enabled an ACK on the output queue (from SEC to DB) so that we
don't lose events should the script handling the DB insertion fail. MQ would
queue up the non-ACK'd entries (which can also be easily measured) until the
issue resolved. It's use all depends on your needs, we just found RabbitMQ
filled our needs more than the raw pipe.
- Steve
From: Clayton Dukes [mailto:[email protected]]
Sent: Monday, February 11, 2013 4:05 PM
To: Busko, Steve
Cc: Risto Vaarandi; Simple-evcorr-users
Subject: Re: [Simple-evcorr-users] extending output types (RabbitMQ)
Hi Steve,
I'm a little confused. Why would you need RabbitMQ to insert only 40 events/sec?
My syslog tool (LogZilla) does 15k events/sec into MySQL without using Rabbit.
We are working now (using RabbitMQ) on getting that number to around 120k eps.
P.S.
I would love it if SEC could process at that rate :-)
______________________________________________________________
Clayton Dukes
______________________________________________________________
On Mon, Feb 11, 2013 at 3:12 PM, Busko, Steve
<[email protected]<mailto:[email protected]>> wrote:
Hello Ristro,
While I don't have specific implementation feedback on this topic, I did want
to throw in that, like Gary, we are also using RabbitMQ and would like to see
sec integrate in some way possibly affecting this thread in part.
Initially we used pipes, but found them not robust enough. So we tried RabbitMQ
and have been much happier with the flexibility and control. We have been
running it for around 6 months in a development environment taking in around
low 40s msgs/sec from a single syslog source and outputs slightly less than
half that into a single DB. In our case we didn't do anything fancy to
integrate, just inserted the MQ perl code/functionality directly into sec and
created an additional "queue" command to use vs. a write.
So we would like to support both the use of RabbitMQ and the consideration of
integrating it (or as a plug-in) with sec.
Regards,
Steve Busko | NOC Tools Manager
> -----Original Message-----
> From: Risto Vaarandi
> [mailto:[email protected]<mailto:[email protected]>]
> Sent: Wednesday, February 06, 2013 12:38 PM
> To: Simple-evcorr-users
> Subject: [Simple-evcorr-users] extending output types
>
> hi all,
>
> as you know, currently there are number of outputs which are not
> directly supported by sec. For example, with 'pipe' action one can feed
> only one event to an external program (with most programs exiting after
> the 'pipe' action closes the pipe). Also, the 'write' action opens a
> given file before each access and closes it afterwards. While this makes
> output file rotation worry-free, it is not very efficient. Finally, sec
> does not directly support various sockets as output (for example, it is
> not possible to connect directly to a syslog server on another host and
> issue syslog events without any intermediaries).
>
> Therefore, would there be any interest to an output action which would
> leave the output open/running/connected, so that the next access of the
> same output would not have to do reopen/restart/connect again? This
> action would have to support a number of output types, e.g.,
>
> write2 udp myserver:514 This is my event
> write2 tcp myserver:514 Another event
> write2 file /var/log/myevents Third event
> write2 program /bin/logger -p user.notice Fourth event
>
> It is probably easy to see that with sockets and files it is not that
> hard to distinguish them from each other, since the filename or server
> name plus port number can serve as unique identifiers. With command
> lines, however, it is somewhat more difficult, since an extra space or
> different ordering of options makes two command lines different, even if
> their effect is exactly the same. Also, since sec uses positional
> parameters, there is an issue with parsing, although this could be
> solved with the use of (). In addition, one workaround for both issues
> would be storing command line in an action list variable.
> The action outlines above would also need an output rotation mechanism,
> but this could be implemented with adding this to SIGUSR2 handler
> (closing all currently open outputs).
>
> Would the action above be of interest, and are there any other ideas for
> handling these situations?
>
> kind regards,
> risto
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Simple-evcorr-users mailing list
> [email protected]<mailto:[email protected]>
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Simple-evcorr-users mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
