Hi Steve,
I'm a little confused. Why would you need RabbitMQ to insert only 40
events/sec?
My syslog tool (LogZilla) does 15k events/sec into MySQL without using
Rabbit.
We are working now (using RabbitMQ) on getting that number to around 120k
eps.
P.S.
I would love it if SEC could process at that rate :-)
______________________________________________________________
Clayton Dukes
______________________________________________________________
On Mon, Feb 11, 2013 at 3:12 PM, Busko, Steve <[email protected]> wrote:
> Hello Ristro,
>
> While I don't have specific implementation feedback on this topic, I did
> want to throw in that, like Gary, we are also using RabbitMQ and would like
> to see sec integrate in some way possibly affecting this thread in part.
>
> Initially we used pipes, but found them not robust enough. So we tried
> RabbitMQ and have been much happier with the flexibility and control. We
> have been running it for around 6 months in a development environment
> taking in around low 40s msgs/sec from a single syslog source and outputs
> slightly less than half that into a single DB. In our case we didn't do
> anything fancy to integrate, just inserted the MQ perl code/functionality
> directly into sec and created an additional "queue" command to use vs. a
> write.
>
> So we would like to support both the use of RabbitMQ and the consideration
> of integrating it (or as a plug-in) with sec.
>
> Regards,
> Steve Busko | NOC Tools Manager
>
> > -----Original Message-----
> > From: Risto Vaarandi [mailto:[email protected]]
> > Sent: Wednesday, February 06, 2013 12:38 PM
> > To: Simple-evcorr-users
> > Subject: [Simple-evcorr-users] extending output types
> >
> > hi all,
> >
> > as you know, currently there are number of outputs which are not
> > directly supported by sec. For example, with 'pipe' action one can feed
> > only one event to an external program (with most programs exiting after
> > the 'pipe' action closes the pipe). Also, the 'write' action opens a
> > given file before each access and closes it afterwards. While this makes
> > output file rotation worry-free, it is not very efficient. Finally, sec
> > does not directly support various sockets as output (for example, it is
> > not possible to connect directly to a syslog server on another host and
> > issue syslog events without any intermediaries).
> >
> > Therefore, would there be any interest to an output action which would
> > leave the output open/running/connected, so that the next access of the
> > same output would not have to do reopen/restart/connect again? This
> > action would have to support a number of output types, e.g.,
> >
> > write2 udp myserver:514 This is my event
> > write2 tcp myserver:514 Another event
> > write2 file /var/log/myevents Third event
> > write2 program /bin/logger -p user.notice Fourth event
> >
> > It is probably easy to see that with sockets and files it is not that
> > hard to distinguish them from each other, since the filename or server
> > name plus port number can serve as unique identifiers. With command
> > lines, however, it is somewhat more difficult, since an extra space or
> > different ordering of options makes two command lines different, even if
> > their effect is exactly the same. Also, since sec uses positional
> > parameters, there is an issue with parsing, although this could be
> > solved with the use of (). In addition, one workaround for both issues
> > would be storing command line in an action list variable.
> > The action outlines above would also need an output rotation mechanism,
> > but this could be implemented with adding this to SIGUSR2 handler
> > (closing all currently open outputs).
> >
> > Would the action above be of interest, and are there any other ideas for
> > handling these situations?
> >
> > kind regards,
> > risto
> >
> >
> ------------------------------------------------------------------------------
> > Free Next-Gen Firewall Hardware Offer
> > Buy your Sophos next-gen firewall before the end March 2013
> > and get the hardware for free! Learn more.
> > http://p.sf.net/sfu/sophos-d2d-feb
> > _______________________________________________
> > Simple-evcorr-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Simple-evcorr-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
