hi Bond, I see now your point and understand better the task you would like to accomplish. Given that your goal is to feed data from one instance of sec to another, I would definitely recommend to avoid the 'logonly' action, since its use would force you to employ sec's internal log as a data transfer media between sec instances. However, sec's internal log might contain many irrelevant messages, and some of them might produce unexpected rule matches (e.g. consider a debug-level message about new synthetic event "Creating event 'Interface 10.1.1.1 down'" which might mistakenly match one of the regular expression patterns of the second sec instance). In order to prevent any side effects, I would recommend to use the 'write' action which is probably the simplest and most efficient way for transferring data between two local sec instances. With newer versions of sec, the 'write' action opens the file at first write and keeps it open during all future writes, closing it only when log rotation occurs. For rotating the logs, the USR2 signal has to be sent to the sec process which closes all output files previously opened by 'write' actions. Nevertheless, the first 'write' on a file after USR2 will reopen the file again, so you will have no data loss. The sec instance which is reading this file will detect the input log rotation, and will switch over to new input file instance automatically without any outside interference (such as sending a signal). The syslog based approach is more valuable if you would intend to run the 2nd sec instance at a central log server, but if your aim is to create a one-node setup, using local files and 'write' action is probably the best choice. Just don't forget to rotate the logs. kind regards, risto
2015-10-17 0:23 GMT+03:00 Bond Masuda <bond.mas...@jlbond.com>: > Thanks Risto for the advice. > > In my particular use case, I'm not using logonly to generate messages > for troubleshooting or observing the process flow of sec. I agree with > you that if that were the use case, then level 4 seems just right and > one can turn it on/off with --debug=<level>. In my use case, I'm also > generating messages that are meant to be processed by another SEC. > Granted, perhaps I am not using the right tool for the job. > > The overall architecture of what we're doing goes something like this: > > [data collectors: generates a lot of data very fast, many collectors > write to many logs] > > =>[1st SEC layer: processes multiple logs, does event correlation, > writes to single log] > > =>[2nd SEC layer: processes log from 1st SEC, makes decision to take > action or alert or both and does it] > > each layer has a trade-off between amount of work vs. urgency. the data > collectors do minimal processing, their goal is to not lose any data and > put it in the logs as fast as they can. the 1st SEC does correlation > across these multiple inputs, so some processing, but there's still a > bit of urgency here because there are limited time windows we have for > "look back" into the data. The 2nd SEC is not so urgent, but it need to > spend a lot more time when it is taking actions (alerts not so much) > automatically chasing down a problem and doing deeper analysis to fix > the problems. > > These are all done locally on each node, so no real need to forward to > any place; in fact, we're trying to minimize the use of centralized > processing because the model creates a lot of expensive scaling problems > for us. So, I naively thought, since SEC processes logs, I would use > 'logonly' to send those messages into the logs and the 2nd SEC would > process that. > > So, now I think I should leave the SEC logging alone for SEC purposes > only, and either use 'write' action or 'udgram /dev/log' to send > messages out to the 2nd SEC. If I don't have a need to forward the > messages anywhere else, do you think I should use 'write' action instead > of 'udgram /dev/log' ? Is one more expensive than the other? Are there > issues I have to consider since the output will be the input to the 2nd > SEC? > > Thanks as usual for your thoughtful advice and suggestions. > Bond > > On 10/16/2015 01:25 AM, Risto Vaarandi wrote: > > The logonly action has been largely used for tracking the event > > processing flow of sec, where a message is left only to sec's internal > > log without any more visible action. For such purposes, the level 4 > > has been an adequate setting for most environments (it's not a warning > > or error, but a notification). It would be somewhat pointless to set > > the level lower, since when you have sec's own logging verbosity set > > to 'debug', you will get a very detailed picture of sec's actions > > anyway, even without the use of 'logonly'. > > > > If you would like to implement a full blown logging scheme, I'd > > recommend to use syslog protocol for this purpose where the messages > > are not just written to sec's internal log but can be potentially > > forwarded to many receivers. Also, apart from just setting a severity, > > you can also take advantage of the facility and tag fields, in order > > to better indicate the type and name of the message source. > > > > If you are on a Linux host, then the 'udgram /dev/log <30>sec: this is > > a test' action produces a syslog message with the daemon-facility and > > info-severity, while the tag (program name) of the message is set to > > "sec" and the message text is "this is a test". This message is read > > by the local syslog server from /dev/log (or systemd-journald which > > forwards it to syslog daemon), and the further routing of the message > > depends on the configuration of the syslog server. > > > > regards, > > risto > > >
------------------------------------------------------------------------------
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
