hi Alberto,
by periodic re-evaluation of file patterns I meant interpretation which
happens in sec event processing loop, and is done after processing each
event (or after very short time frames). This scheme is not very efficient
and introduces computational overhead even for rule bases which do not need
support for dynamic input filenames. In contrast, action based approach is
lot more lightweight and does not introduce any additional overhead if
actions are not used by the rule base.
Naturally, the actions would be available for all rule types, and I would
see Calendar rule as one of the most frequently employed rule with this
action. For example, one could write:
type=Calendar
time=0 0 * * *
desc=open the new input file /var/log/mylog-YYYY-MM-DD
action=register_input /var/log/mylog-%{.year}-%{.mon}-%{.mday}
The question if wildcards should be supported in file names is open at this
point. I think for more complex cases (such as aperiodically created files
with unpredictable names) wildcards alone would not help, and one should
probably need some external scripting anyway. For example, consider files
/var/log/mylog.<timestamp>, where timestamps are seconds since epoch when
file was created:
/var/log/mylog.1521550321
/var/log/mylog.1521551548
/var/log/mylog.1521552170
Wildcard pattern can match all these files, but picking up the most recent
one requires some scripting. If one always needs to monitor the last file,
getting its name would require either checking for new file from Calendar
rule (introduces a delay of up to 60 seconds, though). A more flexible
alternative is running an external process with 'spawn' action which would
issue a synthetic event about the appearance of new file (for example,
"NEW_INPUT_/var/log/mylog.1521552585"). The synthetic event would then be
matched by Single rule which opens the new file with 'register_input'. It
would be straightforward to generate multiple synthetic events if several
files need opening at once, and since the filenames can be entirely
different without any commonalities, this approach is more generic than the
use of wildcards. That's why I am (at least currently) slightly favoring
non-wildcard approach :)
kind regards,
risto
2018-03-20 13:22 GMT+02:00 Alberto Corton <acor...@s21sec.com>:
> Hi,
> Glad to see that this is on the roadmap, I think is quite needed.
>
> I have a few questions about what you have mind. If periodic re-evaluation
> of the input file pattern is to be avoided for the reasons you provided I
> guess that these new actions are not intended to be run via a Calendar
> rule. Or are they?
>
> Also, would they work with glob patterns, as --input does?
>
> Regards,
>
> El 19/03/18 a las 21:31, Risto Vaarandi escribió:
>
> hi all,
>
> in the past, there have been several questions in the mailing list about
> support for input file names with variable parts (such as timestamps).
> While this question can be addressed with setting up a constant symbolic
> link to input file, this approach does not work for platforms which don't
> support symbolic links. Also, symbolic link involves the following caveat -
> if the link does not exist already at sec startup, sec will not be able to
> open the input file immediately.
>
> For the reasons above, I have been thinking of implementing support for
> such file names in the next sec version. One way would be re-evaluation of
> input file pattern (say, /var/log/mylog*) after short time periods, but
> this approach is neither elegant nor lightweight. However, one could also
> address this problem with actions like 'register_input' and
> 'unregister_input'. The first action would simply add the given file name
> into the list of input files and attempt to open it (unsuccessful open
> would still leave the file in the list of inputs and it would continue be
> treated as a regular input file, with attempts to reopen it if
> --reopen_timeout command line option has been specified). Likewise,
> 'unregister_input' would remove the input file from the list of inputs.
>
> Also, for implementing above two actions there are two possible paths.
> Firstly, they could be allowed to disable/modify inputs provided in command
> line with --input options. As the second alternative, they could simply
> allow for creating temporary inputs which do not overlap with the ones from
> command line (for example, with --input=/var/log/messages command line
> option, 'unregister_input /var/log/messages' would fail and produce a
> warning message).
>
> A question to the mailing list members -- would you see the need for such
> actions, and if yes, do you have some remarks and suggestions about
> potential implementation?
>
> kind regards,
> risto
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> Simple-evcorr-users mailing
> listSimple-evcorr-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
