Thanks for the explanation, now I understand why the performance penaty :D
Regarding the wildcards, solutions like fluentd or logstash rely on file
modification times. For example, fluentd has a /limit_recently_modified/
parameter that limit the watching files that the modification time is
within the specified time range. Logstash has /ignore_older: /when the
file input discovers a file that was last modified before the specified
timespan in seconds, the file is ignored. Now, I don't know how portable
this. I guess that in Unix/Linux find+spawn could be used for the same
effect which, as you say, is more generic and flexible (you could
include or ignore symlinks as your wish, for example).
The only caveat I see for the new action is the minimum 60 second delay
when used in a calendar rule. In the past we have struggled with running
rules at periodic, shorter intervals.
Regards,
El 20/03/18 a las 14:52, Risto Vaarandi escribió:
> hi Alberto,
>
> by periodic re-evaluation of file patterns I meant interpretation
> which happens in sec event processing loop, and is done after
> processing each event (or after very short time frames). This scheme
> is not very efficient and introduces computational overhead even for
> rule bases which do not need support for dynamic input filenames. In
> contrast, action based approach is lot more lightweight and does not
> introduce any additional overhead if actions are not used by the rule
> base.
>
> Naturally, the actions would be available for all rule types, and I
> would see Calendar rule as one of the most frequently employed rule
> with this action. For example, one could write:
>
> type=Calendar
> time=0 0 * * *
> desc=open the new input file /var/log/mylog-YYYY-MM-DD
> action=register_input /var/log/mylog-%{.year}-%{.mon}-%{.mday}
>
> The question if wildcards should be supported in file names is open at
> this point. I think for more complex cases (such as aperiodically
> created files with unpredictable names) wildcards alone would not
> help, and one should probably need some external scripting anyway. For
> example, consider files /var/log/mylog.<timestamp>, where timestamps
> are seconds since epoch when file was created:
>
> /var/log/mylog.1521550321
> /var/log/mylog.1521551548
> /var/log/mylog.1521552170
>
> Wildcard pattern can match all these files, but picking up the most
> recent one requires some scripting. If one always needs to monitor the
> last file, getting its name would require either checking for new file
> from Calendar rule (introduces a delay of up to 60 seconds, though). A
> more flexible alternative is running an external process with 'spawn'
> action which would issue a synthetic event about the appearance of new
> file (for example, "NEW_INPUT_/var/log/mylog.1521552585"). The
> synthetic event would then be matched by Single rule which opens the
> new file with 'register_input'. It would be straightforward to
> generate multiple synthetic events if several files need opening at
> once, and since the filenames can be entirely different without any
> commonalities, this approach is more generic than the use of
> wildcards. That's why I am (at least currently) slightly favoring
> non-wildcard approach :)
>
> kind regards,
> risto
>
> 2018-03-20 13:22 GMT+02:00 Alberto Corton <acor...@s21sec.com
> <mailto:acor...@s21sec.com>>:
>
> Hi,
>
> Glad to see that this is on the roadmap, I think is quite needed.
>
> I have a few questions about what you have mind. If periodic
> re-evaluation of the input file pattern is to be avoided for the
> reasons you provided I guess that these new actions are not
> intended to be run via a Calendar rule. Or are they?
>
> Also, would they work with glob patterns, as --input does?
>
> Regards,
>
>
> El 19/03/18 a las 21:31, Risto Vaarandi escribió:
>> hi all,
>>
>> in the past, there have been several questions in the mailing
>> list about support for input file names with variable parts (such
>> as timestamps). While this question can be addressed with setting
>> up a constant symbolic link to input file, this approach does not
>> work for platforms which don't support symbolic links. Also,
>> symbolic link involves the following caveat - if the link does
>> not exist already at sec startup, sec will not be able to open
>> the input file immediately.
>>
>> For the reasons above, I have been thinking of implementing
>> support for such file names in the next sec version. One way
>> would be re-evaluation of input file pattern (say,
>> /var/log/mylog*) after short time periods, but this approach is
>> neither elegant nor lightweight. However, one could also address
>> this problem with actions like 'register_input' and
>> 'unregister_input'. The first action would simply add the given
>> file name into the list of input files and attempt to open it
>> (unsuccessful open would still leave the file in the list of
>> inputs and it would continue be treated as a regular input file,
>> with attempts to reopen it if --reopen_timeout command line
>> option has been specified). Likewise, 'unregister_input' would
>> remove the input file from the list of inputs.
>>
>> Also, for implementing above two actions there are two possible
>> paths. Firstly, they could be allowed to disable/modify inputs
>> provided in command line with --input options. As the second
>> alternative, they could simply allow for creating temporary
>> inputs which do not overlap with the ones from command line (for
>> example, with --input=/var/log/messages command line option,
>> 'unregister_input /var/log/messages' would fail and produce a
>> warning message).
>>
>> A question to the mailing list members -- would you see the need
>> for such actions, and if yes, do you have some remarks and
>> suggestions about potential implementation?
>>
>> kind regards,
>> risto
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> <mailto:Simple-evcorr-users@lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>> <https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> <mailto:Simple-evcorr-users@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
> <https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users>
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
