hi Alberto,
thanks for the suggestions! I will also look into other tools such as
logstash and consider if something can be borrowed from their approach.
kind regards,
risto
2018-03-20 19:40 GMT+02:00 Alberto Corton <acor...@s21sec.com>:
> Thanks for the explanation, now I understand why the performance penaty :D
>
> Regarding the wildcards, solutions like fluentd or logstash rely on file
> modification times. For example, fluentd has a *limit_recently_modified*
> parameter that limit the watching files that the modification time is
> within the specified time range. Logstash has *ignore_older: *when the
> file input discovers a file that was last modified before the specified
> timespan in seconds, the file is ignored. Now, I don't know how portable
> this. I guess that in Unix/Linux find+spawn could be used for the same
> effect which, as you say, is more generic and flexible (you could include
> or ignore symlinks as your wish, for example).
>
> The only caveat I see for the new action is the minimum 60 second delay
> when used in a calendar rule. In the past we have struggled with running
> rules at periodic, shorter intervals.
>
> Regards,
>
> El 20/03/18 a las 14:52, Risto Vaarandi escribió:
>
> hi Alberto,
>
> by periodic re-evaluation of file patterns I meant interpretation which
> happens in sec event processing loop, and is done after processing each
> event (or after very short time frames). This scheme is not very efficient
> and introduces computational overhead even for rule bases which do not need
> support for dynamic input filenames. In contrast, action based approach is
> lot more lightweight and does not introduce any additional overhead if
> actions are not used by the rule base.
>
> Naturally, the actions would be available for all rule types, and I would
> see Calendar rule as one of the most frequently employed rule with this
> action. For example, one could write:
>
> type=Calendar
> time=0 0 * * *
> desc=open the new input file /var/log/mylog-YYYY-MM-DD
> action=register_input /var/log/mylog-%{.year}-%{.mon}-%{.mday}
>
> The question if wildcards should be supported in file names is open at
> this point. I think for more complex cases (such as aperiodically created
> files with unpredictable names) wildcards alone would not help, and one
> should probably need some external scripting anyway. For example, consider
> files /var/log/mylog.<timestamp>, where timestamps are seconds since epoch
> when file was created:
>
> /var/log/mylog.1521550321
> /var/log/mylog.1521551548
> /var/log/mylog.1521552170
>
> Wildcard pattern can match all these files, but picking up the most recent
> one requires some scripting. If one always needs to monitor the last file,
> getting its name would require either checking for new file from Calendar
> rule (introduces a delay of up to 60 seconds, though). A more flexible
> alternative is running an external process with 'spawn' action which would
> issue a synthetic event about the appearance of new file (for example,
> "NEW_INPUT_/var/log/mylog.1521552585"). The synthetic event would then be
> matched by Single rule which opens the new file with 'register_input'. It
> would be straightforward to generate multiple synthetic events if several
> files need opening at once, and since the filenames can be entirely
> different without any commonalities, this approach is more generic than the
> use of wildcards. That's why I am (at least currently) slightly favoring
> non-wildcard approach :)
>
> kind regards,
> risto
>
> 2018-03-20 13:22 GMT+02:00 Alberto Corton <acor...@s21sec.com>:
>
>> Hi,
>> Glad to see that this is on the roadmap, I think is quite needed.
>>
>> I have a few questions about what you have mind. If periodic
>> re-evaluation of the input file pattern is to be avoided for the reasons
>> you provided I guess that these new actions are not intended to be run via
>> a Calendar rule. Or are they?
>>
>> Also, would they work with glob patterns, as --input does?
>>
>> Regards,
>>
>> El 19/03/18 a las 21:31, Risto Vaarandi escribió:
>>
>> hi all,
>>
>> in the past, there have been several questions in the mailing list about
>> support for input file names with variable parts (such as timestamps).
>> While this question can be addressed with setting up a constant symbolic
>> link to input file, this approach does not work for platforms which don't
>> support symbolic links. Also, symbolic link involves the following caveat -
>> if the link does not exist already at sec startup, sec will not be able to
>> open the input file immediately.
>>
>> For the reasons above, I have been thinking of implementing support for
>> such file names in the next sec version. One way would be re-evaluation of
>> input file pattern (say, /var/log/mylog*) after short time periods, but
>> this approach is neither elegant nor lightweight. However, one could also
>> address this problem with actions like 'register_input' and
>> 'unregister_input'. The first action would simply add the given file name
>> into the list of input files and attempt to open it (unsuccessful open
>> would still leave the file in the list of inputs and it would continue be
>> treated as a regular input file, with attempts to reopen it if
>> --reopen_timeout command line option has been specified). Likewise,
>> 'unregister_input' would remove the input file from the list of inputs.
>>
>> Also, for implementing above two actions there are two possible paths.
>> Firstly, they could be allowed to disable/modify inputs provided in command
>> line with --input options. As the second alternative, they could simply
>> allow for creating temporary inputs which do not overlap with the ones from
>> command line (for example, with --input=/var/log/messages command line
>> option, 'unregister_input /var/log/messages' would fail and produce a
>> warning message).
>>
>> A question to the mailing list members -- would you see the need for such
>> actions, and if yes, do you have some remarks and suggestions about
>> potential implementation?
>>
>> kind regards,
>> risto
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>>
>> _______________________________________________
>> Simple-evcorr-users mailing
>> listSimple-evcorr-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
