[ https://issues.apache.org/jira/browse/NIFI-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493204#comment-15493204 ]
ASF GitHub Bot commented on NIFI-1170: -------------------------------------- Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/980#discussion_r78954211 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java --- @@ -117,31 +173,78 @@ .allowableValues(LOCATION_LOCAL, LOCATION_REMOTE) .defaultValue(LOCATION_LOCAL.getValue()) .build(); + static final PropertyDescriptor START_POSITION = new PropertyDescriptor.Builder() .name("Initial Start Position") - .description("When the Processor first begins to tail data, this property specifies where the Processor should begin reading data. Once data has been ingested from the file, " + .description("When the Processor first begins to tail data, this property specifies where the Processor should begin reading data. Once data has been ingested from a file, " + "the Processor will continue from the last point from which it has received data.") .allowableValues(START_BEGINNING_OF_TIME, START_CURRENT_FILE, START_CURRENT_TIME) .defaultValue(START_CURRENT_FILE.getValue()) .required(true) .build(); + static final PropertyDescriptor RECURSIVE = new PropertyDescriptor.Builder() + .name("tailfile-recursive-lookup") + .displayName("Recursive lookup") + .description("When using Multiple files mode, this property defines if files must be listed recursively or not" + + " in the base directory.") + .allowableValues("true", "false") + .defaultValue("true") + .required(true) + .build(); + + static final PropertyDescriptor ROLLING_STRATEGY = new PropertyDescriptor.Builder() + .name("tailfile-rolling-strategy") + .displayName("Rolling Strategy") + .description("Specifies if the files to tail have a fixed name or not.") + .required(true) + .allowableValues(FIXED_NAME, CHANGING_NAME) + .defaultValue(FIXED_NAME.getValue()) + .build(); + + static final PropertyDescriptor LOOKUP_FREQUENCY = new PropertyDescriptor.Builder() + .name("tailfile-lookup-frequency") + .displayName("Lookup frequency") + .description("Only used in Multiple files mode and Changing name rolling strategy, it specifies the minimum " + + "duration the processor will wait before listing again the files to tail.") + .required(false) + .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR) + .defaultValue("10 minutes") --- End diff -- @olegz @joewitt Just updated the PR with a default of 10 minutes for refresh frequency, and 24 hours for maximum age as a daily rollover pattern seems to be indeed the most common case. I updated the property descriptions and additional details page. I also changed a little bit the approach on the regex matching while listing files to tail. Previously I was only matching the regular expression against file name, now I am trying to match the regular expression against the path. It gives more flexibility to the users. I added an example in the additional details. And I squashed my commits. Let me know your thoughts. @trixpan, I understand your point but it sounds like an edge case, no? It would require to tail a very large number of files to get in such a situation. I'd be inclined to see this as an additional improvement in a separate PR if you agree. > TailFile "File to Tail" property should support Wildcards > --------------------------------------------------------- > > Key: NIFI-1170 > URL: https://issues.apache.org/jira/browse/NIFI-1170 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework > Affects Versions: 0.4.0 > Reporter: Andre > > Because of challenges around log rotation of high volume syslog and app > producers, it is customary to logging platform developers to promote file > variables based file names such as DynaFiles (rsyslog), Macros(syslog-ng)as > alternatives to getting SIGHUPs being sent to the syslog daemon upon every > file rotation. > (To certain extent, used even NiFi's has similar patterns, like for example, > when one uses Expression Language to set PutHDFS destination file). > The current TailFile strategy suggests rotation patterns like: > {code} > log_folder/app.log > log_folder/app.log.1 > log_folder/app.log.2 > log_folder/app.log.3 > {code} > It is possible to fool the system to accept wildcards by simply using a > strategy like: > {code} > log_folder/test1 > log_folder/server1 > log_folder/server2 > log_folder/server3 > {code} > And configure *Rolling Filename Pattern* to * but it feels like a hack, > rather than catering for an ever increasingly prevalent use case > (DynaFile/macros/etc). > It would be great if instead, TailFile had the ability to use wildcards on > File to Tail property -- This message was sent by Atlassian JIRA (v6.3.4#6332)