Re: Review Request 50378: FLUME-2960: Support Wildcards in directoryname in TaildirSource
> On 三月 3, 2017, 2:44 p.m., Denes Arvay wrote: > > First of all thank you for your effort by providing and constantly > > improving this patch. > > I have two comments, both of them were already mentioned before, I just > > would like to give them more emphasis: > > - please think a little bit on the `walkFileTree` optimization by > > implementing the `preVisitDirectory` (I'm not 100% sure that it's feasible, > > though) and also please move the `dirMatcher` and `fileMatcher` to the > > enclosing class. > > - Following the symlinks should be added as well to keep the backward > > compatibility. (It's still an open issue from the initial code review > > rounds) OK, I will take into account the two issues you mentioned. - qiao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50378/#review167826 --- On 七月 30, 2016, 5:33 a.m., qiao wen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50378/ > --- > > (Updated 七月 30, 2016, 5:33 a.m.) > > > Review request for Flume. > > > Repository: flume-git > > > Description > --- > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/*/log.* > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst 3f08d8b > > flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirMatcher.java > ad9f720 > > flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirMatcher.java > c341054 > > flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java > 097ee0b > > > Diff: https://reviews.apache.org/r/50378/diff/5/ > > > Testing > --- > > All tests in TestTaildirSource passed. > > > Thanks, > > qiao wen > >
[jira] [Comment Edited] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889507#comment-15889507 ] tinawenqiao edited comment on FLUME-2960 at 3/3/17 3:12 PM: Hi, [~mpercy], [~sati], [~bessbd],[~denes] About the consistence of regex or glob patterns in the path, I have the following strategies. Please vote for which is better or give me some suggestions. 1. Unify regex. Whether it is a directory name or a file name, regex is only supported. 2. Unify glob. Whether it is a directory name or a file name, wildcard is only supported. 3. Wildcard is only supported in directory name, and regex is only supported in file name. 4. Add a flag config. It's up to the user. The user can choose which to use. was (Author: wenqiao): Hi, [~mpercy], [~sati], [~bessbd],[~denes] About the consistence of regex or glob patterns in the path, I have the following three strategies. Please vote for which is better or give me some suggestions. 1. Unify regex. Whether it is a directory name or a file name, regex is only supported. 2. Unify glob. Whether it is a directory name or a file name, wildcard is only supported. 3. Wildcard is only supported in directory name, and regex is only supported in file name. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.8.0 > > Attachments: FLUME-2960_1.patch, FLUME-2960_2.patch, > FLUME-2960_3.patch, FLUME-2960_4.patch, FLUME-2960_5.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894539#comment-15894539 ] tinawenqiao commented on FLUME-2960: Thanks for your idea. I add another strategy. Choose which to use is decided by the user. It's more flexible. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.8.0 > > Attachments: FLUME-2960_1.patch, FLUME-2960_2.patch, > FLUME-2960_3.patch, FLUME-2960_4.patch, FLUME-2960_5.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889507#comment-15889507 ] tinawenqiao edited comment on FLUME-2960 at 3/3/17 3:13 PM: Hi, [~mpercy], [~sati], [~bessbd],[~denes] About the consistence of regex or glob patterns in the path, I have the following strategies. Please vote for which is better or give me some suggestions. 1. Unify regex. Whether it is a directory name or a file name, regex is only supported. 2. Unify glob. Whether it is a directory name or a file name, wildcard is only supported. 3. Wildcard is only supported in directory name, and regex is only supported in file name. 4. Add a flag config. It's up to the user. The user can choose which to use. If set to regex, unify regex. If set to glob, unify glob. was (Author: wenqiao): Hi, [~mpercy], [~sati], [~bessbd],[~denes] About the consistence of regex or glob patterns in the path, I have the following strategies. Please vote for which is better or give me some suggestions. 1. Unify regex. Whether it is a directory name or a file name, regex is only supported. 2. Unify glob. Whether it is a directory name or a file name, wildcard is only supported. 3. Wildcard is only supported in directory name, and regex is only supported in file name. 4. Add a flag config. It's up to the user. The user can choose which to use. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.8.0 > > Attachments: FLUME-2960_1.patch, FLUME-2960_2.patch, > FLUME-2960_3.patch, FLUME-2960_4.patch, FLUME-2960_5.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894488#comment-15894488 ] Denes Arvay commented on FLUME-2960: Hi [~wenqiao], I think it's a good idea to unify the patterns, and because the current implementation supports regexes in the filename I'd go with it to stay backward compatible. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.8.0 > > Attachments: FLUME-2960_1.patch, FLUME-2960_2.patch, > FLUME-2960_3.patch, FLUME-2960_4.patch, FLUME-2960_5.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 50378: FLUME-2960: Support Wildcards in directoryname in TaildirSource
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50378/#review167826 --- First of all thank you for your effort by providing and constantly improving this patch. I have two comments, both of them were already mentioned before, I just would like to give them more emphasis: - please think a little bit on the `walkFileTree` optimization by implementing the `preVisitDirectory` (I'm not 100% sure that it's feasible, though) and also please move the `dirMatcher` and `fileMatcher` to the enclosing class. - Following the symlinks should be added as well to keep the backward compatibility. (It's still an open issue from the initial code review rounds) - Denes Arvay On July 30, 2016, 7:33 a.m., qiao wen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50378/ > --- > > (Updated July 30, 2016, 7:33 a.m.) > > > Review request for Flume. > > > Repository: flume-git > > > Description > --- > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/*/log.* > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst 3f08d8b > > flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirMatcher.java > ad9f720 > > flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirMatcher.java > c341054 > > flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java > 097ee0b > > > Diff: https://reviews.apache.org/r/50378/diff/5/ > > > Testing > --- > > All tests in TestTaildirSource passed. > > > Thanks, > > qiao wen > >
[jira] [Commented] (FLUME-3064) NullPointerException when communication issue with kafka server
[ https://issues.apache.org/jira/browse/FLUME-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894330#comment-15894330 ] Tristan Stevens commented on FLUME-3064: LGTM. Ship It! [~jholoman] Are you able to review and commit this? > NullPointerException when communication issue with kafka server > --- > > Key: FLUME-3064 > URL: https://issues.apache.org/jira/browse/FLUME-3064 > Project: Flume > Issue Type: Bug > Components: Kafka Channel, Sinks+Sources >Affects Versions: v1.7.0 > Environment: logging level must be DEBUG >Reporter: Laszlo Hornyak >Priority: Minor > > When flume has communication issues with kafka and the logging level is set > to DEBUG, additionally to the log of the communication error an NPE will be > thrown. This happens because kafka callback metadata parameter [can be > null|https://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/Callback.html#onCompletion(org.apache.kafka.clients.producer.RecordMetadata,%20java.lang.Exception)], > this is not handled, but it is only evaluated if the log level is DEBUG. > The problem affects the kafka source and the channel. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLUME-3050) add error stats to monitor URL
[
https://issues.apache.org/jira/browse/FLUME-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894321#comment-15894321
]
Tristan Stevens commented on FLUME-3050:
Hi [~yuvalif]
I think what you are asking here is for new counters to be added, rather than
for existing counters not being present on the monitoring URL.
At the moment, most counters are defined for Source, Sink and Channel as
supertypes, and there are only a few specific overrides for counters
(KafkaSink, KafkaChannel and KafkaSource are the only ones that I can see).
You can deduce the failure rate by comparing the attempts with the successes,
but otherwise you'd be in the realms of adding specific counters all over the
sources and sinks that you've specified. It's certainly possible, but it would
require a fair amount of re-working to get right.
What you you think?
By the way, at the very least we should re-name the JIRA to something like "Add
Counters for error conditions"
> add error stats to monitor URL
> --
>
> Key: FLUME-3050
> URL: https://issues.apache.org/jira/browse/FLUME-3050
> Project: Flume
> Issue Type: Improvement
> Components: Channel, Shell, Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Yuval Lifshitz
> Labels: features
>
> currently error counters are not present when getting stats. for example:
> {code}
> > curl http://my-flume-host:4/metrics
> {"SINK.k1":{"ConnectionCreatedCount":"1","ConnectionClosedCount":"0","Type":"SINK","BatchCompleteCount":"0","BatchEmptyCount":"4","EventDrainAttemptCount":"10","StartTime":"1485348138992","EventDrainSuccessCount":"10","BatchUnderflowCount":"1","StopTime":"0","ConnectionFailedCount":"0"},"CHANNEL.c1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"10","EventTakeAttemptCount":"15","StartTime":"1485348138990","EventPutAttemptCount":"10","EventPutSuccessCount":"10","StopTime":"0"},"SOURCE.r1":{"EventReceivedCount":"10","AppendBatchAcceptedCount":"0","Type":"SOURCE","AppendReceivedCount":"0","EventAcceptedCount":"10","StartTime":"1485348138993","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"0","StopTime":"0"}}
> {code}
> return only "good" stats for source, channel and sink.
> to get error you need to look into the log file. this makes it hard to
> integrate flume into automatic monitoring systems, NMS etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889507#comment-15889507 ] tinawenqiao edited comment on FLUME-2960 at 3/3/17 12:05 PM: - Hi, [~mpercy], [~sati], [~bessbd],[~denes] About the consistence of regex or glob patterns in the path, I have the following three strategies. Please vote for which is better or give me some suggestions. 1. Unify regex. Whether it is a directory name or a file name, regex is only supported. 2. Unify glob. Whether it is a directory name or a file name, wildcard is only supported. 3. Wildcard is only supported in directory name, and regex is only supported in file name. was (Author: wenqiao): Hi, [~mpercy][~sati][~bessbd][~denes] About the consistence of regex or glob patterns in the path, I have the following three strategies. Please vote for which is better or give me some suggestions. 1. Unify regex. Whether it is a directory name or a file name, regex is only supported. 2. Unify glob. Whether it is a directory name or a file name, wildcard is only supported. 3. Wildcard is only supported in directory name, and regex is only supported in file name. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.8.0 > > Attachments: FLUME-2960_1.patch, FLUME-2960_2.patch, > FLUME-2960_3.patch, FLUME-2960_4.patch, FLUME-2960_5.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 57278: Enable multiple monitoring types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57278/ --- Review request for Flume. Repository: flume-git Description --- The current flume can specify only one monitoring type. It is nice to enable multiple monitoring types such as HTTP, GANGLIA and other custom monitoring types. Diffs - flume-ng-node/src/main/java/org/apache/flume/node/Application.java d6d92f0 flume-ng-node/src/test/java/org/apache/flume/node/TestApplication.java affbd8c Diff: https://reviews.apache.org/r/57278/diff/1/ Testing --- All tests passed Thanks, Satoshi Iijima
[jira] [Updated] (FLUME-3065) Enable multiple monitoring types
[
https://issues.apache.org/jira/browse/FLUME-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satoshi Iijima updated FLUME-3065:
--
Attachment: FLUME-3065_2.patch
Updated patch and added test code.
I applied this patch to my dev environment and tested it.
> Enable multiple monitoring types
>
>
> Key: FLUME-3065
> URL: https://issues.apache.org/jira/browse/FLUME-3065
> Project: Flume
> Issue Type: Improvement
> Components: Node
>Affects Versions: v1.7.0
>Reporter: Satoshi Iijima
> Attachments: FLUME-3065_2.patch, FLUME-3065.patch
>
>
> The current flume can specify only one monitoring type.
> It is nice to enable multiple monitoring type such as HTTP, GANGLIA and other
> custom monitoring types.
> {code}
> # JAVA_OPTS example
> -Dflume.monitoring.type=HTTP,GANGLIA,org.apache.flume.instrumentation.customMetricsServer
> -Dflume.monitoring.port=34545
> -Dflume.monitoring.hosts=10.129.0.01:8649 -Dflume.monitoring.pollFrequency=180
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
