[
https://issues.apache.org/jira/browse/FLUME-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon updated FLUME-2918:
--------------------------------
Comment: was deleted
(was: patch considered final, ready to be reviewed. contains:
-used java.nio.file.DirectoryStream to filter files
-made pattern match calculation optionally cached
-added junit tests
-added javadoc)
> TaildirSource is underperforming with huge parent directories
> -------------------------------------------------------------
>
> Key: FLUME-2918
> URL: https://issues.apache.org/jira/browse/FLUME-2918
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.6.0
> Reporter: Attila Simon
> Labels: performance
> Fix For: v1.7.0
>
> Attachments: FLUME-2918-2.patch, PerfHugeDir.java, perftest.png,
> profiling_after.png, profiling_before.png, test.csv
>
>
> TailDir source cause high cpu utilization, when large amount of file is
> sitting in the target directory. File pattern matches only a single file, but
> the parent directory contains about 50,000 other file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
