[
https://issues.apache.org/jira/browse/FLUME-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918390#comment-16918390
]
Ming Zhekai commented on FLUME-3334:
------------------------------------
how long does it take for OOM to occur after the program starts?
> TaildirSource tailFiles Map causing OOM when huge amount of files
> -----------------------------------------------------------------
>
> Key: FLUME-3334
> URL: https://issues.apache.org/jira/browse/FLUME-3334
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.7.0, 1.8.0, 1.9.0
> Reporter: ZhengHanyang
> Priority: Major
> Attachments: 20190511173448.png, 20190511173521.png
>
>
> I am using taildir source to monitor a log dir, about 100 new files per
> seconds, I set -xmx 2048m for flume, after 2 hours running, I get OOM error
> with "Failed writing positionFile".
> With a deap dive to heap dump file, i can see tailFiles occupies 1.7G memory,
> so I looked into the source code find that flume remember every file that
> match the file pattern in tailFiles, so can you add a property to filter file
> last modify time, default can be infinity, for example 30min, if the file
> modify time is 30min ago then remove it from tailFiles and do not monitor it.
> My logs come from real time transcation system and one file per transaction,
> file name is trace number, usually a transcation should be completed in
> several seconds, so most of the time there is no more update on the file, for
> some exception flume just read whole file and we can deal with it specially
> too.
> Please consider this scenario, thanks
> !20190511173448.png!
> !20190511173521.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
