belvey created FLUME-3325:
-----------------------------
Summary: TaildirSource causing oom when tailing a lot of files,
where some file is writing frequently but the others are idle.
Key: FLUME-3325
URL: https://issues.apache.org/jira/browse/FLUME-3325
Project: Flume
Issue Type: Bug
Affects Versions: 1.8.0, 1.7.0
Reporter: belvey
I am using taildir source to read file to kafka, but recently i am facing flume
agent oom error.
With a deap dive to OOM dump file, i found 'List<Long> idleInodes ' of
TaildirSource class occupied most of the memory. so i looked into the source
code of TaildirSource, and found that when tailing a lot of files where some
file is writing frequently but the others are idle can cause the idleInodes
List increasing infinitely.
Relative code is shown below:
{code:java}
public Status process() {
Status status = Status.BACKOFF;
try {
existingInodes.clear();
existingInodes.addAll(reader.updateTailFiles());
for (long inode : existingInodes) {
TailFile tf = reader.getTailFiles().get(inode);
if (tf.needTail()) {
boolean hasMoreLines = tailFileProcess(tf, true);
if (hasMoreLines) {
status = Status.READY;
}
}
}
closeTailFiles();
} catch (Throwable t) {
logger.error("Unable to tail files", t);
sourceCounter.incrementEventReadFail();
status = Status.BACKOFF;
}
return status;
}
{code}
when one file is frequently writing, this thread is blocked at this line , and
will not execute closeTailFiles() method to clear the idleInodes List.
{code:java}
boolean hasMoreLines = tailFileProcess(tf, true);{code}
while the other files are idle, and there is anthor thread checking for idle
file every 5 sec by default and if one file is always idle ,it may be add to
list multi times, so this list can grow infinitely.
I will add a pull request to it later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
