[
https://issues.apache.org/jira/browse/FLUME-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
belvey updated FLUME-3325:
--------------------------
Description:
I am using taildir source to read file to kafka, but recently i am facing flume
agent oom error.
With a deap dive to OOM dump file, i found 'List<Long> idleInodes ' of
TaildirSource class occupied most of the memory. so i looked into the source
code of TaildirSource, and found that when tailing a lot of files where some
file is writing frequently but the others are idle can cause the idleInodes
List increasing infinitely.
Relative code is shown below:
{code:java}
public Status process() {
Status status = Status.BACKOFF;
try {
existingInodes.clear();
existingInodes.addAll(reader.updateTailFiles());
for (long inode : existingInodes) {
TailFile tf = reader.getTailFiles().get(inode);
if (tf.needTail()) {
boolean hasMoreLines = tailFileProcess(tf, true);
if (hasMoreLines) {
status = Status.READY;
}
}
}
closeTailFiles();
} catch (Throwable t) {
logger.error("Unable to tail files", t);
sourceCounter.incrementEventReadFail();
status = Status.BACKOFF;
}
return status;
}
{code}
when one file is frequently writing, this thread is blocked at this line , and
will not execute closeTailFiles() method to clear the idleInodes List.
{code:java}
boolean hasMoreLines = tailFileProcess(tf, true);{code}
while the other files are idle, and there is another thread checking for idle
file every 5 sec by default and if one file is always idle ,it may be add to
list multi times, so this list can grow infinitely.
I will add a pull request to it later.
was:
I am using taildir source to read file to kafka, but recently i am facing flume
agent oom error.
With a deap dive to OOM dump file, i found 'List<Long> idleInodes ' of
TaildirSource class occupied most of the memory. so i looked into the source
code of TaildirSource, and found that when tailing a lot of files where some
file is writing frequently but the others are idle can cause the idleInodes
List increasing infinitely.
Relative code is shown below:
{code:java}
public Status process() {
Status status = Status.BACKOFF;
try {
existingInodes.clear();
existingInodes.addAll(reader.updateTailFiles());
for (long inode : existingInodes) {
TailFile tf = reader.getTailFiles().get(inode);
if (tf.needTail()) {
boolean hasMoreLines = tailFileProcess(tf, true);
if (hasMoreLines) {
status = Status.READY;
}
}
}
closeTailFiles();
} catch (Throwable t) {
logger.error("Unable to tail files", t);
sourceCounter.incrementEventReadFail();
status = Status.BACKOFF;
}
return status;
}
{code}
when one file is frequently writing, this thread is blocked at this line , and
will not execute closeTailFiles() method to clear the idleInodes List.
{code:java}
boolean hasMoreLines = tailFileProcess(tf, true);{code}
while the other files are idle, and there is anthor thread checking for idle
file every 5 sec by default and if one file is always idle ,it may be add to
list multi times, so this list can grow infinitely.
I will add a pull request to it later.
> TaildirSource causing oom when tailing a lot of files, where some file is
> writing frequently but the others are idle.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-3325
> URL: https://issues.apache.org/jira/browse/FLUME-3325
> Project: Flume
> Issue Type: Bug
> Affects Versions: 1.8.0, 1.9.0
> Reporter: belvey
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I am using taildir source to read file to kafka, but recently i am facing
> flume agent oom error.
> With a deap dive to OOM dump file, i found 'List<Long> idleInodes ' of
> TaildirSource class occupied most of the memory. so i looked into the source
> code of TaildirSource, and found that when tailing a lot of files where some
> file is writing frequently but the others are idle can cause the idleInodes
> List increasing infinitely.
> Relative code is shown below:
>
> {code:java}
> public Status process() {
> Status status = Status.BACKOFF;
> try {
> existingInodes.clear();
> existingInodes.addAll(reader.updateTailFiles());
> for (long inode : existingInodes) {
> TailFile tf = reader.getTailFiles().get(inode);
> if (tf.needTail()) {
> boolean hasMoreLines = tailFileProcess(tf, true);
> if (hasMoreLines) {
> status = Status.READY;
> }
> }
> }
> closeTailFiles();
> } catch (Throwable t) {
> logger.error("Unable to tail files", t);
> sourceCounter.incrementEventReadFail();
> status = Status.BACKOFF;
> }
> return status;
> }
> {code}
> when one file is frequently writing, this thread is blocked at this line ,
> and will not execute closeTailFiles() method to clear the idleInodes List.
> {code:java}
> boolean hasMoreLines = tailFileProcess(tf, true);{code}
>
> while the other files are idle, and there is another thread checking for idle
> file every 5 sec by default and if one file is always idle ,it may be add to
> list multi times, so this list can grow infinitely.
> I will add a pull request to it later.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
