[
https://issues.apache.org/jira/browse/FLUME-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001644#comment-16001644
]
Hudson commented on FLUME-3085:
-------------------------------
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #247 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/247/])
FLUME-3085. HDFS Sink can skip flushing some BucketWriters, might lead (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ed433ae1b12d40117ca3dca2ffd57389984ede72])
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
> HDFS Sink can skip flushing some BucketWriters, might lead to data loss
> -----------------------------------------------------------------------
>
> Key: FLUME-3085
> URL: https://issues.apache.org/jira/browse/FLUME-3085
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.7.0
> Reporter: Denes Arvay
> Assignee: Denes Arvay
> Priority: Critical
> Fix For: 1.8.0
>
>
> The {{HDFSEventSink.process()}} is already prepared for a rare race
> condition, namely when the BucketWriter acquired in [line
> 389|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L389]
> gets closed by an other thread (e.g. because the {{idleTimeout}} or the
> {{rollInterval}}) before the {{append()}} is called in [line
> 406|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L406].
> If this is the case the {{BucketWriter.append()}} call throws a
> {{BucketClosedException}} and the sink creates a new {{BucketWriter}}
> instance and appends to it.
> But this newly created instance won't be added to the {{writers}} list, which
> means that it won't be flushed after the processing loop finished:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429
> This has multiple consequences:
> - unflushed data might get lost
> - the {{BucketWriter}}'s {{idleAction}} won't be scheduled
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450),
> which means that it won't be closed nor renamed if the idle timeout is the
> only trigger for closing the file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
