[
https://issues.apache.org/jira/browse/FLUME-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488415#comment-13488415
]
Denny Ye commented on FLUME-1665:
---------------------------------
hi Hari, after taking with you, I checked and improved my test cases to track
missing and duplicate. I modified code as I said to write checkpoint even if
restart, part of tests also have failed. Thus, that might be not a problem in
your interpretation. In logically, I will raise an example to describe process
what we have met.
=====================================
Transaction batch size is 1000 in Sink, and SinkRunner was running.
When it was consuming at number#500 in this batch, Sink thread received
'interrupt' notification from modification of configuration file.
Then error happened and rollback to original of this batch, it was looks like
that this batch hasn't been consumed yet.
Actually, that 500 events has already been consumed into HDFS file before
interruption.
Next loop, when Flume restarted all components, it also would sending all
events at that batch to HDFS. Duplicate has happening.
=====================================
Following the example above, I removed the 'interrupt' from modification of
configuration file to ensure the consumption perfectly (100th lines at
SinkRunner.java). Because we have already set the stop single, the Sink thread
will exit at end of this batch. We don't need to interrupt that thread
immediately. In this case, all the test cases has passed normally.
What's the opinion of yours for my interpretation?
> Data from FileChannel will be duplicated when restarting configuration
> ----------------------------------------------------------------------
>
> Key: FLUME-1665
> URL: https://issues.apache.org/jira/browse/FLUME-1665
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0, v1.3.0
> Reporter: Denny Ye
> Labels: FileChannel
>
> When Flume process was running, I changed configuration property and Flume
> rebooted without process restarting. Events will be duplicated in next loop,
> also has been consumed before all components have stopped.
> I found the root cause. When FileChannel was stopping, it should save the
> 'inflightPuts' and 'inflightTakes' into disk for preparing in next loop.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
