[
https://issues.apache.org/jira/browse/FLUME-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156527#comment-13156527
]
[email protected] commented on FLUME-857:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2929/
-----------------------------------------------------------
Review request for jmhsieh.
Summary
-------
The CollectionSink create a chain of sinks on top of the underlying sink to
perform roll and retries upon failures. The stubborn append reopens the
downstream sink and retries the append once. The insistent append keeps
retrying the append.
Now if any errors (like network or HDFS datanodes shutdown) cause the
downstream sink to throw IOException on append, the stubborn append will try to
close that sink and reopen. If the close also throws an IOException, then sink
chain doesn't get properly closed. This results into the collectorsink skipping
the open and perpetually failing on append/close.
The fix is to ignore IOException in when closing downstream sink from the
stubborn append. Also set the close status correctly for decorators and roller.
This addresses bug FLUME-857.
https://issues.apache.org/jira/browse/FLUME-857
Diffs
-----
flume-core/src/main/java/com/cloudera/flume/core/EventSinkDecorator.java
6ab39ba
flume-core/src/main/java/com/cloudera/flume/handlers/debug/StubbornAppendSink.java
9436eb4
flume-core/src/main/java/com/cloudera/flume/handlers/rolling/RollSink.java
27302b1
flume-core/src/test/java/com/cloudera/flume/collector/TestCollectorSink.java
4f25f58
Diff: https://reviews.apache.org/r/2929/diff
Testing
-------
Added new test cases in TestCollectorSink. will run full regression test suite.
Thanks,
Prasad
> CollectorSink can get stuck into perpetual close() if the underlying sinks
> throws IOExceptions for both append and close.
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-857
> URL: https://issues.apache.org/jira/browse/FLUME-857
> Project: Flume
> Issue Type: Bug
> Affects Versions: v0.9.3, v0.9.4
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: v0.9.5
>
>
> The CollectionSink create a chain of sinks on top of the underlying sink to
> perform roll and retries upon failures. The stubborn append reopens the
> downstream sink and retries the append once. The insistent append keeps
> retrying the append.
> Now if any errors (like network or HDFS datanodes shutdown) cause the
> downstream sink to throw IOException on append, the stubborn append will try
> to close that sink and reopen. If the close also throws an IOException, then
> sink chain doesn't get properly closed. This results into the collectorsink
> skipping the open and perpetually failing on append/close.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
