[
https://issues.apache.org/jira/browse/FLUME-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285519#comment-13285519
]
Jarek Jarcec Cecho commented on FLUME-1045:
-------------------------------------------
Hi Sharad,
your understanding of FLUME-1201 is correct. We closed that discussion with
result that entire purpose of the idea is just to support this very specific
use case. Which is the reason why we've opened follow up FLUME-1227.
Thank you for your feedback, it helped me understand your orignal proposal. I
believe that the proposal make sense. I'm just worried here that current
channel interface do not expose number of stored entries. So questions like "is
memory channel full" or "do file channel contains less than memory channel
capacity * (some threshold say 0.2)" might not work.
Jarcec
> Proposal to support disk based spooling
> ---------------------------------------
>
> Key: FLUME-1045
> URL: https://issues.apache.org/jira/browse/FLUME-1045
> Project: Flume
> Issue Type: New Feature
> Affects Versions: v1.0.0
> Reporter: Inder SIngh
> Priority: Minor
> Labels: patch
> Attachments: FLUME-1045-1.patch, FLUME-1045-2.patch
>
>
> 1. Problem Description
> A sink being unavailable at any stage in the pipeline causes it to back-off
> and retry after a while. Channel's associated with such sinks start buffering
> data with the caveat that if you are using a memory channel it can result in
> a domino effect on the entire pipeline. There could be legitimate down times
> eg: HDFS sink being down for name node maintenance, hadoop upgrades.
> 2. Why not use a durable channel (JDBC, FileChannel)?
> Want high throughput and support sink down times as a first class use-case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
