[
https://issues.apache.org/jira/browse/FLUME-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274481#comment-13274481
]
Juhani Connolly commented on FLUME-1201:
----------------------------------------
Due to the current nature of channel instantiation in
PropertiesFileConfigurationProvider, it seems "wrong" to instantiate the
subchannels from the channel itself(which initially seemed the most viable
solution).
One solution would be to implement channelgroups like the existing sinkgroups,
but personally I find this quite unwieldy, further complicating configuration.
I'd prefer some other alternative and am open to ideas.
Unfortunately I do not believe this could not be implemented as a channel
selector(at least not one that maintains event order... If one was to make a
failover channel selector with the sink pulling from both channels that would
make for a rudimentary implementation, dependent on configuration).
I'd be curious to hear other peoples thoughts
> Create a buffer channel, that stores overflow from a fast, low capacity
> channel to a slower high capacity channel
> -----------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-1201
> URL: https://issues.apache.org/jira/browse/FLUME-1201
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
>
> As it stands, users need to make a choice between either slower channels or
> faster ones. The current "middle ground" consists of RecoverableFileChannel
> which still stores everything to disk. MemoryChannel on the other hand has
> limited capacity or a high memory footprint.
> I propose to implement a buffer channel, somewhat like the buffer store in
> scribed.
> It would normally act as a proxy to the primary channel. Should this channel
> be unable to receive events(normally because it is at capacity, but perhaps
> some future channels may have other failure cases) it would switch states to
> buffering, storing new events to the secondary channel.
> In buffering, items continue to be read from the primary channel, and it
> attempts to "refill" itself from the secondary. Once the secondary is found
> to be empty, operation switches back to streaming mode, with items going
> directly to the primary.
> The main objective of this would thus be to have a high throughput channel as
> the primary mode of operation, allowing it to switch over when takes are not
> keeping up with puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
