[
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540116#comment-13540116
]
Roshan Naik commented on FLUME-1227:
------------------------------------
Seeking input ..
The current configuration system does not look conducive to chaining channels.
Here are the config techniques that has been previously talked about :
1) Out-of-line:
agent1.channels = channel1 channel2
agent1.channels.channel1.type = SPILLABLE
agent1.channels.channel1.overflow = channel2
agent1.channels.channel2.type = FILE
agent1.channels.channel2.checkpointDir = /path1
...
The problem here is that ..
- At the time channel1 is configured, channel2 may not have been instantiated
yet. So it is not possible to latch on to an instance of channel2. So it may be
better to defer obtaining a reference to the overflow channel at start time.
- No mechanism to get a reference to one channel from another (in this case, at
start time)
2) Inline: (as suggested by Mike)
agent1.channels = channel1
agent1.channels.channel1.type = SPILLABLE
agent1.channels.channel1.overflowChannel.type = FILE
agent1.channels.channel1.overflowChannel.checkpointDir = /path1
agent1.channels.channel1.overflowChannel.dataDirs = /path2
...
The issue here is that the instantiation and configuration of the overflow
channel will now have to reside inside SpillableChannel::configure(). This
method is not a very conducive place for doing such things.
3) Hard coding
Basically hard code the file channel to be the overflow channel. this allows
the file channel to be easily instantiated and configured. downside is that it
still duplicates the channel instantiation/config logic from
AbstractConfigurationProvider.loadChannels()
Any thoughts ?
> Introduce some sort of SpillableChannel
> ---------------------------------------
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Jarek Jarcec Cecho
> Assignee: Roshan Naik
>
> I would like to introduce new channel that would behave similarly as scribe
> (https://github.com/facebook/scribe). It would be something between memory
> and file channel. Input events would be saved directly to the memory (only)
> and would be served from there. In case that the memory would be full, we
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend
> servers that are generating events. We want to send all events to just
> limited number of machines from where we would send the data to HDFS (some
> sort of staging layer). Reason for this second layer is our need to decouple
> event aggregation and front end code to separate machines. Using memory
> channel is fully sufficient as we can survive lost of some portion of the
> events. However in order to sustain maintenance windows or networking issues
> we would have to end up with a lot of memory assigned to those "staging"
> machines. Referenced "scribe" is dealing with this problem by implementing
> following logic - events are saved in memory similarly as our MemoryChannel.
> However in case that the memory gets full (because of maintenance, networking
> issues, ...) it will spill data to disk where they will be sitting until
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's
> durability guarantees would be same as MemoryChannel - in case that someone
> would remove power cord, this channel would lose data. Based on the
> discussion in FLUME-1201, I would propose to have the implementation
> completely independent on any other channel internal code.
> Jarcec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
