[
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726702#comment-13726702
]
Roshan Naik commented on FLUME-1227:
------------------------------------
Appreciate your feedback Hari.
HARI > It looks like channel can actually return fewer events than total
available in the case where there are only "n" events in the primary queue and
an "n+1"-th take would happen - since the events in a particular txn will
always come from one queue. I think we should be able to pull events from the
other store if it turns out to be required - else we expect the sink to come
back and poll immediately - and also cause sink side transactions to be smaller
than they have to be - which can cause Avro/HDFS batch sizes to be smaller than
configured causing perf issues.
Yes that is correct. The sink's transaction batch size would be smaller in that
case. The case
would only occur in when the take transaction transitions between overflow and
primary.
The alternative, as you sugest, is to pull from both overflow and primary, but
that opens up some fundamental problems similar to distributed transactions.
Essentially the sink needs to have
two transactions open (one each on overflow and primary) which needs to be
atomically committed/rolledback. Thoughts ?
HARI > How the channel recovers from an "overflow" situation.
I have updated the design doc (section 2.1.2) to elaborate on this. The short
version is:
New incoming events will go into primary if the sinks have drained older events
from the primary
even if overflow is not empty.
Let me know if the description addresses your question sufficiently.
> Introduce some sort of SpillableChannel
> ---------------------------------------
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Jarek Jarcec Cecho
> Assignee: Roshan Naik
> Attachments: 1227.patch.1, FLUME-1227.v2.patch, SpillableMemory
> Channel Design 2.pdf, SpillableMemory Channel Design.pdf
>
>
> I would like to introduce new channel that would behave similarly as scribe
> (https://github.com/facebook/scribe). It would be something between memory
> and file channel. Input events would be saved directly to the memory (only)
> and would be served from there. In case that the memory would be full, we
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend
> servers that are generating events. We want to send all events to just
> limited number of machines from where we would send the data to HDFS (some
> sort of staging layer). Reason for this second layer is our need to decouple
> event aggregation and front end code to separate machines. Using memory
> channel is fully sufficient as we can survive lost of some portion of the
> events. However in order to sustain maintenance windows or networking issues
> we would have to end up with a lot of memory assigned to those "staging"
> machines. Referenced "scribe" is dealing with this problem by implementing
> following logic - events are saved in memory similarly as our MemoryChannel.
> However in case that the memory gets full (because of maintenance, networking
> issues, ...) it will spill data to disk where they will be sitting until
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's
> durability guarantees would be same as MemoryChannel - in case that someone
> would remove power cord, this channel would lose data. Based on the
> discussion in FLUME-1201, I would propose to have the implementation
> completely independent on any other channel internal code.
> Jarcec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
