[
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843403#comment-13843403
]
Hari Shreedharan commented on FLUME-1227:
-----------------------------------------
Hi Roshan,
In the takePrimary and takeOverflow methods, there is a
Preconditions.checkArgument method where like you mentioned in takePrimary
method comments, there is an int->Integer->String conversion in a hot path
(this is handled with an if in the takePrimary method, not in takeOverflow) -
can you get rid of the the preconditions call, and just do:
if (...) {
throw IllegalStateException(..)
}.
This for one is cleaner, since the if already checks for the issue and we can
avoid an unneeded method call.
Is this because rolling back the overflow txn will ensure that the event goes
back into the file channel and you don't need to handle it?
{code}
if (!useOverflow) {
takeList.offer(event); // takeList is thd pvt, so no need to do this
in synchronized block
}
{code}
If that is the case the counters are incorrect when the transaction committed
is overflow transaction, since this is how they are updated:
{code}
channelCounter.addToEventTakeSuccessCount(takeList.size());
{code}
Even this is not accurate:
{code}
if (takeList.size() > largestTakeTxSize)
largestTakeTxSize = takeList.size();
{code}
There are also a couple issue with regards to failed transactions when writing
to primary (granted it is a queue and it should not fail, but if a lock acquire
gets interrupted, it can still fail). The memQueueRemaining semaphore has
already been updated before pushing the events to the queue (that is definitely
the right thing to do), but if a queue.offer fails the memQueueRemaining is not
updated. This might be an issue with the current channels too - and is
sufficiently rare to say we can revisit this later.
Also there is a possibility of partially successful transactions right now (if
the queue inserts fail - that I guess is true for all channels right now, so I
guess we can live with it - just mentioning it to ensure that we know it is a
possibility and we can revisit if needed).
> Introduce some sort of SpillableChannel
> ---------------------------------------
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Jarek Jarcec Cecho
> Assignee: Roshan Naik
> Attachments: 1227.patch.1, FLUME-1227.v2.patch, FLUME-1227.v5.patch,
> FLUME-1227.v6.patch, FLUME-1227.v7.patch, SpillableMemory Channel Design
> 2.pdf, SpillableMemory Channel Design.pdf
>
>
> I would like to introduce new channel that would behave similarly as scribe
> (https://github.com/facebook/scribe). It would be something between memory
> and file channel. Input events would be saved directly to the memory (only)
> and would be served from there. In case that the memory would be full, we
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend
> servers that are generating events. We want to send all events to just
> limited number of machines from where we would send the data to HDFS (some
> sort of staging layer). Reason for this second layer is our need to decouple
> event aggregation and front end code to separate machines. Using memory
> channel is fully sufficient as we can survive lost of some portion of the
> events. However in order to sustain maintenance windows or networking issues
> we would have to end up with a lot of memory assigned to those "staging"
> machines. Referenced "scribe" is dealing with this problem by implementing
> following logic - events are saved in memory similarly as our MemoryChannel.
> However in case that the memory gets full (because of maintenance, networking
> issues, ...) it will spill data to disk where they will be sitting until
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's
> durability guarantees would be same as MemoryChannel - in case that someone
> would remove power cord, this channel would lose data. Based on the
> discussion in FLUME-1201, I would propose to have the implementation
> completely independent on any other channel internal code.
> Jarcec
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
