[
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915277#comment-13915277
]
Hudson commented on FLUME-1227:
-------------------------------
SUCCESS: Integrated in flume-trunk #553 (See
[https://builds.apache.org/job/flume-trunk/553/])
FLUME-1227. Introduce Spillable Channel. (hshreedharan:
http://git-wip-us.apache.org/repos/asf/flume/repo/?p=flume.git&a=commit&h=6a50ec2ad33b8cbd057907c67030d855520c5f13)
* flume-ng-doc/sphinx/FlumeUserGuide.rst
* flume-ng-channels/flume-spillable-memory-channel/pom.xml
*
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
* flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java
*
flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java
* flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
* flume-ng-node/pom.xml
*
flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java
* pom.xml
* flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
* flume-ng-dist/pom.xml
*
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
*
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java
*
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java
* flume-ng-channels/pom.xml
> Introduce some sort of SpillableChannel
> ---------------------------------------
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Jarek Jarcec Cecho
> Assignee: Roshan Naik
> Attachments: 1227.patch.1, FLUME-1227.v2.patch, FLUME-1227.v5.patch,
> FLUME-1227.v6.patch, FLUME-1227.v7.patch, FLUME-1227.v8.patch,
> FLUME-1227.v9.patch, SpillableMemory Channel Design 2.pdf, SpillableMemory
> Channel Design.pdf
>
>
> I would like to introduce new channel that would behave similarly as scribe
> (https://github.com/facebook/scribe). It would be something between memory
> and file channel. Input events would be saved directly to the memory (only)
> and would be served from there. In case that the memory would be full, we
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend
> servers that are generating events. We want to send all events to just
> limited number of machines from where we would send the data to HDFS (some
> sort of staging layer). Reason for this second layer is our need to decouple
> event aggregation and front end code to separate machines. Using memory
> channel is fully sufficient as we can survive lost of some portion of the
> events. However in order to sustain maintenance windows or networking issues
> we would have to end up with a lot of memory assigned to those "staging"
> machines. Referenced "scribe" is dealing with this problem by implementing
> following logic - events are saved in memory similarly as our MemoryChannel.
> However in case that the memory gets full (because of maintenance, networking
> issues, ...) it will spill data to disk where they will be sitting until
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's
> durability guarantees would be same as MemoryChannel - in case that someone
> would remove power cord, this channel would lose data. Based on the
> discussion in FLUME-1201, I would propose to have the implementation
> completely independent on any other channel internal code.
> Jarcec
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
