[
https://issues.apache.org/jira/browse/FLUME-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249626#comment-13249626
]
[email protected] commented on FLUME-1085:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4661/
-----------------------------------------------------------
(Updated 2012-04-08 22:34:48.594621)
Review request for Flume.
Changes
-------
There were some races in how Log cleaned up logs. We now find the smallest file
id from the queue and open for write. We then ignore all files greater than
this id. Additionally we insure we have at least two data files per data
directory. Added integration test to TestFileChannel.
Summary
-------
This patch implements a durable file channel. It does by writing all
transaction events to disk and syncing to disk when a commit occurs. It does
have a memory component in that pointers to the event on disk are kept in
memory. This will consume 8 bytes of direct memory (non-heap) per event. Some
basic calculations are in the FileChannel java docs.
This addresses bug FLUME-1085.
https://issues.apache.org/jira/browse/FLUME-1085
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/pom.xml 926e58a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
a279453
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
ab66998
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties
739ecc8
flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java
0622f27
flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java
fa63b73
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java
PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java
225cd34
flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java
0b8a2c0
Diff: https://reviews.apache.org/r/4661/diff
Testing
-------
Unit tests and integration tests added to cover obvious cases.
Thanks,
Brock
> Implement a durable FileChannel
> -------------------------------
>
> Key: FLUME-1085
> URL: https://issues.apache.org/jira/browse/FLUME-1085
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Affects Versions: v1.2.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> FLUME-896 turned into a durable Memory Channel, we need a durable File Channel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
