-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/
-----------------------------------------------------------
Review request for Flume and Brock Noland.
Description
-------
Flume Event Queue now keeps a copy of the event pointers to uncommitted puts
and takes. It serializes these on every checkpoint and deserializes these on
replay and reinserts these into either the event queue(for takes) or to the
replay queue(for puts).
I could have used the PutList and TakeList of the transaction for this, but I
didn't really like the approach. I don't want to be sharing this kind of data
between multiple layers, since that makes it complex to change the
FlumeEventQueue implementation without causing major changes in
FileBackedTransaction. Also it would lead to a number of cross layer calls to
read data - which makes the approach less clean.
With my current approach, by localizing most changes to the FlumeEventQueue
class, only a couple of function calls would need to be removed/modified.
Agreed that this is going to be some memory overhead, but this is insignificant
compared to the event queue size itself. This would be hardly a few MB extra in
memory - but if that gives me cleaner implementation, I would prefer that.
This addresses bug FLUME-1437.
https://issues.apache.org/jira/browse/FLUME-1437
Diffs
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
e7735e8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
9bfee2d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
11f1e1f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
bbca62c
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
7ec5916
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
1d5a0f9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
569b7c7
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
e0b5e3f
Diff: https://reviews.apache.org/r/6683/diff/
Testing
-------
Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case,
and 2 to TestFlumeEventQueue.java to test the actual functionality of
serialization/deserialization).
Thanks,
Hari Shreedharan
