-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/
-----------------------------------------------------------
(Updated 2012-05-21 09:44:35.214547)
Review request for Flume.
Changes
-------
Arvind, thanks for your insightful review. After the latest patch, the
Kerberos-related thread safety issues appear to be gone, and the logs are as
clean as the driven snow.
Changes in this patch:
Incorporated review feedback:
- volatile variables where needed
- Call BucketWriter.flush() before Transaction.commit() as appropriate
- synchronize the public methods of BucketWriter
Additional improvements:
- Unify configuration timeouts to just callTimeout (appendTimeout doesn't
mean anything anymore)
- Increase default callTimeout to 10 seconds
- Acquire lock on getLoginUser() before calling Path.getFileSystem()
- this is because that call is not thread-safe under Kerberos
- in a non-kerberized cluster, this will just be the Unix login credential
Summary
-------
BucketWriter refactoring: append() does all the work of open/close/roll. open()
is a private method that takes no arguments. No abort() call. Only one
constructor.
Far fewer entry points and code paths. I believe I've closed all or many of the
race conditions and clarified the API responsibilities/semantics.
This addresses bug FLUME-1219.
https://issues.apache.org/jira/browse/FLUME-1219
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
91cb822
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
d272f74
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
397138b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
0f78f37
Diff: https://reviews.apache.org/r/5175/diff
Testing
-------
Looks good under load. Unit tests pass.
Thanks,
Mike
