-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31643/
-----------------------------------------------------------
Review request for Flume.
Bugs: FLUME-2437
https://issues.apache.org/jira/browse/FLUME-2437
Repository: flume-git
Description
-------
S3 Source, initial version
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/serialization/FileStreamCreator.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableGenericInputStream.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/StreamCreator.java
PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java
d1240fb
flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableGenericInputStream.java
PRE-CREATION
flume-ng-dist/pom.xml a083fe2
flume-ng-sources/flume-s3-source/pom.xml PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/FileBasedMetadataBackingStore.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/InMemoryMetadataBackingStore.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/MetadataBackingStore.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/MetadataBackingStoreFactory.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3ObjectEventReader.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3Source.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3SourceConfigurationConstants.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3StreamCreator.java
PRE-CREATION
flume-ng-sources/flume-s3-source/src/test/java/org/apache/flume/source/s3/TestS3Source.java
PRE-CREATION
flume-ng-sources/pom.xml ab8eca4
pom.xml ea7ffe3
Diff: https://reviews.apache.org/r/31643/diff/
Testing
-------
TestResettableGenericInputStream and
TestS3Source
Manual testing of scenarios:
1. Created multiple files in S3 Bucket - make sure the source processes all the
files
2. Add more files, after the S3 source starts - make sure the newly created S3
objects are processed
3. Stop the source after a few files are processed - make sure on restart, the
source only processes the rest of the unprocessed files
4. Stop the source in the middle of processing a file - make sure the postion
tracker is read on re-start and processing of the partial file continues from
where it was last marked
Thanks,
Johny Rufus John
