> On March 15, 2015, 11:04 a.m., Pawel wrote: > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3Source.java, > > line 220 > > <https://reviews.apache.org/r/31643/diff/1/?file=882391#file882391line220> > > > > Hmmm... It seems that source runnable will read only all already > > existing files in Bucket and will not track new ones which appear
New files are tracked, in readEvents(), under the comment - "// Check if new files have arrived since last call", getNextFile() takes care of this - Johny Rufus ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31643/#review76506 ----------------------------------------------------------- On March 2, 2015, 10:26 p.m., Johny Rufus John wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/31643/ > ----------------------------------------------------------- > > (Updated March 2, 2015, 10:26 p.m.) > > > Review request for Flume. > > > Bugs: FLUME-2437 > https://issues.apache.org/jira/browse/FLUME-2437 > > > Repository: flume-git > > > Description > ------- > > S3 Source, initial version > > > Diffs > ----- > > > flume-ng-core/src/main/java/org/apache/flume/serialization/FileStreamCreator.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableGenericInputStream.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/serialization/StreamCreator.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java > d1240fb > > flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableGenericInputStream.java > PRE-CREATION > flume-ng-dist/pom.xml a083fe2 > flume-ng-sources/flume-s3-source/pom.xml PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/FileBasedMetadataBackingStore.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/InMemoryMetadataBackingStore.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/MetadataBackingStore.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/MetadataBackingStoreFactory.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3ObjectEventReader.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3Source.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3SourceConfigurationConstants.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/main/java/org/apache/flume/source/s3/S3StreamCreator.java > PRE-CREATION > > flume-ng-sources/flume-s3-source/src/test/java/org/apache/flume/source/s3/TestS3Source.java > PRE-CREATION > flume-ng-sources/pom.xml ab8eca4 > pom.xml ea7ffe3 > > Diff: https://reviews.apache.org/r/31643/diff/ > > > Testing > ------- > > TestResettableGenericInputStream and > TestS3Source > > Manual testing of scenarios: > 1. Created multiple files in S3 Bucket - make sure the source processes all > the files > 2. Add more files, after the S3 source starts - make sure the newly created > S3 objects are processed > 3. Stop the source after a few files are processed - make sure on restart, > the source only processes the rest of the unprocessed files > 4. Stop the source in the middle of processing a file - make sure the postion > tracker is read on re-start and processing of the partial file continues from > where it was last marked > > > Thanks, > > Johny Rufus John > >
