Source with S3 bucket with millions ( billions ) of object ( small files )

[Vishal Santoshi]

Folks,
        I am doing a simple batch job that uses readFile() with
"s3a://[bucket_name]" as the path with setNestedFileEnumeration(true). I am
a little curious about a few things.

In batch mode which I think is turned on by FileProcessingMode.PROCESS_ONCE
mode does the source list all the S3 objects in the bucket to create input
splits *before* it calls downstream operators ?




Thanks.