I sent this a little prematurely. Will the streaming process find new
directories under the parent?
If the input path is
s3://foo.bar/
and directories are added daily, should I expect that the newly added
directories+files will get processed?
Thanks!
Wayne
On 2021/01/21 23:20:41, Billy Bain <b...@gmail.com> wrote:
> I have a Streaming process where new directories are added daily in S3.>
>
> s3://foo/bar/2021-01-18/data.gz>
> s3://foo/bar/2021-01-19/data.gz>
> s3://foo/bar/2021-01-20/data.gz>
>
> If I start the process, it won't pick up anything other than the>
> directories visible when the process was started.>
>
> The textInput has this applied:>
> textInputFormat.setNestedFileEnumeration(true);>
>
> DataStreamSource<String> lines = env.readFile(textInputFormat,
inputPath,>
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000);>
> -- >
> Wayne D. Young>
> aka Billy Bob Bain>
> billybobb...@gmail.com>
>
