I was trying to create a really simple pipeline that read from a
bucket in a filesystem (s3) and writes to a different bucket in the
same filesystem.
S3Options options =
PipelineOptionsFactory.fromArgs(args).create().as(S3Options.class);
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply("ReadLines", TextIO.read().from("s3://src-bucket/*"))
// .apply("AllOtherMagic", ...)
.apply("WriteCounts", TextIO.write().to("s3://dst-bucket/"));
p.run().waitUntilFinish();
I discovered that my original bucket was in a different region so I
needed to pass a different S3Options object to the Write
‘options.setAwsRegion(“dst-region”)’, but I could not find a way to do
it. Can somebody give me a hint on how to do this?
I was wondering that since File-based IOs use the configuration
implied by the Filesystem if this was possible. With non-file based
IOs all the configuration details are explicit in each specific
transform, but this is not the case for these file-based transforms.
Note. I know this question probably belongs more to user@ but since I
couldn’t find an easy way to do it I was wondering if this is an issue
we should consider at dev@ from an API point of view.
