I was trying to create a really simple pipeline that read from a bucket in a filesystem (s3) and writes to a different bucket in the same filesystem.
S3Options options = PipelineOptionsFactory.fromArgs(args).create().as(S3Options.class); Pipeline pipeline = Pipeline.create(options); pipeline .apply("ReadLines", TextIO.read().from("s3://src-bucket/*")) // .apply("AllOtherMagic", ...) .apply("WriteCounts", TextIO.write().to("s3://dst-bucket/")); p.run().waitUntilFinish(); I discovered that my original bucket was in a different region so I needed to pass a different S3Options object to the Write ‘options.setAwsRegion(“dst-region”)’, but I could not find a way to do it. Can somebody give me a hint on how to do this? I was wondering that since File-based IOs use the configuration implied by the Filesystem if this was possible. With non-file based IOs all the configuration details are explicit in each specific transform, but this is not the case for these file-based transforms. Note. I know this question probably belongs more to user@ but since I couldn’t find an easy way to do it I was wondering if this is an issue we should consider at dev@ from an API point of view.