Starting a new thread just for dealing with AWS regions better, context S3
and Redshift.

S3FileSystem.amazonS3 build could be refactored to select region based on
[1]:
1. the flag value region
2. the EC2 region, if found in environment (running in EC2 VM)
3. the default region (us-east-1)

For actually moving data, a Map<String, AmazonS3> could be used to hold an
S3 client per region, new S3 clients created as needed. The "master" client
can be used to find a bucket's region [2].

I think this is pragmatic, looking for feedback before I write a PR. Also,
if someone is already making progress, let me know.

Jacob

[1]
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html

[2]
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#getBucketLocation-java.lang.String-

Reply via email to