Steve Loughran created HADOOP-17771:
---------------------------------------
Summary: S3AFS creation fails without region set ~/.aws/config
Key: HADOOP-17771
URL: https://issues.apache.org/jira/browse/HADOOP-17771
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.1
Reporter: Steve Loughran
Assignee: Steve Loughran
If you don't have {{fs.s3a.endpoint}} set and lack a region set in
env var {{AWS_REGION_ENV_VAR}}, system property {{aws.region}} or the file
~/.aws/config
then S3A FS creation fails with the message
"Unable to find a region via the region provider chain."
This is caused by the move to the AWS S3 client builder API in HADOOP-13551
This is pretty dramatic and no doubt everyone will be asking "why didn't you
notice this?",
But in fact there are some reasons.
# when running in EC2, all is well. Meaning our big test runs were all happy.
# if a developer has fs.s3a.endpoint set for the test bucket, all is well.
Those of us who work with buckets in the "regions tend to do this, not least
because it can save a HEAD request every time an FS is created.
# if you have a region set in ~/.aws/config then all is well
reason #3 is the real surprise and the one which has really caught out. Even my
tests against buckets in usw-2 through central didn't fail because of course I,
like my colleagues, have the AWS S3 client installed locally. This was
sufficient to make the problem go away. It is also why this has been an
intermittent problem on test clusters outside AWS infra: it really depended on
the VM/docker image whether things worked or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
