[ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074461#comment-16074461 ]
Ilya Fourmanov commented on HADOOP-14620: ----------------------------------------- That's extremely interesting. So {code} hadoop fs -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls s3a://dshbasebackup/ {code} fails for me with 403 as described above however if I use format as proposed by [~ste...@apache.org] {code} hadoop fs -D fs.s3a.bucket.dshbasebackup.endpoint=s3.eu-west-1.amazonaws.com -ls s3a://dshbasebackup/ {code} it works as expected. Now, what's the difference between those 2 formats? > S3A authentication failure for regions other than us-east-1 > ----------------------------------------------------------- > > Key: HADOOP-14620 > URL: https://issues.apache.org/jira/browse/HADOOP-14620 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 2.8.0, 2.7.3 > Reporter: Ilya Fourmanov > Attachments: s3-403.txt > > > hadoop fs s3a:// operations fail authentication for s3 buckets hosted in > regions other than default us-east-1 > Steps to reproduce: > # create s3 bucket in eu-west-1 > # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run > following command: > {code} > hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls > s3a://your-eu-west-1-hosted-bucket/ > {code} > Expected behaviour: > You will see listing of the bucket > Actual behaviour: > You will get 403 Authentication Denied response for AWS S3. > Reason is mismatch in string to sign as defined in > http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html > provided by hadoop and expected by AWS. > If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes > returned by AWS, you will see that AWS expects CanonicalizedResource to be in > form > /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/. > Hadoop provides it as /your-eu-west-1-hosted-bucket/ > Note that AWS documentation doesn't explicitly state that endpoint or full > dns address should be appended to CanonicalizedResource however practice > shows it is actually required. > I've also submitted this to AWS for them to correct behaviour or > documentation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org