[ https://issues.apache.org/jira/browse/JCLOUDS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Gaul resolved JCLOUDS-1516. ---------------------------------- Fix Version/s: 2.4.0 Assignee: Andrew Gaul Resolution: Fixed > First putblob should be signed with specific region rather than with default > region during createcontainer API > --------------------------------------------------------------------------------------------------------------- > > Key: JCLOUDS-1516 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1516 > Project: jclouds > Issue Type: New Feature > Components: jclouds-blobstore > Affects Versions: 2.1.2 > Environment: Linux > Reporter: Dileep Dixith > Assignee: Andrew Gaul > Priority: Major > Fix For: 2.4.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > When container in non default region exists, only first time put operation > will be applied to find out whether bucket exists and have proper access or > not. Aws sigv4 signature will be created based on the default region only as > create bucket method was not honoring the region specified. > So, Put request was first signed with default region and if the user does not > have access to default(us-east-1) region,. Then it will be re-directed to > sa-east-1 region, but the request is signed with us-east-1, the request is > rejected and throws "The authorization header is malformed". > Flow in case of user has access to default region: > 17:22:24.460 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking CreateBucket > 17:22:24.460 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking CreateBucket > 17:22:24.461 [bscThread-02] DEBUG jclouds.signature - << PUT > / > content-length:105 > content-type:text/xml > host:test3.s3.amazonaws.com > x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > x-amz-date:20190818T115218Z > content-length;content-type;host;x-amz-content-sha256;x-amz-date > f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > 17:22:24.461 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256 > 20190818T115218Z > 20190818/us-east-1/s3/aws4_request > 089a5248f5eff6e8b6378154acdf07bff7d208029c98c67af44c99b4a8f2df39 > 17:22:24.463 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - > Sending request -1533211628: PUT https://test3.s3.amazonaws.com/ HTTP/1.1 > 17:22:24.463 [bscThread-02] DEBUG jclouds.wire - >> > "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>" > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> PUT > https://test3.s3.amazonaws.com/ HTTP/1.1 > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105 > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Host: > test3.s3.amazonaws.com > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: > f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: > 20190818T115218Z > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Authorization: > AWS4-HMAC-SHA256 > Credential=AKIAIGKQ7V52FQQJFYJQ/20190818/us-east-1/s3/aws4_request, > SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, > Signature=637d42fbf6684430ab0f08fd82cbae69f3261859e0031ad40054bccb829473da > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml > 17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105 > 17:22:25.671 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - > Receiving response -1533211628: HTTP/1.1 409 Conflict > 17:22:25.671 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 409 Conflict > 17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: > chunked > 17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3 > 17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: > 09E5163C51F25F34 > 17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: > WuN84GYMs47Nn6+48XYDpLZNvp0NPszokKyhxlzZk+ub8RhjbLpkfEI8E2tKWVCFKtJiXrhdpkc= > 17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 > 11:52:11 GMT > 17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << x-amz-bucket-region: > sa-east-1 > 17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << Content-Type: > application/xml > 17:22:25.673 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" > encoding="UTF-8"?>[\n]" > 17:22:25.673 [bscThread-02] DEBUG jclouds.wire - << > "<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to > create the named bucket succeeded and you already own > it.</Message><BucketName>test3</BucketName><RequestId>09E5163C51F25F34</RequestId><HostId>WuN84GYMs47Nn6+48XYDpLZNvp0NPszokKyhxlzZk+ub8RhjbLpkfEI8E2tKWVCFKtJiXrhdpkc=</HostId></Error>" > 17:22:25.680 [bscThread-03] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking BucketExists > 17:22:25.681 [bscThread-03] DEBUG jclouds.signature - << HEAD > / > Flow in case of user has access to default region > host:test3.s3-sa-east-1.amazonaws.com > x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 > x-amz-date:20190818T115218Z > > Flow in case of user does not have access to default region. > 17:28:41.464 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking CreateBucket > 17:28:41.464 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking CreateBucket > 17:28:41.466 [bscThread-02] DEBUG jclouds.signature - << PUT > / > content-length:105 > content-type:text/xml > host:test-poc-spectrum-scale.s3.amazonaws.com > x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > x-amz-date:20190818T115835Z > content-length;content-type;host;x-amz-content-sha256;x-amz-date > f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > 17:28:41.466 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256 > 20190818T115835Z > 20190818/us-east-1/s3/aws4_request > d8c68a44175c3b0b49182ca5658bff291b3f382d7d02b3fe25d6970912df1697 > 17:28:41.467 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - > Sending request -541426552: PUT > https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1 > 17:28:41.467 [bscThread-02] DEBUG jclouds.wire - >> > "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>" > 17:28:41.467 [bscThread-02] DEBUG jclouds.headers - >> PUT > https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1 > 17:28:41.467 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105 > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Host: > test-poc-spectrum-scale.s3.amazonaws.com > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: > f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: > 20190818T115835Z > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Authorization: > AWS4-HMAC-SHA256 > Credential=AKIA25YU33MFMAZO3B7F/20190818/us-east-1/s3/aws4_request, > SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, > Signature=8cf0d9f58bd0ba5fc607f5018e701cdf14587741169ba7deaaa7a9cad0548cb6 > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml > 17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105 > 17:28:43.016 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - > Receiving response -541426552: HTTP/1.1 400 Bad Request > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 400 Bad > Request > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: > chunked > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3 > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Connection: close > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: > 43F137234826AA08 > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: > nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc= > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 > 11:58:29 GMT > 17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Content-Type: > application/xml > 17:28:43.019 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" > encoding="UTF-8"?>[\n]" > 17:28:43.019 [bscThread-02] DEBUG jclouds.wire - << > "<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization > header is malformed; the region 'us-east-1' is wrong; expecting > 'sa-east-1'</Message><Region>sa-east-1</Region><RequestId>43F137234826AA08</RequestId><HostId>nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=</HostId></Error>" > EXC:org.jclouds.aws.AWSResponseException: request PUT > https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1 failed with code > 400, error: AWSError\{requestId='43F137234826AA08', > requestToken='nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=', > code='AuthorizationHeaderMalformed', message='The authorization header is > malformed; the region 'us-east-1' is wrong; expecting 'sa-east-1'', > context='{Region=sa-east-1, > HostId=nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=}'} > I have a fix so that the first put operation honors the custom region > specified and request will be signed with specific region rather than with > default region. Tested with default region as well and worked well. > After my fix, flow with stand-alone program is as below. > 17:36:09.043 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking CreateBucket > 17:36:09.043 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> > invoking CreateBucket > 17:36:09.044 [bscThread-02] DEBUG jclouds.signature - << PUT > / > content-length:105 > content-type:text/xml > host:test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com > x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > x-amz-date:20190818T120602Z > content-length;content-type;host;x-amz-content-sha256;x-amz-date > f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > 17:36:09.044 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256 > 20190818T120602Z > 20190818/sa-east-1/s3/aws4_request > 2996a3b4d85ab8c68e1378ab68db2f7d80b244969d1792391a49b9b7390bc920 > 17:36:09.046 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - > Sending request -719121422: PUT > https://test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com/ HTTP/1.1 > 17:36:09.046 [bscThread-02] DEBUG jclouds.wire - >> > "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>" > 17:36:09.046 [bscThread-02] DEBUG jclouds.headers - >> PUT > https://test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com/ HTTP/1.1 > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105 > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Host: > test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: > f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96 > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: > 20190818T120602Z > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Authorization: > AWS4-HMAC-SHA256 > Credential=AKIA25YU33MFMAZO3B7F/20190818/sa-east-1/s3/aws4_request, > SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, > Signature=982a3a62e85e03c68f4ac1e0da5cfa753cb81cf750a6fbff157b34681ed54774 > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml > 17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105 > 17:36:11.076 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - > Receiving response -719121422: HTTP/1.1 409 Conflict > 17:36:11.076 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 409 Conflict > 17:36:11.076 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: > chunked > 17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3 > 17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: > 5FB8D9C1B41E48EB > 17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: > 54oqA5Lc+yl6Y9Ppz6kMd6hZs/iGFGfNFPqrOPX90Q731UH5KkYEYc7RS/4W9btahu0dxQKd3iI= > 17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 > 12:05:57 GMT > 17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-bucket-region: > sa-east-1 > 17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Content-Type: > application/xml > 17:36:11.077 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" > encoding="UTF-8"?>[\n]" > 17:36:11.077 [bscThread-02] DEBUG jclouds.wire - << > "<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to > create the named bucket succeeded and you already own > it.</Message><BucketName>test-poc-spectrum-scale</BucketName><RequestId>5FB8D9C1B41E48EB</RequestId><HostId>54oqA5Lc+yl6Y9Ppz6kMd6hZs/iGFGfNFPqrOPX90Q731UH5KkYEYc7RS/4W9btahu0dxQKd3iI=</HostId></Error>" > 17:36:11.083 [bscThread-03] DEBUG o.j.rest.internal.InvokeHttpMethod - -- This message was sent by Atlassian Jira (v8.3.4#803005)