Mukund Thakur created HADOOP-18353:
--------------------------------------
Summary: HEAD OBJECT returns only 400 BAD REQUEST when token is
expired
Key: HADOOP-18353
URL: https://issues.apache.org/jira/browse/HADOOP-18353
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.3
Reporter: Mukund Thakur
I tried reproducing this today by changing this test
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116]
. Getting a session token for 15 mins and trying every one mins and finally it
fails after 15 mins.
Looks like the AWS SDK is not having the Expired Token error message as I could
see the same in access logs but I see BadRequest on the SDK logs.
*S3A Connector logs with SDK debug enabled.*
2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from
TemporaryAWSCredentialsProvider
2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request
(AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD
[https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
/test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers:
(amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type:
application/octet-stream, Referer:
[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318],
User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
vendor/Oracle_Corporation cfg/retry-mode/legacy, )
2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request
(AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error
response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request
ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID:
IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=;
Proxy: null), S3 Extended Request ID:
IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is closed
*AWS access logs*
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
mthakur-us-west-1 [13/Jul/2022:20:44:15 +0000] 67.79.115.98 - WMGQ0KC4MHEMZTQC
REST.HEAD.OBJECT test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD
/test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400 ExpiredToken*
556 - 5 -
"[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318]";
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
vendor/Oracle_Corporation cfg/retry-mode/legacy" -
IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
SigV4 ECDHE-RSA-AES128-SHA AuthHeader
[mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
TLSv1.2 -
I tested by running repeatedly ITestCustomSigner in S3A, and also just
ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and *I
can reproduce* by running this after credential expiry.
aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key
test-object –debug
To summarise:
* If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
* If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
* If I run GetObject, I get “400 ExpiredToken”
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
