[
https://issues.apache.org/jira/browse/HDDS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18018774#comment-18018774
]
Ivan Andika commented on HDDS-12935:
------------------------------------
It is suspected that this introduces a regression on Presigned PutObject (since
there was no such tests at the time). This should be addressed in HDDS-5195.
Therefore HDDS-12935 and HDDS-5195 should be targetted to the next release.
> Support unsigned chunked upload with STREAMING-UNSIGNED-PAYLOAD-TRAILER
> -----------------------------------------------------------------------
>
> Key: HDDS-12935
> URL: https://issues.apache.org/jira/browse/HDDS-12935
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Critical
> Labels: pull-request-available
> Fix For: 2.1.0
>
>
> HDDS-12488 fixes the issue where the AWS checksum trailer data is silently
> appended to the final payload. This works for http endpoint which will by
> default provide with signed payload (i.e. x-amz-content-sha256 is set to the
> payload signature).
> When the request is sent against https endpoint, the x-amz-content-sha256 is
> set to STREAMING-UNSIGNED-PAYLOAD-TRAILER , probably because https protocol
> already encrypts the HTTP payload and there is no need to further encryption
> using signature. HDDS-12488 handled it by using string "UNSIGNED-PAYLOAD:
> when building the canonical request
> (StringToSignProducer#buildCanonicalRequest) as specified in
> [https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html].
> However, this causes the mismatch between the AWS SDK canonical request and
> the calculated canonical request which results in signature mismatch and all
> PutObject requests for the newer AWS SDK version to fail.
> Did manual testing
> AWS CLI canonical request
> {code:java}
> PUT
> /ivan-bucket/test4.txt
> content-encoding:aws-chunked
> host:<redacted>
> x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
> x-amz-date:20250430T071131Z
> x-amz-decoded-content-length:5
> x-amz-sdk-checksum-algorithm:CRC64NVME
> x-amz-trailer:x-amz-checksum-crc64nvme
> content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer
> STREAMING-UNSIGNED-PAYLOAD-TRAILER
> {code}
> Ozone canonical request
> {code:java}
> PUT
> /ivan-bucket/test4.txt
> content-encoding:aws-chunked
> host:<redacted>
> x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
> x-amz-date:20250430T071131Z
> x-amz-decoded-content-length:5
> x-amz-sdk-checksum-algorithm:CRC64NVME
> x-amz-trailer:x-amz-checksum-crc64nvme
> content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer
> UNSIGNED-PAYLOAD
> {code}
> AWS CLI StrToSign
> {code:java}
> AWS4-HMAC-SHA256
> 20250430T065600Z
> 20250430/us-east-1/s3/aws4_request
> a3b489c10cf8e4878c7d7b870d3eda15c9c17f52541bd4742361a538c2b458fe
> {code}
> Ozone StrToSign
> {code:java}
> AWS4-HMAC-SHA256
> 20250430T065600Z
> 20250430/us-east-1/s3/aws4_request
> cb995d592067f026dc30bb4416340afa7e9d38abee57e8e8a44a5604380a22ab
> {code}
> The correct behavior seems to use the "STREAMING-UNSIGNED-PAYLOAD-TRAILER"
> instead of "UNSIGNED-PAYLOAD" when building the canonical request.
> Ref:
> [https://github.com/aws/aws-sdk-java-v2/blob/f6adeaa5b24a4c203106d122647537f83a5ecfbc/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/AbstractAwsS3V4Signer.java#L206]
> Additionally, unsigned chunked upload (STREAMING-UNSIGNED-PAYLOAD-TRAILER)
> will omit the "chunk-signature". For example:
> {code:java}
> 5
> test
> 0
> x-amz-checksum-crc64nvme:2wstOANdZ/o=
> {code}
> We have SignedChunksInputStream, but it only works for signed chunked upload.
> This might requires us to create a new InputStream to handle unsigned chunk
> uploads. Additionally, currently STREAMING-UNSIGNED-PAYLOAD-TRAILER is
> handled as normal single chunk input stream which causes unexpected key
> content.
> Note that some of the behaviors are not well-documented, and might not catch
> some edge cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
