chungen0126 commented on code in PR #10006: URL: https://github.com/apache/ozone/pull/10006#discussion_r3195130997
########## hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md: ########## @@ -0,0 +1,142 @@ +--- +title: S3 Multi Chunks Verification +summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G. +date: 2026-04-29 +jira: HDDS-12542 +status: proposed +author: Chung-En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Context & Motivation + +Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. + +# Goal + +Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms: +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER + +# Proposed Solution + +Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs. + +## Secret Key + +Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data. Review Comment: > Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern. So is there a way to do some chunk signature validation without knowing the user secret key? Retrieving the signing key from OM is secure due to the following reasons: 1. One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key 2. Time-bound Scoping: The derived key is strictly bound to a specific date. Because the date is a required input for the HMAC calculation, the key is only valid for that particular day and cannot be reused for requests on any other date. As mentioned in the design doc: > From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. Regarding the performance concern of fetching the signing key from OM. For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the `createKey` call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
