chungen0126 commented on code in PR #10006: URL: https://github.com/apache/ozone/pull/10006#discussion_r3195032539
########## hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md: ########## @@ -0,0 +1,142 @@ +--- +title: S3 Multi Chunks Verification +summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G. +date: 2026-04-29 +jira: HDDS-12542 +status: proposed +author: Chung-En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Context & Motivation + +Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. + +# Goal + +Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms: +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER + +# Proposed Solution + +Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs. + +## Secret Key + +Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data. Review Comment: In symmetric cryptography like HMAC, both the client and the server independently hold the shared secret key to prevent: 1. Key interception and unauthorized writes: The user's secret key is never transmitted over the network. If it were, any illegal actor intercepting the traffic could steal the key and perform unauthorized write operations. 2. Spoofed verification and untraceable access: The server must never rely on a secret key provided within the request itself for verification. If it did, an attacker could simply sign a payload with a fake secret key and send it along, making it impossible for the server to detect the unauthorized access. AFAIK, S3G currently does not hold any secret key state locally. If I missed something and there is already a mechanism where S3G stores this, could you please point me to it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
