ivandika3 commented on code in PR #10010: URL: https://github.com/apache/ozone/pull/10010#discussion_r3141864986
########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +66,214 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional CompleteMultipartUpload Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional multipart completion applies to +`CompleteMultipartUpload`. Although the request finalizes an existing +multipart upload, the conditional headers are evaluated against the +current committed object at the destination key, not against the MPU's +temporary part state. -#### Restrictions +#### Supported Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Complete the MPU only if no committed object currently exists at the destination key.|`412 Precondition Failed` if the destination key already exists.| +|`If-Match: "<etag>"`|Complete the MPU only if the current committed object exists and its ETag matches.|`412 Precondition Failed` if the object is missing or the ETag does not match.| Review Comment: I'm wondering how we can get the ETag here. Does the client need to know how MPU ETag is calculated in OM? ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +62,187 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| Review Comment: Also add `If-Match: "*"` as per https://github.com/apache/ozone/pull/9815#discussion_r3012836729 ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +66,214 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional CompleteMultipartUpload Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional multipart completion applies to +`CompleteMultipartUpload`. Although the request finalizes an existing +multipart upload, the conditional headers are evaluated against the +current committed object at the destination key, not against the MPU's +temporary part state. -#### Restrictions +#### Supported Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Complete the MPU only if no committed object currently exists at the destination key.|`412 Precondition Failed` if the destination key already exists.| +|`If-Match: "<etag>"`|Complete the MPU only if the current committed object exists and its ETag matches.|`412 Precondition Failed` if the object is missing or the ETag does not match.| -- Cannot use both headers together in same request -- No additional charges for failed conditional requests +#### Restrictions and Notes + +- `If-Match` and `If-None-Match` must not be used together. +- `CreateMultipartUpload`, `UploadPart`, and `UploadPartCopy` do not + use these destination write headers. The precondition is enforced only + when the client calls `CompleteMultipartUpload`. +- In-progress multipart uploads do not count as an existing committed + object for `If-None-Match: *`. +- The MPU parts remain associated with the multipart upload until the + complete request succeeds or the upload is aborted. ### AWS S3 Conditional Read Specification -TODO +Conditional reads apply to `GetObject` and `HeadObject`. They do not mutate object state; they only decide whether the +current representation should be returned. + +#### Supported Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Return the object only if the current ETag matches|`412 Precondition Failed`| +|`If-None-Match: "<etag>"`|Return the object only if the current ETag does not match|`304 Not Modified`| +|`If-Modified-Since: <http-date>`|Return the object only if it has been modified since the supplied time|`304 Not Modified`| +|`If-Unmodified-Since: <http-date>`|Return the object only if it has not been modified since the supplied time|`412 Precondition Failed`| + +`HeadObject` uses the same validators and status codes as `GetObject`; the only difference is that the successful +response has no body. + +#### Header Combination Rules + +AWS S3 documents two precedence rules that are important for compatibility: + +- If both `If-Match` and `If-Unmodified-Since` are present, and the ETag comparison succeeds while the time check + fails, S3 still returns success. +- If both `If-None-Match` and `If-Modified-Since` are present, and the ETag comparison fails while the time check + succeeds, S3 returns `304 Not Modified`. + +Operationally, this means the ETag-based headers take precedence over the date-based headers within each pair. + +#### Additional Notes + +- Conditional evaluation happens before any response body is streamed. +- `Range` and `partNumber` do not change validator semantics; they are applied only after the conditional checks pass. +- Missing objects still follow the normal `NoSuchKey` / `404` behavior rather than returning `304` or `412`. ### AWS S3 Conditional Copy Specification -TODO +Conditional copy combines two independent sets of preconditions: + +1. **Source-side read conditions**, evaluated against the object named by `x-amz-copy-source` +2. **Destination-side write conditions**, evaluated against the destination key being written + +Both sets must pass for the copy to proceed. + +#### Source Object Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`x-amz-copy-source-if-match: "<etag>"`|Copy only if the source ETag matches|`412 Precondition Failed`| +|`x-amz-copy-source-if-none-match: "<etag>"`|Copy only if the source ETag does not match|`412 Precondition Failed`| +|`x-amz-copy-source-if-modified-since: <http-date>`|Copy only if the source was modified since the supplied time|`412 Precondition Failed`| +|`x-amz-copy-source-if-unmodified-since: <http-date>`|Copy only if the source was not modified since the supplied time|`412 Precondition Failed`| + +AWS S3 documents the same precedence pattern for source conditions as for conditional reads: + +- `x-amz-copy-source-if-match` takes precedence over `x-amz-copy-source-if-unmodified-since` +- `x-amz-copy-source-if-none-match` takes precedence over `x-amz-copy-source-if-modified-since` + +#### Destination Object Headers + +`CopyObject` also supports the destination-side write headers already used by conditional `PutObject`: + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Copy only if the destination object exists and its ETag matches|`412 Precondition Failed`| +|`If-None-Match: "*"`|Copy only if the destination object does not already exist|`412 Precondition Failed`| + +Restrictions and notes: + +- `If-Match` and `If-None-Match` must not be used together. +- Destination conditions are evaluated with the same semantics as conditional writes. +- Source and destination conditions are conjunctive: a request succeeds only when both sides pass. +- `UploadPartCopy` uses the same source-side header family and should follow the same source validation rules. + +### AWS S3 Conditional Delete Specification + +For the scope of [HDDS-14907](https://issues.apache.org/jira/browse/HDDS-14907), +we target `DeleteObject`. + +AWS also supports conditional batch delete through `DeleteObjects`, but +that is a separate API surface and is out of scope for this first step. + +#### Supported Header + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Delete only if the current ETag matches|`412 Precondition Failed`| +|`If-Match: "*"`|Delete only if the object currently exists|`412 Precondition Failed`| + +On success, `DeleteObject` still returns `204 No Content`. + +#### Scope Notes + +- Directory-bucket-only headers `x-amz-if-match-last-modified-time` and + `x-amz-if-match-size` are out of scope. +- Ozone S3 gateway currently behaves like a non-versioned object store + for this operation, so the design evaluates the current committed key + only. AWS delete-marker and object-version semantics are not modeled + here. +- Conditional delete changes only the meaning of requests that carry + `If-Match`. Unconditional delete keeps the existing S3-compatible + "delete missing key succeeds" behavior. ## Implementation ### AWS S3 Conditional Write Implementation -The implementation aims to minimize Redundant RPCs (RTT) while ensuring strict atomicity for conditional operations. - -- **If-None-Match** utilizes the atomic "Create-If-Not-Exists" capability ([HDDS-13963](https://issues.apache.org/jira/browse/HDDS-13963 "null")). -- **If-Match** optimizes the happy path by pushing ETag validation directly into the Ozone Manager's write path, avoiding preliminary read operations. +The implementation aims to minimize redundant RPCs while ensuring +strict atomicity for conditional operations. + +- **If-None-Match** utilizes the atomic "Create-If-Not-Exists" + capability ([HDDS-13963](https://issues.apache.org/jira/browse/HDDS-13963)). +- **If-Match** reuses the same open-key plus commit workflow as normal + writes, but converts the caller's ETag into + `expectedDataGeneration` during `createKey`. The gateway sends the + caller's ETag directly to OM, OM validates it, resolves the current + `updateID`, and then reuses the existing atomic rewrite mechanism from + [HDDS-10656](https://issues.apache.org/jira/browse/HDDS-10656). This + avoids an extra read RPC to fetch the current ETag and generation + before opening the key. Review Comment: Could you help remind me again what will happen if when the data uploaded by the client to the datanodes do not match to the ETag? Seems we don't compare the KeyArgs.metadata.get(ETag) in `commitKey`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
