chungen0126 commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r4489599593
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,637 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: draft +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design adds S3 Object Lock to OBS buckets, so keys can be protected from deletion and overwrite. +Authorization uses Ozone native ACLs or Apache Ranger. + +## Problem statement + +Data stored for security or compliance must be protected from accidental or malicious deletion and overwrite. Object Lock provides this protection. + +The design supports both native ACLs and Ranger. Native ACLs serve deployments without Ranger, and let the core logic be built and tested without the Ranger integration. +To ship the core feature sooner, the first release excludes versioning, Legacy buckets and FSO buckets. + +**Goal:** + +* Object Lock on OBS buckets, with authorization through native ACLs or Ranger. +* Single-version keys only. + +## Non-Goal + +* Versioning: locks across multiple object versions (versioning is still under development). +* Legacy and FSO buckets. FSO needs its own decisions on: + * Lock granularity: files and directories are in different tables (FileTable / DirectoryTable), so locks could apply to files, directories or both. + * Rename and move: they change the namespace without changing the data, so they must either be blocked on locked paths or keep the lock with the same objectID. +* Value-level Ranger authorization, for example allowing a legal hold to be set `ON` but not `OFF`. + +## Differences from AWS + +* Single version: locks apply to the current key, not to object versions. +* Overwrites are rejected: in AWS S3, Put, Copy or MPU on a locked object writes a new version. In Ozone it would replace the locked key, + so Ozone rejects Put, Copy, MPU and rename (native clients only) on locked keys with `403 Access Denied` (`WORMProtectionException`). +* Governance bypass on overwrite and rename: a Governance-locked key can only be overwritten or renamed with the `bypassGovernanceRetention` flag and the bypass permission. + AWS S3 does not need this because it writes a new version (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). +* Authorization: Ozone has no root account, so it uses the owner or the Ozone administrator role instead. + * Native ACLs: changing locks and bypassing Governance need the volume or bucket owner; every change to a Compliance retention needs an Ozone administrator (see [Native ACLs](#1-native-acls)). + * Ranger: action-level permissions need `enableActionMatcherInPoliciesCondition`, which is `false` by default (see [Ranger](#2-ranger)). + Object Lock uses the same access types as data (`READ` / `WRITE`), so without `action-matches`, anyone who can write a key can also change its locks. +* Releasing a Compliance retention: in AWS S3 no one can shorten or remove it. Ozone lets an Ozone administrator do it through `PutObjectRetention`, in both models + (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +## Technical Description + +### Terminology + +**Legal Hold** + +* Locks a key with no end date, until someone allowed to change the legal hold removes it + (the owner with native ACLs; `WRITE` with Ranger, limited to `PutObjectLegalHold` when `action-matches` is used). +* Used for litigation, investigations or audits, when it is not known how long the data must be kept. +* Has no mode and no date, and is independent of retention: a key can have a legal hold, a retention, or both, and stays locked until both are gone. + A legal hold can be set on a key without a retention. +* Blocks: Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients). +* Allows: Get / Head, Get / Put Object Tagging, Get Legal Hold, Put Legal Hold (with permission). + +**Retention** + +* Protects a key until a `RetainUntilDate`. The date is either fixed (set directly, or from days / years) or comes from an event hold. +* Modes: + * Compliance: no one, including administrators, can overwrite or delete the key before it expires. Only an Ozone administrator can shorten it, remove it or change it to Governance. + * Governance: blocks normal users, but users with the bypass permission can bypass it. +* Event hold (AWS S3 *variable retention*): + * Used when the retention depends on an external event, such as a contract ending, an employee leaving or a loan closing. + * While the event has not happened (`EventHold=ON`), the key is locked with no end date. When it happens, the hold is released (`EventHold=OFF`) + and the duration (`EventHoldDuration`, in days or years) becomes a fixed `RetainUntilDate`. + * See [Event Hold Trigger Workflow](#event-hold-trigger-workflow) for the rules. +* Blocks (Governance allows them with `bypassGovernanceRetention`): Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients), + shortening or removing the retention (Compliance: only an Ozone administrator). +* Allows: Get / Head, Get / Put Object Tagging, extending the retention. + +#### Lock Settings by Resource + +| Setting | Bucket default (`PutObjectLockConfiguration`) | Key (`PutObjectRetention`, `PutObjectLegalHold`, `x-amz-object-lock-*` headers) | +|:-----------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------------| +| Mode | `GOVERNANCE` or `COMPLIANCE` | `GOVERNANCE` or `COMPLIANCE` | +| Fixed date | A duration in days or years; a new key gets creation time + duration | `RetainUntilDate` | +| Event hold | `DefaultEventHold` has only a duration, no status; a new key gets the hold `ON` with that duration | `EventHold` `ON` / `OFF` with `EventHoldDuration` | +| Legal hold | Not available | `ON` / `OFF` | + +A bucket default has no `RetainUntilDate` and no legal hold. As in AWS S3 ([DefaultRetention](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-s3-bucket-defaultretention.html)), +a default with both a duration and `DefaultEventHold` is only valid when the event hold duration does not exceed the default duration (`400 InvalidRequest` otherwise). +A default with only `DefaultEventHold` is valid. + +How the settings combine on a key: +* A key is locked while `legalHold = ON`, while its event hold is `ON`, or while its effective `RetainUntilDate` is in the future. Legal hold and retention are independent, so all three must be gone before the key is unlocked. +* `x-amz-object-lock-*` headers at creation replace the bucket default entirely; without them, the key gets the bucket default (mode, `retentionDate` = creation time + duration, and the event hold). +* Effective `RetainUntilDate`: + * Fixed retention: `retentionDate` + * Event hold `ON`: `max(retentionDate, now + EventHoldDuration)`, so it moves forward over time + * Release without `RetainUntilDate`: `retentionDate = max(retentionDate, release time + EventHoldDuration)` + * Release with `RetainUntilDate`: `retentionDate = requested RetainUntilDate` (earlier than the effective date only with the [release permission](#event-hold-trigger-workflow)) + * Shorter `EventHoldDuration` while `ON`: `retentionDate = max(retentionDate, now + old duration)` first, then the new duration applies + +The table, the combination rules and the formulas were verified against AWS S3 (us-east-1, 2026-10-28) in Governance mode. + +#### Event Hold Trigger Workflow + +```text +object created + | +event hold applied (locked with no end date) + | +business event occurs + | +hold released + | +RetainUntilDate is calculated + | +locked until RetainUntilDate +``` + +Event holds use `PutObjectRetention`, as in AWS S3, so no new API or Ranger action is needed (native clients use `OzoneBucket#setRetention`). +OM handles the request under the key write lock and uses the leader-stamped request time as now. + +The key stores `retentionDate` and the event hold (status and duration): +* The duration comes from the request, or from the bucket default when the key is created. Later changes to the bucket default do not affect existing keys. +* If the bucket default has both `duration` and `defaultEventHold`, a new key gets `retentionDate` = creation time + `duration` and an active event hold. + With only `defaultEventHold`, `retentionDate` is unset until the hold is released. +* While the hold is `ON`, the effective `RetainUntilDate` is the later of `retentionDate` and now + duration, so it moves forward over time. + `GetObjectRetention`, `HeadObject` and `GetObject` return this date. + +Below, "earlier" and "later" compare the requested `RetainUntilDate` with the effective date. Making it earlier needs the **release permission**: +the bypass flag and the bypass permission for `GOVERNANCE`, and an Ozone administrator for `COMPLIANCE` (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +1. **Request format**: checked before the lock and the bypass; a request that breaks a rule gets `400 Bad Request` + (`InvalidRequest`; `InvalidArgument` for a `RetainUntilDate` before now + duration; `MalformedXML` for a retention without a mode). + * The request has a mode, and `RetainUntilDate`, `EventHold` or both. + The only exception is an empty retention, which removes the retention: it needs the release permission (`403` without it). + * `RetainUntilDate` is in the future. + * `EventHold=ON` has exactly one `EventHoldDuration` (positive, at most 36,500 days or 100 years). A `RetainUntilDate` in the same request is at least now + duration. + * `EventHold=OFF` has no `EventHoldDuration`, and `EventHoldDuration` needs `EventHold=ON`. + * The same applies to the `x-amz-object-lock-*` headers at creation: the mode header needs the retain-until-date header or the event hold headers, and they need the mode header. +2. **`EventHold=ON`** sets or updates the hold. + * Without the release permission, `RetainUntilDate` can only be later (`403` otherwise). The duration can change; + when it gets shorter, OM first saves the current effective date in `retentionDate`, so the date never moves earlier, even with the release permission. + * With the release permission, `RetainUntilDate` can be earlier, but still at least now + the new duration (rule 1). +3. **`EventHold=OFF`** releases the hold. It is the only way to release it. + * Without `RetainUntilDate`: `retentionDate` = the later of `retentionDate` and now + duration. + * With `RetainUntilDate`: that date is used. An earlier date needs the release permission, otherwise `403`. + * On a key without an active hold: no change if the key has a future `retentionDate`; `400` if neither the key nor the request has a future date. +4. **No `EventHold`** keeps the hold as it is and only changes `RetainUntilDate`. + * A later date is accepted, also while the hold is `ON`. + * An earlier date needs the release permission, otherwise `403`. While the hold is `ON`, it gets `400` even with the release permission: the request must use rule 2 or 3. + +Releasing the hold keeps the mode. A different mode in the request is a mode change (see [Retention Updates](#retention-updates)). + +> _**Note**_: AWS S3 requires versioning for Object Lock, so a Put on a locked object only adds a new version and Object Lock mainly blocks deletes. +> Ozone has no versioning yet, so it rejects every overwrite of a locked key. + +#### Retention Updates + +`PutObjectRetention` on a key that already has a retention follows AWS S3 (see [Event Hold Trigger Workflow](#event-hold-trigger-workflow)). +"Longer" and "shorter" compare the requested `RetainUntilDate` with the current effective date: + +| Change | `GOVERNANCE` | `COMPLIANCE` | +|:------------------------------------------------------------------------------------|:-------------------------------------------------|:-------------------------------------| +| Longer retention (later `RetainUntilDate`, longer event hold duration, `EventHold=ON`) | Allowed | Allowed | +| Shorter event hold duration while the hold is `ON` | Allowed; the effective date is kept | Allowed; the effective date is kept | +| Shorter retention (earlier `RetainUntilDate`, also on release) or removal (an empty retention) | Needs the bypass; while the hold is `ON`, an earlier `RetainUntilDate` must come with `EventHold` | Ozone administrator only; the same `EventHold` rule applies | +| Release the event hold (`EventHold=OFF` without `RetainUntilDate`) | Allowed | Allowed | +| Change the mode | `GOVERNANCE` → `COMPLIANCE`: allowed | `COMPLIANCE` → `GOVERNANCE`: Ozone administrator only | + +Who may send `PutObjectRetention` depends on the model: see [Native ACLs](#1-native-acls) and [Ranger](#2-ranger). + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + optional bool objectLockEnabled = 24 [default = false]; + optional BucketRetention defaultRetention = 25; + } + + message BucketRetention { + required RetentionMode retentionMode = 1; + optional RetentionDuration duration = 2; + optional RetentionDuration defaultEventHold = 3; // only a duration: new keys start with the hold ON + } + + message RetentionDuration { + required TimeUnit timeUnit = 1; + required uint64 duration = 2; + } + + enum TimeUnit { + DAYS = 1; + YEARS = 2; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +Two new fields: retention & legalHold. + +```protobuf +message KeyInfo { + // ... existing fields + optional KeyRetention retention = 23; + optional bool legalHold = 24 [default = false]; +} + +message KeyRetention { + required RetentionMode retentionMode = 1; + optional uint64 retentionDate = 2; // epoch milliseconds; the minimum date while the event hold is ON + optional EventHold eventHold = 3; +} + +message EventHold { + required bool enabled = 1; Review Comment: Sure, I decided to set the default to true. I think this better fits the use case. When an event hold is configured in the default retention settings, any object created under this bucket will have the event hold enabled by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
