fmorg-git commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r4449254297
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,637 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: draft +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design adds S3 Object Lock to OBS buckets, so keys can be protected from deletion and overwrite. +Authorization uses Ozone native ACLs or Apache Ranger. + +## Problem statement + +Data stored for security or compliance must be protected from accidental or malicious deletion and overwrite. Object Lock provides this protection. + +The design supports both native ACLs and Ranger. Native ACLs serve deployments without Ranger, and let the core logic be built and tested without the Ranger integration. +To ship the core feature sooner, the first release excludes versioning, Legacy buckets and FSO buckets. + +**Goal:** + +* Object Lock on OBS buckets, with authorization through native ACLs or Ranger. +* Single-version keys only. + +## Non-Goal + +* Versioning: locks across multiple object versions (versioning is still under development). +* Legacy and FSO buckets. FSO needs its own decisions on: + * Lock granularity: files and directories are in different tables (FileTable / DirectoryTable), so locks could apply to files, directories or both. + * Rename and move: they change the namespace without changing the data, so they must either be blocked on locked paths or keep the lock with the same objectID. +* Value-level Ranger authorization, for example allowing a legal hold to be set `ON` but not `OFF`. + +## Differences from AWS + +* Single version: locks apply to the current key, not to object versions. +* Overwrites are rejected: in AWS S3, Put, Copy or MPU on a locked object writes a new version. In Ozone it would replace the locked key, + so Ozone rejects Put, Copy, MPU and rename (native clients only) on locked keys with `403 Access Denied` (`WORMProtectionException`). +* Governance bypass on overwrite and rename: a Governance-locked key can only be overwritten or renamed with the `bypassGovernanceRetention` flag and the bypass permission. + AWS S3 does not need this because it writes a new version (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). +* Authorization: Ozone has no root account, so it uses the owner or the Ozone administrator role instead. + * Native ACLs: changing locks and bypassing Governance need the volume or bucket owner; every change to a Compliance retention needs an Ozone administrator (see [Native ACLs](#1-native-acls)). + * Ranger: action-level permissions need `enableActionMatcherInPoliciesCondition`, which is `false` by default (see [Ranger](#2-ranger)). + Object Lock uses the same access types as data (`READ` / `WRITE`), so without `action-matches`, anyone who can write a key can also change its locks. +* Releasing a Compliance retention: in AWS S3 no one can shorten or remove it. Ozone lets an Ozone administrator do it through `PutObjectRetention`, in both models + (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +## Technical Description + +### Terminology + +**Legal Hold** + +* Locks a key with no end date, until someone allowed to change the legal hold removes it + (the owner with native ACLs; `WRITE` with Ranger, limited to `PutObjectLegalHold` when `action-matches` is used). +* Used for litigation, investigations or audits, when it is not known how long the data must be kept. +* Has no mode and no date, and is independent of retention: a key can have a legal hold, a retention, or both, and stays locked until both are gone. + A legal hold can be set on a key without a retention. +* Blocks: Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients). +* Allows: Get / Head, Get / Put Object Tagging, Get Legal Hold, Put Legal Hold (with permission). + +**Retention** + +* Protects a key until a `RetainUntilDate`. The date is either fixed (set directly, or from days / years) or comes from an event hold. +* Modes: + * Compliance: no one, including administrators, can overwrite or delete the key before it expires. Only an Ozone administrator can shorten it, remove it or change it to Governance. + * Governance: blocks normal users, but users with the bypass permission can bypass it. +* Event hold (AWS S3 *variable retention*): + * Used when the retention depends on an external event, such as a contract ending, an employee leaving or a loan closing. + * While the event has not happened (`EventHold=ON`), the key is locked with no end date. When it happens, the hold is released (`EventHold=OFF`) + and the duration (`EventHoldDuration`, in days or years) becomes a fixed `RetainUntilDate`. + * See [Event Hold Trigger Workflow](#event-hold-trigger-workflow) for the rules. +* Blocks (Governance allows them with `bypassGovernanceRetention`): Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients), + shortening or removing the retention (Compliance: only an Ozone administrator). +* Allows: Get / Head, Get / Put Object Tagging, extending the retention. + +#### Lock Settings by Resource + +| Setting | Bucket default (`PutObjectLockConfiguration`) | Key (`PutObjectRetention`, `PutObjectLegalHold`, `x-amz-object-lock-*` headers) | +|:-----------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------------| +| Mode | `GOVERNANCE` or `COMPLIANCE` | `GOVERNANCE` or `COMPLIANCE` | +| Fixed date | A duration in days or years; a new key gets creation time + duration | `RetainUntilDate` | +| Event hold | `DefaultEventHold` has only a duration, no status; a new key gets the hold `ON` with that duration | `EventHold` `ON` / `OFF` with `EventHoldDuration` | +| Legal hold | Not available | `ON` / `OFF` | + +A bucket default has no `RetainUntilDate` and no legal hold. As in AWS S3 ([DefaultRetention](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-s3-bucket-defaultretention.html)), +a default with both a duration and `DefaultEventHold` is only valid when the event hold duration does not exceed the default duration (`400 InvalidRequest` otherwise). +A default with only `DefaultEventHold` is valid. + +How the settings combine on a key: +* A key is locked while `legalHold = ON`, while its event hold is `ON`, or while its effective `RetainUntilDate` is in the future. Legal hold and retention are independent, so all three must be gone before the key is unlocked. +* `x-amz-object-lock-*` headers at creation replace the bucket default entirely; without them, the key gets the bucket default (mode, `retentionDate` = creation time + duration, and the event hold). +* Effective `RetainUntilDate`: + * Fixed retention: `retentionDate` + * Event hold `ON`: `max(retentionDate, now + EventHoldDuration)`, so it moves forward over time + * Release without `RetainUntilDate`: `retentionDate = max(retentionDate, release time + EventHoldDuration)` + * Release with `RetainUntilDate`: `retentionDate = requested RetainUntilDate` (earlier than the effective date only with the [release permission](#event-hold-trigger-workflow)) + * Shorter `EventHoldDuration` while `ON`: `retentionDate = max(retentionDate, now + old duration)` first, then the new duration applies + +The table, the combination rules and the formulas were verified against AWS S3 (us-east-1, 2026-10-28) in Governance mode. + +#### Event Hold Trigger Workflow + +```text +object created + | +event hold applied (locked with no end date) + | +business event occurs + | +hold released + | +RetainUntilDate is calculated + | +locked until RetainUntilDate +``` + +Event holds use `PutObjectRetention`, as in AWS S3, so no new API or Ranger action is needed (native clients use `OzoneBucket#setRetention`). +OM handles the request under the key write lock and uses the leader-stamped request time as now. + +The key stores `retentionDate` and the event hold (status and duration): +* The duration comes from the request, or from the bucket default when the key is created. Later changes to the bucket default do not affect existing keys. +* If the bucket default has both `duration` and `defaultEventHold`, a new key gets `retentionDate` = creation time + `duration` and an active event hold. + With only `defaultEventHold`, `retentionDate` is unset until the hold is released. +* While the hold is `ON`, the effective `RetainUntilDate` is the later of `retentionDate` and now + duration, so it moves forward over time. + `GetObjectRetention`, `HeadObject` and `GetObject` return this date. + +Below, "earlier" and "later" compare the requested `RetainUntilDate` with the effective date. Making it earlier needs the **release permission**: +the bypass flag and the bypass permission for `GOVERNANCE`, and an Ozone administrator for `COMPLIANCE` (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +1. **Request format**: checked before the lock and the bypass; a request that breaks a rule gets `400 Bad Request` + (`InvalidRequest`; `InvalidArgument` for a `RetainUntilDate` before now + duration; `MalformedXML` for a retention without a mode). + * The request has a mode, and `RetainUntilDate`, `EventHold` or both. + The only exception is an empty retention, which removes the retention: it needs the release permission (`403` without it). + * `RetainUntilDate` is in the future. + * `EventHold=ON` has exactly one `EventHoldDuration` (positive, at most 36,500 days or 100 years). A `RetainUntilDate` in the same request is at least now + duration. + * `EventHold=OFF` has no `EventHoldDuration`, and `EventHoldDuration` needs `EventHold=ON`. + * The same applies to the `x-amz-object-lock-*` headers at creation: the mode header needs the retain-until-date header or the event hold headers, and they need the mode header. +2. **`EventHold=ON`** sets or updates the hold. + * Without the release permission, `RetainUntilDate` can only be later (`403` otherwise). The duration can change; + when it gets shorter, OM first saves the current effective date in `retentionDate`, so the date never moves earlier, even with the release permission. + * With the release permission, `RetainUntilDate` can be earlier, but still at least now + the new duration (rule 1). +3. **`EventHold=OFF`** releases the hold. It is the only way to release it. + * Without `RetainUntilDate`: `retentionDate` = the later of `retentionDate` and now + duration. + * With `RetainUntilDate`: that date is used. An earlier date needs the release permission, otherwise `403`. + * On a key without an active hold: no change if the key has a future `retentionDate`; `400` if neither the key nor the request has a future date. +4. **No `EventHold`** keeps the hold as it is and only changes `RetainUntilDate`. + * A later date is accepted, also while the hold is `ON`. + * An earlier date needs the release permission, otherwise `403`. While the hold is `ON`, it gets `400` even with the release permission: the request must use rule 2 or 3. + +Releasing the hold keeps the mode. A different mode in the request is a mode change (see [Retention Updates](#retention-updates)). + +> _**Note**_: AWS S3 requires versioning for Object Lock, so a Put on a locked object only adds a new version and Object Lock mainly blocks deletes. +> Ozone has no versioning yet, so it rejects every overwrite of a locked key. + +#### Retention Updates + +`PutObjectRetention` on a key that already has a retention follows AWS S3 (see [Event Hold Trigger Workflow](#event-hold-trigger-workflow)). +"Longer" and "shorter" compare the requested `RetainUntilDate` with the current effective date: + +| Change | `GOVERNANCE` | `COMPLIANCE` | +|:------------------------------------------------------------------------------------|:-------------------------------------------------|:-------------------------------------| +| Longer retention (later `RetainUntilDate`, longer event hold duration, `EventHold=ON`) | Allowed | Allowed | +| Shorter event hold duration while the hold is `ON` | Allowed; the effective date is kept | Allowed; the effective date is kept | +| Shorter retention (earlier `RetainUntilDate`, also on release) or removal (an empty retention) | Needs the bypass; while the hold is `ON`, an earlier `RetainUntilDate` must come with `EventHold` | Ozone administrator only; the same `EventHold` rule applies | +| Release the event hold (`EventHold=OFF` without `RetainUntilDate`) | Allowed | Allowed | +| Change the mode | `GOVERNANCE` → `COMPLIANCE`: allowed | `COMPLIANCE` → `GOVERNANCE`: Ozone administrator only | + +Who may send `PutObjectRetention` depends on the model: see [Native ACLs](#1-native-acls) and [Ranger](#2-ranger). + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + optional bool objectLockEnabled = 24 [default = false]; + optional BucketRetention defaultRetention = 25; + } + + message BucketRetention { + required RetentionMode retentionMode = 1; + optional RetentionDuration duration = 2; + optional RetentionDuration defaultEventHold = 3; // only a duration: new keys start with the hold ON + } + + message RetentionDuration { + required TimeUnit timeUnit = 1; + required uint64 duration = 2; + } + + enum TimeUnit { + DAYS = 1; + YEARS = 2; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +Two new fields: retention & legalHold. + +```protobuf +message KeyInfo { + // ... existing fields + optional KeyRetention retention = 23; + optional bool legalHold = 24 [default = false]; +} + +message KeyRetention { + required RetentionMode retentionMode = 1; + optional uint64 retentionDate = 2; // epoch milliseconds; the minimum date while the event hold is ON + optional EventHold eventHold = 3; +} + +message EventHold { + required bool enabled = 1; Review Comment: should EventHold.enabled have a default, similar to how KeyInfo.eventHold has a default? ########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,637 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: draft +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design adds S3 Object Lock to OBS buckets, so keys can be protected from deletion and overwrite. +Authorization uses Ozone native ACLs or Apache Ranger. + +## Problem statement + +Data stored for security or compliance must be protected from accidental or malicious deletion and overwrite. Object Lock provides this protection. + +The design supports both native ACLs and Ranger. Native ACLs serve deployments without Ranger, and let the core logic be built and tested without the Ranger integration. +To ship the core feature sooner, the first release excludes versioning, Legacy buckets and FSO buckets. + +**Goal:** + +* Object Lock on OBS buckets, with authorization through native ACLs or Ranger. +* Single-version keys only. + +## Non-Goal + +* Versioning: locks across multiple object versions (versioning is still under development). +* Legacy and FSO buckets. FSO needs its own decisions on: + * Lock granularity: files and directories are in different tables (FileTable / DirectoryTable), so locks could apply to files, directories or both. + * Rename and move: they change the namespace without changing the data, so they must either be blocked on locked paths or keep the lock with the same objectID. +* Value-level Ranger authorization, for example allowing a legal hold to be set `ON` but not `OFF`. + +## Differences from AWS + +* Single version: locks apply to the current key, not to object versions. +* Overwrites are rejected: in AWS S3, Put, Copy or MPU on a locked object writes a new version. In Ozone it would replace the locked key, + so Ozone rejects Put, Copy, MPU and rename (native clients only) on locked keys with `403 Access Denied` (`WORMProtectionException`). +* Governance bypass on overwrite and rename: a Governance-locked key can only be overwritten or renamed with the `bypassGovernanceRetention` flag and the bypass permission. + AWS S3 does not need this because it writes a new version (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). +* Authorization: Ozone has no root account, so it uses the owner or the Ozone administrator role instead. + * Native ACLs: changing locks and bypassing Governance need the volume or bucket owner; every change to a Compliance retention needs an Ozone administrator (see [Native ACLs](#1-native-acls)). + * Ranger: action-level permissions need `enableActionMatcherInPoliciesCondition`, which is `false` by default (see [Ranger](#2-ranger)). + Object Lock uses the same access types as data (`READ` / `WRITE`), so without `action-matches`, anyone who can write a key can also change its locks. +* Releasing a Compliance retention: in AWS S3 no one can shorten or remove it. Ozone lets an Ozone administrator do it through `PutObjectRetention`, in both models + (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +## Technical Description + +### Terminology + +**Legal Hold** + +* Locks a key with no end date, until someone allowed to change the legal hold removes it + (the owner with native ACLs; `WRITE` with Ranger, limited to `PutObjectLegalHold` when `action-matches` is used). +* Used for litigation, investigations or audits, when it is not known how long the data must be kept. +* Has no mode and no date, and is independent of retention: a key can have a legal hold, a retention, or both, and stays locked until both are gone. + A legal hold can be set on a key without a retention. +* Blocks: Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients). +* Allows: Get / Head, Get / Put Object Tagging, Get Legal Hold, Put Legal Hold (with permission). + +**Retention** + +* Protects a key until a `RetainUntilDate`. The date is either fixed (set directly, or from days / years) or comes from an event hold. +* Modes: + * Compliance: no one, including administrators, can overwrite or delete the key before it expires. Only an Ozone administrator can shorten it, remove it or change it to Governance. + * Governance: blocks normal users, but users with the bypass permission can bypass it. +* Event hold (AWS S3 *variable retention*): + * Used when the retention depends on an external event, such as a contract ending, an employee leaving or a loan closing. + * While the event has not happened (`EventHold=ON`), the key is locked with no end date. When it happens, the hold is released (`EventHold=OFF`) + and the duration (`EventHoldDuration`, in days or years) becomes a fixed `RetainUntilDate`. + * See [Event Hold Trigger Workflow](#event-hold-trigger-workflow) for the rules. +* Blocks (Governance allows them with `bypassGovernanceRetention`): Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients), + shortening or removing the retention (Compliance: only an Ozone administrator). +* Allows: Get / Head, Get / Put Object Tagging, extending the retention. + +#### Lock Settings by Resource + +| Setting | Bucket default (`PutObjectLockConfiguration`) | Key (`PutObjectRetention`, `PutObjectLegalHold`, `x-amz-object-lock-*` headers) | +|:-----------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------------| +| Mode | `GOVERNANCE` or `COMPLIANCE` | `GOVERNANCE` or `COMPLIANCE` | +| Fixed date | A duration in days or years; a new key gets creation time + duration | `RetainUntilDate` | +| Event hold | `DefaultEventHold` has only a duration, no status; a new key gets the hold `ON` with that duration | `EventHold` `ON` / `OFF` with `EventHoldDuration` | +| Legal hold | Not available | `ON` / `OFF` | + +A bucket default has no `RetainUntilDate` and no legal hold. As in AWS S3 ([DefaultRetention](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-s3-bucket-defaultretention.html)), +a default with both a duration and `DefaultEventHold` is only valid when the event hold duration does not exceed the default duration (`400 InvalidRequest` otherwise). +A default with only `DefaultEventHold` is valid. + +How the settings combine on a key: +* A key is locked while `legalHold = ON`, while its event hold is `ON`, or while its effective `RetainUntilDate` is in the future. Legal hold and retention are independent, so all three must be gone before the key is unlocked. +* `x-amz-object-lock-*` headers at creation replace the bucket default entirely; without them, the key gets the bucket default (mode, `retentionDate` = creation time + duration, and the event hold). +* Effective `RetainUntilDate`: + * Fixed retention: `retentionDate` + * Event hold `ON`: `max(retentionDate, now + EventHoldDuration)`, so it moves forward over time + * Release without `RetainUntilDate`: `retentionDate = max(retentionDate, release time + EventHoldDuration)` + * Release with `RetainUntilDate`: `retentionDate = requested RetainUntilDate` (earlier than the effective date only with the [release permission](#event-hold-trigger-workflow)) + * Shorter `EventHoldDuration` while `ON`: `retentionDate = max(retentionDate, now + old duration)` first, then the new duration applies + +The table, the combination rules and the formulas were verified against AWS S3 (us-east-1, 2026-10-28) in Governance mode. + +#### Event Hold Trigger Workflow + +```text +object created + | +event hold applied (locked with no end date) + | +business event occurs + | +hold released + | +RetainUntilDate is calculated + | +locked until RetainUntilDate +``` + +Event holds use `PutObjectRetention`, as in AWS S3, so no new API or Ranger action is needed (native clients use `OzoneBucket#setRetention`). +OM handles the request under the key write lock and uses the leader-stamped request time as now. + +The key stores `retentionDate` and the event hold (status and duration): +* The duration comes from the request, or from the bucket default when the key is created. Later changes to the bucket default do not affect existing keys. +* If the bucket default has both `duration` and `defaultEventHold`, a new key gets `retentionDate` = creation time + `duration` and an active event hold. + With only `defaultEventHold`, `retentionDate` is unset until the hold is released. +* While the hold is `ON`, the effective `RetainUntilDate` is the later of `retentionDate` and now + duration, so it moves forward over time. + `GetObjectRetention`, `HeadObject` and `GetObject` return this date. + +Below, "earlier" and "later" compare the requested `RetainUntilDate` with the effective date. Making it earlier needs the **release permission**: +the bypass flag and the bypass permission for `GOVERNANCE`, and an Ozone administrator for `COMPLIANCE` (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +1. **Request format**: checked before the lock and the bypass; a request that breaks a rule gets `400 Bad Request` + (`InvalidRequest`; `InvalidArgument` for a `RetainUntilDate` before now + duration; `MalformedXML` for a retention without a mode). + * The request has a mode, and `RetainUntilDate`, `EventHold` or both. + The only exception is an empty retention, which removes the retention: it needs the release permission (`403` without it). + * `RetainUntilDate` is in the future. + * `EventHold=ON` has exactly one `EventHoldDuration` (positive, at most 36,500 days or 100 years). A `RetainUntilDate` in the same request is at least now + duration. + * `EventHold=OFF` has no `EventHoldDuration`, and `EventHoldDuration` needs `EventHold=ON`. + * The same applies to the `x-amz-object-lock-*` headers at creation: the mode header needs the retain-until-date header or the event hold headers, and they need the mode header. +2. **`EventHold=ON`** sets or updates the hold. + * Without the release permission, `RetainUntilDate` can only be later (`403` otherwise). The duration can change; + when it gets shorter, OM first saves the current effective date in `retentionDate`, so the date never moves earlier, even with the release permission. + * With the release permission, `RetainUntilDate` can be earlier, but still at least now + the new duration (rule 1). +3. **`EventHold=OFF`** releases the hold. It is the only way to release it. + * Without `RetainUntilDate`: `retentionDate` = the later of `retentionDate` and now + duration. + * With `RetainUntilDate`: that date is used. An earlier date needs the release permission, otherwise `403`. + * On a key without an active hold: no change if the key has a future `retentionDate`; `400` if neither the key nor the request has a future date. +4. **No `EventHold`** keeps the hold as it is and only changes `RetainUntilDate`. + * A later date is accepted, also while the hold is `ON`. + * An earlier date needs the release permission, otherwise `403`. While the hold is `ON`, it gets `400` even with the release permission: the request must use rule 2 or 3. + +Releasing the hold keeps the mode. A different mode in the request is a mode change (see [Retention Updates](#retention-updates)). + +> _**Note**_: AWS S3 requires versioning for Object Lock, so a Put on a locked object only adds a new version and Object Lock mainly blocks deletes. +> Ozone has no versioning yet, so it rejects every overwrite of a locked key. + +#### Retention Updates + +`PutObjectRetention` on a key that already has a retention follows AWS S3 (see [Event Hold Trigger Workflow](#event-hold-trigger-workflow)). +"Longer" and "shorter" compare the requested `RetainUntilDate` with the current effective date: + +| Change | `GOVERNANCE` | `COMPLIANCE` | +|:------------------------------------------------------------------------------------|:-------------------------------------------------|:-------------------------------------| +| Longer retention (later `RetainUntilDate`, longer event hold duration, `EventHold=ON`) | Allowed | Allowed | +| Shorter event hold duration while the hold is `ON` | Allowed; the effective date is kept | Allowed; the effective date is kept | +| Shorter retention (earlier `RetainUntilDate`, also on release) or removal (an empty retention) | Needs the bypass; while the hold is `ON`, an earlier `RetainUntilDate` must come with `EventHold` | Ozone administrator only; the same `EventHold` rule applies | +| Release the event hold (`EventHold=OFF` without `RetainUntilDate`) | Allowed | Allowed | +| Change the mode | `GOVERNANCE` → `COMPLIANCE`: allowed | `COMPLIANCE` → `GOVERNANCE`: Ozone administrator only | + +Who may send `PutObjectRetention` depends on the model: see [Native ACLs](#1-native-acls) and [Ranger](#2-ranger). + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + optional bool objectLockEnabled = 24 [default = false]; + optional BucketRetention defaultRetention = 25; + } + + message BucketRetention { + required RetentionMode retentionMode = 1; + optional RetentionDuration duration = 2; + optional RetentionDuration defaultEventHold = 3; // only a duration: new keys start with the hold ON + } + + message RetentionDuration { + required TimeUnit timeUnit = 1; + required uint64 duration = 2; + } + + enum TimeUnit { + DAYS = 1; + YEARS = 2; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +Two new fields: retention & legalHold. + +```protobuf +message KeyInfo { + // ... existing fields + optional KeyRetention retention = 23; + optional bool legalHold = 24 [default = false]; +} + +message KeyRetention { + required RetentionMode retentionMode = 1; + optional uint64 retentionDate = 2; // epoch milliseconds; the minimum date while the event hold is ON + optional EventHold eventHold = 3; +} + +message EventHold { + required bool enabled = 1; + required RetentionDuration duration = 2; +} +``` + + + +### Access Control + +A request on a locked key must pass two checks: authorization (native ACLs or Ranger) and the WORM check in OM. +Even with `CREATE`, `WRITE` or `DELETE`, overwriting or deleting a locked key is rejected with `403 Access Denied` (`WORMProtectionException`), +unless the Governance bypass applies (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). + +The cluster uses one of two authorization models: +* [Native ACLs](#1-native-acls) have no per-action permissions, so changing locks needs the owner, and every change to a Compliance retention needs an Ozone administrator. +* [Ranger](#2-ranger) checks access types, and `action-matches` can limit a policy item to specific S3 actions. Use Ranger when each Object Lock operation needs its own permission. + +In both models, releasing a Compliance retention needs an Ozone administrator (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)), +and keys that only get the bucket default retention need no extra check, because the default was authorized when it was set. +Native clients pass the same settings and the `bypassGovernanceRetention` flag as request arguments, and the same rules apply. + +--- + +#### 1. Native ACLs + +| Operation | Resource | Required | +|:----------------------------------------------------------------------------------|:-------------|:--------------------| +| `GetObjectLockConfiguration` | Bucket | `READ` | +| `GetObjectRetention`, `GetObjectLegalHold` | Key | `READ` | +| `PutObjectLockConfiguration` (no default, or `GOVERNANCE`) | Bucket | Owner | +| `PutObjectRetention` on a `GOVERNANCE` key or a key without retention (setting `GOVERNANCE`), `PutObjectLegalHold` | Key | Owner | +| Governance bypass (including shortening or removing a `GOVERNANCE` retention) | Key | Owner | +| `COMPLIANCE`: setting it on a key, `GOVERNANCE` → `COMPLIANCE`, any `PutObjectRetention` on a `COMPLIANCE` key (extending it, shortening, removing or downgrading it, or releasing its event hold), or as the bucket default (also changing a `COMPLIANCE` default) | Bucket / Key | Ozone administrator | + +* **Owner**: the owner of the volume or bucket, the same owner that `OzoneNativeAuthorizer` already lets skip ACL checks. Ozone administrators can do everything an owner can. + Locks use the owner instead of an ACL type like `WRITE_ACL`, because any ACL type can be granted to other users. +* **Ozone administrator**: a user in `ozone.administrators` or a group in `ozone.administrators.groups` (if neither is set, the user that starts OM). + Read-only and S3 administrators are not included. The role is cluster wide; use Ranger to delegate Compliance per bucket. +* Reading lock information only needs `READ`, because `HeadObject` / `GetObject` already return it in the `x-amz-object-lock-*` headers. +* Every `PutObjectRetention` on a Compliance key needs an administrator: setting it, extending it (a later date or a longer event hold duration), releasing its event hold, + and shortening, removing or downgrading it. One rule for Compliance is easier to apply than exceptions for an event hold release with or without a `RetainUntilDate`. + Without it, an owner could lock a key for up to 100 years that only an administrator can unlock. This is stricter than AWS S3, where `PutObjectRetention` is enough. +* Keys that get a Compliance bucket default need no administrator check. To store Compliance data, an administrator sets `COMPLIANCE` as the bucket default once, + and applications upload as normal users. + +##### Extra Checks for Object Lock Headers + +Requests that carry Object Lock headers also need the owner or an administrator, checked in the same request after the normal ACL check: + +| Request | Header or condition | Extra check | +|:--------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------|:---------------------------------------------------------| +| `CreateBucket` | `x-amz-bucket-object-lock-enabled: true` | Owner | +| `PutObject`, `CopyObject` (destination), `CreateMultipartUpload` | `x-amz-object-lock-mode`, `-retain-until-date`, `-event-hold`, `-event-hold-duration-days` / `-years` | Owner (`GOVERNANCE`), Ozone administrator (`COMPLIANCE`) | +| `PutObject`, `CopyObject` (destination), `CreateMultipartUpload` | `x-amz-object-lock-legal-hold` | Owner | +| `DeleteObject`, `DeleteObjects`, overwrite, rename (native clients) | `bypassGovernanceRetention` | Owner | + +If the check fails, the request fails (`DeleteObjects` reports a per-key `AccessDenied`). `GetObject` / `HeadObject` need no extra check and always return the lock headers. +So a user who can only write data cannot set a lock by adding a header. + +--- + +#### 2. Ranger + +> _**Important**_: action-level permissions need the Ranger action matcher, which is `false` by default. +> Without it, every principal with `WRITE` on a key can also change its locks, including setting a Compliance retention. +> ```properties +> ranger.servicedef.ozone.enableActionMatcherInPoliciesCondition=true +> ``` + +* Object Lock uses the same access types as data: `READ` to read lock information, `WRITE` to change it, and `WRITE` and `DELETE` for the Governance bypass. + They apply to every client: S3 Gateway, ofs / o3fs and the Ozone CLI. + There is no separate "remove lock" operation: shortening or removing a Governance retention is a `PutObjectRetention` that also needs `BypassGovernanceRetention` + (see [Dependent Permissions](#dependent-permissions)), so a principal with `WRITE` alone can set or extend locks but not shorten or remove a Governance retention. +* Setting Compliance only needs `PutObjectRetention` (`WRITE`), as in AWS S3, and only `action-matches` can limit who sets it, so a Ranger policy can delegate it to a team for specific buckets. + Native ACLs need an Ozone administrator instead, and `ozone.administrators` is one list for the whole cluster, so it cannot be limited to a bucket. + Releasing a Compliance retention still needs an Ozone administrator: OM checks `ozone.administrators` after the Ranger check, + and no Ranger permission alone is enough (see [Releasing a Compliance Retention](#4-releasing-a-compliance-retention)). + +##### Access Type Mapping + +Policy items without `action-matches` are checked by access type only. +The S3 actions are the AWS permission names; `GetBucketObjectLockConfiguration` / `PutBucketObjectLockConfiguration` authorize the `GetObjectLockConfiguration` / `PutObjectLockConfiguration` APIs. + +| S3 Action | Resource | Access type | Description | +|:-----------------------------------|:---------|:---------------------|:----------------------------------------------------------------------------| +| `GetBucketObjectLockConfiguration` | Bucket | `READ` | Read the bucket's Object Lock settings. | +| `PutBucketObjectLockConfiguration` | Bucket | `WRITE` | Set the bucket's default retention (mode, duration, event hold). | +| `GetObjectRetention` | Key | `READ` | Read the retention mode, date and event hold. | +| `PutObjectRetention` | Key | `WRITE` | Set, extend, shorten or remove a retention; set or release an event hold. | +| `GetObjectLegalHold` | Key | `READ` | Read the legal hold. | +| `PutObjectLegalHold` | Key | `WRITE` | Set the legal hold `ON` or `OFF`. | +| `BypassGovernanceRetention` | Key | `WRITE` and `DELETE` | Bypass Governance retention. | + +##### `action-matches` + +* With `enableActionMatcherInPoliciesCondition`, a policy item can have an `action-matches` condition: a list of S3 actions (`*` and `Prefix*` allowed) + that `RangerActionMatcher` compares with the action of the request. +* It only narrows an item: the item applies when both the access type and the action match. Items without it behave as before. +* Example: grant a legal team `WRITE` with `action-matches: PutObjectLegalHold`, so it can change legal holds but not retention. +* S3 Gateway sets the action in `S3GActionIamMapper`, and OM passes it to Ranger in the `RequestContext`. OM only adds the dependent actions below. +* Today S3 Gateway only sends the action when `ozone.s3g.sts.http.enabled=true`. This design depends on [HDDS-17054](https://issues.apache.org/jira/browse/HDDS-17054), which sends it for every S3 request. +* Requests without an S3 action (ofs, o3fs, Ozone CLI, Java API) never match an item with a specific `action-matches` value. + They are allowed only by items without the condition (or with `*`); otherwise they are denied (see [Trusted Boundary](#trusted-boundary-and-compliance-scope)). + +##### Dependent Permissions + +As in AWS S3 ([required permissions for S3 API operations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-policy-actions.html)), +some requests also need a second action, depending on their headers, the bypass flag or the bucket: + +| S3 API | Main S3 action | Header or condition | Dependent S3 action | Access type | If denied | +|:----------------------------------------------------------|:---------------------|:-------------------------------------------------------------------------------------------------------|:-------------------------------------------|:------------------------|:--------------------------------------------| +| `CreateBucket` | `CreateBucket` | `x-amz-bucket-object-lock-enabled: true` | `PutBucketObjectLockConfiguration` | `WRITE` (bucket) | Request fails | +| `PutObject`, `CopyObject` (destination), `CreateMultipartUpload` | `PutObject` | `x-amz-object-lock-mode`, `-retain-until-date`, `-event-hold`, `-event-hold-duration-days` / `-years` | `PutObjectRetention` | `WRITE` (key) | Request fails | +| `PutObject`, `CopyObject` (destination), `CreateMultipartUpload` | `PutObject` | `x-amz-object-lock-legal-hold` | `PutObjectLegalHold` | `WRITE` (key) | Request fails | +| `GetObject`, `HeadObject` | `GetObject` | The bucket has Object Lock enabled | `GetObjectRetention`, `GetObjectLegalHold` | `READ` (key) | OM drops the lock information, so S3 Gateway omits the `x-amz-object-lock-*` headers; the read succeeds | +| `DeleteObject`, `DeleteObjects` | `DeleteObject` | `x-amz-bypass-governance-retention: true` | `BypassGovernanceRetention` | `WRITE`, `DELETE` (key) | Request fails (`DeleteObjects`: per-key `AccessDenied`) | +| `PutObjectRetention` | `PutObjectRetention` | `x-amz-bypass-governance-retention: true` (required to shorten or remove a Governance retention) | `BypassGovernanceRetention` | `WRITE`, `DELETE` (key) | Request fails | +| `PutObject`, `CopyObject`, `CompleteMultipartUpload`, rename (native clients) | `PutObject` | `bypassGovernanceRetention` (required to overwrite or rename a key under Governance retention) | `BypassGovernanceRetention` | `WRITE`, `DELETE` (key) | Request fails | + +The dependent actions use the same access types as the direct lock changes (`READ` or `WRITE`; `WRITE` and `DELETE` for the bypass), so they mostly matter for policy items with `action-matches`. +With such items, a data writer cannot set its own retention with a header, and a retention manager cannot upload data. + +**How it works**: S3 Gateway sends one RPC with the main S3 action. In `preExecute`, the OM request runs the normal ACL check for that action, then looks at the request itself: +each Object Lock header or flag it carries adds one (access type, S3 action) check on the same resource, for example (`WRITE`, `PutObjectRetention`) for retention headers, +or (`WRITE`, `BypassGovernanceRetention`) and (`DELETE`, `BypassGovernanceRetention`) for the bypass flag. OM calls `checkAccess` once per check. +The main action always comes from S3 Gateway; OM never maps an API to its action. OM only adds dependent actions, as the existing `DEPENDENT_S3_ACTIONS` map already does, +because their conditions are headers and flags that OM parses anyway. + +Whether a check is needed depends only on what the request carries (a header or the bypass flag), not on whether the key is currently locked, so `preExecute` does not read the key. +The WORM check in `validateAndUpdateCache` decides whether the lock allows the operation. +`GetObject` / `HeadObject` also check `GetObjectRetention` and `GetObjectLegalHold` with `READ`, before the lookup, when the bucket has Object Lock enabled. + +What happens when a check is denied is in the "If denied" column of the table above. The dependent checks apply to the target resource only (the parent volume check is not repeated), +and they also run for requests without an S3 action, by access type alone. Listings do not return lock information, so they need no check per key. + +Each `checkAccess` is an in-process evaluation against the Ranger plugin's cached policies and one Ranger audit record, not a round trip. +This generalizes the existing `DEPENDENT_S3_ACTIONS` map in `OmMetadataReader` (today `GetObjectAttributes` → `GetObject`): `OMClientRequest.checkAcls` takes the checks declared by the request, +and `OmMetadataReader.checkAcls` accepts the S3 action per call instead of taking it only from the thread-local `S3Auth`. Review Comment: > `OMClientRequest.checkAcls` takes the checks declared by the request, and `OmMetadataReader.checkAcls` accepts the S3 action per call instead of taking it only from the thread-local `S3Auth`. just curious how this will be done? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
