ChenSammi commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r3954820020
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,291 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: accepted +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design document aims to plan and implement the Object Lock mechanism for OBS buckets integrated with Ranger. +The primary objective is to provide data immutability and tamper-proof protection through the object locking feature. + +## Problem statement + +With growing demands for data security and compliance, ensuring that critical data stored in OBS (Object Storage) is +protected from accidental or malicious deletion and overwriting has become an essential system protection requirement. +To establish a more rigorous data protection mechanism, we plan to introduce the Object Lock feature. + +Considering the current system architecture and access control strategies, +this design integrates with existing Apache Ranger to manage Object Lock permissions on OBS buckets. +Meanwhile, to accelerate core feature delivery, we have decided to exclude complex multi-version locking (Versioning Lock) +and legacy FSO buckets from this initial release. In addition, support for Native ACLs is excluded; Native ACLs typically grant permissions +at the granular bucket or object level, whereas Object Lock permission management favors broad, role-based authorization, +creating a conflict in design philosophies. Narrowing the scope allows us to focus on the core functionality and ensure a rapid, +stable rollout of baseline tamper-proof protection. + +**Goal:** + +* Implement the Object Lock feature on standard OBS buckets, fully integrated with Ranger for permission and access control. +* Support single-version objects only. + +## Non-Goal + +* Versioning Lock: Support for locking across multiple object versions is deferred (multi-version core features are currently under development). +* FSO Legacy Buckets: Object Lock support for legacy FSO buckets is excluded. +* Native ACL Support: Native ACLs will not be used for access control or advanced configuration such as Retention Mode (Governance); access management is centralized exclusively via Ranger. + +## 5. Technical Description + +### Terminology + +**Legal Hold** + +* **Definition**: Applies an indefinite lock status to an object. The object remains protected until an administrator explicitly removes the lock (Remove Legal Hold). +* **Restricted Operations**: + * Put Object + * Delete Object + * Multipart Initial / Complete +* **Allowed Operations**: + * Get Object + * Get Legal Hold + * Put Legal Hold (Depends on permission) + +**Retention** + +* Definition: Configures a fixed retention duration (specified in days or years) for an object and applies a specific retention mode. +* Retention Modes: + * Compliance Mode: The strictest protection tier. Once applied, no user (including root/admin) can remove the lock, shorten the duration, or overwrite the object before the retention period expires. + * Governance Mode: A flexible protection tier. Standard users are restricted by locking rules, but users with bypassgovernance permissions can bypass restrictions to perform modifications or deletions. +* Restricted Operations: + * Put Object + * Delete Object + * Multipart Initial/Complete + * Set Retention Period +* Allowed Operations: + * Get Object + +> _**Note**: +> * Background & Root Cause: A prerequisite for enabling WORM (Write Once, Read Many) in AWS S3 is that Object Versioning must be enabled. Under S3 architecture, executing a Put on a locked object generates a new version without affecting the protected prior version; thus, S3 Object Lock primarily restricts Delete Object. +> * Ozone Implementation Status: Because Ozone's versioning feature is still under development, to guarantee absolute immutability during the lock period, Ozone will directly block and reject all overwrite operations (such as any form of Put or overwrite) on locked objects. + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + required bool objectLockEnabled = 24 [default = false]; + optional RetentionConfig defaultRetention = 25; + } + + message RetentionConfig { + optional RetentionMode retentionMode = 23; + optional uint64 retainUntilDate = 24; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +Two new fields: retentionConfig & legalHold. + +```protobuf + message KeyInfo { + // ... existing fields + optional RetentionConfig retentionConfig = 23; + optional bool legalHold = 24 [default = false]; + } +``` + + + +### Ranger Access Control + +Ozone Object Lock enforces a dual-gate mechanism combining **Ranger authorization checks** and **underlying WORM state validation**: + +* **Standard Data Operations (Put / Delete)**: Even if a user is granted Ranger `WRITE` or `DELETE` permissions, the operation is immediately denied with a `403 Access Denied` (`WORMProtectionException`) if the target object is actively locked by an unexpired retention period or an active Legal Hold. +* **Lock Management and Bypass Authorization**: To adhere to the Principle of The Least Privilege, dedicated Ranger Access Types are introduced to govern Legal Hold state transitions and Governance Mode bypass privileges: + +#### 1. Legal Hold Access Control + +Three dedicated Access Types are added to `accessTypeRestrictions` at the Key level to govern Legal Hold states: + +* **`GET_LEGAL_HOLD`**: Allows querying the current Legal Hold status of an object (maps to the `GetObjectLegalHold` API). +* **`PUT_LEGAL_HOLD`**: Grants permission to apply a Legal Hold to an object (maps to `PutObjectLegalHold` with status ON). Once applied, the object is locked indefinitely, blocking all overwrite and delete operations. +* **`CLEAR_LEGAL_HOLD`**: Grants permission to remove a Legal Hold from an object (maps to `PutObjectLegalHold` with status OFF). Because this clears immutability protection, it is treated as a high-privilege action restricted only to authorized users. + +#### 2. Retention Access Control and Governance Mode Bypass + +For object retention, the interaction with Ranger policies depends on the configured Retention Mode: + +* **Compliance Mode**: Serves as the strictest compliance tier. Until the retention period expires, **no role or Ranger permission can bypass or overwrite the lock**, including cluster administrators. +* **Governance Mode and `BYPASS_GOVERNANCE`**: + * **Mechanism**: Governance Mode allows authorized users to overwrite, delete, or alter the retention duration of a locked object before its expiration date. + * **Authorization Binding**: Introduces the **`BYPASS_GOVERNANCE`** Access Type, configured under `accessTypeRestrictions` at the **Volume level**. + * **Enforcement Flow**: While regular users are strictly blocked from mutating locked objects in Governance Mode, any request attempting to overwrite or delete such objects requires explicit `BYPASS_GOVERNANCE` entitlement granted in the Ranger policy for that Volume. Without this explicit permission, even administrators using standard client tools cannot modify or remove the locked object. +### New Ozone APIs + +**ObjectStore** + +```java + public void addRetentionConfig(OzoneObj obj, RetentionArgs retentionArgs); + public void addLegalHold(OzoneObj obj, bool hold); +``` + +**OzoneBucket** + +```java +public void setRetentionConfig(@Nullable RetentionArgs retentionArgs); +``` + +### Supported S3 APIs + +To ensure seamless integration with existing S3 clients (e.g., AWS CLI, Boto3) and applications, the Ozone S3 Gateway (S3G) will translate and support the following standard AWS S3 Object Lock APIs: + +**Bucket-Level APIs:** +* `GetBucketObjectLockConfiguration`: Retrieves the Object Lock status and the default retention configuration (if any) for a specified OBS bucket. +* `PutBucketObjectLockConfiguration`: Maps to the new Ozone API to enable Object Lock for an existing bucket. Configuring default retention rules (DefaultRetention) is optional. + +**Object-Level APIs:** +* `PutObjectRetention`: Places a retention configuration on an object, specifying the retention mode (COMPLIANCE or GOVERNANCE) and the `RetainUntilDate`. +* `GetObjectRetention`: Retrieves the current retention configuration applied to an object. +* `PutObjectLegalHold`: Applies or removes a Legal Hold configuration to the specified object. +* `GetObjectLegalHold`: Retrieves the current Legal Hold status of an object. + +**Impacted Standard APIs:** +Standard data mutating APIs will intercept the Object Lock state and return the `403 Access Denied` (or specifically `WORMProtectionException`) if a modification is attempted on a locked object: +* `PutObject` / `CopyObject` (Overwrites will be rejected) +* `DeleteObject` / `DeleteObjects` (Deletions will be rejected) +* `CreateMultipartUpload` / `CompleteMultipartUpload` / `UploadPart` (Uploads will be rejected if the object is locked) + +> _**Note**: +> * **Object Lock Immutability & Future Versioning Binding**: +> - **Current State (No Versioning)**: Once Object Lock is enabled on a bucket (`objectLockEnabled = true`), it cannot be disabled. +> - **Future Evolution (With Versioning)**: When S3 Versioning is supported, Object Lock and Versioning will be tightly coupled following AWS S3 semantics: once Object Lock is enabled, Versioning cannot be suspended or disabled. +> * **Bucket Creation Semantics**: Review Comment: There is no more S3 behavior, versioning is enabled automatically when Object Lock is enabled. ########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,291 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: accepted +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design document aims to plan and implement the Object Lock mechanism for OBS buckets integrated with Ranger. +The primary objective is to provide data immutability and tamper-proof protection through the object locking feature. + +## Problem statement + +With growing demands for data security and compliance, ensuring that critical data stored in OBS (Object Storage) is +protected from accidental or malicious deletion and overwriting has become an essential system protection requirement. +To establish a more rigorous data protection mechanism, we plan to introduce the Object Lock feature. + +Considering the current system architecture and access control strategies, +this design integrates with existing Apache Ranger to manage Object Lock permissions on OBS buckets. +Meanwhile, to accelerate core feature delivery, we have decided to exclude complex multi-version locking (Versioning Lock) +and legacy FSO buckets from this initial release. In addition, support for Native ACLs is excluded; Native ACLs typically grant permissions +at the granular bucket or object level, whereas Object Lock permission management favors broad, role-based authorization, +creating a conflict in design philosophies. Narrowing the scope allows us to focus on the core functionality and ensure a rapid, +stable rollout of baseline tamper-proof protection. + +**Goal:** + +* Implement the Object Lock feature on standard OBS buckets, fully integrated with Ranger for permission and access control. +* Support single-version objects only. + +## Non-Goal + +* Versioning Lock: Support for locking across multiple object versions is deferred (multi-version core features are currently under development). +* FSO Legacy Buckets: Object Lock support for legacy FSO buckets is excluded. +* Native ACL Support: Native ACLs will not be used for access control or advanced configuration such as Retention Mode (Governance); access management is centralized exclusively via Ranger. + +## 5. Technical Description + +### Terminology + +**Legal Hold** + +* **Definition**: Applies an indefinite lock status to an object. The object remains protected until an administrator explicitly removes the lock (Remove Legal Hold). +* **Restricted Operations**: + * Put Object + * Delete Object + * Multipart Initial / Complete +* **Allowed Operations**: + * Get Object + * Get Legal Hold + * Put Legal Hold (Depends on permission) + +**Retention** + +* Definition: Configures a fixed retention duration (specified in days or years) for an object and applies a specific retention mode. +* Retention Modes: + * Compliance Mode: The strictest protection tier. Once applied, no user (including root/admin) can remove the lock, shorten the duration, or overwrite the object before the retention period expires. + * Governance Mode: A flexible protection tier. Standard users are restricted by locking rules, but users with bypassgovernance permissions can bypass restrictions to perform modifications or deletions. +* Restricted Operations: + * Put Object + * Delete Object + * Multipart Initial/Complete + * Set Retention Period +* Allowed Operations: + * Get Object + +> _**Note**: +> * Background & Root Cause: A prerequisite for enabling WORM (Write Once, Read Many) in AWS S3 is that Object Versioning must be enabled. Under S3 architecture, executing a Put on a locked object generates a new version without affecting the protected prior version; thus, S3 Object Lock primarily restricts Delete Object. +> * Ozone Implementation Status: Because Ozone's versioning feature is still under development, to guarantee absolute immutability during the lock period, Ozone will directly block and reject all overwrite operations (such as any form of Put or overwrite) on locked objects. + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + required bool objectLockEnabled = 24 [default = false]; + optional RetentionConfig defaultRetention = 25; + } + + message RetentionConfig { + optional RetentionMode retentionMode = 23; + optional uint64 retainUntilDate = 24; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +Two new fields: retentionConfig & legalHold. + +```protobuf + message KeyInfo { + // ... existing fields + optional RetentionConfig retentionConfig = 23; + optional bool legalHold = 24 [default = false]; + } +``` + + + +### Ranger Access Control + +Ozone Object Lock enforces a dual-gate mechanism combining **Ranger authorization checks** and **underlying WORM state validation**: + +* **Standard Data Operations (Put / Delete)**: Even if a user is granted Ranger `WRITE` or `DELETE` permissions, the operation is immediately denied with a `403 Access Denied` (`WORMProtectionException`) if the target object is actively locked by an unexpired retention period or an active Legal Hold. +* **Lock Management and Bypass Authorization**: To adhere to the Principle of The Least Privilege, dedicated Ranger Access Types are introduced to govern Legal Hold state transitions and Governance Mode bypass privileges: + +#### 1. Legal Hold Access Control + +Three dedicated Access Types are added to `accessTypeRestrictions` at the Key level to govern Legal Hold states: + +* **`GET_LEGAL_HOLD`**: Allows querying the current Legal Hold status of an object (maps to the `GetObjectLegalHold` API). +* **`PUT_LEGAL_HOLD`**: Grants permission to apply a Legal Hold to an object (maps to `PutObjectLegalHold` with status ON). Once applied, the object is locked indefinitely, blocking all overwrite and delete operations. +* **`CLEAR_LEGAL_HOLD`**: Grants permission to remove a Legal Hold from an object (maps to `PutObjectLegalHold` with status OFF). Because this clears immutability protection, it is treated as a high-privilege action restricted only to authorized users. + +#### 2. Retention Access Control and Governance Mode Bypass + +For object retention, the interaction with Ranger policies depends on the configured Retention Mode: + +* **Compliance Mode**: Serves as the strictest compliance tier. Until the retention period expires, **no role or Ranger permission can bypass or overwrite the lock**, including cluster administrators. +* **Governance Mode and `BYPASS_GOVERNANCE`**: + * **Mechanism**: Governance Mode allows authorized users to overwrite, delete, or alter the retention duration of a locked object before its expiration date. + * **Authorization Binding**: Introduces the **`BYPASS_GOVERNANCE`** Access Type, configured under `accessTypeRestrictions` at the **Volume level**. + * **Enforcement Flow**: While regular users are strictly blocked from mutating locked objects in Governance Mode, any request attempting to overwrite or delete such objects requires explicit `BYPASS_GOVERNANCE` entitlement granted in the Ranger policy for that Volume. Without this explicit permission, even administrators using standard client tools cannot modify or remove the locked object. +### New Ozone APIs + +**ObjectStore** + +```java + public void addRetentionConfig(OzoneObj obj, RetentionArgs retentionArgs); + public void addLegalHold(OzoneObj obj, bool hold); +``` + +**OzoneBucket** + +```java +public void setRetentionConfig(@Nullable RetentionArgs retentionArgs); +``` + +### Supported S3 APIs + +To ensure seamless integration with existing S3 clients (e.g., AWS CLI, Boto3) and applications, the Ozone S3 Gateway (S3G) will translate and support the following standard AWS S3 Object Lock APIs: + +**Bucket-Level APIs:** +* `GetBucketObjectLockConfiguration`: Retrieves the Object Lock status and the default retention configuration (if any) for a specified OBS bucket. +* `PutBucketObjectLockConfiguration`: Maps to the new Ozone API to enable Object Lock for an existing bucket. Configuring default retention rules (DefaultRetention) is optional. + +**Object-Level APIs:** +* `PutObjectRetention`: Places a retention configuration on an object, specifying the retention mode (COMPLIANCE or GOVERNANCE) and the `RetainUntilDate`. +* `GetObjectRetention`: Retrieves the current retention configuration applied to an object. +* `PutObjectLegalHold`: Applies or removes a Legal Hold configuration to the specified object. +* `GetObjectLegalHold`: Retrieves the current Legal Hold status of an object. + +**Impacted Standard APIs:** +Standard data mutating APIs will intercept the Object Lock state and return the `403 Access Denied` (or specifically `WORMProtectionException`) if a modification is attempted on a locked object: +* `PutObject` / `CopyObject` (Overwrites will be rejected) +* `DeleteObject` / `DeleteObjects` (Deletions will be rejected) +* `CreateMultipartUpload` / `CompleteMultipartUpload` / `UploadPart` (Uploads will be rejected if the object is locked) + +> _**Note**: +> * **Object Lock Immutability & Future Versioning Binding**: +> - **Current State (No Versioning)**: Once Object Lock is enabled on a bucket (`objectLockEnabled = true`), it cannot be disabled. +> - **Future Evolution (With Versioning)**: When S3 Versioning is supported, Object Lock and Versioning will be tightly coupled following AWS S3 semantics: once Object Lock is enabled, Versioning cannot be suspended or disabled. +> * **Bucket Creation Semantics**: Review Comment: There is one more S3 behavior, versioning is enabled automatically when Object Lock is enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
