ChenSammi commented on code in PR #6989: URL: https://github.com/apache/ozone/pull/6989#discussion_r3772977410
########## hadoop-hdds/docs/content/design/storage-policy.md: ########## @@ -0,0 +1,607 @@ +--- +title: Ozone Storage Policy Support +summary: Support storage policy in Ozone to write key data into specified types of storage media. +date: 2026-03-23 +jira: HDDS-11233 +status: draft +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Terminology + +## Definitions + +- Storage Policy: Defines where key data replicas should be stored in specific storage tiers. +- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a + storage type, including SSD, DISK, and ARCHIVE. +- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For + example, a 3-replica SSD tier consists of 3 replicas of SSD type. +- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode. +- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable. + +## Storage Policy vs Storage Type vs Storage Tier + + + +The relationship between Storage Policy, Storage Type, and Storage Tier: + +- The storage policy is the property of key/bucket (managed by OM). +- The storage tier is the property of Pipeline and Container (managed by SCM). +- The storage type is the property of volume and container replica (managed by DN). +- Only the storage policy can be modified by the user directly via the ozone command. + +Example: + +For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which +are of the SSD storage type. + +# User Scenarios + +- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot. + Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster. +- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The + key's data will be distributed across SSD disks in the cluster. +- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file + to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster. + +# Goals + +- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3 + storage class. +- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3, + API, and shell command interfaces. +- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for + containers. +- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the + storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support + checking whether the key storage policy is satisfied. +- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type. + For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK + type volumes. +- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on + Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage + tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container + replicas, and each container replica should reside on an SSD type volume. +- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type. + For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK + type volumes. + +# Design + +## Supported Storage Policies + +- Supported storage policies: Hot / Warm / Cold +- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY +- Supported storage types: SSD / DISK / ARCHIVE +- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY +- S3 storage classes: STANDARD / STANDARD_IA / GLACIER + +### Storage Policy Map to Storage Tier + +| Storage Policy | Storage Tier for Write | Fallback Tier for Write | +|----------------|------------------------|-------------------------| +| Hot | SSD | DISK | +| Warm | DISK | EMPTY | +| Cold | ARCHIVE | EMPTY | + +- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified. +- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM + will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available. + +### Storage Tier Map to Storage Type + +| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type | +|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------| +| SSD | SSD | SSD | 3 SSD | n SSD | +| DISK | DISK | DISK | 3 DISK | n DISK | +| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE | +| EMPTY | - | - | - | - | + +### Fallback Storage Type for Container Replica Replication/Migration + +| Container Replica Storage Type | Fallback Storage Types (ordered) [1] | +|--------------------------------|--------------------------------------| +| SSD | DISK, ARCHIVE | +| DISK | ARCHIVE | +| ARCHIVE | none | + +- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable + volume type that matches the original container replica's storage type, it will attempt to use the fallback storage + types in order. + +[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs +to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage +Types" rather than "Fallback Storage Tier". + +### AWS S3 StorageClass + +| AWS S3 StorageClass | Ozone Storage Policy | +|---------------------|----------------------| +| STANDARD [1] | Hot | +| STANDARD_IA | Warm | +| GLACIER | Cold | +| DEEP_ARCHIVE | Warm | + +> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | +> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE +> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, +> which is not straightforward to map to the Ozone SSD tier. + +[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in +Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics. + +## Component Changes + +### Datanode Container Replica + +A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML +file. + +### Bucket, Key + +A storage policy attribute is added to buckets and keys on OM. + +### SCM Container, Pipeline + +A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers. +For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier +attributes will include both SSD and DISK. + +## Datanode Volume Storage Type + +- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is + configured for a volume, the storage type will be DISK. + - For example: + ```xml + <property> + <name>hdds.datanode.dir</name> + <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value> + </property> + ``` + Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type. + +[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section +Configuration. + +## Writing Keys with Storage Policy + +### Ozone Filesystem + +- Support specifying a storage policy when writing a key. + - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy. + - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket. + If the bucket does not have a storage policy set, the default storage policy is determined by the configuration + `ozone.default.storagepolicy`, the default value is WARM. + +- Fallback strategy: + - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback + storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed. + - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write + to the fallback storage tier. The default is to allow fallback. + +### S3 + +- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request. +- If StorageClass is not specified in the request, the storage policy of the bucket is used. +- If the storage policy of the bucket is null, the storage policy is determined by the configuration + `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM. + +### Supported APIs + +#### Ozone Filesystem + +- createKey +- createStreamKey +- createFile +- createStreamFile +- initiateMultipartUpload + +#### S3 Request + +Refer to +[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) + +- PutObject: + - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the + object. +- CopyObject: + - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage + policy when copying the object from the source location to the target location. If no new storage policy is + specified, inherit the storage policy of the source object. +- Multipart Upload: + - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass + parameter of CreateMultipartUpload in UploadPart. +- GetObject: + - Return the current storage policy of the object in the GetObject response. +- HeadObject: + - Return the metadata of the object, including its storage policy (StorageClass). +- ListObjects: + - Include the storage policy (StorageClass) information of each object in the ListObjects response. + +### Write Key Process + +#### Normal Write (No Fallback) + + + +- Client sends CreateKey request to OM with storage policy Hot. +- OM requests SCM to allocate a container with storage policy Hot. +- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot. +- Client determines the storage type of the chunk to be SSD according to the storage tier SSD. +- DN creates the container or writes the chunk on the specified storage medium according to the storage type. + +#### Fallback Write + + + +- Client sends CreateKey request to OM with storage policy Hot. +- OM requests SCM to allocate a container with storage policy Hot. +- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds + that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK. +- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK. +- DN creates the container or writes the chunk on the specified storage medium according to the storage type. + +### Datanode Container Replica Creation + +- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request. +- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode + creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the + request. +- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be + created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default + is DISK). This typically only occurs when a client from an older version that does not support the storage policy + feature writes data to the Datanode. +- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in + the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to + create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal + operation. + +### SCM Pipeline Creation and Selection + +- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and + storage type of all volumes (the default storage type for a Datanode volume is DISK). +- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto. +- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier + corresponding to the storage policy in the request. + - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the + pipeline will consist of 3 Datanodes that have SSD type volumes. + - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD + type volumes. +- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the + fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough + suitable Datanodes, the creation fails. +- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes + with SSD type volumes, and will not include Datanodes with only DISK type volumes. +- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline + can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline + created on these Datanodes will have supported storage tier attributes of both SSD and DISK. +- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported + storage tiers. + + +- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster + can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier. +- The pipeline count limit is calculated independently for each storage tier. + +## Storage Policy Update + +Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy. + +- `ozone sh bucket update --storagepolicy <storagePolicyStr>` + Allowed values: HOT, WARM, COLD, or null to unset. + +Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the Review Comment: is any state container allowed here to set the storage tier? Is there any action triggered later to move the container to the corresponding tier? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
