chungen0126 commented on code in PR #6989: URL: https://github.com/apache/ozone/pull/6989#discussion_r2999375115
########## hadoop-hdds/docs/content/design/storage-policy.md: ########## @@ -19,379 +20,588 @@ status: draft # Terminology -## Terminology +## Definitions - Storage Policy: Defines where key data replicas should be stored in specific storage tiers. -- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc. -- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy. -- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.. -- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix. +- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a + storage type, including SSD, DISK, and ARCHIVE. +- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For + example, a 3-replica SSD tier consists of 3 replicas of SSD type. +- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode. +- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable. ## Storage Policy vs Storage Type vs Storage Tier  -The relation of Storage Policy, Storage Type and Storage Tier +The relationship between Storage Policy, Storage Type, and Storage Tier: -- The storage policy is the property of key/bucket/ prefix (Managed by OM); -- The storage tier is the property of Pipeline and Container (Managed by SCM); -- The storage type is the property of volume and Container replicas (Managed by DN); -- Only the storage policy can be modified by the user directly via ozone command; +- The storage policy is the property of key/bucket (managed by OM). +- The storage tier is the property of Pipeline and Container (managed by SCM). +- The storage type is the property of volume and container replica (managed by DN). +- Only the storage policy can be modified by the user directly via the ozone command. Example: -For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type. +For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which +are of the SSD storage type. # User Scenarios -- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster. -- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster. -- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks. -- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier +- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot. + Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster. +- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The + key's data will be distributed across SSD disks in the cluster. +- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file + to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster. + +# Goals + +- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3 + storage class. +- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3, + API, and shell command interfaces. +- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for + containers. +- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the + storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support + checking whether the key storage policy is satisfied. +- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type. + For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK + type volumes. +- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on + Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage + tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container + replicas, and each container replica should reside on an SSD type volume. +- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type. + For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK + type volumes. + +# Design + +## Supported Storage Policies + +- Supported storage policies: Hot / Warm / Cold +- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY +- Supported storage types: SSD / DISK / ARCHIVE +- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY +- S3 storage classes: STANDARD / STANDARD_IA / GLACIER + +### Storage Policy Map to Storage Tier -# Current Status +| Storage Policy | Storage Tier for Write | Fallback Tier for Write | +|----------------|------------------------|-------------------------| +| Hot | SSD | DISK | +| Warm | DISK | EMPTY | +| Cold | ARCHIVE | EMPTY | -- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented. -- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks +- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified. +- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM + will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available. -# Goal Requirements Specification +### Storage Tier Map to Storage Type -### **Support for Storage Policy Writing and Management** +| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type | +|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------| +| SSD | SSD | SSD | 3 SSD | n SSD | +| DISK | DISK | DISK | 3 DISK | n DISK | +| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE | +| EMPTY | - | - | - | - | -- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies. -- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set. +### Fallback Storage Type for Container Replica Replication/Migration -### **Support for Data Migration Across Different Storage Policies** +| Container Replica Storage Type | Fallback Storage Types (ordered) [1] | +|--------------------------------|--------------------------------------| +| SSD | DISK, ARCHIVE | +| DISK | ARCHIVE | +| ARCHIVE | none | -- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers. +- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable Review Comment: Got it. I miss that part. Thanks for reminding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
