anuragp010 opened a new pull request, #11951: URL: https://github.com/apache/ozone/pull/11951
## What changes were proposed in this pull request? A Key is initially written with a `StoragePolicy` and when the container is created, the `StorageTier` is chosen. This is then propagated to a datanode which places this container on a volume with a `StorageType`. Currently, nothing tells an operator whether a key's data is really on the intended storage type. A write can fallback to another tier, or a volume can be relabelled amongst other scenarios. This PR adds the command `ozone admin storagepolicy check <volume/bucket/key> [--show-replicas]`. The command returns YES only if each the key's container replica's `storageType` and `volumeStorageType` match the expected `storageType`. 1. The command first gets the `OzoneKeyDetails` of the key. 2. For each unique container in the list of containers that the key's data is present in, a call is made to the SCM to get the container replicas. 3. The expected storageType for a key is derived from it's StoragePolicy (`OzoneStoragePolicy` -> `StorageTier` -> `StorageType`) ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17710 ## How was this patch tested? 1. Unit Tests and Integration Tests. 2. **Docker compose cluster with output:** ```shell bash-5.1$ ozone admin storagepolicy check vol1/bucket1/key1 --show-replicas Storage Policy for key 'vol1/bucket1/key1': HOT{Creation Tier: [SSD x 3], Fallback Tier: [DISK x 3]} Key Match Storage Policy: YES Container ID: 1 Datanode: 5e7318f9-14bc-4db8-b3b7-bc78737746a6 (ozone-datanode-ssd-3.ozone_default, /default-rack) Replica Storage Type: SSD Volume Storage Type: SSD Replica Match: YES Datanode: 5b53ac10-d16b-4a36-a4df-269cc737964c (ozone-datanode-ssd-1.ozone_default, /default-rack) Replica Storage Type: SSD Volume Storage Type: SSD Replica Match: YES Datanode: d675d25c-8ea3-4c36-85d7-df57524e1ee7 (ozone-datanode-ssd-2.ozone_default, /default-rack) Replica Storage Type: SSD Volume Storage Type: SSD Replica Match: YES Container Match Storage Policy: YES ``` **Sample Output for the volume relabelling scenario:** ```shell Storage Policy for key 'vol1/bucket1/key1': HOT{Creation Tier: [SSD x 3], Fallback Tier: [DISK x 3]} Key Match Storage Policy: NO Container ID: 12 Datanode: 1b6e0c52-... (dn1.example.com, /rack1) Replica Storage Type: SSD Volume Storage Type: SSD Replica Match: YES Datanode: 7d41a9f3-... (dn2.example.com, /rack2) Replica Storage Type: SSD Volume Storage Type: DISK Replica Match: NO (volumeStorageType DISK != expected SSD) Datanode: c93f5e18-... (dn3.example.com, /rack3) Replica Storage Type: SSD Volume Storage Type: SSD Replica Match: YES Container Match Storage Policy: NO ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
