Maksim Myskov created HDDS-8410:
-----------------------------------
Summary: Implement storage policies
Key: HDDS-8410
URL: https://issues.apache.org/jira/browse/HDDS-8410
Project: Apache Ozone
Issue Type: Improvement
Reporter: Maksim Myskov
In order to optimize read performance, it is required to implement HDFS-like
storage policies: HOT, WARM, COLD.
{*}HOT{*}: all container’s replicas are stored on fast storage
{*}WARM{*}: one container replica is stored on fast storage, others are stored
on regular storage
{*}COLD{*}: all container’s replicas are stored on slow storage
*Use case example 1:*
There is a cluster of N datanodes. Each datanode has N regular disks and M fast
disks.
*Use case example 2:*
There is a cluster of N regular datanodes and M fast datanodes. Regular
datanodes have regular (HDD) disk and fast datanodes have fast (SSD) disks.
Requirements:
* If a container’s heat state (hot-warm-cold) changes, container can be moved
to a different disk on the same node or to a different node, depending on the
cluster configuration.
* container’s heat is automatically calculated based on data request
statistics.
* A file/directory’s heat state can be set manually. Example: if one set
“key1/subkey/*” to be HOT data, all chunks related to this prefix must be
considered HOT despite heatmap.
Open questions:
* How to calculate data heat map? A possible solution is that statistics are
collected by datanodes and then sent to SCM in reports. SCM analyses these
reports to build an actual data heat map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
