[ 
https://issues.apache.org/jira/browse/HDDS-17818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devesh Kumar Singh updated HDDS-17818:
--------------------------------------
    Description: 
Ozone shows how full SSD and DISK are, but not whether storage policies are 
working. An administrator cannot see:

how much new HOT data is falling back from SSD to DISK
why SCM could not place data on a tier
whether enough SSD datanodes and pipelines are ready
whether repaired replicas or balancer moves stayed on the right medium
which buckets have data that doesn't match its policy
Today no OM, SCM or datanode metric has a storage policy, tier or storage type 
dimension. The only exception is the per-volume StorageType tag in 
VolumeInfoMetrics. SCM exports per-type capacity only for DISK and SSD, not 
ARCHIVE.

This JIRA tracks the work to make storage policy outcomes visible. Ozone emits 
raw counters and gauges with bounded labels: policy, tier, storage type, 
replication and reason. Ratios such as the fallback share are computed in 
PromQL, and a Grafana dashboard explains them. Per-bucket and per-key detail 
stays in the CLI, not in Prometheus labels.

Scope:

OM: committed bytes by policy, tier and fallback outcome.
SCM:
placement failure reasons
ARCHIVE capacity
eligible datanodes per type
pipeline readiness per tier
repair outcome
replica drift from the container tier
Datanode: Disk Balancer results per storage type.
A Storage Policy Grafana dashboard with example alerts.
A bucket and volume compliance summary in ozone admin storagepolicy check.
Rules for all sub-tasks:

Keep existing aggregate metrics and names.
Add new series alongside them.
Report a missing tier or type as UNKNOWN, never as DISK.
Never use volume, bucket, key or exception text as a label.
Related work tracked elsewhere:

Container Balancer per-type metrics: HDDS-17590, under HDDS-17583.
Per-key policy check: HDDS-17710.
Bucket mismatch listing: HDDS-17756.
Fallback contract: HDDS-17535.

cc: [~anuragparvatikar] [~rakeshr] [~rbarooah]

  was:
Ozone shows how full SSD and DISK are, but not whether storage policies are 
working. An administrator cannot see:

how much new HOT data is falling back from SSD to DISK
why SCM could not place data on a tier
whether enough SSD datanodes and pipelines are ready
whether repaired replicas or balancer moves stayed on the right medium
which buckets have data that doesn't match its policy
Today no OM, SCM or datanode metric has a storage policy, tier or storage type 
dimension. The only exception is the per-volume StorageType tag in 
VolumeInfoMetrics. SCM exports per-type capacity only for DISK and SSD, not 
ARCHIVE.

This JIRA tracks the work to make storage policy outcomes visible. Ozone emits 
raw counters and gauges with bounded labels: policy, tier, storage type, 
replication and reason. Ratios such as the fallback share are computed in 
PromQL, and a Grafana dashboard explains them. Per-bucket and per-key detail 
stays in the CLI, not in Prometheus labels.

Scope:

OM: committed bytes by policy, tier and fallback outcome.
SCM:
placement failure reasons
ARCHIVE capacity
eligible datanodes per type
pipeline readiness per tier
repair outcome
replica drift from the container tier
Datanode: Disk Balancer results per storage type.
A Storage Policy Grafana dashboard with example alerts.
A bucket and volume compliance summary in ozone admin storagepolicy check.
Rules for all sub-tasks:

Keep existing aggregate metrics and names.
Add new series alongside them.
Report a missing tier or type as UNKNOWN, never as DISK.
Never use volume, bucket, key or exception text as a label.
Related work tracked elsewhere:

Container Balancer per-type metrics: HDDS-17590, under HDDS-17583.
Per-key policy check: HDDS-17710.
Bucket mismatch listing: HDDS-17756.
Fallback contract: HDDS-17535.

cc: [~anuragparvatikar] [~rakeshr][~rbarooah]


> Storage policy observability: metrics, dashboard and compliance reporting
> -------------------------------------------------------------------------
>
>                 Key: HDDS-17818
>                 URL: https://issues.apache.org/jira/browse/HDDS-17818
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: Ozone Dashboards, Ozone Datanode, Ozone Filesystem, 
> Ozone Manager, Ozone Recon
>            Reporter: Devesh Kumar Singh
>            Assignee: Devesh Kumar Singh
>            Priority: Major
>
> Ozone shows how full SSD and DISK are, but not whether storage policies are 
> working. An administrator cannot see:
> how much new HOT data is falling back from SSD to DISK
> why SCM could not place data on a tier
> whether enough SSD datanodes and pipelines are ready
> whether repaired replicas or balancer moves stayed on the right medium
> which buckets have data that doesn't match its policy
> Today no OM, SCM or datanode metric has a storage policy, tier or storage 
> type dimension. The only exception is the per-volume StorageType tag in 
> VolumeInfoMetrics. SCM exports per-type capacity only for DISK and SSD, not 
> ARCHIVE.
> This JIRA tracks the work to make storage policy outcomes visible. Ozone 
> emits raw counters and gauges with bounded labels: policy, tier, storage 
> type, replication and reason. Ratios such as the fallback share are computed 
> in PromQL, and a Grafana dashboard explains them. Per-bucket and per-key 
> detail stays in the CLI, not in Prometheus labels.
> Scope:
> OM: committed bytes by policy, tier and fallback outcome.
> SCM:
> placement failure reasons
> ARCHIVE capacity
> eligible datanodes per type
> pipeline readiness per tier
> repair outcome
> replica drift from the container tier
> Datanode: Disk Balancer results per storage type.
> A Storage Policy Grafana dashboard with example alerts.
> A bucket and volume compliance summary in ozone admin storagepolicy check.
> Rules for all sub-tasks:
> Keep existing aggregate metrics and names.
> Add new series alongside them.
> Report a missing tier or type as UNKNOWN, never as DISK.
> Never use volume, bucket, key or exception text as a label.
> Related work tracked elsewhere:
> Container Balancer per-type metrics: HDDS-17590, under HDDS-17583.
> Per-key policy check: HDDS-17710.
> Bucket mismatch listing: HDDS-17756.
> Fallback contract: HDDS-17535.
> cc: [~anuragparvatikar] [~rakeshr] [~rbarooah]



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to