devmadhuu opened a new pull request, #12071:
URL: https://github.com/apache/ozone/pull/12071

   <html><head></head><body><h2>What changes were proposed in this pull 
request?</h2>
   <p>Add OM metrics showing how many logical bytes are committed for each 
storage policy, which tier SCM selected, and whether allocation used 
fallback.</p>
   <p>Existing SSD/DISK capacity metrics show storage usage, but cannot 
distinguish WARM data placed on DISK as requested from HOT data placed on DISK 
through fallback. The existing <code>TotalDataCommitted</code> counter also 
provides no policy or tier breakdown.</p>
   <p>This PR adds <code>OMStoragePolicyMetrics.CommittedBytes</code>, exported 
to Prometheus as:</p>
   <pre><code>om_storage_policy_metrics_committed_bytes
   </code></pre>
   <p>The metric has four bounded labels:</p>
   
   Label | Meaning | Values
   -- | -- | --
   policy | Policy recorded on the key | HOT, WARM, COLD, UNKNOWN
   requestedtier | Creation tier requested by that policy | SSD, DISK, ARCHIVE, 
UNKNOWN
   actualtier | SCM-selected allocation tier for the committed block | SSD, 
DISK, ARCHIVE, UNKNOWN
   outcome | Allocation outcome for that block | preferred, fallback, unknown
   
   
   <p>The counters add up to 12 MiB. Only the 4 MiB allocated through fallback 
contributes to the fallback series.</p>
   <h3>Implementation</h3>
   <p>Both ordinary and FSO final commits use the existing successful-commit 
metrics path.</p>
   <p>OM retains request-local access to the original open-key allocation 
metadata before committed client locations replace it. The metric matches 
accepted committed block IDs against that original metadata to obtain the 
selected tier and fallback flag.</p>
   <p>This is necessary because client commit locations can omit those 
allocation attributes. The metric uses OM's allocation evidence rather than 
trusting attributes echoed by the client.</p>
   <p>Unused preallocated blocks contribute no bytes. Missing allocation 
information is reported as UNKNOWN. Bytes not accounted for by accepted block 
lengths are also UNKNOWN. Negative lengths or lengths exceeding the key size 
cause the entire key size to be attributed to UNKNOWN/unknown, avoiding 
overcounting without changing commit behavior.</p>
   <p>Classification runs after the bucket lock is released. Metric 
registration and cleanup are owned by <code>OMMetrics</code>.</p>
   <h3>Counting semantics</h3>
   <ul>
   <li>Count logical bytes from successful final commits, including overwrites 
and recovery completion.</li>
   <li>Exclude failed commits, intermediate hsync calls, MPU part commits and 
MPU completion, matching the existing <code>TotalDataCommitted</code> 
boundary.</li>
   <li>Empty keys add no bytes.</li>
   <li>Preserve the existing <code>TotalDataCommitted</code> metric and its 
behavior.</li>
   <li>Counters reset on OM restart and do not scan or backfill existing 
keys.</li>
   <li>Each OM counts transactions it applies, including follower application. 
Select one OM for the service when measuring write traffic; do not sum 
replicated OM instances as independent writes.</li>
   </ul>
   <p>This is a cumulative write counter. Deleting a key does not subtract its 
previous writes. It does not represent currently retained logical data or 
current physical replica usage.</p>
   <p>For example, deleting the 5 GiB HOT data above leaves the historical 
fallback counter at 5 GiB. Current physical usage decreases separately after 
replica cleanup and refreshed datanode reports. Retained-data gauges are 
separate follow-up work.</p>
   <h2>What is the link to the Apache JIRA?</h2>
   <p>https://issues.apache.org/jira/browse/HDDS-17819</p>
   <p>Parent: https://issues.apache.org/jira/browse/HDDS-17818</p>
   <p>Target branch: <code>HDDS-11233</code></p>
   <h2>How was this patch tested?</h2>
   <p>Local validation using JDK 21 passed:</p>
   <ul>
   <li>137 OM unit/regression tests.</li>
   <li>10 <code>TestOzoneStoragePolicy</code> mini-cluster integration 
tests.</li>
   <li>Reactor compilation and test compilation.</li>
   <li>CI-aligned Checkstyle with zero violations.</li>
   <li>RAT for <code>ozone-manager</code>.</li>
   <li>PMD for <code>ozone-manager</code> and its dependencies.</li>
   <li><code>git diff --check</code>.</li>
   </ul>
   <p>Coverage includes:</p>
   <ul>
   <li>Preferred allocation for HOT, WARM and COLD.</li>
   <li>Unequal SSD/DISK splits and aggregation of multiple blocks.
   Hardcore- Omitted or conflicting client allocation attributes.</li>
   <li>Unused preallocation, empty blocks and unknown/legacy metadata.</li>
   <li>Inconsistent lengths, unaccounted bytes and overflow boundaries.</li>
   <li>Concurrent metric updates and scrapes.</li>
   <li>Logical-byte accounting for EC metadata.</li>
   <li>Source registration, cleanup and reset.</li>
   <li>Ordinary/FSO final commits, hsync followed by close or recovery, quota 
rejection and rejected repeat commits.</li>
   <li>Existing MPU part/completion behavior.</li>
   <li>Cluster writes covering preferred placement, HOT fallback, FSO, multiple 
blocks, empty keys and policy-changing overwrites.</li>
   </ul>
   </body></html>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to