[
https://issues.apache.org/jira/browse/HDDS-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-16288:
----------------------------------
Status: Patch Available (was: Open)
> Convert OMPerformanceMetrics MutableRate latency counters to lock-free
> ConcurrentMutableStat
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-16288
> URL: https://issues.apache.org/jira/browse/HDDS-16288
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> OMPerformanceMetrics declares ~44 latency counters as @Metric MutableRate
> fields
> (OMPerformanceMetrics.java). MutableRate extends MutableStat, whose add(long)
> is
> synchronized on the counter instance:
> public synchronized void add(long) // hadoop-common MutableStat
> These counters are updated on the OM read/write RPC hot path, and several are
> hit
> per request. A single getKeyInfo, for example, records into six or seven of
> them in
> sequence (OmMetadataReader.getKeyInfo, ~lines 160-215):
> getKeyInfoResolveBucketLatencyNs, getKeyInfoAclCheckLatencyNs,
> getKeyInfoReadKeyInfoLatencyNs, getKeyInfoRefreshLocationLatencyNs,
> getKeyInfoGenerateBlockTokenLatencyNs, getKeyInfoSortDatanodesLatencyNs,
> getKeyInfoLatencyNs
> lookup, listKeys, createKey and the ACL/checkAccess paths have their own
> sets. Each
> counter is a distinct monitor, so contention is per-counter across threads:
> when many
> handlers execute the same read op concurrently, they serialize on the same
> synchronized
> add(). This is the same thundering-herd contention that HDDS-9377 (PR #11085)
> removed
> from OMLockMetrics and the PerformanceMetrics wrapper by replacing
> MutableStat with the lock-free ConcurrentMutableStat — but the raw
> MutableRate counters in OMPerformanceMetrics were not part of that change and
> still use the synchronized path.
> h2. Approach
> Reuse the ConcurrentMutableStat introduced in HDDS-9377 (hadoop-hdds/common)
> for the OMPerformanceMetrics latency counters instead of MutableRate,
> preserving the emitted metric names, num/avg samples, and min/max/stddev
> semantics so dashboards and JMX consumers are unaffected. Because these
> fields are currently auto-created by the @Metric annotation factory (which
> produces MutableRate), registration moves to explicit
> construction against the MetricsRegistry (mirroring how OMLockMetrics
> registers its
> ConcurrentMutableStat), and the capture helpers (captureLatencyNs / add*
> methods) are
> retargeted to the new type.
> h2. Scope
> In scope:
> - The MutableRate latency counters in OMPerformanceMetrics on OM request
> paths.
> Out of scope:
> - OMLockMetrics and PerformanceMetrics (already covered by HDDS-9377).
> - Changing hadoop-common MutableRate/MutableStat.
> - MutableGaugeLong service-latency gauges in OMPerformanceMetrics
> (single-writer,
> not contended).
> h2. Testing
> - Unit coverage that a converted counter records count/sum/min/max
> equivalently to
> MutableRate and that JMX/metrics-sink snapshot output is unchanged.
> - Re-run the mixed read/write OM workload used for HDDS-9377 to confirm the
> residual
> per-counter contention on OMPerformanceMetrics is removed and read p99 under
> concurrent load is unaffected or improved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
