[
https://issues.apache.org/jira/browse/HDDS-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127927#comment-18127927
]
Mohammadhani Fouladgar commented on HDDS-17075:
-----------------------------------------------
*Approach*
SCM leader measures its follower replication lag and advertises it as an
advisory signal on the existing {{deleteKeyBlocks}} response. OM reads it and
reduces (never stops) its per-task key deletion limit while the lag is above a
threshold, recovering automatically as the lag clears.
- Lag metric: {{leaderLastAppliedIndex − min(followerNextIndices)}} in Raft log
entries (same shape as OM's existing {{allowFollowerReadLocalLease}}).
- Signal: backward-compatible additive field on
{{DeleteScmKeyBlocksResponseProto}} (absent/old peer → 0 → no throttle).
- Policy split: SCM measures; OM decides (threshold + reduced limit), where the
deletion knobs already live.
*Why this way*
- Carrying the signal on the success response (not an error) means the current
batch still commits — no wasted DB scans and no tripping OM's SCM
failover/retry path, which would otherwise cause retry storms against an
already-struggling SCM.
- Reduce, not stop keeps OM sending smaller batches, so it keeps receiving
fresh lag readings and can un-throttle (avoids getting stuck).
- Scope limited to block deletion (the true async load); block allocation
(foreground client writes) is deliberately untouched.
*Config (enabled by default, conservative)
- {{ozone.om.key.deleting.scm.follower.lag.throttle.threshold}} = 100000
entries (≤0 disables)
- {{ozone.om.key.deleting.scm.lag.throttled.limit.per.task}} = 1000
Compatibility: additive {{optional}} proto field; mixed-version safe (old SCM →
no signal; old OM → ignores it).
> Throttle Ozone manager's load(async ops) to SCM when SCM followers are in
> highly lagged.
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-17075
> URL: https://issues.apache.org/jira/browse/HDDS-17075
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Mohammadhani Fouladgar
> Priority: Major
>
> *Problem*
> OM's background block deletion ({{{}KeyDeletingService{}}} →
> {{{}ScmBlockLocationProtocol.deleteKeyBlocks{}}}) writes transactions into
> SCM's Ratis-replicated {{{}DeletedBlockLog{}}}. When SCM followers lag the
> leader, this async load is exactly what backs up, but OM has no lag-aware
> throttle today — only static limits.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
