Siyao Meng created HDDS-16126:
---------------------------------
Summary: Quota repair applies a stale name-only delta to a
recreated same-name bucket and drives usage counters negative
Key: HDDS-16126
URL: https://issues.apache.org/jira/browse/HDDS-16126
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Siyao Meng
h3. Problem
Online quota repair can apply a stale usage delta to the wrong bucket when a
bucket is deleted and recreated under the same name during the repair window,
driving the recreated bucket's {{usedBytes}} and {{usedNamespace}} negative.
h3. Root cause
Quota repair scans a point-in-time RocksDB checkpoint and submits per-bucket
deltas as {{BucketQuotaCount}}, which carries only {{volName}}, {{bucketName}},
and the deltas, with no bucket identity field
(OmClientProtocol.proto:2411-2419). On apply,
{{OMQuotaRepairRequest.updateBucketInfo}} resolves the bucket by name, guards
only for {{null}}, and applies the delta with {{incrUsedBytes}} and
{{incrUsedNamespace}} (OMQuotaRepairRequest.java:118-126), which are plain
{{+=}} with no clamp (OmBucketInfo.java:259-275). The scan captures the bucket
{{objectID}} locally, but the protobuf discards it, so a delta computed against
one generation is applied to whatever bucket currently holds the name.
h3. Trigger
# Bucket gen1 over-counts (for example usedNamespace=1, usedBytes=1000) with
zero live keys, which is the drift quota repair exists to correct. Bucket
deletion gates on {{isBucketEmpty}} (key table), not on the usage counters, so
an over-counted empty bucket is deletable.
# Repair checkpoints and scans gen1, computing a name-only delta of -1 and
-1000.
# Before the repair transaction commits, the bucket is deleted and recreated
under the same name; gen2 has a new objectID and zero usage.
# The repair transaction commits; {{updateBucketInfo}} resolves the name, finds
gen2, and applies the stale delta, moving gen2 from 0 to -1 and -1000.
h3. Impact
The recreated bucket's durable usage counters go negative. These feed key-write
quota enforcement ({{OMKeyRequest.checkBucketQuotaInBytes}}) and {{ozone
admin}} usage reporting: a negative usedBytes inflates available headroom and
permits writes beyond the configured quota. The wrong count is durable with no
automatic re-correction.
h3. Suggested fix
Add the scanned bucket {{objectID}} (and/or generation or updateID) to
{{BucketQuotaCount}} and, in {{updateBucketInfo}}, skip the delta unless the
loaded bucket's {{objectID}} matches the request's captured identity, reusing
the generation-fencing pattern already used elsewhere in the protocol. As
defense in depth, guard the counters against going negative.
h3. Notes
Discovered via TLA+ formal verification and model checking under HDDS-16123.
All code claims were verified against the source and reproduced with an
ozone-manager JUnit test driving the real {{QuotaRepairTask}} scan and the real
apply path: the scan emitted a name-only delta (diffUsedBytes=-1000,
diffUsedNamespace=-1) that landed on the recreated generation's objectID,
leaving usedNamespace=-1, usedBytes=-1000. Pinned source commit
9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb. Analysis assisted by AI tooling
(Claude Code, Opus 4.8) via the Specula pipeline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
