[
https://issues.apache.org/jira/browse/HDDS-16125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng resolved HDDS-16125.
-------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
> Quota repair aliases the live cached OmBucketInfo and double-counts bucket
> usedBytes after crash and Ratis replay
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16125
> URL: https://issues.apache.org/jira/browse/HDDS-16125
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Siyao Meng
> Assignee: Jason O'Sullivan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
> Attachments: HDDS-16125.001.patch
>
>
> h3. Problem
> On an OM bucket, online quota repair can leave the durable {{usedBytes}}
> double-counted after a crash and Ratis replay: the bucket durably records
> usage 2 for a single 1-byte key.
> h3. Root cause
> {{OMQuotaRepairRequest.updateBucketInfo}} reads the bucket via
> {{getBucketTable().get()}} (a copy), then stores that same object into both
> the cache ({{addCacheEntry}}) and the client response map ({{bucketMap.put}})
> with no {{copyObject()}} (OMQuotaRepairRequest.java:120,144,146). Every other
> mutating request copies its response bucket for replay safety, for example
> {{omBucketInfo.copyObject()}} at OMKeyCommitRequest.java:410. A later key
> commit fetches the same live cached object via {{OMKeyRequest.getBucketInfo}}
> ({{getCacheValue}}, no copy; OMKeyRequest.java:903-910) and mutates it in
> place. If the double buffer serializes the repair batch after that in-place
> mutation but the key-commit batch is not yet durable, a crash strands a
> durable prefix that already contains the key commit's increment under the
> repair transaction index; on restart Ratis replays the committed key commit
> and the count is applied twice.
> h3. Trigger
> # Bucket {{usedBytes=1}} (drifted; true usage 0). Repair computes delta -1,
> live becomes 0; the response and cache entry alias one object.
> # The double buffer begins flushing the repair batch (index1) and pauses
> before serialization.
> # A 1-byte key commit (index2) reads the aliased live object (0) and
> increments it to 1.
> # The repair batch serializes the mutated aliased object, so durable index1
> records usedBytes=1 (repair's own result is 0).
> # A crash before index2 flushes loses the unflushed index2 and the in-memory
> cache.
> # Restart reloads caches from the durable DB (bucket=1) and Ratis replays
> committed index2, driving live and durable usedBytes to 2 while true usage is
> 1.
> h3. Impact
> {{OmBucketInfo.getUsedBytes()}} is read by
> {{OMKeyRequest.checkBucketQuotaInBytes}} on every key write, so an inflated
> count rejects legitimate writes early when a space quota is set, and {{ozone
> admin}} bucket usage reporting shows the wrong value. The wrong count is
> durable, and re-running repair is subject to the same race.
> h3. Suggested fix
> Store {{bucketInfo.copyObject()}} into the response map (and/or add a
> distinct copy to the cache) at OMQuotaRepairRequest.java:146, mirroring
> {{OMKeyCommitRequest}}, so the repair response owns an immutable snapshot and
> a later key commit's mutation cannot bleed into the repair's durable write.
> h3. Notes
> Discovered via TLA+ formal verification and model checking under HDDS-16123.
> All code claims were verified against the source and reproduced
> deterministically on a single-module ozone-manager JUnit test that drives the
> real OM state machine, double buffer, and request handlers (observed
> liveCount=2, durableCount=2, trueUsage=1). Pinned source commit
> 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb. Analysis assisted by AI tooling
> (Claude Code, Opus 4.8) via the Specula pipeline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
