[
https://issues.apache.org/jira/browse/HDDS-16178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang resolved HDDS-16178.
------------------------------------
Resolution: Not A Problem
Not applicable per the discussion on HDDS-16127. The stale createVersion only
arises on a versioned overwrite, which no write path reaches today.
> Allocated blocks can survive under a stale createVersion after a versioned
> overwrite
> ------------------------------------------------------------------------------------
>
> Key: HDDS-16178
> URL: https://issues.apache.org/jira/browse/HDDS-16178
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Major
>
> h3. Problem
> On a versioning\-enabled overwrite, blocks that were allocated but never
> committed can stay attached to the committed key under a stale
> {{createVersion}}. Every caller that flattens a key's location groups, most
> importantly {{OMKeyRequest.sumBlockLengths}}, then counts those blocks at
> their full allocated length in addition to the committed ones.
> h3. Root cause
> Blocks handed out by {{allocateBlock}} carry the default {{createVersion}} of
> 0 and a length of the SCM block size. On an overwrite of a
> versioning\-enabled bucket, {{OMKeyRequest.prepareFileInfo}} calls
> {{addNewVersion\(locations, false, true\)}}, and
> {{OmKeyLocationInfoGroup.generateNextVersion}}
> \({{OmKeyLocationInfoGroup.java:160\-165}}\) files them under map key
> {{version \+ 1}} while the block objects still report {{createVersion=0}}:
> {code}
> OmKeyLocationInfoGroup generateNextVersion\(List newLocationList\) {
> Map> newMap = new HashMap<>\(\);
> newMap.put\(version \+ 1, new ArrayList<>\(newLocationList\)\);
> return new OmKeyLocationInfoGroup\(version \+ 1, newMap\);
> }
> {code}
> If the open key is written to RocksDB and read back before the commit is
> processed, {{OmKeyLocationInfoGroup.getFromProtobuf}}
> \({{OmKeyLocationInfoGroup.java:139\-148}}\) regroups the entries by their
> {{createVersion}}, so the new group's internal map becomes {{{0: \[allocated
> blocks\], 1: \[\]}}}, the empty entry coming from the constructor's
> {{putIfAbsent}}.
> Commit then only touches the latest key. {{OmKeyInfo.updateLocationInfoList}}
> \({{OmKeyInfo.java:330\-333}}\) runs
> {{keyLocationInfoGroup.removeBlocks\(latestVersion\)}} and
> {{addAll\(latestVersion, updatedBlockLocations\)}}, which clears and
> repopulates map key 1 and leaves the entries under map key 0 untouched.
> {{getLocationList\(\)}} flattens the whole map, so those stale entries are
> returned alongside the committed blocks.
> {{OMAllocateBlockRequest}} is not affected, because it appends through
> {{appendNewBlocks}}, which sets {{createVersion}} on each block.
> h3. Evidence
> The regrouping was reproduced in an ozone\-manager unit test. An open key
> built as an overwrite whose new blocks carry {{createVersion=0}}, written to
> the open key table and read back, has groups {{\[v0\[0\], v1\[0, 1\]\]}}: the
> block intended for version 1 sits under map key 0 of the version 1 group.
> The survival of those entries through commit follows from
> {{OmKeyInfo.java:330\-333}} but has not been reproduced end to end. An
> integration test using the real client, with the open key flushed before
> commit, would confirm it.
> h3. Impact
> Any consumer of the flattened block list over\-counts.
> {{OMKeyRequest.sumBlockLengths}} is used to release quota in
> {{OMKeyDeleteRequest}}, {{OMKeysDeleteRequest}}, the FSO variants,
> {{OMDirectoriesPurgeRequestWithFSO}} and {{KeyManagerImpl}}, so a delete can
> refund more than was charged. The quota repair recount in HDDS\-16127 counts
> only the blocks each version created, so it is not affected, which also means
> repair and delete disagree on such a key until this is fixed.
> h3. Notes
> Found while scoping HDDS\-16127. Pinned source commit 4766aa8609. Analysis
> assisted by AI tooling \(Claude Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
