Aleksander Nek created HDFS-18024:
-------------------------------------
Summary: Preserve copy-truncate source during last-snapshot
deletion on standby
Key: HDFS-18024
URL: https://issues.apache.org/jira/browse/HDFS-18024
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, snapshots
Affects Versions: 3.6.0
Reporter: Aleksander Nek
h2. Problem and impact
The copy-on-truncate source must survive until its replacement is completed,
even if the snapshot that retained the source is deleted. During OP_TRUNCATE
replay, a standby leaves the new target UNDER_CONSTRUCTION, whereas the active
uses UNDER_RECOVERY. FileDiffList.combineAndCollectSnapshotBlocks() only
protects a source whose target is UNDER_RECOVERY. Replaying deletion of the
last snapshot can therefore remove a still-needed source from the standby block
map.
After promotion, the missing metadata can cause the source replicas to be
treated as unreferenced during block reports and invalidated before recovery
creates a replacement. The regression directly proves premature metadata loss;
invalidation of every physical replica is a possible consequence, not an
assertion made by this test.
h2. Ordinary HA reproduction
# Start two NameNodes, three JournalNodes and three DataNodes with
MiniQJMHACluster; no feature-specific configuration.
# Create a 1536-byte file, block size 1024, replication 3, and its only
snapshot.
# Disable DataNode heartbeats to hold recovery pending; truncate the file by
one byte.
# Roll and manually tail finalized edits on standby. Verify UNDER_RECOVERY on
active versus UNDER_CONSTRUCTION on standby, both referencing the source block.
# Delete the snapshot and replay on standby.
# Expected: source remains in both block maps until recovery/CLOSE. Actual:
source is absent on standby.
h2. Fix
Protect the source whenever the target is incomplete. Before OP_CLOSE replay
completes the target and clears its under-construction feature, capture the
source. After replay, remove a distinct copy source only when it has no
latest-snapshot reference. Resolve the canonical stored block and detach its
inode owner before BlockManager removal.
No configuration, protocol, layout or fsimage format change. FSNamesystem
active-side cleanup is outside this patch.
This extends the active-side protection introduced by HDFS-7903
(6acb7f2110897264241df44d564db2f85260348f), which checks UNDER_RECOVERY only;
it covers the remaining HA replay case. HDFS-12638 concerns deleting the file
itself, which is not done here. The issue remains reproducible on current trunk
6c8dc8f2216cab134d5546f3327ad5907ef4b7d3.
h2. Validation
JDK 17.0.20.1, Maven 3.9.16, macOS ARM64, pure Java MiniClusters, one fork,
isolated Maven repository. All runs use -ea, -DenableAssertions=true and
-Dmaven.test.failure.ignore=false; the new fixture also checks
INodeFile.class.desiredAssertionStatus().
* Current trunk with the new test only: 3 cases, expected standby
source-retention failure 1, controls pass 2, errors/skips 0.
* Retention-only patch on c815ec8e (the two affected source files are unchanged
in 6c8dc8f): 3 cases, expected unreferenced-source leak after CLOSE failure 1,
controls pass 2, errors/skips 0.
* Complete patch on current trunk: TestTruncateSnapshotSource 3,
TestFileTruncate 24, TestFSEditLogLoader 26 (13 sync + 13 async): all 53 pass,
failures/errors/skips 0. Tests check failover, lease recovery, exact truncated
bytes, CLOSE replay cleanup on both NameNodes, and retaining a snapshot through
CLOSE until its later deletion.
* Checkstyle 11.1.0: zero diagnostics in the new test or on added runtime
lines; 30 existing diagnostics in the selected runtime files.
* Targeted SpotBugs 4.9.7 before/after for the two runtime classes: no new
findings (same 11 findings).
Log caveat: an active-side BlocksMap.removeBlock assertion is logged by
recovery in the new HA case and existing TestFileTruncate, despite their
successful JUnit results. A separate unmodified-trunk
TestFileTruncate#testUpgradeAndRestart control also logs this assertion (1/1
JUnit pass). FSNamesystem is unchanged and that active-side behavior is not
repaired or claimed fixed here. Physical source-GC completion is not claimed.
Native-library and shell/Bats suites were not run; Bats is unavailable locally.
Contains content generated by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]