[
https://issues.apache.org/jira/browse/HDFS-17932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rstest updated HDFS-17932:
--------------------------
Description:
h1. Summary
SecondaryNameNode checkpoint retry can fail indefinitely during non-HA rolling
upgrade after replaying `OP_ROLLING_UPGRADE_START` twice
h1. Bug Symptom
During a non-HA HDFS rolling upgrade from Hadoop 2.10.2 to Hadoop 3.3.6, the
SecondaryNameNode can become stuck failing checkpoint retries after a transient
NameNode RPC failure.
The failure occurs when the SecondaryNameNode has already replayed
`OP_ROLLING_UPGRADE_START` while merging a checkpoint, then the checkpoint
merge fails later when the SecondaryNameNode calls
`namenode.isRollingUpgrade()` (in our buggy case the namenode is under upgrade
downtime and momentarily unreachable). On retry, the same SecondaryNameNode
process reloads the checkpoint inputs and replays `OP_ROLLING_UPGRADE_START`
again, but its local in-memory `FSNamesystem` still has `rollingUpgradeInfo`
active from the failed first attempt.
The retry then fails with a `RollingUpgradeException`, because
`FSNamesystem.checkRollingUpgrade("start rolling upgrade")` rejects starting a
rolling upgrade while one is already in progress.
h3. (1) START replay is unguarded; FINALIZE replay is guarded
In FSEditLogLoader.applyEditLogOp the two ops sit ten lines apart:
{code:java}
case OP_ROLLING_UPGRADE_START: {
...
fsNamesys.startRollingUpgradeInternal(startTime); // unconditional ->
checkRollingUpgrade -> throws if already active
...
}
case OP_ROLLING_UPGRADE_FINALIZE: {
if (fsNamesys.isRollingUpgrade()) { // guarded: "We can get
FINALIZE without corresponding START ..."
fsNamesys.finalizeRollingUpgradeInternal(finalizeTime);
}
...
}
{code}
startRollingUpgradeInternal() -> checkRollingUpgrade() throws
RollingUpgradeException when isRollingUpgrade() is already true. The op class
is annotated // @Idempotent, but this replay path is not idempotent.
h3. (2) Two ways rolling state is already set when START replays
* (a) The rollback fsimage restores it:
FSImageFormatProtobuf.loadNameSystemSection sets
fsn.setRollingUpgradeInfo(true, startTime) when the image has a rolling-upgrade
start time. Loading the rollback image (saved at prepare, after START) makes
isRollingUpgrade() true, then the replayed edits apply START again.
* (b) Stale state survives a failed merge: after a checkpoint-merge failure,
SecondaryNameNode.doCheckpoint forces a reload (hasMergeError()); doMerge ->
FSImage.reloadFromImageFile -> FSNamesystem.clear(). clear() resets the
namespace but does NOT clear rollingUpgradeInfo, so the old rolling state
survives and the replayed START throws.
A single fix - make OP_ROLLING_UPGRADE_START idempotent on replay (and/or reset
rolling-upgrade state in clear()) - covers both paths.
h2. Steps to Reproduce
The only required ingredient is that an SNN checkpoint's RPC to the NameNode
fails at/after the merge while a rolling
upgrade is active. A rolling upgrade restarts the NameNode to swap binaries,
which is exactly such a window, so no
artificial fault is needed:
# Non-HA cluster on 2.10.2 (NameNode + SecondaryNameNode), short checkpoint
period so a checkpoint is frequently in flight.
# Run dfsadmin -rollingUpgrade prepare (writes OP_ROLLING_UPGRADE_START and
saves a rollback fsimage that captures active rolling-upgrade state). The
rolling upgrade is now active.
# Upgrade the NameNode: stop it, swap to the 3.3.6 binary, start it. This
restart is a window in which the NameNode cannot answer the SecondaryNameNode.
# Arrange (by timing / short checkpoint period) for the upgraded
SecondaryNameNode to run a checkpoint whose isRollingUpgrade() RPC
(SecondaryNameNode.doMerge, line 1111) lands during that NameNode-down window,
so it gets an EOF.
*What happens:* the first checkpoint applies START cleanly (rolling state set
in memory), then the isRollingUpgrade() RPC at doMerge:1111 gets an EOF ->
doCheckpoint catches it and calls setMergeError(). The next checkpoint sees
hasMergeError(), reloads from the image -> FSNamesystem.clear() (which does not
reset rollingUpgradeInfo) -> replays OP_ROLLING_UPGRADE_START over the
still-set state.
*Expected:* the retry reloads the image and replays edits idempotently; the
SecondaryNameNode completes the checkpoint and stays up.
*Actual:* the retry replays OP_ROLLING_UPGRADE_START over already-active
rolling state; startRollingUpgradeInternal -> checkRollingUpgrade throws
RollingUpgradeException; every retry repeats; after "Merging failed 4 times"
the SecondaryNameNode exits with status 1.
h2. Exception
{code:java}
org.apache.hadoop.hdfs.protocol.RollingUpgradeException:
Failed to start rolling upgrade since a rolling upgrade is already in
progress.{code}
h2. Buggy code
The following snippets are from Hadoop 3.3.6.
`SecondaryNameNode.doCheckpoint()` intentionally retries by reloading image and
edits after a merge failure:
{code:java}
// SecondaryNameNode.java
// Fetch fsimage and edits. Reload the image if previous merge failed.
loadImage |= downloadCheckpointFiles(
fsName, checkpointImage, sig, manifest) |
checkpointImage.hasMergeError();
try {
doMerge(sig, manifest, loadImage, checkpointImage, namesystem);
} catch (IOException ioe) {
// A merge error occurred. The in-memory file system state may be
// inconsistent, so the image and edits need to be reloaded.
checkpointImage.setMergeError();
throw ioe;
}
checkpointImage.clearMergeError(); {code}
`SecondaryNameNode.doMerge()` first replays edits and saves a local checkpoint
image. Only after that does it call back to the active NameNode:
{code:java}
// SecondaryNameNode.java
if (loadImage) {
File file = dstStorage.findImageFile(NameNodeFile.IMAGE,
sig.mostRecentCheckpointTxId);
dstNamesystem.writeLock();
try {
dstImage.reloadFromImageFile(file, dstNamesystem);
} finally {
dstNamesystem.writeUnlock();
}
dstNamesystem.imageLoadComplete();
}Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);
dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());
if (!namenode.isRollingUpgrade()) {
dstImage.updateStorageVersion();
} {code}
During edit replay, `OP_ROLLING_UPGRADE_START` mutates the
SecondaryNameNode-local `FSNamesystem` by setting rolling-upgrade state:
{code:java}
// FSEditLogLoader.java
case OP_ROLLING_UPGRADE_START: {
final long startTime = ((RollingUpgradeOp) op).getTime();
fsNamesys.startRollingUpgradeInternal(startTime);
fsNamesys.triggerRollbackCheckpoint();
break;
}
// FSNamesystem.java
void startRollingUpgradeInternal(long startTime) throws IOException {
checkRollingUpgrade("start rolling upgrade");
getFSImage().checkUpgrade();
setRollingUpgradeInfo(false, startTime);
}void checkRollingUpgrade(String action) throws RollingUpgradeException {
if (isRollingUpgrade()) {
throw new RollingUpgradeException("Failed to " + action
+ " since a rolling upgrade is already in progress."
+ " Existing rolling upgrade info:\n" + rollingUpgradeInfo);
}
}{code}
The suspected stale-state bug is in the reload path. `reloadFromImageFile()`
calls `FSNamesystem.clear()`, but `clear()` does not clear `rollingUpgradeInfo`:
{code:java}
// FSImage.java
void reloadFromImageFile(File file, FSNamesystem target) throws IOException {
target.clear();
LOG.debug("Reloading namespace from " + file);
loadFSImage(file, target, null, false);
} {code}
{code:java}
// FSNamesystem.java
void clear() {
dir.reset();
dtSecretManager.reset();
leaseManager.removeAllLeases();
snapshotManager.clearSnapshottableDirs();
cacheManager.clear();
setImageLoaded(false);
blockManager.clear();
ErasureCodingPolicyManager.getInstance().clear();
} {code}
Because `rollingUpgradeInfo` is not cleared here, a retry after
`setMergeError()` can replay `OP_ROLLING_UPGRADE_START` while the
SecondaryNameNode-local `FSNamesystem` still reports rolling upgrade as active.
The second replay then fails in `checkRollingUpgrade("start rolling upgrade")`.
was:
h1. Summary
SecondaryNameNode checkpoint retry can fail indefinitely during non-HA rolling
upgrade after replaying `OP_ROLLING_UPGRADE_START` twice
h1. Bug Symptom
During a non-HA HDFS rolling upgrade from Hadoop 2.10.2 to Hadoop 3.3.6, the
SecondaryNameNode can become stuck failing checkpoint retries after a transient
NameNode RPC failure.
The failure occurs when the SecondaryNameNode has already replayed
`OP_ROLLING_UPGRADE_START` while merging a checkpoint, then the checkpoint
merge fails later when the SecondaryNameNode calls
`namenode.isRollingUpgrade()` (for example namenode is under upgrade downtime
and momentarily unreachable). On retry, the same SecondaryNameNode process
reloads the checkpoint inputs and replays `OP_ROLLING_UPGRADE_START` again, but
its local in-memory `FSNamesystem` still has `rollingUpgradeInfo` active from
the failed first attempt.
The retry then fails with a `RollingUpgradeException`, because
`FSNamesystem.checkRollingUpgrade("start rolling upgrade")` rejects starting a
rolling upgrade while one is already in progress.
h3. (1) START replay is unguarded; FINALIZE replay is guarded
In FSEditLogLoader.applyEditLogOp the two ops sit ten lines apart:
{code:java}
case OP_ROLLING_UPGRADE_START: {
...
fsNamesys.startRollingUpgradeInternal(startTime); // unconditional ->
checkRollingUpgrade -> throws if already active
...
}
case OP_ROLLING_UPGRADE_FINALIZE: {
if (fsNamesys.isRollingUpgrade()) { // guarded: "We can get
FINALIZE without corresponding START ..."
fsNamesys.finalizeRollingUpgradeInternal(finalizeTime);
}
...
}
{code}
startRollingUpgradeInternal() -> checkRollingUpgrade() throws
RollingUpgradeException when isRollingUpgrade() is already true. The op class
is annotated // @Idempotent, but this replay path is not idempotent.
h3. (2) Two ways rolling state is already set when START replays
* (a) The rollback fsimage restores it:
FSImageFormatProtobuf.loadNameSystemSection sets
fsn.setRollingUpgradeInfo(true, startTime) when the image has a rolling-upgrade
start time. Loading the rollback image (saved at prepare, after START) makes
isRollingUpgrade() true, then the replayed edits apply START again.
* (b) Stale state survives a failed merge: after a checkpoint-merge failure,
SecondaryNameNode.doCheckpoint forces a reload (hasMergeError()); doMerge ->
FSImage.reloadFromImageFile -> FSNamesystem.clear(). clear() resets the
namespace but does NOT clear rollingUpgradeInfo, so the old rolling state
survives and the replayed START throws.
A single fix - make OP_ROLLING_UPGRADE_START idempotent on replay (and/or reset
rolling-upgrade state in clear()) - covers both paths.
h2. Steps to Reproduce
The only required ingredient is that an SNN checkpoint's RPC to the NameNode
fails at/after the merge while a rolling
upgrade is active. A rolling upgrade restarts the NameNode to swap binaries,
which is exactly such a window, so no
artificial fault is needed:
# Non-HA cluster on 2.10.2 (NameNode + SecondaryNameNode), short checkpoint
period so a checkpoint is frequently in flight.
# Run dfsadmin -rollingUpgrade prepare (writes OP_ROLLING_UPGRADE_START and
saves a rollback fsimage that captures active rolling-upgrade state). The
rolling upgrade is now active.
# Upgrade the NameNode: stop it, swap to the 3.3.6 binary, start it. This
restart is a window in which the NameNode cannot answer the SecondaryNameNode.
# Arrange (by timing / short checkpoint period) for the upgraded
SecondaryNameNode to run a checkpoint whose isRollingUpgrade() RPC
(SecondaryNameNode.doMerge, line 1111) lands during that NameNode-down window,
so it gets an EOF.
*What happens:* the first checkpoint applies START cleanly (rolling state set
in memory), then the isRollingUpgrade() RPC at doMerge:1111 gets an EOF ->
doCheckpoint catches it and calls setMergeError(). The next checkpoint sees
hasMergeError(), reloads from the image -> FSNamesystem.clear() (which does not
reset rollingUpgradeInfo) -> replays OP_ROLLING_UPGRADE_START over the
still-set state.
*Expected:* the retry reloads the image and replays edits idempotently; the
SecondaryNameNode completes the checkpoint and stays up.
*Actual:* the retry replays OP_ROLLING_UPGRADE_START over already-active
rolling state; startRollingUpgradeInternal -> checkRollingUpgrade throws
RollingUpgradeException; every retry repeats; after "Merging failed 4 times"
the SecondaryNameNode exits with status 1.
h2. Exception
{code:java}
org.apache.hadoop.hdfs.protocol.RollingUpgradeException:
Failed to start rolling upgrade since a rolling upgrade is already in
progress.{code}
h2. Buggy code
The following snippets are from Hadoop 3.3.6.
`SecondaryNameNode.doCheckpoint()` intentionally retries by reloading image and
edits after a merge failure:
{code:java}
// SecondaryNameNode.java
// Fetch fsimage and edits. Reload the image if previous merge failed.
loadImage |= downloadCheckpointFiles(
fsName, checkpointImage, sig, manifest) |
checkpointImage.hasMergeError();
try {
doMerge(sig, manifest, loadImage, checkpointImage, namesystem);
} catch (IOException ioe) {
// A merge error occurred. The in-memory file system state may be
// inconsistent, so the image and edits need to be reloaded.
checkpointImage.setMergeError();
throw ioe;
}
checkpointImage.clearMergeError(); {code}
`SecondaryNameNode.doMerge()` first replays edits and saves a local checkpoint
image. Only after that does it call back to the active NameNode:
{code:java}
// SecondaryNameNode.java
if (loadImage) {
File file = dstStorage.findImageFile(NameNodeFile.IMAGE,
sig.mostRecentCheckpointTxId);
dstNamesystem.writeLock();
try {
dstImage.reloadFromImageFile(file, dstNamesystem);
} finally {
dstNamesystem.writeUnlock();
}
dstNamesystem.imageLoadComplete();
}Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);
dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());
if (!namenode.isRollingUpgrade()) {
dstImage.updateStorageVersion();
} {code}
During edit replay, `OP_ROLLING_UPGRADE_START` mutates the
SecondaryNameNode-local `FSNamesystem` by setting rolling-upgrade state:
{code:java}
// FSEditLogLoader.java
case OP_ROLLING_UPGRADE_START: {
final long startTime = ((RollingUpgradeOp) op).getTime();
fsNamesys.startRollingUpgradeInternal(startTime);
fsNamesys.triggerRollbackCheckpoint();
break;
}
// FSNamesystem.java
void startRollingUpgradeInternal(long startTime) throws IOException {
checkRollingUpgrade("start rolling upgrade");
getFSImage().checkUpgrade();
setRollingUpgradeInfo(false, startTime);
}void checkRollingUpgrade(String action) throws RollingUpgradeException {
if (isRollingUpgrade()) {
throw new RollingUpgradeException("Failed to " + action
+ " since a rolling upgrade is already in progress."
+ " Existing rolling upgrade info:\n" + rollingUpgradeInfo);
}
}{code}
The suspected stale-state bug is in the reload path. `reloadFromImageFile()`
calls `FSNamesystem.clear()`, but `clear()` does not clear `rollingUpgradeInfo`:
{code:java}
// FSImage.java
void reloadFromImageFile(File file, FSNamesystem target) throws IOException {
target.clear();
LOG.debug("Reloading namespace from " + file);
loadFSImage(file, target, null, false);
} {code}
{code:java}
// FSNamesystem.java
void clear() {
dir.reset();
dtSecretManager.reset();
leaseManager.removeAllLeases();
snapshotManager.clearSnapshottableDirs();
cacheManager.clear();
setImageLoaded(false);
blockManager.clear();
ErasureCodingPolicyManager.getInstance().clear();
} {code}
Because `rollingUpgradeInfo` is not cleared here, a retry after
`setMergeError()` can replay `OP_ROLLING_UPGRADE_START` while the
SecondaryNameNode-local `FSNamesystem` still reports rolling upgrade as active.
The second replay then fails in `checkRollingUpgrade("start rolling upgrade")`.
> SecondaryNameNode checkpoint stuck in retry infinitely after rolling upgrade
> ----------------------------------------------------------------------------
>
> Key: HDFS-17932
> URL: https://issues.apache.org/jira/browse/HDFS-17932
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, rolling upgrades
> Affects Versions: 2.10.2, 3.3.6, 3.4.2
> Reporter: rstest
> Priority: Major
>
> h1. Summary
> SecondaryNameNode checkpoint retry can fail indefinitely during non-HA
> rolling upgrade after replaying `OP_ROLLING_UPGRADE_START` twice
> h1. Bug Symptom
> During a non-HA HDFS rolling upgrade from Hadoop 2.10.2 to Hadoop 3.3.6, the
> SecondaryNameNode can become stuck failing checkpoint retries after a
> transient NameNode RPC failure.
> The failure occurs when the SecondaryNameNode has already replayed
> `OP_ROLLING_UPGRADE_START` while merging a checkpoint, then the checkpoint
> merge fails later when the SecondaryNameNode calls
> `namenode.isRollingUpgrade()` (in our buggy case the namenode is under
> upgrade downtime and momentarily unreachable). On retry, the same
> SecondaryNameNode process reloads the checkpoint inputs and replays
> `OP_ROLLING_UPGRADE_START` again, but its local in-memory `FSNamesystem`
> still has `rollingUpgradeInfo` active from the failed first attempt.
> The retry then fails with a `RollingUpgradeException`, because
> `FSNamesystem.checkRollingUpgrade("start rolling upgrade")` rejects starting
> a rolling upgrade while one is already in progress.
> h3. (1) START replay is unguarded; FINALIZE replay is guarded
> In FSEditLogLoader.applyEditLogOp the two ops sit ten lines apart:
> {code:java}
> case OP_ROLLING_UPGRADE_START: {
> ...
> fsNamesys.startRollingUpgradeInternal(startTime); // unconditional ->
> checkRollingUpgrade -> throws if already active
> ...
> }
> case OP_ROLLING_UPGRADE_FINALIZE: {
> if (fsNamesys.isRollingUpgrade()) { // guarded: "We can get
> FINALIZE without corresponding START ..."
> fsNamesys.finalizeRollingUpgradeInternal(finalizeTime);
> }
> ...
> }
> {code}
> startRollingUpgradeInternal() -> checkRollingUpgrade() throws
> RollingUpgradeException when isRollingUpgrade() is already true. The op class
> is annotated // @Idempotent, but this replay path is not idempotent.
> h3. (2) Two ways rolling state is already set when START replays
> * (a) The rollback fsimage restores it:
> FSImageFormatProtobuf.loadNameSystemSection sets
> fsn.setRollingUpgradeInfo(true, startTime) when the image has a
> rolling-upgrade start time. Loading the rollback image (saved at prepare,
> after START) makes isRollingUpgrade() true, then the replayed edits apply
> START again.
> * (b) Stale state survives a failed merge: after a checkpoint-merge failure,
> SecondaryNameNode.doCheckpoint forces a reload (hasMergeError()); doMerge ->
> FSImage.reloadFromImageFile -> FSNamesystem.clear(). clear() resets the
> namespace but does NOT clear rollingUpgradeInfo, so the old rolling state
> survives and the replayed START throws.
> A single fix - make OP_ROLLING_UPGRADE_START idempotent on replay (and/or
> reset rolling-upgrade state in clear()) - covers both paths.
> h2. Steps to Reproduce
> The only required ingredient is that an SNN checkpoint's RPC to the NameNode
> fails at/after the merge while a rolling
> upgrade is active. A rolling upgrade restarts the NameNode to swap binaries,
> which is exactly such a window, so no
> artificial fault is needed:
> # Non-HA cluster on 2.10.2 (NameNode + SecondaryNameNode), short checkpoint
> period so a checkpoint is frequently in flight.
> # Run dfsadmin -rollingUpgrade prepare (writes OP_ROLLING_UPGRADE_START and
> saves a rollback fsimage that captures active rolling-upgrade state). The
> rolling upgrade is now active.
> # Upgrade the NameNode: stop it, swap to the 3.3.6 binary, start it. This
> restart is a window in which the NameNode cannot answer the SecondaryNameNode.
> # Arrange (by timing / short checkpoint period) for the upgraded
> SecondaryNameNode to run a checkpoint whose isRollingUpgrade() RPC
> (SecondaryNameNode.doMerge, line 1111) lands during that NameNode-down
> window, so it gets an EOF.
> *What happens:* the first checkpoint applies START cleanly (rolling state set
> in memory), then the isRollingUpgrade() RPC at doMerge:1111 gets an EOF ->
> doCheckpoint catches it and calls setMergeError(). The next checkpoint sees
> hasMergeError(), reloads from the image -> FSNamesystem.clear() (which does
> not reset rollingUpgradeInfo) -> replays OP_ROLLING_UPGRADE_START over the
> still-set state.
> *Expected:* the retry reloads the image and replays edits idempotently; the
> SecondaryNameNode completes the checkpoint and stays up.
> *Actual:* the retry replays OP_ROLLING_UPGRADE_START over already-active
> rolling state; startRollingUpgradeInternal -> checkRollingUpgrade throws
> RollingUpgradeException; every retry repeats; after "Merging failed 4 times"
> the SecondaryNameNode exits with status 1.
> h2. Exception
> {code:java}
> org.apache.hadoop.hdfs.protocol.RollingUpgradeException:
> Failed to start rolling upgrade since a rolling upgrade is already in
> progress.{code}
> h2. Buggy code
> The following snippets are from Hadoop 3.3.6.
> `SecondaryNameNode.doCheckpoint()` intentionally retries by reloading image
> and edits after a merge failure:
> {code:java}
> // SecondaryNameNode.java
> // Fetch fsimage and edits. Reload the image if previous merge failed.
> loadImage |= downloadCheckpointFiles(
> fsName, checkpointImage, sig, manifest) |
> checkpointImage.hasMergeError();
> try {
> doMerge(sig, manifest, loadImage, checkpointImage, namesystem);
> } catch (IOException ioe) {
> // A merge error occurred. The in-memory file system state may be
> // inconsistent, so the image and edits need to be reloaded.
> checkpointImage.setMergeError();
> throw ioe;
> }
> checkpointImage.clearMergeError(); {code}
> `SecondaryNameNode.doMerge()` first replays edits and saves a local
> checkpoint image. Only after that does it call back to the active NameNode:
> {code:java}
> // SecondaryNameNode.java
> if (loadImage) {
> File file = dstStorage.findImageFile(NameNodeFile.IMAGE,
> sig.mostRecentCheckpointTxId);
> dstNamesystem.writeLock();
> try {
> dstImage.reloadFromImageFile(file, dstNamesystem);
> } finally {
> dstNamesystem.writeUnlock();
> }
> dstNamesystem.imageLoadComplete();
> }Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);
> dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());
> if (!namenode.isRollingUpgrade()) {
> dstImage.updateStorageVersion();
> } {code}
> During edit replay, `OP_ROLLING_UPGRADE_START` mutates the
> SecondaryNameNode-local `FSNamesystem` by setting rolling-upgrade state:
> {code:java}
> // FSEditLogLoader.java
> case OP_ROLLING_UPGRADE_START: {
> final long startTime = ((RollingUpgradeOp) op).getTime();
> fsNamesys.startRollingUpgradeInternal(startTime);
> fsNamesys.triggerRollbackCheckpoint();
> break;
> }
> // FSNamesystem.java
> void startRollingUpgradeInternal(long startTime) throws IOException {
> checkRollingUpgrade("start rolling upgrade");
> getFSImage().checkUpgrade();
> setRollingUpgradeInfo(false, startTime);
> }void checkRollingUpgrade(String action) throws RollingUpgradeException {
> if (isRollingUpgrade()) {
> throw new RollingUpgradeException("Failed to " + action
> + " since a rolling upgrade is already in progress."
> + " Existing rolling upgrade info:\n" + rollingUpgradeInfo);
> }
> }{code}
> The suspected stale-state bug is in the reload path. `reloadFromImageFile()`
> calls `FSNamesystem.clear()`, but `clear()` does not clear
> `rollingUpgradeInfo`:
> {code:java}
> // FSImage.java
> void reloadFromImageFile(File file, FSNamesystem target) throws IOException {
> target.clear();
> LOG.debug("Reloading namespace from " + file);
> loadFSImage(file, target, null, false);
> } {code}
> {code:java}
> // FSNamesystem.java
> void clear() {
> dir.reset();
> dtSecretManager.reset();
> leaseManager.removeAllLeases();
> snapshotManager.clearSnapshottableDirs();
> cacheManager.clear();
> setImageLoaded(false);
> blockManager.clear();
> ErasureCodingPolicyManager.getInstance().clear();
> } {code}
> Because `rollingUpgradeInfo` is not cleared here, a retry after
> `setMergeError()` can replay `OP_ROLLING_UPGRADE_START` while the
> SecondaryNameNode-local `FSNamesystem` still reports rolling upgrade as
> active. The second replay then fails in `checkRollingUpgrade("start rolling
> upgrade")`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
