[
https://issues.apache.org/jira/browse/HDFS-17932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rstest updated HDFS-17932:
--------------------------
Description:
h1. Summary
SecondaryNameNode checkpoint retry can fail indefinitely during non-HA rolling
upgrade after replaying `OP_ROLLING_UPGRADE_START` twice
h1. Bug Symptom
During a non-HA HDFS rolling upgrade from Hadoop 2.10.2 to Hadoop 3.3.6, the
SecondaryNameNode can become stuck failing checkpoint retries after a transient
NameNode RPC failure.
The failure occurs when the SecondaryNameNode has already replayed
`OP_ROLLING_UPGRADE_START` while merging a checkpoint, then the checkpoint
merge fails later when the SecondaryNameNode calls
`namenode.isRollingUpgrade()`. On retry, the same SecondaryNameNode process
reloads the checkpoint inputs and replays `OP_ROLLING_UPGRADE_START` again, but
its local in-memory `FSNamesystem` still has `rollingUpgradeInfo` active from
the failed first attempt.
The retry then fails with a `RollingUpgradeException`, because
`FSNamesystem.checkRollingUpgrade("start rolling upgrade")` rejects starting a
rolling upgrade while one is already in progress.
h2. Expected behavior:
- A transient NameNode RPC failure during SecondaryNameNode checkpoint merge
should be recoverable.
- A checkpoint retry should not fail because stale in-memory rolling-upgrade
state from the failed merge attempt remains active.
h2. Actual behavior:
- The running SecondaryNameNode checkpoint loop can remain stuck.
- Checkpoints stop being produced/uploaded by the SecondaryNameNode.
- Edit logs can continue accumulating until the SecondaryNameNode is restarted
or local checkpoint state is manually cleaned up.
h2. Relevant code path:
- `SecondaryNameNode.doCheckpoint()` calls `doMerge(...)`.
- `SecondaryNameNode.doMerge()` calls
`Checkpointer.rollForwardByApplyingLogs(...)`.
- Edit replay sees `OP_ROLLING_UPGRADE_START`.
- `FSEditLogLoader` calls `fsNamesys.startRollingUpgradeInternal(startTime)`.
- `startRollingUpgradeInternal` sets `rollingUpgradeInfo` in the
SecondaryNameNode-local `FSNamesystem`.
- `doMerge()` saves the merged local fsimage, then calls
`namenode.isRollingUpgrade()`.
- If that RPC fails, `doCheckpoint()` calls `checkpointImage.setMergeError()`.
- Retry reloads/replays image+edits, but the local `rollingUpgradeInfo` state
is still active.
- Replaying `OP_ROLLING_UPGRADE_START` again throws because rolling upgrade is
already in progress.
h2. Version pairs tested:
- Hadoop 2.10.2 -> Hadoop 3.3.6: issue observed.
- Hadoop 3.3.6 -> Hadoop 3.4.2: also covered by upgrade testing, but this
specific failure was observed on 2.10.2 -> 3.3.6.
h2. How To Reproduce
One way to reproduce is to force a transient NameNode RPC failure during the
narrow checkpoint window after the SecondaryNameNode has replayed
`OP_ROLLING_UPGRADE_START` but before `SecondaryNameNode.doMerge()` completes.
1. Start a non-HA HDFS cluster on Hadoop 2.10.2 with: one NameNode, one
SecondaryNameNode and one DataNode
2. Prepare a rolling upgrade on the NameNode. This should create a rollback
image and write a rolling-upgrade START operation into the edit log.
3. Upgrade/start the SecondaryNameNode on Hadoop 3.3.6 while the rolling
upgrade is prepared and not finalized.
4. Trigger or wait for a SecondaryNameNode checkpoint. The SecondaryNameNode
should download the rollback fsimage and edit logs, then replay edits
containing `OP_ROLLING_UPGRADE_START`.
5. After the SecondaryNameNode has replayed `OP_ROLLING_UPGRADE_START` and
saved the merged local checkpoint image, but before `doMerge()` completes, make
the NameNode RPC endpoint temporarily unavailable. A practical way to induce
this is to restart the NameNode at this point. The observed failure point is
the SecondaryNameNode RPC call to `namenode.isRollingUpgrade()`, which can fail
with an EOF/connection-closed error if the NameNode is down.
6. Bring the NameNode back and let the same SecondaryNameNode process retry
checkpointing.
7. Observe that the retry reloads/replays the checkpoint inputs and encounters
`OP_ROLLING_UPGRADE_START` again.
8. The retry fails because the SecondaryNameNode-local `rollingUpgradeInfo`
from the failed previous merge attempt is still active.
h2. Exception
{code:java}
org.apache.hadoop.hdfs.protocol.RollingUpgradeException:
Failed to start rolling upgrade since a rolling upgrade is already in
progress.{code}
h2. Buggy code
The following snippets are from Hadoop 3.3.6.
`SecondaryNameNode.doCheckpoint()` intentionally retries by reloading image and
edits after a merge failure:
{code:java}
// SecondaryNameNode.java
// Fetch fsimage and edits. Reload the image if previous merge failed.
loadImage |= downloadCheckpointFiles(
fsName, checkpointImage, sig, manifest) |
checkpointImage.hasMergeError();
try {
doMerge(sig, manifest, loadImage, checkpointImage, namesystem);
} catch (IOException ioe) {
// A merge error occurred. The in-memory file system state may be
// inconsistent, so the image and edits need to be reloaded.
checkpointImage.setMergeError();
throw ioe;
}
checkpointImage.clearMergeError(); {code}
`SecondaryNameNode.doMerge()` first replays edits and saves a local checkpoint
image. Only after that does it call back to the active NameNode:
{code:java}
// SecondaryNameNode.java
if (loadImage) {
File file = dstStorage.findImageFile(NameNodeFile.IMAGE,
sig.mostRecentCheckpointTxId);
dstNamesystem.writeLock();
try {
dstImage.reloadFromImageFile(file, dstNamesystem);
} finally {
dstNamesystem.writeUnlock();
}
dstNamesystem.imageLoadComplete();
}Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);
dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());
if (!namenode.isRollingUpgrade()) {
dstImage.updateStorageVersion();
} {code}
During edit replay, `OP_ROLLING_UPGRADE_START` mutates the
SecondaryNameNode-local `FSNamesystem` by setting rolling-upgrade state:
{code:java}
// FSEditLogLoader.java
case OP_ROLLING_UPGRADE_START: {
final long startTime = ((RollingUpgradeOp) op).getTime();
fsNamesys.startRollingUpgradeInternal(startTime);
fsNamesys.triggerRollbackCheckpoint();
break;
}
// FSNamesystem.java
void startRollingUpgradeInternal(long startTime) throws IOException {
checkRollingUpgrade("start rolling upgrade");
getFSImage().checkUpgrade();
setRollingUpgradeInfo(false, startTime);
}void checkRollingUpgrade(String action) throws RollingUpgradeException {
if (isRollingUpgrade()) {
throw new RollingUpgradeException("Failed to " + action
+ " since a rolling upgrade is already in progress."
+ " Existing rolling upgrade info:\n" + rollingUpgradeInfo);
}
}{code}
The suspected stale-state bug is in the reload path. `reloadFromImageFile()`
calls `FSNamesystem.clear()`, but `clear()` does not clear `rollingUpgradeInfo`:
{code:java}
// FSImage.java
void reloadFromImageFile(File file, FSNamesystem target) throws IOException {
target.clear();
LOG.debug("Reloading namespace from " + file);
loadFSImage(file, target, null, false);
} {code}
{code:java}
// FSNamesystem.java
void clear() {
dir.reset();
dtSecretManager.reset();
leaseManager.removeAllLeases();
snapshotManager.clearSnapshottableDirs();
cacheManager.clear();
setImageLoaded(false);
blockManager.clear();
ErasureCodingPolicyManager.getInstance().clear();
} {code}
Because `rollingUpgradeInfo` is not cleared here, a retry after
`setMergeError()` can replay `OP_ROLLING_UPGRADE_START` while the
SecondaryNameNode-local `FSNamesystem` still reports rolling upgrade as active.
The second replay then fails in `checkRollingUpgrade("start rolling upgrade")`.
was:
h1. Summary
SecondaryNameNode checkpoint retry can fail indefinitely during non-HA rolling
upgrade after replaying `OP_ROLLING_UPGRADE_START` twice
h1. Bug Symptom
During a non-HA HDFS rolling upgrade from Hadoop 2.10.2 to Hadoop 3.3.6, the
SecondaryNameNode can become stuck failing checkpoint retries after a transient
NameNode RPC failure.
The failure occurs when the SecondaryNameNode has already replayed
`OP_ROLLING_UPGRADE_START` while merging a checkpoint, then the checkpoint
merge fails later when the SecondaryNameNode calls
`namenode.isRollingUpgrade()`. On retry, the same SecondaryNameNode process
reloads the checkpoint inputs and replays `OP_ROLLING_UPGRADE_START` again, but
its local in-memory `FSNamesystem` still has `rollingUpgradeInfo` active from
the failed first attempt.
The retry then fails with a `RollingUpgradeException`, because
`FSNamesystem.checkRollingUpgrade("start rolling upgrade")` rejects starting a
rolling upgrade while one is already in progress.
h2. Expected behavior:
- A transient NameNode RPC failure during SecondaryNameNode checkpoint merge
should be recoverable.
- A checkpoint retry should not fail because stale in-memory rolling-upgrade
state from the failed merge attempt remains active.
h2. Actual behavior:
- The running SecondaryNameNode checkpoint loop can remain stuck.
- Checkpoints stop being produced/uploaded by the SecondaryNameNode.
- Edit logs can continue accumulating until the SecondaryNameNode is restarted
or local checkpoint state is manually cleaned up.
h2. Relevant code path:
- `SecondaryNameNode.doCheckpoint()` calls `doMerge(...)`.
- `SecondaryNameNode.doMerge()` calls
`Checkpointer.rollForwardByApplyingLogs(...)`.
- Edit replay sees `OP_ROLLING_UPGRADE_START`.
- `FSEditLogLoader` calls `fsNamesys.startRollingUpgradeInternal(startTime)`.
- `startRollingUpgradeInternal` sets `rollingUpgradeInfo` in the
SecondaryNameNode-local `FSNamesystem`.
- `doMerge()` saves the merged local fsimage, then calls
`namenode.isRollingUpgrade()`.
- If that RPC fails, `doCheckpoint()` calls `checkpointImage.setMergeError()`.
- Retry reloads/replays image+edits, but the local `rollingUpgradeInfo` state
is still active.
- Replaying `OP_ROLLING_UPGRADE_START` again throws because rolling upgrade is
already in progress.
h2. Version pairs tested:
- Hadoop 2.10.2 -> Hadoop 3.3.6: issue observed.
- Hadoop 3.3.6 -> Hadoop 3.4.2: also covered by upgrade testing, but this
specific failure was observed on 2.10.2 -> 3.3.6.
h2. How To Reproduce
One way to reproduce is to force a transient NameNode RPC failure during the
narrow checkpoint window after the SecondaryNameNode has replayed
`OP_ROLLING_UPGRADE_START` but before `SecondaryNameNode.doMerge()` completes.
1. Start a non-HA HDFS cluster on Hadoop 2.10.2 with: one NameNode, one
SecondaryNameNode and one DataNode
2. Prepare a rolling upgrade on the NameNode. This should create a rollback
image and write a rolling-upgrade START operation into the edit log.
3. Upgrade/start the SecondaryNameNode on Hadoop 3.3.6 while the rolling
upgrade is prepared and not finalized.
4. Trigger or wait for a SecondaryNameNode checkpoint. The SecondaryNameNode
should download the rollback fsimage and edit logs, then replay edits
containing `OP_ROLLING_UPGRADE_START`.
5. After the SecondaryNameNode has replayed `OP_ROLLING_UPGRADE_START` and
saved the merged local checkpoint image, but before `doMerge()` completes, make
the NameNode RPC endpoint temporarily unavailable. A practical way to induce
this is to restart the NameNode at this point. The observed failure point is
the SecondaryNameNode RPC call to `namenode.isRollingUpgrade()`, which can fail
with an EOF/connection-closed error if the NameNode is down.
6. Bring the NameNode back and let the same SecondaryNameNode process retry
checkpointing.
7. Observe that the retry reloads/replays the checkpoint inputs and encounters
`OP_ROLLING_UPGRADE_START` again.
8. The retry fails because the SecondaryNameNode-local `rollingUpgradeInfo`
from the failed previous merge attempt is still active.
h2. Exception
{code:java}
org.apache.hadoop.hdfs.protocol.RollingUpgradeException:
Failed to start rolling upgrade since a rolling upgrade is already in
progress.{code}
> SecondaryNameNode checkpoint stuck in retry infinitely after rolling upgrade
> ----------------------------------------------------------------------------
>
> Key: HDFS-17932
> URL: https://issues.apache.org/jira/browse/HDFS-17932
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, rolling upgrades
> Affects Versions: 2.10.2, 3.3.6, 3.4.2
> Reporter: rstest
> Priority: Major
>
> h1. Summary
> SecondaryNameNode checkpoint retry can fail indefinitely during non-HA
> rolling upgrade after replaying `OP_ROLLING_UPGRADE_START` twice
> h1. Bug Symptom
> During a non-HA HDFS rolling upgrade from Hadoop 2.10.2 to Hadoop 3.3.6, the
> SecondaryNameNode can become stuck failing checkpoint retries after a
> transient NameNode RPC failure.
> The failure occurs when the SecondaryNameNode has already replayed
> `OP_ROLLING_UPGRADE_START` while merging a checkpoint, then the checkpoint
> merge fails later when the SecondaryNameNode calls
> `namenode.isRollingUpgrade()`. On retry, the same SecondaryNameNode process
> reloads the checkpoint inputs and replays `OP_ROLLING_UPGRADE_START` again,
> but its local in-memory `FSNamesystem` still has `rollingUpgradeInfo` active
> from the failed first attempt.
> The retry then fails with a `RollingUpgradeException`, because
> `FSNamesystem.checkRollingUpgrade("start rolling upgrade")` rejects starting
> a rolling upgrade while one is already in progress.
> h2. Expected behavior:
> - A transient NameNode RPC failure during SecondaryNameNode checkpoint merge
> should be recoverable.
> - A checkpoint retry should not fail because stale in-memory rolling-upgrade
> state from the failed merge attempt remains active.
> h2. Actual behavior:
> - The running SecondaryNameNode checkpoint loop can remain stuck.
> - Checkpoints stop being produced/uploaded by the SecondaryNameNode.
> - Edit logs can continue accumulating until the SecondaryNameNode is
> restarted or local checkpoint state is manually cleaned up.
> h2. Relevant code path:
> - `SecondaryNameNode.doCheckpoint()` calls `doMerge(...)`.
> - `SecondaryNameNode.doMerge()` calls
> `Checkpointer.rollForwardByApplyingLogs(...)`.
> - Edit replay sees `OP_ROLLING_UPGRADE_START`.
> - `FSEditLogLoader` calls `fsNamesys.startRollingUpgradeInternal(startTime)`.
> - `startRollingUpgradeInternal` sets `rollingUpgradeInfo` in the
> SecondaryNameNode-local `FSNamesystem`.
> - `doMerge()` saves the merged local fsimage, then calls
> `namenode.isRollingUpgrade()`.
> - If that RPC fails, `doCheckpoint()` calls
> `checkpointImage.setMergeError()`.
> - Retry reloads/replays image+edits, but the local `rollingUpgradeInfo`
> state is still active.
> - Replaying `OP_ROLLING_UPGRADE_START` again throws because rolling upgrade
> is already in progress.
> h2. Version pairs tested:
> - Hadoop 2.10.2 -> Hadoop 3.3.6: issue observed.
> - Hadoop 3.3.6 -> Hadoop 3.4.2: also covered by upgrade testing, but this
> specific failure was observed on 2.10.2 -> 3.3.6.
> h2. How To Reproduce
> One way to reproduce is to force a transient NameNode RPC failure during the
> narrow checkpoint window after the SecondaryNameNode has replayed
> `OP_ROLLING_UPGRADE_START` but before `SecondaryNameNode.doMerge()` completes.
> 1. Start a non-HA HDFS cluster on Hadoop 2.10.2 with: one NameNode, one
> SecondaryNameNode and one DataNode
> 2. Prepare a rolling upgrade on the NameNode. This should create a rollback
> image and write a rolling-upgrade START operation into the edit log.
> 3. Upgrade/start the SecondaryNameNode on Hadoop 3.3.6 while the rolling
> upgrade is prepared and not finalized.
> 4. Trigger or wait for a SecondaryNameNode checkpoint. The SecondaryNameNode
> should download the rollback fsimage and edit logs, then replay edits
> containing `OP_ROLLING_UPGRADE_START`.
> 5. After the SecondaryNameNode has replayed `OP_ROLLING_UPGRADE_START` and
> saved the merged local checkpoint image, but before `doMerge()` completes,
> make the NameNode RPC endpoint temporarily unavailable. A practical way to
> induce this is to restart the NameNode at this point. The observed failure
> point is the SecondaryNameNode RPC call to `namenode.isRollingUpgrade()`,
> which can fail with an EOF/connection-closed error if the NameNode is down.
> 6. Bring the NameNode back and let the same SecondaryNameNode process retry
> checkpointing.
> 7. Observe that the retry reloads/replays the checkpoint inputs and
> encounters `OP_ROLLING_UPGRADE_START` again.
> 8. The retry fails because the SecondaryNameNode-local `rollingUpgradeInfo`
> from the failed previous merge attempt is still active.
> h2. Exception
> {code:java}
> org.apache.hadoop.hdfs.protocol.RollingUpgradeException:
> Failed to start rolling upgrade since a rolling upgrade is already in
> progress.{code}
> h2. Buggy code
> The following snippets are from Hadoop 3.3.6.
> `SecondaryNameNode.doCheckpoint()` intentionally retries by reloading image
> and edits after a merge failure:
> {code:java}
> // SecondaryNameNode.java
> // Fetch fsimage and edits. Reload the image if previous merge failed.
> loadImage |= downloadCheckpointFiles(
> fsName, checkpointImage, sig, manifest) |
> checkpointImage.hasMergeError();
> try {
> doMerge(sig, manifest, loadImage, checkpointImage, namesystem);
> } catch (IOException ioe) {
> // A merge error occurred. The in-memory file system state may be
> // inconsistent, so the image and edits need to be reloaded.
> checkpointImage.setMergeError();
> throw ioe;
> }
> checkpointImage.clearMergeError(); {code}
> `SecondaryNameNode.doMerge()` first replays edits and saves a local
> checkpoint image. Only after that does it call back to the active NameNode:
> {code:java}
> // SecondaryNameNode.java
> if (loadImage) {
> File file = dstStorage.findImageFile(NameNodeFile.IMAGE,
> sig.mostRecentCheckpointTxId);
> dstNamesystem.writeLock();
> try {
> dstImage.reloadFromImageFile(file, dstNamesystem);
> } finally {
> dstNamesystem.writeUnlock();
> }
> dstNamesystem.imageLoadComplete();
> }Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);
> dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());
> if (!namenode.isRollingUpgrade()) {
> dstImage.updateStorageVersion();
> } {code}
> During edit replay, `OP_ROLLING_UPGRADE_START` mutates the
> SecondaryNameNode-local `FSNamesystem` by setting rolling-upgrade state:
> {code:java}
> // FSEditLogLoader.java
> case OP_ROLLING_UPGRADE_START: {
> final long startTime = ((RollingUpgradeOp) op).getTime();
> fsNamesys.startRollingUpgradeInternal(startTime);
> fsNamesys.triggerRollbackCheckpoint();
> break;
> }
> // FSNamesystem.java
> void startRollingUpgradeInternal(long startTime) throws IOException {
> checkRollingUpgrade("start rolling upgrade");
> getFSImage().checkUpgrade();
> setRollingUpgradeInfo(false, startTime);
> }void checkRollingUpgrade(String action) throws RollingUpgradeException {
> if (isRollingUpgrade()) {
> throw new RollingUpgradeException("Failed to " + action
> + " since a rolling upgrade is already in progress."
> + " Existing rolling upgrade info:\n" + rollingUpgradeInfo);
> }
> }{code}
> The suspected stale-state bug is in the reload path. `reloadFromImageFile()`
> calls `FSNamesystem.clear()`, but `clear()` does not clear
> `rollingUpgradeInfo`:
> {code:java}
> // FSImage.java
> void reloadFromImageFile(File file, FSNamesystem target) throws IOException {
> target.clear();
> LOG.debug("Reloading namespace from " + file);
> loadFSImage(file, target, null, false);
> } {code}
> {code:java}
> // FSNamesystem.java
> void clear() {
> dir.reset();
> dtSecretManager.reset();
> leaseManager.removeAllLeases();
> snapshotManager.clearSnapshottableDirs();
> cacheManager.clear();
> setImageLoaded(false);
> blockManager.clear();
> ErasureCodingPolicyManager.getInstance().clear();
> } {code}
> Because `rollingUpgradeInfo` is not cleared here, a retry after
> `setMergeError()` can replay `OP_ROLLING_UPGRADE_START` while the
> SecondaryNameNode-local `FSNamesystem` still reports rolling upgrade as
> active. The second replay then fails in `checkRollingUpgrade("start rolling
> upgrade")`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
