cxzl25 commented on code in PR #5460:
URL: https://github.com/apache/hadoop/pull/5460#discussion_r3051817271
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##########
@@ -791,6 +792,9 @@ private void offerService() throws Exception {
shouldServiceRun = false;
return;
}
+ if (InvalidBlockReportLeaseException.class.getName().equals(reClass)) {
+ fullBlockReportLeaseId = 0;
Review Comment:
## Description
HDFS-16942 introduced `InvalidBlockReportLeaseException`, which the NameNode
now throws back to the DataNode via RPC when a block report is rejected due to
an invalid lease. On a DataNode that also includes HDFS-16942, the exception is
caught and `fullBlockReportLeaseId` is reset to 0, allowing the DN to request a
new lease on the next heartbeat and retry.
However, during a rolling upgrade where the NameNode has been upgraded (with
HDFS-16942) but DataNodes are still running an older version (without
HDFS-16942), the old DataNode code does not have the
`InvalidBlockReportLeaseException` handling branch in
`BPServiceActor.offerService()`. This causes the DN to enter an infinite
failure loop where it can never successfully send a full block report.
## Root Cause
In `BPServiceActor.offerService()`, the logic works as follows:
1. The DN requests a block report lease during heartbeat only when
`fullBlockReportLeaseId == 0`:
```java
boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
scheduler.isBlockReportDue(startTime);
```
2. After sending the block report, `fullBlockReportLeaseId` is reset to 0:
```java
if ((fullBlockReportLeaseId != 0) || forceFullBr) {
cmds = blockReport(fullBlockReportLeaseId);
fullBlockReportLeaseId = 0; // not reached if blockReport() throws
}
```
3. When the upgraded NN throws `InvalidBlockReportLeaseException`,
`blockReport()` propagates the exception. The `fullBlockReportLeaseId = 0` line
after the call is never executed.
4. The exception is caught by the generic `RemoteException` catch block. The
old DN code does not recognize `InvalidBlockReportLeaseException`, so
`fullBlockReportLeaseId` remains set to the stale invalid value.
5. On the next heartbeat iteration, because `fullBlockReportLeaseId != 0`,
`requestBlockReportLease` is `false` — the DN does not request a new lease. It
then attempts `blockReport()` again with the same stale lease, which the NN
rejects again. This repeats indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
