yuyanlei created HDFS-16752: ------------------------------- Summary: 2.7.2 Rolling Upgrade 3.3.4 Datanode cannot be Degraded due to version Inconsistency Key: HDFS-16752 URL: https://issues.apache.org/jira/browse/HDFS-16752 Project: Hadoop HDFS Issue Type: Bug Components: datanode, rolling upgrades Affects Versions: 3.3.4, 2.7.2 Reporter: yuyanlei
I am following the official website: HDFS Rolling Upgrade: After the Rolling Upgrade (HDFS dfsAdmin -RollingUpgrade Finalize is not executed), Datanode is degraded. However, Datanode degradation fails. The log of Datanode is displayed as follows: {panel:title=myhost001.datanode.log} INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data01/block/in_use.lock acquired by nodename 37422@myhost001 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data01/block. Reported: -57. Expecting = -56. INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data02/block/in_use.lock acquired by nodename 37422@myhost001 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data02/block. Reported: -57. Expecting = -56. INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data03/block/in_use.lock acquired by nodename 37422@myhost001 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data03/block. Reported: -57. Expecting = -56. INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data01/block/in_use.lock acquired by nodename 37422@myhost001 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data01/block. Reported: -57. Expecting = -56. INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data02/block/in_use.lock acquired by nodename 37422@myhost001 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data02/block. Reported: -57. Expecting = -56. INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data03/block/in_use.lock acquired by nodename 37422@myhost001 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data03/block. Reported: -57. Expecting = -56. FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to myhost002/***:9002. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644) at java.lang.Thread.run(Thread.java:748) FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to myhost002/****:9002. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644) at java.lang.Thread.run(Thread.java:748) WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to myhost002/10.48.20.230:9002 2022-08-30 15:09:04,908 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to myhost002/****:9002 2022-08-30 15:09:05,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned) 2022-08-30 15:09:07,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode 2022-08-30 15:09:07,010 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0 2022-08-30 15:09:07,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at myhost001/**** ************************************************************/ {panel} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org