[
https://issues.apache.org/jira/browse/HDFS-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yuyanlei updated HDFS-16752:
----------------------------
Description:
I am following the official website: HDFS Rolling
Upgrade（[https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html]）:
After the Rolling Upgrade (HDFS dfsAdmin -RollingUpgrade Finalize is not
executed), start degradeing Datanode . However, Datanode degradation fails. The
log of Datanode is displayed as follows:
{panel:title=myhost001.datanode.log}
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data01/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data01/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data02/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data02/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data03/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data03/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data01/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data01/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data02/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data02/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data03/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data03/block. Reported: -57. Expecting = -56.
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed
for Block pool <registering> (Datanode Uuid unassigned) service to
myhost002/***:9002. Exiting.
java.io.IOException: All specified directories are failed to load.
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
at java.lang.Thread.run(Thread.java:748)
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed
for Block pool <registering> (Datanode Uuid unassigned) service to
myhost002/****:9002. Exiting.
java.io.IOException: All specified directories are failed to load.
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
at java.lang.Thread.run(Thread.java:748)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service
for: Block pool <registering> (Datanode Uuid unassigned) service to
myhost002/****:9002
2022-08-30 15:09:04,908 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Ending block pool service for: Block pool <registering> (Datanode Uuid
unassigned) service to myhost002/****:9002
2022-08-30 15:09:05,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Removed Block pool <registering> (Datanode Uuid unassigned)
2022-08-30 15:09:07,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Exiting Datanode
2022-08-30 15:09:07,010 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 0
2022-08-30 15:09:07,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at myhost001/****
************************************************************/
{panel}
was:
I am following the official website: HDFS Rolling
Upgrade（https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html）:
After the Rolling Upgrade (HDFS dfsAdmin -RollingUpgrade Finalize is not
executed), Datanode is degraded. However, Datanode degradation fails. The log
of Datanode is displayed as follows:
{panel:title=myhost001.datanode.log}
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data01/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data01/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data02/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data02/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data03/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data03/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data01/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data01/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data02/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data02/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
/data03/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data03/block. Reported: -57. Expecting = -56.
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed
for Block pool <registering> (Datanode Uuid unassigned) service to
myhost002/***:9002. Exiting.
java.io.IOException: All specified directories are failed to load.
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
at java.lang.Thread.run(Thread.java:748)
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed
for Block pool <registering> (Datanode Uuid unassigned) service to
myhost002/****:9002. Exiting.
java.io.IOException: All specified directories are failed to load.
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
at java.lang.Thread.run(Thread.java:748)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service
for: Block pool <registering> (Datanode Uuid unassigned) service to
myhost002/****:9002
2022-08-30 15:09:04,908 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Ending block pool service for: Block pool <registering> (Datanode Uuid
unassigned) service to myhost002/****:9002
2022-08-30 15:09:05,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Removed Block pool <registering> (Datanode Uuid unassigned)
2022-08-30 15:09:07,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Exiting Datanode
2022-08-30 15:09:07,010 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 0
2022-08-30 15:09:07,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at myhost001/****
************************************************************/
{panel}
> 2.7.2 Rolling Upgrade 3.3.4 Datanode cannot be Degraded due to version
> Inconsistency
> ------------------------------------------------------------------------------------
>
> Key: HDFS-16752
> URL: https://issues.apache.org/jira/browse/HDFS-16752
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, rolling upgrades
> Affects Versions: 2.7.2, 3.3.4
> Reporter: yuyanlei
> Priority: Blocker
>
> I am following the official website: HDFS Rolling
> Upgrade（[https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html]）:
> After the Rolling Upgrade (HDFS dfsAdmin -RollingUpgrade Finalize is not
> executed), start degradeing Datanode . However, Datanode degradation fails.
> The log of Datanode is displayed as follows:
> {panel:title=myhost001.datanode.log}
> INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
> /data01/block/in_use.lock acquired by nodename 37422@myhost001
> WARN org.apache.hadoop.hdfs.server.common.Storage:
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data01/block. Reported: -57. Expecting = -56.
> INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
> /data02/block/in_use.lock acquired by nodename 37422@myhost001
> WARN org.apache.hadoop.hdfs.server.common.Storage:
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data02/block. Reported: -57. Expecting = -56.
> INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
> /data03/block/in_use.lock acquired by nodename 37422@myhost001
> WARN org.apache.hadoop.hdfs.server.common.Storage:
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data03/block. Reported: -57. Expecting = -56.
> INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
> /data01/block/in_use.lock acquired by nodename 37422@myhost001
> WARN org.apache.hadoop.hdfs.server.common.Storage:
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data01/block. Reported: -57. Expecting = -56.
> INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
> /data02/block/in_use.lock acquired by nodename 37422@myhost001
> WARN org.apache.hadoop.hdfs.server.common.Storage:
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data02/block. Reported: -57. Expecting = -56.
> INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on
> /data03/block/in_use.lock acquired by nodename 37422@myhost001
> WARN org.apache.hadoop.hdfs.server.common.Storage:
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data03/block. Reported: -57. Expecting = -56.
> FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed
> for Block pool <registering> (Datanode Uuid unassigned) service to
> myhost002/***:9002. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
> at java.lang.Thread.run(Thread.java:748)
> FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed
> for Block pool <registering> (Datanode Uuid unassigned) service to
> myhost002/****:9002. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
> at java.lang.Thread.run(Thread.java:748)
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool
> service for: Block pool <registering> (Datanode Uuid unassigned) service to
> myhost002/****:9002
> 2022-08-30 15:09:04,908 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool <registering> (Datanode Uuid
> unassigned) service to myhost002/****:9002
> 2022-08-30 15:09:05,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Removed Block pool <registering> (Datanode Uuid unassigned)
> 2022-08-30 15:09:07,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exiting Datanode
> 2022-08-30 15:09:07,010 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 0
> 2022-08-30 15:09:07,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at myhost001/****
> ************************************************************/
> {panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
