[ https://issues.apache.org/jira/browse/HDFS-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154478#comment-17154478 ]
fengwu edited comment on HDFS-14396 at 7/9/20, 12:10 PM: --------------------------------------------------------- Hi, Found in my test roll downgrade from 3.1.3 to 2.7.2, namenode successful , but datanode failed (2.8+ successfully ). because different datanode layout versions is -56 in hdfs 2.7 So, Is there a way to solve datanode roll downgrade from layout version -57 to -56 ? {code:java} // code placeholder 2020-07-06 14:45:01,313 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56. 2020-07-06 14:45:01,315 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data/hadoop/dfs/in_use.lock acquired by nodename 21258@test-v03 2020-07-06 14:45:01,315 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56. 2020-07-06 14:45:01,315 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to test-v01/10.110.228.21:8020. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802) at java.lang.Thread.run(Thread.java:748) {code} was (Author: fengwu99): Hi, Found in my test roll downgrade from 3.1.3 to 2.7.2, namenode successful , but datanode failed (2.8+ successfully ). because different datanode layout versions is -56 in hdfs 2.7. So, Is there a way to solve datanode roll downgrade from layout version -57 to -56 ? // code placeholder2020-07-06 14:45:01,313 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56. 2020-07-06 14:45:01,315 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data/hadoop/dfs/in_use.lock acquired by nodename 21258@test-v03 2020-07-06 14:45:01,315 WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56. 2020-07-06 14:45:01,315 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to test-v01/10.110.228.21:8020. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802) at java.lang.Thread.run(Thread.java:748) > Failed to load image from FSImageFile when downgrade from 3.x to 2.x > -------------------------------------------------------------------- > > Key: HDFS-14396 > URL: https://issues.apache.org/jira/browse/HDFS-14396 > Project: Hadoop HDFS > Issue Type: Bug > Components: rolling upgrades > Reporter: Fei Hui > Assignee: Fei Hui > Priority: Blocker > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14396.001.patch, HDFS-14396.002.patch > > > After fixing HDFS-13596, try to downgrade from 3.x to 2.x. But namenode can't > start because exception occurs. The message follows > {code:java} > 2019-01-23 17:22:18,730 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: > Failed to load image from > FSImageFile(file=/data1/hadoopdata/hadoop-namenode/current/fsimage_0000000000000025310, > cpktTxId=0000000000 > 000025310) > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:885) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:869) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:742) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:673) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:672) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:839) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1517) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1583) > 2019-01-23 17:22:19,023 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception > loading fsimage > java.io.IOException: Failed to load FSImage file, see error(s) above for more > info. > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:688) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612) > {code} > This issue occurs because 3.x namenode saves image with EC fields during > upgrade > Try to fix it -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org