[
https://issues.apache.org/jira/browse/HDFS-15624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222863#comment-17222863
]
Ayush Saxena commented on HDFS-15624:
-------------------------------------
No, I didn't try. I just saw the code and thought it would be a problem and
folks there at HDFS-15025 confirmed too. I was just bothered it should get
fixed and frankly didn't follow much. (Limited Bandwidth, Saw the code again
today, because [~huangtianhua] asked for)
I just tried, First the upgrade issue, due to the change of ordinal,
{noformat}
ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfsadmin -setSpaceQuota 5
-storageType ARCHIVE /dir
2020-10-29 16:14:03,072 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t ARCHIVE /dir
2020-10-29 16:15:43,660 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
5 5 /dir
ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t DISK /dir
2020-10-29 16:15:52,942 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
none inf /dir
Upgrade Namenode : Magic, Had set quota for ARCHIVE not for DISK
ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t DISK /dir
2020-10-29 16:31:39,677 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
5 5 /dir
ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t ARCHIVE /dir
2020-10-29 16:31:48,698 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
none inf /dir
{noformat}
Now the Rolling Upgrade stuff, I had set quota on Provided before finalising
the upgrade, then Rollback, My NN crashed with :
{noformat}
2020-10-29 16:51:06,261 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage:
Error replaying edit log at offset 25. Expected transaction ID was 9
Recent opcode offsets: 25
java.lang.ArrayIndexOutOfBoundsException: 5
at
org.apache.hadoop.fs.StorageType.parseStorageType(StorageType.java:80)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetQuotaByStorageTypeOp.readFields(FSEditLogOp.java:2363)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp(FSEditLogOp.java:5161)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Reader.readOp(FSEditLogOp.java:5021)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:229)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:276)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:200)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:243)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:914)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:761)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1015)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:988)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1757)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1822)
2020-10-29 16:51:06,263 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
loading fsimage
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying
edit log at offset 25. Expected transaction ID was 9
Recent opcode offsets: 25
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:256)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:914)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:761)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1015)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:988)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1757)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1822)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 5
at
org.apache.hadoop.fs.StorageType.parseStorageType(StorageType.java:80)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetQuotaByStorageTypeOp.readFields(FSEditLogOp.java:2363)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp(FSEditLogOp.java:5161)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Reader.readOp(FSEditLogOp.java:5021)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:229)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:276)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:200)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:243)
... 12 more
2020-10-29 16:51:06,265 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.w.WebAppContext@1c39680d{hdfs,/,null,UNAVAILABLE}{file:/Users/ayushsaxena/code/hadoop-code/osCode/hadoop-3.3.1-SNAPSHOT/share/hadoop/hdfs/webapps/hdfs}
2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.AbstractConnector:
Stopped ServerConnector@6b09fb41{HTTP/1.1,[http/1.1]}{0.0.0.0:9870}
2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.session: node0 Stopped
scavenging
2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@6f10d5b6{static,/static,file:///Users/ayushsaxena/code/hadoop-code/osCode/hadoop-3.3.1-SNAPSHOT/share/hadoop/hdfs/webapps/static/,UNAVAILABLE}
2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@1972e513{logs,/logs,file:///Users/ayushsaxena/code/hadoop-code/osCode/hadoop-3.3.1-SNAPSHOT/logs/,UNAVAILABLE}
2020-10-29 16:51:06,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping NameNode metrics system...
2020-10-29 16:51:06,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
NameNode metrics system stopped.
2020-10-29 16:51:06,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
NameNode metrics system shutdown complete.
2020-10-29 16:51:06,270 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
Failed to start namenode.
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying
edit log at offset 25. Expected transaction ID was 9
Recent opcode offsets: 25
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:256)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:914)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:761)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1015)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:988)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1757)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1822)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 5
at
org.apache.hadoop.fs.StorageType.parseStorageType(StorageType.java:80)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetQuotaByStorageTypeOp.readFields(FSEditLogOp.java:2363)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp(FSEditLogOp.java:5161)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Reader.readOp(FSEditLogOp.java:5021)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:229)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:276)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:200)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:243)
... 12 more
2020-10-29 16:51:06,271 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1: org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error
replaying edit log at offset 25. Expected transaction ID was 9
Recent opcode offsets: 25
2020-10-29 16:51:06,272 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ayushsaxena-MBP16.local
************************************************************/{noformat}
I am not able to capture screenshot, Couldn't get the log fit on one screen(I
tried but.) That much discount I can expect from you. :)
Wanted to check HDFS-15660 too, but now my namenode isn't starting now because
of this, and It will take time to sort it. But I am still convinced it is
different, so no need to hold this, But still I will respect opinions on how to
go ahead. But in case people want to hold, I think, If I have not messed up.
It's good to go back.
> Fix the SetQuotaByStorageTypeOp problem after updating hadoop
> ---------------------------------------------------------------
>
> Key: HDFS-15624
> URL: https://issues.apache.org/jira/browse/HDFS-15624
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Reporter: YaYun Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum
> of StorageType. And, setting the quota by storageType depends on the
> ordinal(), therefore, it may cause the setting of quota to be invalid after
> upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
