[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15099130#comment-15099130
]
Tsz Wo Nicholas Sze commented on HDFS-9569:
---
Thanks for the quick fix!
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098968#comment-15098968
]
Hudson commented on HDFS-9569:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9114 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9114/])
HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's (yzhang: rev
817cc1f02a60ef4e372171415058fdc76c0d2e39)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098903#comment-15098903
]
Yongjun Zhang commented on HDFS-9569:
-
Committed HDFS-9648 as the fix.
Thanks [~szetszwo] and [~cnauroth].
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098377#comment-15098377
]
Chris Nauroth commented on HDFS-9569:
-
I missed it too. This patch lingered for a few weeks, and I mistakenly assumed
Jenkins was done by now. [~szetszwo], thanks for pointing it out.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098281#comment-15098281
]
Yongjun Zhang commented on HDFS-9569:
-
Hi [~szetszwo],
Thanks for pointing out, the jenkins test was indeed overlooked, very sorry
about the inconvenience, I will look into and try to resolve it today.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097956#comment-15097956
]
Tsz Wo Nicholas Sze commented on HDFS-9569:
---
It seems that this caused TestStartup failing for the past 12 builds; see
https://builds.apache.org/job/PreCommit-HDFS-Build/14117/testReport/org.apache.hadoop.hdfs.server.namenode/TestStartup/testImageChecksum/
> +1 for patch v005, pending Jenkins. ...
BTW, why there was the Jenkins report before committing?
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094391#comment-15094391
]
Hudson commented on HDFS-9569:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9095 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9095/])
HDFS-9569. Log the name of the fsimage being loaded for better (yzhang: rev
25051c3bd08efc12333a6acb51782cc7800403a4)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/IllegalReservedPathException.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092394#comment-15092394
]
Yongjun Zhang commented on HDFS-9569:
-
HI [~cnauroth], thanks for the reminder, I have been on vacation then spending
time on a critical issue after getting back, will get this committed myself
soon.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092374#comment-15092374
]
Chris Nauroth commented on HDFS-9569:
-
Hi [~yzhangal]. I had +1'd this. Were you planning to commit it, or would you
like me to do it?
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067431#comment-15067431
]
Yongjun Zhang commented on HDFS-9569:
-
Thanks [~cnauroth]!
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066838#comment-15066838
]
Chris Nauroth commented on HDFS-9569:
-
+1 for patch v005, pending Jenkins. Thank you!
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch, HDFS-9569.005.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066766#comment-15066766
]
Chris Nauroth commented on HDFS-9569:
-
OK, let's go with the rev 4 approach. However, I don't understand why this was
added:
{code}
List imageFiles = inspector.getLatestImages();
-
+if (imageFiles.size() == 0) {
+ throw new IOException("Failed to find any FSImage files.");
+}
+
{code}
I think this is unreachable code. The implementations of {{getLatestImages}}
are written to either return a non-empty list or throw an exception, so it
appears to be impossible for the list to be empty by the time execution reaches
here.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch, HDFS-9569.004.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065965#comment-15065965
]
Yongjun Zhang commented on HDFS-9569:
-
HI Chris,
I think your main concern at this point is not to log the trace stack twice, my
main concern has been to ensure that the error issued shows both the trace
stack and the file name. I think separating the error logging and exception
throwing may not give clear cause-effect relationship.
There are different reasons for failure to load fsimage. To be consistent, I
think we can wrap the reason as a cause in an IOException. and let the code
that catches the exception to find out the cause and deal with the cause
accordingly.
I'm uploading rev 4 that wraps cause in IOException and ensure the trace stack
is printed only once. I wonder whether it will work for you.
Thanks.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065941#comment-15065941
]
Hadoop QA commented on HDFS-9569:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 42s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK
v1.8.0_66 generated 2 new issues (was 32, now 32). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 18m 32s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK
v1.7.0_91 generated 2 new issues (was 34, now 34). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 182m 53s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 177m 33s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m 6s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 431m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.ser
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065902#comment-15065902
]
Chris Nauroth commented on HDFS-9569:
-
How about this?
{code}
} catch (IllegalReservedPathException e) {
LOG.error("Failed to load image from " + imageFile);
throw e;
{code}
Notice this doesn't pass {{e}} to {{LOG.error}}. That way, we'll get a log
message with the file name, but it won't double-print the stack trace.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065900#comment-15065900
]
Yongjun Zhang commented on HDFS-9569:
-
Hi [~cnauroth],
Thanks again for your review and comments.
About your suggested change (option 1)
{code}
} catch (IllegalReservedPathException e) {
throw e;
{code}
The intention of my last change (option 2) was to include the fsimage file name
in the exception, the above suggested change won't do it. That's why I did one
error logging before rethrowing, so the logging has the imageFile name:
{code}
} catch (IllegalReservedPathException e) {
LOG.error("Failed to load image from " + imageFile, e);
throw e;
{code}
Another way to do it is (option 3):
{code}
} catch (IllegalReservedPathException e) {
throw new IllegalReservedPathException("Failed to load image from "
+ imageFile + " (" + e.getMessage() +")", e);
}
...
{code}
If we want to have the imageFile name in the exception message, which option
would you prefer?
Thanks.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065871#comment-15065871
]
Chris Nauroth commented on HDFS-9569:
-
[~yzhangal], thank you for the update. This looks better. It still ends up
logging the error and full stack trace twice: once within the {{loadFSImage}}
loop, and once again when the exception gets rethrown, escapes to top level,
and exits the process. Sorry to nit-pick, but I think eliminating the
redundant logging will better help operators identify what they need to do
next. I suggest catching and rethrowing {{IllegalReservedPathException}}
without logging from within the loop:
{code}
...
} catch (IllegalReservedPathException e) {
throw e;
} catch (Exception e) {
LOG.error("Failed to load image from " + imageFile, e);
...
{code}
Also, the test can be simplified to catch {{IllegalReservedPathException}}
directly:
{code}
...
} catch (IllegalReservedPathException e) {
GenericTestUtils.assertExceptionContains(
"reserved path component in this version",
e);
} finally {
...
{code}
I'll be +1 after those changes and another Jenkins run. I'll also give
[~kihwal] a chance to comment again before I commit anything, since he did the
earlier review.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065835#comment-15065835
]
Yongjun Zhang commented on HDFS-9569:
-
Hi [~cnauroth],
Thanks for reviewing and good comments!
Theoretically different fsimages could be corrupted different ways, so trying
all and print error for individual fsimage might be ok. Printing the same error
for each fsimage would not be a big problem IMO. But I agree that the reserved
path exception tend to be the same for all fsimages to be tried, and we can
keep the old control flow of throwing exception at the first fsimage tried.
I uploaded rev 3 with your suggestion of introducing
IllegalReservedPathException, which I think is a good idea.
Would you please take a look?
Thanks.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch,
> HDFS-9569.003.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065499#comment-15065499
]
Chris Nauroth commented on HDFS-9569:
-
Patch v002 causes a subtle change in the control flow that will cause a lot of
extraneous logging. For example, assume there are 3 fsimage directories, and
they're all working correctly. With the current code, the first attempt to
load finds reserved paths in the first directory and immediately exits by
throwing an unchecked {{IllegalArgumentException}}. After applying the patch,
this is converted to a checked {{IOException}}, which is not sufficient to
terminate the loop. It will try all 3 fsimage directories. Each iteration
will log an error, with full stack trace. Then, it will get logged a 4th time
after the loop.
This extra logging is not useful for end users. It's a better user experience
to exit as soon as reserved names are encountered, because we already know at
that point that user action is required.
Maybe at this point we need a more specific exception type, like
{{IllegalReservedPathException}}. That one could be allowed to propagate out
of the loop and terminate early.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064715#comment-15064715
]
Yongjun Zhang commented on HDFS-9569:
-
The test failure appears to be irrelevant to this change. Thanks.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063709#comment-15063709
]
Hadoop QA commented on HDFS-9569:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 18s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66
generated 2 new issues (was 32, now 32). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 3s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91
generated 2 new issues (was 34, now 34). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 42s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 42s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.qjournal.client.TestQuorum
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063572#comment-15063572
]
Yongjun Zhang commented on HDFS-9569:
-
Thanks Chris for catching the test issue and reverting.
Hi [~kihwal] and [~cnauroth],
I just uploaded rev 2 to address the failure. Would you please help taking a
look at the patch?
Thanks.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch, HDFS-9569.002.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062942#comment-15062942
]
Hudson commented on HDFS-9569:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8989 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8989/])
Revert "HDFS-9569. Log the name of the fsimage being loaded for better
(cnauroth: rev db37f02dc704ad9eec8c56e3e466a5f37d138d74)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062920#comment-15062920
]
Chris Nauroth commented on HDFS-9569:
-
Pre-commit did flag the test failure in the last run too.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062358#comment-15062358
]
Yongjun Zhang commented on HDFS-9569:
-
Thanks a lot [~kihwal] for the review and commit!
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062343#comment-15062343
]
Hudson commented on HDFS-9569:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8984 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8984/])
HDFS-9569. Log the name of the fsimage being loaded for better (kihwal: rev
eb6939cea0343840c62b930d4adb377f5eaf879f)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062272#comment-15062272
]
Kihwal Lee commented on HDFS-9569:
--
+1 looks good to me.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HDFS-9569.001.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061588#comment-15061588
]
Hadoop QA commented on HDFS-9569:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 28s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 131m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.TestEncryptedTransfer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12778168/HDFS-9569.001.patch |
| JIRA Issue | HDFS-9569
