[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055154#comment-13055154 ] Jim Plush commented on HDFS-979: There seem to be several other places where a generic message like this is displayed... ➺ Ack --java "All specified directories are not accessible" hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java 134: "All specified directories are not accessible or do not exist."); hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java 188: "All specified directories are not accessible or do not exist."); hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java 201: "All specified directories are not accessible or do not exist."); > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Priority: Minor > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055240#comment-13055240 ] Hadoop QA commented on HDFS-979: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483878/HDFS-979-take1.txt against trunk revision 1139715. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/844//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/844//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/844//console This message is automatically generated. > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055276#comment-13055276 ] Hadoop QA commented on HDFS-979: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483882/HDFS-979-take2.txt against trunk revision 1139715. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/845//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/845//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/845//console This message is automatically generated. > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055514#comment-13055514 ] Steve Loughran commented on HDFS-979: - seems reasonable and is close to committing. Some observations # the code would need some minor reformatting to meet the normal Hadoop style rules (primarily spacing in if() clauses) # the test shouldn't call Throwable.getMessage(), and then compare it, as some exceptions return null there. Safer to call toString() and assert on that, and include text in the assertions for better failure diagnosis # One bigger change would be should the initial condition be broken up into two -one for dataDirs and one for editDirs? In favour of this -it's easier to see what is at fault, but against it users may end up fixing one bug and then go straight into the next one. It's probably simpler to leave the test as is, with Jim's extra diagnostics. > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055699#comment-13055699 ] Jim Plush commented on HDFS-979: sounds good Steve, I'll refactor items 1,2 and 3. I do think 3 would probably clean some of the logic up so I'll take a crack at it. Also, for #1(code formatting) are you referring to the Sun standards for if spacing or is there another doc I should take a look at? > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055880#comment-13055880 ] Hadoop QA commented on HDFS-979: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12484010/HDFS-979-take3.txt against trunk revision 1140030. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/850//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/850//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/850//console This message is automatically generated. > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt, > HDFS-979-take3.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056640#comment-13056640 ] Matt Foley commented on HDFS-979: - Please hold off on committing for a few hours while I review existing logging that may intersect with this. Thanks. > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt, > HDFS-979-take3.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058059#comment-13058059 ] Matt Foley commented on HDFS-979: - Sorry for the delay, I was at the Hadoop Summit. Thanks for waiting. In general, more specific logging is good. I wanted to check that this enhancement does not overlap with the reporting of bad dataDirs and editsDirs from e.g. the callers of NNStorage#reportErrorsOnDirectories(). I now believe there is no collision, so go for it! Here are some additional comments on this patch: FSImage: recoverTransitionRead() * Please use dataDirs.isEmpty() rather than dataDirs.size() == 0; same for editsDirs. But it might be best to also check that they are non-null before checking for emptiness. * The enveloping check for "if (startOpt != StartupOption.IMPORT) {...}" concerns me, since the old code did not have this check. It's confusing because this method is called from both loadFSImage() and doImportCheckpoint(), and in either case the dataDirs and editsDirs should be non-null and non-empty. Here's the question: When IMPORT is set, and loadFSImage calls recoverTransitionRead for the FIRST time (before the recursive call from doImportCheckpoint), isn't it important that dataDirs and editsDirs be checked for non-emptiness? I'm not sure, because FSNamesystem#getStorageDirs does have a comment that says "//When importing image from a checkpoint, the name-node can start with empty set of storage directories." But it also logs a warning that this is a really bad idea, since no edits log could be recorded. Can you explain why the old code (without this check) was considered okay? * Looking further at FSNamesystem#getStorageDirs, when IMPORT is not set, it seems you'll never get an empty directory list. It does {code} if (dirNames.isEmpty()) dirNames.add("file:///tmp/hadoop/dfs/name"); {code} getEmptyRecoveryIOExceptionMsg() * Comments have mis-spellings: editsDirs (not editDirs) and necessarily. * Suggest that the dirType strings should be "fsimage" and "edits" rather than "data" and "edit", in call from recoverTransitionRead(). Better yet, since this really should be an enum, why not use the NameNodeDirType values IMAGE and EDITS? * There's no need to pass configKey and directoriesToCheck args. Since this is a single-use method, keep the signature clean by fetching them within the getEmptyRecoveryIOExceptionMsg() method, based on the dirType arg. * In-line the value of "confValue". * The test for dirsCollection.size() == 0 isn't necessary. If dirsCollection isn't empty, this method won't be called, right? But if for some reason it gets called when dirsCollection isn't empty, the log string would be suddenly terminated at "Specifically: ". It's better to just proceed with constructing the log string. * In fact, why bother sending dirsCollection as an arg at all? It will always be empty. The dirType has all the info needed. * If dir.exists() print "EXISTS BUT IS APPARENTLY INACCESSIBLE", rather than just "EXISTS". * Instead of limiting the method to URIs with scheme == "file", can you use the FileContext stuff to test .exists() on the directories regardless of scheme? > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt, > HDFS-979-take3.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178118#comment-13178118 ] Harsh J commented on HDFS-979: -- Hey Jim, did you get a chance to apply Matt's review comments to the patch? :) > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Fix For: 0.24.0 > > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt, > HDFS-979-take3.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[ https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354165#comment-14354165 ] Hadoop QA commented on HDFS-979: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12484010/HDFS-979-take3.txt against trunk revision 82db334. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9807//console This message is automatically generated. > FSImage should specify which dirs are missing when refusing to come up > -- > > Key: HDFS-979 > URL: https://issues.apache.org/jira/browse/HDFS-979 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Jim Plush >Priority: Minor > Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt, > HDFS-979-take3.txt > > > When {{FSImage}} can't come up as either it has no data or edit dirs, it > tells me this > {code} > java.io.IOException: All specified directories are not accessible or do not > exist. > {code} > What it doesn't do is say which of the two attributes are missing. This would > be beneficial to anyone trying to track down the problem. Also, I don't think > the message is correct. It's bailing out because dataDirs.size() == 0 || > editsDirs.size() == 0 , because a list is empty -not because the dirs aren't > there, as there hasn't been any validation yet. > More useful would be > # Explicit mention of which attributes are null > # Declare that this is because they are not in the config -- This message was sent by Atlassian JIRA (v6.3.4#6332)