[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882426#action_12882426 ] Konstantin Shvachko commented on HDFS-729: -- Could you please update components and version fields for this jira. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch, HDFS-729.6.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12853851#action_12853851 ] Hudson commented on HDFS-729: - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/]) fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch, HDFS-729.6.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12853636#action_12853636 ] Hudson commented on HDFS-729: - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/]) fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch, HDFS-729.6.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842362#action_12842362 ] dhruba borthakur commented on HDFS-729: --- Thanks Rodrigo, I will commit this patch. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch, HDFS-729.6.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842188#action_12842188 ] Hadoop QA commented on HDFS-729: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438071/HDFS-729.6.patch against trunk revision 919680. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/128/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/128/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/128/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/128/console This message is automatically generated. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch, HDFS-729.6.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842202#action_12842202 ] Rodrigo Schmidt commented on HDFS-729: -- The failure in the contrib test was the same error attempting to download Tomcat I've seen in several other HDFS Jiras. It's completely unrelated to this patch: Failed to download [http://apache.osuosl.org/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.zip] fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch, HDFS-729.6.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840804#action_12840804 ] dhruba borthakur commented on HDFS-729: --- +1 Code looks good. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch, HDFS-729.5.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[
https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840378#action_12840378
]
dhruba borthakur commented on HDFS-729:
---
Code looks good. The only question I have is that BlockManager.getCorruptInodes
does the following:
{code}
LinkedHashSetINode set =
new LinkedHashSetINode(this.maxCorruptFilesReturned*2);
{code}
Can you pl explain why the multiplication by 2 is needed?
fsck option to list only corrupted files
Key: HDFS-729
URL: https://issues.apache.org/jira/browse/HDFS-729
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: Rodrigo Schmidt
Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt,
HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch
An option to fsck to list only corrupted files will be very helpful for
frequent monitoring.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840406#action_12840406 ] Rodrigo Schmidt commented on HDFS-729: -- Dhruba convinced me that starting with a big hash table is not a good idea. I'm uploading a new patch. Turns out 3 is not my lucky number any more. :-) fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840439#action_12840439 ] Hadoop QA commented on HDFS-729: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437662/HDFS-729.4.patch against trunk revision 916902. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/122/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/122/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/122/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/122/console This message is automatically generated. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840509#action_12840509 ] Hadoop QA commented on HDFS-729: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437665/HDFS-729.4.patch against trunk revision 916902. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/261/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/261/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/261/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/261/console This message is automatically generated. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch, HDFS-729.4.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12839974#action_12839974 ] Hadoop QA commented on HDFS-729: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437530/HDFS-729.3.patch against trunk revision 916902. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/256/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/256/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/256/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/256/console This message is automatically generated. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12839975#action_12839975 ] Rodrigo Schmidt commented on HDFS-729: -- The errors are the same as before, and the same other patches seem to be going through. Dhruba, could you please double check that everything is fine with this patch? fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch, HDFS-729.3.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12836563#action_12836563 ] Hadoop QA commented on HDFS-729: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436392/HDFS-729.2.patch against trunk revision 911744. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/240/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/240/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/240/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/240/console This message is automatically generated. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12836570#action_12836570 ] Rodrigo Schmidt commented on HDFS-729: -- I went through the logs and there was no failed unit test. The build failed with the following error, which has nothing to do with this patch: BUILD FAILED [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:569: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:292: org.codehaus.cargo.container.ContainerException: Failed to download [http://apache.osuosl.org/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.zip] fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12837071#action_12837071 ] dhruba borthakur commented on HDFS-729: --- Code looks good. A few comments: 1. TestDFSClientRetries.java, you do not need to import java.util.Map 2. ClientProtocol javadocs still say @return Map from FileStatus to LocatedBlocks object with locations of * corrupted blocks.. 3. BlockManager.getCorruptInodes() returns a map of Inodes to BlockList. Instead, won't it be sufficient to return just a list of INodes? fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12837072#action_12837072 ] dhruba borthakur commented on HDFS-729: --- Oh, and maybe the unit test can check that no more than 500 files are being returned by this call (even if there are plenty more corrupted files). For this unit test, you might want to initiaize MAX_CORRUPT_FILES_RETURNED from a config parameter (default 500). fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12837103#action_12837103 ] Rodrigo Schmidt commented on HDFS-729: -- Thanks for the comments, Dhruba. 1 and 2 are ghosts from previous code. Good catches! 3 is partly because it was the original code and partly because I thought that returning the blocks could be useful in future extensions. But I think you are right. Let's keep it simple and extend it in the future if needed. I'll make the changes you propose, create the new unit test and resubmit the patch. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch, HDFS-729.2.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12834107#action_12834107 ] dhruba borthakur commented on HDFS-729: --- One option would be to add ClientProtocol.getBadFiles public FileStatus[] getBadFiles() Also, please note that some changes to FileStatus are ocuring via HDFS-946 fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt, HDFS-729.1.patch An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[
https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12833658#action_12833658
]
Rodrigo Schmidt commented on HDFS-729:
--
I was looking at the current patch and I think I found a bug on it.
On UnderReplicatedBlocks.java, the following method was added:
+ /**
+ * Return an iterator of all blocks that have no valid replicas.
+ * These are either blocks with no replicas or all existing replicas
+ * are corrupted. Such blocks are at level 2.
+ */
+ public synchronized IteratorBlock iteratorBadBlocks() {
+return priorityQueues.get(2).iterator();
+ }
It assumes all blocks on queue 2 have 0 replicas. However according to get
getPriority() on the same source file, we can see that level 2 is also used for
blocks whose number of replicas times 3 is bigger than expected replicas:
} else if(curReplicas*3expectedReplicas) {
return 1;
} else {
return 2;
}
So, if a block has 2 replicas, but it is expected to have 3, it will also be
kept in the queue with priority 2. I'm fixing that by adding an extra check on
the real number of replicas a block has before adding it to the list returned
by BlockManager.getCorruptFiles() (previously BockManager.getBadFiles()).
Please let me know what you guys think about it.
fsck option to list only corrupted files
Key: HDFS-729
URL: https://issues.apache.org/jira/browse/HDFS-729
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: Rodrigo Schmidt
Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt
An option to fsck to list only corrupted files will be very helpful for
frequent monitoring.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12833668#action_12833668 ] Rodrigo Schmidt commented on HDFS-729: -- Something else I noticed on this API is that we are returning all block locations of the corrupt files, and not only the locations of those blocks that are corrupt. And besides block locations, we don't give any other information to the client about the file. I suggest that we return a map from FileStatus to corrupt block locations, listing all the files that are corrupt together with its corrupt blocks (leaving out the blocks that have at least one replica). fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Rodrigo Schmidt Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806646#action_12806646 ] dhruba borthakur commented on HDFS-729: --- It appears that I have consensus from Raghu, Konstantin and Rodrigo. anybody else wants to weigh on in this one? fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806433#action_12806433 ] Allen Wittenauer commented on HDFS-729: --- Hmm. So basically the burden on removing duplicates is passed on to the caller? How does the caller know that it has them all? fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806441#action_12806441 ] dhruba borthakur commented on HDFS-729: --- o basically the burden on removing duplicates is passed on to the c This is correct. fsck will actually invoke this API only once and will print out the list of first 500 corrupted files. It will not invoke this API multiple times. This list actually helps the adminstrator because he/she can get a partial list of corrupted files very quickly. My theory is that this partial list is better than waiting for a total fsck to finish which can take hours. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806459#action_12806459 ] Raghu Angadi commented on HDFS-729: --- As I understand, list of files returned in one call will not have duplicates. 500 is a lot.. note that these are files with 'hard-corruption', ie, HDFS could not repair them. Once a cluster has so many corrupt files, I would think there would be a lot more urgent things to worry about than finding rest of the corrupt files. In practice, most likely reason for such a scenario would be a large number of datanodes go missing. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806596#action_12806596 ] Rodrigo Schmidt commented on HDFS-729: -- Returning a quick list of 500 corrupted files is a great idea. Most of the time, this will cover all corrupted files in the system. +1 to this idea. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12805520#action_12805520 ] dhruba borthakur commented on HDFS-729: --- Hi Hairong and Raghu, I had a discussion with Konstantin and he suggested simplifying the API by making this API always return the first 500 corrupted files. Even if there are more than 500 corrupted files in the system, successive calls to this API will always return only 500 files. There is no relationship between the files returned by one invocation of this API with the ones returned by the next invocation of this API. The above proposal address Hairongs (1) above. It also is in line with Raghu's suggestion t is probably good enough to always return up to (100 or 500 files. Regarding Hairong's (2) above, this should be addressed and will fixed in the next patch (if not already there). Does this sound like a reasonable approach to follow? fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12792683#action_12792683 ] Hairong Kuang commented on HDFS-729: I have two concerns about the approach: 1. If the neededReplicationQueue changes while issuing getBadFiles calls, consecutive calls may not be able to return all bad files; 2. Because neededReplicationQueue stores blocks that may belong to the same file, so two consecutive badFiles calls may contain duplicate files. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12789653#action_12789653 ] Raghu Angadi commented on HDFS-729: --- Hi Dhruba, Patch looks good. What would be an alternative for badFile? A corruptFile, might not imply one of the blocks no good replica. But in general, corrupt file implies something that could not be recovered by filesystem. Between these two, my vote is for corruptFile. The current API is fine : one minor nit is that even when it returns less than numExpectedFiles it does not imply there aren't any more. In practice, it is probably good enough not to have this limit and always return up to (100 or 500 files).. This would simplify the interface. Your choice. I will double check if all the blocks in 'pri 2' bucket includes all the and only the blocks with no good replica left. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: badFiles.txt, badFiles2.txt, corruptFiles.txt An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[
https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12771715#action_12771715
]
dhruba borthakur commented on HDFS-729:
---
I am planning to follow Raghu's advice and add the following API to the
namenode:
{quote}
/**
* Returns a list of files that are corrupted.
* p
* Returns a list of files that have at least one block that has no valid
replicas.
* The returned list has numExpectedFiles files in it. If the number of files
* returned is smaller than numExpectedFiles, then it implies that no more
* corrupted files are available in the system. The startingNumber is the
* startingNumber-th corrupted file in the system.
*
* @param numExpectedFiles the maximum number of files to be returned
* @param startingNumber list files starting from startingNumberth to
* (startingNumber + numExpectedFiles)th in the
* list of corrupted files
* @throws AccessControlException if the superuser privilege is violated.
* @throws IOException if unable to retrieve information of a corrupt file
*/
public LocatedBlocks[] getCorruptFiles(int numExpectedFiles, int
startingNumber) throws IOException;
{quote}
This will be used by fsck (or any other application) to quickly detect
corrupted files.
fsck option to list only corrupted files
Key: HDFS-729
URL: https://issues.apache.org/jira/browse/HDFS-729
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
An option to fsck to list only corrupted files will be very helpful for
frequent monitoring.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769124#action_12769124 ] dhruba borthakur commented on HDFS-729: --- There are two existing options to handle corrupted files, one option moves the file to lost+found and the other option deletes the corrupted file. I would like to add another option listCorruptedFiles that will list the corrupted files if any. An alternative is to running a fsck -files and then filter the output on the client to display only corrupted files; but on a cluster with 20 million files, the total amount of data (one for every line of output) to be transferred to the client is huge and introduces lots of latency. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769356#action_12769356 ] Raghu Angadi commented on HDFS-729: --- Is this a regular fsck with less output? That might still be prohibitively long and expensive for regular poll. Name node already has a list of all the corrupt/missing/underreplicated blocks. It might be better to list those blocks and files they belong to (a user friendly -metaSave). Would be nice to have jsp as well. fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769403#action_12769403 ] Allen Wittenauer commented on HDFS-729: --- Using -metaSave as the tell me what is broke option is not obvious. (I'm not even user what -metaSave is supposed to mean!) I'd rather have '-listCorruptedFiles' and '-listCorruptedBlocks' or a -listCorrupted that takes an option of files or blocks with the default being files. [Random rant: if I had a time machine, I'd like to go back and take the keyboard away from the person who decided camelcase for shell options was OK. Even if this means it is Doug. :) ] fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-729) fsck option to list only corrupted files
[ https://issues.apache.org/jira/browse/HDFS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769480#action_12769480 ] Raghu Angadi commented on HDFS-729: --- Using -metaSave as the tell me what is broke option is not obvious. (I'm not even user what -metaSave is supposed to mean!) I'd rather have '-listCorruptedFiles' and '-listCorruptedBlocks' or a -listCorrupted that takes an option of files or blocks with the default being files. yes. this is what I meant. metaSave was mentioned only as a quick reference (for a few that used it in the past). fsck option to list only corrupted files Key: HDFS-729 URL: https://issues.apache.org/jira/browse/HDFS-729 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur An option to fsck to list only corrupted files will be very helpful for frequent monitoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
