[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834724#comment-13834724 ] Hudson commented on HDFS-5568: -- FAILURE: Integrated in Hadoop-Yarn-trunk #405 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/405/]) HDFS-5568. Support includeSnapshots option with Fsck command. Contributed by Vinay (umamahesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1545987) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Fix For: 3.0.0, 2.3.0, 2.2.1 > > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833778#comment-13833778 ] Hudson commented on HDFS-5568: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1621 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1621/]) HDFS-5568. Support includeSnapshots option with Fsck command. Contributed by Vinay (umamahesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1545987) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833770#comment-13833770 ] Hudson commented on HDFS-5568: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1595 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1595/]) HDFS-5568. Support includeSnapshots option with Fsck command. Contributed by Vinay (umamahesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1545987) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833674#comment-13833674 ] Vinay commented on HDFS-5568: - Thanks a lot Uma, Jing and sathish > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833661#comment-13833661 ] Hudson commented on HDFS-5568: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4798 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4798/]) HDFS-5568. Support includeSnapshots option with Fsck command. Contributed by Vinay (umamahesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1545987) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833656#comment-13833656 ] Uma Maheswara Rao G commented on HDFS-5568: --- I have just committed this to trunk, branch-2 and 2.2. Thanks a lot, Vinay for the patch and thanks to Jing and Sathish for review. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833643#comment-13833643 ] Uma Maheswara Rao G commented on HDFS-5568: --- +1 on the latest patch > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833607#comment-13833607
]
Hadoop QA commented on HDFS-5568:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616007/HDFS-5568.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5588//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5588//console
This message is automatically generated.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch,
> HDFS-5568.patch, HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833535#comment-13833535 ] Jing Zhao commented on HDFS-5568: - Since .snapshot is not a directory actually, how about saying something like "-includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it" ? +1 after addressing the comments. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch, > HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833487#comment-13833487
]
Hadoop QA commented on HDFS-5568:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615991/HDFS-5568.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5586//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5586//console
This message is automatically generated.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch,
> HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833361#comment-13833361 ] Jing Zhao commented on HDFS-5568: - Hmm. My original thought is that since we now do not support nested snapshottable directories, for a given directory, we only have at most one snapshottable directory and we should be able to check the corresponding snapshot paths. But after more thinking, looks like the challenge here is that the given directory can be a renamed directory. In that case, it is hard to identify the correct snapshot paths. So I think the current solution is good and we do not need to support that case here. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833329#comment-13833329 ] Vinay commented on HDFS-5568: - You mean to say for a directory /dir1 ( / is snapshottable dir) findout snapshot paths and include them in fsck ( ex: /.snaphost/s1/dir1, /.snaphost/s2/dir1) ..? > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833315#comment-13833315 ] Jing Zhao commented on HDFS-5568: - So maybe we can also support this case here? Basically for a given path we can check if it is in some snapshots (by check its corresponding INodesInPath). And if it is, we can identify the corresponding snapshottable directory and then check all the snapshot paths under the given path. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833299#comment-13833299 ] Vinay commented on HDFS-5568: - bq. One comment is the following code cannot cover the case that the initial given path for fsck is under a snapshottable directory. Yes. -includeSnapshots is to include the ".snapshot" directory and its tree if the path is a snapshottable directory. Not to find out the corresponding snapshot path of directory/files under a snapshottable directory and include them in fsck report. If the path is not exactly a snapshottable directory or a parent of it, then fsck will not include any ".snapshot" paths in it. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833179#comment-13833179
]
Jing Zhao commented on HDFS-5568:
-
The patch looks good overall. One comment is the following code cannot cover
the case that the initial given path for fsck is under a snapshottable
directory.
{code}
if (snapshottableDirs != null && snapshottableDirs.contains(path)) {
String snapshotPath = (path.endsWith(Path.SEPARATOR) ? path : path
+ Path.SEPARATOR)
{code}
A testcase can be:
{code}
@Test
public void testFsckForSnapshotFiles2() throws Exception {
final Configuration conf = new HdfsConfiguration();
MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(1)
.build();
try {
String runFsck = runFsck(conf, 0, true, "/", "-includeSnapshots",
"-files");
assertTrue(runFsck.contains("HEALTHY"));
final String fileName = "/dir1/srcdat";
DistributedFileSystem hdfs = cluster.getFileSystem();
Path file1 = new Path(fileName);
DFSTestUtil.createFile(hdfs, file1, 1024, (short) 1, 1000L);
hdfs.allowSnapshot(new Path("/"));
hdfs.createSnapshot(new Path("/"), "mySnapShot");
runFsck = runFsck(conf, 0, true, "/", "-includeSnapshots", "-files");
assertTrue(runFsck.contains("/.snapshot/mySnapShot/dir1/srcdat"));
runFsck = runFsck(conf, 0, true, "/dir1", "-includeSnapshots", "-files");
assertTrue(runFsck.contains("/.snapshot/mySnapShot/dir1/srcdat"));
} finally {
cluster.shutdown();
}
}
{code}
Thus here we may need to check if the path is under a snapshottable directory.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832549#comment-13832549
]
Hadoop QA commented on HDFS-5568:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615815/HDFS-5568.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5576//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5576//console
This message is automatically generated.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832525#comment-13832525 ] sathish commented on HDFS-5568: --- Thanks vinay for posting the patch +1 patch looks good,latest patch covered UMA comments also > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832517#comment-13832517
]
Hadoop QA commented on HDFS-5568:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615806/HDFS-5568-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5575//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5575//console
This message is automatically generated.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568-1.patch, HDFS-5568.patch, HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832459#comment-13832459 ] Vinay commented on HDFS-5568: - I found that, if we include snapshot paths in fsck report, cluster health check will be done properly. But the small problem is number of files and number of blocks validated will be more because it may contain duplicates. So It will be better to make it on demand, and in that case number of blocks and number of files will not be accurate. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832442#comment-13832442 ] Vinay commented on HDFS-5568: - bq. they are something special files and not as part of main file tree logically. On user interest we can include them like openfiles. Yes, I agree, they are special files. Since blocks inside only snapshots are also counted in deciding the overall cluster health, I think it will be good to consider these always. For openfiles we already have a separate flag. If you strongly feel flagging is enough, then also I am ok with it. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832437#comment-13832437 ] Uma Maheswara Rao G commented on HDFS-5568: --- I think flagging is good option as they are something special files and not as part of main file tree logically. On user interest we can include them like openfiles. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832436#comment-13832436 ] Vinay commented on HDFS-5568: - What if we support inclusion of snapshot paths without any flags..? I mean always... Any thoughts? > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832434#comment-13832434 ] Vinay commented on HDFS-5568: - Thanks Uma for verifying the patch. I will verify the updated patch and repost if required. > Support inclusion of snapshot paths in Namenode fsck > > > Key: HDFS-5568 > URL: https://issues.apache.org/jira/browse/HDFS-5568 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5568-1.patch, HDFS-5568.patch > > > Support Fsck to check the snapshot paths also for inconsistency. > Currently Fsck supports snapshot paths if path given explicitly refers to a > snapshot path. > We have seen safemode problems in our clusters which were due to blocks > missing which were only present inside snapshots. But "hdfs fsck /" shows > HEALTHY. > So supporting snapshot paths also during fsck (may be by default or on > demand) would be helpful in these cases instead of specifying each and every > snapshottable directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832431#comment-13832431
]
Uma Maheswara Rao G commented on HDFS-5568:
---
Hi Vinay, Thanks for the patch. I just reviewed the patch.
Comments:
1) if user don't have any snapshots, then it will end failing with NPE. as
getSnapshottableDirListing returns null.
{noformat}
2013-11-26 13:13:22,522 WARN namenode.NameNode (NamenodeFsck.java:fsck(254))
- Fsck on path '/' FAILED
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:208)
at
org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:69)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
at
org.apache.hadoop.hdfs.server.namenode.FsckServlet.doGet(FsckServlet.java:58)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1220)
{noformat}
2) when we take snapshot for root dir, getFIleInfo will fail saying invalid
path. That is because, you are doing path + Pathsepaator _ snapshotdir ===
//.snapshotdir ...Right now path validation does not allow '//' or ':'
3) Test cases need to catch this kind of small issues.
I am just including the test here with the fixups above mentioned as I wrote
that while just verifying my findings.
Also please take care of above failure if still exists with this patch.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
[
https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832389#comment-13832389
]
Hadoop QA commented on HDFS-5568:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615794/HDFS-5568.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFsck
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5573//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5573//console
This message is automatically generated.
> Support inclusion of snapshot paths in Namenode fsck
>
>
> Key: HDFS-5568
> URL: https://issues.apache.org/jira/browse/HDFS-5568
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a
> snapshot path.
> We have seen safemode problems in our clusters which were due to blocks
> missing which were only present inside snapshots. But "hdfs fsck /" shows
> HEALTHY.
> So supporting snapshot paths also during fsck (may be by default or on
> demand) would be helpful in these cases instead of specifying each and every
> snapshottable directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
