[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[ https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924878#comment-13924878 ] Hudson commented on HDFS-5986: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1720 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1720/]) HDFS-5986. Capture the number of blocks pending deletion on namenode webUI. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1575384) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java > Capture the number of blocks pending deletion on namenode webUI > --- > > Key: HDFS-5986 > URL: https://issues.apache.org/jira/browse/HDFS-5986 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Suresh Srinivas >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5986-screenshot.png, HDFS-5986.1.patch > > > When a directory that has large number of directories and files are deleted, > the namespace deletes the corresponding inodes immediately. However it is > hard to to know when the invalidated blocks are actually deleted on the > datanodes, which could take a while. > I propose adding on namenode webUI, along with under replicated blocks, the > number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[ https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924851#comment-13924851 ] Hudson commented on HDFS-5986: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/]) HDFS-5986. Capture the number of blocks pending deletion on namenode webUI. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1575384) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java > Capture the number of blocks pending deletion on namenode webUI > --- > > Key: HDFS-5986 > URL: https://issues.apache.org/jira/browse/HDFS-5986 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Suresh Srinivas >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5986-screenshot.png, HDFS-5986.1.patch > > > When a directory that has large number of directories and files are deleted, > the namespace deletes the corresponding inodes immediately. However it is > hard to to know when the invalidated blocks are actually deleted on the > datanodes, which could take a while. > I propose adding on namenode webUI, along with under replicated blocks, the > number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[ https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924794#comment-13924794 ] Hudson commented on HDFS-5986: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #503 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/503/]) HDFS-5986. Capture the number of blocks pending deletion on namenode webUI. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1575384) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java > Capture the number of blocks pending deletion on namenode webUI > --- > > Key: HDFS-5986 > URL: https://issues.apache.org/jira/browse/HDFS-5986 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Suresh Srinivas >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5986-screenshot.png, HDFS-5986.1.patch > > > When a directory that has large number of directories and files are deleted, > the namespace deletes the corresponding inodes immediately. However it is > hard to to know when the invalidated blocks are actually deleted on the > datanodes, which could take a while. > I propose adding on namenode webUI, along with under replicated blocks, the > number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[ https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924259#comment-13924259 ] Hudson commented on HDFS-5986: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5283 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5283/]) HDFS-5986. Capture the number of blocks pending deletion on namenode webUI. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1575384) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java > Capture the number of blocks pending deletion on namenode webUI > --- > > Key: HDFS-5986 > URL: https://issues.apache.org/jira/browse/HDFS-5986 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Suresh Srinivas >Assignee: Chris Nauroth > Attachments: HDFS-5986-screenshot.png, HDFS-5986.1.patch > > > When a directory that has large number of directories and files are deleted, > the namespace deletes the corresponding inodes immediately. However it is > hard to to know when the invalidated blocks are actually deleted on the > datanodes, which could take a while. > I propose adding on namenode webUI, along with under replicated blocks, the > number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[ https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924181#comment-13924181 ] Haohui Mai commented on HDFS-5986: -- Looks good to me. +1. Can you post a screenshot for a record? > Capture the number of blocks pending deletion on namenode webUI > --- > > Key: HDFS-5986 > URL: https://issues.apache.org/jira/browse/HDFS-5986 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Suresh Srinivas >Assignee: Chris Nauroth > Attachments: HDFS-5986.1.patch > > > When a directory that has large number of directories and files are deleted, > the namespace deletes the corresponding inodes immediately. However it is > hard to to know when the invalidated blocks are actually deleted on the > datanodes, which could take a while. > I propose adding on namenode webUI, along with under replicated blocks, the > number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[
https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924029#comment-13924029
]
Chris Nauroth commented on HDFS-5986:
-
The failure in {{TestCacheDirectives}} is unrelated. That test has been
failing intermittently.
> Capture the number of blocks pending deletion on namenode webUI
> ---
>
> Key: HDFS-5986
> URL: https://issues.apache.org/jira/browse/HDFS-5986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Suresh Srinivas
>Assignee: Chris Nauroth
> Attachments: HDFS-5986.1.patch
>
>
> When a directory that has large number of directories and files are deleted,
> the namespace deletes the corresponding inodes immediately. However it is
> hard to to know when the invalidated blocks are actually deleted on the
> datanodes, which could take a while.
> I propose adding on namenode webUI, along with under replicated blocks, the
> number of blocks that are pending deletion.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[
https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923742#comment-13923742
]
Hadoop QA commented on HDFS-5986:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633327/HDFS-5986.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6338//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6338//console
This message is automatically generated.
> Capture the number of blocks pending deletion on namenode webUI
> ---
>
> Key: HDFS-5986
> URL: https://issues.apache.org/jira/browse/HDFS-5986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Suresh Srinivas
>Assignee: Chris Nauroth
> Attachments: HDFS-5986.1.patch
>
>
> When a directory that has large number of directories and files are deleted,
> the namespace deletes the corresponding inodes immediately. However it is
> hard to to know when the invalidated blocks are actually deleted on the
> datanodes, which could take a while.
> I propose adding on namenode webUI, along with under replicated blocks, the
> number of blocks that are pending deletion.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[ https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907636#comment-13907636 ] Suresh Srinivas commented on HDFS-5986: --- Yes. It is the PendingDeletetionBlocksCount from invalidateBlocks. I like what @atm suggested as well. I do not think there is a metrics corresponding to this. > Capture the number of blocks pending deletion on namenode webUI > --- > > Key: HDFS-5986 > URL: https://issues.apache.org/jira/browse/HDFS-5986 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Suresh Srinivas > > When a directory that has large number of directories and files are deleted, > the namespace deletes the corresponding inodes immediately. However it is > hard to to know when the invalidated blocks are actually deleted on the > datanodes, which could take a while. > I propose adding on namenode webUI, along with under replicated blocks, the > number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
[
https://issues.apache.org/jira/browse/HDFS-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907627#comment-13907627
]
Kihwal Lee commented on HDFS-5986:
--
The jmx on NN already has {{PendingDeletionBlocks}} and [~wheat9] made NN webUI
render on the client-side using the jmx data, so it should be relatively a
simple change. Is {{PendingDeletionBlocks}} what we want, or is it something
else?
> Capture the number of blocks pending deletion on namenode webUI
> ---
>
> Key: HDFS-5986
> URL: https://issues.apache.org/jira/browse/HDFS-5986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Suresh Srinivas
>
> When a directory that has large number of directories and files are deleted,
> the namespace deletes the corresponding inodes immediately. However it is
> hard to to know when the invalidated blocks are actually deleted on the
> datanodes, which could take a while.
> I propose adding on namenode webUI, along with under replicated blocks, the
> number of blocks that are pending deletion.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
