[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144160#comment-15144160 ] Vinayakumar B commented on HDFS-9038: - Ping [~cnauroth]. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177050#comment-15177050 ] Brahma Reddy Battula commented on HDFS-9038: bq. I suggest that we leave this issue open for a while for others to comment. I could be swayed if others think I'm approaching this incorrectly [~cnauroth]/[~arpitagarwal]/[~vinayrpet] can we discuss in mailing list Or do you want specific people to comment here.. Please let me know.. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177060#comment-15177060 ] Arpit Agarwal commented on HDFS-9038: - Hi [~brahmareddy], I am sorry this patch is stuck after all the effort you put into it. There is a disagreement on what the definition of non-DFS used space. Quoting [~cnauroth]: bq. Essentially, it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used". I don't see any harm in starting a discussion on hdfs-dev. I will think about this some more. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186681#comment-15186681 ] Akira AJISAKA commented on HDFS-9038: - bq. it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used". I prefer (I'm +0 for) the following calculation, which is straightforward for me. {noformat} Non-DFS used = f.getTotalSpace() - f.getFreeSpace() - getDfsUsed() {noformat} When we reach an agreement, it should be well-documented. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051260#comment-15051260 ] Arpit Agarwal commented on HDFS-9038: - I also edited the Jira title to mention "DFS reserved space", to distinguish it from system-reserved space. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051473#comment-15051473 ] Brahma Reddy Battula commented on HDFS-9038: [~arpitagarwal] updated previous patch(004) as per Chris comments and added check for -ve values.. bq. I'll file a separate Jira to discuss fixing this part. will discuss further in seperate jira. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051734#comment-15051734 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 8 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 5s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 53s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 16s {color} | {color:red} hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 50s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 6s {color} | {color:red} hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s {color} | {color:red} Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 292, now 294). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 17s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. {color} | | {color:red}-1{color}
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051894#comment-15051894 ] Chris Nauroth commented on HDFS-9038: - I took patch v006 for a more thorough test run. Unfortunately, I'm still seeing the bug that was introduced by HDFS-5215. Even though non-DFS used is now calculated explicitly by the DataNode and sent in a dedicated RPC field, it's still using an incorrect calculation. To help illustrate the problem better, I'd like to walk through the math for calculating non-DFS used, both before HDFS-5215 and after HDFS-5215. Here is a bit of pseudo-code, prefixed with line numbers. *Before HDFS-5215:* {code} 1: nonDfsUsed = capacity - dfsUsed - available 2:= (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - available 3:= (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - File#getUsableSpace 4:= File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace {code} *After HDFS-5215* {code} 1: nonDfsUsed = capacity - dfsUsed - available 2:= (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - available 3:= (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace - dfs.datanode.du.reserved) 4:= File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace + dfs.datanode.du.reserved 5:= File#getTotalSpace - dfsUsed - File#getUsableSpace {code} The most important point to note is the difference in line 3 of the expansions. After HDFS-5215, the definition of {{available}} removes {{dfs.datanode.du.reserved}}. However, the definition of {{capacity}} already removed {{dfs.datanode.du.reserved}}. Carrying out the expansion further in lines 4 and 5, we see that the 2 occurrences of {{dfs.datanode.du.reserved}} cancel each other out. That means implicitly that {{nonDfsUsed}} now includes (instead of excludes) {{dfs.datanode.du.reserved}}. Patch v006 still has the same fundamental problem in the logic, because {{FsVolumeImpl#getNonDfsUsed}} still reuses the implementation of {{FsVolumeImpl#getAvailable}}, which in turn removes {{dfs.datanode.du.reserved}}. I think {{FsVolumeImpl#getNonDfsUsed}} essentially needs to go back to the old pre-HDFS-5215 calculation of {{remaining}} instead of calling {{FsVolumeImpl#getAvailable}}. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052039#comment-15052039 ] Vinayakumar B commented on HDFS-9038: - Thanks [~cnauroth] for the explanation. Lets go through one simple example. Lets consider following 1. Disk Capacity - 10GB -- {{File#getTotalSpace}} 2. DFS reserved - 1GB -- {{dfs.datanode.du.reserved}} 3. DFS Usage - 2GB -- {{dfsUsed}} 4. Usage by other files - 3GB -- Actual {{non-dfsusage}}. Assuming, System reserved ( which is ~5%) as 0 for this example. total disk used is 5GB = 2GB by dfs files + 3GB by non-dfs files. {{File.getUsableSpace()}} returns 5GB. Lets substitute this values in derived expressions. *Before HDFS-5215* {noformat} nonDfsUsage=File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace nonDfsUsage = 10G - 1G - 2G - 5G nonDfsUsage = 2G. {noformat} {color:red}{{nonDfsUsage=2GB}} is not same as actual non-dfs usage, 3GB.{color} *After HDFS-5215* and as per patch. {noformat} nonDfsUsage=(File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace - dfs.datanode.du.reserved) nonDfSUsage= (10G - 1G ) - 2G - (5G - 1G) nonDfSUsage = 9G - 2G - 4G nonDfsUsage = 3G {noformat} {color:green}{{nonDfsUsage = 3G}} is same as expected actual usage. i.e. 3G{color} By seeing the above example, I believe, post HDFS-5215, calculation turns out to be the correct one. {{dfs.datanode,du.reserved}} is expected to be the freespace, which should not be used by DFS. So its reasonable to subtract this in getAvailable(), but not while finding the non-dfs usage. Do you agree [~cnauroth]? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052049#comment-15052049 ] Vinayakumar B commented on HDFS-9038: - Meanwhile, [~brahmareddy], forgot to update the test {{TestFsVolumeList.testNonDfsUsedMetricForVolume}}? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052994#comment-15052994 ] Brahma Reddy Battula commented on HDFS-9038: [~vinayrpet] thanks for pointing..Corrected and uploaded the patch. Apart from that added null check in {{TestNameNodeMetrics#tearDown}} and fixed javadoc error in {{FsVolumeImpl#getDataset}} which are not related this jira. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053202#comment-15053202 ] Chris Nauroth commented on HDFS-9038: - bq. I believe, post HDFS-5215, calculation turns out to be the correct one. I disagree, because the post-HDFS-5215 calculation has broken previously established operations workflows. Non-DFS usage is "space unexpectedly consumed on data volumes by things that are not HDFS". As a cluster administrator, I would monitor the non-DFS usage value. If a node consistently showed a non-zero non-DFS usage, then that would signal me to login to the box, figure out where the space was consumed, free it up, and then address root cause (probably a rogue process running on the wrong box or misconfigured to write to the wrong volume). It's important to fix this, because high non-DFS usage reduces disk capacity that had been planned for HDFS. After HDFS-5215, this workflow no longer works. The monitoring will show false positives because of inclusion of {{dfs.datanode.du.reserved}}. An administrator would need to use additional checks, or simply disable this monitoring due to the noise. {{dfs.datanode.du.reserved}} is a special case on top of what I described above. Setting a non-zero {{dfs.datanode.du.reserved}} means that the administrator intentionally wants to hold back a portion of the volume, essentially making it invisible to HDFS. This is the same reason that {{dfs.datanode.du.reserved}} gets subtracted from the capacity calculation. This is why I think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS usage". By setting {{dfs.datanode.du.reserved}} to 1 GB, you have stated that up to 1 GB of usage by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected non-DFS usage". Bottom line: I have seen the post-HDFS-5215 calculation cause confusion for administrators who had built a workflow around the old calculation. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053672#comment-15053672 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 8 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 3s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 56s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 30s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 9s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 51s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 46s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 42s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 0s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 0s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 0s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 19s {color} | {color:red} hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 0s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 35s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 35s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 23m 54s {color} | {color:red} hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 35s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 44s {color} | {color:red} Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 54s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 55s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 50s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053765#comment-15053765 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 8 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 49s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 50s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 31s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 39s {color} | {color:red} hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 36s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 16s {color} | {color:red} hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 36s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s {color} | {color:red} Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m 27s {color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053971#comment-15053971 ] Vinayakumar B commented on HDFS-9038: - bq. dfs.datanode.du.reserved means that the administrator intentionally wants to hold back a portion of the volume, essentially making it invisible to HDFS. This is the same reason that dfs.datanode.du.reserved gets subtracted from the capacity calculation. This is why I think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS usage". By setting dfs.datanode.du.reserved to 1 GB, you have stated that up to 1 GB of usage by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected non-DFS usage". So you mean, when 1GB is configured as reserved, 'Non-DFS' usage metric should show 2GB, even though actual usage by files other than HDFS is 3GB.? I was thinking 'NonDfsUsage' should show actual non-dfs usage. Not only unexpected part of it. So if actual non-dfs usage is within this reserved limit, metric should show 0. right? In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage' to make it more clear? Or mention somewhere to clear confusion in people like me. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056966#comment-15056966 ] Chris Nauroth commented on HDFS-9038: - bq. So if actual non-dfs usage is within this reserved limit, metric should show 0. right? Yes, that's my understanding, and that's the behavior pre-HDFS-5215. bq. In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage' to make it more clear? Or mention somewhere to clear confusion in people like me. I don' t think it could be renamed easily due to backwards-compatibility, but I do think we could update Metrics.md. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057077#comment-15057077 ] Arpit Agarwal commented on HDFS-9038: - [~brahma], thanks for your patience as we work through the math. It may be useful to describe your proposed derivation as a Jira comment before you post another patch. [~cnauroth] I think you are right in conclusion but there is a misstep in the equations in [this comment|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15051894&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15051894]. {{reserved}} _should_ cancel out and it should not factor in the final computation. The problem with the v005 patch we missed earlier is that getCapacity() subtracts reserved space. We should use the raw capacity. One way to fix it is: {code} public long getNonDfsUsed() throws IOException { long totalFreeSpace = currentDir.getUsableSpace(); long nonDfsUsed = getCapacity() + reserved - getDfsUsed() - totalFreeSpace; return (nonDfsUsed >= 0) ? nonDfsUsed : 0; } {code} Then the derivation becomes: {code} 1: non-DFS used = getCapacity() + reserved - getDfsUsed() - totalFreeSpace 2: = usage.getCapacity() - reserved + reserved - getDfsUsed() - totalFreeSpace 3: = usage.getCapacity() - getDfsUsed() - totalFreeSpace 4: = File#getTotalSpace - getDfsUsed() - File#getFreeSpace {code} Hope that makes sense. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057367#comment-15057367 ] Vinayakumar B commented on HDFS-9038: - [~arpitagarwal], thanks for jumping in for calculation. I think considering We use {{File#getUsableSpace}} instead of {{File#getFreeSpace}}, current code after HDFS-5215 also comes to same equation as mentioned by [~arpitagarwal], brahma's patch doesn't change anything in this, except subtracting 'reservedForReplicas' also. But, I think what [~cnauroth] expects is, *Since {{reserved}} is hidden from the HDFS in {{getCapacity()}} itself, we can think that's already used for {{nonDfs}} and subtract it from actual ondisk {{nonDfsUsage}}, showing only HDFS visible {{nonDfsUsage}}*. {{nonDfsUsage}} metric will be positive, only if the ondisk nonDfsUsage crosses beyond {{reserved}}, and it shows only excess usage beyond {{reserved}}. i.e. nonDfsUsage will be {{2GB}} instead of {{3GB}} in my earlier example, where {{1GB}} was reserved. So final equation would be {code}nonDfsUsage=File#getTotalSpace - dfsUsed - File#getUsableSpace - reserved{code} Code would look like this. {code} public long getNonDfsUsed() throws IOException { long nonDfsUsed = getCapacity() - getDfsUsed() - getAvailable() - reserved - getReservedForReplicas(); return (nonDfsUsed > 0) ? nonDfsUsed : 0; } {code} Am I right [~cnauroth]? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058764#comment-15058764 ] Chris Nauroth commented on HDFS-9038: - Thanks everyone for sticking with this. This has turned out to be much trickier than I anticipated when I filed the issue. I'd like to summarize current status. Arpit and I are in agreement about my analysis of how the calculation changed after HDFS-5215. However, we are not yet in agreement about which calculation is truly correct. I believe the pre-HDFS-5215 calculation (subtracting {{dfs.datanode.du.reserved}}) is correct, because it allowed me to monitor for unexpected non-zero non-DFS usage and react. Since this was an established operations workflow (at least for me), I argue that we have a responsibility to restore that behavior. Arpit believes that it's correct to cancel out {{dfs.datanode.du.reserved}}, because then non-DFS used would report space used for non-HDFS purposes more accurately. Essentially, it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used". We also discovered an interesting side issue about {{File#getUsableSpace}} vs. {{File#getFreeSpace}}. Pre-HDFS-5215, it could be considered a bug that we did not account for system reserved space. Interestingly, it seems in our testing that ext holds back 5% by default, but xfs does not. I pushed pretty hard for restoring the pre-HDFS-5215 behavior in my earlier comments, but I'm just one voice. I suggest that we leave this issue open for a while for others to comment. I could be swayed if others think I'm approaching this incorrectly. Meanwhile, [~brahmareddy], would you please hold off on posting more patches? Let's wait for the discussion to settle a little more first. Thanks for your patience. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058805#comment-15058805 ] Allen Wittenauer commented on HDFS-9038: I'd really like for someone to attach a simple test case so that we can run and see the differences on different file systems. In particular, every time we change this code, we only ever test on one or two Linux file systems and end up causing really oddball behaviors on others, especially those that don't use the traditional disk->partition->filesystem layout (e.g., ZFS). > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059597#comment-15059597 ] Vinayakumar B commented on HDFS-9038: - Hi [~umamaheswararao]/[~hitliuyi]/[~szetszwo]/[~wheat9], Do You have any opinions to here? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105047#comment-15105047 ] Vinayakumar B commented on HDFS-9038: - Ping. Any update on this guys? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437020#comment-15437020 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 52s{color} | {color:red} hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new + 583 unchanged - 5 fixed = 584 total (was 588) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 0s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 7s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.mover.TestStorageMover | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12825464/HDFS-9038-010.patch | | JIRA Issue | HDFS-9038 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 4f982e9d4706 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 525d52b | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | javac | https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16537/
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446268#comment-15446268 ] Brahma Reddy Battula commented on HDFS-9038: [~arpiagariu] can you please look into latest patch.. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450427#comment-15450427 ] Arpit Agarwal commented on HDFS-9038: - Thanks for sticking with this difficult fix [~brahmareddy]. The patch looks good to me (I am still reviewing unit tests). Nitpick - {{convert(DatanodeInfoProto di)}} can use the new {{DataNodeInfo}} constructor. [~vinayrpet], [~cnauroth], [~szetszwo], do you have any comments on the latest patch. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460609#comment-15460609 ] Brahma Reddy Battula commented on HDFS-9038: [~arpitagarwal] thanks for review.. will update patch once your review completed.. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460919#comment-15460919 ] Arpit Agarwal commented on HDFS-9038: - The tests also look fine to me. Thanks [~brahmareddy]. We should hold off committing for until next week in case Chris or Vinayakumar want to take a look at the new patch since they reviewed earlier revisions. It'd be great to have a targeted unit test for {{PBHelperClient#convert(DatanodeInfoProto di)}} that verifies deserialization of protobufs with and without the new non-dfs used field. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461883#comment-15461883 ] Chris Nauroth commented on HDFS-9038: - [~brahmareddy], [~arpitagarwal] and [~vinayrpet], thank you for your dedication working through this issue. I am +1 to proceed with the patch after addressing Arpit's last round of feedback. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15465359#comment-15465359 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 10 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 26s{color} | {color:red} hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new + 639 unchanged - 5 fixed = 640 total (was 644) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m 35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}108m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-9038 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827080/HDFS-9038-011.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 3ecb99211ffb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 07650bc | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | javac | https://builds.apache.org/job/PreCommit-HDFS-Build/16636/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16636/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt | | Test Results | ht
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15466291#comment-15466291 ] Arpit Agarwal commented on HDFS-9038: - +1 lgtm. The findbugs warning is due to a deprecated function which we must invoke to handle wire compatibility. The checkstyle issue was also not introduced by this patch. Thanks again [~brahmareddy] for persistently driving this issue to closure. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15466360#comment-15466360 ] Vinayakumar B commented on HDFS-9038: - +1 for the latest changes. Thanks [~brahmareddy], [~arpitagarwal], [~cnauroth] for the co-operations. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15468552#comment-15468552 ] Hudson commented on HDFS-9038: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10401 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10401/]) HDFS-9038. DFS reserved space is erroneously counted towards non-DFS (arp: rev 5f23abfa30ea29a5474513c463b4d462c0e824ee) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageReport.java > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15468897#comment-15468897 ] Andrew Wang commented on HDFS-9038: --- Gentle reminder, please set the appropriate 3.0.0 fix version when committing to trunk. Thanks! > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470539#comment-15470539 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 0s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 54s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 56s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_101 with JDK v1.8.0_101 generated 1 new + 44 unchanged - 1 fixed = 45 total (was 45) {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 59s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111 with JDK v1.7.0_111 generated 1 new + 47 unchanged - 1 fixed = 48 total (was 48) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 523 unchanged - 3 fixed = 527 total (was 526) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 7569 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 3m 11s{color} | {color:red} The patch 137 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 22s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s{color} | {color:red} The patch generated 3 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 57s{color} | {color:black} {color} | \\ \\ || Reason ||
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470742#comment-15470742 ] Brahma Reddy Battula commented on HDFS-9038: {{checkstyle}} : can be ignored ,those are related to file length,method parameters (>7) and hiddenfield. {{javac}}: which is not introduced by this patch. {{whitespace}} : Seems to be some problem with report..? {{asflicense}} : Not related to this patch {{unittest failures}} : Not related to this patch [~arpitagarwal] can please review branch-2.7 patch..? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038-branch-2.7.patch, > HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15471751#comment-15471751 ] Arpit Agarwal commented on HDFS-9038: - +1 for the branch-2.7 patch also. Thanks Brahma. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038-branch-2.7.patch, > HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996340#comment-15996340 ] maobaolong commented on HDFS-9038: -- ```java public long getNonDfsUsed() throws IOException { long actualNonDfsUsed = getActualNonDfsUsed(); if (actualNonDfsUsed < reserved) { return 0L; } return actualNonDfsUsed - reserved; } ``` The code block above is the function to caculate nonDfsUsed, but in fact it will let the result to be 0L out of expect. Such as this following situation: du.reserved = 50G Disk Capacity = 2048G Disk Available = 2000G Dfs used = 30G usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace() = 2048G - 2000G = 48G getActualNonDfsUsed = usage.getUsed() - getDfsUsed() = 48G - 30G = 18G 18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and the NonDfsUsed will return 0, is that logic make sense? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038-branch-2.7.patch, > HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996609#comment-15996609 ] Vinayakumar B commented on HDFS-9038: - {code} /** * Unplanned Non-DFS usage, i.e. Extra usage beyond reserved. * * @return * @throws IOException */ public long getNonDfsUsed() throws IOException { long actualNonDfsUsed = getActualNonDfsUsed(); if (actualNonDfsUsed < reserved) { return 0L; } return actualNonDfsUsed - reserved; }{code} As the javadoc says, its the UnPlanned Non-DFS usage beyond reserved. If the actual NonDFS usage is within reserved limits, this returns 0. You can see multiple discussions happened earlier in comments. So YES, this logic makes sense. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038-branch-2.7.patch, > HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004400#comment-16004400 ] maobaolong commented on HDFS-9038: -- [~vshreyas] Thank you, I have known the NonDFS used mean. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, > HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038-branch-2.7.patch, > HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321892#comment-15321892 ] Arpit Agarwal commented on HDFS-9038: - I spent a few hours looking at this today. I now think that the current logic in FsVolumeImpl#getAvailable is too pessimistic the impact of bug may be more a more severe than reported. {code} public long getAvailable() throws IOException { long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get(); long available = usage.getAvailable() - reserved - reservedForReplicas.get(); if (remaining > available) { remaining = available; } return (remaining > 0) ? remaining : 0; } {code} Specifically {{available = usage.getAvailable() - reserved ... }} can account for the non-DFS usage twice. e.g. say 10GB is reserved and YARN is using all of that 10GB. usage.getAvailable() has already accounted for that 10GB and further subtracting it makes us underestimate available. Instead the calculation should only subtract the unused portion of non-DFS used. What do you guys think? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321907#comment-15321907 ] Vinayakumar B commented on HDFS-9038: - bq. Instead the calculation should only subtract the unused portion of non-DFS used. What do you guys think? I think below code change suggested by brahma in the mailing list does exactly what you suggesting. Right? {code} public long getAvailable() throws IOException { long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get(); -long available = usage.getAvailable() - reserved +long available = usage.getAvailable() - getRemainingReserved() - reservedForReplicas.get(); if (remaining > available) { remaining = available; @@ -391,6 +391,31 @@ public long getAvailable() throws IOException { return (remaining > 0) ? remaining : 0; } + private long getActualNonDfsUsed() throws IOException { +return usage.getUsed() - getDfsUsed(); + } + + private long getRemainingReserved() throws IOException { +long actualNonDfsUsed = getActualNonDfsUsed(); +if (actualNonDfsUsed < reserved) { + return reserved - actualNonDfsUsed; +} +return 0L; + } + + /** + * Unplanned Non-DFS usage, i.e. Extra usage beyond reserved. + * @return + * @throws IOException + */ + public long getNonDfsUsed() throws IOException { +long actualNonDfsUsed = getActualNonDfsUsed(); +if (actualNonDfsUsed < reserved) { + return 0L; +} +return actualNonDfsUsed - reserved; + } + {code} > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322988#comment-15322988 ] Arpit Agarwal commented on HDFS-9038: - Thanks Vinay. I see [~szetszwo] already brought this up on the mailing list thread. Brahma's proposed update looks good to me. Will this delta be applied on top of the v7 patch? > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323871#comment-15323871 ] Brahma Reddy Battula commented on HDFS-9038: Cool.. I will update v7 patch. thanks vinay and arpit.. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407668#comment-15407668 ] Brahma Reddy Battula commented on HDFS-9038: Uploaded the patch.. [~arpitagarwal] and [~vinayrpet] kindly review.. and sorry for long delay. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407816#comment-15407816 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 8 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 23s{color} | {color:red} hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 33s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new + 540 unchanged - 2 fixed = 547 total (was 542) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 20s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 86m 46s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | | | hadoop.tracing.TestTracing | | | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | | | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12822051/HDFS-9038-008.patch | | JIRA Issue | HDFS-9038 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 039d0e6011f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 70c2781 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | javac | https://builds.ap
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410512#comment-15410512 ] Hadoop QA commented on HDFS-9038: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 59s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 48s{color} | {color:red} hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new + 594 unchanged - 5 fixed = 596 total (was 599) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 40s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 41s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency | | | hadoop.tracing.TestTracing | | | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | | | hadoop.hdfs.TestFileCreationDelete | | | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | | | hadoop.security.TestRefreshUserMappings | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12822418/HDFS-9038-009.patch | | JIRA Issue | HDFS-9038 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 16bca1a0834f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk
[jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
[ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414078#comment-15414078 ] Arpit Agarwal commented on HDFS-9038: - Hi [~brahmareddy], thanks for the updated patch. I will try to review this next week. > DFS reserved space is erroneously counted towards non-DFS used. > --- > > Key: HDFS-9038 > URL: https://issues.apache.org/jira/browse/HDFS-9038 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Chris Nauroth >Assignee: Brahma Reddy Battula > Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, > HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, > HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch, HDFS-9038.patch > > > HDFS-5215 changed the DataNode volume available space calculation to consider > the reserved space held by the {{dfs.datanode.du.reserved}} configuration > property. As a side effect, reserved space is now counted towards non-DFS > used. I don't believe it was intentional to change the definition of non-DFS > used. This issue proposes restoring the prior behavior: do not count > reserved space towards non-DFS used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org