[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058858#comment-15058858
]
Arpit Agarwal edited comment on HDFS-9038 at 12/15/15 9:34 PM:
---------------------------------------------------------------
tl; dr - We need agreement on the definition of non-DFS used.
The pre HDFS-5215 calculation had two bugs.
# It incorrectly subtracted reserved space from the non-DFS used. (net
negative). Chris suggests this is not really an issue as non-DFS used should be
shown as zero unless it exceeds the DFS reserved value.
# It used File#getUsableSpace to calculate the volume free space instead of
File#getFreeSpace. (net positive)
The net effect was that non-DFS used was displayed as zero unless the actual
non-DFS used exceeded {{DFS reserved - system reserved}}.
HDFS-5215 fixed the first issue and the value that is now erroneously counted
towards non-DFS used is in fact the system reserved 5%.
Also attached a trivial utility that dumps the free/available space.
>From a mostly empty 40GB Ext4 partition:
{code}
$ java GetFree /mnt/sdb/hadoop/
Free space : 42,090,229,760
Available space : 39,925,968,896
{code}
Same partition reformatted as XFS:
{code}
Free space : 42,894,983,168
Available space : 42,894,983,168
{code}
So Ext derivatives hold back 5% free space while XFS does not.
Edit: Removed statement about Jira description being inaccurate since it too
depends on how we define non-DFS used. :)
was (Author: arpitagarwal):
tl; dr - We need agreement on the definition of non-DFS used.
IMO the original Jira description is inaccurate. The pre HDFS-5215 calculation
had two bugs.
# It incorrectly subtracted reserved space from the non-DFS used. (net
negative). Chris suggests this is not really an issue as non-DFS used should be
shown as zero unless it exceeds the DFS reserved value.
# It used File#getUsableSpace to calculate the volume free space instead of
File#getFreeSpace. (net positive)
The net effect was that non-DFS used was displayed as zero unless the actual
non-DFS used exceeded {{DFS reserved - system reserved}}.
HDFS-5215 fixed the first issue and the value that is now erroneously counted
towards non-DFS used is in fact the system reserved 5%.
Also attached a trivial utility that dumps the free/available space.
>From a mostly empty 40GB Ext4 partition:
{code}
$ java GetFree /mnt/sdb/hadoop/
Free space : 42,090,229,760
Available space : 39,925,968,896
{code}
Same partition reformatted as XFS:
{code}
Free space : 42,894,983,168
Available space : 42,894,983,168
{code}
So Ext derivatives hold back 5% free space while XFS does not.
> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch,
> HDFS-9038-007.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
