[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13892807#comment-13892807
]
Andrew Ash commented on HDFS-3570:
----------------------------------
+1 I'm observing this issue on cdh4.4.0
My setup is that there are 4 nodes, 3 of which are dedicated solely to HDFS and
the 4th is partially HDFS and partially other data. When running the balancer
it moves data from the lesser-used 3 nodes onto the higher-used 4th node
because its Used(%) is lower, and eventually fills up the 4th node's disk while
the remaining 3 nodes are underutilized. I'd rather the balancing be done
based on DFS Used %, which I'd define as:
DFS Used % = UsedTB / (ConfiguredCapacityTB - NonDFSUsedTB)
This hasn't been addressed in another ticket somewhere has it?
> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
> space
> --------------------------------------------------------------------------------
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Priority: Minor
>
> Report from a user here:
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
> post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among
> dfs.data.dirs, and very little DFS usage (which is computed against total
> possible capacity).
> Balancer apparently only looks at the usage, and ignores to consider that
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
> DFS Usage report from DN is 8% only, its got a lot of free space to write
> more blocks, when that isn't true as shown by the case of this user. It went
> on scheduling writes to the DN to balance it out, but the DN simply can't
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space
> being used for non DN data, but it does expose a valid bug.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
