[
https://issues.apache.org/jira/browse/HDFS-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700607#comment-14700607
]
Hudson commented on HDFS-8278:
------------------------------
FAILURE: Integrated in Hadoop-trunk-Commit #8315 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8315/])
HDFS-8278. When computing max-size-to-move in Balancer, count only the storage
with remaining >= default block size. (szetszwo: rev
51a00964da0e399718d1cec25ff692a32d7642b7)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> HDFS Balancer should consider remaining storage % when checking for
> under-utilized machines
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-8278
> URL: https://issues.apache.org/jira/browse/HDFS-8278
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
> Affects Versions: 2.8.0
> Reporter: Gopal V
> Assignee: Tsz Wo Nicholas Sze
> Fix For: 2.8.0
>
> Attachments: h8278_20150817.patch
>
>
> DFS balancer mistakenly identifies a node with very little storage space
> remaining as an "underutilized" node and tries to move large amounts of data
> to that particular node.
> All these block moves fail to execute successfully, as the % utilization is
> less relevant than the dfs remaining storage on that node.
> {code}
> 15/04/24 04:25:55 INFO balancer.Balancer: 0 over-utilized: []
> 15/04/24 04:25:55 INFO balancer.Balancer: 1 underutilized:
> [172.19.1.46:50010:DISK]
> 15/04/24 04:25:55 INFO balancer.Balancer: Need to move 47.68 GB to make the
> cluster balanced.
> 15/04/24 04:25:55 INFO balancer.Balancer: Decided to move 413.08 MB bytes
> from 172.19.1.52:50010:DISK to 172.19.1.46:50010:DISK
> 15/04/24 04:25:55 INFO balancer.Balancer: Will move 413.08 MB in this
> iteration
> 15/04/24 04:25:55 WARN balancer.Dispatcher: Failed to move
> blk_1078689321_1099517353638 with size=131146 from 172.19.1.52:50010:DISK to
> 172.19.1.46:50010:DISK through 172.19.1.53:50010: Got error, status message
> opReplaceBlock
> BP-942051088-172.18.1.41-1370508013893:blk_1078689321_1099517353638 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Out of
> space: The volume with the most available space (=225042432 B) is less than
> the block size (=268435456 B)., block move is failed
> {code}
> The machine in concern is under-full when it comes to the BP utilization, but
> has very little free space available for blocks.
> {code}
> Decommission Status : Normal
> Configured Capacity: 3826907185152 (3.48 TB)
> DFS Used: 2817262833664 (2.56 TB)
> Non DFS Used: 1000621305856 (931.90 GB)
> DFS Remaining: 9023045632 (8.40 GB)
> DFS Used%: 73.62%
> DFS Remaining%: 0.24%
> Configured Cache Capacity: 8589934592 (8 GB)
> Cache Used: 0 (0 B)
> Cache Remaining: 8589934592 (8 GB)
> Cache Used%: 0.00%
> Cache Remaining%: 100.00%
> Xceivers: 3
> Last contact: Fri Apr 24 04:28:36 PDT 2015
> {code}
> The machine has 0.40 Gb of non-RAM storage available on that node, so it is
> futile to attempt to move any blocks to that particular machine.
> This is a similar concern when a machine loses disks, since the comparisons
> of utilization always compare percentages per-node. Even that scenario needs
> to cap data movement to that node to the "DFS Remaining %" variable.
> Trying to move any more data than that to a given node will always fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
