[ https://issues.apache.org/jira/browse/HDFS-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Binglin Chang updated HDFS-5580: -------------------------------- Attachment: HDFS-5580.v1.patch Bug analysis: In Balancer.PendingBlockMove.chooseProxySource() {code} boolean find = false; for (BalancerDatanode loc : block.getLocations()) { // check if there is replica which is on the same rack with the target if (cluster.isOnSameRack(loc.getDatanode(), targetDN) && addTo(loc)) { find = true; // if cluster is not nodegroup aware or the proxy is on the same // nodegroup with target, then we already find the nearest proxy if (!cluster.isNodeGroupAware() || cluster.isOnSameNodeGroup(loc.getDatanode(), targetDN)) { return true; } } if (!find) { // find out a non-busy replica out of rack of target find = addTo(loc); } } {code} PendingBlockMove may be added to mulitple locations instead of one, but consumer thread pool only remove a pair of PendingBlockMove at a time, left some wild PendingBlockMove in the queue, Balancer.waitForMoveCompletion wait the queue become empty, which will never happen, causing dead lock. > Infinite loop in Balancer.waitForMoveCompletion > ----------------------------------------------- > > Key: HDFS-5580 > URL: https://issues.apache.org/jira/browse/HDFS-5580 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Binglin Chang > Assignee: Binglin Chang > Attachments: HDFS-5580.v1.patch, TestBalancerWithNodeGroupTimeout.log > > > In recent > [build|https://builds.apache.org/job/PreCommit-HDFS-Build/5592//testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerWithNodeGroup/testBalancerWithNodeGroup/] > in HDFS-5574, TestBalancerWithNodeGroup timeout, this is also mentioned in > HDFS-4376 > [here|https://issues.apache.org/jira/browse/HDFS-4376?focusedCommentId=13799402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13799402]. > > Looks like the bug is introduced by HDFS-4376. -- This message was sent by Atlassian JIRA (v6.1#6144)