[
https://issues.apache.org/jira/browse/HADOOP-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403275#comment-16403275
]
Xiao Chen commented on HADOOP-15317:
------------------------------------
Discussed with [~eddyxu] about this a little bit, uploaded
[^HADOOP-15317.02.patch]
We have several options:
# Similar to patch 1 to use a data structure to track the choose results so we
can terminate
# Use a ridiculous big number like (10 * numOfDatanodes) to guarantee
termination
# Instead of random choosing in every loop, we only randomize the initial
index, and then loop through every node exactly once
# Random once to get a number N, and only choose the Nth valid node
#1 will result in potentially largest memory consumption, #2 is minimal change
but could be the most CPU-heavy since it's just adding a termination to the
current while loop.
#3 and #4 should be the least resource-heavy. But #3 has the problem that the
probability of each node being chosen isn't evenly distributed. Specifically,
the node immediately after excluded node(s) will have a higher probability of
being chosen. Patch went with #4, whose only draw back is to make best case
slower.
Perhaps we can have a threshold (e.g. when availableNodes > numInScopeNodes /
2, we go with the simple stupid random loop in existing code, so #4's best case
slowness is covered.
Also thanks for the review [~ajayydv], comments addressed.
> Improve NetworkTopology chooseRandom's loop
> -------------------------------------------
>
> Key: HADOOP-15317
> URL: https://issues.apache.org/jira/browse/HADOOP-15317
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Xiao Chen
> Assignee: Xiao Chen
> Priority: Major
> Attachments: HADOOP-15317.01.patch, HADOOP-15317.02.patch
>
>
> Recently we found a postmortem case where the ANN seems to be in an infinite
> loop. From the logs it seems it just went through a rolling restart, and DNs
> are getting registered.
> Later the NN become unresponsive, and from the stacktrace it's inside a
> do-while loop inside {{NetworkTopology#chooseRandom}} - part of what's done
> in HDFS-10320.
> Going through the code and logs I'm not able to come up with any theory
> (thought about incorrect locking, or the Node object being modified outside
> of NetworkTopology, both seem impossible) why this is happening, but we
> should eliminate this loop.
> stacktrace:
> {noformat}
> Stack:
> java.util.HashMap.hash(HashMap.java:338)
> java.util.HashMap.containsKey(HashMap.java:595)
> java.util.HashSet.contains(HashSet.java:203)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:786)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:732)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:757)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:692)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:573)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:461)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:368)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:243)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:115)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4AdditionalDatanode(BlockManager.java:1596)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3599)
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:717)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
