[
https://issues.apache.org/jira/browse/HDFS-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doris Gu updated HDFS-11104:
----------------------------
Affects Version/s: 3.0.0-alpha1
> BlockPlacementPolicyDefault choose favoredNodes in turn which may cause
> imbalance
> ---------------------------------------------------------------------------------
>
> Key: HDFS-11104
> URL: https://issues.apache.org/jira/browse/HDFS-11104
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha1
> Reporter: Doris Gu
> Assignee: Ajith S
>
> if client transfer favoredNodes when it writes files into hdfs,chooseTarget
> in BlockPlacementPolicyDefault prior chooseTarget in turn:
> {quote}
> DatanodeStorageInfo[] chooseTarget(String src,
> int numOfReplicas,
> Node writer,
> Set<Node> excludedNodes,
> long blocksize,
> List<DatanodeDescriptor> favoredNodes,
> BlockStoragePolicy storagePolicy) {
> try {
> ...
> *for (int i = 0; i < favoredNodes.size() && results.size() <
> numOfReplicas; i++)* {
> DatanodeDescriptor favoredNode = favoredNodes.get(i);
> // Choose a single node which is local to favoredNode.
> // 'results' is updated within chooseLocalNode
> final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
> favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
> results, avoidStaleNodes, storageTypes, false);
> ...
> {quote}
> why not shuffle it here? Make block more balanced, save the cost balancer
> will pay and make cluster more stable.
> {quote}
> for (DatanodeDescriptor favoredNode :
> DFSUtil.shuffle(favoredNodes.toArray(new
> DatanodeDescriptor[favoredNodes.size()])))
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
