[
https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724528#comment-14724528
]
Hudson commented on HDFS-8946:
------------------------------
FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8375/])
HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu:
rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
> Improve choosing datanode storage for block placement
> -----------------------------------------------------
>
> Key: HDFS-8946
> URL: https://issues.apache.org/jira/browse/HDFS-8946
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Yi Liu
> Assignee: Yi Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8946.001.patch, HDFS-8946.002.patch,
> HDFS-8946.003.patch
>
>
> This JIRA is to:
> Improve chooseing datanode storage for block placement:
> In {{BlockPlacementPolicyDefault}} ({{chooseLocalStorage}},
> {{chooseRandom}}), we have following logic to choose datanode storage to
> place block.
> For given storage type, we iterate storages of the datanode. But for
> datanode, it only cares about the storage type. In the loop, we check
> according to Storage type and return the first storage if the storages of the
> type on the datanode fit in requirement. So we can remove the iteration of
> storages, and just need to do once to find a good storage of given type, it's
> efficient if the storages of the type on the datanode don't fit in
> requirement since we don't need to loop all storages and do the same check.
> Besides, no need to shuffle the storages, since we only need to check
> according to the storage type on the datanode once.
> This also improves the logic and make it more clear.
> {code}
> if (excludedNodes.add(localMachine) // was not in the excluded list
> && isGoodDatanode(localDatanode, maxNodesPerRack, false,
> results, avoidStaleNodes)) {
> for (Iterator<Map.Entry<StorageType, Integer>> iter = storageTypes
> .entrySet().iterator(); iter.hasNext(); ) {
> Map.Entry<StorageType, Integer> entry = iter.next();
> for (DatanodeStorageInfo localStorage : DFSUtil.shuffle(
> localDatanode.getStorageInfos())) {
> StorageType type = entry.getKey();
> if (addIfIsGoodTarget(localStorage, excludedNodes, blocksize,
> results, type) >= 0) {
> int num = entry.getValue();
> ...
> {code}
> (current logic above)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
