[
https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15676336#comment-15676336
]
Nandakumar commented on HDFS-10206:
-----------------------------------
bq. From the below code, it seems each level will increase by 2.
Yes, you're right. Distance from a node to it's parent is assumed to be 1, so
adding the value of both the nodes will make it 2.
Sorry for the confusion on that.
{quote}
one is the reader being a datanode in a remote rack in a large cluster; for
that NetworkTopology already has the reader in its tree, it will be faster to
compare parents reference.
{quote}
We can use {{NetworkTopology.contains(node)}} to check if the reader is a
datanode and use {{NetworkTopology.getDistance(node1, node2)}} to get the
distance (which also calculates the distance by summing up the nodes distances
to their closest common ancestor), but both of these methods use
{{ReadWriteLock.readLock()}} which might again impact the performance.
Any comments on using {{NetworkTopology.contains(node)}} to check and use
{{NetworkTopology.getDistance(node1, node2)}} to get the distance in case if
the reader is an off rack datanode?
I'm currently working on the benchmarking, will update it once it's done.
> getBlockLocations might not sort datanodes properly by distance
> ---------------------------------------------------------------
>
> Key: HDFS-10206
> URL: https://issues.apache.org/jira/browse/HDFS-10206
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ming Ma
> Assignee: Nandakumar
> Attachments: HDFS-10206.000.patch
>
>
> If the DFSClient machine is not a datanode, but it shares its rack with some
> datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}}
> might not put the local-rack datanodes at the beginning of the sorted list.
> That is because the function didn't call {{networktopology.add(client);}} to
> properly set the node's parent node; something required by
> {{networktopology.sortByDistance}} to compute distance between two nodes in
> the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only
> distinguishes local rack from remote rack, but it doesn't support general
> distance calculation to tell how remote the rack is.
> {noformat}
> NetworkTopology.java
> protected int getWeight(Node reader, Node node) {
> // 0 is local, 1 is same rack, 2 is off rack
> // Start off by initializing to off rack
> int weight = 2;
> if (reader != null) {
> if (reader.equals(node)) {
> weight = 0;
> } else if (isOnSameRack(reader, node)) {
> weight = 1;
> }
> }
> return weight;
> }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to
> address another issue. Regardless of where we do the sorting, we still need
> fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object
> used by DatanodeManager and requires Nodes stored in the topology to be
> {{DatanodeDescriptor}} for block placement. So we need to make sure we don't
> pollute the NetworkTopology if we plan to fix it on the server side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
