[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11530:
------------------------------
Attachment: HDFS-11530.008.patch
Attaching v008 patch to fix the issue. [~linyiqun] [~arpitagarwal] would you
please take a look on this one some time? As v008 patch made a few small
changes but they kind of break some earlier assumptions.
More specifically, {{TestBlockManager.testSafeModeIBR}} fail seems to be caused
by that the {{add(Node node)}} was called more than once on the same node, with
different storage type info. This violates the earlier assumption that nodes
only get called by add once, or at least always the same storage info. The
reason seems that this tests is simulating datanode fails and this is why nodes
get added twice. This seems to be a valid case so I added
{{DFSTopologyNodeImpl.updateExistingDatanode}} for this.
On the other hand,
{{TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed}} fail seems to be
caused by that the {{excludedNodes}} set does not have to be
{{DatanodeDescriptor}}, but can also be {{DatanodeInfo}} instead. (Although
it's a bit suspicious to me why this is the only test that failed due to this).
Assuming this is valid, I modified
{{DFSNetworkTopology.chooseRandomWithStorageType}} to do some more check.
I do appreciate any comments. Basically I made the changes here assuming the
tests are doing the right thing, but it might very well be that the tests
themselves should be modified instead.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
> ------------------------------------------------------------------------------------
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Affects Versions: 3.0.0-alpha2
> Reporter: Yiqun Lin
> Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
