[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975343#comment-15975343
]
Chen Liang edited comment on HDFS-11530 at 4/19/17 7:25 PM:
Thanks [~linyiqun] for the updated patch!
Regarding the earlier discussion about the point#1 [~arpitagarwal] made. I
think the suspicious part is like this (please correct me if I make any wrong
statement here):
the original logic:
# find a random node (if no node found, there is no node available at all,
break the while loop)
# loop through all the storage requirements, to see if the found node can
satisfy (if it satisfies any of the storage type, {{numOfReplicas--;}} and
proceed)
the new logic from the patch:
# loop through all the storage type, for each type, find a node, if it
successfully finds a node for any storage type (say X), it breaks
# loop through all the storage requirements, to see if the found node can
satisfy (if it satisfies any of the storage type, {{numOfReplicas--;}} and
proceed)
There are two things about this difference:
# since now we pick node based on storage type (instead blindly picking a node
by chance), we probably don't need the #2 step in the original logic that loops
through all the storage requirements. i.e.
{code}
for (Iterator> iter = storageTypes
.entrySet().iterator(); iter.hasNext();) {...}
{code}
loop seems a bit unnecessary for new logic. Because we knew the node we got is
because it has type X, so we can probably just check for storage type X.
# in the old logic, when the while exits, all storage type requirement would be
found, unless there is no node to pick at all (in which case it breaks without
{{numOfReplicas}} down to 0). We should be careful about what is the proper
behaviour in the new logic.
was (Author: vagarychen):
Thanks [~linyiqun] for the updated patch!
Regarding the earlier discussion about the point#1 [~arpitagarwal] made. I
think the suspicious part is like this:
the original logic:
# find a random node (if no node found, there is no node available at all,
break the while loop)
# loop through all the storage requirements, to see if the found node can
satisfy (if it satisfies any of the storage type, {{numOfReplicas--;}} and
proceed)
the new logic from the patch:
# loop through all the storage type, for each type, find a node
# loop through all the storage requirements, to see if the found node can
satisfy (if it satisfies any of the storage type, {{numOfReplicas--;}} and
proceed)
There are two things about this difference:
# since now we pick node based on storage type (instead blindly picking a node
by chance), we probably don't need the #2 step in the original logic that loops
through all the storage requirements. i.e.
{code}
for (Iterator> iter = storageTypes
.entrySet().iterator(); iter.hasNext();) {...}
{code}
loop seems a bit unnecessary for new logic.
# in the old logic, when the while exits, all storage type requirement would be
found, unless there is no node to pick at all (in which case it breaks without
{{numOfReplicas}} down to 0). We should be careful about what is the proper
behaviour in the new logic.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch,
> HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch,
> HDFS-11530.012.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967462#comment-15967462
]
Yiqun Lin edited comment on HDFS-11530 at 4/13/17 11:45 AM:
Thanks [~arpitagarwal] for the suggestion. The comment makes sense to me. The
followings are what I did in my latest patch and will help you review.
* Add the new setting {{dfs.net.topology.impl}} that only used to choose nodes
to place replicas but not used to balance/move block replicas.
* The default topology of the setting is {{NetworkTopology}} since we still
need more testings to test {{DFSNetworkTopology}} before making it as the
default(as [~arpitagarwal] suggested).
* Update place that used the old config name from
{{CommonConfigurationKeysPublic.NET_TOPOLOGY_IMPL_KEY}} to
{{DFSConfigKeys.DFS_NET_TOPOLOGY_IMPL_KEY}} and add an additional test to test
the function of the new DFSNetworkTopology which parsed from the new setting.
* Please look the v011 patch that fix some minors based on v010 patch.
Any comments are welcomed. Thanks.
was (Author: linyiqun):
Thanks [~arpitagarwal] for the suggestion. The comment makes sense to me. The
followings are what I did in my latest patch and will help you review.
* Add the new setting {{dfs.net.topology.impl}} that only used to choose nodes
to place replicas but not used to balance/move block replicas.
* The default topology of the setting is {{NetworkTopology}} since we still
need more testings to test {{DFSNetworkTopology}} before making it as the
default(as [~arpitagarwal] suggested).
* Update place that used the old config name from
{{CommonConfigurationKeysPublic.NET_TOPOLOGY_IMPL_KEY}} to
{{DFSConfigKeys.DFS_NET_TOPOLOGY_IMPL_KEY}} and add an additional test to test
the function of the new DFSNetworkTopology which parsed from the new setting.
* Please look the v009 patch that fix some minors based on v008 patch.
Any comments are welcomed. Thanks.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch,
> HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967462#comment-15967462
]
Yiqun Lin edited comment on HDFS-11530 at 4/13/17 11:44 AM:
Thanks [~arpitagarwal] for the suggestion. The comment makes sense to me. The
followings are what I did in my latest patch and will help you review.
* Add the new setting {{dfs.net.topology.impl}} that only used to choose nodes
to place replicas but not used to balance/move block replicas.
* The default topology of the setting is {{NetworkTopology}} since we still
need more testings to test {{DFSNetworkTopology}} before making it as the
default(as [~arpitagarwal] suggested).
* Update place that used the old config name from
{{CommonConfigurationKeysPublic.NET_TOPOLOGY_IMPL_KEY}} to
{{DFSConfigKeys.DFS_NET_TOPOLOGY_IMPL_KEY}} and add an additional test to test
the function of the new DFSNetworkTopology which parsed from the new setting.
* Please look the v009 patch that fix some minors based on v008 patch.
Any comments are welcomed. Thanks.
was (Author: linyiqun):
Thanks [~arpitagarwal] for the suggestion. The comment makes sense to me. The
followings are what I did in my latest patch and will help you review.
* Add the new setting {{dfs.net.topology.impl}} that only used to choose nodes
to place replicas but not used to balance/move block replicas.
* The default topology of the setting is {{NetworkTopology}} since we still
need more testings to test {{DFSNetworkTopology}} before making it as the
default(as [~arpitagarwal] suggested).
* Update place that used the old config name from
{{CommonConfigurationKeysPublic.NET_TOPOLOGY_IMPL_KEY}} to
{{DFSConfigKeys.DFS_NET_TOPOLOGY_IMPL_KEY}} and add an additional test to test
the function of the new DFSNetworkTopology which parsed from the new setting.
Any comments are welcomed. Thanks.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch,
> HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965870#comment-15965870
]
Yiqun Lin edited comment on HDFS-11530 at 4/12/17 1:52 PM:
---
[~vagarychen], thanks a lot for helping fix the related test failure.
The latest patch overall looks good. Now it will be better to let
[~arpitagarwal] to take a look as well.
The units test are not broken now.
I have one thought: we can add the additional test for the special case we are
found when we are working on this JIRA(e.g. node added twice but different
storage type or the exclude node is not DatanodeDescriptor). By completing
this, we can test {{DFSNettopology}} in more different scenarios. BTW, I don't
mean we have to do this in this JIRA since the work of this JIRA is just to
apply {{DFSNettopology}} into default block placement and works well meanwhile .
was (Author: linyiqun):
[~vagarychen], thanks a lot for helping fix the related test failure.
The latest patch overall looks good. Now it will be better to let
[~arpitagarwal] to take a look as well.
The units test are not broken now.
I have one thought: we can add the additional test for the special case when we
are working on this JIRA(e.g. node added twice but different storage type or
the exclude node is not DatanodeDescriptor). By completing this, we can test
{{DFSNettopology}} in more different scenarios. BTW, I don't mean we have to do
this in this JIRA since the work of this JIRA is just to apply
{{DFSNettopology}} into default block placement and works well meanwhile .
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965870#comment-15965870
]
Yiqun Lin edited comment on HDFS-11530 at 4/12/17 1:51 PM:
---
[~vagarychen], thanks a lot for helping fix the related test failure.
The latest patch overall looks good. Now it will be better to let
[~arpitagarwal] to take a look as well.
The units test are not broken now.
I have one thought: we can add the additional test for the special case when we
are working on this JIRA(e.g. node added twice but different storage type or
the exclude node is not DatanodeDescriptor). By completing this, we can test
{{DFSNettopology}} in more different scenarios. BTW, I don't mean we have to do
this in this JIRA since the work of this JIRA is just to apply
{{DFSNettopology}} into default block placement and works well meanwhile .
was (Author: linyiqun):
[~vagarychen], thanks a lot for helping fix the related test failure.
The latest patch overall looks good. Now it will be better to let
[~arpitagarwal] to take a look as well.
The units test are not broken now.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964185#comment-15964185
]
Yiqun Lin edited comment on HDFS-11530 at 4/11/17 11:08 AM:
Post the new patch to fix findbug warning and update default policy used in
{{DatanodeManager}}.
was (Author: linyiqun):
Post the new patch to fix findbug warning and update default policy used in
{{DatanodeManager}}).
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962704#comment-15962704
]
Yiqun Lin edited comment on HDFS-11530 at 4/10/17 10:59 AM:
It seems that some tests are still failed due to different {{BlockPlacement}}
plociy used in HDFS.
For example when the different policy {{BlockPlacementPolicyWithNodeGroup}}
used, it will
cast networkTopology instance to {{NetworkTopologyWithNodeGroup}} and then test
will be failed,
I am thinking one reasonable way that we can use {{DFSNetworkTopology}} only if
the current block placement is the default
way(BlockPlacementPolicyRackFaultTolerant). Otherwise, we use the
{{NetworkTopology#getInstance(conf)}}.
Attach the updated patch.
Any comments? [~vagarychen], [~arpitagarwal].
was (Author: linyiqun):
It seems that some tests are still failed due to different {{BlockPlacement}}
plociy used in HDFS.
I am thinking one reasonable way that we can use {{DFSNetworkTopology}} only if
the current block placement is the default
way(BlockPlacementPolicyRackFaultTolerant). Otherwise, we use the
{{NetworkTopology#getInstance(conf)}}.
Attach the updated patch.
Any comments? [~vagarychen], [~arpitagarwal].
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958352#comment-15958352
]
Yiqun Lin edited comment on HDFS-11530 at 4/6/17 6:13 AM:
--
Thanks for your analysis, [~vagarychen]!
I think there is no need to use {{DFSNetworkTopology}} in {{Dispatcher}}. As I
look into the code, the {{NetworkTopology}} that used in class {{Dispatcher}}
is completely independent. It will add the nodes into topology in the
{{Dispatcher#init}} method. In addition, the method
{{NetworkTopology#isNodeGroupAware()}} will also be invoked in {{Dispatcher}}.
Attach the new patch to address this and fix checkstyle warning.
was (Author: linyiqun):
Thanks for your analysis, [~vagarychen]!
I think there is no need to use {{DFSNetworkTopology}} in {{Dispatcher}}. As I
look into the code, the {{NetworkTopology}} that used in class {{Dispatcher}}
is completely independent. It will add the nodes into topology in the
{{.Dispatcher#init}} method. In addition, the method
{{NetworkTopology#isNodeGroupAware()}} will also be invoked in {{Dispatcher}}.
Attach the new patch to address this and fix checkstyle warning.
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957343#comment-15957343
]
Chen Liang edited comment on HDFS-11530 at 4/5/17 6:22 PM:
---
Hi [~linyiqun], after revisiting the code, I have a question about the change
in the earlier patch.
I think when {{getInstance()}} is called, it is always a fresh network topology
instance being created, this applies to both the old and the new
{{NetworkTopology}} classes. Given this, I wonder, for classes where the new
{{chooseRandom}} is *not* going to be called, is it necessary at all to switch
to new topology class ?
More specifically, in the changes from v2 patch, {{Dispatcher}} switched to the
new topology class, but they never called the new {{chooseRandom}} method. So
in this case, is it really needed to switch to the new topology at all?
I digged down to this when I was looking at {{TestMover}} fail, which failed
because it adds {{DatanodeInfo}}, instead of {{DatanodeDescriptor}} to the
topology. However {{DFSNetworkTopology}} requires a {{DatanodeDescriptor}} as
the leaf nodes, because this class is where storage type info lives. This
appears to me that certain classes run with a slightly more abstracted datanode
representation that is not {{DatanodeDescriptor}}, in which case a more
detailed {{DFSNetworkTopology}} class might not be necessary.
An alternative way is to still replacing with {{DFSNetworkTopology}}, but
within {{DFSNetworkTopology}}, it checks if it is {{DatanodeInfo}} instead, if
yes, then it does not do anything fancy about storage type info, but pretty
much just stick to whatever it was in {{NetworkTopology}}.
Any comments? cc. [~arpitagarwal].
was (Author: vagarychen):
Hi [~linyiqun], after revisiting the code, I have a question about the change
in the earlier patch.
I think when {{getInstance()}} is called, it is always a fresh network topology
instance being created, this applies to both the old and the new
{{NetworkTopology}} classes. Given this, I wonder, for classes where the new
{{chooseRandom}} is *not* going to be called, is it necessary at all to switch
to new topology class ?
More specifically, in the changes from v2 patch, {{DatanodeManager}} and
{{Dispatcher}} switched to the new topology class, but they never called the
new {{chooseRandom}} method. So in this case, is it really needed to switch to
the new topology at all? Appears to me that only the changes to
{{BlockPlacementPolicyDefault}} are required.
I digged down to this when I was looking at {{TestMover}} fail, which failed
because it adds {{DatanodeInfo}}, instead of {{DatanodeDescriptor}} to the
topology. However {{DFSNetworkTopology}} requires a {{DatanodeDescriptor}} as
the leaf nodes, because this class is where storage type info lives. This
appears to me that certain classes run with a slightly more abstracted datanode
representation that is not {{DatanodeDescriptor}}, in which case a more
detailed {{DFSNetworkTopology}} class might not be necessary.
An alternative way is to still replacing with {{DFSNetworkTopology}}, but
within {{DFSNetworkTopology}}, it checks if it is {{DatanodeInfo}} instead, if
yes, then it does not do anything fancy about storage type info, but pretty
much just stick to whatever it was in {{NetworkTopology}}.
Any comments? cc. [~arpitagarwal].
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948903#comment-15948903
]
Yiqun Lin edited comment on HDFS-11530 at 3/30/17 11:43 AM:
Thanks [~vagarychen] for the comments.
I take some time to looked into this. I picked up one other type test failure
{{TestDatanodeRegistration#testChangeIpcPort}} to have a debug. The removing
node operation is failed if the node registers to namenode again with different
node name. Maybe this test failure leads other test failed.
[~vagarychen], could you share your thought of this or we can file a new JIRA
to have a track to cover this case? It seems there are still some work we
should do before making {{DFSNetworkTopology}} as the default HDFS
{{NetworkTopology}}.
In addition, I am thinking if we use {{DFSNetworkTopology}} as the default
class, is that mean the other subclass of {{NetworkTopology}}(e.g.
NetworkTopologyWithNodeGroup) will be not allowed used in HDFS?
was (Author: linyiqun):
Thanks [~vagarychen] for the comments.
I take some time to looked into this. I picked up one other type test failure
{{TestDatanodeRegistration#testChangeIpcPort}} to have a debug. The removing
node operation is failed if the node registers to namenode again with different
node name. Maybe this test failure leads other test failed.
[~vagarychen], could you share your thought of this or we can file a new JIRA
to have a track to cover this case? It seems there are still some works we
should do before making {{DFSNetworkTopology}} as the default NetworkTopology
{{NetworkTopology}}.
In addition, I am thinking if we use {{DFSNetworkTopology}} as the default
class, is that mean the other subclass of {{NetworkTopology}}(e.g.
NetworkTopologyWithNodeGroup) will be not allowed used in HDFS?
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
