[jira] [Comment Edited] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962758#comment-16962758
]
Xiaoqiao He edited comment on HDFS-14882 at 10/30/19 7:08 AM:
--
Thanks [~pifta] for your suggestions,
{quote}I would suggest two more things to do, we might deprecate the old sorter
methods, as we most likely won't need them on the long run, as their use
effectively overrides the new setting, and an update would be nice to the
APIDoc of these methods.{quote}
It make sense for me. I agree that the following methods should be removed.
BTW, this common interface is invoke by different classes. I would like to
update that later.
{code:java}
public void sortByDistance(Node reader, Node[] nodes, int activeLen)
public void sortByDistanceUsingNetworkLocation(Node reader, Node[] nodes,
int activeLen)
{code}
Check the failed unit test {{TestNetworkTopology}} is related to this changes.
Will fix that later.
cc [~ayushtkn],[~elgoiri] any other comments?
was (Author: hexiaoqiao):
Thanks [~pifta] for your suggestions,
{quote}I would suggest two more things to do, we might deprecate the old sorter
methods, as we most likely won't need them on the long run, as their use
effectively overrides the new setting, and an update would be nice to the
APIDoc of these methods.{quote}
It make sense for me. I agree that the following methods should be removed.
BTW, this common interface is invoke by different classes. I would like to
update that later.
{code:java}
public void sortByDistance(Node reader, Node[] nodes, int activeLen)
public void sortByDistanceUsingNetworkLocation(Node reader, Node[] nodes,
int activeLen)
{code}
cc [~ayushtkn],[~elgoiri] any other comments?
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch,
> HDFS-14882.009.patch, HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956085#comment-16956085 ] Xiaoqiao He edited comment on HDFS-14882 at 10/21/19 1:31 PM: -- Check failed unit tests, most of them failed due to OOM, Please help to double check. Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. was (Author: hexiaoqiao): Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955493#comment-16955493 ] Ayush Saxena edited comment on HDFS-14882 at 10/20/19 1:27 PM: --- Thanx [~hexiaoqiao] IMO we shouldn't use the same configuration, someone turning on the old configuration, will now after this will turn this feature on too, which doesn't use to happen earlier. In general for anything new we usually keep the new feature turned off by default and I see the default for the config is true. I don't think we should force people into using this by default, since sorting too has some performance impact, So I would prefer them turning this on explicitly. Though things are quiet similar but not for same thing, I think we should have a separate config. Moreover for the test, You may add a case having decommissioned or stale datanodes and verify the case that they stay at end irrespective of the distance. was (Author: ayushtkn): Thanx [~hexiaoqiao] IMO we shouldn't use the same configuration, someone turning on the old configuration, will now after this will turn this feature on too, which doesn't use to happen earlier. In general for anything new we usually keep the new feature turned off by default and I see the default for the config is true. I don't think we should force people into using this by default, since sorting too has some performance impact, So I would prefer them turning this on explicitly. Though things are quiet similar but not for same thing, I think we should have a seperate config. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
