[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[
https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376824#comment-17376824
]
Íñigo Goiri commented on HDFS-16100:
This is a pretty sensitive part of the code.
We need to make sure we have tests covering this.
In terms of readability, I think we may want to use the complement of the if:
{code}
if (reportedState != ReplicaState.RBW ||
storedBlock.getGenerationStamp() <= block.getGenerationStamp()) {
{code}
In addition, we may want to have comments explaining the reasoning.
> HA: Improve performance of Standby node transition to Active
> -
>
> Key: HDFS-16100
> URL: https://issues.apache.org/jira/browse/HDFS-16100
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.1
>Reporter: wudeyu
>Assignee: wudeyu
>Priority: Major
> Attachments: HDFS-16100.001.patch, HDFS-16100.patch
>
>
> pendingDNMessages in Standby is used to support process postponed block
> reports. Block reports in pendingDNMessages would be processed:
> # If GS of replica is in the future, Standby Node will process it when
> corresponding edit log(e.g add_block) is loaded.
> # If replica is corrupted, Standby Node will process it while it transfer to
> Active.
> # If DataNode is removed, corresponding of block reports will be removed in
> pendingDNMessages.
> Obviously, if num of corrupted replica grows, more time cost during
> transferring. In out situation, there're 60 millions block reports in
> pendingDNMessages before transfer. Processing block reports cost almost 7mins
> and it's killed by zkfc. The replica state of the most block reports is RBW
> with wrong GS(less than storedblock in Standby Node).
> In my opinion, Standby Node could ignore the block reports that replica state
> is RBW with wrong GS. Because Active node/DataNode will remove it later.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376251#comment-17376251 ] Xiaoqiao He commented on HDFS-16100: [^HDFS-16100.001.patch] it is safe to checkin and improve it for me. Kindly ping [~weichiu], [~inigoiri] do you mind to give another reviews? CI seems work not well and trigger it manually, refer: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/671/ > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.001.patch, HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[
https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17374323#comment-17374323
]
Xiaoqiao He commented on HDFS-16100:
Thanks [~ayushtkn] for your comments.
IMO, it is safe to to queue when `storedBlock.getGenerationStamp() <=
iblk.getGenerationStamp()` rather than `storedBlock.getGenerationStamp() ==
iblk.getGenerationStamp()` here.
{code:java}
+ if (!(reportedState == ReplicaState.RBW &&
+ storedBlock.getGenerationStamp() != iblk.getGenerationStamp())) {
+..
+ }
{code}
others look good to me. I will give my +1 when fix that. Thanks.
> HA: Improve performance of Standby node transition to Active
> -
>
> Key: HDFS-16100
> URL: https://issues.apache.org/jira/browse/HDFS-16100
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.1
>Reporter: wudeyu
>Assignee: wudeyu
>Priority: Major
> Attachments: HDFS-16100.patch
>
>
> pendingDNMessages in Standby is used to support process postponed block
> reports. Block reports in pendingDNMessages would be processed:
> # If GS of replica is in the future, Standby Node will process it when
> corresponding edit log(e.g add_block) is loaded.
> # If replica is corrupted, Standby Node will process it while it transfer to
> Active.
> # If DataNode is removed, corresponding of block reports will be removed in
> pendingDNMessages.
> Obviously, if num of corrupted replica grows, more time cost during
> transferring. In out situation, there're 60 millions block reports in
> pendingDNMessages before transfer. Processing block reports cost almost 7mins
> and it's killed by zkfc. The replica state of the most block reports is RBW
> with wrong GS(less than storedblock in Standby Node).
> In my opinion, Standby Node could ignore the block reports that replica state
> is RBW with wrong GS. Because Active node/DataNode will remove it later.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373232#comment-17373232 ] Ayush Saxena commented on HDFS-16100: - I am also not very confident about this part of code. In general looks ok, but might miss some corner case or so, not very sure > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372912#comment-17372912 ] Xiaoqiao He commented on HDFS-16100: I am not familiar to this logic, but it sounds reasonable at first glance. cc [~weichiu],[~ayushtkn] what do you think about? > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372103#comment-17372103 ] Xiaoqiao He commented on HDFS-16100: Thanks [~g20141821] for your report. Add [~g20141821] to contributors list and assign this issue to him. > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
