[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368830#comment-17368830 ] Duo Zhang commented on HBASE-26029: --- After talkling with [~sunxin] offline, we think that the most clear way to fix this, is to add a step for SCP to assign the replication queue of the dead server to different region servers. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371044#comment-17371044 ] Bharath Vissapragada commented on HBASE-26029: -- Hooking this up with SCP is neat, should clean up all the dead server management code from ReplicationSourceManager, +1. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371994#comment-17371994 ] Hudson commented on HBASE-26029: Results for branch master [build #335 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372142#comment-17372142 ] Hudson commented on HBASE-26029: Results for branch branch-2 [build #289 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (x) {color:red}-1 client integration test{color} -- Something went wrong with this stage, [check relevant console output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289//console]. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)