[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575921#comment-15575921 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1283 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1283/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev 09b89eedb585517d734bf3af3875e0e1ff5359e0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575250#comment-15575250 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-0.98-matrix #410 (See [https://builds.apache.org/job/HBase-0.98-matrix/410/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev 09b89eedb585517d734bf3af3875e0e1ff5359e0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573577#comment-15573577 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1796 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1796/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev ba6e7ddd31b3fe6b965cb04a8033185382459b7c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573509#comment-15573509 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-1.1-JDK8 #1880 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1880/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev ba6e7ddd31b3fe6b965cb04a8033185382459b7c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572876#comment-15572876 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #42 (See [https://builds.apache.org/job/HBase-1.2-JDK8/42/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev bcc74e5ee5fa113d1815077f276ed61cb704b99b) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572737#comment-15572737 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #44 (See [https://builds.apache.org/job/HBase-1.3-JDK8/44/]) HBASE-16807 RegionServer will fail to report new active Hmaster until (chenheng: rev e3ded33a52574d532780594e85be8fddb6259671) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572735#comment-15572735 ] Hudson commented on HBASE-16807: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #47 (See [https://builds.apache.org/job/HBase-1.2-JDK7/47/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev bcc74e5ee5fa113d1815077f276ed61cb704b99b) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572709#comment-15572709 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-1.4 #467 (See [https://builds.apache.org/job/HBase-1.4/467/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev 27398ea525be88f9d8089071e0d7f212f33c2ca8) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572588#comment-15572588 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #39 (See [https://builds.apache.org/job/HBase-1.3-JDK7/39/]) HBASE-16807 RegionServer will fail to report new active Hmaster until (chenheng: rev e3ded33a52574d532780594e85be8fddb6259671) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572257#comment-15572257 ] Hudson commented on HBASE-16807: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1779 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1779/]) HBASE-16807, RegionServer will fail to report new active Hmaster until (chenheng: rev 90d83d5b38beb4ce35db0d7d75eb6fcef477fbf7) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572248#comment-15572248 ] Pankaj Kumar commented on HBASE-16807: -- Thanks @Heng Chen. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572129#comment-15572129 ] Heng Chen commented on HBASE-16807: --- commit to all branches > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch, > HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572040#comment-15572040 ] Pankaj Kumar commented on HBASE-16807: -- Yeah, better to have this fix in all the branches. will upload patch for b-1.1 & b-1.2. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.24 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572030#comment-15572030 ] Heng Chen commented on HBASE-16807: --- Will you upload patch for branch-1.1 and branch-1.2 ? > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572025#comment-15572025 ] Heng Chen commented on HBASE-16807: --- That's great you have patch for branch-1+, let me commit it. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.3.patch, > HBASE-16807-branch-1.patch, HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571528#comment-15571528 ] Heng Chen commented on HBASE-16807: --- push to master. Thanks all the guys! > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571080#comment-15571080 ] Pankaj Kumar commented on HBASE-16807: -- Thanks [~ashish singhi] for reviewing the patch. Test case failed because of " java.lang.OutOfMemoryError: Java heap space". I am observing OOM problem is multiple JIRAs, we need to look on this it may prevent test case execution. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571018#comment-15571018 ] Ashish Singhi commented on HBASE-16807: --- Test failures doesn't seems to be related to this patch. +1 > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570911#comment-15570911 ] Pankaj Kumar commented on HBASE-16807: -- Thanks [~chenheng] for reviewing the patch. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570885#comment-15570885 ] Hadoop QA commented on HBASE-16807: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 17s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 31m 20s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 15s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 131m 49s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries | | | org.apache.hadoop.hbase.snapshot.TestExportSnapshot | | | org.apache.hadoop.hbase.client.TestHCM | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12833026/HBASE-16807.patch | | JIRA Issue | HBASE-16807 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 295de781389b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 92ef234 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3977/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/3977/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/3977/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/3977/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > RegionServer will fail to report new acti
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570846#comment-15570846 ] Heng Chen commented on HBASE-16807: --- Got it. +1 for it. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570844#comment-15570844 ] Pankaj Kumar commented on HBASE-16807: -- On ServiceException, instead of using cached data it will refresh the master address znode from ZK and create the RS stub. > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570829#comment-15570829 ] Heng Chen commented on HBASE-16807: --- You patch seems just skip the cache without consider ServiceException, right? > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker will always return old active HM detail to Region server > on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570663#comment-15570663 ] Ted Yu commented on HBASE-16807: lgtm > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 2.0.0 > > Attachments: HBASE-16807.patch > > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker > will always return old active HM detail to Region server on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567756#comment-15567756 ] Pankaj Kumar commented on HBASE-16807: -- Thanks [~stack], will upload the patch soon. [~chenheng], In my opinion we just need to refresh the cached data on ServiceException which will occur while sending report to active master. WDYT? > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker > will always return old active HM detail to Region server on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567648#comment-15567648 ] Heng Chen commented on HBASE-16807: --- Just skip the cache is OK here? > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker > will always return old active HM detail to Region server on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[ https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15566740#comment-15566740 ] stack commented on HBASE-16807: --- Nice analysis. Have a patch [~pankaj_kumar]? > RegionServer will fail to report new active Hmaster until > HMaster/RegionServer failover > --- > > Key: HBASE-16807 > URL: https://issues.apache.org/jira/browse/HBASE-16807 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > > It's little weird, but it happened in the product environment that few > RegionServer missed master znode create notification on master failover. In > that case ZooKeeperNodeTracker will not refresh the cached data and > MasterAddressTracker > will always return old active HM detail to Region server on ServiceException. > Though We create region server stub on failure but without refreshing the > MasterAddressTracker data. > In HRegionServer.createRegionServerStatusStub() > {code} > boolean refresh = false; // for the first time, use cached data > RegionServerStatusService.BlockingInterface intf = null; > boolean interrupted = false; > try { > while (keepLooping()) { > sn = this.masterAddressTracker.getMasterAddress(refresh); > if (sn == null) { > if (!keepLooping()) { > // give up with no connection. > LOG.debug("No master found and cluster is stopped; bailing out"); > return null; > } > if (System.currentTimeMillis() > (previousLogTime + 1000)) { > LOG.debug("No master found; retry"); > previousLogTime = System.currentTimeMillis(); > } > refresh = true; // let's try pull it from ZK directly > if (sleep(200)) { > interrupted = true; > } > continue; > } > {code} > Here we refresh node only when 'sn' is NULL otherwise it will use same cached > data. > So in above case RegionServer will never report active HMaster successfully > until HMaster failover or RegionServer restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)