[ https://issues.apache.org/jira/browse/HBASE-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021535#comment-15021535 ]
Sumit Nigam commented on HBASE-8675: ------------------------------------ I'd like to understand that is it guaranteed to be Kerberos being unreachable issue? I have similar problem but my error message is: 15/11/15 15:46:53 ERROR client.ZooKeeperSaslClient: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Connection reset)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. 15/11/15 15:46:53 ERROR zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Connection reset)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. The mechanism level points to connection reset. Is that error being reported for kerberos server or for zookeeper client's inability to connect with zookeeper quorum? > Two active Hmasters for AUTH_FAILED in secure hbase cluster > ----------------------------------------------------------- > > Key: HBASE-8675 > URL: https://issues.apache.org/jira/browse/HBASE-8675 > Project: HBase > Issue Type: Bug > Components: master > Reporter: Liu Shaohui > Priority: Critical > Attachments: HBASE-8675-0.94-v1.patch > > > In our product cluster, because of the net problem to kerberos server, the > ZooKeeperWatcher in active hmaster fails to Auth , gets a connection Event of > AUTH_FAILED and loose the master lock. But the zookeeper watcher ignores the > event, so the old active hmaster keeps to be active. After the net problem is > fixed, the backup hmaster gets the master lock and becomes active. There are > two two active hmasters in the cluster. > 2013-05-30 09:44:21,004 ERROR > org.apache.zookeeper.client.ZooKeeperSaslClient: An error: > (java.security.PrivilegedActionException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: krb1.xiaomi.net)]) occurred when evaluating Zookeeper > Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED > state. > 2013-05-30 09:54:07,755 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: > hconnection-0x3e10d98be405bc Unable to set watcher on znode /hbase/master > org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = > AuthFailed for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:123) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:231) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:595) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:850) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:825) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:286) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:201) > at > org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) > at > org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) > at > org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705) > at > org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) > at > org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:168) > at > org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:123) > at > org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:134) > at > org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:92) > at org.apache.hadoop.hbase.Chore.run(Chore.java:67) > at java.lang.Thread.run(Thread.java:662) > I want to just abort the hmaster server if AuthFailed or SaslAuthenticated. > Any better idea about this issue? > For ZookeeperWatcher is used in many classes, will the aborting will bring > more problems? Any more problems we need consider? -- This message was sent by Atlassian JIRA (v6.3.4#6332)