Hi All, we seem to have overcome this occasional exception by changing HTable from being called through ConnectionManager vs. directly instantiating HTable.
OLD BAD: connection = HConnectionManager.createConnection(config); protected HTableInterface getHTable(String tableName) throws IOException { //Connection might have been closed since we acquired it...create a //new one if this is the case if (connection == null || connection.isClosed()) { log.info("HConnection null or closed...reopening"); connection = HConnectionManager.createConnection(config); } return connection.getTable(tableName); } NEW GOOD: public HTableInterface getHTable(byte[] tableName) throws IOException { return new HTable(conf, tableName); } Comment on this link [0] is why they opted down the initial - BAD path: ". Instances of this class can be constructed directly but it is encouraged that users get instances via HConnection and HConnectionManager . See HConnectionManager class comment for an example. " [0] http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html We run the workload for a few days and it seems fine. We are not sure why this occurs but somebody listens this thread may have a better idea. Thanks for all your help! Best regards, Wei --------------------------------- Wei Tan, PhD Research Staff Member IBM T. J. Watson Research Center http://researcher.ibm.com/person/us-wtan From: Zesheng Wu <wuzeshen...@gmail.com> To: user@hbase.apache.org, Date: 03/13/2014 09:33 PM Subject: Re: Occasional GSSException that brings down region server Thank you all the same :) 2014-03-13 22:50 GMT+08:00 Andrew Purtell <apurt...@apache.org>: > Then nothing immediately comes to mind, sorry. > > On Wednesday, March 12, 2014, Zesheng Wu <wuzeshen...@gmail.com> wrote: > > > Hi Andrew, > > > > We use keytab file for hbase, there's no ticket cache file under /tmp, > but > > we still encounter the same error as above, how about this? > > > > > > 2014-03-12 7:37 GMT+08:00 Andrew Purtell <apurt...@apache.org > <javascript:;> > > >: > > > > > If you might have more than one security enabled Java process running > > under > > > the same UNIX user on the servers, then this and other weirdness can > > happen > > > because they share the same ticket cache file in /tmp. Someone does a > > > (re)login and another process concurrently accessing the cache gets > > garbage > > > or unexpected state. > > > > > > > > > On Mon, Mar 10, 2014 at 2:26 PM, Wei Tan <w...@us.ibm.com< javascript:;>> > > wrote: > > > > > > > Hi, > > > > > > > > We are running a HBase cluster in these settings and with kerberos > > > > enabled. > > > > HBase: 0.96.1.1 > > > > Zookeeper: 3.4.5 > > > > Hadoop: 1.1.1 > > > > > > > > > > > > We constantly put data into HBase and every several hours we get the > > > error > > > > below on a random region server; this error arises and the region > > server > > > > kills itself. > > > > > > > > ERROR: > > > > 2014-02-28 09:32:39,755 ERROR > > > [hconnection-0x116987ad-shared--pool1378-t9] > > > > security.UserGroupInformation: PriviledgedActionException > > > > as:XXXXXXXX@DOMAIN cause:javax.security.sasl.SaslException: GSS > > initiate > > > > failed [Caused by GSSException: No valid credentials provided > > (Mechanism > > > > level: The ticket isn't for us (35) - BAD TGS SERVER NAME)] > > > > > > > > > > > > > > > > We also tried with multiple version of kdc - all the way up to latest > > > > 1.12.1 - still see this error. What is weird is that most put gets > > > > processed successfully until this error occurs and kills the RS. > > > > > > > > Thanks, > > > > Wei > > > > --------------------------------- > > > > Wei Tan, PhD > > > > Research Staff Member > > > > IBM T. J. Watson Research Center > > > > http://researcher.ibm.com/person/us-wtan > > > > > > > > > > > > > > > -- > > > Best regards, > > > > > > - Andy > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein > > > (via Tom White) > > > > > > > > > > > -- > > Best Wishes! > > > > Yours, Zesheng > > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- Best Wishes! Yours, Zesheng