Thanks Mikhail! It turned out not having unlimited strength JCE files everywhere on the cluster was root cause (I reproduced it by leaving default files on regionservers on the newly built cluster).
On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <manoj.murum...@gmail.com> wrote: > Thanks. We took care of above items (all principals were generated using > Ambari tool) although I am not sure about 3rd item. One of my coworkers > decided to reinstall HBase, so I am waiting for the cluster to come up to > see if we still have the issue. Funny thing is, everything else worked in > secure environment (MR, Hive), just not HBase. The error message didn't > mention any details why the GSSException was raised (sample below): > > > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] > ipc.RpcServer: Kerberos principal name is hbase/ > sfdvgctsn001.xx...@sfdvgct.com > > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > ipc.RpcServer: Created SASL server with mechanism = GSSAPI > > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > ipc.RpcServer: Have read input token of size 627 for processing by > saslServer.evaluateResponse() > *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while > reading:GSS initiate fail* > > > On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <olorinb...@gmail.com> > wrote: > >> I don't see region server log file in attachment for some reason.. May >> be the file is too big and gets rejected by mail server? Without it >> it'd be hard to say. Also master log looks a bit incomplete, are there >> no more traces? "GSS initiate fail" is normally followed by something >> like (just for example) - "[Caused by GSSException: No valid >> credentials provided (Mechanism level: Failed to find any Kerberos >> tgt)];" or so. Nothing like that? >> >> But in general, GSS initialization most often fails for following reasons: >> >> - wrong linux path or file permissions on keytab file >> - typo in kerberos principal/realm (that includes lower/upper case >> differences) >> - JCE unlimited strength file aren't installed >> >> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar >> <manoj.murum...@gmail.com> wrote: >> > Attached region server log again. >> > >> > Ted, >> > >> > Corresponding log for master is: >> > >> > java.io.IOException: Couldn't setup connection for >> > hbase/sfdvgctmn001.gid.gap....@sfdvgct.com to >> > hbase/sfdvgctsn001.gid.gap.com@S >> > >> > FDVGCT.COM >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) >> > >> > at java.security.AccessController.doPrivileged(Native Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> > >> > at >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) >> > >> > at >> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) >> > >> > at >> > >> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176) >> > >> > at >> > >> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670) >> > >> > at >> > >> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004) >> > >> > at >> > >> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119) >> > >> > at >> > >> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743) >> > >> > at >> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) >> > >> > at >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > >> > at java.lang.Thread.run(Thread.java:745) >> > >> > Caused by: >> > >> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): >> > GSS initiate failed >> > >> > at >> > >> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151) >> > >> > at >> > >> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) >> > >> > at java.security.AccessController.doPrivileged(Native Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> > >> > at >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> > >> > at >> > >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) >> > >> > >> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> >> >> bq. Attached both files >> >> >> >> I only found hbase-site.xml in the attachment. >> >> >> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar < >> manoj.murum...@gmail.com> >> >> wrote: >> >> >> >> > Attached both files. >> >> > >> >> > >> >> > >> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov < >> olorinb...@gmail.com> >> >> > wrote: >> >> > >> >> >> Hi, >> >> >> >> >> >> Traces (especially one for region server) look a bit incomplete, did >> >> >> you copy them fully? >> >> >> >> >> >> Also may help if you post relevant pieces of hbase-site.xml (with >> >> >> security configs). >> >> >> >> >> >> Thanks, >> >> >> Mikhail >> >> >> >> >> >> >> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar >> >> >> <manoj.murum...@gmail.com> wrote: >> >> >> > Hi, >> >> >> > >> >> >> > We have a secured cluster. All components are working well, except >> >> >> hbase. >> >> >> > Specifically, this is what I see on regionserver: >> >> >> > >> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] >> >> >> > ipc.RpcServer: Kerberos principal name is hbase/ >> >> >> > sfdvgctsn001.xx...@sfdvgct.com >> >> >> > >> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI >> >> >> > >> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> >> >> > ipc.RpcServer: Have read input token of size 627 for processing by >> >> >> > saslServer.evaluateResponse() >> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] >> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception >> while >> >> >> > reading:GSS initiate fail >> >> >> > >> >> >> > >> >> >> > and on the master: >> >> >> > >> >> >> > >> >> >> > java.io.IOException: Couldn't setup connection for >> >> >> > hbase/sfdvgctmn004. >> >> >> > xx...@sfdvgct.com to hbase/sfdvgctsn001.xx...@sfdvgct.com >> >> >> > >> >> >> > at >> >> >> > >> >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) >> >> >> > >> >> >> > at java.security.AccessController.doPrivileged(Native >> Method) >> >> >> > >> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> >> >> > >> >> >> > at >> >> >> > >> >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> >> >> > >> >> >> > at >> >> >> > >> >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) >> >> >> > >> >> >> > at >> >> >> > >> >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) >> >> >> > >> >> >> > at >> >> >> > >> >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) >> >> >> > >> >> >> > at >> >> >> > >> >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) >> >> >> > >> >> >> > All the keytabs are set properly. Has anyone seen this before? >> >> >> Appreciate >> >> >> > the time. >> >> >> > >> >> >> > Manoj >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Thanks, >> >> >> Michael Antonov >> >> >> >> >> > >> >> > >> > >> > >> >> >> >> -- >> Thanks, >> Michael Antonov >> > >
