I don't see region server log file in attachment for some reason.. May be the file is too big and gets rejected by mail server? Without it it'd be hard to say. Also master log looks a bit incomplete, are there no more traces? "GSS initiate fail" is normally followed by something like (just for example) - "[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)];" or so. Nothing like that?
But in general, GSS initialization most often fails for following reasons: - wrong linux path or file permissions on keytab file - typo in kerberos principal/realm (that includes lower/upper case differences) - JCE unlimited strength file aren't installed On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar <manoj.murum...@gmail.com> wrote: > Attached region server log again. > > Ted, > > Corresponding log for master is: > > java.io.IOException: Couldn't setup connection for > hbase/sfdvgctmn001.gid.gap....@sfdvgct.com to > hbase/sfdvgctsn001.gid.gap.com@S > > FDVGCT.COM > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) > > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) > > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) > > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176) > > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004) > > at > org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119) > > at > org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743) > > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: > org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): > GSS initiate failed > > at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151) > > at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) > > > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> bq. Attached both files >> >> I only found hbase-site.xml in the attachment. >> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murum...@gmail.com> >> wrote: >> >> > Attached both files. >> > >> > >> > >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <olorinb...@gmail.com> >> > wrote: >> > >> >> Hi, >> >> >> >> Traces (especially one for region server) look a bit incomplete, did >> >> you copy them fully? >> >> >> >> Also may help if you post relevant pieces of hbase-site.xml (with >> >> security configs). >> >> >> >> Thanks, >> >> Mikhail >> >> >> >> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar >> >> <manoj.murum...@gmail.com> wrote: >> >> > Hi, >> >> > >> >> > We have a secured cluster. All components are working well, except >> >> hbase. >> >> > Specifically, this is what I see on regionserver: >> >> > >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: Kerberos principal name is hbase/ >> >> > sfdvgctsn001.xx...@sfdvgct.com >> >> > >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI >> >> > >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: Have read input token of size 627 for processing by >> >> > saslServer.evaluateResponse() >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while >> >> > reading:GSS initiate fail >> >> > >> >> > >> >> > and on the master: >> >> > >> >> > >> >> > java.io.IOException: Couldn't setup connection for >> >> > hbase/sfdvgctmn004. >> >> > xx...@sfdvgct.com to hbase/sfdvgctsn001.xx...@sfdvgct.com >> >> > >> >> > at >> >> > >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) >> >> > >> >> > at java.security.AccessController.doPrivileged(Native Method) >> >> > >> >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> >> > >> >> > at >> >> > >> >> >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> >> > >> >> > at >> >> > >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) >> >> > >> >> > at >> >> > >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) >> >> > >> >> > at >> >> > >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) >> >> > >> >> > at >> >> > >> >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) >> >> > >> >> > All the keytabs are set properly. Has anyone seen this before? >> >> Appreciate >> >> > the time. >> >> > >> >> > Manoj >> >> >> >> >> >> >> >> -- >> >> Thanks, >> >> Michael Antonov >> >> >> > >> > > > -- Thanks, Michael Antonov
