Attached region server log again. Ted,
Corresponding log for master is: java.io.IOException: Couldn't setup connection for hbase/ sfdvgctmn001.gid.gap....@sfdvgct.com to hbase/sfdvgctsn001.gid.gap.com@S FDVGCT.COM at org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004) at org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119) at org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhih...@gmail.com> wrote: > bq. Attached both files > > I only found hbase-site.xml in the attachment. > > On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murum...@gmail.com> > wrote: > > > Attached both files. > > > > > > > > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <olorinb...@gmail.com> > > wrote: > > > >> Hi, > >> > >> Traces (especially one for region server) look a bit incomplete, did > >> you copy them fully? > >> > >> Also may help if you post relevant pieces of hbase-site.xml (with > >> security configs). > >> > >> Thanks, > >> Mikhail > >> > >> > >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar > >> <manoj.murum...@gmail.com> wrote: > >> > Hi, > >> > > >> > We have a secured cluster. All components are working well, except > >> hbase. > >> > Specifically, this is what I see on regionserver: > >> > > >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] > >> > ipc.RpcServer: Kerberos principal name is hbase/ > >> > sfdvgctsn001.xx...@sfdvgct.com > >> > > >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI > >> > > >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > >> > ipc.RpcServer: Have read input token of size 627 for processing by > >> > saslServer.evaluateResponse() > >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] > >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while > >> > reading:GSS initiate fail > >> > > >> > > >> > and on the master: > >> > > >> > > >> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004. > >> > xx...@sfdvgct.com to hbase/sfdvgctsn001.xx...@sfdvgct.com > >> > > >> > at > >> > > >> > org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) > >> > > >> > at java.security.AccessController.doPrivileged(Native Method) > >> > > >> > at javax.security.auth.Subject.doAs(Subject.java:415) > >> > > >> > at > >> > > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > >> > > >> > at > >> > > >> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) > >> > > >> > at > >> > > >> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) > >> > > >> > at > >> > > >> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) > >> > > >> > at > >> > > >> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) > >> > > >> > All the keytabs are set properly. Has anyone seen this before? > >> Appreciate > >> > the time. > >> > > >> > Manoj > >> > >> > >> > >> -- > >> Thanks, > >> Michael Antonov > >> > > > > >