Looks like region server log didn't go through. Can you use pastebin ?
On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar <manoj.murum...@gmail.com> wrote: > Attached region server log again. > > Ted, > > Corresponding log for master is: > > java.io.IOException: Couldn't setup connection for hbase/ > sfdvgctmn001.gid.gap....@sfdvgct.com to hbase/sfdvgctsn001.gid.gap.com@S > > FDVGCT.COM > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) > > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) > > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) > > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176) > > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670) > > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004) > > at > org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119) > > at > org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743) > > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: > org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): > GSS initiate failed > > at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151) > > at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) > > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> bq. Attached both files >> >> I only found hbase-site.xml in the attachment. >> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murum...@gmail.com >> > >> wrote: >> >> > Attached both files. >> > >> > >> > >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <olorinb...@gmail.com> >> > wrote: >> > >> >> Hi, >> >> >> >> Traces (especially one for region server) look a bit incomplete, did >> >> you copy them fully? >> >> >> >> Also may help if you post relevant pieces of hbase-site.xml (with >> >> security configs). >> >> >> >> Thanks, >> >> Mikhail >> >> >> >> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar >> >> <manoj.murum...@gmail.com> wrote: >> >> > Hi, >> >> > >> >> > We have a secured cluster. All components are working well, except >> >> hbase. >> >> > Specifically, this is what I see on regionserver: >> >> > >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: Kerberos principal name is hbase/ >> >> > sfdvgctsn001.xx...@sfdvgct.com >> >> > >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI >> >> > >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: Have read input token of size 627 for processing by >> >> > saslServer.evaluateResponse() >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while >> >> > reading:GSS initiate fail >> >> > >> >> > >> >> > and on the master: >> >> > >> >> > >> >> > java.io.IOException: Couldn't setup connection for >> hbase/sfdvgctmn004. >> >> > xx...@sfdvgct.com to hbase/sfdvgctsn001.xx...@sfdvgct.com >> >> > >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) >> >> > >> >> > at java.security.AccessController.doPrivileged(Native Method) >> >> > >> >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> >> > >> >> > at >> >> > >> >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> >> > >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) >> >> > >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) >> >> > >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) >> >> > >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) >> >> > >> >> > All the keytabs are set properly. Has anyone seen this before? >> >> Appreciate >> >> > the time. >> >> > >> >> > Manoj >> >> >> >> >> >> >> >> -- >> >> Thanks, >> >> Michael Antonov >> >> >> > >> > >> > >
