Attached region server log again.
Ted,
Corresponding log for master is:
java.io.IOException: Couldn't setup connection for hbase/
sfdvgctmn001.gid.gap....@sfdvgct.com to hbase/sfdvgctsn001.gid.gap.com@S
FDVGCT.COM
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
at
org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
at
org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
GSS initiate failed
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> bq. Attached both files
>
> I only found hbase-site.xml in the attachment.
>
> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murum...@gmail.com>
> wrote:
>
> > Attached both files.
> >
> >
> >
> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <olorinb...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Traces (especially one for region server) look a bit incomplete, did
> >> you copy them fully?
> >>
> >> Also may help if you post relevant pieces of hbase-site.xml (with
> >> security configs).
> >>
> >> Thanks,
> >> Mikhail
> >>
> >>
> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> >> <manoj.murum...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > We have a secured cluster. All components are working well, except
> >> hbase.
> >> > Specifically, this is what I see on regionserver:
> >> >
> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Kerberos principal name is hbase/
> >> > sfdvgctsn001.xx...@sfdvgct.com
> >> >
> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >> >
> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Have read input token of size 627 for processing by
> >> > saslServer.evaluateResponse()
> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> >> > reading:GSS initiate fail
> >> >
> >> >
> >> > and on the master:
> >> >
> >> >
> >> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
> >> > xx...@sfdvgct.com to hbase/sfdvgctsn001.xx...@sfdvgct.com
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >> >
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> >
> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >> >
> >> > All the keytabs are set properly. Has anyone seen this before?
> >> Appreciate
> >> > the time.
> >> >
> >> > Manoj
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Michael Antonov
> >>
> >
> >
>
