I did that, to no avail. I am surprised by level of verbosity (lack thereof) too, something new you learn every day :) On Mar 16, 2015 3:44 PM, "Mikhail Antonov" <olorinb...@gmail.com> wrote:
> Glad it helped! > > In general though, in this cases, if you turn on java kerberos debug > logging (via something like export > HADOOP_OPTS="-Dsun.security.krb5.debug=true"), in your log you would > see error like: "unsupported key type found the default TGT: 18" > > (which means encryption type 18 per kerberos specification, i.e. > aes256-cts-hmac-sha1-96, isn't supported by Java - due to lack of > unlimited strength JCE) > > -Mikhail > > On Mon, Mar 16, 2015 at 3:19 PM, Manoj Murumkar > <manoj.murum...@gmail.com> wrote: > > Thanks Mikhail! It turned out not having unlimited strength JCE files > > everywhere on the cluster was root cause (I reproduced it by leaving > > default files on regionservers on the newly built cluster). > > > > On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar < > manoj.murum...@gmail.com> > > wrote: > > > >> Thanks. We took care of above items (all principals were generated using > >> Ambari tool) although I am not sure about 3rd item. One of my coworkers > >> decided to reinstall HBase, so I am waiting for the cluster to come up > to > >> see if we still have the issue. Funny thing is, everything else worked > in > >> secure environment (MR, Hive), just not HBase. The error message didn't > >> mention any details why the GSSException was raised (sample below): > >> > >> > >> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] > >> ipc.RpcServer: Kerberos principal name is hbase/ > >> sfdvgctsn001.xx...@sfdvgct.com > >> > >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > >> ipc.RpcServer: Created SASL server with mechanism = GSSAPI > >> > >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > >> ipc.RpcServer: Have read input token of size 627 for processing by > >> saslServer.evaluateResponse() > >> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] > >> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while > >> reading:GSS initiate fail* > >> > >> > >> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <olorinb...@gmail.com> > >> wrote: > >> > >>> I don't see region server log file in attachment for some reason.. May > >>> be the file is too big and gets rejected by mail server? Without it > >>> it'd be hard to say. Also master log looks a bit incomplete, are there > >>> no more traces? "GSS initiate fail" is normally followed by something > >>> like (just for example) - "[Caused by GSSException: No valid > >>> credentials provided (Mechanism level: Failed to find any Kerberos > >>> tgt)];" or so. Nothing like that? > >>> > >>> But in general, GSS initialization most often fails for following > reasons: > >>> > >>> - wrong linux path or file permissions on keytab file > >>> - typo in kerberos principal/realm (that includes lower/upper case > >>> differences) > >>> - JCE unlimited strength file aren't installed > >>> > >>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar > >>> <manoj.murum...@gmail.com> wrote: > >>> > Attached region server log again. > >>> > > >>> > Ted, > >>> > > >>> > Corresponding log for master is: > >>> > > >>> > java.io.IOException: Couldn't setup connection for > >>> > hbase/sfdvgctmn001.gid.gap....@sfdvgct.com to > >>> > hbase/sfdvgctsn001.gid.gap.com@S > >>> > > >>> > FDVGCT.COM > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) > >>> > > >>> > at java.security.AccessController.doPrivileged(Native Method) > >>> > > >>> > at javax.security.auth.Subject.doAs(Subject.java:415) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) > >>> > > >>> > at > >>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743) > >>> > > >>> > at > >>> > > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > >>> > > >>> > at > >>> > > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >>> > > >>> > at > >>> > > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >>> > > >>> > at java.lang.Thread.run(Thread.java:745) > >>> > > >>> > Caused by: > >>> > > >>> > org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): > >>> > GSS initiate failed > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) > >>> > > >>> > at java.security.AccessController.doPrivileged(Native Method) > >>> > > >>> > at javax.security.auth.Subject.doAs(Subject.java:415) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > >>> > > >>> > at > >>> > > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) > >>> > > >>> > > >>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >>> >> > >>> >> bq. Attached both files > >>> >> > >>> >> I only found hbase-site.xml in the attachment. > >>> >> > >>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar < > >>> manoj.murum...@gmail.com> > >>> >> wrote: > >>> >> > >>> >> > Attached both files. > >>> >> > > >>> >> > > >>> >> > > >>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov < > >>> olorinb...@gmail.com> > >>> >> > wrote: > >>> >> > > >>> >> >> Hi, > >>> >> >> > >>> >> >> Traces (especially one for region server) look a bit incomplete, > did > >>> >> >> you copy them fully? > >>> >> >> > >>> >> >> Also may help if you post relevant pieces of hbase-site.xml (with > >>> >> >> security configs). > >>> >> >> > >>> >> >> Thanks, > >>> >> >> Mikhail > >>> >> >> > >>> >> >> > >>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar > >>> >> >> <manoj.murum...@gmail.com> wrote: > >>> >> >> > Hi, > >>> >> >> > > >>> >> >> > We have a secured cluster. All components are working well, > except > >>> >> >> hbase. > >>> >> >> > Specifically, this is what I see on regionserver: > >>> >> >> > > >>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] > >>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/ > >>> >> >> > sfdvgctsn001.xx...@sfdvgct.com > >>> >> >> > > >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > >>> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI > >>> >> >> > > >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] > >>> >> >> > ipc.RpcServer: Have read input token of size 627 for > processing by > >>> >> >> > saslServer.evaluateResponse() > >>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] > >>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception > >>> while > >>> >> >> > reading:GSS initiate fail > >>> >> >> > > >>> >> >> > > >>> >> >> > and on the master: > >>> >> >> > > >>> >> >> > > >>> >> >> > java.io.IOException: Couldn't setup connection for > >>> >> >> > hbase/sfdvgctmn004. > >>> >> >> > xx...@sfdvgct.com to hbase/sfdvgctsn001.xx...@sfdvgct.com > >>> >> >> > > >>> >> >> > at > >>> >> >> > > >>> >> >> > >>> >> >> > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) > >>> >> >> > > >>> >> >> > at java.security.AccessController.doPrivileged(Native > >>> Method) > >>> >> >> > > >>> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415) > >>> >> >> > > >>> >> >> > at > >>> >> >> > > >>> >> >> > >>> >> >> > >>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > >>> >> >> > > >>> >> >> > at > >>> >> >> > > >>> >> >> > >>> >> >> > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) > >>> >> >> > > >>> >> >> > at > >>> >> >> > > >>> >> >> > >>> >> >> > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) > >>> >> >> > > >>> >> >> > at > >>> >> >> > > >>> >> >> > >>> >> >> > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) > >>> >> >> > > >>> >> >> > at > >>> >> >> > > >>> >> >> > >>> >> >> > >>> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) > >>> >> >> > > >>> >> >> > All the keytabs are set properly. Has anyone seen this before? > >>> >> >> Appreciate > >>> >> >> > the time. > >>> >> >> > > >>> >> >> > Manoj > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Thanks, > >>> >> >> Michael Antonov > >>> >> >> > >>> >> > > >>> >> > > >>> > > >>> > > >>> > >>> > >>> > >>> -- > >>> Thanks, > >>> Michael Antonov > >>> > >> > >> > > > > -- > Thanks, > Michael Antonov >
