Re: Kerberos exception

2015-03-16 Thread Manoj Murumkar
Thanks Mikhail! It turned out not having unlimited strength JCE files everywhere on the cluster was root cause (I reproduced it by leaving default files on regionservers on the newly built cluster). On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar manoj.murum...@gmail.com wrote: Thanks. We took

Re: Kerberos exception

2015-03-16 Thread Mikhail Antonov
Glad it helped! In general though, in this cases, if you turn on java kerberos debug logging (via something like export HADOOP_OPTS=-Dsun.security.krb5.debug=true), in your log you would see error like: unsupported key type found the default TGT: 18 (which means encryption type 18 per kerberos

Re: Kerberos exception

2015-03-16 Thread Manoj Murumkar
I did that, to no avail. I am surprised by level of verbosity (lack thereof) too, something new you learn every day :) On Mar 16, 2015 3:44 PM, Mikhail Antonov olorinb...@gmail.com wrote: Glad it helped! In general though, in this cases, if you turn on java kerberos debug logging (via

Re: Kerberos exception

2015-03-15 Thread Ted Yu
bq. Attached both files I only found hbase-site.xml in the attachment. On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar manoj.murum...@gmail.com wrote: Attached both files. On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov olorinb...@gmail.com wrote: Hi, Traces (especially one for

Re: Kerberos exception

2015-03-15 Thread Manoj Murumkar
Attached both files. On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov olorinb...@gmail.com wrote: Hi, Traces (especially one for region server) look a bit incomplete, did you copy them fully? Also may help if you post relevant pieces of hbase-site.xml (with security configs). Thanks,

Re: Kerberos exception

2015-03-15 Thread Manoj Murumkar
Btw, ignore ganglia exceptions. We don't have that configured. On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar manoj.murum...@gmail.com wrote: Attached both files. On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov olorinb...@gmail.com wrote: Hi, Traces (especially one for region server)

Re: Kerberos exception

2015-03-15 Thread Manoj Murumkar
Attached region server log again. Ted, Corresponding log for master is: java.io.IOException: Couldn't setup connection for hbase/ sfdvgctmn001.gid.gap@sfdvgct.com to hbase/sfdvgctsn001.gid.gap.com@S FDVGCT.COM at

Re: Kerberos exception

2015-03-15 Thread Ted Yu
Looks like region server log didn't go through. Can you use pastebin ? On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar manoj.murum...@gmail.com wrote: Attached region server log again. Ted, Corresponding log for master is: java.io.IOException: Couldn't setup connection for hbase/

Re: Kerberos exception

2015-03-15 Thread Mikhail Antonov
I don't see region server log file in attachment for some reason.. May be the file is too big and gets rejected by mail server? Without it it'd be hard to say. Also master log looks a bit incomplete, are there no more traces? GSS initiate fail is normally followed by something like (just for

Re: Kerberos exception

2015-03-15 Thread Manoj Murumkar
Thanks. We took care of above items (all principals were generated using Ambari tool) although I am not sure about 3rd item. One of my coworkers decided to reinstall HBase, so I am waiting for the cluster to come up to see if we still have the issue. Funny thing is, everything else worked in

Kerberos exception

2015-03-14 Thread Manoj Murumkar
Hi, We have a secured cluster. All components are working well, except hbase. Specifically, this is what I see on regionserver: 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] ipc.RpcServer: Kerberos principal name is hbase/ sfdvgctsn001.xx...@sfdvgct.com 2015-03-14 02:16:11,658

Re: Kerberos exception

2015-03-14 Thread Mikhail Antonov
Hi, Traces (especially one for region server) look a bit incomplete, did you copy them fully? Also may help if you post relevant pieces of hbase-site.xml (with security configs). Thanks, Mikhail On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar manoj.murum...@gmail.com wrote: Hi, We have a