Hi Irfan, this is very helpful, appreciate you responding with your findings. It definitely will help folks in the future!
Could you explain, how did you debug this issue and identify the correct changes? I'm interested to know how you were able to track this one down. Patrick On Thu, Aug 18, 2016 at 9:48 AM, Irfan Hamid <iha...@salesforce.com> wrote: > Thanks Patrick, > > Actually it wasn't a DNS issue but I believe this is something that others > in the community could find useful since the issue had to do with default > realms domain-to-realm conversions and krb5.conf issues. > > Our setup has multiple domains (one for prod and one for internal testing > environments). I'm not a security expert so my terminology may be off. The > TGT is of the format krbtgt/REALM@REALM. In our krb5.conf we had: > > [domain_realms] > .a.b.c = A.B.C > a.b.c = A.B.C > .b.c = B.C > b.c = B.C > > All our machine names end in b.c, and only some end in a.b.c (the client > machines end in b.c). So the first realm that was being selected for the > krbtgt was B.C and the second was from [libdefaults] default_realm which > was A.B.C. I replaced the standard krb5.conf with a custom one that had: > > [domain_realms] > .b.c = A.B.C > b.c = A.B.C > > With this change and pointing to this file via jvm arg > java.security.krb5.conf everything worked fine. > > Thanks, > Irfan. > > On Wed, Aug 17, 2016 at 7:56 PM, Patrick Hunt <ph...@apache.org> wrote: > > > I'm not an expert but I believe this indicates an environmental issue, > see > > "service ticket not found in the subject" here: > > https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/ > > errors.html > > > > Patrick > > > > > > On Tue, Aug 16, 2016 at 12:11 PM, Irfan Hamid <iha...@salesforce.com> > > wrote: > > > > > Hi, > > > > > > I'm very certain this is due to an error in my jaas or krb5 conf files > > but > > > I can't seem to figure out where. My jaas.conf looks something like > this: > > > > > > Client { > > > com.sun.security.auth.module.Krb5LoginModule required > > > useKeyTab=true > > > keyTab="/etc/zkcli_app_eng.keytab" > > > storeKey=true > > > useTicketCache=false > > > debug=true > > > principal="zk...@ay.bee.see.net"; > > > }; > > > > > > But when ZK starts up it sends a TGT request as krbtgt/ > > > bee.see....@ay.bee.see.net, i.e., it shaves off the AY from the realm > > > name ( > > > BEE.SEE.NET is a valid realm in our setup but not the one I want to > > use). > > > I > > > see the following log lines: > > > msgType is 30 > > > sname is krbtgt/BEE.SEE.NET > > > realm is AY.BEE.SEE.NET > > > cname is zkcli > > > crealm is AY.BEE.SEE.NET > > > error Message is Server not found in Kerberos database > > > error code is 7 > > > suSec is 157006 > > > sTime is Tue Aug 16 19:00:48 GMT 2016 1471374048000 > > > cTime is Fri Sep 30 18:19:26 GMT 2016 1475259566000 > > > > > > > > > And a little earlier there was: > > > Credentials acquireServiceCreds: main loop: [0] tempService=krbtgt/ > > > bee.see....@ay.bee.see.net > > > Realm parseCapaths: no cfg entry > > > Realm doInitialParse: cRealm=[AY.BEE.SEE.NET], sRealm=[BEE.SEE.NET] > > > Service ticket not found in the subject > > > Found ticket for zk...@ay.bee.see.net to go to krbtgt/ > > > ay.bee.see....@ay.bee.see.net expiring on Tue Aug 16 21:00:06 GMT 2016 > > > > > > I looked in the source code and the place I see that might be relevant > is > > > Login.java where we have Login.getTGT() which tries to obtain a TGT by > > > seeing if there's a ticket in the Subject of the form > krbtgt/REALM@REALM. > > > However, that part doesn't even get called since I don't even see the > log > > > line at the of the Login thread "TGT refresh thread started". > > > > > > Any help would be much appreciated. > > > > > > Thanks, > > > Irfan. > > > > > >