Ok, thanks Irfan! Patrick
On Wed, Aug 24, 2016 at 2:58 PM, Irfan Hamid <iha...@salesforce.com> wrote: > Sure. It was not very straightforward. I ended up setting the logging level > on the client side by adding the -Dlogging.level=DEBUG and > -Dsun.security.krb5.debug=true. Additionally I set debug=true in the > jaas.conf. > > With this I got log lines similar to the following on the client: > > KrbException: Fail to create credential. (63) - No service creds > >>> Credentials acquireServiceCreds: no tgt; cannot get creds > >>> Credentials acquireServiceCreds: no tgt; searching backwards > msgType is 30 > sname is krbtgt/B.C.NET <http://app.eng.sfdc.net/> > realm is A.B.C.NET <http://APP.ENG.SFDC.NET> > cname is zkcli > crealm is A.B.C.NET <http://APP.ENG.SFDC.NET> > error Message is Server not found in Kerberos database > error code is 7 > suSec is 356619 > sTime is Wed Aug 17 18:32:34 GMT 2016 1471458754000 > cTime is Wed Feb 17 16:12:06 GMT 2010 1266423126000 > > > The Internet had very little information about all that sname, realm, cname > and crealm etc. mean. However, a grep for krbtgt on the ZK sources led me > to Login.java and this code: > > Set<KerberosTicket> tickets = > subject.getPrivateCredentials(KerberosTicket.class); > for(KerberosTicket ticket: tickets) { > KerberosPrincipal server = ticket.getServer(); > if (server.getName().equals("krbtgt/" + server.getRealm() + > "@" > + server.getRealm())) { > LOG.debug("Found tgt " + ticket + "."); > return ticket; > } > } > return null; > > Logs didn't indicate that my code had gotten to this point but it did get > me thinking that this is a problem with my jaas or my krb5.conf since the > KerberTicket objects are being retrieved from a Java API. The man page on > krb5.conf gave me the next clues. Playing around with the [domain_realm] > and default_realm etc. got me to the solution. > > Happy to help, > Irfan. > > On Wed, Aug 24, 2016 at 12:37 PM, Patrick Hunt <ph...@apache.org> wrote: > > > Hi Irfan, this is very helpful, appreciate you responding with your > > findings. It definitely will help folks in the future! > > > > Could you explain, how did you debug this issue and identify the correct > > changes? I'm interested to know how you were able to track this one down. > > > > Patrick > > > > > > On Thu, Aug 18, 2016 at 9:48 AM, Irfan Hamid <iha...@salesforce.com> > > wrote: > > > > > Thanks Patrick, > > > > > > Actually it wasn't a DNS issue but I believe this is something that > > others > > > in the community could find useful since the issue had to do with > default > > > realms domain-to-realm conversions and krb5.conf issues. > > > > > > Our setup has multiple domains (one for prod and one for internal > testing > > > environments). I'm not a security expert so my terminology may be off. > > The > > > TGT is of the format krbtgt/REALM@REALM. In our krb5.conf we had: > > > > > > [domain_realms] > > > .a.b.c = A.B.C > > > a.b.c = A.B.C > > > .b.c = B.C > > > b.c = B.C > > > > > > All our machine names end in b.c, and only some end in a.b.c (the > client > > > machines end in b.c). So the first realm that was being selected for > the > > > krbtgt was B.C and the second was from [libdefaults] default_realm > which > > > was A.B.C. I replaced the standard krb5.conf with a custom one that > had: > > > > > > [domain_realms] > > > .b.c = A.B.C > > > b.c = A.B.C > > > > > > With this change and pointing to this file via jvm arg > > > java.security.krb5.conf everything worked fine. > > > > > > Thanks, > > > Irfan. > > > > > > On Wed, Aug 17, 2016 at 7:56 PM, Patrick Hunt <ph...@apache.org> > wrote: > > > > > > > I'm not an expert but I believe this indicates an environmental > issue, > > > see > > > > "service ticket not found in the subject" here: > > > > https://steveloughran.gitbooks.io/kerberos_and_ > > hadoop/content/sections/ > > > > errors.html > > > > > > > > Patrick > > > > > > > > > > > > On Tue, Aug 16, 2016 at 12:11 PM, Irfan Hamid <iha...@salesforce.com > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm very certain this is due to an error in my jaas or krb5 conf > > files > > > > but > > > > > I can't seem to figure out where. My jaas.conf looks something like > > > this: > > > > > > > > > > Client { > > > > > com.sun.security.auth.module.Krb5LoginModule required > > > > > useKeyTab=true > > > > > keyTab="/etc/zkcli_app_eng.keytab" > > > > > storeKey=true > > > > > useTicketCache=false > > > > > debug=true > > > > > principal="zk...@ay.bee.see.net"; > > > > > }; > > > > > > > > > > But when ZK starts up it sends a TGT request as krbtgt/ > > > > > bee.see....@ay.bee.see.net, i.e., it shaves off the AY from the > > realm > > > > > name ( > > > > > BEE.SEE.NET is a valid realm in our setup but not the one I want > to > > > > use). > > > > > I > > > > > see the following log lines: > > > > > msgType is 30 > > > > > sname is krbtgt/BEE.SEE.NET > > > > > realm is AY.BEE.SEE.NET > > > > > cname is zkcli > > > > > crealm is AY.BEE.SEE.NET > > > > > error Message is Server not found in Kerberos database > > > > > error code is 7 > > > > > suSec is 157006 > > > > > sTime is Tue Aug 16 19:00:48 GMT 2016 1471374048000 > > > > > cTime is Fri Sep 30 18:19:26 GMT 2016 1475259566000 > > > > > > > > > > > > > > > And a little earlier there was: > > > > > Credentials acquireServiceCreds: main loop: [0] tempService=krbtgt/ > > > > > bee.see....@ay.bee.see.net > > > > > Realm parseCapaths: no cfg entry > > > > > Realm doInitialParse: cRealm=[AY.BEE.SEE.NET], sRealm=[BEE.SEE.NET > ] > > > > > Service ticket not found in the subject > > > > > Found ticket for zk...@ay.bee.see.net to go to krbtgt/ > > > > > ay.bee.see....@ay.bee.see.net expiring on Tue Aug 16 21:00:06 GMT > > 2016 > > > > > > > > > > I looked in the source code and the place I see that might be > > relevant > > > is > > > > > Login.java where we have Login.getTGT() which tries to obtain a TGT > > by > > > > > seeing if there's a ticket in the Subject of the form > > > krbtgt/REALM@REALM. > > > > > However, that part doesn't even get called since I don't even see > the > > > log > > > > > line at the of the Login thread "TGT refresh thread started". > > > > > > > > > > Any help would be much appreciated. > > > > > > > > > > Thanks, > > > > > Irfan. > > > > > > > > > > > > > > >
