Thanks Patrick,
Actually it wasn't a DNS issue but I believe this is something that others
in the community could find useful since the issue had to do with default
realms domain-to-realm conversions and krb5.conf issues.
Our setup has multiple domains (one for prod and one for internal testing
environments). I'm not a security expert so my terminology may be off. The
TGT is of the format krbtgt/REALM@REALM. In our krb5.conf we had:
[domain_realms]
.a.b.c = A.B.C
a.b.c = A.B.C
.b.c = B.C
b.c = B.C
All our machine names end in b.c, and only some end in a.b.c (the client
machines end in b.c). So the first realm that was being selected for the
krbtgt was B.C and the second was from [libdefaults] default_realm which
was A.B.C. I replaced the standard krb5.conf with a custom one that had:
[domain_realms]
.b.c = A.B.C
b.c = A.B.C
With this change and pointing to this file via jvm arg
java.security.krb5.conf everything worked fine.
Thanks,
Irfan.
On Wed, Aug 17, 2016 at 7:56 PM, Patrick Hunt <ph...@apache.org> wrote:
> I'm not an expert but I believe this indicates an environmental issue, see
> "service ticket not found in the subject" here:
> https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/
> errors.html
>
> Patrick
>
>
> On Tue, Aug 16, 2016 at 12:11 PM, Irfan Hamid <iha...@salesforce.com>
> wrote:
>
> > Hi,
> >
> > I'm very certain this is due to an error in my jaas or krb5 conf files
> but
> > I can't seem to figure out where. My jaas.conf looks something like this:
> >
> > Client {
> > com.sun.security.auth.module.Krb5LoginModule required
> > useKeyTab=true
> > keyTab="/etc/zkcli_app_eng.keytab"
> > storeKey=true
> > useTicketCache=false
> > debug=true
> > principal="zk...@ay.bee.see.net";
> > };
> >
> > But when ZK starts up it sends a TGT request as krbtgt/
> > bee.see....@ay.bee.see.net, i.e., it shaves off the AY from the realm
> > name (
> > BEE.SEE.NET is a valid realm in our setup but not the one I want to
> use).
> > I
> > see the following log lines:
> > msgType is 30
> > sname is krbtgt/BEE.SEE.NET
> > realm is AY.BEE.SEE.NET
> > cname is zkcli
> > crealm is AY.BEE.SEE.NET
> > error Message is Server not found in Kerberos database
> > error code is 7
> > suSec is 157006
> > sTime is Tue Aug 16 19:00:48 GMT 2016 1471374048000
> > cTime is Fri Sep 30 18:19:26 GMT 2016 1475259566000
> >
> >
> > And a little earlier there was:
> > Credentials acquireServiceCreds: main loop: [0] tempService=krbtgt/
> > bee.see....@ay.bee.see.net
> > Realm parseCapaths: no cfg entry
> > Realm doInitialParse: cRealm=[AY.BEE.SEE.NET], sRealm=[BEE.SEE.NET]
> > Service ticket not found in the subject
> > Found ticket for zk...@ay.bee.see.net to go to krbtgt/
> > ay.bee.see....@ay.bee.see.net expiring on Tue Aug 16 21:00:06 GMT 2016
> >
> > I looked in the source code and the place I see that might be relevant is
> > Login.java where we have Login.getTGT() which tries to obtain a TGT by
> > seeing if there's a ticket in the Subject of the form krbtgt/REALM@REALM.
> > However, that part doesn't even get called since I don't even see the log
> > line at the of the Login thread "TGT refresh thread started".
> >
> > Any help would be much appreciated.
> >
> > Thanks,
> > Irfan.
> >
>
