Re: Failing krb5-related tests on el6

[Todd Lipcon]

On Mon, Nov 7, 2016 at 4:35 PM, Alexey Serbin  wrote:

> Ah, I see -- thank you for the clarification.
>
> BTW, could 'ignore_acceptor_hostname' parameter help there?  Like adding
> the following into the krb5.conf:
>
> [libdefaults]
>   ignore_acceptor_hostname = true
>

We've actually already got that there. But it doesn't help -- the issue is
that the client can't figure out what realm it's connecting to in krb 1.10


>
>
> Best regards,
>
> Alexey
>
> On Mon, Nov 7, 2016 at 3:42 PM, Todd Lipcon  wrote:
>
> > On Mon, Nov 7, 2016 at 3:37 PM, Alexey Serbin 
> > wrote:
> >
> > > May be, as a workaround we could run our mini_kdc only on at 127.0.0.1
> > > address.
> > >
> >
> > The issue is the actual Kudu servers, not the KDC. The KDC is on
> 127.0.0.1,
> > but we have service principals like kudu/127.2.3.4@TESTREALM that are
> > causing the issue.
> >
> > -Todd
> >
> >
> > >
> > >
> > > Best regards,
> > >
> > > Alexey
> > >
> > > On Sun, Nov 6, 2016 at 9:57 PM, Todd Lipcon  wrote:
> > >
> > > > FWIW it looks like there's already some code out there that can do
> the
> > > > appropriate "fake DNS" wrapping: https://cwrap.org/nss_wrapper.html
> > > >
> > > >
> > > > On Sun, Nov 6, 2016 at 9:13 PM, Todd Lipcon 
> wrote:
> > > >
> > > > > Hey folks
> > > > >
> > > > > I've been looking into why our kerberos-dependent tests are failing
> > on
> > > > el6
> > > > > and it looks like it will be unfortunately difficult to fix.
> > > > >
> > > > > The first issue was that krb5 1.10 (on el6) doesn't automatically
> > > create
> > > > > the directory for a DIR: type ticket cache. That one was easy
> > to
> > > > fix
> > > > > and got minikdc-test passing.
> > > > >
> > > > > The more insidious issue, though, is the following:
> > > > > In miniclusters we start daemons on weird local IP addresses
> > > (127.x.y.z)
> > > > > which don't have any corresponding domain names. So, we have
> > configured
> > > > the
> > > > > MiniKDC to disable reverse DNS, and are creating service principals
> > > kudu/
> > > > > 127.x@krbtest.com. This works great on krb5-1.12.
> > > > >
> > > > > However, there's a bug in krb5 1.10 (http://krbdev.mit.edu/rt/
> > > > > Ticket/Display.html?id=7603) which is preventing this from working
> on
> > > > > el6. When 'rdns = false', Kerberos is unable to figure out which
> > realm
> > > > the
> > > > > server is in (and doesn't fall back to the default realm). So, our
> > > > > connections are failing with: "Cannot determine realm for numeric
> > host
> > > > > address".
> > > > >
> > > > > If we change to 'rdns = true', then it will first try to reverse
> > lookup
> > > > > the weird loopback, and then when it fails, still give the same
> error
> > > > about
> > > > > numeric hosts.
> > > > >
> > > > > It's possible there's some way to explicitly set the target realm,
> > but
> > > if
> > > > > so I can't seem to find it. So, the only workarounds I can think of
> > > are:
> > > > >
> > > > > 1) Use a local build of krb5 1.12 or later on el6 for test
> purposes.
> > I
> > > > > verified that if I built krb5 1.14 and put its libraries in
> > > > > LD_LIBRARY_PATH, and modified MiniKDC to use corresponding
> binaries,
> > > the
> > > > > tests passed fine.
> > > > >
> > > > > The downside here is that we would lose test coverage of the
> library
> > > > > version actually installed on a lot of end-user systems. So another
> > > > option
> > > > > might be to test against a patched version of 1.10 (with only the
> fix
> > > for
> > > > > this numeric hostname issue).
> > > > >
> > > > > 2) Somehow monkey-patch getnameinfo() to provide reverse DNS
> entries
> > > for
> > > > > 127.x.y.z mapping to '127-x-y-z.kudu.local' or something of that
> > > nature.
> > > > > We'd have to do this via LD_PRELOAD probably, which is a bummer,
> but
> > I
> > > > > can't see any other way to override the resolver on a per-user
> basis
> > > > > ($HOSTALIASES only does forward DNS).
> > > > >
> > > > > Any other ideas?
> > > > >
> > > > > -Todd
> > > > > --
> > > > > Todd Lipcon
> > > > > Software Engineer, Cloudera
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Todd Lipcon
> > > > Software Engineer, Cloudera
> > > >
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Failing krb5-related tests on el6

[Todd Lipcon]

Here's one more workaround which might actually be simplest: it seems like
if we LD_PRELOAD (or otherwise override) krb5_get_host_realm(), we can
detect this case and return our default KRBTEST.COM realm.

I'll give that one a try and see if it works, since it's only 10-15 lines
of code vs importing new dependencies/build complexity.

-Todd

On Mon, Nov 7, 2016 at 6:49 PM, Todd Lipcon  wrote:

> On Mon, Nov 7, 2016 at 4:35 PM, Alexey Serbin 
> wrote:
>
>> Ah, I see -- thank you for the clarification.
>>
>> BTW, could 'ignore_acceptor_hostname' parameter help there?  Like adding
>> the following into the krb5.conf:
>>
>> [libdefaults]
>>   ignore_acceptor_hostname = true
>>
>
> We've actually already got that there. But it doesn't help -- the issue is
> that the client can't figure out what realm it's connecting to in krb 1.10
>
>
>>
>>
>> Best regards,
>>
>> Alexey
>>
>> On Mon, Nov 7, 2016 at 3:42 PM, Todd Lipcon  wrote:
>>
>> > On Mon, Nov 7, 2016 at 3:37 PM, Alexey Serbin 
>> > wrote:
>> >
>> > > May be, as a workaround we could run our mini_kdc only on at 127.0.0.1
>> > > address.
>> > >
>> >
>> > The issue is the actual Kudu servers, not the KDC. The KDC is on
>> 127.0.0.1,
>> > but we have service principals like kudu/127.2.3.4@TESTREALM that are
>> > causing the issue.
>> >
>> > -Todd
>> >
>> >
>> > >
>> > >
>> > > Best regards,
>> > >
>> > > Alexey
>> > >
>> > > On Sun, Nov 6, 2016 at 9:57 PM, Todd Lipcon 
>> wrote:
>> > >
>> > > > FWIW it looks like there's already some code out there that can do
>> the
>> > > > appropriate "fake DNS" wrapping: https://cwrap.org/nss_wrapper.html
>> > > >
>> > > >
>> > > > On Sun, Nov 6, 2016 at 9:13 PM, Todd Lipcon 
>> wrote:
>> > > >
>> > > > > Hey folks
>> > > > >
>> > > > > I've been looking into why our kerberos-dependent tests are
>> failing
>> > on
>> > > > el6
>> > > > > and it looks like it will be unfortunately difficult to fix.
>> > > > >
>> > > > > The first issue was that krb5 1.10 (on el6) doesn't automatically
>> > > create
>> > > > > the directory for a DIR: type ticket cache. That one was
>> easy
>> > to
>> > > > fix
>> > > > > and got minikdc-test passing.
>> > > > >
>> > > > > The more insidious issue, though, is the following:
>> > > > > In miniclusters we start daemons on weird local IP addresses
>> > > (127.x.y.z)
>> > > > > which don't have any corresponding domain names. So, we have
>> > configured
>> > > > the
>> > > > > MiniKDC to disable reverse DNS, and are creating service
>> principals
>> > > kudu/
>> > > > > 127.x@krbtest.com. This works great on krb5-1.12.
>> > > > >
>> > > > > However, there's a bug in krb5 1.10 (http://krbdev.mit.edu/rt/
>> > > > > Ticket/Display.html?id=7603) which is preventing this from
>> working on
>> > > > > el6. When 'rdns = false', Kerberos is unable to figure out which
>> > realm
>> > > > the
>> > > > > server is in (and doesn't fall back to the default realm). So, our
>> > > > > connections are failing with: "Cannot determine realm for numeric
>> > host
>> > > > > address".
>> > > > >
>> > > > > If we change to 'rdns = true', then it will first try to reverse
>> > lookup
>> > > > > the weird loopback, and then when it fails, still give the same
>> error
>> > > > about
>> > > > > numeric hosts.
>> > > > >
>> > > > > It's possible there's some way to explicitly set the target realm,
>> > but
>> > > if
>> > > > > so I can't seem to find it. So, the only workarounds I can think
>> of
>> > > are:
>> > > > >
>> > > > > 1) Use a local build of krb5 1.12 or later on el6 for test
>> purposes.
>> > I
>> > > > > verified that if I built krb5 1.14 and put its libraries in
>> > > > > LD_LIBRARY_PATH, and modified MiniKDC to use corresponding
>> binaries,
>> > > the
>> > > > > tests passed fine.
>> > > > >
>> > > > > The downside here is that we would lose test coverage of the
>> library
>> > > > > version actually installed on a lot of end-user systems. So
>> another
>> > > > option
>> > > > > might be to test against a patched version of 1.10 (with only the
>> fix
>> > > for
>> > > > > this numeric hostname issue).
>> > > > >
>> > > > > 2) Somehow monkey-patch getnameinfo() to provide reverse DNS
>> entries
>> > > for
>> > > > > 127.x.y.z mapping to '127-x-y-z.kudu.local' or something of that
>> > > nature.
>> > > > > We'd have to do this via LD_PRELOAD probably, which is a bummer,
>> but
>> > I
>> > > > > can't see any other way to override the resolver on a per-user
>> basis
>> > > > > ($HOSTALIASES only does forward DNS).
>> > > > >
>> > > > > Any other ideas?
>> > > > >
>> > > > > -Todd
>> > > > > --
>> > > > > Todd Lipcon
>> > > > > Software Engineer, Cloudera
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Todd Lipcon
>> > > > Software Engineer, Cloudera
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
>
>
>
> --
> Todd Lipcon
> 

Re: Failing krb5-related tests on el6

[Todd Lipcon]

FWIW it looks like there's already some code out there that can do the
appropriate "fake DNS" wrapping: https://cwrap.org/nss_wrapper.html


On Sun, Nov 6, 2016 at 9:13 PM, Todd Lipcon  wrote:

> Hey folks
>
> I've been looking into why our kerberos-dependent tests are failing on el6
> and it looks like it will be unfortunately difficult to fix.
>
> The first issue was that krb5 1.10 (on el6) doesn't automatically create
> the directory for a DIR: type ticket cache. That one was easy to fix
> and got minikdc-test passing.
>
> The more insidious issue, though, is the following:
> In miniclusters we start daemons on weird local IP addresses (127.x.y.z)
> which don't have any corresponding domain names. So, we have configured the
> MiniKDC to disable reverse DNS, and are creating service principals kudu/
> 127.x@krbtest.com. This works great on krb5-1.12.
>
> However, there's a bug in krb5 1.10 (http://krbdev.mit.edu/rt/
> Ticket/Display.html?id=7603) which is preventing this from working on
> el6. When 'rdns = false', Kerberos is unable to figure out which realm the
> server is in (and doesn't fall back to the default realm). So, our
> connections are failing with: "Cannot determine realm for numeric host
> address".
>
> If we change to 'rdns = true', then it will first try to reverse lookup
> the weird loopback, and then when it fails, still give the same error about
> numeric hosts.
>
> It's possible there's some way to explicitly set the target realm, but if
> so I can't seem to find it. So, the only workarounds I can think of are:
>
> 1) Use a local build of krb5 1.12 or later on el6 for test purposes. I
> verified that if I built krb5 1.14 and put its libraries in
> LD_LIBRARY_PATH, and modified MiniKDC to use corresponding binaries, the
> tests passed fine.
>
> The downside here is that we would lose test coverage of the library
> version actually installed on a lot of end-user systems. So another option
> might be to test against a patched version of 1.10 (with only the fix for
> this numeric hostname issue).
>
> 2) Somehow monkey-patch getnameinfo() to provide reverse DNS entries for
> 127.x.y.z mapping to '127-x-y-z.kudu.local' or something of that nature.
> We'd have to do this via LD_PRELOAD probably, which is a bummer, but I
> can't see any other way to override the resolver on a per-user basis
> ($HOSTALIASES only does forward DNS).
>
> Any other ideas?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Failing krb5-related tests on el6

[Todd Lipcon]

Hey folks

I've been looking into why our kerberos-dependent tests are failing on el6
and it looks like it will be unfortunately difficult to fix.

The first issue was that krb5 1.10 (on el6) doesn't automatically create
the directory for a DIR: type ticket cache. That one was easy to fix
and got minikdc-test passing.

The more insidious issue, though, is the following:
In miniclusters we start daemons on weird local IP addresses (127.x.y.z)
which don't have any corresponding domain names. So, we have configured the
MiniKDC to disable reverse DNS, and are creating service principals kudu/
127.x@krbtest.com. This works great on krb5-1.12.

However, there's a bug in krb5 1.10 (
http://krbdev.mit.edu/rt/Ticket/Display.html?id=7603) which is preventing
this from working on el6. When 'rdns = false', Kerberos is unable to figure
out which realm the server is in (and doesn't fall back to the default
realm). So, our connections are failing with: "Cannot determine realm for
numeric host address".

If we change to 'rdns = true', then it will first try to reverse lookup the
weird loopback, and then when it fails, still give the same error about
numeric hosts.

It's possible there's some way to explicitly set the target realm, but if
so I can't seem to find it. So, the only workarounds I can think of are:

1) Use a local build of krb5 1.12 or later on el6 for test purposes. I
verified that if I built krb5 1.14 and put its libraries in
LD_LIBRARY_PATH, and modified MiniKDC to use corresponding binaries, the
tests passed fine.

The downside here is that we would lose test coverage of the library
version actually installed on a lot of end-user systems. So another option
might be to test against a patched version of 1.10 (with only the fix for
this numeric hostname issue).

2) Somehow monkey-patch getnameinfo() to provide reverse DNS entries for
127.x.y.z mapping to '127-x-y-z.kudu.local' or something of that nature.
We'd have to do this via LD_PRELOAD probably, which is a bummer, but I
can't see any other way to override the resolver on a per-user basis
($HOSTALIASES only does forward DNS).

Any other ideas?

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera