OK, this new workaround seems to be working on my el6 box. The next issue I ran into is that KRB5_CLIENT_KTNAME is new in 1.11, so we need to explicitly login from the keytab in order for servers to act as a client.
Put a WIP patch that addresses both of these here: https://gerrit.cloudera.org/#/c/4990/ Dan's checking to see if this also addresses OSX krb5 test failures. Assuming people are OK with the general direction, will finish this up tomorrow, so we can hopefully have green tests in the 1.1 release. On Mon, Nov 7, 2016 at 7:08 PM, Todd Lipcon <t...@cloudera.com> wrote: > Here's one more workaround which might actually be simplest: it seems like > if we LD_PRELOAD (or otherwise override) krb5_get_host_realm(), we can > detect this case and return our default KRBTEST.COM realm. > > I'll give that one a try and see if it works, since it's only 10-15 lines > of code vs importing new dependencies/build complexity. > > -Todd > > On Mon, Nov 7, 2016 at 6:49 PM, Todd Lipcon <t...@cloudera.com> wrote: > >> On Mon, Nov 7, 2016 at 4:35 PM, Alexey Serbin <aser...@cloudera.com> >> wrote: >> >>> Ah, I see -- thank you for the clarification. >>> >>> BTW, could 'ignore_acceptor_hostname' parameter help there? Like adding >>> the following into the krb5.conf: >>> >>> [libdefaults] >>> ignore_acceptor_hostname = true >>> >> >> We've actually already got that there. But it doesn't help -- the issue >> is that the client can't figure out what realm it's connecting to in krb >> 1.10 >> >> >>> >>> >>> Best regards, >>> >>> Alexey >>> >>> On Mon, Nov 7, 2016 at 3:42 PM, Todd Lipcon <t...@cloudera.com> wrote: >>> >>> > On Mon, Nov 7, 2016 at 3:37 PM, Alexey Serbin <aser...@cloudera.com> >>> > wrote: >>> > >>> > > May be, as a workaround we could run our mini_kdc only on at >>> 127.0.0.1 >>> > > address. >>> > > >>> > >>> > The issue is the actual Kudu servers, not the KDC. The KDC is on >>> 127.0.0.1, >>> > but we have service principals like kudu/127.2.3.4@TESTREALM that are >>> > causing the issue. >>> > >>> > -Todd >>> > >>> > >>> > > >>> > > >>> > > Best regards, >>> > > >>> > > Alexey >>> > > >>> > > On Sun, Nov 6, 2016 at 9:57 PM, Todd Lipcon <t...@cloudera.com> >>> wrote: >>> > > >>> > > > FWIW it looks like there's already some code out there that can do >>> the >>> > > > appropriate "fake DNS" wrapping: https://cwrap.org/nss_wrapper. >>> html >>> > > > >>> > > > >>> > > > On Sun, Nov 6, 2016 at 9:13 PM, Todd Lipcon <t...@cloudera.com> >>> wrote: >>> > > > >>> > > > > Hey folks >>> > > > > >>> > > > > I've been looking into why our kerberos-dependent tests are >>> failing >>> > on >>> > > > el6 >>> > > > > and it looks like it will be unfortunately difficult to fix. >>> > > > > >>> > > > > The first issue was that krb5 1.10 (on el6) doesn't automatically >>> > > create >>> > > > > the directory for a DIR:<path> type ticket cache. That one was >>> easy >>> > to >>> > > > fix >>> > > > > and got minikdc-test passing. >>> > > > > >>> > > > > The more insidious issue, though, is the following: >>> > > > > In miniclusters we start daemons on weird local IP addresses >>> > > (127.x.y.z) >>> > > > > which don't have any corresponding domain names. So, we have >>> > configured >>> > > > the >>> > > > > MiniKDC to disable reverse DNS, and are creating service >>> principals >>> > > kudu/ >>> > > > > 127.x....@krbtest.com. This works great on krb5-1.12. >>> > > > > >>> > > > > However, there's a bug in krb5 1.10 (http://krbdev.mit.edu/rt/ >>> > > > > Ticket/Display.html?id=7603) which is preventing this from >>> working on >>> > > > > el6. When 'rdns = false', Kerberos is unable to figure out which >>> > realm >>> > > > the >>> > > > > server is in (and doesn't fall back to the default realm). So, >>> our >>> > > > > connections are failing with: "Cannot determine realm for numeric >>> > host >>> > > > > address". >>> > > > > >>> > > > > If we change to 'rdns = true', then it will first try to reverse >>> > lookup >>> > > > > the weird loopback, and then when it fails, still give the same >>> error >>> > > > about >>> > > > > numeric hosts. >>> > > > > >>> > > > > It's possible there's some way to explicitly set the target >>> realm, >>> > but >>> > > if >>> > > > > so I can't seem to find it. So, the only workarounds I can think >>> of >>> > > are: >>> > > > > >>> > > > > 1) Use a local build of krb5 1.12 or later on el6 for test >>> purposes. >>> > I >>> > > > > verified that if I built krb5 1.14 and put its libraries in >>> > > > > LD_LIBRARY_PATH, and modified MiniKDC to use corresponding >>> binaries, >>> > > the >>> > > > > tests passed fine. >>> > > > > >>> > > > > The downside here is that we would lose test coverage of the >>> library >>> > > > > version actually installed on a lot of end-user systems. So >>> another >>> > > > option >>> > > > > might be to test against a patched version of 1.10 (with only >>> the fix >>> > > for >>> > > > > this numeric hostname issue). >>> > > > > >>> > > > > 2) Somehow monkey-patch getnameinfo() to provide reverse DNS >>> entries >>> > > for >>> > > > > 127.x.y.z mapping to '127-x-y-z.kudu.local' or something of that >>> > > nature. >>> > > > > We'd have to do this via LD_PRELOAD probably, which is a bummer, >>> but >>> > I >>> > > > > can't see any other way to override the resolver on a per-user >>> basis >>> > > > > ($HOSTALIASES only does forward DNS). >>> > > > > >>> > > > > Any other ideas? >>> > > > > >>> > > > > -Todd >>> > > > > -- >>> > > > > Todd Lipcon >>> > > > > Software Engineer, Cloudera >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Todd Lipcon >>> > > > Software Engineer, Cloudera >>> > > > >>> > > >>> > >>> > >>> > >>> > -- >>> > Todd Lipcon >>> > Software Engineer, Cloudera >>> > >>> >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > > > > -- > Todd Lipcon > Software Engineer, Cloudera > -- Todd Lipcon Software Engineer, Cloudera
