Bug#863260: kstart: k5start does not recognize network changes

[Kai Wohlfahrt]

Package: kstart
Version: 4.2-1
Severity: important
Tags: upstream

Dear Maintainer,

The k5start program repeats attempts to contact the KDC server if it is
unavailable. However, it continues to fail if a new network is available.

Steps to reproduce:
- Disable network interface
- Run k5start (e.g. k5start -f /etc/krb5.keytab -K 10 -u host/jason -k test.tkt)
- Enable network interface

I expect to see the error below repeated until the network comes back up, and
then it should stop:
k5start: error getting credentials: Cannot contact any KDC for realm 
'MY.REALM.NAME'

Instead, I continue to get errors until I stop the k5start process. Starting it
again after the network is available shows no errors and gets the ticket
correctly.

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64
 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages kstart depends on:
ii  libc6  2.24-10
ii  libkrb5-3  1.15-1

kstart recommends no packages.

kstart suggests no packages.

-- no debconf information

Bug#863260: kstart: k5start does not recognize network changes

[Russ Allbery]

Control: reassign -1 libkrb5-3
Control: affects -1 kstart

Kai Wohlfahrt  writes:

> Package: kstart
> Version: 4.2-1
> Severity: important
> Tags: upstream

> Dear Maintainer,

> The k5start program repeats attempts to contact the KDC server if it is
> unavailable. However, it continues to fail if a new network is
> available.

> Steps to reproduce:
> - Disable network interface
> - Run k5start (e.g. k5start -f /etc/krb5.keytab -K 10 -u host/jason -k 
> test.tkt)
> - Enable network interface

> I expect to see the error below repeated until the network comes back
> up, and then it should stop:

> k5start: error getting credentials: Cannot contact any KDC for realm 
> 'MY.REALM.NAME'

> Instead, I continue to get errors until I stop the k5start
> process. Starting it again after the network is available shows no
> errors and gets the ticket correctly.

k5start just calls krb5_get_init_creds_*, so if something is being
inappropriately cached, this would be a bug in the underlying Kerberos
library (rather than in kstart).  Reassigning accordingly.

-- 
Russ Allbery (r...@debian.org)   

Bug#863260: kstart: k5start does not recognize network changes

[Benjamin Kaduk]

On Wed, May 24, 2017 at 11:44:12AM -0700, Russ Allbery wrote:
> 
> Kai Wohlfahrt  writes:
> 
> > Package: kstart
> > Version: 4.2-1
> > Severity: important
> > Tags: upstream
> 
> > Dear Maintainer,
> 
> > The k5start program repeats attempts to contact the KDC server if it is
> > unavailable. However, it continues to fail if a new network is
> > available.
> 
> > Steps to reproduce:
> > - Disable network interface
> > - Run k5start (e.g. k5start -f /etc/krb5.keytab -K 10 -u host/jason -k 
> > test.tkt)
> > - Enable network interface
> 
> > I expect to see the error below repeated until the network comes back
> > up, and then it should stop:
> 
> > k5start: error getting credentials: Cannot contact any KDC for realm 
> > 'MY.REALM.NAME'
> 
> > Instead, I continue to get errors until I stop the k5start
> > process. Starting it again after the network is available shows no
> > errors and gets the ticket correctly.
> 
> k5start just calls krb5_get_init_creds_*, so if something is being
> inappropriately cached, this would be a bug in the underlying Kerberos
> library (rather than in kstart).  Reassigning accordingly.

I could not reproduce this behavior using any of 'ifconfig 
down', adding loopback routes to all configured KDCs, or modifying
the profile (KRB5_CONFIG pointing to a custom path) to use
inaccessible KDC addresses as a way of making the KDC inaccessible.
The 'ifconfig  down' method requires manual intervention to
restore a default route after bringing the interface back up,
though.

(Code inspection also did not find a place where such caching would
occur, though of course there are probably some places I didn't
look.)

Kai, can you confirm that you can reproduce and that have proper
network connectivity (DNS resolution, ping, etc.) during the case
where k5start continues to be unable to contact any KDCs?

Also, is krb5.conf or DNS being used to locate KDCs (not that this
appears to matter).

-Ben

Bug#863260: kstart: k5start does not recognize network changes

[Kai Wohlfahrt]

Thanks for looking into this.

Having tried the ifconfig method, k5start correctly detects the new link.
However, unplugging the ethernet cable, running k5start, and then plugging
it back in shows the error (i.e. continued errors even after I can run
kinit/k5start in another terminal).

The original method I used to reproduce the issue was to comment out
`/etc/systemd/network/50-wired.network` (my only network configuration
file) and then restart systemd-networkd + systemd-resolved. To re-enable
the network, I un-commented the lines and restarted the same services. That
was the method suggested here
https://lists.freedesktop.org/archives/systemd-devel/2015-September/034327.html

I have a block of the style `MY.REALM = {kdc = xyz.fqdn}` in  my krb5.conf,
is that what you meant by locating KDCs?

Not sure how helpful this link is, but perhaps it is an issue like this:
https://sourceware.org/bugzilla/show_bug.cgi?id=3675

Best wishes,
Kai


On Thu, 1 Jun 2017 at 05:03 Benjamin Kaduk  wrote:

> On Wed, May 24, 2017 at 11:44:12AM -0700, Russ Allbery wrote:
> >
> > Kai Wohlfahrt  writes:
> >
> > > Package: kstart
> > > Version: 4.2-1
> > > Severity: important
> > > Tags: upstream
> >
> > > Dear Maintainer,
> >
> > > The k5start program repeats attempts to contact the KDC server if it is
> > > unavailable. However, it continues to fail if a new network is
> > > available.
> >
> > > Steps to reproduce:
> > > - Disable network interface
> > > - Run k5start (e.g. k5start -f /etc/krb5.keytab -K 10 -u host/jason -k
> test.tkt)
> > > - Enable network interface
> >
> > > I expect to see the error below repeated until the network comes back
> > > up, and then it should stop:
> >
> > > k5start: error getting credentials: Cannot contact any KDC for realm '
> MY.REALM.NAME'
> >
> > > Instead, I continue to get errors until I stop the k5start
> > > process. Starting it again after the network is available shows no
> > > errors and gets the ticket correctly.
> >
> > k5start just calls krb5_get_init_creds_*, so if something is being
> > inappropriately cached, this would be a bug in the underlying Kerberos
> > library (rather than in kstart).  Reassigning accordingly.
>
> I could not reproduce this behavior using any of 'ifconfig 
> down', adding loopback routes to all configured KDCs, or modifying
> the profile (KRB5_CONFIG pointing to a custom path) to use
> inaccessible KDC addresses as a way of making the KDC inaccessible.
> The 'ifconfig  down' method requires manual intervention to
> restore a default route after bringing the interface back up,
> though.
>
> (Code inspection also did not find a place where such caching would
> occur, though of course there are probably some places I didn't
> look.)
>
> Kai, can you confirm that you can reproduce and that have proper
> network connectivity (DNS resolution, ping, etc.) during the case
> where k5start continues to be unable to contact any KDCs?
>
> Also, is krb5.conf or DNS being used to locate KDCs (not that this
> appears to matter).
>
> -Ben
>

Bug#863260: kstart: k5start does not recognize network changes

[Sam Hartman]

I wonder if your nss stack is somehow caching something about the
network and the name servers and that kstart process is no longer able
to resolve KDCs.
It would be interesting to set KRB5_TRACE to a file, run kstart such
that it is failing and see what specifically is not working.
My bet is on DNS

Bug#863260: kstart: k5start does not recognize network changes

[Kai Wohlfahrt]

See below the KRB5_TRACE file (domains/realms cleaned up). It could well be
DNS, but other programs (e.g. dig) respond correctly and recognize when the
network comes back up.

Kai

> [18010] 1497006307.510633: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006307.510733: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006307.510807: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006307.510877: Sending request (193 bytes) to MY.REALM
> [18010] 1497006307.510906: Resolving hostname my.kdc.fqdn
> [18010] 1497006307.511120: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006307.511154: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006307.511199: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006307.511225: Sending request (193 bytes) to MY.REALM
> [18010] 1497006307.511236: Resolving hostname my.kdc.fqdn
> [18010] 1497006308.512362: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006308.512531: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006308.512679: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006308.512761: Sending request (193 bytes) to MY.REALM
> [18010] 1497006308.512796: Resolving hostname my.kdc.fqdn
> [18010] 1497006310.514982: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006310.515150: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006310.515300: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006310.515378: Sending request (193 bytes) to MY.REALM
> [18010] 1497006310.515412: Resolving hostname my.kdc.fqdn
> [18010] 1497006314.516629: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006314.516779: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006314.516917: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006314.516993: Sending request (193 bytes) to MY.REALM
> [18010] 1497006314.517027: Resolving hostname my.kdc.fqdn
> [18010] 1497006322.525217: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006322.525387: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006322.525529: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006322.525612: Sending request (193 bytes) to MY.REALM
> [18010] 1497006322.525647: Resolving hostname my.kdc.fqdn
> [18010] 1497006338.541837: Getting initial credentials for
host/jason@MY.REALM
> [18010] 1497006338.542026: Setting initial creds service to
krbtgt/MY.REALM@MY.REALM
> [18010] 1497006338.542185: Looked up etypes in keytab: aes256-cts,
rc4-hmac, des3-cbc-sha1, des-cbc-crc
> [18010] 1497006338.542262: Sending request (193 bytes) to MY.REALM
> [18010] 1497006338.542302: Resolving hostname my.kdc.fqdn

On Fri, 9 Jun 2017 at 10:56 Sam Hartman  wrote:

> I wonder if your nss stack is somehow caching something about the
> network and the name servers and that kstart process is no longer able
> to resolve KDCs.
> It would be interesting to set KRB5_TRACE to a file, run kstart such
> that it is failing and see what specifically is not working.
> My bet is on DNS
>