So to respond to my own email, but a co-worker did finally find some references to that bizarre name ZZZKBTDURBOL8.
[root@nwpllv8bu100 post_install]# klist -kte Keytab name: FILE:/etc/krb5.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM (aes128-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM (aes256-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 host/ zzzkbtdurbol8.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 host/ zzzkbtdurbol8.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 host/zzzkbtdurb...@amer.company.com (aes128-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 host/zzzkbtdurb...@amer.company.com (aes256-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 RestrictedKrbHost/zzzkbtdurb...@amer.company.com (aes128-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 RestrictedKrbHost/zzzkbtdurb...@amer.company.com (aes256-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 RestrictedKrbHost/ zzzkbtdurbol8.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) 2 07/13/2021 16:42:17 RestrictedKrbHost/ zzzkbtdurbol8.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM (DEPRECATED:arcfour-hmac) 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM (aes128-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM (aes256-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company....@amer.company.com (DEPRECATED:arcfour-hmac) 3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com (DEPRECATED:arcfour-hmac) 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com (aes128-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com (aes256-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com (DEPRECATED:arcfour-hmac) 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com (aes128-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com (aes256-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 RestrictedKrbHost/ nwpllv8bu100.amer.company....@amer.company.com (DEPRECATED:arcfour-hmac) 3 07/29/2021 13:38:27 RestrictedKrbHost/ nwpllv8bu100.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) 3 07/29/2021 13:38:27 RestrictedKrbHost/ nwpllv8bu100.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) I realize this is reflecting the literal entries in the /etc/krb5.keytab file. So it appears that when this VM was born (on July 13th), it was named zzzkbtdurbo18.amer.company.com. (I see other supporting evidence for this). On or before July 29th, it was renamed to final FQDN nwpllv8bu100.amer.company.com. /etc/hostname, /etc/hosts, /etc/sysconfig/network etc were all updated and it was rejoined to AD. kinit -k works fine. It picks up the current hostname and apparently uses host/nwpllv8bu100.amer.company....@amer.company.com as its service principal. Since there's a valid entry in /etc/krb5.keytab file for this, it uses this and all is good. (I'm guessing it uses the 14th or 15th /etc/krb5.keytab file entry above.) sssd works, because it has this line: ldap_sasl_authid = host/nwpllv8bu100.amer.company....@amer.company.com But when sssd invokes adcli update to refresh the machine account password, adcli update fails. Also, I see that adcli testjoin fails. [root@nwpllv8bu100 tmp]# adcli testjoin -D AMER.COMPANY.COM adcli: couldn't connect to AMER.COMPANY.COM domain: Couldn't authenticate as machine account: ZZZKBTDURBOL8: Preauthentication failed [root@nwpllv8bu100 tmp]# >From a strace of this adcli testjoin, it appears that adcli is opening the /etc/krb5.keytab file to determine the default service principal to use and is pulling the old server name. (instead of using the correct service principal, as kinit -k somehow does.) Maybe when sssd constructs this adcli update invocation, it's not passing the ldap_sasl_authid, so the adcli update is doing the above logic to pull the old server name? Sounds like an adcli problem. Adcli should do as 'kinit -k' does when it's passed no explicit service principal. Should dive into /etc/krb5.keytab file and use the most recent set of entries (KVNO = 3 in above example). Maybe derive the default service principal off the current FQDN and Kerberos realm? Spike PS As a general policy, we are not supposed to clone a VM and rename it to another FQDN/IP address. I'll be trying to track down who did this and for what reason. On Wed, Sep 1, 2021 at 10:08 AM Spike White <spikewhit...@gmail.com> wrote: > Ok, this is *very* illuminating! > > I see this in sssd_amer.company.com.log" > > (2021-09-01 3:44:46): [be[amer.company.com]] > [ad_machine_account_password_renewal_done] (0x1000): --- adcli output > start--- > adcli: couldn't connect to amer.company.com domain: Couldn't authenticate > as machine account: ZZZKBTDURBOL8: Preauthentication failed > ---adcli output end--- > > However, I don't find that host name ZZZKBTDURBOL8 anywhere on the > system. (By company convention, servers named ZZZ* are test servers that > linux SEs spin up themselves). > > This server that's not renewing its creds is named: > nwpllv8bu100.amer.company.com. it's a std dev server. in > /etc/sssd/sssd.conf file, it has that as its sasl auth ID: > > [root@nwpllv8bu100 sssd]# grep sasl /etc/sssd/sssd.conf > ldap_sasl_authid = host/nwpllv8bu100.amer.dell....@amer.company.com > [root@nwpllv8bu100 sssd]# > > If I do 'kinit -k', the /etc/krb5.keytab file has that name as well: > > [root@nwpllv8bu100 sssd]# kinit -k > [root@nwpllv8bu100 sssd]# klist > Ticket cache: KCM:0 > Default principal: host/nwpllv8bu100.amer.dell....@amer.company.com > > Valid starting Expires Service principal > 09/01/2021 11:04:16 09/01/2021 21:04:16 krbtgt/ > amer.dell....@amer.company.com > renew until 09/08/2021 11:04:16 > [root@nwpllv8bu100 sssd]# > > I searched /etc/sssd/sssd.conf -- no "zzz" or "ZZZ" string is anywhere in > there. So where is sssd picking up this name ZZZKBTDURBOL8 and passing it > to adcli update? > > Spike > > > > > > > > On Wed, Sep 1, 2021 at 2:46 AM Sumit Bose <sb...@redhat.com> wrote: > >> Am Tue, Aug 31, 2021 at 09:53:01PM +0200 schrieb Alexey Tikhonov: >> > On Tue, Aug 31, 2021 at 6:47 PM Spike White <spikewhit...@gmail.com> >> wrote: >> > >> > > All, >> > > >> > > OK we have a query we run in AD for machine account passwords for a >> > > certain age. In today's run, 31 - 32 days. Then we verify it's >> pingable. >> > > >> > > We have found such one such suspicious candidate today (two actually, >> but >> > > the other Linux server is quite sick). So one good research >> candidate. >> > > According to both AD and /etc/krb5.keytab file, the machine account >> > > password was last set on 7/29. Today is 8/31, so that would be 32 >> days. >> > > This 'automatic machine account keytab renewal' background task >> should >> > > trigger again today. >> > > >> > > sssd service was last started 2 weeks ago and, by all appearances, >> appears >> > > healthy. sssctl domain-status <domain> shows online, connected to AD >> > > servers (both domain and GC servers).. All logins and group >> enumerations >> > > working as expected. >> > > >> > > Just now, we dynamically set the debug level to 9 with 'sssctl >> debug-level >> > > 9'. This particular server is Oracle Linux 8.4, >> > > running sssd-*-2.4.0-9.0.1.el8_4.1.x86_64. Installed July 13th, >> 2021. So >> > > -- very recent sssd version. (This problem occurs with both RHEL & OL >> > > 6/7/8, it's just today's candidate happens to be OL8.) >> > > >> > > We can't keep debug level 9 up for a great many days; it swamps the >> > > /var/log filesystem. But we can leave up for a few days. We >> purposely did >> > > not restart sssd server as we know that would trigger a machine >> account >> > > renewal. >> > > >> > > Speaking of that -- from Sumit's sssd source code in >> > > ad_provider/ad_machine_pw_renewal.c, it appears that sssd is creating >> a >> > > back-end task to call external program /usr/sbin/adcli with certain >> args. >> > > What string can I look for in which sssd log file (now that I have >> debug >> > > level 9 enabled) to tell me when this 'adcli update' task (aka >> 'automatic >> > > machine account keytab renewal') is triggered? >> > > >> > >> > It seems SSSD itself only logs in case of errors. I didn't find any >> > explicit logs around `ad_machine_account_password_renewal_send()`. >> > But perhaps there will be something like "[be_ptask_execute] (0x0400): >> Task >> > [AD machine account password renewal]: executing task" from generic >> > be_ptask_* helpers in the sssd_$domain.log (I'm not sure). >> > >> > Also at this verbosity level `--verbose` should be supplied to adcli >> itself >> > and I guess output should be captured in sssd_$domain.log as well. I'm >> not >> > familiar with `adcli` internals, you can take a glance at >> > https://gitlab.freedesktop.org/realmd/adcli to find its log messages. >> >> Hi, >> >> if SSSD's debug_level is 7 or higher the '--verbose' option is set >> when calling adcli and the output is added to the backend logs. It will >> start with log message "--- adcli output start---". >> >> HTH >> >> bye, >> Sumit >> >> > >> > >> > > >> > > I'm less certain now that we've surveyed our env that this background >> > > 'adcli update' task is the reason behind 70 - 80 servers / month >> dropping >> > > off the domain. It might be a slight contributor, but I find only a >> very >> > > few pingable servers with machine account last renewal date between >> 30 and >> > > 40 days. >> > > >> > > Yes, I can disable this default 30 day automatic update and roll my >> own >> > > 'adcli update' cron. But that's a mass deployment, to fix what might >> not >> > > be the problem. I want to verify this is the actual culprit before >> I take >> > > those drastic steps. >> > > >> > > Spike >> > > >> > > >> >> > _______________________________________________ >> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org >> > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org >> > Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> > List Archives: >> https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org >> > Do not reply to spam on the list, report it: >> https://pagure.io/fedora-infrastructure >> _______________________________________________ >> sssd-users mailing list -- sssd-users@lists.fedorahosted.org >> To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org >> Do not reply to spam on the list, report it: >> https://pagure.io/fedora-infrastructure >> >
_______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
