Am Wed, Sep 01, 2021 at 11:39:30AM -0500 schrieb Spike White: > So to respond to my own email, but a co-worker did finally find some > references to that bizarre name ZZZKBTDURBOL8. > > [root@nwpllv8bu100 post_install]# klist -kte > Keytab name: FILE:/etc/krb5.keytab > KVNO Timestamp Principal > ---- ------------------- > ------------------------------------------------------ > 2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM > (aes128-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM > (aes256-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 host/ > zzzkbtdurbol8.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 host/ > zzzkbtdurbol8.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 host/zzzkbtdurb...@amer.company.com > (aes128-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 host/zzzkbtdurb...@amer.company.com > (aes256-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 RestrictedKrbHost/zzzkbtdurb...@amer.company.com > (aes128-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 RestrictedKrbHost/zzzkbtdurb...@amer.company.com > (aes256-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 RestrictedKrbHost/ > zzzkbtdurbol8.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) > 2 07/13/2021 16:42:17 RestrictedKrbHost/ > zzzkbtdurbol8.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM > (DEPRECATED:arcfour-hmac) > 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM > (aes128-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM > (aes256-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company....@amer.company.com > (DEPRECATED:arcfour-hmac) > 3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company....@amer.company.com > (aes128-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company....@amer.company.com > (aes256-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com > (DEPRECATED:arcfour-hmac) > 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com > (aes128-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com > (aes256-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com > (DEPRECATED:arcfour-hmac) > 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com > (aes128-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com > (aes256-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 RestrictedKrbHost/ > nwpllv8bu100.amer.company....@amer.company.com (DEPRECATED:arcfour-hmac) > 3 07/29/2021 13:38:27 RestrictedKrbHost/ > nwpllv8bu100.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) > 3 07/29/2021 13:38:27 RestrictedKrbHost/ > nwpllv8bu100.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) > > I realize this is reflecting the literal entries in the /etc/krb5.keytab > file. So it appears that when this VM was born (on July 13th), it was > named zzzkbtdurbo18.amer.company.com. (I see other supporting evidence > for this). On or before July 29th, it was renamed to final FQDN > nwpllv8bu100.amer.company.com. /etc/hostname, /etc/hosts, > /etc/sysconfig/network etc were all updated and it was rejoined to AD. > > kinit -k works fine. It picks up the current hostname and apparently uses > host/nwpllv8bu100.amer.company....@amer.company.com as its service > principal. Since there's a valid entry in /etc/krb5.keytab file for this, > it uses this and all is good. (I'm guessing it uses the 14th or 15th > /etc/krb5.keytab file entry above.) > > sssd works, because it has this line: > > ldap_sasl_authid = host/nwpllv8bu100.amer.company....@amer.company.com > > But when sssd invokes adcli update to refresh the machine account password, > adcli update fails. > > Also, I see that adcli testjoin fails. > > [root@nwpllv8bu100 tmp]# adcli testjoin -D AMER.COMPANY.COM > adcli: couldn't connect to AMER.COMPANY.COM domain: Couldn't authenticate > as machine account: ZZZKBTDURBOL8: Preauthentication failed > [root@nwpllv8bu100 tmp]# > > From a strace of this adcli testjoin, it appears that adcli is opening the > /etc/krb5.keytab file to determine the default service principal to use and > is pulling the old server name. (instead of using the correct service > principal, as kinit -k somehow does.)
Hi, it looks like your environment is a bit special. I guess you have added 'host/nwpllv8bu100.amer.company....@amer.company.com' to the 'userPrincipalName' LDAP attribute in the AD computer object for this host. # klist -k Keytab name: FILE:/etc/krb5.keytab KVNO Principal ---- -------------------------------------------------------------------------- 2 MASTER$@CHILD.AD.VM 2 MASTER$@CHILD.AD.VM 2 host/mas...@child.ad.vm 2 host/mas...@child.ad.vm 2 host/master.client...@child.ad.vm 2 host/master.client...@child.ad.vm 2 RestrictedKrbHost/mas...@child.ad.vm 2 RestrictedKrbHost/mas...@child.ad.vm 2 RestrictedKrbHost/master.client...@child.ad.vm 2 RestrictedKrbHost/master.client...@child.ad.vm # kdestroy -A # # # kinit -k kinit: Client 'host/master.client...@child.ad.vm' not found in Kerberos database while getting initial credentials # kinit -k 'MASTER$@CHILD.AD.VM' The reason is that 'kinit -k' constructs the principal by calling gethostname() or similar, adding the 'host/' prefix and the realm. But by default this principal in AD is only a service principal can cannot be used to request a TGT as kinit does. AD only allows user principals for request a TGT and this is by default 'SHORT$@AD.REALM'. If the userPrincipalName attribute is set, this principal given here is allowed as well. That's why adcli is checking the keytab for a principal with a '$' character and by default it uses the first it finds because it is expected there is only one. Adding some heuristics in case there are more '$' principals in the keytab like highest KVNO might help in some cases but would fail in other cases so I think just using the first is good enough. > > Maybe when sssd constructs this adcli update invocation, it's not passing > the ldap_sasl_authid, so the adcli update is doing the above logic to pull > the old server name? Adding a new option to adcli and using the value from ldap_sasl_authid might be a solution, I will think about it. HTH bye, Sumit > > Sounds like an adcli problem. Adcli should do as 'kinit -k' does when it's > passed no explicit service principal. Should dive into /etc/krb5.keytab > file and use the most recent set of entries (KVNO = 3 in above example). > Maybe derive the default service principal off the current FQDN and > Kerberos realm? > > Spike > PS As a general policy, we are not supposed to clone a VM and rename it to > another FQDN/IP address. I'll be trying to track down who did this and for > what reason. > > On Wed, Sep 1, 2021 at 10:08 AM Spike White <spikewhit...@gmail.com> wrote: > > > Ok, this is *very* illuminating! > > > > I see this in sssd_amer.company.com.log" > > > > (2021-09-01 3:44:46): [be[amer.company.com]] > > [ad_machine_account_password_renewal_done] (0x1000): --- adcli output > > start--- > > adcli: couldn't connect to amer.company.com domain: Couldn't authenticate > > as machine account: ZZZKBTDURBOL8: Preauthentication failed > > ---adcli output end--- > > > > However, I don't find that host name ZZZKBTDURBOL8 anywhere on the > > system. (By company convention, servers named ZZZ* are test servers that > > linux SEs spin up themselves). > > > > This server that's not renewing its creds is named: > > nwpllv8bu100.amer.company.com. it's a std dev server. in > > /etc/sssd/sssd.conf file, it has that as its sasl auth ID: > > > > [root@nwpllv8bu100 sssd]# grep sasl /etc/sssd/sssd.conf > > ldap_sasl_authid = host/nwpllv8bu100.amer.dell....@amer.company.com > > [root@nwpllv8bu100 sssd]# > > > > If I do 'kinit -k', the /etc/krb5.keytab file has that name as well: > > > > [root@nwpllv8bu100 sssd]# kinit -k > > [root@nwpllv8bu100 sssd]# klist > > Ticket cache: KCM:0 > > Default principal: host/nwpllv8bu100.amer.dell....@amer.company.com > > > > Valid starting Expires Service principal > > 09/01/2021 11:04:16 09/01/2021 21:04:16 krbtgt/ > > amer.dell....@amer.company.com > > renew until 09/08/2021 11:04:16 > > [root@nwpllv8bu100 sssd]# > > > > I searched /etc/sssd/sssd.conf -- no "zzz" or "ZZZ" string is anywhere in > > there. So where is sssd picking up this name ZZZKBTDURBOL8 and passing it > > to adcli update? > > > > Spike > > > > > > > > > > > > > > > > On Wed, Sep 1, 2021 at 2:46 AM Sumit Bose <sb...@redhat.com> wrote: > > > >> Am Tue, Aug 31, 2021 at 09:53:01PM +0200 schrieb Alexey Tikhonov: > >> > On Tue, Aug 31, 2021 at 6:47 PM Spike White <spikewhit...@gmail.com> > >> wrote: > >> > > >> > > All, > >> > > > >> > > OK we have a query we run in AD for machine account passwords for a > >> > > certain age. In today's run, 31 - 32 days. Then we verify it's > >> pingable. > >> > > > >> > > We have found such one such suspicious candidate today (two actually, > >> but > >> > > the other Linux server is quite sick). So one good research > >> candidate. > >> > > According to both AD and /etc/krb5.keytab file, the machine account > >> > > password was last set on 7/29. Today is 8/31, so that would be 32 > >> days. > >> > > This 'automatic machine account keytab renewal' background task > >> should > >> > > trigger again today. > >> > > > >> > > sssd service was last started 2 weeks ago and, by all appearances, > >> appears > >> > > healthy. sssctl domain-status <domain> shows online, connected to AD > >> > > servers (both domain and GC servers).. All logins and group > >> enumerations > >> > > working as expected. > >> > > > >> > > Just now, we dynamically set the debug level to 9 with 'sssctl > >> debug-level > >> > > 9'. This particular server is Oracle Linux 8.4, > >> > > running sssd-*-2.4.0-9.0.1.el8_4.1.x86_64. Installed July 13th, > >> 2021. So > >> > > -- very recent sssd version. (This problem occurs with both RHEL & OL > >> > > 6/7/8, it's just today's candidate happens to be OL8.) > >> > > > >> > > We can't keep debug level 9 up for a great many days; it swamps the > >> > > /var/log filesystem. But we can leave up for a few days. We > >> purposely did > >> > > not restart sssd server as we know that would trigger a machine > >> account > >> > > renewal. > >> > > > >> > > Speaking of that -- from Sumit's sssd source code in > >> > > ad_provider/ad_machine_pw_renewal.c, it appears that sssd is creating > >> a > >> > > back-end task to call external program /usr/sbin/adcli with certain > >> args. > >> > > What string can I look for in which sssd log file (now that I have > >> debug > >> > > level 9 enabled) to tell me when this 'adcli update' task (aka > >> 'automatic > >> > > machine account keytab renewal') is triggered? > >> > > > >> > > >> > It seems SSSD itself only logs in case of errors. I didn't find any > >> > explicit logs around `ad_machine_account_password_renewal_send()`. > >> > But perhaps there will be something like "[be_ptask_execute] (0x0400): > >> Task > >> > [AD machine account password renewal]: executing task" from generic > >> > be_ptask_* helpers in the sssd_$domain.log (I'm not sure). > >> > > >> > Also at this verbosity level `--verbose` should be supplied to adcli > >> itself > >> > and I guess output should be captured in sssd_$domain.log as well. I'm > >> not > >> > familiar with `adcli` internals, you can take a glance at > >> > https://gitlab.freedesktop.org/realmd/adcli to find its log messages. > >> > >> Hi, > >> > >> if SSSD's debug_level is 7 or higher the '--verbose' option is set > >> when calling adcli and the output is added to the backend logs. It will > >> start with log message "--- adcli output start---". > >> > >> HTH > >> > >> bye, > >> Sumit > >> > >> > > >> > > >> > > > >> > > I'm less certain now that we've surveyed our env that this background > >> > > 'adcli update' task is the reason behind 70 - 80 servers / month > >> dropping > >> > > off the domain. It might be a slight contributor, but I find only a > >> very > >> > > few pingable servers with machine account last renewal date between > >> 30 and > >> > > 40 days. > >> > > > >> > > Yes, I can disable this default 30 day automatic update and roll my > >> own > >> > > 'adcli update' cron. But that's a mass deployment, to fix what might > >> not > >> > > be the problem. I want to verify this is the actual culprit before > >> I take > >> > > those drastic steps. > >> > > > >> > > Spike > >> > > > >> > > > >> > >> > _______________________________________________ > >> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > >> > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > >> > Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> > List Archives: > >> https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > >> > Do not reply to spam on the list, report it: > >> https://pagure.io/fedora-infrastructure > >> _______________________________________________ > >> sssd-users mailing list -- sssd-users@lists.fedorahosted.org > >> To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > >> Do not reply to spam on the list, report it: > >> https://pagure.io/fedora-infrastructure > >> > > > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
