On the failing node, the output of "getcert list" does not show any expired certs. I have hand-copied the info info this email below (it's interesting to note that while the other IdM servers are tracking 9 certs, the problem server is only tracking 8):
Number of certificates and requests being tracked: 8 Request ID '<###>': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=<servername>,O=<domain> subject: CN=<servername>,O=<domain> expires: 2020-09-12 19:51:34 UTC principal name: krbtgt/<domain> certificate template/profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token=NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<domain> subject: CN=CA Audit,O=<domain> expires: 2021-08-10 17:20:21 UTC key usage: digitialSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token=NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<domain> subject: CN=OCSP Subsystem,O=<domain> expires: 2021-08-10 17:19:42 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token=NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<domain> subject: CN=CA Subsystem,O=<domain> expires: 2021-08-10 17:19:51 UTC key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token=NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<domain> subject: CN=Certificate Authority,O=<domain> expires: 2037-09-28 14:29:02 UTC key usage: digitialSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/ipa/ra-agent.key' certificate: type=NSSDB,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<domain> subject: CN=IPA RA,,O=<domain> expires: 2021-08-10 17:20:41 UTC key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<domain>/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<domain> subject: CN=<server>,O=<domain> expires: 2021-09-09 19:53:33 UTC principal name: ldap/<serverFQDN@domain> key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <domain> track: yes auto-renew: yes Request ID '<###>': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<domain> subject: CN=<server>,O=<domain> expires: 2021-09-09 19:51:45 UTC principal name: HTTP/<serverFQDN@domain> key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Thank you so much again! Scot ________________________________ From: Florence Blanc-Renaud <f...@redhat.com> Sent: Thursday, August 6, 2020 2:46 AM To: FreeIPA users list <freeipa-users@lists.fedorahosted.org> Cc: Scott Z. <sud...@hotmail.com> Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote: > Thanks much for the assistance. Here is where I am with your suggestions: > 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n > 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old > (almost a year old actually, I assume IPA only checks it when it first > starts up so it didn't care that it was expired until the server was > rebooted?) certmonger checks the certificate validity periodically (configurable in certmonger.conf) and tries multiple times to renew soon-to-expire certs. The system probably had an issue that was not detected and the cert reached its expiration date. > > 2) ran ipactl start --ignore-service-failures >       a. most services started, obviously pki-tomcatd did not > 3) ran "kinit admin" >       a. was forced to change the password, but otherwise nothing > happened > 4) Ran "ipa config-show |grep -i master >      a. I see that the IPA CA renewal master is a different idm > machine. > 5) Ran "getcert list | grep -E "Request|certificate:|expires:" >      a.I see all certs are currently valid (none expired) > 6) Ran the command "getcert list" on the problem server, but I cannot > paste the output here because it's on an airgaped environment so while I > apologize for this and realize it makes things more difficult, perhaps > if you tell me what I should be looking for or more specifically what > you're interested in I can pluck that out and manually include it here? > So in summary, it is indeed an expired "Server-Cert cert-pki-ca' > certificate on the problem server, and it can theoretically be renew by > the Master at this time. The interesting part is the list of expired certs on the failing node (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed instructions are available here: https://access.redhat.com/solutions/3357331 How do I manually renew Identity Management (IPA) certificates on RHEL7 after they have expired? (Replica IPA Server) flo > Many thanks! > Scott > > ------------------------------------------------------------------------ > *From:* Florence Blanc-Renaud <f...@redhat.com> > *Sent:* Monday, August 3, 2020 9:34 PM > *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org> > *Cc:* Scott Z. <sud...@hotmail.com> > *Subject:* Re: [Freeipa-users] pki-tomcatd not starting > On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote: >> Not sure I'm sending this to the right place, but here it goes. I >> inherited a FreeIPA/Identity Manager setup in an enclave (no internet >> access) environment that is running into problems. There are at least 3 >> different IdM servers running in the environment spread out across >> different geographical areas. One of those areas suffered an unschedule >> power outage recently, and ever since we brought everything back up, the >> IdM server for this region is having an issue. Please bear with me as I >> have zero formal experience, training, or real knowledge with IdM. >> >> Logging in to the serverv (it's a VM server, running Centos 7.5), I run >> "ipactl status" and it shows "Directory Service: STOPPED". I then run >> "ipactl restart", and things go fine until it gets to "Starting >> pki-tomcatd Service", where it hangs for quite some time before failing >> to start and killing all the other services. I check the log at >> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as >> (forgive any mistypings, I have to manually type these in as I can't >> import or screen capure the logs and put them in this message): >> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid: >> Invalid certificate: (-8181) Peer's Certificate has expired/" >> And slightly further down in the same log: >> "/Cannot reset factory: connections not all returned/" >> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset >> LDAP connection factory because some connections are still outstanding/" >> ... still further down" >> "/returnConn:mNumConns now 3 Invalid class name repositorytop/" >> >> Assuming I have some weird certificate issue with this server in >> particular, I try to run a few more commands: >> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing >> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C >> for it's attributes. Comparing to a second IdM server in this >> environment, it seems to be missing a "Signing-Cert"? >> > Hi, > PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert > has the nickname 'Server-Cert cert-pki-ca'. You should check that this > one is not expired with: > # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' > | grep 'Not ' > > If the certificate is indeed expired, it will have to be renewed but you > need first to find which IPA server is the CA renewal master. On your > server, force a service start and check the CA renewal master: > # ipactl start --ignore-service-failures > # kinit admin > # ipa config-show | grep "renewal master" >   IPA CA renewal master: server.domain.com > > You need to make sure that all the certificates are valid on the CA > renewal master: > (on the CA renewal master)# getcert list | grep -E > "Request|certificate:|expires:" > > - if the CA renewal master is not OK, please post the output of "# > getcert list" (without the grep) on the CA renewal master. This node > will have to be repaired first. > - if the CA renewal master is OK, please post the output of "# getcert > list" (also without the grep) on the failing node. > > We'll be able to help based on this information. > flo > >> I also did a "getcert list", and all certs it has show that they expire >> in the future (nothing shows as bein currently expired). >> >> I'm confused; it seems to that it is seeing an expired cert *somewhere*, >> but how do I track down which 'peer' the log file is talking about that >> has an expired cert? Meanwhile none of the linux clients that point to >> this IdM server are allowing people to log in/authenticate. >> Many thanks for any help! >> Scott >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >> > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org