Hi, On Wed, Apr 16, 2025 at 1:25 AM Mark Selby <[email protected]> wrote:
> Thank you very much for your response. I have done some further debugging > and have more info in out situation and what the genesis of the issue is. I > am hoping you have some suggestions on a resolution > > I am in a bit of a pickle > > (1) We have nightly full backups of IPA but they have been running on a > NON CA server. So we do not have the ipaca tree in our backups. > > (2) We had a broken CA replication agreement between out two CA servers > and mostly all requests went to only one of the servers. > > (3) We regenerated our CAs using the server that did not get the > replicated requests. So we lost all certs and requests from "time A" onward. > > The situation on our two CA servers now is that we have the below certs in > /etc/pki/pki-tomcat/alias and /var/lib/certmonger/requests, but not in ldap > > "CN=CA Audit,O=SNAFU.NET" Wed Jun 17 17:45:30 2026 > "CN=CA Subsystem,O=SNAFU.NET" Wed Jul 01 17:52:00 2026 > "CN=OCSP Subsystem,O=SNAFU.NET" Thu Apr 03 13:00:01 2025 > > (4) This is what I have on the CA server(s) > > certutil -L -d . -n 'ocspSigningCert cert-pki-ca' > > Serial Number: 580 (0x244) > Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption > Issuer: "CN=Certificate Authority,O=SNAFU.NET" > Validity: > Not Before: Fri Apr 14 13:00:01 2023 > Not After : Thu Apr 03 13:00:01 2025 > Subject: "CN=OCSP Subsystem,O=SNAFU.NET" > If you have the cert in the NSSDB, you can export it to a pem format with certutil -L -d . -n 'ocspSigningCert cert-pki-ca' -a As you need to have the content without the header/footer and in a single line, you can use: certutil -L -d /etc/pki/pki-tomcat/alias -n 'ocspSigningCert cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n' > /tmp/cert.out Then you can re-create the LDAP entry with this content in the "userCertificate;binary" attribute. You can copy-paste from an existing entry in order to add the proper attributes and objectclass. flo /var/lib/certmonger/requests/20241109195943 - which is the req for the above > > > (5) The OCSP is expired and ipa-cert-fix fails because we do not have > either of the below in ldap > > cn=580,ou=certificateRepository,ou=ca,o=ipaca (with metaInfo: > requestId:####) > > cn=####,ou=ca,ou=requests,o=ipaca > > (6) What I do have is an older version of the cert and request in ldap, > but I do not have that cert locally in the nss db or in certmonger > > dn: cn=558,ou=certificateRepository,ou=ca,o=ipaca > issuerName: CN=Certificate Authority,O=VOLEON.NET > subjectName: CN=OCSP Subsystem,O=VOLEON.NET > duration: 1162208000000 > notAfter: 20240707001455Z > notBefore: 20220718001455Z > metaInfo: requestId:685 > metaInfo: profileId:caOCSPCert > serialno: 03558 > objectClass: top > objectClass: certificateRecord > > dn: cn=685,ou=ca,ou=requests,o=ipaca > cn: 685 > dateOfCreate: 20220718001455Z > requestId: 03685 > extdata-authenticatedname: CN=OCSP Subsystem,O=SNAFU.NET > > I was thinking that I could use ipa-cert-fix against the older cert pair > but I am unsure of how do accomplish this. Is it as simple as replacing > the cert in the nss db? Do I need to hack a /var/lib/certmonger/requests > file. > > I will take any suggestion even if it is a painful one,. > > > On 4/14/25 1:19 AM, Florence Blanc-Renaud wrote: > > Hi, > > On Sun, Apr 13, 2025 at 7:25 PM Mark Selby via FreeIPA-users < > [email protected]> wrote: > >> Our old FreeIPA cluster ipa-server-4.6.8-5.el7 (which we will upgrade >> if/when this issue resolved) has a non functional CA due to the >> ocspSigningCert being expired. >> >> I have tried all of the suggested fixes that others with this issue have >> suggested. ipa-cert-fix and running pki-server cert-fix directly all fail. >> >> ipa-cert-fix >> pki-server cert-fix --ldapi-socket /var/run/slapd-SNAFU-NET.socket >> --agent-uid ipara --cert ca_ocsp_signing >> >> I have also tried setting back the clock on one of the CA servers and >> running both ipa-cert-fix and getcert resubmit. This sugesstion >> https://access.redhat.com/solutions/3939431 was also tried. >> > > What is the output of ipa-cert-fix --verbose or pki-server cert-fix > --verbose? > > >> When I turn back the clock and restart the pki server at least I can look >> at the certificates as the CA. When the clock is normal the pki server will >> not start due to the invalid cert >> >> [13/Apr/2025:09:10:03][localhost-startStop-1]: CertUtils: >> verifySystemCertsByTag() failed: java.lang.Exception: >> Certutils.verifySystemCertValidityByNickname: faliled: >> nickname:ocspSigningCert cert-pki-cacause: java.lang.Exception: >> Certutils.verifySystemCertValidityByNickname: failed: nickname: >> ocspSigningCert cert-pki-ca >> >> I turned on verbose=4 logging in CS.cfg and received the below debug >> output. The serial number of the ocsp cert is 580 as fas the certmonger is >> concerned. The pki server is seemingly connecting to LDAP to lookup that >> cert and ends up with the Record not found error. I can not seem to find >> which ldap tree the server is looking in for this record or which record it >> is searching for. >> > > The cert-fix tool is looking for the CSR in CS.cfg and for the cert in the > LDAP tree. The CSR are also stored below ou=ca,ou=requests,o=ipaca and the > certs below ou=certificateRepository,ou=ca,o=ipaca. > Do you have your ocsp cert in the subtree > ou=certificateRepository,ou=ca,o=ipaca? You can check with > ldapsearch -D cn=directory\ manager -W -b > "ou=certificateRepository,ou=ca,o=ipaca" "(subjectname=cn=ocsp*)" > > We had a few bugs with cert-fix, happening when the CSR was missing from > /etc/pki/pki-tomcat/ca/CS.cfg (1780782 > <https://bugzilla.redhat.com/show_bug.cgi?id=1780782>) or when some > parameters were missing from CS.cfg (1930586 > <https://bugzilla.redhat.com/show_bug.cgi?id=1930586>). With your verbose > output from cert-fix we may be able to find if you're hitting one of those > issues. > > flo > >> >> I believe that I have all the correct certs in ldap under cn=ca_renewal, >> but of course the ocspSigningCert in ldap is expired. >> >> Does anyone know what is happening here and what a possible fix might be. >> All and any help is greatly appreciated >> >> cd /etc/pki/pki-tomcat/alias >> certutil -L -d . -n 'ocspSigningCert cert-pki-ca' >> >> Validity: >> Not Before: Fri Apr 14 13:00:01 2023 >> Not After : Thu Apr 03 13:00:01 2025 >> Subject: "CN=OCSP Subsystem,O=SNAFU.NET" >> >> Version: 3 (0x2) >> Serial Number: 580 (0x244) >> >> dn: cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net >> dn: cn=ipaCert,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net >> dn: cn=subsystemCert >> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net >> dn: cn=auditSigningCert >> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net >> dn: cn=ocspSigningCert >> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net >> >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> CAProcessor.java:252:printParameterValues() CAProcessor: Input Parameters: >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> CAProcessor.java:264:printParameterValues() CAProcessor: - isRenewal: true >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> CAProcessor.java:264:printParameterValues() CAProcessor: - remoteHost: >> 10.17.1.18 >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> CAProcessor.java:264:printParameterValues() CAProcessor: - profileId: >> caManualRenewal >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> CAProcessor.java:264:printParameterValues() CAProcessor: - serial_num: 580 >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> CAProcessor.java:264:printParameterValues() CAProcessor: - remoteAddr: >> 10.17.1.18 >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> RenewalProcessor.java:90:processRenewal() RenewalProcessor: processRenewal() >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> RenewalProcessor.java:98:processRenewal() RenewalProcessor: profile: >> caManualRenewal >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> RenewalProcessor.java:149:processRenewal() RenewalProcessor: found >> SerialNumRenewInput >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> RenewalProcessor.java:157:processRenewal() RenewalProcessor: profile input >> serial_num value: 580 >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> RenewalProcessor.java:181:processRenewal() processRenewal: serial number of >> cert to renew:580 >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> LdapBoundConnFactory.java:324:getConn() In LdapBoundConnFactory::getConn() >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> LdapBoundConnFactory.java:326:getConn() masterConn is connected: true >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> LdapBoundConnFactory.java:368:getConn() getConn: conn is connected true >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> LdapBoundConnFactory.java:398:getConn() getConn: mNumConns now 2 >> [12/Apr/2025:17:55:31][http-bio-8443-exec-1]: >> LdapBoundConnFactory.java:444:returnConn() returnConn: mNumConns now 3 >> Record not found >> at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:182) >> at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:137) >> at >> com.netscape.cmscore.dbs.CertificateRepository.readCertificateRecord(CertificateRepository.java:1023) >> at >> com.netscape.cms.servlet.cert.RenewalProcessor.processRenewal(RenewalProcessor.java:182) >> at >> com.netscape.cms.servlet.cert.CertRequestDAO.submitRequest(CertRequestDAO.java:194) >> at >> org.dogtagpki.server.ca.rest.CertRequestService.enrollCert(CertRequestService.java:155) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137) >> at >> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280) >> at >> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234) >> at >> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221) >> at >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) >> at >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) >> at >> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >> at >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) >> at >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> at >> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> at >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297) >> at >> org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) >> at >> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) >> at >> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) >> at >> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> at >> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> at >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) >> at >> org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) >> at >> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) >> at >> org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) >> at >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498) >> -- >> _______________________________________________ >> FreeIPA-users mailing list -- [email protected] >> To unsubscribe send an email to >> [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/[email protected] >> Do not reply to spam, report it: >> https://pagure.io/fedora-infrastructure/new_issue >> >
-- _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
