Thank you very much for your response. I have done some further
debugging and have more info in out situation and what the genesis of
the issue is. I am hoping you have some suggestions on a resolution
I am in a bit of a pickle
(1) We have nightly full backups of IPA but they have been running on a
NON CA server. So we do not have the ipaca tree in our backups.
(2) We had a broken CA replication agreement between out two CA servers
and mostly all requests went to only one of the servers.
(3) We regenerated our CAs using the server that did not get the
replicated requests. So we lost all certs and requests from "time A" onward.
The situation on our two CA servers now is that we have the below certs
in /etc/pki/pki-tomcat/alias and /var/lib/certmonger/requests, but not
in ldap
"CN=CA Audit,O=SNAFU.NET" Wed Jun 17 17:45:30 2026
"CN=CA Subsystem,O=SNAFU.NET" Wed Jul 01 17:52:00 2026
"CN=OCSP Subsystem,O=SNAFU.NET" Thu Apr 03 13:00:01 2025
(4) This is what I have on the CA server(s)
certutil -L -d . -n 'ocspSigningCert cert-pki-ca'
Serial Number: 580 (0x244)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=SNAFU.NET"
Validity:
Not Before: Fri Apr 14 13:00:01 2023
Not After : Thu Apr 03 13:00:01 2025
Subject: "CN=OCSP Subsystem,O=SNAFU.NET"
/var/lib/certmonger/requests/20241109195943 - which is the req for the above
(5) The OCSP is expired and ipa-cert-fix fails because we do not have
either of the below in ldap
cn=580,ou=certificateRepository,ou=ca,o=ipaca (with metaInfo:
requestId:####)
cn=####,ou=ca,ou=requests,o=ipaca
(6) What I do have is an older version of the cert and request in ldap,
but I do not have that cert locally in the nss db or in certmonger
dn: cn=558,ou=certificateRepository,ou=ca,o=ipaca
issuerName: CN=Certificate Authority,O=VOLEON.NET
subjectName: CN=OCSP Subsystem,O=VOLEON.NET
duration: 1162208000000
notAfter: 20240707001455Z
notBefore: 20220718001455Z
metaInfo: requestId:685
metaInfo: profileId:caOCSPCert
serialno: 03558
objectClass: top
objectClass: certificateRecord
dn: cn=685,ou=ca,ou=requests,o=ipaca
cn: 685
dateOfCreate: 20220718001455Z
requestId: 03685
extdata-authenticatedname: CN=OCSP Subsystem,O=SNAFU.NET
I was thinking that I could use ipa-cert-fix against the older cert pair
but I am unsure of how do accomplish this. Is it as simple as replacing
the cert in the nss db? Do I need to hack a /var/lib/certmonger/requests
file.
I will take any suggestion even if it is a painful one,.
On 4/14/25 1:19 AM, Florence Blanc-Renaud wrote:
Hi,
On Sun, Apr 13, 2025 at 7:25 PM Mark Selby via FreeIPA-users
<[email protected]> wrote:
Our old FreeIPA cluster ipa-server-4.6.8-5.el7 (which we will
upgrade if/when this issue resolved) has a non functional CA due
to the ocspSigningCert being expired.
I have tried all of the suggested fixes that others with this
issue have suggested. ipa-cert-fix and running pki-server cert-fix
directly all fail.
ipa-cert-fix
pki-server cert-fix --ldapi-socket /var/run/slapd-SNAFU-NET.socket
--agent-uid ipara --cert ca_ocsp_signing
I have also tried setting back the clock on one of the CA servers
and running both ipa-cert-fix and getcert resubmit. This
sugesstion https://access.redhat.com/solutions/3939431 was also tried.
What is the output of ipa-cert-fix --verbose or pki-server cert-fix
--verbose?
When I turn back the clock and restart the pki server at least I
can look at the certificates as the CA. When the clock is normal
the pki server will not start due to the invalid cert
[13/Apr/2025:09:10:03][localhost-startStop-1]: CertUtils:
verifySystemCertsByTag() failed: java.lang.Exception:
Certutils.verifySystemCertValidityByNickname: faliled:
nickname:ocspSigningCert cert-pki-cacause: java.lang.Exception:
Certutils.verifySystemCertValidityByNickname: failed: nickname:
ocspSigningCert cert-pki-ca
I turned on verbose=4 logging in CS.cfg and received the below
debug output. The serial number of the ocsp cert is 580 as fas the
certmonger is concerned. The pki server is seemingly connecting to
LDAP to lookup that cert and ends up with the Record not found
error. I can not seem to find which ldap tree the server is
looking in for this record or which record it is searching for.
The cert-fix tool is looking for the CSR in CS.cfg and for the cert in
the LDAP tree. The CSR are also stored below ou=ca,ou=requests,o=ipaca
and the certs below ou=certificateRepository,ou=ca,o=ipaca.
Do you have your ocsp cert in the subtree
ou=certificateRepository,ou=ca,o=ipaca? You can check with
ldapsearch -D cn=directory\ manager -W -b
"ou=certificateRepository,ou=ca,o=ipaca" "(subjectname=cn=ocsp*)"
We had a few bugs with cert-fix, happening when the CSR was missing
from /etc/pki/pki-tomcat/ca/CS.cfg (1780782
<https://bugzilla.redhat.com/show_bug.cgi?id=1780782>) or when some
parameters were missing from CS.cfg (1930586
<https://bugzilla.redhat.com/show_bug.cgi?id=1930586>). With your
verbose output from cert-fix we may be able to find if you're hitting
one of those issues.
flo
I believe that I have all the correct certs in ldap under
cn=ca_renewal, but of course the ocspSigningCert in ldap is expired.
Does anyone know what is happening here and what a possible fix
might be. All and any help is greatly appreciated
cd /etc/pki/pki-tomcat/alias
certutil -L -d . -n 'ocspSigningCert cert-pki-ca'
Validity:
Not Before: Fri Apr 14 13:00:01 2023
Not After : Thu Apr 03 13:00:01 2025
Subject: "CN=OCSP Subsystem,O=SNAFU.NET <http://SNAFU.NET>"
Version: 3 (0x2)
Serial Number: 580 (0x244)
dn: cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net
dn: cn=ipaCert,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net
dn: cn=subsystemCert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net
dn: cn=auditSigningCert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net
dn: cn=ocspSigningCert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=snafu,dc=net
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
CAProcessor.java:252:printParameterValues() CAProcessor: Input
Parameters:
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
CAProcessor.java:264:printParameterValues() CAProcessor: -
isRenewal: true
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
CAProcessor.java:264:printParameterValues() CAProcessor: -
remoteHost: 10.17.1.18
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
CAProcessor.java:264:printParameterValues() CAProcessor: -
profileId: caManualRenewal
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
CAProcessor.java:264:printParameterValues() CAProcessor: -
serial_num: 580
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
CAProcessor.java:264:printParameterValues() CAProcessor: -
remoteAddr: 10.17.1.18
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
RenewalProcessor.java:90:processRenewal() RenewalProcessor:
processRenewal()
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
RenewalProcessor.java:98:processRenewal() RenewalProcessor:
profile: caManualRenewal
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
RenewalProcessor.java:149:processRenewal() RenewalProcessor: found
SerialNumRenewInput
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
RenewalProcessor.java:157:processRenewal() RenewalProcessor:
profile input serial_num value: 580
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
RenewalProcessor.java:181:processRenewal() processRenewal: serial
number of cert to renew:580
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
LdapBoundConnFactory.java:324:getConn() In
LdapBoundConnFactory::getConn()
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
LdapBoundConnFactory.java:326:getConn() masterConn is connected: true
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
LdapBoundConnFactory.java:368:getConn() getConn: conn is connected
true
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
LdapBoundConnFactory.java:398:getConn() getConn: mNumConns now 2
[12/Apr/2025:17:55:31][http-bio-8443-exec-1]:
LdapBoundConnFactory.java:444:returnConn() returnConn: mNumConns now 3
Record not found
at
com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:182)
at
com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:137)
at
com.netscape.cmscore.dbs.CertificateRepository.readCertificateRecord(CertificateRepository.java:1023)
at
com.netscape.cms.servlet.cert.RenewalProcessor.processRenewal(RenewalProcessor.java:182)
at
com.netscape.cms.servlet.cert.CertRequestDAO.submitRequest(CertRequestDAO.java:194)
at
org.dogtagpki.server.ca.rest.CertRequestService.enrollCert(CertRequestService.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297)
at
org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
--
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to
[email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
