On 05/02/2018 19:44, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone wrote:
On 31/01/2018 20:36, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote:
On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
Hi
Our freeipa certificates need to be renewed due to passing
their
expiry
dates.
While some certificates have renewed ok, the ipaCert and
auditSigningCert are renewing but the new certificates
have the
wrong
Subject.
Environment is:
serverA (CRL, first, master) RHEL 7.3, ipa 4.4
serverB (replica) RHEL 7.3, ipa 4.4
serverC (replica) RHEL 7.4, ipa 4.5
Once there are renewed certificates with the wrong Subject
present,
there are various problems with renewing the remaining
certificates,
which I think might be related to the bad Subject:
1) When just ipaCert has the wrong subject no further
renewals
happen
2) When auditSigningCert has the wrong subject the ipa
pki-tomcatd
service will not start and no further renewals happen.
I've been round the following loop many times on
ServerA, our
first
master:
1) Restore good certificates from backup
2) Put the clock back to a time when certificates are all
valid
3) Resubmit certificates for renewal
Each time the ipaCert renews it has the same wrong
Subject. The
wrong
Subject includes the host name of one of our ipa client
systems.
Each time the auditSigningCert renews it has the same wrong
Subject
but
a different subject to the ipaCert. The wrong Subject in
this
case
includes the host name of a system which has never been an
ipa
client,
but might have been added and removed with ipa host-add
and ipa
host-del
for testing something, a while ago.
As far as I can see, the "cert_subject" is set correctly
in the
file
/var/lib/certmonger/<request id> until the point at
which the
certificate is actually renewed.
I'd be very grateful for some pointers as to which
configuration
options
and logs to check through to resolve this problem on our
production
system.
If its of any relevance we did change which server is the
first
master
some time ago.
I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger
to see
what
the subject is.
I'm not seeing any obvious CSR fields in the
/etc/pki/pki-tomcat/ca/CS.cfg file.
foo.bar.certreq=
The CSR in the certmonger requests file for the
auditSigningCert
seems
to be showing with the correct Subject. This is different
from
the bad
subject showing in the requests file field:
cert_subject=
The value of cert_subject comes from the issued certificate.
and the Subject which is showing in the 'getcert list' output
(which is
the same as that in the cert_subject= field.>
I'm not quite sure what this all means.
It is displayed from the data within the tracked certmonger
request.
certmonger logs to syslog so you can check there or you can
stop
the
process and run it manually with: certmonger -n -d 9 2>&1 |
tee
certmonger.log
That will provide a lot of debugging output that may show
what is
going on.
I've restored certificate databases from backup and put the
clock
back
to a time when certificates are valid and renewed the
ocspSigining
certificate with:
getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i
20161124081302
(I've previously tried without the -N with similar results)
What I am seeing in the certmonger logs is:
2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert
cert-pki-ca'.
2017-10-23 00:05:28 [438] Converted private key
'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [439] Located the certificate
"ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" will not be valid after
20171025122401.
2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert
cert-pki-ca'.
2017-10-23 00:05:28 [442] Converted private key
'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [443] Located the certificate
"ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert
cert-pki-ca'.
2017-10-23 00:05:28 [444] Converted private key
'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:39 [581] Found a certificate with the same
nickname but
different subject, removing certificate "ocspSigningCert
cert-pki-ca"
with subject "CN=OCSP Subsystem,O=<REALM>".
2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
2017-10-23 00:05:39 [583] Located the certificate
"ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:39 [48576] Adding hook
"/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"" (0).
2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" issued by CA and saved.
I now have a date valid ocspSigningCertificate, but with the
wrong
subject, and a broken certificate system which will no longer
start.
ipactl status
...
pki-tomcatd Service: STOPPED
I can't renew other expired certificates.
I also note that there is now no key for
ocspSigningCertificate as
shown
by:
certutil -K -d /etc/pki/pki-tomcat/alias
I wonder if this is because the certificate subject changed?
There
was a
key before the certificate renewed.
The ca debug logs are showing:
[23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by
nickname:
'ocspSigningCert cert-pki-ca' with serial number: 268370108
[23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
x509CertImpl
[23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
Certificate
object not found
Certificate object not found
at
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
Any help in repairing my broken ipa system will be much
appreciated.
Sorry for the delay. I think my previous reading of the
certmonger
csrgen code was wrong.
IIRC in your certmonger entry the cert_subject has the hostname
value
right? Do you also have cert_subject_der?
You can decode that by:
1. Running a hex-to-bin, something hacky like this in python:
import binascii
hex_string = "<hex value>"
binary_string = binascii.unhexlify(hex_string)
fd = open("out", "wb")
fd.write(binary_string)
fd.close()
2. Run: openssl asn1parse -in out -inform der
I'm going to assume this also has the hostname encoded in it.
Hi Again
Yes, correct. The cert_subject_der does have the bad CN with the
hostname encoded in it.
Can you try this:
1. Make a backup of the request file (just in case) > 2. Remove
cert_subject_der
3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
3. Try the renewal again
I ran though all of this, checking that the request file was
still
what
we wanted after restarting certmonger with verbose logging,
restoring
all the databases, fixing permissions, checking keys etc., and
ran the
getcert resubmit.
It still generates a certificate with the bad CN including a
hostname.
Then the pki-tomcat server fails to start again.
Other things I have checked are:
1) The csr= field in the (modified) request file has the correct
subject.
2) The dogtag ca debug log file is showing:
processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
...
RenewalSubmitter: renewal original profileId=caIPAserviceCert
RenewalSubmitter: for renewal, original authenticator raCertAuth
found
CertProcessor: No authenticator credentials required
processRenewal: set original Inputs into profile Context
RenewalSubmitter: setInputsIntoContext() getting input name=
cert_request_type
RenewalSubmitter: setInputsIntoContext() setting value in
ctx:pkcs10
RenewalSubmitter: setInputsIntoContext() getting input name=
cert_request
RenewalSubmitter: setInputsIntoContext() setting value in
ctx:<hex
encoded csr>
...
Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
The hex encoded csr in the last line decodes to have the bad
subject
where the hostname is one of our other ipa servers.
The certificate always getthe same hostname in the bad subject
cn.
Do you have any other ideas of things to check to find out what's
generating the bad subject?
The order certmonger uses for the subject is:
1. cert_subject_der
2. If there is no DER, try to encode cert_subject
3. If that fails try again a different way
4. If it fails yet again use CN=localhost
It is baffling that it would pick a hostname much less one that
certmonger shouldn't even know about.
Indeed, and if I try to renew other certificates for some of
them it
chooses other host names that should not be known about. Each
certificate seems to get the same bad hostname each time I try to
renew.
I assume the CSR found in the dogtag logs matches the csr value
in the
certmonger request?
No, its different. The issued certificate matches the csr seen in
dogtag
which makes sense. But the csr in the dogtag logs has the bad
subject.
The csr in the request directory file has the good subject.
Can you share the certmonger request file privately?
Yes, certainly.
Thanks for your continued help on this.
Let's try this. We'll drop the current request and create a new one.
Stop tomcat, restore the old cert database, start tomcat, then:
# getcert stop-tracking -i <request id>
# getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
"ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P
<pin> -B
/usr/libexec/ipa/certmonger/stop_pkicad -C
'/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"'
Then try resubmitting the request.
Hi Rob
When you say 'stop tomcat', I'm just doing an ipactl stop. That is
stopiing the ipa pki-tomcat service. Is that good enough or are there
other services that need stopping too?
I tried the stop-tracking/start-tracking. Exactly the same result as
before. Same hostname appears in the subject CN field of the new
certificate and then the pki-tomcat service won't start.
I've also been back and redone the manual submits changing the
point at
which I copied in the edited request as I wondered if there was some
caching of csr's going on in certmonger and it was remembering a
previous request it had read on startup before I replaced the
request file.
But nothing seems to stop dogtag issuing a certificate with the
Subject
CN being that host name.
Is it possible that dogtag is somehow overriding the Subject its
being
explicitly given?
FWIW the CSR in the dogtag debug log is always identical, but I guess
thats expected if the CSR information is always the same.
I'm not sure if its possible to increase the verbosity on the dogtag
side. Can you advise please?
I've been assuming that its the bad subject that is preventing the
pki-tomcat from starting after the new certificate is issued. Does
that
make sense to you?
I wonder where we can go from here? There must be a good reason for
whats happening!
Let's step back and gather some more information.
Can you restore the files again and send me the output of:
# getcert list
# certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
cert-pki-ca"
Hi Rob
You should have the output you requested by private email now.
And the exact commands you are using to do all this, including
re-issuing the cert
Thats a great idea. Maybe I'm just doing something in the wrong order.
Using ipactl is fine. You just don't want to touch the NSS databases
while the service is running. Similarly you probably don't want to
mess
with certmonger requests while it is running as it won't see the
changes
until it restarts.
Here is the procedure I'm using to renew the certificates. One of the
tricky things is to not have the certificate system working when
certmonger is started so that it doesn't renew a certificate on
startup that might not be one I have changed the request file for
(other certificates have similar problems with bad subject CN and stop
the certificate system.)
I've found it convenient to keep reinitializing the log files because
they get very confusing when the clock keeps getting set back.
The request number has changed since we did the stop-tracking /
start-tracking.
Roderick
ipactl stop
# Set clock back:
systemctl stop ntpd
timedatectl set-time "2017-10-23 00:00:00"
date
#Stop certificate renewals the systemd way
systemctl stop certmonger
# Sort out certmonger log
mv /var/log/certmonger.log /var/log/certmonger.log-$(date
+%Y%m%d:%H:%M)
# ****** Copy in the fixed certmonger request
\cp /root/20161124081302.ocsp_backup
/var/lib/certmonger/requests/20161124081302
# In another window
certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
# In the original window
# Sort out dogtag log
mv /var/log/pki/pki-tomcat/ca/debug
/var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
touch /var/log/pki/pki-tomcat/ca/debug
chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
mv /var/log/pki/pki-tomcat/ca/selftests.log
/var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
touch /var/log/pki/pki-tomcat/ca/selftests.log
chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
\cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
\cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
/etc/pki/pki-tomcat/alias/cert8.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
/etc/pki/pki-tomcat/alias/key3.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg
/etc/pki/pki-tomcat/ca/CS.cfg
# Fix up certificate permissions and check all is ok
certutil -M -n "caSigningCert cert-pki-ca" -d
/etc/pki/pki-tomcat/alias -t CTu,Cu,Cu
certutil -L -d /etc/pki/pki-tomcat/alias
certutil -K -d /etc/pki/pki-tomcat/alias
<pin>
# Fix dogtag so it starts
pki-server subsystem-enable ca >
# Start IdM services
ipactl start
ipactl status
#Resubmit certificate ocsp
date;getcert resubmit -i 20161124081302
Hi Rob
Does it look like there is anything I should be doing differently in
this command sequence to try to get the certificate renewed correctly?
I'm starting to wonder whether I should look at alternative strategies
for recovering the freeipa servers if the certificates won't renew
correctly. Do you have any ideas on this please?
This looks reasonable. Have you tried stop-tracking and start-tracking I
suggested?
Yes,I tried the stop-tracking / start-tracking. Same result, bad Subject
CN.
I'm still baffled how the name(s) are getting mixed up.
Indeed, something is going round changing the subject. In the dogtag
debug log is:
[23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal:
origNotAfter =Wed Oct 25 13:24:01 BST 2017
[23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal: orig subj
dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
and then very shortly afterwards there is a csr listed that decodes to
have the bad Subject CN.
The listed csr doesn't match any I have been able to find on the system,
so something must be going round and changing it.
Something else you might want to try would be to move all the other
requests out of the way in case there is some sort of memory corruption
causing the hostname to get in there somehow.
Thanks for the suggestion which I have now tried. Same result I'm afraid.
I've now been battling this issue since the beginning of November.
I'm wondering whether, just to get our IPA system fully operational
again, whether we might be able to add the correct CN into a Subject
Alternative name?
Does this sound like a possible way to at least get all the IPA services
started properly?
Do you think it would cause further problems down the line?
Would you be able to advise how to get the certificates to have the
correct CN in the Subject Alternative Name.
Maybe there are some other tricks we could use just to get going again.
Can you advise please?
I'm really stumped here.
Do you have the CA installed on any other masters? You might try setting
it as the renewal master and trying the renewals there instead.
Hi Rob
Thanks for all your thoughts on this.
Some of the certificates have renewed automatically (but incorrectly) on
the other masters. I have two other replicas, but the same sort of thing
happens.
At the moment I have an RHEL 7.3 replica with:
auditSigningCert renewed but with a bad subject CN
ocspSigningCert expired
subsystemCert expired
ipaCert renewed but with bad subject CN
and an RHEL 7.4 replica with:
auditSigningCert renewed but with a same bad subject CN as RHEL 7.3 replica
ocspSigningCert expired
subsystemCert expired
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem', which I
think is the equivalent of ipaCert, renewed but with the same bad
subject CN as RHEL 7.3 replica
All the other certificates seem to have plausible Subjects and are valid.
I'm not sure that this this helps me much though.
I wondered whether the CSR is getting the bad subject because its
accessing a wrong certificate profile. Its almost like its trying to get
an IPA service certificate, but I can't see how a certificate is tied to
a profile.
For the oscpSigningCert I can see in the dogtag access logs on the host
we have been using all along:
"GET
/ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=2&renewal=true&xml=true
HTTP/1.1" 200 139
"GET /ca/agent/ca/profileReview?requestId=9997550&xml=true HTTP/1.1" 200
14421
"GET
/ca/agent/ca/profileProcess?requestId=9997550&xml=true&op=approve&name=CN%3D<fqdn
of host3>%2CO%3D<REALM>
Anyway, I clearly need to trace the certificate renewal process in more
detail. In the certmonger logs it just says:
Request2('20161124081302') moved to state 'GENERATING_CSR
and doesn't give any clue as to how its doing this.
Do you think there is anyone else who might be able to help with this?
Thanks.
Roderick
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
