On 11/20/19 8:13 PM, Natxo Asenjo via FreeIPA-users wrote:
hi,
after patching our centos 7 hosts to the latest version today, one of
the two replicas is having trouble.
[root@kdc2 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
named Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: STOPPED
pki-tomcatd Service: RUNNING
smb Service: STOPPED
winbind Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful
and after digging in the logs I come across this in /var/log/ipaupgrade.log:
2019-11-20T18:18:29Z DEBUG stderr=
2019-11-20T18:18:31Z INFO Certmonger certificate renewal configuration
already up-to-date
2019-11-20T18:18:31Z INFO [Enable PKIX certificate path discovery and
validation]
2019-11-20T18:18:31Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-11-20T18:18:31Z INFO PKIX already enabled
2019-11-20T18:18:31Z INFO [Authorizing RA Agent to modify profiles]
2019-11-20T18:18:31Z INFO [Authorizing RA Agent to manage lightweight CAs]
2019-11-20T18:18:31Z INFO [Ensuring Lightweight CAs container exists in
Dogtag database]
2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740162547472
2019-11-20T18:18:31Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache
2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc24b638>
2019-11-20T18:18:31Z DEBUG Destroyed connection
context.ldap2_139740162547472
2019-11-20T18:18:31Z INFO [Adding default OCSP URI configuration]
2019-11-20T18:18:31Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2019-11-20T18:18:31Z INFO [Migrating certificate profiles to LDAP]
2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740160021648
2019-11-20T18:18:31Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache
2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc289b00>
2019-11-20T18:18:31Z DEBUG Destroyed connection
context.ldap2_139740160021648
2019-11-20T18:18:31Z DEBUG request GET
https://kdc2.l.domain.it:8443/ca/rest/account/login
2019-11-20T18:18:31Z DEBUG request body ''
2019-11-20T18:18:31Z DEBUG response status 401
2019-11-20T18:18:31Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 01:00:00 CET
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Wed, 20 Nov 2019 18:18:31 GMT
2019-11-20T18:18:31Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 401 - </h1><HR size="1"
noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
<u></u></p><p><b>description</b> <u>This request requires HTTP
authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-11-20T18:18:31Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2019-11-20T18:18:31Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
server.upgrade()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2146, in upgrade
upgrade_configuration()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2018, in upgrade_configuration
ca_enable_ldap_profile_subsystem(ca)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 406, in ca_enable_ldap_profile_subsystem
cainstance.migrate_profiles_to_ldap()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
2027, in migrate_profiles_to_ldap
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
2033, in _create_dogtag_profile
with api.Backend.ra_certprofile as profile_api:
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",
line 1315, in __enter__
raise errors.RemoteRetrieveError(reason=_('Failed to authenticate
to CA REST API'))
2019-11-20T18:18:31Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2019-11-20T18:18:31Z ERROR Unexpected error - see
/var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
The authentication between IPA and dogtag is done using the ra-agent
cert located in /var/lib/ipa/ra-agent.pem. As its expiration date is
near, it's possible that the renewal process for this cert started but
did not complete successfully.
You need to check the following:
- note the serial ID of the cert, its subject and issuer:
$ openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem
You can also check if it has already been renewed (look at the date Not
Before / Not After).
If it has been renewed, check the content of the entry
uid=ipara,ou=people,o=ipaca:
$ ldapsearch -D cn=directory\ manager -W -b uid=ipara,ou=people,o=ipaca
There are 2 things to check:
- The userCertificate attribute must contain the cert (same value as in
ra-agent.pem, in a single line and without the -----BEGIN
CERTIFICATE----- and -----END CERTIFICATE---- lines).
- The description attribute must have the foollowing value:
description: 2;<serial number retrieved in previous step>;<issuer>;<subject>
If it's not the case, it's likely that the renewal failed to update the
entry and that may be causing your issue. You will need to manually fix
the entry using ldapmodify.
After that, restart ipa with ipactl stop / ipactl start and check if
certmonger is able to renew the other certs that will expire soon.
HTH,
flo
In this kdc I see these errors in getcert list:
Request ID '20190220182014':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
subject: CN=CA Audit,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
expires: 2019-12-05 13:58:24 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190220182015':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
subject: CN=OCSP Subsystem,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
expires: 2019-12-05 13:58:24 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190220182016':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
subject: CN=CA Subsystem,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
expires: 2019-12-05 13:58:24 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190220182018':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
subject: CN=IPA RA,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
expires: 2019-12-05 13:58:44 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20190220182019':
status: MONITORING
ca-error: Server at
"https://kdc2.l.domain.it:8443/ca/agent/ca/profileProcess"; replied: 1:
Invalid Credential.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
subject: CN=kdc2.l.domain.it
<http://kdc2.l.domain.it>,O=L.DOMAIN.IT <http://L.DOMAIN.IT>
expires: 2019-12-10 10:57:52 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
I still have a working replica, so I could just reinstall and have a
working set in a couple of minutes, but I would like to find out what
has gone wrong.
The systems are running ipa-server-4.6.5-11.el7.centos.3.x86_64
Any help welcome ;-)
Thanks,
--
Groeten,
natxo
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
