> -----Original Message----- > From: Martin Kosek [mailto:mko...@redhat.com] > Sent: Wednesday, 25 February 2015 10:35 PM > To: Les Stott; Rob Crittenden; freeipa-users@redhat.com; Endi Dewata; Jan > Cholasta > Subject: Re: [Freeipa-users] ipa-getcert list fails to report correctly - > RESOLVED > > On 02/25/2015 03:11 AM, Les Stott wrote: > > > > > >> -----Original Message----- > >> From: freeipa-users-boun...@redhat.com [mailto:freeipa-users- > >> boun...@redhat.com] On Behalf Of Les Stott > >> Sent: Monday, 23 February 2015 8:01 PM > >> To: Rob Crittenden; Martin Kosek; freeipa-users@redhat.com; Endi > >> Dewata; Jan Cholasta > >> Subject: Re: [Freeipa-users] ipa-getcert list fails to report > >> correctly > >> > >> > >> > >>> -----Original Message----- > >>> From: freeipa-users-boun...@redhat.com [mailto:freeipa-users- > >>> boun...@redhat.com] On Behalf Of Les Stott > >>> Sent: Monday, 23 February 2015 12:18 PM > >>> To: Rob Crittenden; Martin Kosek; freeipa-users@redhat.com; Endi > >>> Dewata; Jan Cholasta > >>> Subject: Re: [Freeipa-users] ipa-getcert list fails to report > >>> correctly > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: Rob Crittenden [mailto:rcrit...@redhat.com] > >>>> Sent: Saturday, 21 February 2015 1:39 AM > >>>> To: Martin Kosek; Les Stott; freeipa-users@redhat.com; Endi Dewata; > >>>> Jan Cholasta > >>>> Subject: Re: [Freeipa-users] ipa-getcert list fails to report > >>>> correctly > >>>> > >>>> Martin Kosek wrote: > >>>>> On 02/20/2015 06:56 AM, Les Stott wrote: > >>>>>> Hi all, > >>>>>> > >>>>>> The following is blocking the ability for me to install a CA replica. > >>>>>> > >>>>>> Environment: > >>>>>> > >>>>>> RHEL 6.6 > >>>>>> > >>>>>> IPA 3.0.0-42 > >>>>>> > >>>>>> PKI 9.0.3-38 > >>>>>> > >>>>>> On the master the following is happening: > >>>>>> > >>>>>> ipa-getcert list > >>>>>> > >>>>>> Number of certificates and requests being tracked: 5. > >>>>>> > >>>>>> (but it shows no certificate details in the output) > >>>>>> > >>>>>> Running "getcert list" shows complete output. > >>>>>> > >>>>>> Also, when trying to browse > >>>>>> https://master.mydomain.com/ca/ee/ca/getCertChain i get a failed > >>>>>> response. The apache error logs on the master show.... > >>>>>> > >>>>>> [Thu Feb 19 23:23:23 2015] [error] SSL Library Error: -12271 SSL > >>>>>> client cannot verify your certificate > >>>>>> > >>>>>> The reason I am trying to browse that address is because that's > >>>>>> what the ipa-ca-install setup is failing at (it complains that > >>>>>> the CA certificate is not in proper format, in fact it's not able > >>>>>> to get it at all). > >>>>>> > >>>>>> I know from another working ipa setup that .... > >>>>>> > >>>>>> Browsing to the above address provides valid xml content and > >>>>>> ipa-getcert list shows certificate details and not just the > >>>>>> number of tracked certificates. > >>>>>> > >>>>>> Been trying for a long time to figure out the issues without luck. > >>>>>> > >>>>>> I would greatly appreciate any help to troubleshoot and resolve > >>>>>> the above issues. > >>>>>> > >>>>>> Regards, > >>>>>> > >>>>>> Les > >>>>> > >>>>> Endi or JanC, would you have any advise for Les? To me, it looks > >>>>> like the Apache does not have proper certificate installed. > >>>>> > >>>>> My ipa-getcert on RHEL-6.6 shows 3 Server-Certs tracked, making it > >>>>> in total of 8 certs tracked: > >>>>> > >>>>> # ipa-getcert list > >>>>> Number of certificates and requests being tracked: 8. > >>>>> Request ID '20141111000002': > >>>>> status: MONITORING > >>>>> stuck: no > >>>>> key pair storage: > >>>>> type=NSSDB,location='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT- > >>>> COM',nicknam > >>>>> e='Server-Cert',token='NSS > >>>>> Certificate > >>>>> DB',pinfile='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT- > COM/pwdfile.txt' > >>>>> certificate: > >>>>> type=NSSDB,location='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT- > >>>> COM',nicknam > >>>>> e='Server-Cert',token='NSS > >>>>> Certificate DB' > >>>>> CA: IPA > >>>>> issuer: CN=Certificate Authority,O=EXAMPLE.COM > >>>>> subject: CN=vm-086.example.com,O=EXAMPLE.COM > >>>>> expires: 2016-11-11 00:00:01 UTC > >>>>> key usage: > >>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >>>>> eku: id-kp-serverAuth,id-kp-clientAuth > >>>>> pre-save command: > >>>>> post-save command: > >>>>> track: yes > >>>>> auto-renew: yes > >>>>> Request ID '20141111000047': > >>>>> status: MONITORING > >>>>> stuck: no > >>>>> key pair storage: > >>>>> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server- > >> Cert' > >>>>> ,token='NSS Certificate > >>>>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > >>>>> certificate: > >>>>> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server- > >> Cert' > >>>>> ,token='NSS > >>>>> Certificate DB' > >>>>> CA: IPA > >>>>> issuer: CN=Certificate Authority,O=EXAMPLE.COM > >>>>> subject: CN=vm-086.example.com,O=EXAMPLE.COM > >>>>> expires: 2016-11-11 00:00:46 UTC > >>>>> key usage: > >>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >>>>> eku: id-kp-serverAuth,id-kp-clientAuth > >>>>> pre-save command: > >>>>> post-save command: > >>>>> track: yes > >>>>> auto-renew: yes > >>>>> Request ID '20141111000302': > >>>>> status: MONITORING > >>>>> stuck: no > >>>>> key pair storage: > >>>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',toke > >>>>> n= 'N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > >>>>> certificate: > >>>>> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',toke > >>>>> n= > >>>>> 'N > >>>>> SS > >>>>> Certificate DB' > >>>>> CA: IPA > >>>>> issuer: CN=Certificate Authority,O=EXAMPLE.COM > >>>>> subject: CN=vm-086.example.com,O=EXAMPLE.COM > >>>>> expires: 2016-11-11 00:03:02 UTC > >>>>> key usage: > >>>>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >>>>> eku: id-kp-serverAuth,id-kp-clientAuth > >>>>> pre-save command: > >>>>> post-save command: > >>>>> track: yes > >>>>> auto-renew: yes > >>>>> > >>>>> > >>>>> What is actually in your Apache NSS database? > >>>>> > >>>>> # certutil -L -d /etc/httpd/alias/ > >>>>> > >>>>> Martin > >>>>> > >>>> > >>>> Remember ipa-getcert is just a shortcut for certificates using the > >>>> certmonger CA named IPA, so it's more a filter than anything else. > >>>> I don't know why it wouldn't display any output but I'd file a bug. > >>>> > >>>> I think we'd need to see the getcert list output to try to figure > >>>> out what is going on. > >>>> > >>>> As for the SSL error fetching the cert chain I think Martin may be > >>>> onto something. The request is proxied through Apache. I think the > >>>> client here might be the Apache proxy client. > >>>> > >>>> I believe this command replicates what Apache is doing, you might > >>>> give it a try on the master. This will get the chain directly from > >>>> dogtag, bypassing > >>>> Apache: > >>>> > >>>> $ curl -v --cacert /etc/ipa/ca.crt > >>>> https://`hostname`:9444/ca/ee/ca/getCertChain > >>>> > >>>> rob > >>> > >>> Certutil shows.... > >>> > >>> certutil -L -d /etc/httpd/alias/ > >>> > >>> Certificate Nickname Trust > >>> Attributes > >>> > >>> SSL,S/MIME,JAR/XPI > >>> > >>> MYDOMAIN.COM IPA CA CT,C,C > >>> ipaCert u,u,u > >>> Signing-Cert u,u,u > >>> Server-Cert u,u,u > >>> > >>> curl -v --cacert /etc/ipa/ca.crt > >>> https://`hostname`:9444/ca/ee/ca/getCertChain > >>> * About to connect() to `hostname` port 9444 (#0) > >>> * Trying 192.168.1.1... connected > >>> * Connected to `hostname` (192.168.1.1) port 9444 (#0) > >>> * Initializing NSS with certpath: sql:/etc/pki/nssdb > >>> * CAfile: /etc/ipa/ca.crt > >>> CApath: none > >>> * SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA > >>> * Server certificate: > >>> * subject: CN=`hostname`,O=MYDOMAIN.COM > >>> * start date: Dec 13 01:21:30 2013 GMT > >>> * expire date: Dec 03 01:21:30 2015 GMT > >>> * common name: `hostname` > >>> * issuer: CN=Certificate Authority,O=MYDOMAIN.COM > >>>> GET /ca/ee/ca/getCertChain HTTP/1.1 > >>>> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 > >>>> NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > >>>> Host: `hostname`:9444 > >>>> Accept: */* > >>>> > >>> < HTTP/1.1 200 OK > >>> < Server: Apache-Coyote/1.1 > >>> < Content-Type: application/xml > >>> < Content-Length: 1434 > >>> < Date: Mon, 23 Feb 2015 01:04:29 GMT < <?xml version="1.0" > >>> encoding="UTF-8" > >>> > >> > standalone="no"?><XMLResponse><Status>0</Status><ChainBase64>MIID > >>> > >> > zwYJKoZIhvcNAQcCoIIDwDCCA7wCAQExADAPBgkqhkiG9w0BBwGgAgQAoII > >>> > >> > DoDCCA5wwggKEoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwOjEYMBYGA1U > >>> > >> > EChMPREVSSVZBVElWRVMuQ09NMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSB > >>> > >> > BdXRob3JpdHkwHhcNMTMxMjEzMDEyMTI5WhcNMzMxMjEzMDEyMTI5Wj > >>> > >> > A6MRgwFgYDVQQKEw9ERVJJVkFUSVZFUy5DT00xHjAcBgNVBAMTFUNlcnRp > >>> > >> > ZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCg > >>> > >> > gEBAMAA8EaYhmpjSA8o3/1kB/W1+0K6+FrwCS+njOgRtXhiTdmtSddXSDVxH > >>> > >> > OafFwqN26BR+QRPZbbpJY70gP3SG8W+J6+c37PMVNshWz6UfChGt6ubgFxlS > >>> > >> > TGUUre2Osr9I4C836MXpGJvRx2VDEuMUxv8j7B9iDRnTDglseqPqrMct2No4w > >>> > >> > k4cLtA9puBJb0Es76SOHP9edXlf6GBnuYwR8YMc1yJLqpP8IGpHhEkVxMsRpqk > >>> > >> > EpuuRwEFa7uBcTDhqVV24BpFlseZVubpiOdEgfb3IRBTjvI1Mum9OCJbuj9P/W > >>> > >> > mqMnrA0sQsmF/R3WBwFdMAsN3+bQCRw73+rwoeDNcCAwEAAaOBrDCBq > >>> > >> > TAfBgNVHSMEGDAWgBSO8J+j2jAuyg3a0yE+3oVCQJCWUTAPBgNVHRMBAf8 > >>> > >> > EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUjvCfo9owLsoN > >>> > >> > 2tMhPt6FQkCQllEwRgYIKwYBBQUHAQEEOjA4MDYGCCsGAQUFBzABhipodHR > >>> wOi8vc2I! > >>> ybW9uMDEuZGVyaXZhdGl2ZXMuY29tOjgwL2* Connection #0 to host > >> `hostname` > >>> left intact > >>> * Closing connection #0 > >>> > >> > NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAKH8YkoTAzX2xNYMkZSDK84EK3 > >>> > >> > e4FUixdXxc/EC5ehjrtaqXT1KT9Fl9DAF5/jYNKqgmEmtHnPGlfQ7/Y1ESdhEGcB > >>> > >> > ZjU4qLe4HaFXuw5c9odDYxhtjQUd1g7ifY8SKOcHDCY+6Xx6F/rhFgzrXXMndn8 > >>> > >> > ZaYryctPoOAj/5INnLrJq8S4XyLmb2BHM4e1ORQbOhDi8xjhfK2veYXvIu55Brhp > >>> > >> > RSS/goz5oSE8e+QE/H9afRmeV2+WkS/YDhSyoUDb7CYjklRuONzX3GopKtp1y > >>> > >> > yLXQZnBFjCvIJvja0mo3ik3AXxSZuOwUIlV23U8CyPU/rDeiV00iUyA/fLvdkEtZkx > >>> AA==</ChainBase64></XMLResponse> > >>> > >>> > >>> In any event, I've decided to rebuilt my DR IPA environment. Late > >>> last year the master in DR had to be rebuilt due to a disk issue. > >>> While IPA was restored manually and appeared to be working fine, CA > >>> replication hasn't worked. I finally got CA replication working in > >>> Prod after enabling needed apache modules and performing a yum > >>> update to update related packages, but these things didn't help in > >>> DR. It's my strong suspicion that something got missed when > >>> restoring the DR master IPA server and this is what is causing all > >>> my grief. Therefore, I'm going to wipe it > >> out and start from scratch in DR. > >>> There are other benefits for me to do this anyway. > >>> > >> > >> Well things have gone from bad to worse. > >> > >> I removed IPA in DR. uninstalled all ipa clients, uninstalled > >> replicas, removed replication agreements and removed the master. Ran > >> pki-remove to clear any leftover pki instances and used certutil -D > >> to remove left behind ipa entries in /etc/httpd/alias. > >> > >> So, clean slate to start again. > >> > >> This time, in order to mirror config with prod, I began an > >> installation for the master on a different server, let's call it > >> serverb. It was previously a replica (in my prod environment, serverb > >> is the true master, servera, serverc, and serverd are replicas). > >> > >> So, trying to install a new fresh instance of IPA and it still fails > >> to configure a CA. > >> > >> Attached is the relevant portion of the server install log file > >> (ipa-server- install.txt). I have removed certificate and copyright info to > reduce its size. > >> Also my server to install is serverb.mydomain.com > >> > >> Apache logs at the time of the error show: > >> [Mon Feb 23 03:05:31 2015] [error] SSL Library Error: -12195 Peer > >> does not recognize and trust the CA that issued your certificate > >> > >> Certificate databases only show the following (note that "Server-Cert > >> cert- pki-ca" got installed before the installer crashed). Prior to > >> trying installation I had to manually remove server certs left behind > >> from the previous installation via ... > >> certutil -d /etc/httpd/alias -D -n "Server-Cert" > >> certutil -d /etc/httpd/alias -D -n "MYDOMAIN.COM IPA CA" > >> certutil -d /etc/httpd/alias -D -n ipaCert > >> > >> certutil -L -d /var/lib/pki-ca/alias > >> Certificate Nickname Trust > >> Attributes > >> > >> SSL,S/MIME,JAR/XPI > >> Server-Cert cert-pki-ca CTu,Cu,Cu > >> > >> certutil -L -d /etc/pki/nssdb > >> Certificate Nickname Trust > >> Attributes > >> > >> SSL,S/MIME,JAR/XPI > >> > >> > >> Selinux is in permissive mode. > >> Ausearch -m avc does show some selinux issues, but its permissive > >> mode so it should be ok right? In any event I have previously tried > >> installing a CA replica with selinux disabled and it didn't help. > >> > >> I have tried removing ipa and pki rpms and reinstalling. Then > >> rerunning the ipa server install script but the same error occurs. > >> > >> I noticed that /etc/ipa/ca.crt was still old, and referencing the original > master. > >> I removed that and again reran the installer but the same error occurred. > >> > >> Note also that /etc/ipa/cr.crt was not recreated when ipa-python was > >> reinstalled. > >> > >> Other logs: > >> > >> /var/log/pki-ca/system shows > >> 5042.main - [23/Feb/2015:03:05:12 EST] [3] [3] Cannot build CA chain. > >> Error > >> java.security.cert.CertificateException: Certificate is not a PKCS > >> #11 certificate 5042.main - [23/Feb/2015:03:05:12 EST] [13] [3] authz > >> instance DirAclAuthz initialization failed and skipped, > >> error=Property internaldb.ldapconn.port missing value > >> 5042.http-9445-1 - [23/Feb/2015:03:05:26 EST] [3] [3] Cannot build CA > chain. > >> Error java.security.cert.CertificateException: Certificate is not a > >> PKCS #11 certificate > >> 5042.http-9445-1 - [23/Feb/2015:03:05:35 EST] [3] [3] CASigningUnit: > >> Object certificate not found. Error > >> org.mozilla.jss.crypto.ObjectNotFoundException > >> > >> /var/log/pki-ca/catalina.out > >> Feb 23, 2015 3:05:11 AM org.apache.catalina.startup.HostConfig > >> deployDirectory > >> INFO: Deploying web application directory ca 64-bit osutil library > >> loaded 64-bit osutil library loaded CMS Warning: FAILURE: Cannot > >> build CA chain. Error > >> java.security.cert.CertificateException: Certificate is not a PKCS > >> #11 > >> certificate|FAILURE: authz instance DirAclAuthz initialization failed > >> certificate|and > >> skipped, error=Property internaldb.ldapconn.port missing value| > >> Server is started. > >> Feb 23, 2015 3:05:12 AM org.apache.coyote.http11.Http11Protocol start > >> INFO: Starting Coyote HTTP/1.1 on http-9180 Feb 23, 2015 3:05:12 AM > >> org.apache.coyote.http11.Http11Protocol start > >> INFO: Starting Coyote HTTP/1.1 on http-9443 Feb 23, 2015 3:05:12 AM > >> org.apache.coyote.http11.Http11Protocol start > >> INFO: Starting Coyote HTTP/1.1 on http-9445 Feb 23, 2015 3:05:12 AM > >> org.apache.coyote.http11.Http11Protocol start > >> INFO: Starting Coyote HTTP/1.1 on http-9444 Feb 23, 2015 3:05:12 AM > >> org.apache.coyote.http11.Http11Protocol start > >> INFO: Starting Coyote HTTP/1.1 on http-9446 Feb 23, 2015 3:05:12 AM > >> org.apache.jk.common.ChannelSocket init > >> INFO: JK: ajp13 listening on /0.0.0.0:9447 Feb 23, 2015 3:05:12 AM > >> org.apache.jk.server.JkMain start > >> INFO: Jk running ID=0 time=0/25 config=null Feb 23, 2015 3:05:12 AM > >> org.apache.catalina.startup.Catalina start > >> INFO: Server startup in 1655 ms > >> > >> I have no idea where to look next. There must be some remnant of the > >> old system hanging around screwing things up but I cannot figure it > >> out. This will drive me insane! > >> > >> I can provide more logs if needed. > >> > >> Thanks in advance for any help. > >> > > > > Have resolved this. > > Great! Thanks for reaching back to us. > > > Here is the procedure to completely remove FreeIPA so you can start > again. > > To me, that sounds like the FreeIPA uninstaller is missing some clean up > steps. > I would personally rather resolve it in the the actual code than just having > this > information in the list archives. > > > > > ipa-server-install --uninstall > > certutil -d /etc/httpd/alias -D -n "Server-Cert" > > certutil -d /etc/httpd/alias -D -n "DERIVATIVES.COM IPA CA" > > certutil -d /etc/httpd/alias -D -n ipaCert certutil -d > > /etc/httpd/alias -D -n Signing-Cert > > This sounds like https://fedorahosted.org/freeipa/ticket/4639. We should > bump the priority if it is really causing issues. >
Yes, definitely experienced this behaviour. > > yum -y remove pki-selinux pki-ca pki-common pki-setup pki-silent > > pki-java-tools pki-symkey pki-util pki-native-tools ipa-server-selinux > > ipa-server ipa-client ipa-admintools ipa-python ipa-pki-ca-theme > > ipa-pki-common-theme 389-ds-base 389-ds-base-libs userdel pkisrv > > userdel pkiuser > > This should not be needed at all, AFAIK. > Possibly not, but wanted to start with a clean system without having to reinstall the OS from scratch. > > rm -rf /etc/pki-ca /var/lib/pki-ca /var/log/pki-ca /etc/certmonger > > /etc/sysconfig/pki-ca /etc/sysconfig/pki /var/run/pki-ca.pid > > /usr/share/pki /etc/ipa /var/log/ipa* reboot > > > > Now you have a clean slate. > > Do you know which step of the steps above actually helped you resolve the > reinstall issue? > The reboot I think was key to the whole process, but pki remnants seemed left behind too which caused grief. Previously I had never rebooted the system in between uninstall/reinstall. /etc/ipa/ca.crt was also left behind. It caused an issue during one reinstall as it never got updated and the install bombed out because it found a mismatched cert. This led me to deleting all possible ipa/pki directories and then removing/reinstalling rpms to restore to default state. I noticed that in some cases (I went through this same process on 6 servers to reinstall and setup CA replicas) I could still see a left over process running as the pkiuser (tomcat/java) which stopped the "userdel pkiuser" command from completing. I had to kill that process and then userdel pkiuser worked. Regards, Les > > > > Then install works as normal for IPA Server, Replica and CA Replica > installations. > > > > Hope this saves someone else time in the future. > > > > Regards, > > > > Les > > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project