Forgot to reply again - ugh! Hmmmm, so my domain is actually "idm.project.its.srv2", so I was literally typing "systemctl start dir...@idm.project.its.srv2" I see what you're saying, I need to put in dashes instead of periods! DOH! Done. Moving on... 4) Ran systemctl start krb5kdc 5) Ran systemctl start kadmin 6) Ran systemctl start named-pkcs11 7) Ran systemctl start httpd - got an error here, nothing really useful in the logs or journalctl, it says it's starting the Apache HTTP server, then throws "httpd.service: main process exited, code=exited, status=1/FAILURE", and "Failed to start The Apache HTTP Server". Finally there is a mention of 'too much time skew'. I assume the problem is that I'm trying to start HTTPD on a system where the date is almost a year old. Although now that I'm looking at /var/log/httpd/error_log, I see mention of "SSL Library Error: -8181 Certificate has expired". CERTIFICATES!!! "Unable to verify certificate 'Server-Cert'. Add "NSSEnfroceValideCerts off" to nss.conf so the server can start until the problem can be resolved", so maybe I'll try that. Scott
________________________________ From: Florence Blanc-Renaud <f...@redhat.com> Sent: Tuesday, August 11, 2020 6:55 AM To: Scott Z. <sud...@hotmail.com>; FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Rob Crittenden <rcrit...@redhat.com> Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting On 8/11/20 6:39 PM, Scott Z. wrote: > First thing I did when I logged in this morning (I'm on Hawaii Standard > Time) was run "ipactl status". The return was "Directory Services: > STOPPED", and "Directory Service must running in order to obtain status > of other services". > 1) Ran "getcert list", and it shows the 9 certs being tracked (all the > previous 8 plus the 1 expired guy I added yesterday). All look good > except of course my problem child, who's status is CA_UNREACHABLE and > ca-error is Internal error. > 2) Ran "ipa stop", looks like all service stopped successfully. > 2) Changed date back to Sept. 1, 2019. > 3) Ran the "systemctl start dirsrv@<domain> and got back "Job for > dirsrv@<domain> failed because a configured resource limit was exceeded." >     a. when I looked at "journalctl -xe", I just see a couple of > messages that don't tell me much... "Registered Authentication Agent for > unix-process:<blahblah>", followed by "Failed to load environment files: > no such files or directory". Then, "dirsrv@<domain> filed to run > 'start-pre' task: No such files or directory" and finally "Failed to > start 389 Directory Server <domain>". > If your domain is domain.com, you need to run systemctl start dirsrv@DOMAIN-COM I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM which would produce the error you're seeing. flo > Not sure now how to proceed at this point. > > BTW, I have decided that once I get through this slog and have a working > server again, I'm going to donate $50 to the Hawaiian Food Bank or the > charity of your choice in appreciation. > Scott > > > ------------------------------------------------------------------------ > *From:* Florence Blanc-Renaud <f...@redhat.com> > *Sent:* Monday, August 10, 2020 8:55 PM > *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Rob > Crittenden <rcrit...@redhat.com> > *Cc:* Scott Z. <sud...@hotmail.com> > *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting > On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote: >> I stopped the ntp service with the command "timedatectl set_ntp 0" >> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time >> 2019-09-01" >> I waiting a minute and then checked with the "date" command; the problem >> server believes it is Sept. 1st, 2019. >> >> Now when you say 'restart services', I assume you're only referring to >> the ipactl services? In that case I ran "ipactl start >> --ignore-service-failures". Interestingly, when I ran this command it >> not only failed to start pki-tomcatd (which I expected), but actually >> reset the date back to the present/correct time and date. Thus, I >> re-ran the command to set it back to Sept. 1st, 2019. >> > If the server was configured with ntp, "ipactl start" will also restart > ntpd. You need to do the following: > ipactl stop > change date in the past > systemctl start dirsrv@DOMAIN-COM (replace with your domain name) > systemctl start krb5kdc > systemctl start kadmin > systemctl start named-pkcs11 (if IPA is hosting the DNS server) > systemctl start httpd > systemctl start pki-tomcatd@pki-tomcat > > Then try getcert resubmit. > >> I then ran the "getcert resubmit -i <reqID> command. I just now went >> through these steps again, and it's showing "status: CA_UNREACHABLE" and >> "ca-error: Internal Error". Stuck now shows 'no'. >> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert >> cert-pki-ca' now yields a new error message, "certutil: could not find >> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not >> found" > The cert nickname should contain a dash: "Server-Cert cert-pki-ca" > > HTH, > flo >> >> Many Mahalos for your continued support and patience! >> Scott >> >> >> >> >> ------------------------------------------------------------------------ >> *From:* Rob Crittenden <rcrit...@redhat.com> >> *Sent:* Monday, August 10, 2020 11:36 AM >> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>; >> Florence Blanc-Renaud <f...@redhat.com> >> *Cc:* Scott Z. <sud...@hotmail.com> >> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting >> Scott Z. via FreeIPA-users wrote: >>> Whoops! Using the additional command to start tracking this >>> paritcular >>> cert that you included in a different message, I got it in the "getcert" >>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d >>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B >>> /usr/libexec/ipa/certmonger/stop_pkicad -C >>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P >>> <pin>" command). >>> >>> I have the date rolled back to Sept. 1st, 2019. I guess I have >>> 'some' >>> progress now at least, but still have an issue; checking on the >>> cert >>> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and >>> "stuck: yes". >> >> How did you roll the date back? Did you restart services? What date did >> you pick and does it overlap so that all certs are valid? >> >> rob >> >>> >>> Any additional thoughts or help would be greatly appreciated! And >>> thanks for the help so far. >>> Scott >>> >>> ------------------------------------------------------------------------ >>> *From:* Scott Z. via FreeIPA-users <freeipa-users@lists.fedorahosted.org> >>> *Sent:* Monday, August 10, 2020 10:37 AM >>> *To:* Florence Blanc-Renaud <f...@redhat.com> >>> *Cc:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Scott >>> Z. <sud...@hotmail.com> >>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting >>>  >>> Sorry, I didn't realize I had dropped the mailing list - my mistake! >>> >>> I backed up the files/directories you mentioned below, then I checked on >>> the ra-agent.pem to see if it was still valid (openssl x509 -in >>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed >>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: >>> Aug >>> 10 17:20:41 2021 GMT). >>> >>> Based on that information, and knowing that the bad cert is valid from >>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019 >>> since all certs will see that date as valid. >>> >>> The only issue I have now is getting the request ID for the expired >>> cert; it doesn't show up in the list of certs when I do "getcert -list", >>> I can only see it by running "certutil -L -d >>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when >>> I run that it does not show any Request ID associated for it? >>> Scott >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Florence Blanc-Renaud <f...@redhat.com> >>> *Sent:* Monday, August 10, 2020 8:45 AM >>> *To:* Scott Z. <sud...@hotmail.com> >>> *Cc:* FreeIPA users list <freeipa-users@lists.fedorahosted.org> >>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting >>>  >>> Hi, >>> >>> re-adding the mailing list as the conversation could also help others. >>> >>> On 8/8/20 12:06 AM, Scott Z. wrote: >>>> I did notice when I compare it to another IdM server in the environment, >>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a >>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that >>>> I'm comparing against has a "Signing-Cert" certificate in >>>> addition. Is >>>> this because it's the 'Master' or whatever? Should my >>>> 'bad' server have >>>> this same Signing-Cert listed? >>> >>> /etc/httpd/alias only needs its own Server-Cert + IPA CA. >>> >>>> Scott >>>> >>>> ------------------------------------------------------------------------ >>>> *From:* Scott Z. <sud...@hotmail.com> >>>> *Sent:* Friday, August 7, 2020 10:44 AM >>>> *To:* Florence Blanc-Renaud <f...@redhat.com> >>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting >>>> /"The interesting part is the list of expired certs on the failing node >>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed >>>> instructions are available here: >>>> https://access.redhat.com/solutions/3357331 How do I manually renew >>>> Identity Management (IPA) certificates on RHEL7 after they have expired? >>>> (Replica IPA Server)"/ >>> >>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias, >>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where >>> the certificates are stored). >>> >>> If the RA cert is valid, you need to find a time window during which the >>> RA cert is already valid (date > notbefore) and the other certs are not >>> expired yet (date < notafter). When you have identified a proper date, >>> stop ntpd (or chronyd, depending on which service is used for time >>> synchronization), move the date back in time to the identified date, >>> start all the services except ntpd, then call "getcert resubmit -i >>> <request id>" for the expired cert(s). >>> >>> Check that the cert has been renewed with "getcert list -i <request >>> id>", the state should display MONITORING. When all the certs are good, >>> you can restart ntpd and the clock will go back to the current date. >>> >>> It's really important to find a date where all the certs are valid >>> because this ensures that the services are able to start and the RA cert >>> allows the authentication that is mandatory for certificate renewal. >>> >>> HTH, >>> flo >>>> >>>> Sadly, after I log in, it's only telling me that it's "Subscriber >>>> Exclusive Content". Not sure what happened with my >>>> account, I used to >>>> be able to access these docs with no problem but since I took a RHEL >>>> class a couple of weeks back now it's not working any >>>> more. I guess >>>> they did something to screw up my account when I took the class. Grrrrr!!! >>>> Scott >>>> >>>> ------------------------------------------------------------------------ >>>> *From:* Florence Blanc-Renaud <f...@redhat.com> >>>> *Sent:* Thursday, August 6, 2020 2:46 AM >>>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org> >>>> *Cc:* Scott Z. <sud...@hotmail.com> >>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting >>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote: >>>>> Thanks much for the >>>>> assistance. Here is where I >>>>> am with your suggestions: >>>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old >>>>> (almost a year old actually, I assume IPA only checks it when it first >>>>> starts up so it didn't care that it was expired until the server was >>>>> rebooted?) >>>> >>>> certmonger checks the certificate validity periodically (configurable in >>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs. >>>> The system probably had an issue that was not detected and the cert >>>> reached its expiration date. >>>> >>>>> >>>>> 2) ran ipactl start --ignore-service-failures >>>>> Â >>>>> >>>>> >>>>> >>>>> >>>>> a. most services started, >>>>>obviously pki-tomcatd did not >>>>> 3) ran "kinit admin" >>>>> Â >>>>> >>>>> >>>>> >>>>> >>>>> a. was forced to change the >>>>>password, but otherwise nothing happened >>>>> 4) Ran "ipa config-show |grep -i master >>>>> Â >>>>> >>>>> >>>>> >>>>> a. I see that the IPA CA >>>>>renewal master is a different idm machine. >>>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:" >>>>> Â >>>>> >>>>> >>>>> >>>>> a.I see all certs are >>>>>currently valid (none expired) >>>>> 6) Ran the command "getcert list" on the problem server, but I cannot >>>>> paste the output here because it's on an airgaped environment so while I >>>>> apologize for this and realize it makes things more difficult, perhaps >>>>> if you tell me what I should be looking for or more specifically what >>>>> you're interested in I can pluck that out and manually include it here? >>>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca' >>>>> certificate on the problem server, and it can theoretically be renew by >>>>> the Master at this time. >>>> The interesting part is the list of expired certs on the failing node >>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed >>>> instructions are available here: >>>> https://access.redhat.com/solutions/3357331 How do I manually renew >>>> Identity Management (IPA) certificates on RHEL7 after they have expired? >>>> (Replica IPA Server) >>>> >>>> flo >>>> >>>>> Many thanks! >>>>> Scott >>>>> >>>>> ------------------------------------------------------------------------ >>>>> *From:* Florence Blanc-Renaud <f...@redhat.com> >>>>> *Sent:* Monday, August 3, 2020 9:34 PM >>>>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org> >>>>> *Cc:* Scott Z. <sud...@hotmail.com> >>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting >>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote: >>>>>> Not sure I'm sending this to the right place, but here it >>>>>> goes.ÃÆ>>>>>> ’‚ >>>>>> I >>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet >>>>>> access) environment that is running into problems.ÃÆ>>>>>> ’‚ >>>>>> There are at least 3 >>>>>> different IdM servers running in the environment spread out across >>>>>> different geographical areas.ÃÆ>>>>>> ’‚ >>>>>> One of those areas suffered an unschedule >>>>>> power outage recently, and ever since we brought everything back up, the >>>>>> IdM server for this region is having an issue.ÃÆ>>>>>> ’‚ >>>>>> Please bear with me as I >>>>>> have zero formal experience, training, or real knowledge with IdM. >>>>>> >>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run >>>>>> "ipactl status" and it shows "Directory Service: STOPPED".ÃÆ>>>>>> ’‚ >>>>>> I then run >>>>>> "ipactl restart", and things go fine until it gets to "Starting >>>>>> pki-tomcatd Service", where it hangs for quite some time before failing >>>>>> to start and killing all the other services.ÃÆ>>>>>> ’‚ >>>>>> I check the log at >>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as >>>>>> (forgive any mistypings, I have to manually type these in as I can't >>>>>> import or screen capure the logs and put them in this message): >>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid: >>>>>> Invalid certificate: (-8181) Peer's Certificate has expired/" >>>>>> And slightly further down in the same log: >>>>>> "/Cannot reset factory: connections not all returned/" >>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset >>>>>> LDAP connection factory because some connections are still outstanding/" >>>>>> ... still further down" >>>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/" >>>>>> >>>>>> Assuming I have some weird certificate issue with this server in >>>>>> particular, I try to run a few more commands: >>>>>> "certutil -L -d /etc/httpd/alias"ÃÆ>>>>>> ’‚ >>>>>> --> returns a Server-Cert listing >>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C >>>>>> for it's attributes.ÃÆ>>>>>> ’‚ >>>>>> Comparing to a second IdM server in this >>>>>> environment, it seems to be missing a "Signing-Cert"? >>>>>> >>>>> Hi, >>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert >>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this >>>>> one is not expired with: >>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' >>>>> | grep 'Not ' >>>>> >>>>> If the certificate is indeed expired, it will have to be renewed but you >>>>> need first to find which IPA server is the CA renewal master. On your >>>>> server, force a service start and check the CA renewal master: >>>>> # ipactl start --ignore-service-failures >>>>> # kinit admin >>>>> # ipa config-show | grep "renewal master" >>>>> Â >>>>> IPA CA renewal master: >>>>>server.domain.com >>>>> >>>>> You need to make sure that all the certificates are valid on the CA >>>>> renewal master: >>>>> (on the CA renewal master)# getcert list | grep -E >>>>> "Request|certificate:|expires:" >>>>> >>>>> - if the CA renewal master is not OK, please post the output of "# >>>>> getcert list" (without the grep) on the CA renewal master. This node >>>>> will have to be repaired first. >>>>> - if the CA renewal master is OK, please post the output of "# getcert >>>>> list" (also without the grep) on the failing node. >>>>> >>>>> We'll be able to help based on this information. >>>>> flo >>>>> >>>>>> I also did a "getcert list", and all certs it has show that they expire >>>>>> in the future (nothing shows as bein currently expired). >>>>>> >>>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*, >>>>>> but how do I track down which 'peer' the log file is talking about that >>>>>> has an expired cert?ÃÆ>>>>>> ’‚ >>>>>> Meanwhile none of the linux clients that point to >>>>>> this IdM server are allowing people to log in/authenticate. >>>>>> Many thanks for any help! >>>>>> Scott >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >>>>>> To unsubscribe send an email to >>>>>> freeipa-users-le...@lists.fedorahosted.org >>>>>> Fedora Code of Conduct: >>>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>> List Archives: >>>>>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >>>>> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >>>>> >>>> >>> >>> >>> _______________________________________________ >>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >>> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >>> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >> >
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
