[Freeipa-users] pki-tomcat won't start + expired certificates

[Basile Pinsard via FreeIPA-users]

Hi freeipa experts.

I have been using freeipa for the past 5 years running in a docker container, 
no replicas.
currently  on VERSION: 4.9.6, API_VERSION: 2.245

I have the following issue, not sure what caused this: pki-tomcat service is 
not starting, and it is no longer possible to login through the web-ui.
Auth through ldap (some websites) and through sssd on linux servers is still 
working, kerberos tickets are generated when logging with password or when 
running kinit, so critical operations are still possible.

The messages in `systemctl status pki-tomcatd@pki-tomcat.service` are
```
Apr 12 13:50:33 ipa.domain.com ipa-pki-wait-running[17869]: 
ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:  for url: 
http://ipa.domain.com:8080/ca/admin/ca/getStatus
Apr 12 13:50:34 ipa.domain.com systemd[1]: pki-tomcatd@pki-tomcat.service: 
start-post operation timed out. Terminating.
Apr 12 13:50:34 ipa.domain.com systemd[1]: pki-tomcatd@pki-tomcat.service: 
Control process exited, code=killed, status=15/TERM
Apr 12 13:50:34 ipa.domain.com systemd[1]: pki-tomcatd@pki-tomcat.service: 
Failed with result 'timeout'.
Apr 12 13:50:34 ipa.domain.com systemd[1]: Failed to start PKI Tomcat Server 
pki-tomcat.
```

journalctl give other errors (filtered what seems relevant).
```
Apr 12 13:49:05 ipa.domain.com server[17868]: WARNING: Problem with JAR file 
[/usr/share/pki/server/common/lib/commons-collections.jar], exists: [false], 
canRead: [false]
Apr 12 13:49:07 ipa.domain.com java[17868]: usr/lib/api/apiutil.c Could not 
open /run/lock/opencryptoki/LCK..APIlock
Apr 12 13:49:18 ipa.domain.com server[17868]: SEVERE: Context [/acme] startup 
failed due to previous errors

```


`/var/log/pki/pki-tomcat/pki/debug.2024-04-12.log`
contains the following errors 
```
2024-04-12 15:01:12 [main] SEVERE: Exception initializing random number 
generator using provider [Mozilla-JSS]
java.security.NoSuchProviderException: no such provider: Mozilla-JSS
at 
java.base/sun.security.jca.GetInstance.getService(GetInstance.java:83)
at 
java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:206)

```

`/var/log/pki/pki-tomcat/ca/debug.2024-04-12.log`
contains the following type of errors

```
2024-04-12 00:17:37 [main] SEVERE: Unable to start CA engine: Property 
instanceRoot missing value
Property instanceRoot missing value
at 
com.netscape.cmscore.base.PropConfigStore.getString(PropConfigStore.java:297)
at 
com.netscape.cmscore.apps.EngineConfig.getInstanceDir(EngineConfig.java:55)
at com.netscape.cmscore.apps.CMSEngine.loadConfig(CMSEngine.java:233)
at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1025)


2024-04-12 17:49:21 [main] SEVERE: Exception sending context initialized event 
to listener instance of class [org.dogtagpki.server.ca.CAEngine]
java.lang.RuntimeException: Unable to start CA engine: Property instanceRoot 
missing value
at 
com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1672)
at 
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
```

`getcert list` reports all entries except the caCACert as expired.

I tried pretty much everything I could find on the internet (though most of the 
threads I found were never resolved).
Tried ipa-cert-fix.
Tried ipa-restoring a backup in a new container, same problem occurs.

My guess is that an upgrade years back did break the certificate auto-renewal 
and went undetected, and now everything is expired it's failing.
 
If you have any ideas of what to check/try I would be very grateful as I am 
losing my sanity here.
Also, I am a bit scared of breaking what is currently working (ldap+sssd) and 
critical to our operations, so if anything can be tested on a copy of the data 
in a container that would be great. 

Thanks!
--
___
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

[Freeipa-users] Re: ipaclient-install.log certutil: Could not find cert:

[Christian Heimes via FreeIPA-users]

On 12/04/2024 18.46, C Wilson via FreeIPA-users wrote:

Hello

I'm trying to roll out a new IPA server for our development environment and 
have nicely automated the server installation process with Ansible but when 
I've come to rolling out the clients I'm hitting this problem.

When running ipa-client-install:
ipa-client-install -N --fixed-primary --server server.domain.local --realm 
DOMAIN.LOCAL --domain DOMAIN.local --principal admin --password 'adminpassword' 
-U


I recommend against use of .local TLD for an IPA installation. The 
.local addresses are reserved for link-local networks, mDNS and 
zeroconf. Host lookups for .local behave differently and may result in 
surprising behavior.


Instead use one of the recommended TLDs from 
https://www.rfc-editor.org/rfc/rfc6762#appendix-G or 
https://www.rfc-editor.org/rfc/rfc2606.html .


Christian

--
Christian Heimes
Principal Software Engineer, Identity Management and Platform Security

Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
--
___
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

[Freeipa-users] Re: ipaclient-install.log certutil: Could not find cert:

[Rob Crittenden via FreeIPA-users]

C Wilson via FreeIPA-users wrote:
> Hello
> 
> I'm trying to roll out a new IPA server for our development environment and 
> have nicely automated the server installation process with Ansible but when 
> I've come to rolling out the clients I'm hitting this problem. 
> 
> When running ipa-client-install:
> ipa-client-install -N --fixed-primary --server server.domain.local --realm 
> DOMAIN.LOCAL --domain DOMAIN.local --principal admin --password 
> 'adminpassword' -U
> 
> I get the following error:
> Please make sure the following ports are opened in the firewall settings:
>  TCP: 80, 88, 389
>  UDP: 88 (at least one of TCP/UDP ports 88 has to be open)
> Also note that following ports are necessary for ipa-client working properly 
> after enrollment:
>  TCP: 464
>  UDP: 464, 123 (if NTP enabled)
> Installation failed. Rolling back changes.
> Disabling client Kerberos and LDAP configurations
> nscd daemon is not installed, skip configuration
> nslcd daemon is not installed, skip configuration
> Client uninstall complete.
> Kerberos authentication failed: kinit: Cannot contact any KDC for realm 
> 'DOMAIN.LOCAL' while getting initial credentials
> 
> 
> I've disabled the firewall on both systems, DNS resolves the server name. I 
> can nmap and telnet to the ports listed so I don't think it's a networking 
> issue. The ipa server appears to be running fine:
> 
> [root@server tmp]# service ipa status
> Redirecting to /bin/systemctl status ipa.service
> ● ipa.service - Identity, Policy, Audit
>  Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; preset: 
> disabled)
>  Active: active (exited) since Wed 2024-04-10 15:49:49 UTC; 2 days ago
>Main PID: 18336 (code=exited, status=0/SUCCESS)
> CPU: 1.610s
> 
> Apr 10 15:49:48 server ipactl[18336]: Assuming stale, cleaning and proceeding
> Apr 10 15:49:49 server ipactl[18336]: ipa: INFO: The ipactl command was 
> successful
> Apr 10 15:49:49 server ipactl[18336]: Starting Directory Service
> Apr 10 15:49:49 server ipactl[18336]: Starting krb5kdc Service
> Apr 10 15:49:49 server ipactl[18336]: Starting kadmin Service
> Apr 10 15:49:49 server ipactl[18336]: Starting httpd Service
> Apr 10 15:49:49 server ipactl[18336]: Starting ipa-custodia Service
> Apr 10 15:49:49 server ipactl[18336]: Starting pki-tomcatd Service
> Apr 10 15:49:49 server ipactl[18336]: Starting ipa-otpd Service
> Apr 10 15:49:49 server systemd[1]: Finished Identity, Policy, Audit.
> 
> 
> Looking at the ipaclient-install.log there are lines that are semi 
> interesting but I can't see how to progress from here to resolve the issue:
> 
> 2024-04-12T16:25:51Z DEBUG stderr=kinit: Cannot contact any KDC for realm 
> 'DOMAIN.LOCAL' while getting initial credentials
> 2024-04-12T16:25:51Z ERROR Installation failed. Rolling back changes.
> 2024-04-12T16:25:52Z DEBUG stderr=
> 2024-04-12T16:25:52Z DEBUG stderr=certutil: Could not find cert: IPA Machine 
> Certificate - virt01.domain.local
> : PR_FILE_NOT_FOUND_ERROR: File not found
> 
> 
> but if I run `kinit admin@server.domain.local` it authenticates. 

The cert error is a red herring. It is looking to see if there is one
that needs to be cleaned up (there isn't).

Do you already have krb5.conf configured? Otherwise I don't know how the
KDC is contacted.

You can find the temporary krb5.conf that is used by the installer in
the log. You can put that into a file and try something like:

KRB5_CONFIG=/tmp/krb.conf KRB5_TRACE=/dev/stderr kinit admin

This should fail since this is doing the same thing as
ipa-client-install. The output may help identify what it's doing.

rob
--
___
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

[Freeipa-users] ipaclient-install.log certutil: Could not find cert:

[C Wilson via FreeIPA-users]

Hello

I'm trying to roll out a new IPA server for our development environment and 
have nicely automated the server installation process with Ansible but when 
I've come to rolling out the clients I'm hitting this problem. 

When running ipa-client-install:
ipa-client-install -N --fixed-primary --server server.domain.local --realm 
DOMAIN.LOCAL --domain DOMAIN.local --principal admin --password 'adminpassword' 
-U

I get the following error:
Please make sure the following ports are opened in the firewall settings:
 TCP: 80, 88, 389
 UDP: 88 (at least one of TCP/UDP ports 88 has to be open)
Also note that following ports are necessary for ipa-client working properly 
after enrollment:
 TCP: 464
 UDP: 464, 123 (if NTP enabled)
Installation failed. Rolling back changes.
Disabling client Kerberos and LDAP configurations
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
Kerberos authentication failed: kinit: Cannot contact any KDC for realm 
'DOMAIN.LOCAL' while getting initial credentials


I've disabled the firewall on both systems, DNS resolves the server name. I can 
nmap and telnet to the ports listed so I don't think it's a networking issue. 
The ipa server appears to be running fine:

[root@server tmp]# service ipa status
Redirecting to /bin/systemctl status ipa.service
● ipa.service - Identity, Policy, Audit
 Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; preset: 
disabled)
 Active: active (exited) since Wed 2024-04-10 15:49:49 UTC; 2 days ago
   Main PID: 18336 (code=exited, status=0/SUCCESS)
CPU: 1.610s

Apr 10 15:49:48 server ipactl[18336]: Assuming stale, cleaning and proceeding
Apr 10 15:49:49 server ipactl[18336]: ipa: INFO: The ipactl command was 
successful
Apr 10 15:49:49 server ipactl[18336]: Starting Directory Service
Apr 10 15:49:49 server ipactl[18336]: Starting krb5kdc Service
Apr 10 15:49:49 server ipactl[18336]: Starting kadmin Service
Apr 10 15:49:49 server ipactl[18336]: Starting httpd Service
Apr 10 15:49:49 server ipactl[18336]: Starting ipa-custodia Service
Apr 10 15:49:49 server ipactl[18336]: Starting pki-tomcatd Service
Apr 10 15:49:49 server ipactl[18336]: Starting ipa-otpd Service
Apr 10 15:49:49 server systemd[1]: Finished Identity, Policy, Audit.


Looking at the ipaclient-install.log there are lines that are semi interesting 
but I can't see how to progress from here to resolve the issue:

2024-04-12T16:25:51Z DEBUG stderr=kinit: Cannot contact any KDC for realm 
'DOMAIN.LOCAL' while getting initial credentials
2024-04-12T16:25:51Z ERROR Installation failed. Rolling back changes.
2024-04-12T16:25:52Z DEBUG stderr=
2024-04-12T16:25:52Z DEBUG stderr=certutil: Could not find cert: IPA Machine 
Certificate - virt01.domain.local
: PR_FILE_NOT_FOUND_ERROR: File not found


but if I run `kinit admin@server.domain.local` it authenticates. 

I seem to be at a dead end, How do I troubleshoot this further? 
--
___
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

[Freeipa-users] Re: Cannot retrieve CRL from new EL9 IPA replica

[Florence Blanc-Renaud via FreeIPA-users]

Hi,

On Thu, Apr 11, 2024 at 6:02 PM Orion Poplawski  wrote:

> On 4/11/24 09:03, Florence Blanc-Renaud wrote:
> > Hi,
> >
> > On Thu, Apr 11, 2024 at 12:34 AM Orion Poplawski via FreeIPA-users
> >  > > wrote:
> >
> > I've just added an EL9 IPA replica into our domain.  I seems to
> generally be
> > working fine, but trying to download the MasterCRL.bin fails:
> >
> > ==> /var/log/httpd/access_log <==
> > 10.20.0.37 - - [10/Apr/2024:14:13:17 -0700] "GET
> /ipa/crl/MasterCRL.bin
> > HTTP/1.1" 301 293 "-" "curl/7.76.1"
> >
> > ==> /var/log/httpd/error_log <==
> > [Wed Apr 10 14:14:17.830119 2024] [proxy_ajp:error] [pid 28001:tid
> 28040]
> > (70007)The timeout specified has expired: AH01030:
> ajp_ilink_receive() can't
> > receive header
> > [Wed Apr 10 14:14:17.830249 2024] [proxy_ajp:error] [pid 28001:tid
> 28040]
> > [client 10.20.0.37:35124 ] AH00992:
> > ajp_read_header: ajp_ilink_receive failed
> > [Wed Apr 10 14:14:17.830261 2024] [proxy_ajp:error] [pid 28001:tid
> 28040]
> > (70007)The timeout specified has expired: [client 10.20.0.37:35124
> > ] AH00878:
> > read response failed from [::1]:8009 (localhost:8009)
> >
> > ==> /var/log/httpd/access_log <==
> > 10.20.0.37 - - [10/Apr/2024:14:13:17 -0700] "GET
> > /ca/ee/ca/getCRL?op=getCRL=MasterCRL HTTP/1.1" 500
> 527 "-"
> > "curl/7.76.1"
> >
> > I'm not sure where else to look for logs.
> >
> >
> > If you are requesting the MasterCRL.bin file on a replica that is not
> the CRL
> > generation master, the URL is transferred to the local CA server
> > at
> http://replica.ipa.test/ca/ee/ca/getCRL?op=getCRL=MasterCRL
> > <
> http://replica.ipa.test/ca/ee/ca/getCRL?op=getCRL=MasterCRL
> >
> > (this is configured in /etc/httpd/conf.d/ipa-pki-proxy.conf).
> >
> > Then the calls to /ca/ee/ca/getCRL are handled by an AJP connector
> > (LocationMatch defined in /etc/httpd/conf.d/ipa-pki-proxy.conf using
> > ajp://localhost:8009). The AJP connector is defined
> > in /etc/pki/pki-tomcat/server.xml and should be using the loopback
> address.
> > There can be issues if your /etc/hosts does not contain the following
> lines:
> > 127.0.0.1   localhost localhost.localdomain localhost4
> localhost4.localdomain4
> > ::1 localhost localhost.localdomain localhost6
> localhost6.localdomain6
> >
> > You can have a look
> in /var/log/pki/pki-tomcat/localhost_access_log.$DATE.txt
> > and check if the request really reached the PKI server. Then check logs
> > in /var/log/pki/pki-tomcat/ca/debug.$DATE.log
>
> The machine in question is not the CRL generator.  We are getting
> redirected
> to /ca/ee/ca/getCRL?op=getCRL=MasterCRL on that machine.
> But
> it is that request that is timing out.
>
> Looks like the tomcat server may be hosed:
>
> Apr 05 00:01:00 server[5758]: java.util.logging.ErrorManager: 1:
> FileHandler
> is closed or not yet initialized, unable to log [2024-04-05 00:01:00
> [Timer-0]
> INFO: SessionTimer: checking security domain sessions
> Apr 05 00:01:00 server[5758]: ]
> Apr 05 00:01:02 server[5758]: java.util.logging.ErrorManager: 1:
> FileHandler
> is closed or not yet initialized, unable to log [2024-04-05 00:01:02
> [pool-1-thread-1] SEVERE: Unable to run maintenance task: access denied
> ("java.io.FilePermission" "/var/lib/pki/pki-tomcat/logs/acme" "read")
> Apr 05 00:01:02 server[5758]: java.security.AccessControlException: access
> denied ("java.io.FilePermission" "/var/lib/pki/pki-tomcat/logs/acme"
> "read")
> Apr 05 00:01:02 server[5758]: at
>
> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
>
> Apr 06 00:01:13 server[16841]: java.util.logging.ErrorManager: 1:
> FileHandler
> is closed or not yet initialized, unable to log [2024-04-06 00:01:13
> [pool-1-thread-1] SEVERE: Unable to run maintenance task: access denied
> ("java.io.FilePermission" "/var/lib/pki/pki-tomcat/logs/acme" "read")
> Apr 06 00:01:13 server[16841]: java.security.AccessControlException: access
> denied ("java.io.FilePermission" "/var/lib/pki/pki-tomcat/logs/acme"
> "read")
> Apr 06 00:01:13 server[16841]: at
> java.base/java.security.AccessControlContext.checkPermis
>
> Apr 06 00:01:14 server[16841]: java.util.logging.ErrorManager: 1:
> FileHandler
> is closed or not yet initialized, unable to log [2024-04-06 00:01:14
> [KeyStatusUpdateTask] WARNING: Repository: Unable to check next range:
> access
> denied ("java.io.FilePermission" "/var/lib/pki/pki-tomcat/logs/kra" "read")
> Apr 06 00:01:14 server[16841]: java.security.AccessControlException: access
> denied ("java.io.FilePermission" "/var/lib/pki/pki-tomcat/logs/kra" "read")
> Apr 06 00:01:14 server[16841]: at
>
> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
> Apr 06 00:01:14 server[16841]: at
>
>