On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Hi Team,
We have 2 IPA servers in Mater-Master setup are we facing the below issue
on these servers.
Isuue1:
Our httpd certificate has expired because of which our IPA1 UI wasn't
working, we are getting “*loging failed due to an unknown reason*” error
while we log in to the UI
1. First, the IPA console was not working as httpd service was stopped,
httpd was not starting as HTTP certificate is expired. Added
*NSSEnforceValidCerts
off* line in nss.conf to start the service.
2. After the change IPA console was loading we are not able to login to the
console as pki-tomcatd service was not running,
[root@ipa1 ca]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l
● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago
Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited,
status=0/SUCCESS)
Main PID: 97233 (java)
CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
└─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
-DRESTEASY_LIB=/usr/share/java/resteasy-base
-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.security.manager
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception
processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background
process
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]:
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug
31/Oct/2019:13:24:23][localhost-startStop-1]:
SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert
cert-pki-ca
[31/Oct/2019:13:24:23][localhost-startStop-1]:
SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error
netscape.ldap.LDAPException: Authentication failed (49)
Authentication failed means the RA agent certificate dogtag uses to
authenticate to LDAP server is not the same as the one mentioned in the
LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy.
Also, you did not specify what versions of FreeIPA you run.
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Internal Database Error encountered: Could not connect to LDAP server host
ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException: Authentication
failed (49)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
# getcert list
Request ID '20180412150739':
status: SUBMITTING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN=
ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate
DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN=
ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=xxx.xxxxx.COM
subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM
expires: 2019-10-25 20:16:38 UTC
principal name: krbtgt/xxxx.xxxx....@xxxx.xxxx.com
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Status SUBMITTING means the renewal is not yet completed. It will not
complete until you get Dogtag working.
Issue2:
On the IPA2 server, we are unable to login with the admin user credentials
without OTP, but when an AD user is trying to login with 2FA (i.e,
password and OTP) we are getting this error *"The password you entered is
incorrect."*
AD users cannot use multifactor authentication defined in IPA.
# [root@ipa2 log]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l
● ipa-otpd.socket - ipa-otpd socket
Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled;
vendor preset: disabled)
Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; 1h
31min ago
Listen: /var/run/krb5kdc/DEFAULT.socket (Stream)
Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd socket.
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed to
queue service startup job (Maybe the service file is missing or not a
template unit?): Resource temporarily unavailable
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket entered
failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket
[Unit]
Description=ipa-otpd socket
[Socket]
ListenStream=/var/run/krb5kdc/DEFAULT.socket
RemoveOnStop=true
SocketMode=0600
Accept=true
[Install]
WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as the
changes made on IPA2 is not reflecting on IPA1
This is most likely because your LDAP server certificate expired as
well.
We the below errors as well.
IPA1
Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes
{rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx....@xxxx.xxxxx.com for ldap/
ipa2.xxxx.xxxx....@xxxx.xxxx.com
Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ (8
etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863,
etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx....@xxxx.xxxx.com for
ldap/ipa2.xxxx.xxxx....@xxxx.xxxx.com
These aren't errors. They are normal operations: ldap/ipa1 service (LDAP
server on IPA1) asked for a Kerberos service ticket to LDAP service on
IPA2 and was granted it. This is just as it should be for replication.
IPA2
# tailf krb5kdc.log
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/
ipa2.xxxx.xxxx....@xxx.xxxx.com for krbtgt/xxxx.xxxx....@xxxx.xxxx.com,
Additional pre-authentication required
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes
{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx....@xxx.xxxx.com for krbtgt/
xxx.xxxx....@xxx.xxxx.com
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes
{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx....@xxxx.xxxx.com for ldap/
ipa2.xxxx.xxxx....@xxx.xxxx.com
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Same here. LDAP server on IPA2 operated against itself here.
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
