Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-18 Thread Michael Plemmons 
SOLVED!

Thank you Flo!  That did the trick.  Once I made the change to the
certificate and restarted the IPA services everything came back up like it
was supposed to.

High five!


*Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
614.427.2411
mike.plemm...@crosschx.com
www.crosschx.com

On Thu, May 18, 2017 at 10:28 AM, Florence Blanc-Renaud 
wrote:

> On 05/18/2017 03:49 PM, Michael Plemmons wrote:
>
>>
>>
>>
>>
>> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX
>> *
>> 614.427.2411
>> mike.plemm...@crosschx.com 
>> www.crosschx.com 
>>
>> On Thu, May 18, 2017 at 8:02 AM, Florence Blanc-Renaud > > wrote:
>>
>> On 05/15/2017 08:33 PM, Michael Plemmons wrote:
>>
>> I have done more searching in my logs and I see the following
>> errors.
>>
>> This is in the localhost log file /var/lib/pki/pki-tomcat/logs
>>
>> May 15, 2017 3:08:08 PM
>> org.apache.catalina.core.ApplicationContext log
>> SEVERE: StandardWrapper.Throwable
>> java.lang.NullPointerException
>>
>> May 15, 2017 3:08:08 PM org.apache.catalina.core.StandardContext
>> loadOnStartup
>> SEVERE: Servlet [castart] in web application [/ca] threw load()
>> exception
>> java.lang.NullPointerException
>>
>> May 15, 2017 3:08:09 PM
>> org.apache.catalina.core.StandardHostValve invoke
>> SEVERE: Exception Processing /ca/admin/ca/getStatus
>> javax.ws.rs 
>> .ServiceUnavailableException: Subsystem
>> unavailable
>>
>>
>> Looking at the debug log it says Authentication failed for port
>> 636.
>>
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo:
>> init()
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo:
>> init begins
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo:
>> init ends
>> [15/May/2017:17:39:25][localhost-startStop-1]: init: before
>> makeConnection errorIfDown is true
>> [15/May/2017:17:39:25][localhost-startStop-1]: makeConnection:
>> errorIfDown true
>> [15/May/2017:17:39:25][localhost-startStop-1]:
>> SSLClientCertificateSelectionCB: Setting desired cert nickname
>> to:
>> subsystemCert cert-pki-ca
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapJssSSLSocket:
>> set
>> client auth cert nickname subsystemCert cert-pki-ca
>> [15/May/2017:17:39:25][localhost-startStop-1]:
>> SSLClientCertificatSelectionCB: Entering!
>> [15/May/2017:17:39:25][localhost-startStop-1]:
>> SSLClientCertificateSelectionCB: returning: null
>> [15/May/2017:17:39:25][localhost-startStop-1]: SSL handshake
>> happened
>> Could not connect to LDAP server host ipa12.mgmt.crosschx.com
>> 
>> > > port 636 Error
>> netscape.ldap.LDAPException: Authentication failed (48)
>> at
>> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConne
>> ction(LdapBoundConnFactory.java:205)
>>
>>
>> I looked at the validity of the cert it mentions and it is fine.
>>
>> (root)>getcert status -v -d /etc/pki/pki-tomcat/alias -n
>> 'subsystemCert
>> cert-pki-ca'
>> State MONITORING, stuck: no.
>>
>>
>> I then looked at the ldap errors around the time of this failure
>> and I
>> am seeing this log entry.
>>
>>
>> [15/May/2017:17:38:42.063080758 +] set_krb5_creds - Could
>> not get
>> initial credentials for principal
>> [ldap/ipa12.mgmt.crosschx@mgmt.crosschx.com
>> 
>> > >] in keytab
>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
>> KDC for
>> requested realm)
>>
>> When I perform a klist against that keytab nothing appears out
>> of the
>> ordinary compared to working IPA servers.
>>
>> I am not sure what to look at next.
>>
>>
>> Hi,
>>
>> you can try the following to manually replay the connection
>> established by Dogtag to LDAP server:
>>
>> root$ export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>> root$ export LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>
>> The above commands specify the NSSDB containing the user certificate
>> and its name for SASL-EXTERNAL authentication.
>>
>> Then note the value obtained below as it will be used for the next
>> step as the password to access the private key in the NSSDB:
>> root$ grep

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-18 Thread Florence Blanc-Renaud 

On 05/18/2017 03:49 PM, Michael Plemmons wrote:





*Mike Plemmons | Senior DevOps Engineer | CROSSCHX
*
614.427.2411
mike.plemm...@crosschx.com 
www.crosschx.com 

On Thu, May 18, 2017 at 8:02 AM, Florence Blanc-Renaud > wrote:

On 05/15/2017 08:33 PM, Michael Plemmons wrote:

I have done more searching in my logs and I see the following
errors.

This is in the localhost log file /var/lib/pki/pki-tomcat/logs

May 15, 2017 3:08:08 PM
org.apache.catalina.core.ApplicationContext log
SEVERE: StandardWrapper.Throwable
java.lang.NullPointerException

May 15, 2017 3:08:08 PM org.apache.catalina.core.StandardContext
loadOnStartup
SEVERE: Servlet [castart] in web application [/ca] threw load()
exception
java.lang.NullPointerException

May 15, 2017 3:08:09 PM
org.apache.catalina.core.StandardHostValve invoke
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs 
.ServiceUnavailableException: Subsystem
unavailable


Looking at the debug log it says Authentication failed for port 636.

[15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init()
[15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo:
init begins
[15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo:
init ends
[15/May/2017:17:39:25][localhost-startStop-1]: init: before
makeConnection errorIfDown is true
[15/May/2017:17:39:25][localhost-startStop-1]: makeConnection:
errorIfDown true
[15/May/2017:17:39:25][localhost-startStop-1]:
SSLClientCertificateSelectionCB: Setting desired cert nickname to:
subsystemCert cert-pki-ca
[15/May/2017:17:39:25][localhost-startStop-1]: LdapJssSSLSocket: set
client auth cert nickname subsystemCert cert-pki-ca
[15/May/2017:17:39:25][localhost-startStop-1]:
SSLClientCertificatSelectionCB: Entering!
[15/May/2017:17:39:25][localhost-startStop-1]:
SSLClientCertificateSelectionCB: returning: null
[15/May/2017:17:39:25][localhost-startStop-1]: SSL handshake
happened
Could not connect to LDAP server host ipa12.mgmt.crosschx.com

> port 636 Error
netscape.ldap.LDAPException: Authentication failed (48)
at

com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)


I looked at the validity of the cert it mentions and it is fine.

(root)>getcert status -v -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
cert-pki-ca'
State MONITORING, stuck: no.


I then looked at the ldap errors around the time of this failure
and I
am seeing this log entry.


[15/May/2017:17:38:42.063080758 +] set_krb5_creds - Could
not get
initial credentials for principal
[ldap/ipa12.mgmt.crosschx@mgmt.crosschx.com

>] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
KDC for
requested realm)

When I perform a klist against that keytab nothing appears out
of the
ordinary compared to working IPA servers.

I am not sure what to look at next.


Hi,

you can try the following to manually replay the connection
established by Dogtag to LDAP server:

root$ export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
root$ export LDAPTLS_CERT='subsystemCert cert-pki-ca'

The above commands specify the NSSDB containing the user certificate
and its name for SASL-EXTERNAL authentication.

Then note the value obtained below as it will be used for the next
step as the password to access the private key in the NSSDB:
root$ grep internal /etc/pki/pki-tomcat/password.conf
internal=

root$ ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL
-Q -LLL dn namingcontexts
Please enter pin, password, or pass phrase for security token
'ldap(0)': here supply the value found above
dn:
namingcontexts: cn=changelog
namingcontexts: dc=ipadomain,dc=com
namingcontexts: o=ipaca



So I guess I found my problem.

(root)>ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q
-LLL dn namingcontexts
Please enter pin, password, or pass phrase for security token 'ldap(0)':
ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)
  additional info: TLS error

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-18 Thread Michael Plemmons 
*Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
614.427.2411
mike.plemm...@crosschx.com
www.crosschx.com

On Thu, May 18, 2017 at 8:02 AM, Florence Blanc-Renaud 
wrote:

> On 05/15/2017 08:33 PM, Michael Plemmons wrote:
>
>> I have done more searching in my logs and I see the following errors.
>>
>> This is in the localhost log file /var/lib/pki/pki-tomcat/logs
>>
>> May 15, 2017 3:08:08 PM org.apache.catalina.core.ApplicationContext log
>> SEVERE: StandardWrapper.Throwable
>> java.lang.NullPointerException
>>
>> May 15, 2017 3:08:08 PM org.apache.catalina.core.StandardContext
>> loadOnStartup
>> SEVERE: Servlet [castart] in web application [/ca] threw load() exception
>> java.lang.NullPointerException
>>
>> May 15, 2017 3:08:09 PM org.apache.catalina.core.StandardHostValve invoke
>> SEVERE: Exception Processing /ca/admin/ca/getStatus
>> javax.ws.rs .ServiceUnavailableException: Subsystem
>> unavailable
>>
>>
>> Looking at the debug log it says Authentication failed for port 636.
>>
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init()
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init begins
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init ends
>> [15/May/2017:17:39:25][localhost-startStop-1]: init: before
>> makeConnection errorIfDown is true
>> [15/May/2017:17:39:25][localhost-startStop-1]: makeConnection:
>> errorIfDown true
>> [15/May/2017:17:39:25][localhost-startStop-1]:
>> SSLClientCertificateSelectionCB: Setting desired cert nickname to:
>> subsystemCert cert-pki-ca
>> [15/May/2017:17:39:25][localhost-startStop-1]: LdapJssSSLSocket: set
>> client auth cert nickname subsystemCert cert-pki-ca
>> [15/May/2017:17:39:25][localhost-startStop-1]:
>> SSLClientCertificatSelectionCB: Entering!
>> [15/May/2017:17:39:25][localhost-startStop-1]:
>> SSLClientCertificateSelectionCB: returning: null
>> [15/May/2017:17:39:25][localhost-startStop-1]: SSL handshake happened
>> Could not connect to LDAP server host ipa12.mgmt.crosschx.com
>>  port 636 Error
>> netscape.ldap.LDAPException: Authentication failed (48)
>> at
>> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConne
>> ction(LdapBoundConnFactory.java:205)
>>
>>
>> I looked at the validity of the cert it mentions and it is fine.
>>
>> (root)>getcert status -v -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>> cert-pki-ca'
>> State MONITORING, stuck: no.
>>
>>
>> I then looked at the ldap errors around the time of this failure and I
>> am seeing this log entry.
>>
>>
>> [15/May/2017:17:38:42.063080758 +] set_krb5_creds - Could not get
>> initial credentials for principal
>> [ldap/ipa12.mgmt.crosschx@mgmt.crosschx.com
>> ] in keytab
>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
>> requested realm)
>>
>> When I perform a klist against that keytab nothing appears out of the
>> ordinary compared to working IPA servers.
>>
>> I am not sure what to look at next.
>>
>>
> Hi,
>
> you can try the following to manually replay the connection established by
> Dogtag to LDAP server:
>
> root$ export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
> root$ export LDAPTLS_CERT='subsystemCert cert-pki-ca'
>
> The above commands specify the NSSDB containing the user certificate and
> its name for SASL-EXTERNAL authentication.
>
> Then note the value obtained below as it will be used for the next step as
> the password to access the private key in the NSSDB:
> root$ grep internal /etc/pki/pki-tomcat/password.conf
> internal=
>
> root$ ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q
> -LLL dn namingcontexts
> Please enter pin, password, or pass phrase for security token 'ldap(0)':
>    here supply the value found above
> dn:
> namingcontexts: cn=changelog
> namingcontexts: dc=ipadomain,dc=com
> namingcontexts: o=ipaca
>
>

So I guess I found my problem.

(root)>ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q
-LLL dn namingcontexts
Please enter pin, password, or pass phrase for security token 'ldap(0)':
ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)
  additional info: TLS error -12195:Peer does not recognize and trust the
CA that issued your certificate.


I looked at our certs in /etc/dirsrv/slapd-IPADOMAIN-COM and found the
following.

IPA12 - problem server
(root)>certutil -L -d /etc/dirsrv/slapd-IPADOMAIN-COM

Certificate Nickname Trust
Attributes

 SSL,S/MIME,JAR/XPI

Server-Cert  u,u,u
IPADOMAIN-COM IPA CA C,,



IPA11/IPA13 - 11 was the master and 13 is the new master
(root)>certutil -L -d /etc/dirsrv/slapd-IPADOMAIN-COM

Certificate Nickname Trust
Attributes

 SSL,S/MIME,JAR/XPI

Server-Cert

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-18 Thread Florence Blanc-Renaud 

On 05/15/2017 08:33 PM, Michael Plemmons wrote:

I have done more searching in my logs and I see the following errors.

This is in the localhost log file /var/lib/pki/pki-tomcat/logs

May 15, 2017 3:08:08 PM org.apache.catalina.core.ApplicationContext log
SEVERE: StandardWrapper.Throwable
java.lang.NullPointerException

May 15, 2017 3:08:08 PM org.apache.catalina.core.StandardContext
loadOnStartup
SEVERE: Servlet [castart] in web application [/ca] threw load() exception
java.lang.NullPointerException

May 15, 2017 3:08:09 PM org.apache.catalina.core.StandardHostValve invoke
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs .ServiceUnavailableException: Subsystem
unavailable


Looking at the debug log it says Authentication failed for port 636.

[15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init()
[15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init begins
[15/May/2017:17:39:25][localhost-startStop-1]: LdapAuthInfo: init ends
[15/May/2017:17:39:25][localhost-startStop-1]: init: before
makeConnection errorIfDown is true
[15/May/2017:17:39:25][localhost-startStop-1]: makeConnection:
errorIfDown true
[15/May/2017:17:39:25][localhost-startStop-1]:
SSLClientCertificateSelectionCB: Setting desired cert nickname to:
subsystemCert cert-pki-ca
[15/May/2017:17:39:25][localhost-startStop-1]: LdapJssSSLSocket: set
client auth cert nickname subsystemCert cert-pki-ca
[15/May/2017:17:39:25][localhost-startStop-1]:
SSLClientCertificatSelectionCB: Entering!
[15/May/2017:17:39:25][localhost-startStop-1]:
SSLClientCertificateSelectionCB: returning: null
[15/May/2017:17:39:25][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host ipa12.mgmt.crosschx.com
 port 636 Error
netscape.ldap.LDAPException: Authentication failed (48)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)


I looked at the validity of the cert it mentions and it is fine.

(root)>getcert status -v -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca'
State MONITORING, stuck: no.


I then looked at the ldap errors around the time of this failure and I
am seeing this log entry.


[15/May/2017:17:38:42.063080758 +] set_krb5_creds - Could not get
initial credentials for principal
[ldap/ipa12.mgmt.crosschx@mgmt.crosschx.com
] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)

When I perform a klist against that keytab nothing appears out of the
ordinary compared to working IPA servers.

I am not sure what to look at next.



Hi,

you can try the following to manually replay the connection established 
by Dogtag to LDAP server:


root$ export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
root$ export LDAPTLS_CERT='subsystemCert cert-pki-ca'

The above commands specify the NSSDB containing the user certificate and 
its name for SASL-EXTERNAL authentication.


Then note the value obtained below as it will be used for the next step 
as the password to access the private key in the NSSDB:

root$ grep internal /etc/pki/pki-tomcat/password.conf
internal=

root$ ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q 
-LLL dn namingcontexts
Please enter pin, password, or pass phrase for security token 'ldap(0)': 
    here supply the value found above

dn:
namingcontexts: cn=changelog
namingcontexts: dc=ipadomain,dc=com
namingcontexts: o=ipaca


In the LDAP server access log (in 
/etc/dirsrv/slapd-IPADOMAIN.COM/access), you should see the 
corresponding connection:


[18/May/2017:13:35:14.822090417 +0200] conn=297 fd=150 slot=150 SSL 
connection from xxx to yyy
[18/May/2017:13:35:15.789414017 +0200] conn=297 TLS1.2 128-bit AES-GCM; 
client CN=CA Subsystem,O=IPADOMAIN.COM; issuer CN=Certificate 
Authority,O=IPADOMAIN.COM
[18/May/2017:13:35:15.793108509 +0200] conn=297 TLS1.2 client bound as 
uid=pkidbuser,ou=people,o=ipaca
[18/May/2017:13:35:15.798101505 +0200] conn=297 op=0 BIND dn="" 
method=sasl version=3 mech=EXTERNAL
[18/May/2017:13:35:15.800322076 +0200] conn=297 op=0 RESULT err=0 tag=97 
nentries=0 etime=0 dn="uid=pkidbuser,ou=people,o=ipaca"


HTH,
Flo.





*Mike Plemmons | Senior DevOps Engineer | CROSSCHX
*
614.427.2411
mike.plemm...@crosschx.com 
www.crosschx.com 

On Wed, May 10, 2017 at 3:35 PM, Michael Plemmons
>
wrote:

The PKI service came up successfully but only when it uses BasicAuth
rather than SSL auth.  I am not sure about what I need to do in
order to get the auth working over SSL again.

None of the certs are expired when I run getcert list and
ipa-getcert list.

Since the failure is with attempts to login to LDAP over 636.  I
have been attempting to auth to LDAP via port

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-05 Thread Rob Crittenden 
Michael Plemmons wrote:
> I just realized that I sent the reply directly to Rob and not to the
> list.  My response is inline

Ok, this is actually good news.

I made a similar proposal in another case and I was completely wrong.
Flo had the user do something and it totally fixed their auth error, I
just can't remember what it was or find the e-mail thread. I'm pretty
sure it was this calendar year though.

rob

> 
> 
> 
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX
> *
> 614.427.2411
> mike.plemm...@crosschx.com 
> www.crosschx.com 
> 
> On Thu, May 4, 2017 at 9:39 AM, Michael Plemmons
> >
> wrote:
> 
> 
> 
> 
> 
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX
> *
> 614.427.2411
> mike.plemm...@crosschx.com 
> www.crosschx.com 
> 
> On Thu, May 4, 2017 at 9:24 AM, Rob Crittenden  > wrote:
> 
> Michael Plemmons wrote:
> > I realized that I was not very clear in my statement about
> testing with
> > ldapsearch.  I had initially run it without logging in with a
> DN.  I was
> > just running the local ldapsearch -x command.  I then tested on
> > ipa12.mgmt and ipa11.mgmt logging in with a full DN for the
> admin and
> > "cn=Directory Manager" from ipa12.mgmt (broken server) and
> ipa11.mgmt
> > and both ldapsearch command succeeded.
> >
> > I ran the following from ipa12.mgmt and ipa11.mgmt as a non
> root user.
> > I also ran the command showing a line count for the output and
> the line
> > counts for each were the same when run from ipa12.mgmt and
> ipa11.mgmt.
> >
> > ldapsearch -LLL -h ipa12.mgmt.crosschx.com
> 
> >  > -D "DN" -w PASSWORD -b
> > "cn=users,cn=accounts,dc=mgmt,dc=crosschx,dc=com" dn
> >
> > ldapsearch -LLL -h ipa12.mgmt.crosschx.com
> 
> >  > -D "cn=directory manager" -w
> PASSWORD dn
> 
> The CA has its own suffix and replication agreements. Given the auth
> error and recent (5 months) renewal of CA credentials I'd check
> that the
> CA agent authentication entries are correct.
> 
> Against each master with a CA run:
> 
> $ ldapsearch -LLL -x -D 'cn=directory manager' -W -b
> uid=ipara,ou=people,o=ipaca description
> 
> The format is 2;serial#,subject,issuer
> 
> Then on each run:
> 
> # certutil -L -d /etc/httpd/alias -n ipaCert |grep Serial
> 
> The serial # should match that in the description everywhere.
> 
> rob
> 
> 
> 
> On the CA (IPA13.MGMT) I ran the ldapsearch command and see that the
> serial number is 7.  I then ran the certutil command on all three
> servers and the serial number is 7 as well.
> 
>  
> I also ran the ldapsearch command against the other two servers and
> they also showed a serial number of 7. 
> 
>  
> 
> 
> >
> >
> >
> >
> >
> > *Mike Plemmons | Senior DevOps Engineer | CROSSCHX
> > *
> > 614.427.2411
> > mike.plemm...@crosschx.com 
>  >
> > www.crosschx.com 
> 
> >
> > On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons
> >  
>  >>
> > wrote:
> >
> > I have a three node IPA cluster.
> >
> > ipa11.mgmt - was a master over 6 months ago
> > ipa13.mgmt - current master
> > ipa12.mgmt
> >
> > ipa13 has agreements with ipa11 and ipa12.  ipa11 and
> ipa12 do not
> > have agreements between each other.
> >
> > It appears that either ipa12.mgmt lost some level of its
> replication
> > agreement with ipa13.  I saw some level because users /
> hosts were
> > replicated between all systems but we started seeing DNS
> was not
> > resolving properly from ipa12.  I do not know when this
> started.
> >
> > When looking at replication agreements on ipa12 I did not
> see any
>

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-05 Thread Michael Plemmons 
I just realized that I sent the reply directly to Rob and not to the list.
My response is inline




*Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
614.427.2411
mike.plemm...@crosschx.com
www.crosschx.com

On Thu, May 4, 2017 at 9:39 AM, Michael Plemmons <
michael.plemm...@crosschx.com> wrote:

>
>
>
>
>
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
> 614.427.2411
> mike.plemm...@crosschx.com
> www.crosschx.com
>
> On Thu, May 4, 2017 at 9:24 AM, Rob Crittenden 
> wrote:
>
>> Michael Plemmons wrote:
>> > I realized that I was not very clear in my statement about testing with
>> > ldapsearch.  I had initially run it without logging in with a DN.  I was
>> > just running the local ldapsearch -x command.  I then tested on
>> > ipa12.mgmt and ipa11.mgmt logging in with a full DN for the admin and
>> > "cn=Directory Manager" from ipa12.mgmt (broken server) and ipa11.mgmt
>> > and both ldapsearch command succeeded.
>> >
>> > I ran the following from ipa12.mgmt and ipa11.mgmt as a non root user.
>> > I also ran the command showing a line count for the output and the line
>> > counts for each were the same when run from ipa12.mgmt and ipa11.mgmt.
>> >
>> > ldapsearch -LLL -h ipa12.mgmt.crosschx.com
>> >  -D "DN" -w PASSWORD -b
>> > "cn=users,cn=accounts,dc=mgmt,dc=crosschx,dc=com" dn
>> >
>> > ldapsearch -LLL -h ipa12.mgmt.crosschx.com
>> >  -D "cn=directory manager" -w PASSWORD
>> dn
>>
>> The CA has its own suffix and replication agreements. Given the auth
>> error and recent (5 months) renewal of CA credentials I'd check that the
>> CA agent authentication entries are correct.
>>
>> Against each master with a CA run:
>>
>> $ ldapsearch -LLL -x -D 'cn=directory manager' -W -b
>> uid=ipara,ou=people,o=ipaca description
>>
>> The format is 2;serial#,subject,issuer
>>
>> Then on each run:
>>
>> # certutil -L -d /etc/httpd/alias -n ipaCert |grep Serial
>>
>> The serial # should match that in the description everywhere.
>>
>> rob
>>
>>
>
> On the CA (IPA13.MGMT) I ran the ldapsearch command and see that the
> serial number is 7.  I then ran the certutil command on all three servers
> and the serial number is 7 as well.
>
>
> I also ran the ldapsearch command against the other two servers and they
> also showed a serial number of 7.
>

>

> >
>> >
>> >
>> >
>> >
>> > *Mike Plemmons | Senior DevOps Engineer | CROSSCHX
>> > *
>> > 614.427.2411
>> > mike.plemm...@crosschx.com 
>> > www.crosschx.com 
>> >
>> > On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons
>> > >
>> > wrote:
>> >
>> > I have a three node IPA cluster.
>> >
>> > ipa11.mgmt - was a master over 6 months ago
>> > ipa13.mgmt - current master
>> > ipa12.mgmt
>> >
>> > ipa13 has agreements with ipa11 and ipa12.  ipa11 and ipa12 do not
>> > have agreements between each other.
>> >
>> > It appears that either ipa12.mgmt lost some level of its replication
>> > agreement with ipa13.  I saw some level because users / hosts were
>> > replicated between all systems but we started seeing DNS was not
>> > resolving properly from ipa12.  I do not know when this started.
>> >
>> > When looking at replication agreements on ipa12 I did not see any
>> > agreement with ipa13.
>> >
>> > When I run ipa-replica-manage list all three hosts show has master.
>> >
>> > When I run ipa-replica-manage ipa11.mgmt I see ipa13.mgmt is a
>> replica.
>> >
>> > When I run ipa-replica-manage ipa12.mgmt nothing returned.
>> >
>> > I ran ipa-replica-manage connect --cacert=/etc/ipa/ca.crt
>> > ipa12.mgmt.crosschx.com 
>> > ipa13.mgmt.crosschx.com  on
>> ipa12.mgmt
>> >
>> > I then ran the following
>> >
>> > ipa-replica-manage force-sync --from ipa13.mgmt.crosschx.com
>> > 
>> >
>> > ipa-replica-manage re-initialize --from ipa13.mgmt.crosschx.com
>> > 
>> >
>> > I was still seeing bad DNS returns when dig'ing against ipa12.mgmt.
>> > I was able to create user and DNS records and see the information
>> > replicated properly across all three nodes.
>> >
>> > I then ran ipactl stop on ipa12.mgmt and then ipactl start on
>> > ipa12.mgmt because I wanted to make sure everything was running
>> > fresh after the changes above.  While IPA was staring up (DNS
>> > started) we were able to see valid DNS queries returned but
>> > pki-tomcat would not start.
>> >
>> > I am not sure what I need to do in order to get this working.  I
>> > have included the output of certutil and getcert below from all
>> > three servers as well as the debug output for pki.
>> >
>> >
>> > While the IPA system is coming up I am able to

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-04 Thread Rob Crittenden 
Michael Plemmons wrote:
> I realized that I was not very clear in my statement about testing with
> ldapsearch.  I had initially run it without logging in with a DN.  I was
> just running the local ldapsearch -x command.  I then tested on
> ipa12.mgmt and ipa11.mgmt logging in with a full DN for the admin and
> "cn=Directory Manager" from ipa12.mgmt (broken server) and ipa11.mgmt
> and both ldapsearch command succeeded. 
> 
> I ran the following from ipa12.mgmt and ipa11.mgmt as a non root user. 
> I also ran the command showing a line count for the output and the line
> counts for each were the same when run from ipa12.mgmt and ipa11.mgmt.
> 
> ldapsearch -LLL -h ipa12.mgmt.crosschx.com
>  -D "DN" -w PASSWORD -b
> "cn=users,cn=accounts,dc=mgmt,dc=crosschx,dc=com" dn
> 
> ldapsearch -LLL -h ipa12.mgmt.crosschx.com
>  -D "cn=directory manager" -w PASSWORD dn

The CA has its own suffix and replication agreements. Given the auth
error and recent (5 months) renewal of CA credentials I'd check that the
CA agent authentication entries are correct.

Against each master with a CA run:

$ ldapsearch -LLL -x -D 'cn=directory manager' -W -b
uid=ipara,ou=people,o=ipaca description

The format is 2;serial#,subject,issuer

Then on each run:

# certutil -L -d /etc/httpd/alias -n ipaCert |grep Serial

The serial # should match that in the description everywhere.

rob

> 
> 
> 
> 
> 
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX
> *
> 614.427.2411
> mike.plemm...@crosschx.com 
> www.crosschx.com 
> 
> On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons
> >
> wrote:
> 
> I have a three node IPA cluster.
> 
> ipa11.mgmt - was a master over 6 months ago
> ipa13.mgmt - current master
> ipa12.mgmt
> 
> ipa13 has agreements with ipa11 and ipa12.  ipa11 and ipa12 do not
> have agreements between each other.
> 
> It appears that either ipa12.mgmt lost some level of its replication
> agreement with ipa13.  I saw some level because users / hosts were
> replicated between all systems but we started seeing DNS was not
> resolving properly from ipa12.  I do not know when this started.
> 
> When looking at replication agreements on ipa12 I did not see any
> agreement with ipa13.
> 
> When I run ipa-replica-manage list all three hosts show has master.
> 
> When I run ipa-replica-manage ipa11.mgmt I see ipa13.mgmt is a replica.
> 
> When I run ipa-replica-manage ipa12.mgmt nothing returned.
> 
> I ran ipa-replica-manage connect --cacert=/etc/ipa/ca.crt
> ipa12.mgmt.crosschx.com 
> ipa13.mgmt.crosschx.com  on ipa12.mgmt
> 
> I then ran the following
> 
> ipa-replica-manage force-sync --from ipa13.mgmt.crosschx.com
> 
> 
> ipa-replica-manage re-initialize --from ipa13.mgmt.crosschx.com
> 
> 
> I was still seeing bad DNS returns when dig'ing against ipa12.mgmt. 
> I was able to create user and DNS records and see the information
> replicated properly across all three nodes.
> 
> I then ran ipactl stop on ipa12.mgmt and then ipactl start on
> ipa12.mgmt because I wanted to make sure everything was running
> fresh after the changes above.  While IPA was staring up (DNS
> started) we were able to see valid DNS queries returned but
> pki-tomcat would not start.
> 
> I am not sure what I need to do in order to get this working.  I
> have included the output of certutil and getcert below from all
> three servers as well as the debug output for pki.
> 
> 
> While the IPA system is coming up I am able to successfully run
> ldapsearch -x as the root user and see results.  I am also able to
> login with the "cn=Directory Manager" account and see results.
> 
> 
> The debug log shows the following error.
> 
> 
> [03/May/2017:21:22:01][localhost-startStop-1]:
> 
> [03/May/2017:21:22:01][localhost-startStop-1]: =  DEBUG
> SUBSYSTEM INITIALIZED   ===
> [03/May/2017:21:22:01][localhost-startStop-1]:
> 
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
> autoShutdown? false
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine:
> autoShutdown crumb file path?
> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to
> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
> cert:auditSigningCert cert-pki-ca
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine:

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-03 Thread Michael Plemmons 
I also looked at RUVs and here is what I found.  I do not know if anything
here is helpful.

ldapsearch -ZZ -h ipa11.mgmt.crosschx.com -D "cn=Directory Manager" -W -b
"o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
| grep "nsds50ruv\|nsDS5ReplicaId"
nsDS5ReplicaId: 1095
nsds50ruv: {replicageneration} 583445980060
nsds50ruv: {replica 1095 ldap://ipa11.mgmt.crosschx.com:389}
5865323f04470
nsds50ruv: {replica 86 ldap://ipa13.mgmt.crosschx.com:389}
58651fdb0056000
nsds50ruv: {replica 96 ldap://ipa11.mgmt.crosschx.com:389}
5834459c006
nsds50ruv: {replica 91 ldap://ipa13.mgmt.crosschx.com:389}
58344997005b000
nsds50ruv: {replica 97 ldap://ipa12.mgmt.crosschx.com:389}
583445c30061000
nsds50ruv: {replica 81 ldap://ipa12.mgmt.crosschx.com:389}
586529560051000

IPA12 - this is the problem node.
ldapsearch -ZZ -h ipa12.mgmt.crosschx.com -D "cn=Directory Manager" -W -b
"o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
| grep "nsds50ruv\|nsDS5ReplicaId"
nsDS5ReplicaId: 81
nsds50ruv: {replicageneration} 583445980060
nsds50ruv: {replica 81 ldap://ipa12.mgmt.crosschx.com:389}
586529560051000
nsds50ruv: {replica 96 ldap://ipa11.mgmt.crosschx.com:389}
5834459c006
nsds50ruv: {replica 86 ldap://ipa13.mgmt.crosschx.com:389}
58651fdb0056000
nsds50ruv: {replica 91 ldap://ipa13.mgmt.crosschx.com:389}
58344997005b000
nsds50ruv: {replica 97 ldap://ipa12.mgmt.crosschx.com:389}
583445c30061000

ldapsearch -ZZ -h ipa13.mgmt.crosschx.com -D "cn=Directory Manager" -W -b
"o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
| grep "nsds50ruv\|nsDS5ReplicaId"
nsDS5ReplicaId: 86
nsds50ruv: {replicageneration} 583445980060
nsds50ruv: {replica 86 ldap://ipa13.mgmt.crosschx.com:389}
58651fdb0056000
nsds50ruv: {replica 1095 ldap://ipa11.mgmt.crosschx.com:389}
5865323f04470
nsds50ruv: {replica 96 ldap://ipa11.mgmt.crosschx.com:389}
5834459c006
nsds50ruv: {replica 91 ldap://ipa13.mgmt.crosschx.com:389}
58344997005b000
nsds50ruv: {replica 97 ldap://ipa12.mgmt.crosschx.com:389}
583445c30061000
nsds50ruv: {replica 81 ldap://ipa12.mgmt.crosschx.com:389}
586529560051000





*Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
614.427.2411
mike.plemm...@crosschx.com
www.crosschx.com

On Wed, May 3, 2017 at 10:52 PM, Michael Plemmons <
michael.plemm...@crosschx.com> wrote:

> I ran another test.  I started IPA with the ignore service failure option
> and I tired doing ldap searches like this.
>
> ldapsearch -H ldaps://ipa12.mgmt.crosschx.com
>
> from both my laptop and from ipa11.mgmt and I get successful returns when
> logging in as the admin user and as the directory manager.
>
> I then looked closer at the LDAP access logs for the last time I tried to
> start up PKI and got the auth failure and i see this.
>
>
> [04/May/2017:02:22:45.859021005 +] conn=12 fd=101 slot=101 SSL
> connection from 10.71.100.92 to 10.71.100.92
> [04/May/2017:02:22:45.875672450 +] conn=12 TLS1.2 256-bit AES
> [04/May/2017:02:22:45.940908536 +] conn=12 op=0 BIND dn=""
> method=sasl version=3 mech=EXTERNAL
> [04/May/2017:02:22:45.942441120 +] conn=12 op=0 RESULT err=48 tag=97
> nentries=0 etime=0
>
> Is dn="" supposed to be empty?
>
>
>
>
>
>
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
> 614.427.2411
> mike.plemm...@crosschx.com
> www.crosschx.com
>
> On Wed, May 3, 2017 at 10:16 PM, Michael Plemmons <
> michael.plemm...@crosschx.com> wrote:
>
>> I realized that I was not very clear in my statement about testing with
>> ldapsearch.  I had initially run it without logging in with a DN.  I was
>> just running the local ldapsearch -x command.  I then tested on ipa12.mgmt
>> and ipa11.mgmt logging in with a full DN for the admin and "cn=Directory
>> Manager" from ipa12.mgmt (broken server) and ipa11.mgmt and both ldapsearch
>> command succeeded.
>>
>> I ran the following from ipa12.mgmt and ipa11.mgmt as a non root user.  I
>> also ran the command showing a line count for the output and the line
>> counts for each were the same when run from ipa12.mgmt and ipa11.mgmt.
>>
>> ldapsearch -LLL -h ipa12.mgmt.crosschx.com -D "DN" -w PASSWORD -b
>> "cn=users,cn=accounts,dc=mgmt,dc=crosschx,dc=com" dn
>>
>> ldapsearch -LLL -h ipa12.mgmt.crosschx.com -D "cn=directory manager" -w
>> PASSWORD dn
>>
>>
>>
>>
>>
>>
>> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
>> 614.427.2411
>> mike.plemm...@crosschx.com
>> www.crosschx.com
>>
>> On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons <
>> michael.plemm...@crosschx.com> wrote:
>>
>>> I have a three node IPA cluster.
>>>
>>> ipa11.mgmt - was a master over 6 months ago
>>> ipa13.mgmt - current master
>>> ipa12.mgmt
>>>
>>> ipa13 has agreements with ipa11 and ipa12.  ipa11 and ipa12 do not have
>>> agreements between each other.
>>>
>>> It appears that either ipa12.mgmt lost some level of its

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-03 Thread Michael Plemmons 
I ran another test.  I started IPA with the ignore service failure option
and I tired doing ldap searches like this.

ldapsearch -H ldaps://ipa12.mgmt.crosschx.com

from both my laptop and from ipa11.mgmt and I get successful returns when
logging in as the admin user and as the directory manager.

I then looked closer at the LDAP access logs for the last time I tried to
start up PKI and got the auth failure and i see this.


[04/May/2017:02:22:45.859021005 +] conn=12 fd=101 slot=101 SSL
connection from 10.71.100.92 to 10.71.100.92
[04/May/2017:02:22:45.875672450 +] conn=12 TLS1.2 256-bit AES
[04/May/2017:02:22:45.940908536 +] conn=12 op=0 BIND dn="" method=sasl
version=3 mech=EXTERNAL
[04/May/2017:02:22:45.942441120 +] conn=12 op=0 RESULT err=48 tag=97
nentries=0 etime=0

Is dn="" supposed to be empty?






*Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
614.427.2411
mike.plemm...@crosschx.com
www.crosschx.com

On Wed, May 3, 2017 at 10:16 PM, Michael Plemmons <
michael.plemm...@crosschx.com> wrote:

> I realized that I was not very clear in my statement about testing with
> ldapsearch.  I had initially run it without logging in with a DN.  I was
> just running the local ldapsearch -x command.  I then tested on ipa12.mgmt
> and ipa11.mgmt logging in with a full DN for the admin and "cn=Directory
> Manager" from ipa12.mgmt (broken server) and ipa11.mgmt and both ldapsearch
> command succeeded.
>
> I ran the following from ipa12.mgmt and ipa11.mgmt as a non root user.  I
> also ran the command showing a line count for the output and the line
> counts for each were the same when run from ipa12.mgmt and ipa11.mgmt.
>
> ldapsearch -LLL -h ipa12.mgmt.crosschx.com -D "DN" -w PASSWORD -b
> "cn=users,cn=accounts,dc=mgmt,dc=crosschx,dc=com" dn
>
> ldapsearch -LLL -h ipa12.mgmt.crosschx.com -D "cn=directory manager" -w
> PASSWORD dn
>
>
>
>
>
>
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
> 614.427.2411
> mike.plemm...@crosschx.com
> www.crosschx.com
>
> On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons <
> michael.plemm...@crosschx.com> wrote:
>
>> I have a three node IPA cluster.
>>
>> ipa11.mgmt - was a master over 6 months ago
>> ipa13.mgmt - current master
>> ipa12.mgmt
>>
>> ipa13 has agreements with ipa11 and ipa12.  ipa11 and ipa12 do not have
>> agreements between each other.
>>
>> It appears that either ipa12.mgmt lost some level of its replication
>> agreement with ipa13.  I saw some level because users / hosts were
>> replicated between all systems but we started seeing DNS was not resolving
>> properly from ipa12.  I do not know when this started.
>>
>> When looking at replication agreements on ipa12 I did not see any
>> agreement with ipa13.
>>
>> When I run ipa-replica-manage list all three hosts show has master.
>>
>> When I run ipa-replica-manage ipa11.mgmt I see ipa13.mgmt is a replica.
>>
>> When I run ipa-replica-manage ipa12.mgmt nothing returned.
>>
>> I ran ipa-replica-manage connect --cacert=/etc/ipa/ca.crt
>> ipa12.mgmt.crosschx.com ipa13.mgmt.crosschx.com on ipa12.mgmt
>>
>> I then ran the following
>>
>> ipa-replica-manage force-sync --from ipa13.mgmt.crosschx.com
>>
>> ipa-replica-manage re-initialize --from ipa13.mgmt.crosschx.com
>>
>> I was still seeing bad DNS returns when dig'ing against ipa12.mgmt.  I
>> was able to create user and DNS records and see the information replicated
>> properly across all three nodes.
>>
>> I then ran ipactl stop on ipa12.mgmt and then ipactl start on ipa12.mgmt
>> because I wanted to make sure everything was running fresh after the
>> changes above.  While IPA was staring up (DNS started) we were able to see
>> valid DNS queries returned but pki-tomcat would not start.
>>
>> I am not sure what I need to do in order to get this working.  I have
>> included the output of certutil and getcert below from all three servers as
>> well as the debug output for pki.
>>
>>
>> While the IPA system is coming up I am able to successfully run
>> ldapsearch -x as the root user and see results.  I am also able to login
>> with the "cn=Directory Manager" account and see results.
>>
>>
>> The debug log shows the following error.
>>
>>
>> [03/May/2017:21:22:01][localhost-startStop-1]:
>> 
>> [03/May/2017:21:22:01][localhost-startStop-1]: =  DEBUG SUBSYSTEM
>> INITIALIZED   ===
>> [03/May/2017:21:22:01][localhost-startStop-1]:
>> 
>> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
>> autoShutdown? false
>> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: autoShutdown
>> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to look
>> for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
>> cert:auditSigningCert cert-pki-ca
>>

Re: [Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-03 Thread Michael Plemmons 
I realized that I was not very clear in my statement about testing with
ldapsearch.  I had initially run it without logging in with a DN.  I was
just running the local ldapsearch -x command.  I then tested on ipa12.mgmt
and ipa11.mgmt logging in with a full DN for the admin and "cn=Directory
Manager" from ipa12.mgmt (broken server) and ipa11.mgmt and both ldapsearch
command succeeded.

I ran the following from ipa12.mgmt and ipa11.mgmt as a non root user.  I
also ran the command showing a line count for the output and the line
counts for each were the same when run from ipa12.mgmt and ipa11.mgmt.

ldapsearch -LLL -h ipa12.mgmt.crosschx.com -D "DN" -w PASSWORD -b
"cn=users,cn=accounts,dc=mgmt,dc=crosschx,dc=com" dn

ldapsearch -LLL -h ipa12.mgmt.crosschx.com -D "cn=directory manager" -w
PASSWORD dn






*Mike Plemmons | Senior DevOps Engineer | CROSSCHX*
614.427.2411
mike.plemm...@crosschx.com
www.crosschx.com

On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons <
michael.plemm...@crosschx.com> wrote:

> I have a three node IPA cluster.
>
> ipa11.mgmt - was a master over 6 months ago
> ipa13.mgmt - current master
> ipa12.mgmt
>
> ipa13 has agreements with ipa11 and ipa12.  ipa11 and ipa12 do not have
> agreements between each other.
>
> It appears that either ipa12.mgmt lost some level of its replication
> agreement with ipa13.  I saw some level because users / hosts were
> replicated between all systems but we started seeing DNS was not resolving
> properly from ipa12.  I do not know when this started.
>
> When looking at replication agreements on ipa12 I did not see any
> agreement with ipa13.
>
> When I run ipa-replica-manage list all three hosts show has master.
>
> When I run ipa-replica-manage ipa11.mgmt I see ipa13.mgmt is a replica.
>
> When I run ipa-replica-manage ipa12.mgmt nothing returned.
>
> I ran ipa-replica-manage connect --cacert=/etc/ipa/ca.crt
> ipa12.mgmt.crosschx.com ipa13.mgmt.crosschx.com on ipa12.mgmt
>
> I then ran the following
>
> ipa-replica-manage force-sync --from ipa13.mgmt.crosschx.com
>
> ipa-replica-manage re-initialize --from ipa13.mgmt.crosschx.com
>
> I was still seeing bad DNS returns when dig'ing against ipa12.mgmt.  I was
> able to create user and DNS records and see the information replicated
> properly across all three nodes.
>
> I then ran ipactl stop on ipa12.mgmt and then ipactl start on ipa12.mgmt
> because I wanted to make sure everything was running fresh after the
> changes above.  While IPA was staring up (DNS started) we were able to see
> valid DNS queries returned but pki-tomcat would not start.
>
> I am not sure what I need to do in order to get this working.  I have
> included the output of certutil and getcert below from all three servers as
> well as the debug output for pki.
>
>
> While the IPA system is coming up I am able to successfully run ldapsearch
> -x as the root user and see results.  I am also able to login with the
> "cn=Directory Manager" account and see results.
>
>
> The debug log shows the following error.
>
>
> [03/May/2017:21:22:01][localhost-startStop-1]:
> 
> [03/May/2017:21:22:01][localhost-startStop-1]: =  DEBUG SUBSYSTEM
> INITIALIZED   ===
> [03/May/2017:21:22:01][localhost-startStop-1]:
> 
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
> autoShutdown? false
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: autoShutdown
> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to look
> for cert for auto-shutdown support:auditSigningCert cert-pki-ca
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
> cert:auditSigningCert cert-pki-ca
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: done init
> id=debug
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initialized
> debug
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initSubsystem
> id=log
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: ready to init
> id=log
> [03/May/2017:21:22:01][localhost-startStop-1]: Creating
> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
> [03/May/2017:21:22:01][localhost-startStop-1]: Creating
> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
> [03/May/2017:21:22:01][localhost-startStop-1]: Creating
> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
> autoShutdown? false
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: autoShutdown
> crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to look
> for cert for auto-shutdown support:auditSigningCert cert-pki-ca
> [03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
> cert:auditSigningCert cert-pki-ca
>

[Freeipa-users] Could not connect to LDAP server host - IO Error creating JSS SSL Socket:

2017-05-03 Thread Michael Plemmons 
I have a three node IPA cluster.

ipa11.mgmt - was a master over 6 months ago
ipa13.mgmt - current master
ipa12.mgmt

ipa13 has agreements with ipa11 and ipa12.  ipa11 and ipa12 do not have
agreements between each other.

It appears that either ipa12.mgmt lost some level of its replication
agreement with ipa13.  I saw some level because users / hosts were
replicated between all systems but we started seeing DNS was not resolving
properly from ipa12.  I do not know when this started.

When looking at replication agreements on ipa12 I did not see any agreement
with ipa13.

When I run ipa-replica-manage list all three hosts show has master.

When I run ipa-replica-manage ipa11.mgmt I see ipa13.mgmt is a replica.

When I run ipa-replica-manage ipa12.mgmt nothing returned.

I ran ipa-replica-manage connect --cacert=/etc/ipa/ca.crt
ipa12.mgmt.crosschx.com ipa13.mgmt.crosschx.com on ipa12.mgmt

I then ran the following

ipa-replica-manage force-sync --from ipa13.mgmt.crosschx.com

ipa-replica-manage re-initialize --from ipa13.mgmt.crosschx.com

I was still seeing bad DNS returns when dig'ing against ipa12.mgmt.  I was
able to create user and DNS records and see the information replicated
properly across all three nodes.

I then ran ipactl stop on ipa12.mgmt and then ipactl start on ipa12.mgmt
because I wanted to make sure everything was running fresh after the
changes above.  While IPA was staring up (DNS started) we were able to see
valid DNS queries returned but pki-tomcat would not start.

I am not sure what I need to do in order to get this working.  I have
included the output of certutil and getcert below from all three servers as
well as the debug output for pki.


While the IPA system is coming up I am able to successfully run ldapsearch
-x as the root user and see results.  I am also able to login with the
"cn=Directory Manager" account and see results.


The debug log shows the following error.


[03/May/2017:21:22:01][localhost-startStop-1]:

[03/May/2017:21:22:01][localhost-startStop-1]: =  DEBUG SUBSYSTEM
INITIALIZED   ===
[03/May/2017:21:22:01][localhost-startStop-1]:

[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
autoShutdown? false
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: autoShutdown
crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to look for
cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
cert:auditSigningCert cert-pki-ca
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: done init id=debug
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initialized debug
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initSubsystem
id=log
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: ready to init
id=log
[03/May/2017:21:22:01][localhost-startStop-1]: Creating
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[03/May/2017:21:22:01][localhost-startStop-1]: Creating
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[03/May/2017:21:22:01][localhost-startStop-1]: Creating
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
autoShutdown? false
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: autoShutdown
crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to look for
cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
cert:auditSigningCert cert-pki-ca
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: done init id=log
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initialized log
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initSubsystem
id=jss
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: ready to init
id=jss
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: restart at
autoShutdown? false
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: autoShutdown
crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: about to look for
cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: found
cert:auditSigningCert cert-pki-ca
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: done init id=jss
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initialized jss
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: initSubsystem
id=dbs
[03/May/2017:21:22:01][localhost-startStop-1]: CMSEngine: ready to init
id=dbs
[03/May/2017:21:22:01][localhost-startStop-1]: DBSubsystem: init()
 mEnableSerialMgmt=true