[Freeipa-users] Re: Failed Upgrade?

Fri, 04 Aug 2017 02:18:12 -0700 

On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:

On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:

On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:

On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:

On 08/02/2017 01:43 AM, Ian Harding wrote:

On 08/01/2017 12:03 PM, Rob Crittenden wrote:

Ian Harding wrote:

On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:

On 08/01/2017 03:11 PM, Ian Harding wrote:

On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:

On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:



On 07/31/2017 11:34 AM, Rob Crittenden wrote:

Ian Harding via FreeIPA-users wrote:

I had an unexpected restart of an IPA server that had 
apparently had
updates run but had not been restarted.  ipactl says 
pki-tomcatd

would
not start.

Strangely, the actual service appears to be running:




dogtag is an application within tomcat so tomcat can run 
without

dogtag
running.


We need to see more of the dogtag debug log to see what is 
going on.




It looks like an authentication problem...


[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake 
happened
Could not connect to LDAP server host seattlenfs.bpt.rocks 
port 636

Error netscape.ldap.LDAPException: Authentication failed (49)



Hi,

dogtag stores its internal data in the LDAP server and needs to
establish a secure LDAP connection. You can check how this

connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, 
look for

the lines:

internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
internaldb.ldapconn.host=vm-...
internaldb.ldapconn.port=636
internaldb.ldapconn.secureConn

authtype can be SslClientAuth (authentication with a ssl
certificate) or BasicAuth (authentication with a bind DN and
password stored in /var/lib/pki/pki-tomcat/conf/password.conf).


You can use this information to manually check the 
credentials. For

instance with sslclientauth:

export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
export LDAPTLS_CERT='subsystemCert cert-pki-ca'

ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL
(provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)



I found this:

internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
internaldb.ldapconn.cloneReplicationPort=389
...


and when I try the ldapsearch I am presented with a prompt to 
provide

a pin/password


Please enter pin, password, or pass phrase for security token 
'ldap(0)':


but there is no password file...


Hi,


you are right, in 4.4. there is no pwdfile.txt and the password 
can be

found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag
internal=...)


Can you check if the password with the tag internal=... allows 
to read

the keys from the NSS db?
certutil -K -d /etc/pki/pki-tomcat/alias
(provide password)


That works...

# certutil -K -d /etc/pki/pki-tomcat/alias

certutil: Checking token "NSS Certificate DB" in slot "NSS User 
Private

Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa      0f327e760a7eecdcf6973f5dc57ca5367c592d64   (orphan)

< 1> rsa      b12580c7c696cfcd8aefc9405a7a870b24b7b96a   NSS 
Certificate

DB:auditSigningCert cert-pki-ca
< 2> rsa      881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert
cert-pki-ca

< 3> rsa      fa9a255a1d15585ac28064c0f4986e416bc48403   NSS 
Certificate

DB:ocspSigningCert cert-pki-ca
< 4> rsa      3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f   Server-Cert
cert-pki-ca

< 5> rsa      1e9479a9556af9339bb5e4552ccbd381d3c38856   NSS 
Certificate

DB:subsystemCert cert-pki-ca

But this doesn't (with the same password from password.conf)

# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL

Please enter pin, password, or pass phrase for security token 
'ldap(0)':

SASL/EXTERNAL authentication started
ldap_sasl_interactive_bind_s: Invalid credentials (49)

That password is getting me somewhere though, since if I put in a
nonsense or incorrect password it just prompts over and over.


Let's step back a second. You upgraded from what to what?



There wasn't much of a change... I just assumed someone ran yum 
upgrade and didn't restart, then the power outage... it looks like 
not much of a version change though.


# grep ipa /var/log/yum.log
Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:45:32 Installed: 
ipa-client-common-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64
Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch

Jan 08 04:47:04 Installed: 
python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Installed: 
python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:47:05 Installed: 
ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:06 Installed: 
python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64
Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64
Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64
Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:47:36 Installed: 
ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch

Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch

Jul 28 09:55:42 Updated: 
ipa-client-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:43 Updated: 
ipa-server-common-4.4.0-14.el7.centos.7.noarch

Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch

Jul 28 09:55:44 Updated: 
python2-ipaclient-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:45 Updated: 
python2-ipaserver-4.4.0-14.el7.centos.7.noarch

Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64
Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64
Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch

Jul 28 09:55:48 Updated: 
ipa-python-compat-4.4.0-14.el7.centos.7.noarch


# grep pki /var/log/yum.log
Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64
Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch
Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64
Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch
Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch
Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64
Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch
Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch




Are you running in SELinux enforcing mode?


# getenforce
Enforcing



If so can you run:

# ausearch -m AVC



# ausearch -m AVC

The NOLOG option to log_format is deprecated. Please use the 
write_logs option.

The NOLOG option is overriding the write_logs current setting.
----
time->Mon Mar 14 10:39:01 2016

type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e 
syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 
a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 
gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 
fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" 
exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 
key=(null)
type=AVC msg=audit(1457977141.767:17537): avc:  denied  { read } 
for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" 
ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 
tcontext=system_u:object_r:unlabeled_t:s0 tclass=file




I suspect that the subsystem cert was renewed and everything wasn't

updated properly. If I'm right this is unrelated to the upgrade it 
just

shone a spotlight on it.

Can you run these commands:

$ ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description



This returns


$ ldapsearch -LLL -D 'cn=directory manager' -W -b 
uid=pkidbuser,ou=people,o=ipaca userCertificate description

Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca

userCertificate:: 
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC

... stuff ...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=

description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA 
Subsystem,O=BPT.ROCKS



and

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' | grep Serial

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert 
cert-pki-ca' -a




These both return


# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert 
cert-pki-ca' | grep Serial

Enter Password or Pin for "NSS Certificate DB":

certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: 
Unrecognized Object Identifier.




# certutil -K -d /etc/pki/pki-tomcat/alias  -n 'subsystemCert 
cert-pki-ca' -a
certutil: Checking token "NSS Certificate DB" in slot "NSS User 
Private Key and Certificate Services"

Enter Password or Pin for "NSS Certificate DB":

certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: 
Unrecognized Object Identifier.



Hi,


you made a typo and requested the keys (-K) instead of the certs 
(-L). Please retry with -L and you should be able to see the 
certificate serial.





[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W 
-b uid=pkidbuser,ou=people,o=ipaca userCertificate description

Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca

userCertificate:: 
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC

...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=

description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA 
Subsystem,O=BPT.ROCKS


So the serial is 4?


[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 
'subsystemCert cert-pki-ca' | grep Serial

         Serial Number: 1341718541 (0x4ff9000d)

Which is NOT 4...


[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias  -n 
'subsystemCert cert-pki-ca' -a

-----BEGIN CERTIFICATE-----
MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
sjeiFbUFtoMnfmHLIYpe5QAR
-----END CERTIFICATE-----

And this cert is NOT the same.


Hi,


when certmonger renews a certificate, it runs pre-save and post-save 
commands (which you can see in the output of getcert list). In your 
case, the post-save command for subsystemCert cert-pki-ca probably 
failed.



This command (on the renewal master) is responsible for copying the 
new cert from the NSS db to the LDAP server. You need first to check 
which IPA server is the renewal master (as the serial numbers are in a 
completely different range, I assume that you have multiple IPA 
servers configured with a CA instance):


$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-`
$ kinit admin

$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b 
"cn=masters,cn=ipa,cn=etc,$BASEDN" 
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
dn: 
cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com 



In this example the renewal master is vm-ipaserver.



I get this


$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b 
"cn=masters,cn=ipa,cn=etc,$BASEDN" 
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn

dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks

Which is the other FreeIPA server.


Hi Ian,


it's getting difficult to follow this thread, so could we step back and 
summarize?



- On server freeipa-sea (=renewal master), the CA component is installed 
and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB 
/etc/pki/pki-tomcat/alias



- On server seattlenfs (where PKI fails to restart), the CA component is 
installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 
in the NSSDB /etc/pki/pki-tomcat/alias.



- In the LDAP servers, the cert is stored in the entry 
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the 
cert from freeipa-sea).



If you check the certificates dates, I guess that the most recent one is 
on server seattlenfs (because of serial numbers). Can you confirm this 
(because our goal is to have the most recent cert stored in all the 
NSSDBs and in LDAP)?


Thanks,
Flo



If your renewal master is the machine on which the NSS DB 
/etc/pki/pki-tomcat/alias contains the newer cert, then you can 
manually run the scripts:



If I'm interpreting this correctly, then it is... so I ran this on the 
broken machine (seattlenfs)



$ sudo /usr/libexec/ipa/certmonger/stop_pkicad

$ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert 
cert-pki-ca"





I ran this, it took a while, then I re-checked and the serial still 
seems to be 4



$ ldapsearch -LLL -D 'cn=directory manager' -W -b 
uid=pkidbuser,ou=people,o=ipaca userCertificate description

Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca

userCertificate:: 
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC

...
  ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=

description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA 
Subsystem,O=BPT.ROCKS


Here's what was in /var/log/pki/pki-tomcat/ca/debug


[03/Aug/2017:13:52:51][localhost-startStop-1]: 
============================================
[03/Aug/2017:13:52:51][localhost-startStop-1]: =====  DEBUG SUBSYSTEM 
INITIALIZED   =======
[03/Aug/2017:13:52:51][localhost-startStop-1]: 
============================================
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at 
autoShutdown? false
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown 
crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look 
for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found 
cert:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init 
id=debug

[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug

[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem 
id=log
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init 
id=log
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating 
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating 
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating 
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at 
autoShutdown? false
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown 
crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look 
for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found 
cert:auditSigningCert cert-pki-ca

[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log

[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem 
id=jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init 
id=jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at 
autoShutdown? false
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown 
crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look 
for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found 
cert:auditSigningCert cert-pki-ca

[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss

[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem 
id=dbs
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init 
id=dbs
[03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() 
mEnableSerialMgmt=true
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating 
LdapBoundConnFactor(DBSubsystem)

[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init

[03/Aug/2017:13:52:51][localhost-startStop-1]: 
LdapBoundConnFactory:doCloning true

[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init()
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends

[03/Aug/2017:13:52:51][localhost-startStop-1]: init: before 
makeConnection errorIfDown is true
[03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: 
errorIfDown true

[03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true

[03/Aug/2017:13:52:51][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: Setting desired cert nickname to: 
subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set 
client auth cert nickname subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: 
SSLClientCertificatSelectionCB: Entering!
[03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: 
ocspSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: 
subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: desired cert found in list: 
subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca

[03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()


I really appreciate you sticking with me through this.  I owe you.

- Ian


After that, check that the LDAP entry has been updated. If it is the 
case, you should be able to restart pki.


Flo


Thank you again for your time!

Ian


Flo.


which seems weird.


The description attribute in the ldapsearch output is of the form 
2;#;DN;DN


The # should match the serial number in the first certutil output

The ASCII blob (minus the BEGIN/END headers) should match one of the
userCertificate attributes.

rob












_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org

























					Reply via email to