On 22.6.2016 23:09, Sean Hogan wrote: > SLAPD showing > > 22/Jun/2016:17:01:59 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) > [22/Jun/2016:17:06:59 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 > (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: > gss_accept_sec_context) errno 0 (Success) > > > where would these creds be and what ID? I am using SASL so I assume it to > be sasl_user DNS/FirstMaster.watson.local or something like that?
These are in /etc/dirsrv/ds.keytab. I would start with # klist -kt /etc/dirsrv/ds.keytab and try to proceed with kinit etc. (very similarly to the bind-dyndb-ldap how-to). I hope it helps. Petr^2 Spacek > From: Sean Hogan/Durham/IBM@IBMUS > To: Petr Spacek <pspa...@redhat.com> > Cc: freeipa-users@redhat.com > Date: 06/22/2016 08:36 AM > Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem > Sent by: freeipa-users-boun...@redhat.com > > > > Hi Peter... > > Yes..... this has me doing loops in my head to /dev/null > > You are correct I could not complete the BIND steps... I did them yesterday > but did not post results as I wanted to stop bugging you all :) > The initial credential section of that I could not complete nor can I get > an keytab without it and I don't think I have an issue with cert versions > (used the SASL section). The upgrade log from 3.47 to 3.50 on this one > server did show an error with named though. > > I had the box powered down again last night after testing the BIND > procedures... and its been up since then. Which makes we really not sure > what is going on(DNS DOS from internal maybe? I get a lot of outside > requests showing network unreachable and I don't forward to a outside DNS). > If it was a password/cert/cipher/file perm issue then I don't see how it > can work at all after a reboot. > > I am thinking it needs a rebuild.. I have not done this on a First Master > IPA is there anything I need to be take into consider with it being first > master? Right now I have 8 IPAs all DNS, NTP and CAs on differ vlans but > the first master is the fail back IPA(on the only vlan that can talk to the > others) in case there local vlan IPA dies. First Master is also the master > CA in the realm where everything is enrolled to originally. We then mod > everything to point to the vlan IPA with the Firstmaster as secondary with > our vlan-specific scripts we run after ipa client install. > > With the box rebooted last night I am now getting normal functionality but > it prob wont last long as indicated from the past... > > Working > [bob@FirstMaster ~]# kinit admin > Password for admin@DOMAIN.LOCAL: > Warning: Your password will expire in 6 days on Tue Jun 28 14:55:52 2016 > [bob@FirstMaster ~]# > > I did post ldap logs in my first email though... will readd them to this > and when it dies off again I will add more. > > >> [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time >> Directory Server was running, recovering database. >> [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries > set >> up under cn=computers, cn=compat,dc=domain,dc=local >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV >> [database RUV] does not contain element [{replica 7} 55ca26a0000900070000 >> 5688d8e6001000070000] which is present in RUV [changelog max RUV] >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local >> there were some differences between the changelog max RUV and the > database >> RUV. If there are obsolete elements in the database RUV, you should > remove >> them using the CLEANALLRUV task. If they are not obsolete, you should > check >> their status to see why there are no changes from those servers in the >> changelog. >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meTobldvxl0011.domain.local" (1server:389): Replication bind > with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver7.domain.local" (server7:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:48 -0400] - slapd started. Listening on All Interfaces >> port 389 for LDAP requests >> [20/Jun/2016:13:59:48 -0400] - Listening on All Interfaces port 636 for >> LDAPS requests >> [20/Jun/2016:13:59:48 -0400] - Listening >> on /var/run/slapd-DOMAIN-LOCAL.socket for LDAPI requests >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver4.domain.local" (server4:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 0 (Success) >> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with >> GSSAPI auth resumed >> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 >> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: >> gss_accept_sec_context) errno 0 (Success) >> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) >> [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with >> GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): >> authentication failure: GSSAPI Failure: gss_accept_sec_context) >> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (No credentials cache >> found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (No credentials cache >> found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (No credentials cache >> found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (No credentials cache >> found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (No credentials cache >> found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:59:57 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 >> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: >> gss_accept_sec_context) errno 0 (Success) >> [20/Jun/2016:13:59:57 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) >> [20/Jun/2016:13:59:57 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with >> GSSAPI auth resumed > > > > Sean Hogan > > > > > > Inactive hide details for Petr Spacek ---06/21/2016 10:20:43 PM---On > 22.6.2016 02:56, Sean Hogan wrote: > More infoPetr Spacek ---06/21/2016 > 10:20:43 PM---On 22.6.2016 02:56, Sean Hogan wrote: > More info > > From: Petr Spacek <pspa...@redhat.com> > To: freeipa-users@redhat.com > Date: 06/21/2016 10:20 PM > Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem > Sent by: freeipa-users-boun...@redhat.com > > > > On 22.6.2016 02:56, Sean Hogan wrote: >> More info >> >> >> Krb5 log is showing: >> Jun 21 20:42:47 Firstmaster.domain.local krb5kdc[2141](info): AS_REQ (4 >> etypes {18 17 16 23}) 10.x.x.x: LOOKING_UP_CLIENT: admin@domain.LOCAL for >> krbtgt/DOMAIN.LOCAL@DOMAIN.LOCAL, Server error > > > Hello, > > this is really fishy. I would bet that there is a problem with LDAP server > and > DNS errors are just consequence of it. > > I suspect that you will not be able to finish steps mentioned in > https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart#a3.FailedtoinitcredentialsorFailedtogetinitialcredentialsDecryptintegritycheckfailedorClientscredentialshavebeenrevoked > > > If it is the case I would turn your attention to krb5kdc.log and LDAP > server > logs in /var/log/dirsrv/* > > There must be something wrong with the LDAP server. > > Petr^2 Spacek > > >> >> [bob@Firstmaster etc]# kinit -v admin >> kinit: Credentials cache file '/tmp/krb5cc_0' not found while validating >> credentials >> >> >> >> >> >> >> Sean Hogan >> >> >> >> >> >> >> From: Sean Hogan/Durham/IBM >> To: freeipa-users <freeipa-users@redhat.com> >> Date: 06/21/2016 12:02 PM >> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem >> >> >> Has anyone seen these before? >> >> >> >> First Master IPA DNS logs show: Looks like the host names are getting > the >> domain twice domain.local.domain.local >> >> >> client 10.x.x.x#58094: query failed (SERVFAIL) for >> server1.domain.local.domain.local/IN/AAAA at query.c:6569 >> timeout in ldap_pool_getconnection(): try to raise 'connections' > parameter; >> potential deadlock? >> client 10.x.x.x#44147: query failed (SERVFAIL) for >> x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569 >> timeout in ldap_pool_getconnection(): try to raise 'connections' > parameter; >> potential deadlock? >> client 10.x.x.x#56466: query failed (SERVFAIL) for >> x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569 >> timeout in ldap_pool_getconnection(): try to raise 'connections' > parameter; >> potential deadlock? >> client 10.x.x.x53367: query failed (SERVFAIL) for >> server2.domain.local.domain.local/IN/A at query.c:6569 >> timeout in ldap_pool_getconnection(): try to raise 'connections' > parameter; >> potential deadlock? >> client 10.x.x.x#53367: query failed (SERVFAIL) for >> server2.domain.local.domain.local/IN/AAAA at query.c:6569 >> >> >> >> So enrolls are failing at this point when tyring to enroll to a replica: >> >> [bob@server1 log]# ipa-client-install –enable-dns-updates >> Discovery was successful! >> Hostname: server1.watson.local >> Realm: DOMAIN.LOCAL >> DNS Domain: domain.local >> IPA Server: ipareplica.domain.local >> BaseDN: dc=domain,dc=local >> >> Continue to configure the system with these values? [no]: yes >> User authorized to enroll computers: bob >> Synchronizing time with KDC... >> Password for bob@DOMAIN.LOCAL: >> Successfully retrieved CA cert >> Subject: CN=Certificate Authority,O=DOMAIN.LOCAL >> Issuer: CN=Certificate Authority,O=DOMAIN.LOCAL >> Valid From: Tue Jan 06 19:37:09 2015 UTC >> Valid Until: Sat Jan 06 19:37:09 2035 UTC >> >> Enrolled in IPA realm DOMAIN.LOCAL >> Attempting to get host TGT... >> Created /etc/ipa/default.conf >> New SSSD config will be created >> Configured sudoers in /etc/nsswitch.conf >> Configured /etc/sssd/sssd.conf >> Configured /etc/krb5.conf for IPA realm DOMAIN.LOCAL >> trying https://ipareplica.domain.local/ipa/xml >> Cannot connect to the server due to Kerberos error: Kerberos error: >> Kerberos error: ('Unspecified GSS failure. Minor code may provide more >> information', 851968)/('KDC returned error string: PROCESS_TGS', >> -1765328324)/. Trying with delegate=True >> trying https://ipareplica.domain.local/ipa/xml >> Second connect with delegate=True also failed: Kerberos error: Kerberos >> error: ('Unspecified GSS failure. Minor code may provide more >> information', 851968)/('KDC returned error string: PROCESS_TGS', >> -1765328324)/ >> Cannot connect to the IPA server XML-RPC interface: Kerberos error: >> Kerberos error: ('Unspecified GSS failure. Minor code may provide more >> information', 851968)/('KDC returned error string: PROCESS_TGS', >> -1765328324)/ >> Installation failed. Rolling back changes. >> Unenrolling client from IPA server >> Unenrolling host failed: Error obtaining initial credentials: Generic > error >> (see e-text). >> >> Removing Kerberos service principals from /etc/krb5.keytab >> Disabling client Kerberos and LDAP configurations >> Redundant SSSD configuration file /etc/sssd/sssd.conf was moved >> to /etc/sssd/sssd.conf.deleted >> Restoring client configuration files >> nscd daemon is not installed, skip configuration >> nslcd daemon is not installed, skip configuration >> Client uninstall complete. >> >> >> Sean Hogan >> >> >> >> >> >> >> >> >> From: Sean Hogan/Durham/IBM >> To: Sean Hogan/Durham/IBM@IBMUS >> Cc: freeipa-users <freeipa-users@redhat.com> >> Date: 06/20/2016 12:49 PM >> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem >> >> >> Also seeing this in the upgrade log on the first master but not on the 7 >> ipas. >> >> ERROR Failed to restart named: Command '/sbin/service named restart ' >> returned non-zero exit status 7 >> >> >> which led me to >> >> https://bugzilla.redhat.com/show_bug.cgi?id=895298 >> >> >> >> >> >> Sean Hogan >> >> >> >> >> >> >> >> From: Sean Hogan/Durham/IBM@IBMUS >> To: freeipa-users <freeipa-users@redhat.com> >> Date: 06/20/2016 11:46 AM >> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem >> Sent by: freeipa-users-boun...@redhat.com >> >> >> >> Hi All.. >> >> I thought we fixed this issue by rebooting the KVM host but it is showing >> again. Our First Master IPA is being rebooted 2 -5 times a day now just > to >> keep it alive. >> >> What we are seeing: >> >> God@FirstMaster log]# kinit admin >> kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting >> initial credentials >> >> DNS is not working as nslookup is failing to a replica.... think once we >> lose DNS it all goes down hill which makes sense. >> >> [god@FirstMaster log]# ipactl stop -----> Just hangs forever.. no > replies.. >> no error.. nothing >> >> I try service named stop and nothing happens >> >> I have the box hard shutdown from KVM console. Reboot it and it works for > a >> little while but eventually back to same behavior. >> >> At this point I can service named stop and it responds... ipactl status > and >> it responds.. but when if I try service named restart I get >> >> [god@FirstMaster log]# service named stop >> Stopping named: ...... >> >> [god@Firstmaster log]# service named start >> Starting named: [FAILED] >> >> [god@FirstMaster log]# service named status >> rndc: connect failed: 127.0.0.1#953: connection refused >> named dead but pid file exists >> >> Rebooted box and it is hung on shutting down domain-local and never fully >> shuts down.. have to get it hard shutdown again. >> During an attempt to gracefully shut down we see this >> >> Shutting Down dirsrv: >> PKI-IPA OK >> DOMAIN-LOCAL FAILED >> *** Error: 1 instance(s) unsuccessfully stopped FAILED >> >> Then it moves on to shut other things down and returns to dirsrv >> Shutting Down dirsrv: >> PKI-IPA....server already stopped FAILED {Makes sense.. it died earlier} >> DOMAIN-LOCAL... {this sits here til we hard shutdown} >> >> >> >> bind-libs-9.8.2-0.47.rc1.el6.x86_64 >> bind-9.8.2-0.47.rc1.el6.x86_64 >> bind-utils-9.8.2-0.47.rc1.el6.x86_64 >> >> >> ipa-client-3.0.0-50.el6.1.x86_64 >> ipa-server-selinux-3.0.0-50.el6.1.x86_64 >> ipa-server-3.0.0-50.el6.1.x86_64 >> sssd-ipa-1.13.3-22.el6.x86_64 >> >> >> /var/log/dirsrv/slapd-DOMAIN-LOCAL >> [20/Jun/2016:13:29:06 -0400] - 389-Directory/1.2.11.15 B2016.063.2110 >> starting up >> [20/Jun/2016:13:29:06 -0400] schema-compat-plugin - warning: no entries > set >> up under cn=computers, cn=compat,dc=domain,dc=local >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV >> [database RUV] does not contain element [{replica 7} 55ca26a0000900070000 >> 5688d8e6001000070000] which is present in RUV [changelog max RUV] >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local >> there were some differences between the changelog max RUV and the > database >> RUV. If there are obsolete elements in the database RUV, you should > remove >> them using the CLEANALLRUV task. If they are not obsolete, you should > check >> their status to see why there are no changes from those servers in the >> changelog. >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] - slapd started. Listening on All Interfaces >> port 389 for LDAP requests >> [20/Jun/2016:13:29:07 -0400] - Listening on All Interfaces port 636 for >> LDAPS requests >> [20/Jun/2016:13:29:07 -0400] - Listening >> on /var/run/slapd-DOMAIN-LOCAL.socket for LDAPI requests >> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial >> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in >> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC >> for requested realm) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver4.domain.local" (server4:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meTo1server.domain.local" (1server:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver7.domain.local" (server7:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >> failure. Minor code may provide more information (Credentials cache file >> '/tmp/krb5cc_495' not found)) errno 0 (Success) >> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error -2 (Local error) >> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - >> agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with >> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic > failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Credentials cache file '/tmp/krb5cc_495' not found)) >> [20/Jun/2016:13:29:10 -0400] NSMMReplicationPlugin - > -- Petr Spacek @ Red Hat -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project