And yes, I learned to stop using kadmin after I made that note On Thu, Feb 23, 2017 at 11:56 AM, Aaron Young <ayo...@marketfactory.com> wrote:
> on ld4ipa01, I removed it with ipa-server-install --uninstall > > this was an attempt to recreate the replica from nyc02ipa02 > > On Thu, Feb 23, 2017 at 3:17 AM, Martin Basti <mba...@redhat.com> wrote: > >> >> >> On 22.02.2017 23:26, Aaron Young wrote: >> >> Hello Everyone >> >> I recently lost the master master IPA server setup by the previous >> administrator. >> As it stands now, if I try to add a new client, in order to standup a new >> replica, I get errors while trying to setup DNS. This led me to look at how >> authentication worked (I'm new to IPA) and I learned about the kerberos >> tools >> >> I don't know if I'm familiar enough with the terminology to adequately >> describe what I'm experiencing, so I'll give you some of the commands and >> their results >> >> but first, a bit on the design >> >> before I got to this, we had >> >> a <-> b <-> c <-> d >> >> b was the master master >> >> a, happened to point to two test servers nyc02ipa01 and nyc02ipa02 (not >> pictured, I discovered them later when c and d started having problems) >> >> a - nyc01ipa02 >> b - nyc01ipa01 >> c - ld4ipa01 >> d - ld4ipa02 >> >> currently, I have nyc02ipa02 <-> nyc01ipa02 >> >> the reason I have it limited like this is because all the other servers >> stopped replicating for one reason or another (mainly that they can't >> authenticate or in one case, there was a database record corruption) >> >> Anyway, here are some activities and logs from the latest round of fixes >> and information activities I've been engaging in >> >> 22:54:32 root@nyc01ipa02:~# kinit admin >> kinit: Clients credentials have been revoked while getting initial >> credentials >> >> Reading through this >> <http://web.mit.edu/Kerberos/krb5-1.13/doc/admin/lockout.html> tells me >> that >> >> # kadmin: modprinc -unlock PRINCNAME >> >> will unlock an account...but if I can't get in.... >> >> 22:54:37 root@nyc01ipa02:~# kadmin >> Authenticating as principal root/admin@MF with password. >> kadmin: Client 'root/admin@MF' not found in Kerberos database while >> initializing kadmin interface >> >> on ld4ipa02, did a >> >> # ipa-client-install --uninstall >> >> then >> >> # ipa-client-install --force-join --enable-dns-updates --permit -f >> --ssh-trust-dns --request-cert --automount-location=LD4 --enable-dns-updates >> >> DNS did not update, here is the relevant portion from >> /var/log/ipaclient-install.log >> >> 2017-02-20T18:46:49Z DEBUG Writing nsupdate commands to >> /etc/ipa/.dns_update.txt: >> 2017-02-20T18:46:49Z DEBUG debug >> >> update delete ld4ipa02.mf. IN A >> show >> send >> >> update delete ld4ipa02.mf. IN AAAA >> show >> send >> >> update add ld4ipa02.mf. 1200 IN A 10.102.100.140 >> show >> send >> >> 2017-02-20T18:46:49Z DEBUG Starting external process >> 2017-02-20T18:46:49Z DEBUG args=/usr/bin/nsupdate -g /etc/ipa/.dns_update.txt >> 2017-02-20T18:46:49Z DEBUG Process finished, return code=1 >> 2017-02-20T18:46:49Z DEBUG stdout=Outgoing update query: >> ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0 >> ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0 >> ;; UPDATE SECTION: >> ld4ipa02.mf. 0 ANY A >> >> 2017-02-20T18:46:49Z DEBUG stderr=Reply from SOA query: >> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34702 >> ;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 >> ;; QUESTION SECTION: >> ;ld4ipa02.mf. IN SOA >> >> ;; AUTHORITY SECTION: >> mf. 1800 IN SOA ld4ipa01.mf. hostmaster.mf. 1487615509 3600 900 1209600 3600 >> >> Found zone name: mf >> The master is: ld4ipa01.mf >> start_gssrequest >> tkey query failed: GSSAPI error: Major = Unspecified GSS failure. Minor code >> may provide more information, Minor = Server DNS/ld4ipa01.mf@MF not found in >> Kerberos database. >> >> 2017-02-20T18:46:49Z DEBUG nsupdate failed: Command '/usr/bin/nsupdate -g >> /etc/ipa/.dns_update.txt' returned non-zero exit status 1 >> 2017-02-20T18:46:49Z ERROR Failed to update DNS records. >> 2017-02-20T18:46:49Z DEBUG DNS resolver: Query: ld4ipa02.mf IN A >> 2017-02-20T18:46:49Z DEBUG DNS resolver: No record. >> 2017-02-20T18:46:49Z DEBUG DNS resolver: Query: ld4ipa02.mf IN AAAA >> 2017-02-20T18:46:49Z DEBUG DNS resolver: No record. >> 2017-02-20T18:46:49Z DEBUG DNS resolver: Query: 140.100.102.10.in-addr.arpa. >> IN PTR >> 2017-02-20T18:46:49Z DEBUG DNS resolver: No record. >> 2017-02-20T18:46:49Z WARNING Missing A/AAAA record(s) for host ld4ipa02.mf: >> 10.102.100.140. >> 2017-02-20T18:46:49Z WARNING Missing reverse record(s) for address(es): >> 10.102.100.140. >> >> Why isn't there an entry for "DNS/ld4ipa01.mf@MF" in the Kerberos >> database? >> >> klist -ktK /etc/dirsrv/ds.keytab on ld4ipa01 returns >> >> Keytab name: FILE:/etc/dirsrv/ds.keytab >> <http://file/etc/dirsrv/ds.keytab> >> KVNO Timestamp Principal >> ---- ------------------- ------------------------------ >> ------------------------ >> 2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF (0x696a502bc73d209acdd36c42242 >> f7f8aff9dbba1073b34ea018ed3bd9cdfd970) >> 2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF (0xe031464b6948ea34f4291d40fca >> 7a21e) >> 2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF (0xe94a1c98fe79b6317901435d9e9 >> e0257cefe438ff2ec527f) >> 2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF (0x6aaf4c7fa6b51b9de032b7c6428 >> 307b5) >> 2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF (0x5e0702f44aef9e0633e09eede7c >> a8041) >> 2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF (0x6e3a9d29ee3f129a156ae6228ab >> 7728df8ce5de923a61eba6a2e7802b8d230b6) >> >> >> Tried to test connectivity using ldapsearch found that I could connect >> to other hosts on 389 but not 636 >> >> # ldapsearch -H ldap://nyc02ipa02:389 -D "cn=directory manager" -W -b "" -s >> base >> >> # ldapsearch -H ldaps://nyc02ipa02:686 -D "cn=directory manager" -W -b "" -s >> base >> >> Testing the kvno >> >> 02:10:00 root@ld4ipa01:~# kvno DNS/ld4ipa01.mf@MF >> DNS/ld4ipa01.mf@MF: kvno = 2 >> >> 02:10:52 root@ld4ipa02:~# kvno DNS/ld4ipa01.mf@MF >> kvno: Server DNS/ld4ipa01.mf@MF not found in Kerberos database while >> getting credentials for DNS/ld4ipa01.mf@MF >> >> Add this to any command line to get debug on kerberos commands >> >> KRB5_TRACE=/dev/stdout kvno DNS/ld4ipa01.mf@MF >> >> So, looking at the debug >> kvno from ld4ipa02, does not return tickets. It does this because it >> contacts the KDC which is nyc02ipa02, and nyc02ipa02 does not recognize >> ldipa02 as an IPA server. It doesn't recognize ld4ipa01 either. >> >> >> >> right now, if I try to connect nyc02ipa02 to ld4ipa01 I get >> >> 21:56:27 root@nyc02ipa02:~# ipa topologysegment-add domain >> ld4ipa01-to-nyc02ipa02 --leftnode ld4ipa01.mf --rightnode nyc02ipa02.mf >> ipa: ERROR: invalid 'leftnode': left node is not a topology node: >> ld4ipa01.mf >> >> ipa privilege-show 'DNS Servers' --all --raw >> >> dn: cn=DNS Servers,cn=privileges,cn=pbac,dc=mf >> cn: DNS Servers >> description: DNS Servers >> member: krbprincipalname=DNS/nyc01ipa02.mf@MF,cn=services,cn=accounts,dc=mf >> member: >> krbprincipalname=ipa-dnskeysyncd/nyc01ipa02.mf@MF,cn=services,cn=accounts,dc=mf >> member: krbprincipalname=DNS/nyc02ipa02.mf@MF,cn=services,cn=accounts,dc=mf >> member: >> krbprincipalname=ipa-dnskeysyncd/nyc02ipa02.mf@MF,cn=services,cn=accounts,dc=mf >> member: >> krbprincipalname=ipa-ods-exporter/nyc01ipa02.mf@MF,cn=services,cn=accounts,dc=mf >> memberof: cn=System: Read DNS Configuration,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Write DNS Configuration,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Add DNS Entries,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Manage DNSSEC keys,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Manage DNSSEC metadata,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Read DNS Entries,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Remove DNS Entries,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Update DNS Entries,cn=permissions,cn=pbac,dc=mf >> memberof: cn=System: Read DNS Servers >> Configuration,cn=permissions,cn=pbac,dc=mf >> objectClass: top >> objectClass: groupofnames >> objectClass: nestedgroup >> >> >> -- >> Aaron Young >> MarketFactory, Manager of Site Reliability Engineering >> 425 Broadway, 3FL >> New York, NY 10013 >> Office: +1 212 625 9988 <(212)%20625-9988> >> Direct +1 646 779 3710 <(646)%20779-3710> >> US Support: +1 (212) 625-0688 | UK Support: +44 (0) 203 695-7997 >> >> >> >> Hello, >> >> please don't use kadmin utility, it is not integrated very well with IPA >> (or unsupported is a better word) and may break it even more. It looks that >> you have corrupted kerberos credentials for id4ipa01 >> >> I see you are installing client on id4ipa01, was IPA server removed >> properly previosly? >> >> Martin >> > > > > -- > Aaron Young > MarketFactory, Manager of Site Reliability Engineering > 425 Broadway, 3FL > New York, NY 10013 > Office: +1 212 625 9988 <(212)%20625-9988> > Direct +1 646 779 3710 <(646)%20779-3710> > US Support: +1 (212) 625-0688 | UK Support: +44 (0) 203 695-7997 > -- Aaron Young MarketFactory, Manager of Site Reliability Engineering 425 Broadway, 3FL New York, NY 10013 Office: +1 212 625 9988 Direct +1 646 779 3710 US Support: +1 (212) 625-0688 | UK Support: +44 (0) 203 695-7997
-- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project