Hi Florence, Thank you for taking the time to respond. We have a number of replicas in various campus locations around our region, and our state and country is currently locked-down with COVID restrictions, and it would take days to get to the replicas even without restrictions. We have had wide-scale WAN outages over weeks for this system.
To describe our problem in a simplified manner; We had the IPA replicas ‘Location1R1’ and ‘Location2R1’ talking happily when the WAN between them went away, while it was down we added ‘Location2R2’ to ‘Location2R1’, and manually added some user-ranges for accounts at Location2. We also added a number of ipa-clients at Location2. When the WAN came back, ‘Location1R1’ and ‘Location2R1’ stopped sharing data because the CSN updates were dropped weeks ago. We re-initialized ‘Location2R1’ from ‘Location1R1’, however that has resulted in isolation of ‘Location2R2’. Most logins at Location2 fail, depending on where the DNS round-robin points SSSD at the time of login. We have multiple locations, each with a flavor of this issue. No replica has a complete picture of all replication agreements, host accounts and user accounts. Though we have had some success with continually re-initializing replicas. It was my hope, once the WAN was stabilized to remove and re ipa-server-install at our first server, then remove and re-run ipa-client-install then ipa-replica-install at each secondary server. I believe this would re-constitute the system, effectively by re-installation of that specific service. And yes, this would all be done via SSH. However, we meet the obstacles described in my initial email when performing the install a second time. Sincerely, and with many thanks in advance, Rob From: Florence Renaud via FreeIPA-users <[email protected]> Sent: Wednesday, 2 June 2021 2:16 AM To: FreeIPA users list <[email protected]> Cc: Mattson, Robert (FP) <[email protected]>; Florence Renaud <[email protected]> Subject: [EXTERNAL] [Freeipa-users] Re: IPA Reinstall Hi, the recommended way to uninstall a replica and reinstall it is described in the doc: 1. Uninstall the replica (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/Uninstalling_IPA_Servers) with ipa server-del and ipa-server-install --uninstall 2. re-install the replica as if it was a new one: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/creating-the-replica Was there any reason to backup files and restore them? The replica installation should re-create everything. flo On Wed, May 26, 2021 at 7:32 AM Robert.Mattson--- via FreeIPA-users <[email protected]<mailto:[email protected]>> wrote: Dear Community, I'd like to uninstall and reinstall IPA from a CentOS box because its easier than reinstalling the OS completely. We have a number of replicas, and this host is installed using ipa-client-install and then ipa-replica-install. To remove it, I backup some data like /var/kerberos/krb5kdc/{cacert.pem,kd*} and /etc/httpd/conf/password.conf and then run '/usr/sbin/ipa-server-install --uninstall -U --ignore-topology-disconnect'. I then sed '/Environment=K/d', '/ExecStartPre/d', '/ExecStopPost/d' /etc/systemd/system/httpd.service I recreate the host-account on another replica using ipa host-add, then ipa hostgroup-add-member. On the now-removed host, I do some housekeeping like restoring the backed up files and then I run; /usr/sbin/ipa-client-install \ --password=${otp} \ --mkhomedir \ --no-ntp \ --unattended \ --domain=realm.name<http://realm.name> \ --realm=REALM.NAME<http://REALM.NAME> \ --ca-cert-file=/etc/pki/ca-trust/source/ca.crt then /usr/sbin/ipa-replica-install \ --dirsrv-cert-file=/etc/pki/tls/private/ipa.pkcs12 \ --http-cert-file=/etc/pki/tls/private/ipa.pkcs12 \ --dirsrv-pin=pwd \ --http-pin=pwd \ --unattended \ --no-pkinit \ --no-ntp I seem to get the following keytab request problem followed by dirsrv failure. from ipa-replica-install (4.6.4-10.el7.centos.3.x86_64). If I upgrade to 4.6.8-5.el7.centos.4.noarch.rpm, I get the same problem.[1] On serverb, the host which receives the binding request for the reinstall, I get permission denied the bind dn “” does not have permission in dirsrv error log…? Does anyone have any thoughts, Cheers and many thanks in advance, Rob [1] 2021-05-26T02:50:56Z DEBUG Backing up system configuration file '/etc/httpd/conf.d/ipa.conf' 2021-05-26T02:50:56Z DEBUG -> Not backing up - '/etc/httpd/conf.d/ipa.conf' doesn't exist 2021-05-26T02:50:56Z DEBUG Backing up system configuration file '/etc/httpd/conf.d/ipa-rewrite.conf' 2021-05-26T02:50:56Z DEBUG -> Not backing up - '/etc/httpd/conf.d/ipa-rewrite.conf' doesn't exist 2021-05-26T02:50:56Z DEBUG duration: 0 seconds 2021-05-26T02:50:56Z DEBUG [10/21]: setting up httpd keytab 2021-05-26T02:50:56Z DEBUG raw: service_add(u'HTTP/[email protected]<mailto:[email protected]>', force=True, version=u'2.230') 2021-05-26T02:50:56Z DEBUG service_add(ipapython.kerberos.Principal('HTTP/[email protected]<mailto:[email protected]>'), force=True, all=False, raw=False, version=u'2.230', no_members=False) 2021-05-26T02:50:56Z DEBUG flushing ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket from SchemaCache 2021-05-26T02:50:56Z DEBUG retrieving schema for SchemaCache url=ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f0d29f50368> 2021-05-26T02:50:57Z DEBUG raw: host_show(u'servera.system', version=u'2.230') 2021-05-26T02:50:57Z DEBUG host_show(u'servera.system', rights=False, all=False, raw=False, version=u'2.230', no_members=False) 2021-05-26T02:50:57Z DEBUG Backing up system configuration file '/var/lib/ipa/gssproxy/http.keytab' 2021-05-26T02:50:57Z DEBUG -> Not backing up - '/var/lib/ipa/gssproxy/http.keytab' doesn't exist 2021-05-26T02:50:57Z DEBUG Starting external process 2021-05-26T02:50:57Z DEBUG args=/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[email protected]<mailto:HTTP/[email protected]> -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL 2021-05-26T02:50:57Z DEBUG Process finished, return code=9 2021-05-26T02:50:57Z DEBUG stdout= 2021-05-26T02:50:57Z DEBUG stderr=Failed to parse result: unsupported extended operation Retrying with pre-4.0 keytab retrieval method... Failed to parse result: unsupported extended operation Failed to get keytab! Failed to get keytab 2021-05-26T02:50:57Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 570, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 560, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 637, in request_service_keytab super(HTTPInstance, self).request_service_keytab() File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 742, in request_service_keytab self.run_getkeytab(self.api.env.ldap_uri, self.keytab, self.principal) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 732, in run_getkeytab ipautil.run(args, nolog=nolog) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run raise CalledProcessError(p.returncode, arg_string, str(output)) CalledProcessError: Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[email protected]<mailto:HTTP/[email protected]> -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9 2021-05-26T02:50:57Z DEBUG [error] CalledProcessError: Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[email protected]<mailto:HTTP/[email protected]> -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9 2021-05-26T02:50:57Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run return cfgr.run() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 364, in run return self.execute() exc_handler(exc_info) <snip /> File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 622, in main replica_install(self) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 406, in decorated func(installer) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1487, in install fstore=fstore) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 173, in install_http subject_base=config.subject_base, master_fqdn=config.master_host_name) File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 188, in create_instance self.start_creation() File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 570, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 560, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 637, in request_service_keytab super(HTTPInstance, self).request_service_keytab() File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 742, in request_service_keytab self.run_getkeytab(self.api.env.ldap_uri, self.keytab, self.principal) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 732, in run_getkeytab ipautil.run(args, nolog=nolog) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run raise CalledProcessError(p.returncode, arg_string, str(output)) 2021-05-26T02:50:57Z DEBUG The ipa-replica-install command failed, exception: CalledProcessError: Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[email protected]<mailto:HTTP/[email protected]> -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9 2021-05-26T02:50:57Z ERROR Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[email protected]<mailto:HTTP/[email protected]> -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9 2021-05-26T02:50:57Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information [2] [26/May/2021:12:50:47.240285166 +1000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToservera.system" (servera:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [26/May/2021:12:50:47.858057379 +1000] - INFO - NSMMReplicationPlugin - repl5_tot_run - Beginning total update of replica "agmt="cn=meToservera.system" (servera:389)". [26/May/2021:12:50:50.679652092 +1000] - INFO - NSMMReplicationPlugin - repl5_tot_run - Finished total update of replica "agmt="cn=meToservera.system" (servera:389)". Sent 582 entries. [26/May/2021:12:50:52.158394667 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. [26/May/2021:12:50:55.079367688 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. [26/May/2021:12:50:58.084381230 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. [26/May/2021:12:51:01.092727541 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient and may contain material that is proprietary, confidential, privileged or otherwise legally protected or restricted under applicable government laws. Any review, disclosure, distributing or other use without expressed permission of the sender is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies without reading, printing, or saving. _______________________________________________ FreeIPA-users mailing list -- [email protected]<mailto:[email protected]> To unsubscribe send an email to [email protected]<mailto:[email protected]> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
