Hi Rob, Thanks for the tip, it came in handy with what i was working on (the python code).
Here is the error_log on the new server (ipa011) [Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid 55597] AH00492: caught SIGWINCH, shutting down gracefully [Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor [Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming normal operations [Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cr...@ad.companyx.fm: ping(): SUCCESS [Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cr...@ad.companyx.fm: ping/1(version='2.251'): SUCCESS [Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cr...@ad.companyx.fm: server_conncheck('ipa008.ad.companyx.fm', ' ipa011.ad.companyx.fm', version='2.162'): ValidationError I tweaked the code a little to do some old skool debugging... here: https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L927 This enabled me to dump the vars in question to why the error was raised: ERROR: Remote master check failed with following error message(s): invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:(' ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]: ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm So: keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm As the code states: if keys[-2] != api.env.host: raise errors.ValidationError( name='cn', error=_("must be \"%s\"") % api.env.host) and we can see that in fact ipa008.ad.companyx.fm != ipa011.ad.companyx.fm So, that's about as far as i have gotten so far. Do we think the keys swapped around for some reason? thanks, Nick On Thu, 25 May 2023 at 13:52, Rob Crittenden <rcrit...@redhat.com> wrote: > Florence Blanc-Renaud via FreeIPA-users wrote: > > Hi, > > > > On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cr...@dice.fm > > <mailto:nicholas.cr...@dice.fm>> wrote: > > > > Hi, thanks for the reply. > > > > This was the process. > > 1. prep > > 2. test connectivity - works ok > > 3. install replica (successfully) - no DNS no CA at this point > > 4. test connectivity - fails. > > > > If i replace step 4 for the following: > > > > 4. install DNS (successfully) > > 5. install CA (fails - with the "invalid 'cn': must be" error) > > > > If we can ignore the step 4 where the comms check fails and > > concentrate on step 5 where the CA fails. I am happy to do so. > > But in step 5. it is the comms check that the `ipa-ca-install` > > command runs that also fails as far as i can see (no expert here > > myself, though my company thinks so LOL) > > > > Successful DNS install logs http://sprunge.us/7WCtpW > > > > Failed CA install logs http://sprunge.us/hTVLzL > > > > I believe this error needs to be investigated: > > > > 2023-05-25T07:52:07Z INFO Connection to > https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: > SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or > the credentials were unavailable or inaccessible (Credential cache is empty) > > > > We can see that the process is able to connect to the master and get a > > cookie but later on GSSAPI auth fails. > > > > Are there any corresponding logs in the master in > > /var/log/httpd/error_log? You can enable debug logs on the master with > > the following procedure: > > - create /etc/ipa/server.conf: > > [global] > > debug=True > > > > then restart apache with systemctl restart httpd. > > You can also run ipa-healthcheck on the master/the replica to find any > > config issue. > > It would also be interesting to see the entries around the > 'server_conncheck' call in /var/log/httpd/error_log to see what > arguments were passed in (on ipa011) since you're enabling debug mode > anyway. > > rob > > > > > flo > > > > [root@ipa011 ~]# ipa-ca-install > > Directory Manager (existing master) password: > > > > Run connection check to master > > > > Your system may be partly configured. > > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > > > Connection check failed! > > See /var/log/ipareplica-conncheck.log for more information. > > If the check results are not valid it can be skipped with > > --skip-conncheck parameter. > > > > > > > > thanks, > > Nick > > > > On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <f...@redhat.com > > <mailto:f...@redhat.com>> wrote: > > > > Hi, > > > > On Wed, May 24, 2023 at 3:29 PM Nicholas Cross > > <nicholas.cr...@dice.fm <mailto:nicholas.cr...@dice.fm>> wrote: > > > > Hi Flo (and other helpful people on this list), > > > > After fixing the SID/PAC issue, i am back to looking as to > > why the ipa-replica-conncheck fails when installing the CA > > to a (working) replica. > > > > I ran your suggested commands and they look fine now. > > > > [root@ipa011 ~]# kdestroy > > kdestroy: No credentials cache found while destroying cache > > > > [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab > > host/ipa011.ad.companyx...@ad.companyx.fm > > <mailto:ipa011.ad.companyx...@ad.companyx.fm> > > Using default cache: 0 > > Using principal: host/ipa011.ad.companyx...@ad.companyx.fm > > <mailto:ipa011.ad.companyx...@ad.companyx.fm> > > Using keytab: /etc/krb5.keytab > > Authenticated to Kerberos v5 > > > > [root@ipa011 ~]# kvno > > HTTP/ipa011.ad.companyx...@ad.companyx.fm > > <mailto:ipa011.ad.companyx...@ad.companyx.fm> > > HTTP/ipa011.ad.companyx...@ad.companyx.fm > > <mailto:ipa011.ad.companyx...@ad.companyx.fm>: kvno = 1 > > > > I can run the following BEFORE i install the replica and it > > works 100%, it is part of our pre-reqs in our docs to make > > sure comms work. > > > > ipa-replica-conncheck --master=ipa008.ad.companyx.fm > > <http://ipa008.ad.companyx.fm> --auto-master-check > > --realm=AD.companyx.FM <http://AD.companyx.FM> > > --principal=nicholas.cross > > > > After i install the replica with ipa-replica-install. (no > > DNS, no CA) then running the above ipa-replica-conncheck > > command fails and we see the errors as described. > > > > > > ipa-replica-conncheck is intended to be run *before* replica > > installation, as stated in the man page: > > ipa-replica-conncheck - Check a replica-master network > > connection before installation > > > > The reason is that it opens server sockets listening on the > > tested ports, for the master to try to reach the replica. If the > > replica is already installed, the ports are already in use by > > the ldap server, the apache web server etc... > > > > I understand that you tried to run ipa-replica-conncheck to > > ensure that ipa-dns-install or ipa-ca-install won't hit any > > issue. Did you actually try to install the DNS service and the > > CA service on the replica? If you see errors in those steps we > > can help troubleshoot, based on the produced logs, but > > ipa-replica-conncheck won't be of any use here. > > > > flo > > > > > > Some logs: > > 1. before ipa-replica-install - conn check works. > > http://sprunge.us/4yrInD > > 2. after successful ipa-replica-install - but conn check > fails > > http://sprunge.us/gvqEle > > > > new replica versions > > [root@ipa011 ~]# rpm -qa | grep ipa | sort > > almalinux-logos-ipa-90.5.1-1.1.el9.noarch > > ipa-client-4.10.1-6.el9.x86_64 > > ipa-client-common-4.10.1-6.el9.noarch > > ipa-common-4.10.1-6.el9.noarch > > ipa-healthcheck-0.12-1.el9.noarch > > ipa-healthcheck-core-0.12-1.el9.noarch > > ipa-selinux-4.9.8-7.el9_0.noarch > > ipa-server-4.10.1-6.el9.x86_64 > > ipa-server-common-4.10.1-6.el9.noarch > > ipa-server-dns-4.10.1-6.el9.noarch > > libipa_hbac-2.8.2-2.el9.x86_64 > > python3-ipaclient-4.10.1-6.el9.noarch > > python3-ipalib-4.10.1-6.el9.noarch > > python3-ipaserver-4.10.1-6.el9.noarch > > python3-libipa_hbac-2.8.2-2.el9.x86_64 > > sssd-ipa-2.8.2-2.el9.x86_64 > > > > current master versions > > [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort > > almalinux-logos-ipa-90.5.1-1.1.el9.noarch > > ipa-client-4.10.0-8.el9_1.x86_64 > > ipa-client-common-4.10.0-8.el9_1.noarch > > ipa-common-4.10.0-8.el9_1.noarch > > ipa-healthcheck-0.9-9.el9.noarch > > ipa-healthcheck-core-0.9-9.el9.noarch > > ipa-selinux-4.9.8-7.el9_0.noarch > > ipa-server-4.10.0-8.el9_1.x86_64 > > ipa-server-common-4.10.0-8.el9_1.noarch > > ipa-server-dns-4.10.0-8.el9_1.noarch > > libipa_hbac-2.7.3-4.el9_1.3.x86_64 > > python3-ipaclient-4.10.0-8.el9_1.noarch > > python3-ipalib-4.10.0-8.el9_1.noarch > > python3-ipaserver-4.10.0-8.el9_1.noarch > > python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 > > sssd-ipa-2.7.3-4.el9_1.3.x86_64 > > > > thanks, > > Nick > > > > On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud > > <f...@redhat.com <mailto:f...@redhat.com>> wrote: > > > > Hi, > > > > the replica-conncheck error means that a call to > > server_conncheck reached the wrong server. > > ipa-replica-conncheck performs multiple checks: > > - first from the replica to the existing master (here we > > seem to be good) > > - then from the existing master to the replica, by doing > > a call to the XMLRPC api server_conncheck on the master. > > If the connection from replica to master fails, another > > server is tried (in this case, the replica launches > > server_conncheck on itself), but there is a security > > that ensures the right server is handling the call. > > The logs shows that the connection fails because of SASL > > auth failure: > > 2023-05-22T18:14:03Z INFO Connection to > > https://ipa010.ad.companyx.fm/ipa/json failed with > > Insufficient access: SASL(-1): generic failure: GSSAPI > > Error: No credentials were supplied, or the credentials > > were unavailable or inaccessible (Credential cache is > empty) > > > > Are you able to do kinit -kt /etc/krb5.keytab > > host/<replicafqdn>@<REALM> on the replica? And then kvno > > HTTP/<serverfqdn>@<REALM> ? > > flo > > > > On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via > > FreeIPA-users <freeipa-users@lists.fedorahosted.org > > <mailto:freeipa-users@lists.fedorahosted.org>> wrote: > > > > That was the /var/log/ipareplica-conncheck.log log > file > > > > it does looks like a DNs issue, but im not sure > where. > > > > dns resolves the host fine on the host > > > > [root@ipa011 ~]# host ipa011 > > ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> > > has address 10.32.225.7 > > > > [root@ipa011 ~]# grep ipa /etc/ipa/default.conf > > host = ipa011.ad.companyx.fm > > <http://ipa011.ad.companyx.fm> > > xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml > > ca_host = ipa010.ad.companyx.fm > > <http://ipa010.ad.companyx.fm> > > > > it's odd as i run the connection check before the > > start of the install, to check ports and routes. > > it works fine. > > replica install works. > > dns install works. > > just the ca installer comes back with this error. > > > > As an additional test i added the dns record for > > this host into IPA before the install. Normally we > > don't need to, but just as a test, but it made no > > difference. > > > > > > We do have new DNS forwarders on the network - these > > are in front of the IPA servers. They are there > > just take the load from the k8s clusters away from > > IPA DNS. > > Would the CA install break if the DNS lookups are > > "proxied" by the DNS forwarders? > > All DNS tests i can think of work via the > > forwarders. The IPA clients (100s) are all fine > > with them. > > > > I will update the client to ignore the forwarders, > > but if you can think of anything else to try? > > > > thanks, Nick > > _______________________________________________ > > FreeIPA-users mailing list -- > > freeipa-users@lists.fedorahosted.org > > <mailto:freeipa-users@lists.fedorahosted.org> > > To unsubscribe send an email to > > freeipa-users-le...@lists.fedorahosted.org > > <mailto:freeipa-users-le...@lists.fedorahosted.org> > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > Do not reply to spam, report it: > > https://pagure.io/fedora-infrastructure/new_issue > > > > > > _______________________________________________ > > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > > To unsubscribe send an email to > freeipa-users-le...@lists.fedorahosted.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > > > >
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue