Re: [Freeipa-devel] Check if server is fully installed and ready
On 14.4.2016 12:23, Martin Babinsky wrote: > On 04/14/2016 11:16 AM, Christian Heimes wrote: >> Hi, >> >> while I was working on my Ansible playbook I ran into an issue. It is >> hard to detect if a FreeIPA server instance is fully installed and all >> its services are ready to handle requests. It's even harder to check it >> remotely. I have figured out some heuristics to detect that a sever is >> *not* fully installed (e.g. /etc/ipa/default.conf is missing or >> http://ipa-ca.ipa.example/ipa/crl/MasterCRL.bin returns 404). The >> presence of these resources is no guarantee that all FreeIPA services >> fully up and running. >> >> Two days ago on IRC Jan came up with the same problem with containers. >> He ran into a problem related to containers and DNS updates. Since I'm >> no longer alone with the problem and my own workarounds are not >> completely stable, I like to address the problem in FreeIPA directly. >> Now you might wonder why it is so hard to check if FreeIPA is ready or >> why nobody ran into the issue before. >> >> Let's start with the second question. A typical admin first installs a >> FreeIPA server on one machine. It takes a couple of seconds until he >> notices that the installer has finished. The admin ssh-es into another >> machine, sudo -s and then runs ipa-client-install with some arguments. >> It takes a couple of seconds, maybe even a minute. With containers and >> automation tools it's more like milliseconds. >> >> Now for the first question. Under some conditions a FreeIPA service >> might be started but not yet ready to serve requests or aren't fully >> operation yet. For example ticket >> https://fedorahosted.org/freeipa/ticket/5813 is an example of a problem >> with ipa-kra-install, 389-DS restarts and bind-dyndb-ldap. >> >> >> Proposal >> >> 1) A new boolean attribute ipaReady=TRUE/FALSE in >> cn=$FQDN,cn=masters,cn=ipa,cn=etc,$SUFFIX tracks whether or not an >> FreeIPA instance is ready to handle requests. >> > > Each service entry in the service container contains > 'ipaConfigString=enabledService' attribute that *should* tell that the service > is enabled and running. > > When the Server Roles[1] feature you could check whether the role that > interests you (in this case implicit "IPA Master" role) is enabled (which > means that all component services are marked as enabled, read the linked > design for more info). > > However the fact that something is marked as enabled/ready in LDAP backend > does not reflect the real status of systemd service on local system it just > tells you that someone (typically installer) marked this service as enabled. > > For that we would need a daemon that would synchronize the state of local > services with the state of LDAP backend and even then things may not be 100% > guaranteed. We can use systemd's feature 'notify' to inform systemd that service is actually running and accepting requests (compare this with 'process is running'). See man sd_notify I have a bind-dyndb-ldap ticket for this but it did not get sufficient priority up until now: https://fedorahosted.org/bind-dyndb-ldap/ticket/150 The very same thing can be done with all other services. When this is implemented for all services, it will be very easy to add systemd unit like 'ipa-ready' which can be used either for local check or to trigger a file generation somewhere so you can read the file over HTTP or so. Petr^2 Spacek > >> 2) A new HTTP route http[s]://$FQDN/ipa/ready is added. The route does >> not need authentication. When ipaReady=TRUE the route simple returns 200 >> OK with some text like READY. When the attribute is not present or >> FALSE, it returns an error to the client (412, 408?). >> > >> 3) All ipa install and upgrade commands set the attribute to FALSE >> before any tasks. >> > Installers already mark the corresponding service as enabled when finished > succesfully. >> 4) A final step in all ipa install and upgrade commands checks that all >> services have been started and are ready to handle requests. Eventually >> the ipaReady attribute is set to true. >> > This is basically what ipactl status does. >> Christian >> >> >> > http://www.freeipa.org/page/V4/Server_Roles > -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Check if server is fully installed and ready
On Thu, Apr 14, 2016 at 11:16:01AM +0200, Christian Heimes wrote: > > Two days ago on IRC Jan came up with the same problem with containers. > He ran into a problem related to containers and DNS updates. Since I'm My issue was mostly PEBKAC. In container environments that I use, the IP address of the IPA server itself can change with new start, so the fact that SSSD couldn't talk to IPA was actually correct -- at that time, my client it was trying to talk to the IPA server on IP address that was no longer valid. -- Jan Pazdziora Senior Principal Software Engineer, Identity Management Engineering, Red Hat -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Check if server is fully installed and ready
On 04/14/2016 11:16 AM, Christian Heimes wrote: Hi, while I was working on my Ansible playbook I ran into an issue. It is hard to detect if a FreeIPA server instance is fully installed and all its services are ready to handle requests. It's even harder to check it remotely. I have figured out some heuristics to detect that a sever is *not* fully installed (e.g. /etc/ipa/default.conf is missing or http://ipa-ca.ipa.example/ipa/crl/MasterCRL.bin returns 404). The presence of these resources is no guarantee that all FreeIPA services fully up and running. Two days ago on IRC Jan came up with the same problem with containers. He ran into a problem related to containers and DNS updates. Since I'm no longer alone with the problem and my own workarounds are not completely stable, I like to address the problem in FreeIPA directly. Now you might wonder why it is so hard to check if FreeIPA is ready or why nobody ran into the issue before. Let's start with the second question. A typical admin first installs a FreeIPA server on one machine. It takes a couple of seconds until he notices that the installer has finished. The admin ssh-es into another machine, sudo -s and then runs ipa-client-install with some arguments. It takes a couple of seconds, maybe even a minute. With containers and automation tools it's more like milliseconds. Now for the first question. Under some conditions a FreeIPA service might be started but not yet ready to serve requests or aren't fully operation yet. For example ticket https://fedorahosted.org/freeipa/ticket/5813 is an example of a problem with ipa-kra-install, 389-DS restarts and bind-dyndb-ldap. Proposal 1) A new boolean attribute ipaReady=TRUE/FALSE in cn=$FQDN,cn=masters,cn=ipa,cn=etc,$SUFFIX tracks whether or not an FreeIPA instance is ready to handle requests. Each service entry in the service container contains 'ipaConfigString=enabledService' attribute that *should* tell that the service is enabled and running. When the Server Roles[1] feature you could check whether the role that interests you (in this case implicit "IPA Master" role) is enabled (which means that all component services are marked as enabled, read the linked design for more info). However the fact that something is marked as enabled/ready in LDAP backend does not reflect the real status of systemd service on local system it just tells you that someone (typically installer) marked this service as enabled. For that we would need a daemon that would synchronize the state of local services with the state of LDAP backend and even then things may not be 100% guaranteed. 2) A new HTTP route http[s]://$FQDN/ipa/ready is added. The route does not need authentication. When ipaReady=TRUE the route simple returns 200 OK with some text like READY. When the attribute is not present or FALSE, it returns an error to the client (412, 408?). 3) All ipa install and upgrade commands set the attribute to FALSE before any tasks. Installers already mark the corresponding service as enabled when finished succesfully. 4) A final step in all ipa install and upgrade commands checks that all services have been started and are ready to handle requests. Eventually the ipaReady attribute is set to true. This is basically what ipactl status does. Christian http://www.freeipa.org/page/V4/Server_Roles -- Martin^3 Babinsky -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
[Freeipa-devel] Check if server is fully installed and ready
Hi, while I was working on my Ansible playbook I ran into an issue. It is hard to detect if a FreeIPA server instance is fully installed and all its services are ready to handle requests. It's even harder to check it remotely. I have figured out some heuristics to detect that a sever is *not* fully installed (e.g. /etc/ipa/default.conf is missing or http://ipa-ca.ipa.example/ipa/crl/MasterCRL.bin returns 404). The presence of these resources is no guarantee that all FreeIPA services fully up and running. Two days ago on IRC Jan came up with the same problem with containers. He ran into a problem related to containers and DNS updates. Since I'm no longer alone with the problem and my own workarounds are not completely stable, I like to address the problem in FreeIPA directly. Now you might wonder why it is so hard to check if FreeIPA is ready or why nobody ran into the issue before. Let's start with the second question. A typical admin first installs a FreeIPA server on one machine. It takes a couple of seconds until he notices that the installer has finished. The admin ssh-es into another machine, sudo -s and then runs ipa-client-install with some arguments. It takes a couple of seconds, maybe even a minute. With containers and automation tools it's more like milliseconds. Now for the first question. Under some conditions a FreeIPA service might be started but not yet ready to serve requests or aren't fully operation yet. For example ticket https://fedorahosted.org/freeipa/ticket/5813 is an example of a problem with ipa-kra-install, 389-DS restarts and bind-dyndb-ldap. Proposal 1) A new boolean attribute ipaReady=TRUE/FALSE in cn=$FQDN,cn=masters,cn=ipa,cn=etc,$SUFFIX tracks whether or not an FreeIPA instance is ready to handle requests. 2) A new HTTP route http[s]://$FQDN/ipa/ready is added. The route does not need authentication. When ipaReady=TRUE the route simple returns 200 OK with some text like READY. When the attribute is not present or FALSE, it returns an error to the client (412, 408?). 3) All ipa install and upgrade commands set the attribute to FALSE before any tasks. 4) A final step in all ipa install and upgrade commands checks that all services have been started and are ready to handle requests. Eventually the ipaReady attribute is set to true. Christian signature.asc Description: OpenPGP digital signature -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
