Re: [openstack-dev] Do all OpenStack daemons support sd_notify?
On 12/16/2014 11:59 AM, Alan Pevec wrote: > There's one issue with Type=notify that Dan Prince discovered and > where Type=simple works better for his use-case: > if service startup takes more than DefaultTimeoutStartSec (90s by > default) and systemd does not get notification from the service, it > will consider it failed and try to restart it if Restart is defined in > the service unit file. > I tried to fix that by disabling timeout (example in Nova package > https://review.gerrithub.io/13054 ) but then systemctl start blocks > until notification is received. > Copying Dan's review comment: "This still isn't quite right. It is > better in that the systemctl doesn't think the service fails... rather > it just seems to hang indefinately on 'systemctl start > openstack-nova-compute', as in I never get my terminal back. > My test case is: > 1) Stop Nova compute. 2) Stop RabbitMQ. 3) Try to start Nova compute > via systemctl. > Could the RabbitMQ retry loop be preventing the notification? > " > > Current implementation in oslo service sends notification only just > before entering the wait loop, because at that point all > initialization should be done and service ready to serve. Does anyone > have a suggestion for the better place where to notify service > readiness? > Or should just Dan's test-case step 3) be modified as: > 3) Start Nova compute via systemctl start ... & (i.e. in the background) ? > > > Cheers, > Alan What you describe above looks like a defect in the implementation. Of course, waiting for more than 90s should be considered as failed, and I wouldn't want in any case to have this timeout increased. Failed attempts to connect to Rabbit shouldn't, IMO, be the cause for not sending the notify signal. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Do all OpenStack daemons support sd_notify?
Thanks for responding, does this make it longest by time duration thread ever? :) 2015-10-15 22:44 GMT+02:00 Thomas Goirand: > On 12/16/2014 11:59 AM, Alan Pevec wrote: ... >> Current implementation in oslo service sends notification only just >> before entering the wait loop, because at that point all >> initialization should be done and service ready to serve. Does anyone >> have a suggestion for the better place where to notify service >> readiness? > What you describe above looks like a defect in the implementation. Of > course, waiting for more than 90s should be considered as failed, and I > wouldn't want in any case to have this timeout increased. Failed > attempts to connect to Rabbit shouldn't, IMO, be the cause for not > sending the notify signal. But if service requires message bus for normal operations, it is not ready to server requests, is it? Cheers, Alan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Do all OpenStack daemons support sd_notify?
On 12/15/2014 04:21 PM, Ihar Hrachyshka wrote: > On 14/12/14 09:45, Thomas Goirand wrote: >> Hi, > >> As I am slowing fixing all systemd issues for the daemons of >> OpenStack in Debian (and hopefully, have this ready before the >> freeze of Jessie), I was wondering what kind of Type= directive to >> put on the systemd .service files. I have noticed that in Fedora, >> there's Type=notify. So my question is: > >> Do all OpenStack daemons, as a rule, support the DBus sd_notify >> thing? Should I always use Type=notify for systemd .service files? >> Can this be called a general rule with no exception? > > (I will talk about neutron only.) > > I guess Type=notify is supposed to be used with daemons that use > Service class from oslo-incubator that provides systemd notification > mechanism, or call to systemd.notify_once() otherwise. > > In terms of Neutron, neutron-server process is doing it, metadata > agent also seems to do it, while OVS agent seems to not. So it really > should depend on each service and the way it's implemented. You cannot > just assume that every Neutron service reports back to systemd. > > In terms of Fedora, we have Type=notify for neutron-server service only. > > BTW now that more distributions are interested in shipping unit files > for services, should we upstream them and ship the same thing in all > interested distributions? In Debian & Ubuntu, we use a system which, from a sysv-rc init script, generates startup files for systemd, sysv-rc, and upstart. So if you where shipping .service files, I'm not sure I'd use them (though they may serve as good documentation...). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Do all OpenStack daemons support sd_notify?
2014-12-15 17:00 GMT+01:00 Clint Byrum cl...@fewbar.com: Excerpts from Ihar Hrachyshka's message of 2014-12-15 07:21:04 -0800: I guess Type=notify is supposed to be used with daemons that use Service class from oslo-incubator that provides systemd notification mechanism, or call to systemd.notify_once() otherwise. ... BTW now that more distributions are interested in shipping unit files for services, should we upstream them and ship the same thing in all interested distributions? Since we can expect the five currently implemented OS's in TripleO to all have systemd by default soon (Debian, Fedora, openSUSE, RHEL, Ubuntu), it would make a lot of sense for us to make the systemd unit files that TripleO generates set Type=notify wherever possible. So hopefully we can actually make such a guarantee upstream sometime in the not-so-distant future, especially since our CI will run two of the more distinct forks, Ubuntu and Fedora. There's one issue with Type=notify that Dan Prince discovered and where Type=simple works better for his use-case: if service startup takes more than DefaultTimeoutStartSec (90s by default) and systemd does not get notification from the service, it will consider it failed and try to restart it if Restart is defined in the service unit file. I tried to fix that by disabling timeout (example in Nova package https://review.gerrithub.io/13054 ) but then systemctl start blocks until notification is received. Copying Dan's review comment: This still isn't quite right. It is better in that the systemctl doesn't think the service fails... rather it just seems to hang indefinately on 'systemctl start openstack-nova-compute', as in I never get my terminal back. My test case is: 1) Stop Nova compute. 2) Stop RabbitMQ. 3) Try to start Nova compute via systemctl. Could the RabbitMQ retry loop be preventing the notification? Current implementation in oslo service sends notification only just before entering the wait loop, because at that point all initialization should be done and service ready to serve. Does anyone have a suggestion for the better place where to notify service readiness? Or should just Dan's test-case step 3) be modified as: 3) Start Nova compute via systemctl start ... (i.e. in the background) ? Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Do all OpenStack daemons support sd_notify?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 14/12/14 09:45, Thomas Goirand wrote: Hi, As I am slowing fixing all systemd issues for the daemons of OpenStack in Debian (and hopefully, have this ready before the freeze of Jessie), I was wondering what kind of Type= directive to put on the systemd .service files. I have noticed that in Fedora, there's Type=notify. So my question is: Do all OpenStack daemons, as a rule, support the DBus sd_notify thing? Should I always use Type=notify for systemd .service files? Can this be called a general rule with no exception? (I will talk about neutron only.) I guess Type=notify is supposed to be used with daemons that use Service class from oslo-incubator that provides systemd notification mechanism, or call to systemd.notify_once() otherwise. In terms of Neutron, neutron-server process is doing it, metadata agent also seems to do it, while OVS agent seems to not. So it really should depend on each service and the way it's implemented. You cannot just assume that every Neutron service reports back to systemd. In terms of Fedora, we have Type=notify for neutron-server service only. BTW now that more distributions are interested in shipping unit files for services, should we upstream them and ship the same thing in all interested distributions? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJUjvxgAAoJEC5aWaUY1u57N1gH/RsYPqmGpyoZ8fe8CwXcnz+R Rvfo7FHpcEZ9+Idvr9qitoPhKtGjzwgJC27EIQ6NCvgZZT462f+/jYHlxW0dX5Cz Fm9Zg/Hv50ukDOC1nT3nfDKH8uMwuPMrQsfRuXTGKhwqsfgnFfExozydgVeC2XFw WB9B3tBblp+7PRzaGyN9Bpe3gQnHUm3lyXaziK+wLbf7NTROzATlVCZ4xpPWu/5C ArfzwXICp9Dk5Juy75mwYwh37gw26w0VWfvPzn2WjkSVHKymNVn9GRdflVOrV3fq wnhu08e/wup8XF1/eKkWUJyF+hEsN5E0kO2x5CvavvMS3HSTm3Viuhz5tKC6ZAs= =WiBi -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Do all OpenStack daemons support sd_notify?
Excerpts from Ihar Hrachyshka's message of 2014-12-15 07:21:04 -0800: Hash: SHA512 On 14/12/14 09:45, Thomas Goirand wrote: Hi, As I am slowing fixing all systemd issues for the daemons of OpenStack in Debian (and hopefully, have this ready before the freeze of Jessie), I was wondering what kind of Type= directive to put on the systemd .service files. I have noticed that in Fedora, there's Type=notify. So my question is: Do all OpenStack daemons, as a rule, support the DBus sd_notify thing? Should I always use Type=notify for systemd .service files? Can this be called a general rule with no exception? (I will talk about neutron only.) I guess Type=notify is supposed to be used with daemons that use Service class from oslo-incubator that provides systemd notification mechanism, or call to systemd.notify_once() otherwise. In terms of Neutron, neutron-server process is doing it, metadata agent also seems to do it, while OVS agent seems to not. So it really should depend on each service and the way it's implemented. You cannot just assume that every Neutron service reports back to systemd. In terms of Fedora, we have Type=notify for neutron-server service only. BTW now that more distributions are interested in shipping unit files for services, should we upstream them and ship the same thing in all interested distributions? Since we can expect the five currently implemented OS's in TripleO to all have systemd by default soon (Debian, Fedora, openSUSE, RHEL, Ubuntu), it would make a lot of sense for us to make the systemd unit files that TripleO generates set Type=notify wherever possible. So hopefully we can actually make such a guarantee upstream sometime in the not-so-distant future, especially since our CI will run two of the more distinct forks, Ubuntu and Fedora. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Do all OpenStack daemons support sd_notify?
Hi, As I am slowing fixing all systemd issues for the daemons of OpenStack in Debian (and hopefully, have this ready before the freeze of Jessie), I was wondering what kind of Type= directive to put on the systemd .service files. I have noticed that in Fedora, there's Type=notify. So my question is: Do all OpenStack daemons, as a rule, support the DBus sd_notify thing? Should I always use Type=notify for systemd .service files? Can this be called a general rule with no exception? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev