[Group.of.nepali.translators] [Bug 1761096] Re: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed
This bug was fixed in the package squid3 - 3.5.12-1ubuntu7.14 --- squid3 (3.5.12-1ubuntu7.14) xenial; urgency=medium * d/squid.resolvconf: Invoke "systemctl reload --no-block" if we are using systemd. This prevents squid from blocking if the reload action is being issued indirectly because of another service (e.g., because dnsmasq has been restarted), which may cause a deadlock and prevent the whole transaction to complete. (LP: #1761096) -- Sergio Durigan Junior Fri, 04 Sep 2020 08:31:36 -0400 ** Changed in: squid3 (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1761096 Title: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed Status in dnsmasq package in Ubuntu: Fix Released Status in squid package in Ubuntu: Fix Released Status in squid3 package in Ubuntu: Invalid Status in dnsmasq source package in Xenial: Invalid Status in squid source package in Xenial: Invalid Status in squid3 source package in Xenial: Fix Released Status in dnsmasq source package in Bionic: Invalid Status in squid source package in Bionic: Invalid Status in squid3 source package in Bionic: Incomplete Bug description: [Impact] When using dnsmasq along with squid on Ubuntu Xenial, the user will experience a deadlock while performing on every second execution of the "systemctl start dnsmasq.service" command. The deadlock will be caused by an attempt to invoke, by nss-lookup.target, a "systemctl reload squid.service", which will itself try to start the nss- lookup.target, which will be waiting on squid, therefore eventually leading to a timeout. The underlying cause of this deadlock is related to how systemd used to handle dependencies between multiple jobs being started in the same transaction. [Test Case] One can reproduce the issue on a Xenial container: $ lxc launch ubuntu-daily:xenial squid-bug1761096 $ lxc shell squid-bug1761096 # apt update # apt install squid dnsmasq -y It is quite possible that during "apt install" the bug will manifest, and dnsmasq will fail to start due to a timeout. The user might see a message like: Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. If the bug doesn't manifest itself during installation, the following commands in sequence should trigger it: # systemctl restart dnsmasq.service # systemctl restart dnsmasq.service [Regression Potential] This change only touches the mechanism by which squid has its configuration reloaded in case of a DNS resolver change. Because "systemctl reload --no-block" returns practically immediately, if squid's configuration file is invalid the user won't see any notifications. However, this behaviour is already present currently, because "systemctl reload squid" invokes "/etc/init.d/squid reload"; the user has to check "journalctl -u squid.service" if she wants to verify whether there were any failures during the reload. Other than that, and because systemctl will offload the service to the SysV script as usual (in the case of squid), I don't foresee any potential regressions. [Original Description] Setup to reproduce: Ubuntu Xenial amd64 net install iso from http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer- amd64/current/images/netboot/mini.iso Install system with mostly defaults + LVM + OpenSSH server Note that this bug applies to both DHCP and static IP+DNS network configurations Once server rebooted and is available, log in and install dnsmasq + squid: apt-get update && apt-get install squid dnsmasq output of this can be found at https://pastebin.com/9Atuipju journalctl -xe output at https://pastebin.com/uLhfM4jN Furthermore at this point I can run alternating errors root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:18:07 CEST 2018 Wed Apr 4 09:18:07 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq stop ; date Wed Apr 4 09:18:39 CEST 2018 Wed Apr 4 09:18:39 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:19:10 CEST 2018 Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. Wed Apr 4 09:20:40 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:42:57 CEST 2018 Wed Apr 4 09:42:57 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq stop ; date Wed Apr 4 09:43:14 CEST 2018 Wed Apr 4 09:43:14 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:43:26 CEST 2018 Job for dnsmasq.service failed because a timeout was exceeded. See
[Group.of.nepali.translators] [Bug 1761096] Re: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed
On Xenial the source package is squid3, not squid, so I'll add/adjust tasks for that so the tracking works correctly. On Bionic the code suggests that the same problem might exist, but it appears that nobody has reproduced it there so there is no need for an SRU to Bionic. However, if somebody is affected then we can look again. I'll leave an Incomplete task for Bionic to try to highlight that. We don't have any specific policy or pattern to track this kind of thing so I'm being inventive with the bug status here. Excerpts from #ubuntu-devel: 17:32 cpaelzer, sergiodj: in bug 1761096, did you ever find out why Bionic are unaffected, or did you decide it wasn't worth investigating further? 17:32 *is* 17:38 rbasak: we could not find exactly why Bionic is not affected. my hypothesis is that between Xenial and Bionic there was a fix on systemd which addressed this issue, but I could not pinpoint exactly what was done 17:39 arguably, the best scenario would be to backport this fix to Xenial's systemd, but since we had already spent a non-trivial amount of time investigating this (without finding anything concrete on systemd's side), I opted for the second best thing to do, which is to use --no-block to workaround this bug ** Also affects: squid3 (Ubuntu) Importance: Undecided Status: New ** Also affects: squid (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: dnsmasq (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: squid3 (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: squid3 (Ubuntu) Status: New => Invalid ** Changed in: squid3 (Ubuntu Bionic) Status: New => Incomplete ** Changed in: squid3 (Ubuntu Xenial) Status: New => In Progress ** Changed in: squid3 (Ubuntu Xenial) Assignee: (unassigned) => Sergio Durigan Junior (sergiodj) ** Changed in: squid (Ubuntu Xenial) Assignee: Sergio Durigan Junior (sergiodj) => (unassigned) ** Changed in: squid (Ubuntu Xenial) Importance: High => Undecided ** Changed in: squid (Ubuntu Xenial) Status: Confirmed => Invalid ** Changed in: squid (Ubuntu Bionic) Status: New => Invalid ** Changed in: squid3 (Ubuntu Xenial) Importance: Undecided => High ** Changed in: dnsmasq (Ubuntu Bionic) Status: New => Invalid -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1761096 Title: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed Status in dnsmasq package in Ubuntu: Fix Released Status in squid package in Ubuntu: Fix Released Status in squid3 package in Ubuntu: Invalid Status in dnsmasq source package in Xenial: Invalid Status in squid source package in Xenial: Invalid Status in squid3 source package in Xenial: Fix Committed Status in dnsmasq source package in Bionic: Invalid Status in squid source package in Bionic: Invalid Status in squid3 source package in Bionic: Incomplete Bug description: [Impact] When using dnsmasq along with squid on Ubuntu Xenial, the user will experience a deadlock while performing on every second execution of the "systemctl start dnsmasq.service" command. The deadlock will be caused by an attempt to invoke, by nss-lookup.target, a "systemctl reload squid.service", which will itself try to start the nss- lookup.target, which will be waiting on squid, therefore eventually leading to a timeout. The underlying cause of this deadlock is related to how systemd used to handle dependencies between multiple jobs being started in the same transaction. [Test Case] One can reproduce the issue on a Xenial container: $ lxc launch ubuntu-daily:xenial squid-bug1761096 $ lxc shell squid-bug1761096 # apt update # apt install squid dnsmasq -y It is quite possible that during "apt install" the bug will manifest, and dnsmasq will fail to start due to a timeout. The user might see a message like: Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. If the bug doesn't manifest itself during installation, the following commands in sequence should trigger it: # systemctl restart dnsmasq.service # systemctl restart dnsmasq.service [Regression Potential] This change only touches the mechanism by which squid has its configuration reloaded in case of a DNS resolver change. Because "systemctl reload --no-block" returns practically immediately, if squid's configuration file is invalid the user won't see any notifications. However, this behaviour is already present currently, because "systemctl reload squid" invokes "/etc/init.d/squid reload"; the user has to check "journalctl -u squid.service" if she wants to verify whether there were any failures during the rel
[Group.of.nepali.translators] [Bug 1761096] Re: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed
I'm marking the dnsmasq bug as Invalid since this is a problem with squid/systemd. ** Changed in: squid (Ubuntu Xenial) Assignee: (unassigned) => Sergio Durigan Junior (sergiodj) ** Changed in: dnsmasq (Ubuntu Xenial) Status: Confirmed => Invalid -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1761096 Title: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed Status in dnsmasq package in Ubuntu: Fix Released Status in squid package in Ubuntu: Fix Released Status in dnsmasq source package in Xenial: Invalid Status in squid source package in Xenial: Confirmed Bug description: [Impact] When using dnsmasq along with squid on Ubuntu Xenial, the user will experience a deadlock while performing on every second execution of the "systemctl start dnsmasq.service" command. The deadlock will be caused by an attempt to invoke, by nss-lookup.target, a "systemctl reload squid.service", which will itself try to start the nss- lookup.target, which will be waiting on squid, therefore eventually leading to a timeout. The underlying cause of this deadlock is related to how systemd used to handle dependencies between multiple jobs being started in the same transaction. [Test Case] One can reproduce the issue on a Xenial container: $ lxc launch ubuntu-daily:xenial squid-bug1761096 $ lxc shell squid-bug1761096 # apt update # apt install squid dnsmasq -y It is quite possible that during "apt install" the bug will manifest, and dnsmasq will fail to start due to a timeout. The user might see a message like: Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. If the bug doesn't manifest itself during installation, the following commands in sequence should trigger it: # systemctl restart dnsmasq.service # systemctl restart dnsmasq.service [Regression Potential] This change only touches the mechanism by which squid has its configuration reloaded in case of a DNS resolver change. Because "systemctl reload --no-block" returns practically immediately, if squid's configuration file is invalid the user won't see any notifications. However, this behaviour is already present currently, because "systemctl reload squid" invokes "/etc/init.d/squid reload"; the user has to check "journalctl -u squid.service" if she wants to verify whether there were any failures during the reload. Other than that, and because systemctl will offload the service to the SysV script as usual (in the case of squid), I don't foresee any potential regressions. [Original Description] Setup to reproduce: Ubuntu Xenial amd64 net install iso from http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer- amd64/current/images/netboot/mini.iso Install system with mostly defaults + LVM + OpenSSH server Note that this bug applies to both DHCP and static IP+DNS network configurations Once server rebooted and is available, log in and install dnsmasq + squid: apt-get update && apt-get install squid dnsmasq output of this can be found at https://pastebin.com/9Atuipju journalctl -xe output at https://pastebin.com/uLhfM4jN Furthermore at this point I can run alternating errors root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:18:07 CEST 2018 Wed Apr 4 09:18:07 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq stop ; date Wed Apr 4 09:18:39 CEST 2018 Wed Apr 4 09:18:39 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:19:10 CEST 2018 Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. Wed Apr 4 09:20:40 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:42:57 CEST 2018 Wed Apr 4 09:42:57 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq stop ; date Wed Apr 4 09:43:14 CEST 2018 Wed Apr 4 09:43:14 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:43:26 CEST 2018 Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. Wed Apr 4 09:44:56 CEST 2018 and so on... Each and every 1 out of 2 stop/start cycle fails in 1m30s timeout Complete journalctl -xe output attached To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1761096/+subscriptions ___ Mailing list: https://launchpad.net/~group.of.nepali.translators Post to : group.of.nepali.translators@lists.launchpad.net Unsubscribe : https://launchpad.net/~group.of.nepali.translators More help : https://help.launch
[Group.of.nepali.translators] [Bug 1761096] Re: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed
I confirm that every second $ systemctl restart dnsmasq hangs Also I confirm that removing squid resolved the situation. I found that it does this: 1. /bin/sh /etc/init.d/dnsmasq systemd-stop-resolvconf 2. /bin/sh /etc/init.d/dnsmasq systemd-start-resolvconf Both will call hooks like: 0 0 11642 11634 20 0 4364 660 wait S? 0:00 \_ run-parts --arg=-a --arg=lo.dnsmasq /etc/resolvconf/update.d 0 0 11659 11642 20 0 4364 664 wait S? 0:00 \_ run-parts /etc/resolvconf/update-libc.d 0 0 11672 11659 20 0 4504 704 wait S? 0:00 \_ /bin/sh /etc/resolvconf/update-libc.d/squid 0 0 11673 11672 20 0 4504 1648 wait S? 0:00 \_ /bin/sh /usr/sbin/invoke-rc.d squid reload 4 0 11691 11673 20 0 24888 1388 poll_s S? 0:00 \_ systemctl reload squid.service This is what actually times out - the squid hook. The "systemctl reload squid.service" call seems to take too long every second time?! A single "systemctl reload squid.service" for these seems to take ~100 seconds for me. OTOH a direct call to it is fast every time. This needs to be debugged in detail, but seems well reproducible. ** Also affects: dnsmasq (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: dnsmasq (Ubuntu Xenial) Status: New => Confirmed ** Changed in: dnsmasq (Ubuntu) Status: New => Fix Released ** Changed in: dnsmasq (Ubuntu Xenial) Importance: Undecided => High ** Tags added: server-next ** Also affects: squid (Ubuntu) Importance: Undecided Status: New ** Changed in: squid (Ubuntu Xenial) Status: New => Confirmed ** Changed in: squid (Ubuntu Xenial) Importance: Undecided => High ** Changed in: squid (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1761096 Title: dnsmasq starts with error on Ubuntu Xenial amd64 when squid installed Status in dnsmasq package in Ubuntu: Fix Released Status in squid package in Ubuntu: Fix Released Status in dnsmasq source package in Xenial: Confirmed Status in squid source package in Xenial: Confirmed Bug description: Setup to reproduce: Ubuntu Xenial amd64 net install iso from http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer- amd64/current/images/netboot/mini.iso Install system with mostly defaults + LVM + OpenSSH server Note that this bug applies to both DHCP and static IP+DNS network configurations Once server rebooted and is available, log in and install dnsmasq + squid: apt-get update && apt-get install squid dnsmasq output of this can be found at https://pastebin.com/9Atuipju journalctl -xe output at https://pastebin.com/uLhfM4jN Furthermore at this point I can run alternating errors root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:18:07 CEST 2018 Wed Apr 4 09:18:07 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq stop ; date Wed Apr 4 09:18:39 CEST 2018 Wed Apr 4 09:18:39 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:19:10 CEST 2018 Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. Wed Apr 4 09:20:40 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:42:57 CEST 2018 Wed Apr 4 09:42:57 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq stop ; date Wed Apr 4 09:43:14 CEST 2018 Wed Apr 4 09:43:14 CEST 2018 root@ubuntu-min:~# date ; service dnsmasq start ; date Wed Apr 4 09:43:26 CEST 2018 Job for dnsmasq.service failed because a timeout was exceeded. See "systemctl status dnsmasq.service" and "journalctl -xe" for details. Wed Apr 4 09:44:56 CEST 2018 and so on... Each and every 1 out of 2 stop/start cycle fails in 1m30s timeout Complete journalctl -xe output attached To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1761096/+subscriptions ___ Mailing list: https://launchpad.net/~group.of.nepali.translators Post to : group.of.nepali.translators@lists.launchpad.net Unsubscribe : https://launchpad.net/~group.of.nepali.translators More help : https://help.launchpad.net/ListHelp
