[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Tracked in Github Issues as https://github.com/canonical/cloud- init/issues/2737 ** Bug watch added: github.com/canonical/cloud-init/issues #2737 https://github.com/canonical/cloud-init/issues/2737 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Released Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Given https://bugs.freedesktop.org/show_bug.cgi?id=98254 and https://github.com/systemd/systemd/pull/7609 and https://bugs.launchpad.net/ubuntu/artful/+source/systemd/+bug/1734167 this is not solved, is it? we cannot pull dbus earlier, and pulling resolved earlier means it will not reconnect to the bus. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Released Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Changed in: dbus (Ubuntu) Milestone: ubuntu-16.11 => None -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Released Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
This is fixed in cloud-init 0.7.9. ** Changed in: cloud-init Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Released Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
This is fixed in cloud-init 0.7.9. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Released Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Marking this verified. I booted an instance in gce. ## launch an instance project="smoser-00" # from gcloud compute images list ubuntu-1604-xenial-v20161115 img="/ubuntu-os-cloud/ubuntu-1604-xenial-v20161115" name="smfoo3" zone="us-east1-b" mtype="f1-micro" gcloud compute "--project=$project" instances create "$name" \ "--zone=$zone" "--machine-type=$mtype" --network=default \ "--maintenance-policy=MIGRATE" \ --image="$img" \ --boot-disk-size=10 --boot-disk-type=pd-standard \ "--boot-disk-device-name=$name" ## ssh in # get htools for saving logs and such % git clone https://gist.github.com/29ea35a797c0df1fcb6ac875a024efa9.git htools % sudo ./htools/save-old-data orig-boot new instance local: not found new instance net : not found reformattable: not found disk_setup ran: true mounts ran: true proc-mounts: /etc/fstab: % sudo ./htools/enable-proposed deb http://us-east1.gce.archive.ubuntu.com/ubuntu/ xenial-proposed main universe % sudo apt-get update -qy && sudo apt-get install cloud-init -qy % dpkg-query --show cloud-init cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.1 % sudo ./htools/do-reboot clean cleared /var/lib/cloud cleared logs rebooting # ssh back in % cat /proc/uptime 29.78 19.66 % journalctl --full --no-pager | grep -i "ordering" || echo no order no order % journalctl --full --no-pager | grep -i "break" || echo no break % -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Merge proposal linked: https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/311163 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Merge proposal linked: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/310547 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Also affects: dbus (Ubuntu Yakkety) Importance: Undecided Status: New ** Also affects: cloud-init (Ubuntu Yakkety) Importance: Undecided Status: New ** Also affects: dbus (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: cloud-init (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu Xenial) Status: New => Confirmed ** Changed in: cloud-init (Ubuntu Yakkety) Status: New => Confirmed ** Changed in: cloud-init (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: cloud-init (Ubuntu Yakkety) Importance: Undecided => Medium ** Changed in: cloud-init (Ubuntu Yakkety) Status: Confirmed => Fix Released ** Changed in: dbus (Ubuntu Xenial) Status: New => Invalid ** Changed in: dbus (Ubuntu Yakkety) Status: New => Invalid ** No longer affects: dbus (Ubuntu Xenial) ** No longer affects: dbus (Ubuntu Yakkety) ** Description changed: - During boot, cloud-init does DNS resolution checks to if particular - metadata services are available (in order to determine which cloud it is - running on). These checks happen before systemd-resolved is up[0] and - if they resolve unsuccessfully they take 25 seconds to complete. + === Begin SRU Template === + [Impact] + In cases where cloud-init used dns during early boot and system was + configured in nsswitch.conf to use systemd-resolvd, the system would + timeout on dns attempts making system boot terribly slow. + + [Test Case] + Boot a system on GCE. + check for WARN in /var/log/messages + check that time to boot is reasonable (<30 seconds). In failure case the + times would be minutes. + + [Regression Potential] + Changing order in boot can be dangerous. There is real chance for + regression here, but it should be fairly small as xenial does not include + systemd-resolved usage. This was first noticed on yakkety where it did. + + [Other Info] + It seems useful to SRU this in the event that systemd-resolvd is used + on 16.04 or the case where user upgrades components (admittedly small use + case). + + === End SRU Template === + + + + During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud- init attempts to resolve three known-invalid addresses ("does-not- exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud- init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Status in cloud-init source package in Xenial: Confirmed Status in cloud-init source package in Yakkety: Fix Released Bug description: === Begin SRU Template === [Impact] In cases where cloud-init used dns during early boot and system was configured in nsswitch.conf to use systemd-resolvd, the system would timeout on dns attempts making system boot terribly slow. [Test Case] Boot a system on GCE. check for WARN in /var/log/messages check that time to boot is reasonable (<30 seconds). In failure case the times would be minutes. [Regression Potential] Changing order in boot can be dangerous. There is real chance for regression here, but it should be fairly small as xenial does not include systemd-resolved usage. This was first noticed on yakkety where it did. [Other Info] It seems useful to SRU this in the event that systemd-resolvd is used on 16.04 or the case where user upgrades components (admittedly small use case). === End SRU Template === During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("do
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Discussed that upstream: The gist of it is: - 8< -- if you want to be an early boot service, then you should use something like this: Before=sysinit.target And that's all. Inserting yourself between the sockets and the regular services (which your suggested deps do) is highly problematic, if you actually intend to make use of the sockets, as then you will make the system hang, as to fulfill your requests you need the services you are delaying... - 8< -- So replacing cloud-init.service's Before=basic.target Before=dbus.socket with Before=sysinit.target should DTRT. dbus.socket (and all other sockets) will start after sysinit.target as part of basic.target. ** Changed in: dbus (Ubuntu) Status: In Progress => Won't Fix ** Changed in: dbus (Ubuntu) Assignee: Martin Pitt (pitti) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Won't Fix Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Bug watch added: freedesktop.org Bugzilla #98254 https://bugs.freedesktop.org/show_bug.cgi?id=98254 ** Also affects: dbus via https://bugs.freedesktop.org/show_bug.cgi?id=98254 Importance: Unknown Status: Unknown ** Changed in: dbus (Ubuntu) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in D-Bus: Unknown Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: In Progress Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
This bug was fixed in the package cloud-init - 0.7.8-15-g6e45ffb- 0ubuntu1 --- cloud-init (0.7.8-15-g6e45ffb-0ubuntu1) yakkety; urgency=medium * New upstream snapshot. - systemd: Run cloud-init.service Before dbus.socket not dbus.target [Daniel Watkins] (LP: #1629797). -- Scott Moser Fri, 07 Oct 2016 12:41:38 -0400 ** Changed in: cloud-init (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
To determine if this has been fixed, boot an image that has the GCE data source enabled (e.g. the image from cloud-images.ubuntu.com) but not on GCE. Examine the output of `journalctl` and look for the following lines: Oct 07 16:17:39 ubuntu cloud-init[1009]: [CLOUDINIT] __init__.py[DEBUG]: Seeing if we can get any data from Oct 07 16:19:19 ubuntu cloud-init[1009]: [CLOUDINIT] DataSourceGCE.py[DEBUG]: http://metadata.google.internal/computeMetadata/v1/ is not resolvable The timestamps on them should be no more than fractions of a second apart (the above example is on a _broken_ instance). (Note that, (a) there may be lines in between these two, and (b) you have to use `journalctl` because cloud-init.log doesn't have correct timestamps.) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: In Progress Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Changed in: cloud-init (Ubuntu) Status: Fix Released => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: In Progress Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Merge proposal linked: https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/307927 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
** Also affects: cloud-init Importance: Undecided Status: New ** Changed in: cloud-init Status: New => Fix Committed ** Changed in: cloud-init Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
This bug was fixed in the package cloud-init - 0.7.8-14-g94fd35e- 0ubuntu1 --- cloud-init (0.7.8-14-g94fd35e-0ubuntu1) yakkety; urgency=medium * New upstream snapshot. - systemd: run cloud-init.service Before dbus.service (LP: #1629797) - unittests: fix use of mock 2.0 'assert_called' when running make check [Ryan Harper] - Improve module documentation and doc cleanup. [Wesley Wiedenmeier] -- Scott Moser Tue, 04 Oct 2016 16:46:05 -0400 ** Changed in: cloud-init (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: Fix Released Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Early in z-series we should look into starting D-Bus ealier, to fix this in a more generic fashion. ** Package changed: systemd (Ubuntu) => dbus (Ubuntu) ** Changed in: dbus (Ubuntu) Milestone: None => ubuntu-16.11 ** Changed in: dbus (Ubuntu) Assignee: (unassigned) => Martin Pitt (pitti) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dbus in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: In Progress Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Intent is to work around this in cloud-init via 'Before=dbus.socket'. ** Changed in: cloud-init (Ubuntu) Status: Invalid => In Progress ** Changed in: cloud-init (Ubuntu) Importance: Undecided => Critical ** Changed in: cloud-init (Ubuntu) Assignee: Dan Watkins (daniel-thewatkins) => Scott Moser (smoser) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: In Progress Status in dbus package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
So, the root cause is completely clear: dbus.socket starts early, then cloud-init starts which blocks the entire basic.target (early boot) on network operations, thus dbus.service cannot start. nss-resolve already sees dbus.socket and thus can connect (instead of failing fast), and then gets the 25s D-Bus timeout as D-Bus is blocked. - Moving dbus.service into early boot would be a bold step, and I don't think such a large change is appropriate two weeks before release. - Rearranging nsswitch.conf and modify it on the fly also sounds like a big no. - I'd also not like to generally move dbus.socket into late boot, as that would break other services during early boot which queue up a connection to D-Bus. - So far the cleanest way out of this would be to also make dbus.socket wait for cloud-init.service, as that already blocks dbus.service. I verified that name resolution is then fast again. This also doesn't cause dependency loops as both cloud-init.service and sockets.target run in early boot. Could you try adding "Before=dbus.socket" to /lib/systemd/system/cloud- init.service and confirm that this helps? (Does for me in a container, but I don't have access to GCE or EC2) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: Invalid Status in systemd package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Simpler reproducer without cloud-init: Add "ExecStartPre=/bin/sleep 1000" to /lib/systemd/system/dbus.service. This happens if dbus.socket is already running, but dbus.service isn't yet as it's blocked by cloud- init.service. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: Invalid Status in systemd package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
A convenient way to test this is to install libnss-resolve and cloud- init into a yakkety container. Then cloud-init will basically hang forever, looping on 2016-10-04 12:58:48,716 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [100/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance- id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] No route to host',))] and since cloud-init.service runs during early boot, not much else (dbus, resolved, etc.) has started at that time. During that, name resolution is indeed broken. I think nss-resolve should quickly fall back to DNS if D-Bus isn't running yet. ** Changed in: systemd (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: Invalid Status in systemd package in Ubuntu: Triaged Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1629797] Re: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up
Adding After=systemd-resolved.service to /lib/systemd/system/cloud- init.service causes the following in journalctl: Oct 03 12:05:04 yakkety-161003-0945 systemd[1]: basic.target: Found ordering cycle on basic.target/start Oct 03 12:05:04 yakkety-161003-0945 systemd[1]: basic.target: Found dependency on cloud-init.service/start Oct 03 12:05:04 yakkety-161003-0945 systemd[1]: basic.target: Found dependency on systemd-resolved.service/start Oct 03 12:05:04 yakkety-161003-0945 systemd[1]: basic.target: Found dependency on basic.target/start Oct 03 12:05:04 yakkety-161003-0945 systemd[1]: basic.target: Breaking ordering cycle by deleting job cloud-init.service/start Oct 03 12:05:04 yakkety-161003-0945 systemd[1]: cloud-init.service: Job cloud-init.service/start deleted to break ordering cycle starting with basic.target/start ** Description changed: - Seems yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, - compared to the ~30 seconds of the first boot and ~10 seconds thereafter - in xenial. + During boot, cloud-init does DNS resolution checks to if particular + metadata services are available (in order to determine which cloud it is + running on). These checks happen before systemd-resolved is up[0] and + if they resolve unsuccessfully they take 25 seconds to complete. + + This has substantial impact on boot time in all contexts, because cloud- + init attempts to resolve three known-invalid addresses ("does-not- + exist.example.com.", "example.invalid." and a random string) to enable + it to detect when it's running in an environment where a DNS server will + always return some sort of redirect. As such, we're talking a minimum + impact of 75 seconds in all environments. This increases when cloud- + init is configured to check for multiple environments. + + This means that yakkety is consistently taking 2-3 minutes to boot on + EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 + seconds thereafter in xenial. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1629797 Title: resolve service in nsswitch.conf adds 25 seconds to failed lookups before systemd-resolved is up Status in cloud-init package in Ubuntu: Invalid Status in systemd package in Ubuntu: New Bug description: During boot, cloud-init does DNS resolution checks to if particular metadata services are available (in order to determine which cloud it is running on). These checks happen before systemd-resolved is up[0] and if they resolve unsuccessfully they take 25 seconds to complete. This has substantial impact on boot time in all contexts, because cloud-init attempts to resolve three known-invalid addresses ("does- not-exist.example.com.", "example.invalid." and a random string) to enable it to detect when it's running in an environment where a DNS server will always return some sort of redirect. As such, we're talking a minimum impact of 75 seconds in all environments. This increases when cloud-init is configured to check for multiple environments. This means that yakkety is consistently taking 2-3 minutes to boot on EC2 and GCE, compared to the ~30 seconds of the first boot and ~10 seconds thereafter in xenial. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp