[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
This bug was fixed in the package systemd - 237-3ubuntu10 --- systemd (237-3ubuntu10) bionic; urgency=medium * Create tmpfiles for persistent journal in postinst only when running systemd (LP: #1748659) -- Balint Reczey Fri, 20 Apr 2018 18:55:56 +0200 ** Changed in: systemd (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: systemd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
The semantics of systemd-networkd-wait-online.service are that the service is ready when "the Internet" is reachable. In the past, the implementation of network-online.target has been less correct on Ubuntu, *because* ipv6 RA was handled asynchronously, and this target did not block waiting to find out if ipv6 had been configured. That was mostly ok up until now in an IPv4-dominated world, but over the course of 18.04 LTS's lifetime, I expect we will increasingly see IPv6-enabled deployments where the default IPv6 route is as important as, or more important than, the default IPv4 route, and to not block waiting for SLAAC results in an RFC-compliant manner would make the boot racy. We cannot regard IPv6 as an afterthought. I agree with systemd upstream's decision to block on IPv6 results by default. However, in most environments where cloud-init is running, we shouldn't need a configurable timeout for SLAAC at all because we know by design whether or not we can expect ipv6 and if we know that the substrate does not provide ipv6, the netplan yaml for this provider should specify 'accept-ra: false'. Then the timeout is the correct 0, and there is no adverse effect on boot speed. For public clouds that don't provide netplan yaml today and instead have default netplan yaml generated by cloud-init, this should be included as part of the default policy in cloud-init. For any public clouds that are ipv6 opt-in, it is probably still appropriate to emit accept-ra: false by default in order to ensure a good default experience; and deal with opt-in ipv6 support via user- provided (or, preferably: cloud-provided) override cloud config. For the lxd case, the default should be 'accept-ra: false', with logic added later in lxd to set 'accept-ra: true' when lxd knows that the bridge supports ipv6. I think we will want to extend networkd and netplan to allow expression of additional network policies beyond the options of 'accept-ra: true' and 'accept-ra: false'. It should be possible to express that the kernel should try SLAAC, but that networkd should not block on SLAAC results for purposes of systemd-networkd-wait-online, while still blocking on *ipv4* connectivity for this interface. However, this will take some time to design properly, and I do not think that this should be the default for networkd+netplan once implemented: the forward- looking default is still that we should wait for a firm answer from SLAAC before considering the network up, and that if the user wants a different policy from this, it should be an override of the default netplan configuration. So I believe the quick path for addressing this issue in 18.04 is for cloud-init to declare 'accept-ra: false' everywhere that this is reasonable, with a possible relaxation to a future opportunistic accept- ra policy once this is available in netplan+networkd. ** Also affects: cloud-init (Ubuntu) Importance: Undecided Status: New ** Also affects: netplan.io (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
The default will remain, implicit, RequiredForOnline=yes. And it will be up to netplan / cloud metdata / cloud-init, to pass via netplan whatever the right syntax there needs to be to tell networkd that RequiredForOnline needs to be relaxed for ipv4, or ipv6, or for the whole interface altogether. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
NetworkManager I believe already supports settings if ipv6/ipv4 should be attempted, and whether or not ipv4/ipv6 needs to be considered for a networkmanager-network to be considered up. networkd wise, I am currently inclined to extend following options passed to RequiredForOnline=no|ipv4|ipv6|yes. This work should be implementable and SRUable between GA and .1 releases. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
I disagree that SLAAC is something that we should by default block reaching network-online.target. In general we cannot know in advance whether or not another system in the same subnet is providing Router Advertisements to other instances. If one instance in my VPC is running radvd, ec2 has no knowledge of this meaning that if cloud-init sets accept-ra: false; that disables accepting RAs and that breaks this use-case; this is similar for LXD as well. Additionally, accept-ra:false doesn't mean networkd doesn't handle RA, it also disables accept-ra in the kernel as well; I believe that *IF* you are running an instance where you expect to use SLAAC and know that you won't use other means of network configuration then that instance should set some sort of flag to indicate that you want network-online.target to wait for a SLAAC response. No matter how forward looking we want to be; it doesn't make sense to punish the majority of boots today with a 10 second timeout. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
I am wary of any design decision in netplan that fundamentally prioritizes ipv4 over ipv6. I'd like to suggest the following working definition of network-online.target: - all "non-optional" interfaces have an address configured for at least one address family or have timed out, and - there is at least one default route for at least one address family, and - there is at least one DNS server known. Does this sound like a correct, and complete, definition of what network-online.target should provide, by default, with further knobs in netplan to enforce specific address families? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
Addendum: we also need to be sure that the DNS server address is one that we have a route for. It does no good to start trying to talk to the DNS server if the DNS address is IPv4 but we only have routes for IPv6. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
I don't think not blocking for RAs during boot as a decision that prefers ipv4 over ipv6. networkd is just doing something else by default that the kernel doesn't do today, or historically; *and* there is no toggle for the blocking; only do you want RAs or not *and* if you say you don't want RAs it disables them in the kernel too. That's just not very flexible and breaks at least one known use-case w.r.t radvd use. W.r.t the the online-target; what you describe sounds complete. I think it's worth mentioning that this state can be reached by combining different interfaces; one may have a default route, a second has DNS, and a third has an address. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
in summary the consensus is as follows: - cloud-init should not provide accept-ra key to netplan.io - netplan.io should not emit any accept-ra keys to networkd - networkd, when not receiving any accept-ra keys should use kernel implicit default - networkd, when in implicit RA mode, should not block on getting RA as a pre-condition for a link to become ready I believe cloud-init & netplan.io portions are done already. ** Changed in: cloud-init (Ubuntu) Status: New => Fix Released ** Changed in: netplan.io (Ubuntu) Status: New => Fix Released ** Changed in: systemd (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
** Description changed: + [Impact] + + * On ipv6 less hosts, boot stalls for 10s + * This is due to implicit RA being on, and wait-online awaiting for RA to timeout + * Expectation is, that since explicit request for RA was not done, it should not block network-online.target, whilst all other aspects of network-online.target should be honored. + + [Test Case] + + * lxc network set lxdbr0 ipv6.address none + * lxc launch ubuntu-daily:bionic lp1765173 + * sleep 20 && lxc exec lp1765173 systemd-analyze blame | head + + Bad result is: + 12.157s systemd-networkd-wait-online.service + Good result is: + less than 3s + + [Regression Potential] + + * If explicit IPv6 dhcp/ra was enabled in .network file, the boot will + be blocked awaiting RA response + + * Only kernel implicit RA configuration will now become "async" + + * This is inline with behaviour of xenial systems + + [Other Info] + + * Original bug report + 1. $ lsb_release -rd Description: Ubuntu Bionic Beaver (development branch) Release: 18.04 2. $ apt-cache policy systemd systemd: - Installed: 237-3ubuntu8 - Candidate: 237-3ubuntu8 - Version table: - *** 237-3ubuntu8 500 - 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages - 100 /var/lib/dpkg/status + Installed: 237-3ubuntu8 + Candidate: 237-3ubuntu8 + Version table: + *** 237-3ubuntu8 500 + 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages + 100 /var/lib/dpkg/status 3. systemd-networkd-wait-online doesn't block for 10 seconds waiting on an IPV6 Router Advertisement 4. systemd-networkd sends at least two RA solicitation packets waiting for a response before setting the link to configured. This blocks the boot for every Ubuntu system where it does not have a IPV6 router responding to solicitations. THis includes bionic containers, cloud instances, vms and physical machines. -- We can see that we're spending 13 seconds waiting for networkd to say the network is up. % systemd-analyze blame - 13.326s systemd-networkd-wait-online.service -999ms cloud-init-local.service -954ms cloud-init.service -887ms cloud-config.service -749ms dev-vda1.device -666ms cloud-final.service -248ms keyboard-setup.service -175ms systemd-udev-trigger.service -171ms lxd-containers.service -165ms snapd.service -154ms apparmor.service -147ms ssh.service -144ms systemd-timesyncd.service -133ms grub-common.service -130ms systemd-journald.service -130ms accounts-daemon.service - 99ms systemd-modules-load.service - 98ms systemd-resolved.service - 94ms apport.service - 92ms rsyslog.service - 88ms systemd-logind.service - 80ms lvm2-monitor.service - 75ms iscsid.service - 64ms ebtables.service - 62ms user@1000.service - 54ms dev-mqueue.mount - 53ms ufw.service - 52ms systemd-remount-fs.service - 52ms kmod-static-nodes.service - 34ms systemd-journal-flush.service - 34ms polkit.service - 32ms systemd-tmpfiles-setup-dev.service - 27ms systemd-udevd.service - 26ms systemd-networkd.service - 26ms systemd-sysctl.service - 25ms systemd-tmpfiles-setup.service - 21ms dev-hugepages.mount - 17ms sys-kernel-debug.mount - 16ms console-setup.service - 15ms plymouth-read-write.service - 15ms snapd.socket - 15ms plymouth-quit.service - 14ms systemd-update-utmp.service - 10ms systemd-user-sessions.service - 9ms boot-efi.mount - 8ms sys-fs-fuse-connections.mount - 8ms systemd-update-utmp-runlevel.service - 8ms lxd.socket - 7ms blk-availability.service - 5ms systemd-random-seed.service - 3ms setvtrgb.service - 3ms plymouth-quit-wait.service - 3ms sys-kernel-config.mount - + 13.326s systemd-networkd-wait-online.service + 999ms cloud-init-local.service + 954ms cloud-init.service + 887ms cloud-config.service + 749ms dev-vda1.device + 666ms cloud-final.service + 248ms keyboard-setup.service + 175ms systemd-udev-trigger.service + 171ms lxd-containers.service + 165ms snapd.service + 154ms apparmor.service + 147ms ssh.service + 144ms systemd-timesyncd.service + 133ms grub-common.service + 130ms systemd-journald.service + 130ms accounts-daemon.service + 99ms systemd-modules-loa
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
Test PPA is available at $ sudo add-apt-repository ppa:ci-train-ppa- service/3243 ** Changed in: systemd (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1765173] Re: networkd waits 10 seconds for ipv6 network discovery by default
I've given this a test in LXD on my system which triggers the RA stall; after upgrading it does not wait for an RA before marking the interface configured and also confirmed that it does not regress the case where there are existing RA's on the network; those are still received. Note that we see eth0: Configured before the Discovering IPv6 routers, and Received Router Advertisement. Apr 20 20:55:21.062761 b1 systemd[1]: Started Network Service. Apr 20 20:55:21.904103 b1 systemd-networkd[163]: eth0: Configured Apr 20 20:55:21.904190 b1 systemd-networkd[163]: eth0: Discovering IPv6 routers Apr 20 20:55:21.904252 b1 systemd-networkd[163]: NDISC: Started IPv6 Router Solicitation client Apr 20 20:55:21.904632 b1 systemd-networkd[163]: NDISC: Sent Router Solicitation, next solicitation in 3s Apr 20 20:55:21.906538 b1 systemd-networkd[163]: NDISC: Received Router Advertisement: flags none preference medium lifetime 30 sec root@b1:~# networkctl status ●State: routable Address: 10.8.107.88 on eth0 2001:db8:100:f101:216:3eff:fe5f:d776 on eth0 fe80::216:3eff:fe5f:d776 on eth0 Gateway: 10.8.107.1 on eth0 fe80::216:3eff:fe31:e211 (Xensource, Inc.) on eth0 DNS: 10.8.107.1 Search Domains: lxd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1765173 Title: networkd waits 10 seconds for ipv6 network discovery by default To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1765173/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs