So, after much discussion [1] we came to the realization of why 'start networking' was here. The issue was that the normal case for cloud-images (and ubuntu as a whole) is that / is mounted "ro" when /sbin/init starts. When this is the case, /run will be mounted before /. However, if / is mounted "rw" then "mountall first scans for all filesystems that don't require further processing and reports them mounted", so "mounted MOUNTPOINT=/" occurs before virtual-filesystems.
if mounted MOUNTPOINT=/ occurs before virtual-filesystems, then udev will not start because it depends on virtual-filesystems. That means means networking wont get started either, as ifup occurs via udev events for discovered network adapters. As a result, we hang in cloud-init- nonet waiting for networking devices to come up that never will. Note, slangasek pointed out that while /run is guaranteed to be mounted before / (except withotu 'ro'), virtual-filesystems is *not* guaranteed. So, even with 'ro' root, a race condition could occur here. It would also, then imply that cloud-init cannot actually use some filesystems in cloud-init-local (such has /proc or /sys). slangasek suggested that a fix for bug 643289 would improve the situation here as the virtual-filesystems could all move forward even while the 'mounted MOUNTPOINT=/' were processing (currently it blocks *everything*). That would mean we wouldn't deadlock here. -- [1] http://irclogs.ubuntu.com/2012/09/05/%23ubuntu-devel.html#t18:16 ** Description changed: In development of 'overlayroot' package, I was mounting / as rw in the initramfs. This was causing a different order of execution of mounts, and as a result a different order of networking resolvconf and networking bringup. It exposed a bug in resolvconf, which was being called on boot in this order: ==== Mon Jul 30 19:08:58 UTC 2012 /sbin/resolvconf -a lo.inet ==== ==== Mon Jul 30 19:08:59 UTC 2012 /sbin/dhclient-script ==== ==== Mon Jul 30 19:08:59 UTC 2012 /sbin/resolvconf -a eth0.dhclient ==== ==== Mon Jul 30 19:08:59 UTC 2012 /sbin/resolvconf -a eth0.inet ==== ==== Mon Jul 30 19:08:59 UTC 2012 /sbin/resolvconf --enable-updates ==== ==== Mon Jul 30 19:08:59 UTC 2012 /etc/resolvconf/update.d/libc -u ==== The normal order is for resolvconf --enable-updates to be called (from /etc/init/resolvconf.conf) before anything else. As a result, I was seeing errors like: resolvconf: Error: /run/resolvconf/interface either does not exist or is not a directory This may be exposing a more grave issue, in that I believe the reason for dhclient coming up before resolvconf.conf started was that it was being run as a result of /etc/init/network-interface.conf. I don't immediately see how that is guarnateed to have /run ounted at all. ProblemType: Bug DistroRelease: Ubuntu 12.10 Package: resolvconf 1.67ubuntu1 ProcVersionSignature: User Name 3.5.0-6.6-generic 3.5.0 Uname: Linux 3.5.0-6-generic x86_64 Architecture: amd64 Date: Mon Jul 30 20:07:04 2012 PackageArchitecture: all ProcEnviron: TERM=xterm PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: resolvconf UpgradeStatus: No upgrade log present (probably fresh install) Related Bugs: * bug 800824: cloud-init-nonet times out in lxc * bug 925122: container's udevadm trigger --add affects the host + * bug 643289: idmapd does not starts to work after system reboot -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1031065 Title: cloud-init-nonet runs 'start networking' explicitly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1031065/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
