** Also affects: cloud-utils (Ubuntu Xenial) Importance: Undecided Status: New
** Changed in: cloud-utils (Ubuntu Xenial) Status: New => In Progress ** Description changed: + === Begin SRU Template === + [Impact] + The 16.10 kernel dropped a legacy kernel module alias that allowed usage of + the 'overlay' filesystem via name 'overlayfs'. This broke overlayroot as + it explicitly tried to to use 'overlayfs' by name in loading of modules and + also in entry in /etc/fstab. + + Without this fix, overlayroot will simply not work on any upstream kernel + or Ubuntu kernel of 16.10 (yakkety) or later. + + [Test Case] + Note, not applying proposed as shown in step 3 below will recreate failure. + + 1.) Start an instance of a cloud image. + + 2.) get a suitable 4.8 kernel + On 16.10 or later, this is already done. On 16.04, we currently need to + install the kernel team's PPA to get one. + + $ sudo apt-add-repository -y ppa:canonical-kernel-team/ppa + $ sudo apt update -q && sudo apt install -y linux-virtual-hwe-16.04-edge </dev/null + + 3.) Enable proposed and install overlayroot to show fix. + $ rel=$(lsb_release -sc) + $ echo "deb http://archive.ubuntu.com/ubuntu $rel-proposed main" | + $ sudo tee /etc/apt/sources.list.d/proposed.list + $ sudo apt update -qy && sudo apt install -qy overlayroot </dev/null + $ dpkg-query --show overlayroot + overlayroot 0.27ubuntu1.3 + + 4.) Enable overlayroot and reboot + # remove the cloud-init written mount options for /dev/vdb + # if we do not do this, then /mnt ends up not mounted due to ordering. + $ sudo sed -i.dist s/,x-systemd.requires=cloud-init.service,comment=cloudconfig// /etc/fstab + $ echo "overlayroot=tmpfs" | sudo tee /etc/overlayroot.local.conf + $ sudo reboot + + 5.) log back in and look around. + a.) check that 'overlayroot' is in /proc/mounts + $ awk '$1 == "overlayroot" { print $0 }' /proc/mounts + overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 + + b.) check /run/initramfs/overlayroot.log + $ grep success /run/initramfs/overlayroot.log + [success]: configured root with 'tmpfs' using overlay per /dev/vda1/etc/overlayroot.local.conf + + 6.) try with recurse disabled + + Assuming you're on the same system and in an overlayroot, to change the + file necessary, we use overlayroot-chroot. + + $ echo overlayroot=tmpfs:recurse=0 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf + $ sudo reboot + + 7.) log back in and look around. + This time the /mnt should not have overlay on it. + + $ grep vdb /proc/mounts + /dev/vdb /mnt ext4 rw,relatime,data=ordered 0 0 + $ grep overlay /proc/mounts + overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 + $ cat /etc/overlayroot.local.conf + overlayroot=tmpfs:recurse=0 + + [Regression Potential] + The most likely regression is just in failure for overlayroot to work. + That was the case 100% of the time on any kernel without 'overlayfs' + filesystem, so this can't really make things worse from that perspective. + + Some of the code change was related to fixing another issue, with 'recurse'. + Testing recurse (where not just / is mounted as an overlayroot) is done + above + echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf + + [Other Info] + The full overlayroot/init-bottom/overlayroot can be seen at [1]. The + specific change that fixed the issue is in revision 115 at [2]. + + Related bugs: + * bug 1630274 adjusted the mechanism for determining if overlay was supported. + The change to do so is included in the xenial backport for this bug. + + [1] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/view/head:/overlayroot/scripts/init-bottom/overlayroot + [2] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/115 + + === End SRU Template === + + + === Original Description === + As mentioned in LP: #1411294, it's now called 'overlay' instead of 'overlayfs'. + Ubuntu had patched the kernel for compatibility. + + The Ubuntu kernels as of 4.8 (16.10 kernel) and possibly a bit before no + longer have a overlayfs module either. Thus, this is now affecting + yakkety. + + (The original reporter is @~gpo-9.) + The changes recently added to mount-image-callback under bug 1493188 that were targeted at supporting 'overlay' filesystem type while still supporting 'overlayfs' may result in exiting with "unsupported" when a mount would work. The changes basically said: if overlay in /proc/filesystems: return GOOD modprobe --quiet --use-blacklist overlay if overlay in /proc/filesystems: return GOOD return BAD the path that fails now that previously succeeded is when 'modprobe' would not work, but 'mount -t overlay' *would*. This occurs in build systems where the running kernel is of one version and code is running in a chroot that does not have access to the kernel modules in /lib/modules/. 'mount -t overlay' works in this scenario because the mount abi call passes the filesystem type 'overlay', and the kernel then does: fs/filesystems.c: if (!fs && (request_module("fs-%.*s", len, name) == 0)) The kernel's scope is not chrooted, so the module gets loaded and everything is happy. The new changes are to instead basically do: - for fstype in overlay overlayfs; do - mount -t $fstype <options> && return GOOD - if /proc/filesystems is readable and $fstype in /proc/filesystems; then - return FAIL - fi - done + for fstype in overlay overlayfs; do + mount -t $fstype <options> && return GOOD + if /proc/filesystems is readable and $fstype in /proc/filesystems; then + return FAIL + fi + done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1630274 Title: mount-image-callback may not mount overlay when it could To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-utils/+bug/1630274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs