Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 09/11/2017 06:45 AM, Petr Mladek wrote: >> Except for the second printk line: If you boot with rdinit=/bin/hush >> then the first time you mount -t devtmpfs /dev /dev after boot (with >> CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return >> value but the last printk() doesn't output? The second and later times >> you repeat it, both printk() lines are output. >> >> What's up with printk? >> >> (I added the second printk because the _first_ one wasn't outputting >> that first time. Something is happening to flush the printk() queue >> instead of writing it out? > > You need to add "\n" at the end of the line. Otherwise, it expects > that the message would continue and puts it into a cont buffer. > The buffer is flushed only when another non-continuous message > is added. Ah. The next one flushes the previous one, meaning when I repeat the command I get the output I expected the second time but I'm seeing the _previous_ instance of it, not the current one. > This problem is more visible since the commit 5c2992ee7fd8a29d0412 > ("printk: remove console flushing special cases for partial buffered > lines"). Gotcha. My bad. Thanks, Rob
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 09/11/2017 06:45 AM, Petr Mladek wrote: >> Except for the second printk line: If you boot with rdinit=/bin/hush >> then the first time you mount -t devtmpfs /dev /dev after boot (with >> CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return >> value but the last printk() doesn't output? The second and later times >> you repeat it, both printk() lines are output. >> >> What's up with printk? >> >> (I added the second printk because the _first_ one wasn't outputting >> that first time. Something is happening to flush the printk() queue >> instead of writing it out? > > You need to add "\n" at the end of the line. Otherwise, it expects > that the message would continue and puts it into a cont buffer. > The buffer is flushed only when another non-continuous message > is added. Ah. The next one flushes the previous one, meaning when I repeat the command I get the output I expected the second time but I'm seeing the _previous_ instance of it, not the current one. > This problem is more visible since the commit 5c2992ee7fd8a29d0412 > ("printk: remove console flushing special cases for partial buffered > lines"). Gotcha. My bad. Thanks, Rob
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On (09/11/17 13:45), Petr Mladek wrote: [..] > > Except for the second printk line: If you boot with rdinit=/bin/hush > > then the first time you mount -t devtmpfs /dev /dev after boot (with > > CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return > > value but the last printk() doesn't output? The second and later times > > you repeat it, both printk() lines are output. > > > > What's up with printk? > > > > (I added the second printk because the _first_ one wasn't outputting > > that first time. Something is happening to flush the printk() queue > > instead of writing it out? > Maybe, we could setup the timer via an irq_work. We already use > this trick for flushing deferred printk and waking klogd. > It is not nice but it would be easier than the previous mess. printk() and printf() have similar behaviour here. both flush on \n. so let's keep it the way it is? -ss
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On (09/11/17 13:45), Petr Mladek wrote: [..] > > Except for the second printk line: If you boot with rdinit=/bin/hush > > then the first time you mount -t devtmpfs /dev /dev after boot (with > > CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return > > value but the last printk() doesn't output? The second and later times > > you repeat it, both printk() lines are output. > > > > What's up with printk? > > > > (I added the second printk because the _first_ one wasn't outputting > > that first time. Something is happening to flush the printk() queue > > instead of writing it out? > Maybe, we could setup the timer via an irq_work. We already use > this trick for flushing deferred printk and waking klogd. > It is not nice but it would be easier than the previous mess. printk() and printf() have similar behaviour here. both flush on \n. so let's keep it the way it is? -ss
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On Sun 2017-09-10 18:43:24, Rob Landley wrote: > On 05/25/2017 01:13 AM, Michael Ellerman wrote: > > Hi Rob, > > > > This is breaking a bunch of my powerpc boxes, for the exact same > > reason, they use a config that has DEVTMPFS_MOUNT=y and that trips > > up the initramfs. > > I've continued to use this locally but should probably make another > stab at submitting upstream. The obvious workaround until debian fixes > its 100% obvious bug seems to be: > > diff --git a/fs/namespace.c b/fs/namespace.c > index f8893dc..f57d5df 100644 > --- a/fs/namespace.c > +++ b/fs/namespace.c > @@ -2417,7 +2417,17 @@ static int do_add_mount(struct mount *newmnt, struct > path *path, int mnt_flags) > err = -EBUSY; > if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && > path->mnt->mnt_root == path->dentry) > + { > + if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT) && > + !strcmp(path->mnt->mnt_sb->s_type->name, "devtmpfs")) > + { > + printk(KERN_WARNING "Debian bug workaround for devtmpfs > overmount."); > + printk(KERN_WARNING "This line doesn't output for some > reason."); > + > + err = 0; > + } > goto unlock; > + } > > err = -EINVAL; > if (d_is_symlink(newmnt->mnt.mnt_root)) > > Except for the second printk line: If you boot with rdinit=/bin/hush > then the first time you mount -t devtmpfs /dev /dev after boot (with > CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return > value but the last printk() doesn't output? The second and later times > you repeat it, both printk() lines are output. > > What's up with printk? > > (I added the second printk because the _first_ one wasn't outputting > that first time. Something is happening to flush the printk() queue > instead of writing it out? You need to add "\n" at the end of the line. Otherwise, it expects that the message would continue and puts it into a cont buffer. The buffer is flushed only when another non-continuous message is added. This problem is more visible since the commit 5c2992ee7fd8a29d0412 ("printk: remove console flushing special cases for partial buffered lines"). Hmm, Linus wanted to add a timer that would always flush the cont buffer in a reasonable time frame. But there was a risk of deadlocks because of circular timer<->printk dependency, so it never happened. Maybe, we could setup the timer via an irq_work. We already use this trick for flushing deferred printk and waking klogd. It is not nice but it would be easier than the previous mess. Best Regards, Petr
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On Sun 2017-09-10 18:43:24, Rob Landley wrote: > On 05/25/2017 01:13 AM, Michael Ellerman wrote: > > Hi Rob, > > > > This is breaking a bunch of my powerpc boxes, for the exact same > > reason, they use a config that has DEVTMPFS_MOUNT=y and that trips > > up the initramfs. > > I've continued to use this locally but should probably make another > stab at submitting upstream. The obvious workaround until debian fixes > its 100% obvious bug seems to be: > > diff --git a/fs/namespace.c b/fs/namespace.c > index f8893dc..f57d5df 100644 > --- a/fs/namespace.c > +++ b/fs/namespace.c > @@ -2417,7 +2417,17 @@ static int do_add_mount(struct mount *newmnt, struct > path *path, int mnt_flags) > err = -EBUSY; > if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && > path->mnt->mnt_root == path->dentry) > + { > + if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT) && > + !strcmp(path->mnt->mnt_sb->s_type->name, "devtmpfs")) > + { > + printk(KERN_WARNING "Debian bug workaround for devtmpfs > overmount."); > + printk(KERN_WARNING "This line doesn't output for some > reason."); > + > + err = 0; > + } > goto unlock; > + } > > err = -EINVAL; > if (d_is_symlink(newmnt->mnt.mnt_root)) > > Except for the second printk line: If you boot with rdinit=/bin/hush > then the first time you mount -t devtmpfs /dev /dev after boot (with > CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return > value but the last printk() doesn't output? The second and later times > you repeat it, both printk() lines are output. > > What's up with printk? > > (I added the second printk because the _first_ one wasn't outputting > that first time. Something is happening to flush the printk() queue > instead of writing it out? You need to add "\n" at the end of the line. Otherwise, it expects that the message would continue and puts it into a cont buffer. The buffer is flushed only when another non-continuous message is added. This problem is more visible since the commit 5c2992ee7fd8a29d0412 ("printk: remove console flushing special cases for partial buffered lines"). Hmm, Linus wanted to add a timer that would always flush the cont buffer in a reasonable time frame. But there was a risk of deadlocks because of circular timer<->printk dependency, so it never happened. Maybe, we could setup the timer via an irq_work. We already use this trick for flushing deferred printk and waking klogd. It is not nice but it would be easier than the previous mess. Best Regards, Petr
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
Taking another stab at this old issue from last merge window... > Rob Landleywrites: >> On 05/23/2017 03:01 AM, Yury Norov wrote: >>> On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: Your userspace mounted a tmpfs over /dev when it couldn't mount a second identical instance of devtmpfs over itself. If you had a static /dev in initramfs but didn't configure _in_ devtmpfs to your kernel, your broken error path would have taken that out too with a pointless tmpfs mount. >>> >>> CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your >>> suggestion is correct. But I didn't do that specifically - I run >>> almost default kernel based on Ubuntu 14.04 config and environment. >> >> I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then >> launchd an initramfs instead (which didn't do the automount they >> requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually >> starts working in initramfs they have an insane error path that breaks >> the system, and does nothing _except_ break the system. ... On 05/25/2017 01:13 AM, Michael Ellerman wrote: > Hi Rob, > > This is breaking a bunch of my powerpc boxes, for the exact same > reason, they use a config that has DEVTMPFS_MOUNT=y and that trips > up the initramfs. I've continued to use this locally but should probably make another stab at submitting upstream. The obvious workaround until debian fixes its 100% obvious bug seems to be: diff --git a/fs/namespace.c b/fs/namespace.c index f8893dc..f57d5df 100644 --- a/fs/namespace.c +++ b/fs/namespace.c @@ -2417,7 +2417,17 @@ static int do_add_mount(struct mount *newmnt, struct path *path, int mnt_flags) err = -EBUSY; if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && path->mnt->mnt_root == path->dentry) + { + if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT) && + !strcmp(path->mnt->mnt_sb->s_type->name, "devtmpfs")) + { + printk(KERN_WARNING "Debian bug workaround for devtmpfs overmount."); + printk(KERN_WARNING "This line doesn't output for some reason."); + + err = 0; + } goto unlock; + } err = -EINVAL; if (d_is_symlink(newmnt->mnt.mnt_root)) Except for the second printk line: If you boot with rdinit=/bin/hush then the first time you mount -t devtmpfs /dev /dev after boot (with CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return value but the last printk() doesn't output? The second and later times you repeat it, both printk() lines are output. What's up with printk? (I added the second printk because the _first_ one wasn't outputting that first time. Something is happening to flush the printk() queue instead of writing it out? Built for x86-64, miniconfig attached for reference. I tested commit 4dfc2788033d from yesterday.) Rob # make ARCH=x86 allnoconfig KCONFIG_ALLCONFIG=x86_64.miniconf # make ARCH=x86 -j $(nproc) # boot arch/x86/boot/bzImage CONFIG_64BIT=y CONFIG_PCI=y CONFIG_BLK_DEV_SD=y CONFIG_ATA=y CONFIG_ATA_SFF=y CONFIG_ATA_BMDMA=y CONFIG_ATA_PIIX=y CONFIG_NET_VENDOR_INTEL=y CONFIG_E1000=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_RTC_CLASS=y # CONFIG_EMBEDDED is not set CONFIG_EARLY_PRINTK=y CONFIG_BINFMT_ELF=y CONFIG_BINFMT_SCRIPT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_BLK_DEV=y CONFIG_BLK_DEV_INITRD=y CONFIG_RD_GZIP=y CONFIG_BLK_DEV_LOOP=y CONFIG_EXT4_FS=y CONFIG_EXT4_USE_FOR_EXT2=y CONFIG_VFAT_FS=y CONFIG_FAT_DEFAULT_UTF8=y CONFIG_MISC_FILESYSTEMS=y CONFIG_SQUASHFS=y CONFIG_SQUASHFS_XATTR=y CONFIG_SQUASHFS_ZLIB=y CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_NET=y CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_INET=y CONFIG_IPV6=y CONFIG_NETDEVICES=y #CONFIG_NET_CORE=y #CONFIG_NETCONSOLE=y CONFIG_ETHERNET=y
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
Taking another stab at this old issue from last merge window... > Rob Landley writes: >> On 05/23/2017 03:01 AM, Yury Norov wrote: >>> On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: Your userspace mounted a tmpfs over /dev when it couldn't mount a second identical instance of devtmpfs over itself. If you had a static /dev in initramfs but didn't configure _in_ devtmpfs to your kernel, your broken error path would have taken that out too with a pointless tmpfs mount. >>> >>> CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your >>> suggestion is correct. But I didn't do that specifically - I run >>> almost default kernel based on Ubuntu 14.04 config and environment. >> >> I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then >> launchd an initramfs instead (which didn't do the automount they >> requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually >> starts working in initramfs they have an insane error path that breaks >> the system, and does nothing _except_ break the system. ... On 05/25/2017 01:13 AM, Michael Ellerman wrote: > Hi Rob, > > This is breaking a bunch of my powerpc boxes, for the exact same > reason, they use a config that has DEVTMPFS_MOUNT=y and that trips > up the initramfs. I've continued to use this locally but should probably make another stab at submitting upstream. The obvious workaround until debian fixes its 100% obvious bug seems to be: diff --git a/fs/namespace.c b/fs/namespace.c index f8893dc..f57d5df 100644 --- a/fs/namespace.c +++ b/fs/namespace.c @@ -2417,7 +2417,17 @@ static int do_add_mount(struct mount *newmnt, struct path *path, int mnt_flags) err = -EBUSY; if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && path->mnt->mnt_root == path->dentry) + { + if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT) && + !strcmp(path->mnt->mnt_sb->s_type->name, "devtmpfs")) + { + printk(KERN_WARNING "Debian bug workaround for devtmpfs overmount."); + printk(KERN_WARNING "This line doesn't output for some reason."); + + err = 0; + } goto unlock; + } err = -EINVAL; if (d_is_symlink(newmnt->mnt.mnt_root)) Except for the second printk line: If you boot with rdinit=/bin/hush then the first time you mount -t devtmpfs /dev /dev after boot (with CONFIG_DEVTMPFS_MOUNT already having mounted it), you get the 0 return value but the last printk() doesn't output? The second and later times you repeat it, both printk() lines are output. What's up with printk? (I added the second printk because the _first_ one wasn't outputting that first time. Something is happening to flush the printk() queue instead of writing it out? Built for x86-64, miniconfig attached for reference. I tested commit 4dfc2788033d from yesterday.) Rob # make ARCH=x86 allnoconfig KCONFIG_ALLCONFIG=x86_64.miniconf # make ARCH=x86 -j $(nproc) # boot arch/x86/boot/bzImage CONFIG_64BIT=y CONFIG_PCI=y CONFIG_BLK_DEV_SD=y CONFIG_ATA=y CONFIG_ATA_SFF=y CONFIG_ATA_BMDMA=y CONFIG_ATA_PIIX=y CONFIG_NET_VENDOR_INTEL=y CONFIG_E1000=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_RTC_CLASS=y # CONFIG_EMBEDDED is not set CONFIG_EARLY_PRINTK=y CONFIG_BINFMT_ELF=y CONFIG_BINFMT_SCRIPT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_BLK_DEV=y CONFIG_BLK_DEV_INITRD=y CONFIG_RD_GZIP=y CONFIG_BLK_DEV_LOOP=y CONFIG_EXT4_FS=y CONFIG_EXT4_USE_FOR_EXT2=y CONFIG_VFAT_FS=y CONFIG_FAT_DEFAULT_UTF8=y CONFIG_MISC_FILESYSTEMS=y CONFIG_SQUASHFS=y CONFIG_SQUASHFS_XATTR=y CONFIG_SQUASHFS_ZLIB=y CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_NET=y CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_INET=y CONFIG_IPV6=y CONFIG_NETDEVICES=y #CONFIG_NET_CORE=y #CONFIG_NETCONSOLE=y CONFIG_ETHERNET=y
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
The previous email was rejected by LKML. So resend again. On Thu, May 25, 2017 at 09:39:32PM +0800, Leo Yan wrote: > Hi Rob, Yury, > > On Mon, May 22, 2017 at 8:05 PM, Yury Norov> wrote: > > > Hi Rob, > > > > I found that next-20170522 fails to boot on arm64 machine with the > > following log: > > [...] > > [4.179509] Freeing unused kernel memory: 1088K > > Loading, please wait... > > mount: mounting udev on /dev failed: Device or resource busy > > W: devtmpfs not available, falling back to tmpfs for /dev > > Couldn't get a file descriptor referring to the console > > Begin: Loading essential drivers ... done. > > Begin: Running /scripts/init-premount ... done. > > Begin: Mounting root file system ... Begin: Running > > /scripts/local-top ... done. > > chvt: can't open console > > Gave up waiting for root device. Common problems: > > - Boot args (cat /proc/cmdline) > >- Check rootdelay= (did the system wait long enough?) > >- Check root= (did the system wait for the right device?) > > - Missing modules (cat /proc/modules; ls /dev) > > chvt: can't open console > > ALERT! /dev/sda does not exist. Dropping to a shell! > > Couldn't get a file descriptor referring to the console > > > > > > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) > > Enter 'help' for a list of built-in commands. > > > > (initramfs) > > > > Bisect points to your patch (attached below). If I revert it, everything > > becomes fine. If you need to know something more about my environment, > > feel free to ask me. > > > > At my side, I also can reproduce this failure on Hikey board with Debian > rootfs, > finally the debian rootfs cannot be mounted successfully, the booting log > is: > http://termbin.com/akar > > From the log, the failures are exactly same with Yury's reporting: > mount: mounting udev on /dev failed: Device or resource busy > ... > ALERT! /dev/disk/by-partlabel/system does not exist. Dropping to a > shell! > > After applied this patch, the kernel mounts mount devtmpfs on '/dev', then > later > udev cannot mount again for devfs. Finally cannot find rootfs path from > '/dev/disk/by-partlabel/system'. So after I comment out below sentence, the > rootfs can boot up again. > > diff --git a/init/main.c b/init/main.c > index 9ec09ff..2ea93d1 100644 > --- a/init/main.c > +++ b/init/main.c > @@ -1050,8 +1050,8 @@ static noinline void __init kernel_init_freeable(void) > ramdisk_execute_command = NULL; > prepare_namespace(); > } else if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)) { > - sys_mkdir("/dev", 0755); > - devtmpfs_mount("/dev"); > + //sys_mkdir("/dev", 0755); > + //devtmpfs_mount("/dev"); > } > > /* Open the /dev/console on the rootfs, this should never fail */ > > Thanks, > Leo Yan
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
The previous email was rejected by LKML. So resend again. On Thu, May 25, 2017 at 09:39:32PM +0800, Leo Yan wrote: > Hi Rob, Yury, > > On Mon, May 22, 2017 at 8:05 PM, Yury Norov > wrote: > > > Hi Rob, > > > > I found that next-20170522 fails to boot on arm64 machine with the > > following log: > > [...] > > [4.179509] Freeing unused kernel memory: 1088K > > Loading, please wait... > > mount: mounting udev on /dev failed: Device or resource busy > > W: devtmpfs not available, falling back to tmpfs for /dev > > Couldn't get a file descriptor referring to the console > > Begin: Loading essential drivers ... done. > > Begin: Running /scripts/init-premount ... done. > > Begin: Mounting root file system ... Begin: Running > > /scripts/local-top ... done. > > chvt: can't open console > > Gave up waiting for root device. Common problems: > > - Boot args (cat /proc/cmdline) > >- Check rootdelay= (did the system wait long enough?) > >- Check root= (did the system wait for the right device?) > > - Missing modules (cat /proc/modules; ls /dev) > > chvt: can't open console > > ALERT! /dev/sda does not exist. Dropping to a shell! > > Couldn't get a file descriptor referring to the console > > > > > > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) > > Enter 'help' for a list of built-in commands. > > > > (initramfs) > > > > Bisect points to your patch (attached below). If I revert it, everything > > becomes fine. If you need to know something more about my environment, > > feel free to ask me. > > > > At my side, I also can reproduce this failure on Hikey board with Debian > rootfs, > finally the debian rootfs cannot be mounted successfully, the booting log > is: > http://termbin.com/akar > > From the log, the failures are exactly same with Yury's reporting: > mount: mounting udev on /dev failed: Device or resource busy > ... > ALERT! /dev/disk/by-partlabel/system does not exist. Dropping to a > shell! > > After applied this patch, the kernel mounts mount devtmpfs on '/dev', then > later > udev cannot mount again for devfs. Finally cannot find rootfs path from > '/dev/disk/by-partlabel/system'. So after I comment out below sentence, the > rootfs can boot up again. > > diff --git a/init/main.c b/init/main.c > index 9ec09ff..2ea93d1 100644 > --- a/init/main.c > +++ b/init/main.c > @@ -1050,8 +1050,8 @@ static noinline void __init kernel_init_freeable(void) > ramdisk_execute_command = NULL; > prepare_namespace(); > } else if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)) { > - sys_mkdir("/dev", 0755); > - devtmpfs_mount("/dev"); > + //sys_mkdir("/dev", 0755); > + //devtmpfs_mount("/dev"); > } > > /* Open the /dev/console on the rootfs, this should never fail */ > > Thanks, > Leo Yan
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
Hi Rob, This is breaking a bunch of my powerpc boxes, for the exact same reason, they use a config that has DEVTMPFS_MOUNT=y and that trips up the initramfs. Rob Landleywrites: > On 05/23/2017 03:01 AM, Yury Norov wrote: >> On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: >>> Your userspace mounted a tmpfs over /dev when it couldn't mount a second >>> identical instance of devtmpfs over itself. If you had a static /dev in >>> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken >>> error path would have taken that out too with a pointless tmpfs mount. >> >> CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your >> suggestion is correct. But I didn't do that specifically - I run >> almost default kernel based on Ubuntu 14.04 config and environment. > > I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then > launchd an initramfs instead (which didn't do the automount they > requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually > starts working in initramfs they have an insane error path that breaks > the system, and does nothing _except_ break the system. Sure. But prior to your patch the only reason mounting devtmpfs could fail was because of something being broken, it was never automatically mounted. So we've changed the set of possibilities out from under the initramfs scripts. I agree it's not the greatest error handling ever, but it's pretty rude to break booting on existing working setups. If nothing else it makes bisection a nightmare. >> So the proper way is to remove broken >> config option and introduce new one. BTW, I see it is used once in >> drivers/base/devtmpfs.c. > > How does removing the broken config option (or renaming it to > CONFIG_DEFTMPFS_UBUNTU_IS_BROKEN) _not_ impact systems that were > previously happily using it in the contexts where it already worked? I don't think that's what Yury's suggesting, and indeed it wouldn't work. What would work, is to add a new config option, perhaps CONFIG_DEVTMPFS_MOUNT_INITRAMFS, which is 'default n'. Then existing setups continue to work, and people who want it mounted in the initramfs can enable the new option. cheers
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
Hi Rob, This is breaking a bunch of my powerpc boxes, for the exact same reason, they use a config that has DEVTMPFS_MOUNT=y and that trips up the initramfs. Rob Landley writes: > On 05/23/2017 03:01 AM, Yury Norov wrote: >> On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: >>> Your userspace mounted a tmpfs over /dev when it couldn't mount a second >>> identical instance of devtmpfs over itself. If you had a static /dev in >>> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken >>> error path would have taken that out too with a pointless tmpfs mount. >> >> CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your >> suggestion is correct. But I didn't do that specifically - I run >> almost default kernel based on Ubuntu 14.04 config and environment. > > I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then > launchd an initramfs instead (which didn't do the automount they > requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually > starts working in initramfs they have an insane error path that breaks > the system, and does nothing _except_ break the system. Sure. But prior to your patch the only reason mounting devtmpfs could fail was because of something being broken, it was never automatically mounted. So we've changed the set of possibilities out from under the initramfs scripts. I agree it's not the greatest error handling ever, but it's pretty rude to break booting on existing working setups. If nothing else it makes bisection a nightmare. >> So the proper way is to remove broken >> config option and introduce new one. BTW, I see it is used once in >> drivers/base/devtmpfs.c. > > How does removing the broken config option (or renaming it to > CONFIG_DEFTMPFS_UBUNTU_IS_BROKEN) _not_ impact systems that were > previously happily using it in the contexts where it already worked? I don't think that's what Yury's suggesting, and indeed it wouldn't work. What would work, is to add a new config option, perhaps CONFIG_DEVTMPFS_MOUNT_INITRAMFS, which is 'default n'. Then existing setups continue to work, and people who want it mounted in the initramfs can enable the new option. cheers
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 05/23/2017 06:08 PM, Yury Norov wrote: >> It was 2 years ago, but AFAIR I took the Ubuntu image here: >> http://cdimage.ubuntu.com/ubuntu-base/releases/14.04.1/release/ubuntu-base-14.04.1-core-arm64.tar.gz Have you applied updates since then? (Maybe they fixed their init script since 2 years ago?) >> Kernel config is attached. I build the kernel with simple 'make'. >> >> Yury > > Sorry, config is here. $ diff -u yury.conf /boot/config-4.4.0-78-generic | grep '^[-+]' | wc -l 10384 So that's not Ubuntu's current 14.04 kernel config. $ diff -u yury.conf /boot/config-4.2.0-36-generic | grep '^[-+]' | wc -l 10212 And it's not the oldest Ubuntu 14.04 config I have lying around (from a year ago). $ cd linux && make defconfig $ diff -u ~/yury.conf .config | grep '^[-+]' | wc -l 4369 It's much closer to the current defconfig, but still significantly different. So you're using a custom config, and can't switch off a symbol. Rob
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 05/23/2017 06:08 PM, Yury Norov wrote: >> It was 2 years ago, but AFAIR I took the Ubuntu image here: >> http://cdimage.ubuntu.com/ubuntu-base/releases/14.04.1/release/ubuntu-base-14.04.1-core-arm64.tar.gz Have you applied updates since then? (Maybe they fixed their init script since 2 years ago?) >> Kernel config is attached. I build the kernel with simple 'make'. >> >> Yury > > Sorry, config is here. $ diff -u yury.conf /boot/config-4.4.0-78-generic | grep '^[-+]' | wc -l 10384 So that's not Ubuntu's current 14.04 kernel config. $ diff -u yury.conf /boot/config-4.2.0-36-generic | grep '^[-+]' | wc -l 10212 And it's not the oldest Ubuntu 14.04 config I have lying around (from a year ago). $ cd linux && make defconfig $ diff -u ~/yury.conf .config | grep '^[-+]' | wc -l 4369 It's much closer to the current defconfig, but still significantly different. So you're using a custom config, and can't switch off a symbol. Rob
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
> It was 2 years ago, but AFAIR I took the Ubuntu image here: > http://cdimage.ubuntu.com/ubuntu-base/releases/14.04.1/release/ubuntu-base-14.04.1-core-arm64.tar.gz > > Kernel config is attached. I build the kernel with simple 'make'. > > Yury Sorry, config is here. config.tar.gz Description: application/gzip
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
> It was 2 years ago, but AFAIR I took the Ubuntu image here: > http://cdimage.ubuntu.com/ubuntu-base/releases/14.04.1/release/ubuntu-base-14.04.1-core-arm64.tar.gz > > Kernel config is attached. I build the kernel with simple 'make'. > > Yury Sorry, config is here. config.tar.gz Description: application/gzip
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On Tue, May 23, 2017 at 12:40:04PM -0500, Rob Landley wrote: > On 05/23/2017 03:01 AM, Yury Norov wrote: > > On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: > >> Your userspace mounted a tmpfs over /dev when it couldn't mount a second > >> identical instance of devtmpfs over itself. If you had a static /dev in > >> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken > >> error path would have taken that out too with a pointless tmpfs mount. > > > > CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your > > suggestion is correct. But I didn't do that specifically - I run > > almost default kernel based on Ubuntu 14.04 config and environment. > > I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then > launchd an initramfs instead (which didn't do the automount they > requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually > starts working in initramfs they have an insane error path that breaks > the system, and does nothing _except_ break the system. > > > Grepping the kernel code shows that arc, arm, arm64, m86k, metag, > > mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa > > enable it by default. > > Most of which Ubuntu doesn't support, so none of them could trigger the > broken error path in ubuntu's init script. > > Wait, are you saying you're doing a "make defconfig" on x86-64 and > booting ubuntu from the result? (Or is this arm?) Is _that_ the config > you still haven't specified in this conversation? I thought you were > using the /boot/config-4.4.0-78-generic and friends ubuntu installs. > (Which yes, also switch this symbol on.) No, I'm not saying it. I run arm64 with the config attached below. > I can add a "default n" line to drivers/base/Kconfig if "make defconfig" > is what you're building from. (This symbol never specified a default in > the first place, so I dunno which way it falls, but it's repeated in a > gazillion defconfig files and not present in others... meaning I still > dunno which way the default goes. When I do a "make defconfig" it uses > arch/x86/configs/x86_64_defconfig because the kernel has multiple > codepaths to accomplish the same thing. I'm not sure the built-in > "default y" lines are used at all anymore? What a mess...) > > But again, I'm just guessing what config you're using because you still > haven't _said_. I'm still trying to guess what you're doing when you hit > Ubuntu's bug. > > > So it means for me that (at least) users why run > > Ubuntu 14.04 will have bricked system one day after updating the > > kernel. > > Unless when they build their new kernel they open up menuconfig and > switch this symbol off. Which can't be done because...? > > Or you could add the devtmpfs.mount=0 argument to your kernel command > line, as documented in the CONFIG_DEVTMPFS_MOUNT menuconfig help text. > > The kernel already provides multiple workarounds for Ubuntu's bug, and > the issue only hits people who are manually building a new kernel from > source. If ubuntu provides a new kernel, I assume they'll tweak their > config _and_ fix their initramfs error path (which is just plain wrong). > > > If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel, > > It's not _me_ saying it, it's the kernel doing it. The patch is > conceptually a straightforward fix on the kernel side to make it _not_ > ignore that symbol in that context. > > > I think you cannot relay on it anymore because people may have it > > enabled or disabled randomly. > > I expected configs would have it randomly set, but the bug here is a > broken error path that does something actively harmful rather than going > "oh, we got a static /dev from somewhere, let's just leave it alone". > This error path goes out of its way to blank the contents of /dev by > mounting an empty tmpfs over it and leaving it empty, and then > complaining that /dev is blank _because_it_blanked_it_. > > If Ubuntu meant to intentionally halt the proceedings the script could > have done that explicitly. What did the author of that error path think > would happen, exactly? > > > So the proper way is to remove broken > > config option and introduce new one. BTW, I see it is used once in > > drivers/base/devtmpfs.c. > > How does removing the broken config option (or renaming it to > CONFIG_DEFTMPFS_UBUNTU_IS_BROKEN) _not_ impact systems that were > previously happily using it in the contexts where it already worked? > > If it's too much to ask people to switch it off when it was previously > on (but shouldn't have been), how is asking them to manually switch it > back on when it was previously on and needs to stay on better? (And if > you arrange it so "make oldconfig" migrates the old symbol to the new > one automatically, how would that work around the broken error path in > ubuntu's initramfs script? The rename becomes a NOP.) > > If you're saying it should default to "n" I can send a patch. If you > want me to tweak every
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On Tue, May 23, 2017 at 12:40:04PM -0500, Rob Landley wrote: > On 05/23/2017 03:01 AM, Yury Norov wrote: > > On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: > >> Your userspace mounted a tmpfs over /dev when it couldn't mount a second > >> identical instance of devtmpfs over itself. If you had a static /dev in > >> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken > >> error path would have taken that out too with a pointless tmpfs mount. > > > > CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your > > suggestion is correct. But I didn't do that specifically - I run > > almost default kernel based on Ubuntu 14.04 config and environment. > > I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then > launchd an initramfs instead (which didn't do the automount they > requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually > starts working in initramfs they have an insane error path that breaks > the system, and does nothing _except_ break the system. > > > Grepping the kernel code shows that arc, arm, arm64, m86k, metag, > > mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa > > enable it by default. > > Most of which Ubuntu doesn't support, so none of them could trigger the > broken error path in ubuntu's init script. > > Wait, are you saying you're doing a "make defconfig" on x86-64 and > booting ubuntu from the result? (Or is this arm?) Is _that_ the config > you still haven't specified in this conversation? I thought you were > using the /boot/config-4.4.0-78-generic and friends ubuntu installs. > (Which yes, also switch this symbol on.) No, I'm not saying it. I run arm64 with the config attached below. > I can add a "default n" line to drivers/base/Kconfig if "make defconfig" > is what you're building from. (This symbol never specified a default in > the first place, so I dunno which way it falls, but it's repeated in a > gazillion defconfig files and not present in others... meaning I still > dunno which way the default goes. When I do a "make defconfig" it uses > arch/x86/configs/x86_64_defconfig because the kernel has multiple > codepaths to accomplish the same thing. I'm not sure the built-in > "default y" lines are used at all anymore? What a mess...) > > But again, I'm just guessing what config you're using because you still > haven't _said_. I'm still trying to guess what you're doing when you hit > Ubuntu's bug. > > > So it means for me that (at least) users why run > > Ubuntu 14.04 will have bricked system one day after updating the > > kernel. > > Unless when they build their new kernel they open up menuconfig and > switch this symbol off. Which can't be done because...? > > Or you could add the devtmpfs.mount=0 argument to your kernel command > line, as documented in the CONFIG_DEVTMPFS_MOUNT menuconfig help text. > > The kernel already provides multiple workarounds for Ubuntu's bug, and > the issue only hits people who are manually building a new kernel from > source. If ubuntu provides a new kernel, I assume they'll tweak their > config _and_ fix their initramfs error path (which is just plain wrong). > > > If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel, > > It's not _me_ saying it, it's the kernel doing it. The patch is > conceptually a straightforward fix on the kernel side to make it _not_ > ignore that symbol in that context. > > > I think you cannot relay on it anymore because people may have it > > enabled or disabled randomly. > > I expected configs would have it randomly set, but the bug here is a > broken error path that does something actively harmful rather than going > "oh, we got a static /dev from somewhere, let's just leave it alone". > This error path goes out of its way to blank the contents of /dev by > mounting an empty tmpfs over it and leaving it empty, and then > complaining that /dev is blank _because_it_blanked_it_. > > If Ubuntu meant to intentionally halt the proceedings the script could > have done that explicitly. What did the author of that error path think > would happen, exactly? > > > So the proper way is to remove broken > > config option and introduce new one. BTW, I see it is used once in > > drivers/base/devtmpfs.c. > > How does removing the broken config option (or renaming it to > CONFIG_DEFTMPFS_UBUNTU_IS_BROKEN) _not_ impact systems that were > previously happily using it in the contexts where it already worked? > > If it's too much to ask people to switch it off when it was previously > on (but shouldn't have been), how is asking them to manually switch it > back on when it was previously on and needs to stay on better? (And if > you arrange it so "make oldconfig" migrates the old symbol to the new > one automatically, how would that work around the broken error path in > ubuntu's initramfs script? The rename becomes a NOP.) > > If you're saying it should default to "n" I can send a patch. If you > want me to tweak every
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 05/23/2017 03:01 AM, Yury Norov wrote: > On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: >> Your userspace mounted a tmpfs over /dev when it couldn't mount a second >> identical instance of devtmpfs over itself. If you had a static /dev in >> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken >> error path would have taken that out too with a pointless tmpfs mount. > > CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your > suggestion is correct. But I didn't do that specifically - I run > almost default kernel based on Ubuntu 14.04 config and environment. I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then launchd an initramfs instead (which didn't do the automount they requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually starts working in initramfs they have an insane error path that breaks the system, and does nothing _except_ break the system. > Grepping the kernel code shows that arc, arm, arm64, m86k, metag, > mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa > enable it by default. Most of which Ubuntu doesn't support, so none of them could trigger the broken error path in ubuntu's init script. Wait, are you saying you're doing a "make defconfig" on x86-64 and booting ubuntu from the result? (Or is this arm?) Is _that_ the config you still haven't specified in this conversation? I thought you were using the /boot/config-4.4.0-78-generic and friends ubuntu installs. (Which yes, also switch this symbol on.) I can add a "default n" line to drivers/base/Kconfig if "make defconfig" is what you're building from. (This symbol never specified a default in the first place, so I dunno which way it falls, but it's repeated in a gazillion defconfig files and not present in others... meaning I still dunno which way the default goes. When I do a "make defconfig" it uses arch/x86/configs/x86_64_defconfig because the kernel has multiple codepaths to accomplish the same thing. I'm not sure the built-in "default y" lines are used at all anymore? What a mess...) But again, I'm just guessing what config you're using because you still haven't _said_. I'm still trying to guess what you're doing when you hit Ubuntu's bug. > So it means for me that (at least) users why run > Ubuntu 14.04 will have bricked system one day after updating the > kernel. Unless when they build their new kernel they open up menuconfig and switch this symbol off. Which can't be done because...? Or you could add the devtmpfs.mount=0 argument to your kernel command line, as documented in the CONFIG_DEVTMPFS_MOUNT menuconfig help text. The kernel already provides multiple workarounds for Ubuntu's bug, and the issue only hits people who are manually building a new kernel from source. If ubuntu provides a new kernel, I assume they'll tweak their config _and_ fix their initramfs error path (which is just plain wrong). > If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel, It's not _me_ saying it, it's the kernel doing it. The patch is conceptually a straightforward fix on the kernel side to make it _not_ ignore that symbol in that context. > I think you cannot relay on it anymore because people may have it > enabled or disabled randomly. I expected configs would have it randomly set, but the bug here is a broken error path that does something actively harmful rather than going "oh, we got a static /dev from somewhere, let's just leave it alone". This error path goes out of its way to blank the contents of /dev by mounting an empty tmpfs over it and leaving it empty, and then complaining that /dev is blank _because_it_blanked_it_. If Ubuntu meant to intentionally halt the proceedings the script could have done that explicitly. What did the author of that error path think would happen, exactly? > So the proper way is to remove broken > config option and introduce new one. BTW, I see it is used once in > drivers/base/devtmpfs.c. How does removing the broken config option (or renaming it to CONFIG_DEFTMPFS_UBUNTU_IS_BROKEN) _not_ impact systems that were previously happily using it in the contexts where it already worked? If it's too much to ask people to switch it off when it was previously on (but shouldn't have been), how is asking them to manually switch it back on when it was previously on and needs to stay on better? (And if you arrange it so "make oldconfig" migrates the old symbol to the new one automatically, how would that work around the broken error path in ubuntu's initramfs script? The rename becomes a NOP.) If you're saying it should default to "n" I can send a patch. If you want me to tweak every arch/*/configs file that redundantly includes the same darn symbol, I can do that too. (Makes the patch big but it's just a sed invocation to do it.) >> By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_? >> That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.) > > Have no idea.
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 05/23/2017 03:01 AM, Yury Norov wrote: > On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: >> Your userspace mounted a tmpfs over /dev when it couldn't mount a second >> identical instance of devtmpfs over itself. If you had a static /dev in >> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken >> error path would have taken that out too with a pointless tmpfs mount. > > CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your > suggestion is correct. But I didn't do that specifically - I run > almost default kernel based on Ubuntu 14.04 config and environment. I.E. ubuntu has a bug: they enabled CONFIG_DEVTMPFS_MOUNT and then launchd an initramfs instead (which didn't do the automount they requested so why request it), but if CONFIG_DEVTMPFS_MOUNT actually starts working in initramfs they have an insane error path that breaks the system, and does nothing _except_ break the system. > Grepping the kernel code shows that arc, arm, arm64, m86k, metag, > mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa > enable it by default. Most of which Ubuntu doesn't support, so none of them could trigger the broken error path in ubuntu's init script. Wait, are you saying you're doing a "make defconfig" on x86-64 and booting ubuntu from the result? (Or is this arm?) Is _that_ the config you still haven't specified in this conversation? I thought you were using the /boot/config-4.4.0-78-generic and friends ubuntu installs. (Which yes, also switch this symbol on.) I can add a "default n" line to drivers/base/Kconfig if "make defconfig" is what you're building from. (This symbol never specified a default in the first place, so I dunno which way it falls, but it's repeated in a gazillion defconfig files and not present in others... meaning I still dunno which way the default goes. When I do a "make defconfig" it uses arch/x86/configs/x86_64_defconfig because the kernel has multiple codepaths to accomplish the same thing. I'm not sure the built-in "default y" lines are used at all anymore? What a mess...) But again, I'm just guessing what config you're using because you still haven't _said_. I'm still trying to guess what you're doing when you hit Ubuntu's bug. > So it means for me that (at least) users why run > Ubuntu 14.04 will have bricked system one day after updating the > kernel. Unless when they build their new kernel they open up menuconfig and switch this symbol off. Which can't be done because...? Or you could add the devtmpfs.mount=0 argument to your kernel command line, as documented in the CONFIG_DEVTMPFS_MOUNT menuconfig help text. The kernel already provides multiple workarounds for Ubuntu's bug, and the issue only hits people who are manually building a new kernel from source. If ubuntu provides a new kernel, I assume they'll tweak their config _and_ fix their initramfs error path (which is just plain wrong). > If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel, It's not _me_ saying it, it's the kernel doing it. The patch is conceptually a straightforward fix on the kernel side to make it _not_ ignore that symbol in that context. > I think you cannot relay on it anymore because people may have it > enabled or disabled randomly. I expected configs would have it randomly set, but the bug here is a broken error path that does something actively harmful rather than going "oh, we got a static /dev from somewhere, let's just leave it alone". This error path goes out of its way to blank the contents of /dev by mounting an empty tmpfs over it and leaving it empty, and then complaining that /dev is blank _because_it_blanked_it_. If Ubuntu meant to intentionally halt the proceedings the script could have done that explicitly. What did the author of that error path think would happen, exactly? > So the proper way is to remove broken > config option and introduce new one. BTW, I see it is used once in > drivers/base/devtmpfs.c. How does removing the broken config option (or renaming it to CONFIG_DEFTMPFS_UBUNTU_IS_BROKEN) _not_ impact systems that were previously happily using it in the contexts where it already worked? If it's too much to ask people to switch it off when it was previously on (but shouldn't have been), how is asking them to manually switch it back on when it was previously on and needs to stay on better? (And if you arrange it so "make oldconfig" migrates the old symbol to the new one automatically, how would that work around the broken error path in ubuntu's initramfs script? The rename becomes a NOP.) If you're saying it should default to "n" I can send a patch. If you want me to tweak every arch/*/configs file that redundantly includes the same darn symbol, I can do that too. (Makes the patch big but it's just a sed invocation to do it.) >> By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_? >> That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.) > > Have no idea.
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: > On 05/22/2017 07:05 AM, Yury Norov wrote: > > Hi Rob, > > > > I found that next-20170522 fails to boot on arm64 machine with the > > following log: > > I don't know anything about your kernel config (is CONFIG_DEVTMPFS_MOUNT > enabled or disabled?) or what userspace you're booting with, but it > seems I can guess: > > > [...] > > [4.179509] Freeing unused kernel memory: 1088K > > Loading, please wait... > > At this point, the kernel has launched init and your userspace is > running. During that boot,the kernel mounted devtmpfs on /dev (you > edited the part where it did that out of your boot log), but the next line: > > > mount: mounting udev on /dev failed: Device or resource busy > > has an error that says you already have devtmpfs mounted on /dev, and > your userspace tries to mount devtmpfs on it _again_ and it fails > because you can't mount the exact same filesystem over itself due to a > sanity check in the kernel in fs/namespace.s line 2475 or so: > > /* Refuse the same filesystem on the same mount point */ > err = -EBUSY; > if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && > path->mnt->mnt_root == path->dentry) > goto unlock; > > > W: devtmpfs not available, falling back to tmpfs for /dev > > Couldn't get a file descriptor referring to the console > > At which point your userspace does a "fixup" mounting something else > over the previously working devtmpfs, which succeeds (because you're > mounting a _different_ filesystem and not hitting the above sanity > test), thus breaking your userspace. > > > Begin: Loading essential drivers ... done. > > Begin: Running /scripts/init-premount ... done. > > Begin: Mounting root file system ... Begin: Running > > /scripts/local-top ... done. > > chvt: can't open console > > And then your userspace didn't notice for a while. > > > Gave up waiting for root device. Common problems: > > - Boot args (cat /proc/cmdline) > >- Check rootdelay= (did the system wait long enough?) > >- Check root= (did the system wait for the right device?) > > - Missing modules (cat /proc/modules; ls /dev) > > chvt: can't open console > > ALERT! /dev/sda does not exist. Dropping to a shell! > > Couldn't get a file descriptor referring to the console > > And then it died. > > > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) > > Enter 'help' for a list of built-in commands. > > > > (initramfs) > > > > Bisect points to your patch (attached below). If I revert it, everything > > becomes fine. If you need to know something more about my environment, > > feel free to ask me. > > You were inappropriately specifying CONFIG_DEVTMPFS_MOUNT in your > config, now that it's no longer being ignored your init script is having > an allergic reaction to it. Either yank it from your config or fix your > userspace. It looks to me like my patch triggered a bug in your setup. > > Your userspace mounted a tmpfs over /dev when it couldn't mount a second > identical instance of devtmpfs over itself. If you had a static /dev in > initramfs but didn't configure _in_ devtmpfs to your kernel, your broken > error path would have taken that out too with a pointless tmpfs mount. CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your suggestion is correct. But I didn't do that specifically - I run almost default kernel based on Ubuntu 14.04 config and environment. Grepping the kernel code shows that arc, arm, arm64, m86k, metag, mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa enable it by default. So it means for me that (at least) users why run Ubuntu 14.04 will have bricked system one day after updating the kernel. If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel, I think you cannot relay on it anymore because people may have it enabled or disabled randomly. So the proper way is to remove broken config option and introduce new one. BTW, I see it is used once in drivers/base/devtmpfs.c. > By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_? > That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.) Have no idea. This is how default Ubuntu works. Yury
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote: > On 05/22/2017 07:05 AM, Yury Norov wrote: > > Hi Rob, > > > > I found that next-20170522 fails to boot on arm64 machine with the > > following log: > > I don't know anything about your kernel config (is CONFIG_DEVTMPFS_MOUNT > enabled or disabled?) or what userspace you're booting with, but it > seems I can guess: > > > [...] > > [4.179509] Freeing unused kernel memory: 1088K > > Loading, please wait... > > At this point, the kernel has launched init and your userspace is > running. During that boot,the kernel mounted devtmpfs on /dev (you > edited the part where it did that out of your boot log), but the next line: > > > mount: mounting udev on /dev failed: Device or resource busy > > has an error that says you already have devtmpfs mounted on /dev, and > your userspace tries to mount devtmpfs on it _again_ and it fails > because you can't mount the exact same filesystem over itself due to a > sanity check in the kernel in fs/namespace.s line 2475 or so: > > /* Refuse the same filesystem on the same mount point */ > err = -EBUSY; > if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && > path->mnt->mnt_root == path->dentry) > goto unlock; > > > W: devtmpfs not available, falling back to tmpfs for /dev > > Couldn't get a file descriptor referring to the console > > At which point your userspace does a "fixup" mounting something else > over the previously working devtmpfs, which succeeds (because you're > mounting a _different_ filesystem and not hitting the above sanity > test), thus breaking your userspace. > > > Begin: Loading essential drivers ... done. > > Begin: Running /scripts/init-premount ... done. > > Begin: Mounting root file system ... Begin: Running > > /scripts/local-top ... done. > > chvt: can't open console > > And then your userspace didn't notice for a while. > > > Gave up waiting for root device. Common problems: > > - Boot args (cat /proc/cmdline) > >- Check rootdelay= (did the system wait long enough?) > >- Check root= (did the system wait for the right device?) > > - Missing modules (cat /proc/modules; ls /dev) > > chvt: can't open console > > ALERT! /dev/sda does not exist. Dropping to a shell! > > Couldn't get a file descriptor referring to the console > > And then it died. > > > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) > > Enter 'help' for a list of built-in commands. > > > > (initramfs) > > > > Bisect points to your patch (attached below). If I revert it, everything > > becomes fine. If you need to know something more about my environment, > > feel free to ask me. > > You were inappropriately specifying CONFIG_DEVTMPFS_MOUNT in your > config, now that it's no longer being ignored your init script is having > an allergic reaction to it. Either yank it from your config or fix your > userspace. It looks to me like my patch triggered a bug in your setup. > > Your userspace mounted a tmpfs over /dev when it couldn't mount a second > identical instance of devtmpfs over itself. If you had a static /dev in > initramfs but didn't configure _in_ devtmpfs to your kernel, your broken > error path would have taken that out too with a pointless tmpfs mount. CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your suggestion is correct. But I didn't do that specifically - I run almost default kernel based on Ubuntu 14.04 config and environment. Grepping the kernel code shows that arc, arm, arm64, m86k, metag, mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa enable it by default. So it means for me that (at least) users why run Ubuntu 14.04 will have bricked system one day after updating the kernel. If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel, I think you cannot relay on it anymore because people may have it enabled or disabled randomly. So the proper way is to remove broken config option and introduce new one. BTW, I see it is used once in drivers/base/devtmpfs.c. > By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_? > That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.) Have no idea. This is how default Ubuntu works. Yury
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 05/22/2017 07:05 AM, Yury Norov wrote: > Hi Rob, > > I found that next-20170522 fails to boot on arm64 machine with the > following log: I don't know anything about your kernel config (is CONFIG_DEVTMPFS_MOUNT enabled or disabled?) or what userspace you're booting with, but it seems I can guess: > [...] > [4.179509] Freeing unused kernel memory: 1088K > Loading, please wait... At this point, the kernel has launched init and your userspace is running. During that boot,the kernel mounted devtmpfs on /dev (you edited the part where it did that out of your boot log), but the next line: > mount: mounting udev on /dev failed: Device or resource busy has an error that says you already have devtmpfs mounted on /dev, and your userspace tries to mount devtmpfs on it _again_ and it fails because you can't mount the exact same filesystem over itself due to a sanity check in the kernel in fs/namespace.s line 2475 or so: /* Refuse the same filesystem on the same mount point */ err = -EBUSY; if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && path->mnt->mnt_root == path->dentry) goto unlock; > W: devtmpfs not available, falling back to tmpfs for /dev > Couldn't get a file descriptor referring to the console At which point your userspace does a "fixup" mounting something else over the previously working devtmpfs, which succeeds (because you're mounting a _different_ filesystem and not hitting the above sanity test), thus breaking your userspace. > Begin: Loading essential drivers ... done. > Begin: Running /scripts/init-premount ... done. > Begin: Mounting root file system ... Begin: Running > /scripts/local-top ... done. > chvt: can't open console And then your userspace didn't notice for a while. > Gave up waiting for root device. Common problems: > - Boot args (cat /proc/cmdline) >- Check rootdelay= (did the system wait long enough?) >- Check root= (did the system wait for the right device?) > - Missing modules (cat /proc/modules; ls /dev) > chvt: can't open console > ALERT! /dev/sda does not exist. Dropping to a shell! > Couldn't get a file descriptor referring to the console And then it died. > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) > Enter 'help' for a list of built-in commands. > > (initramfs) > > Bisect points to your patch (attached below). If I revert it, everything > becomes fine. If you need to know something more about my environment, > feel free to ask me. You were inappropriately specifying CONFIG_DEVTMPFS_MOUNT in your config, now that it's no longer being ignored your init script is having an allergic reaction to it. Either yank it from your config or fix your userspace. It looks to me like my patch triggered a bug in your setup. Your userspace mounted a tmpfs over /dev when it couldn't mount a second identical instance of devtmpfs over itself. If you had a static /dev in initramfs but didn't configure _in_ devtmpfs to your kernel, your broken error path would have taken that out too with a pointless tmpfs mount. By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_? That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.) Feel free to send more context if you think I'm wrong about this. > Yury Rob
Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
On 05/22/2017 07:05 AM, Yury Norov wrote: > Hi Rob, > > I found that next-20170522 fails to boot on arm64 machine with the > following log: I don't know anything about your kernel config (is CONFIG_DEVTMPFS_MOUNT enabled or disabled?) or what userspace you're booting with, but it seems I can guess: > [...] > [4.179509] Freeing unused kernel memory: 1088K > Loading, please wait... At this point, the kernel has launched init and your userspace is running. During that boot,the kernel mounted devtmpfs on /dev (you edited the part where it did that out of your boot log), but the next line: > mount: mounting udev on /dev failed: Device or resource busy has an error that says you already have devtmpfs mounted on /dev, and your userspace tries to mount devtmpfs on it _again_ and it fails because you can't mount the exact same filesystem over itself due to a sanity check in the kernel in fs/namespace.s line 2475 or so: /* Refuse the same filesystem on the same mount point */ err = -EBUSY; if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb && path->mnt->mnt_root == path->dentry) goto unlock; > W: devtmpfs not available, falling back to tmpfs for /dev > Couldn't get a file descriptor referring to the console At which point your userspace does a "fixup" mounting something else over the previously working devtmpfs, which succeeds (because you're mounting a _different_ filesystem and not hitting the above sanity test), thus breaking your userspace. > Begin: Loading essential drivers ... done. > Begin: Running /scripts/init-premount ... done. > Begin: Mounting root file system ... Begin: Running > /scripts/local-top ... done. > chvt: can't open console And then your userspace didn't notice for a while. > Gave up waiting for root device. Common problems: > - Boot args (cat /proc/cmdline) >- Check rootdelay= (did the system wait long enough?) >- Check root= (did the system wait for the right device?) > - Missing modules (cat /proc/modules; ls /dev) > chvt: can't open console > ALERT! /dev/sda does not exist. Dropping to a shell! > Couldn't get a file descriptor referring to the console And then it died. > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) > Enter 'help' for a list of built-in commands. > > (initramfs) > > Bisect points to your patch (attached below). If I revert it, everything > becomes fine. If you need to know something more about my environment, > feel free to ask me. You were inappropriately specifying CONFIG_DEVTMPFS_MOUNT in your config, now that it's no longer being ignored your init script is having an allergic reaction to it. Either yank it from your config or fix your userspace. It looks to me like my patch triggered a bug in your setup. Your userspace mounted a tmpfs over /dev when it couldn't mount a second identical instance of devtmpfs over itself. If you had a static /dev in initramfs but didn't configure _in_ devtmpfs to your kernel, your broken error path would have taken that out too with a pointless tmpfs mount. By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_? That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.) Feel free to send more context if you think I'm wrong about this. > Yury Rob
Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
Hi Rob, I found that next-20170522 fails to boot on arm64 machine with the following log: [...] [4.179509] Freeing unused kernel memory: 1088K Loading, please wait... mount: mounting udev on /dev failed: Device or resource busy W: devtmpfs not available, falling back to tmpfs for /dev Couldn't get a file descriptor referring to the console Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. chvt: can't open console Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Check root= (did the system wait for the right device?) - Missing modules (cat /proc/modules; ls /dev) chvt: can't open console ALERT! /dev/sda does not exist. Dropping to a shell! Couldn't get a file descriptor referring to the console BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) Bisect points to your patch (attached below). If I revert it, everything becomes fine. If you need to know something more about my environment, feel free to ask me. Yury On Thu, May 11, 2017 at 12:05:03PM -0500, Rob Landley wrote: > From: Rob Landley> > Make initramfs honor CONFIG_DEVTMPFS_MOUNT, move /dev/console > open after devtmpfs mount, and update help text. > > Signed-off-by: Rob Landley > --- > > drivers/base/Kconfig | 14 -- > init/main.c | 15 +-- > 2 files changed, 13 insertions(+), 16 deletions(-) > > diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig > index d718ae4..74779ee 100644 > --- a/drivers/base/Kconfig > +++ b/drivers/base/Kconfig > @@ -48,16 +48,10 @@ config DEVTMPFS_MOUNT > bool "Automount devtmpfs at /dev, after the kernel mounted the rootfs" > depends on DEVTMPFS > help > - This will instruct the kernel to automatically mount the > - devtmpfs filesystem at /dev, directly after the kernel has > - mounted the root filesystem. The behavior can be overridden > - with the commandline parameter: devtmpfs.mount=0|1. > - This option does not affect initramfs based booting, here > - the devtmpfs filesystem always needs to be mounted manually > - after the rootfs is mounted. > - With this option enabled, it allows to bring up a system in > - rescue mode with init=/bin/sh, even when the /dev directory > - on the rootfs is completely empty. > + Automatically mount devtmpfs at /dev on the root filesystem, which > + lets the system come up in rescue mode with [rd]init=/bin/sh. > + Override with devtmpfs.mount=0 on the commandline. Initramfs can > + create a /dev dir as needed, other rootfs needs the mount point. > > config STANDALONE > bool "Select only drivers that don't need compile-time external > firmware" > diff --git a/init/main.c b/init/main.c > index f866510..9ec09ff 100644 > --- a/init/main.c > +++ b/init/main.c > @@ -1038,12 +1038,6 @@ static noinline void __init kernel_init_freeable(void) > > do_basic_setup(); > > - /* Open the /dev/console on the rootfs, this should never fail */ > - if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0) > - pr_err("Warning: unable to open an initial console.\n"); > - > - (void) sys_dup(0); > - (void) sys_dup(0); > /* >* check if there is an early userspace init. If yes, let it do all >* the work > @@ -1055,8 +1049,17 @@ static noinline void __init kernel_init_freeable(void) > if (sys_access((const char __user *) ramdisk_execute_command, 0) != 0) { > ramdisk_execute_command = NULL; > prepare_namespace(); > + } else if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)) { > + sys_mkdir("/dev", 0755); > + devtmpfs_mount("/dev"); > } > > + /* Open the /dev/console on the rootfs, this should never fail */ > + if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0) > + pr_err("Warning: unable to open an initial console.\n"); > + (void) sys_dup(0); > + (void) sys_dup(0); > + > /* >* Ok, we have completed the initial bootup, and >* we're essentially up and running. Get rid of the
Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot
Hi Rob, I found that next-20170522 fails to boot on arm64 machine with the following log: [...] [4.179509] Freeing unused kernel memory: 1088K Loading, please wait... mount: mounting udev on /dev failed: Device or resource busy W: devtmpfs not available, falling back to tmpfs for /dev Couldn't get a file descriptor referring to the console Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. chvt: can't open console Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Check root= (did the system wait for the right device?) - Missing modules (cat /proc/modules; ls /dev) chvt: can't open console ALERT! /dev/sda does not exist. Dropping to a shell! Couldn't get a file descriptor referring to the console BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) Bisect points to your patch (attached below). If I revert it, everything becomes fine. If you need to know something more about my environment, feel free to ask me. Yury On Thu, May 11, 2017 at 12:05:03PM -0500, Rob Landley wrote: > From: Rob Landley > > Make initramfs honor CONFIG_DEVTMPFS_MOUNT, move /dev/console > open after devtmpfs mount, and update help text. > > Signed-off-by: Rob Landley > --- > > drivers/base/Kconfig | 14 -- > init/main.c | 15 +-- > 2 files changed, 13 insertions(+), 16 deletions(-) > > diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig > index d718ae4..74779ee 100644 > --- a/drivers/base/Kconfig > +++ b/drivers/base/Kconfig > @@ -48,16 +48,10 @@ config DEVTMPFS_MOUNT > bool "Automount devtmpfs at /dev, after the kernel mounted the rootfs" > depends on DEVTMPFS > help > - This will instruct the kernel to automatically mount the > - devtmpfs filesystem at /dev, directly after the kernel has > - mounted the root filesystem. The behavior can be overridden > - with the commandline parameter: devtmpfs.mount=0|1. > - This option does not affect initramfs based booting, here > - the devtmpfs filesystem always needs to be mounted manually > - after the rootfs is mounted. > - With this option enabled, it allows to bring up a system in > - rescue mode with init=/bin/sh, even when the /dev directory > - on the rootfs is completely empty. > + Automatically mount devtmpfs at /dev on the root filesystem, which > + lets the system come up in rescue mode with [rd]init=/bin/sh. > + Override with devtmpfs.mount=0 on the commandline. Initramfs can > + create a /dev dir as needed, other rootfs needs the mount point. > > config STANDALONE > bool "Select only drivers that don't need compile-time external > firmware" > diff --git a/init/main.c b/init/main.c > index f866510..9ec09ff 100644 > --- a/init/main.c > +++ b/init/main.c > @@ -1038,12 +1038,6 @@ static noinline void __init kernel_init_freeable(void) > > do_basic_setup(); > > - /* Open the /dev/console on the rootfs, this should never fail */ > - if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0) > - pr_err("Warning: unable to open an initial console.\n"); > - > - (void) sys_dup(0); > - (void) sys_dup(0); > /* >* check if there is an early userspace init. If yes, let it do all >* the work > @@ -1055,8 +1049,17 @@ static noinline void __init kernel_init_freeable(void) > if (sys_access((const char __user *) ramdisk_execute_command, 0) != 0) { > ramdisk_execute_command = NULL; > prepare_namespace(); > + } else if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)) { > + sys_mkdir("/dev", 0755); > + devtmpfs_mount("/dev"); > } > > + /* Open the /dev/console on the rootfs, this should never fail */ > + if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0) > + pr_err("Warning: unable to open an initial console.\n"); > + (void) sys_dup(0); > + (void) sys_dup(0); > + > /* >* Ok, we have completed the initial bootup, and >* we're essentially up and running. Get rid of the