Re: [Bug 818177] Re: boot failures caused by udev race
On Thu, Oct 06, 2011 at 01:18:27AM -, Serge Hallyn wrote: see comment #42 for one disk layout that reproduces it for me in a kvm VM. This describes a physical disk layout... it doesn't describe a partition or LV layout, which is what is key to reproducing the LVM-related hang. Is this VM using LVM? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures caused by udev race
On Thu, Oct 06, 2011 at 01:20:03AM -, Serge Hallyn wrote: also see bug 833891 as a udev bug specifically for the LVM case. Do you mean you're already tracking the LVM case on that bug instead, and that in the setup you're using there is *no* use of LVM? I guess no one told Adam that this is a different bug, because it was his LVM-based setup that I was responding to... :) In fact, Eduard toward the end speculated precisely the semaphore as a cause as you just did. That was Eduard's speculation, copied from the *other* bug I linked to. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Adam/@Serge: could you try the following to see if it solves / reduces the occurrence of the problem. Also, can you report back if you see any interesting processes as logged by the change below: 1) Change initramfs to bind mount /dev and dump devices and running processes before udevd exits: $ sudo sed -i.bak -e '/^udevadm control --exit/i echo XXX: ps=`ps -ef`, \/dev=`ls -l \/dev` /dev/kmsg' -e 's!^mount -n -o move /dev!mount -n -o bind /dev!g' /usr/share/initramfs-tools/scripts/init-bottom/udev $ sudo mkinitramfs -o /boot/initramfs.foo 2) Reboot setting initrd /boot/initramfs.foo in grub. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Steve, (re comment #52) I'm not sure what you were asking for then, but as I said in that comment, partition 1 is a simple ext3 filesystem. Partition 2 is just an extended. Partition 5 (the only one on the extended) is swap. There is no LVM. With standard (non-instrumented) udev it hangs roughly every 5 boots. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures caused by udev race
Quoting Steve Langasek (steve.langa...@canonical.com): On Thu, Oct 06, 2011 at 01:20:03AM -, Serge Hallyn wrote: also see bug 833891 as a udev bug specifically for the LVM case. Do you mean you're already tracking the LVM case on that bug instead, and I was, yes. I filed that one some time ago to track the hang on my own laptop. I realize there are too many open bugs on this, and we should mark some as dupes. that in the setup you're using there is *no* use of LVM? I guess no one Yes. told Adam that this is a different bug, because it was his LVM-based setup that I was responding to... :) Right. In fact, Eduard toward the end speculated precisely the semaphore as a cause as you just did. That was Eduard's speculation, copied from the *other* bug I linked to. Right. NOTE though: The LVM authors have admonished us for calling vgscan/vgchange in a udev rule, as that has its own deadlock potential. My debdiff in bug 833891 is meant to address that. It sounds like that deadlock is separate from the speculation above. Although perhaps not - such a deadlock could explain why a semaphore in udevd is not getting Vd (since udevd never finishes its processing of that rule, but just gets killed on timeout)? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures caused by udev race
Quoting James Hunt (818...@bugs.launchpad.net): @Adam/@Serge: could you try the following to see if it solves / reduces the occurrence of the problem. It certainly didn't solve it, hung on first try :) I'm not sure what you would deem interesting, but here is the output from the script as obtained from dmesg. ... [0.651771] udevd[79]: starting version 173 [0.812351] FDC 0 is a S82078B [0.815764] EXT3-fs: barriers not enabled [0.818981] kjournald starting. Commit interval 5 seconds [0.819002] EXT3-fs (vda1): mounted filesystem with ordered data mode [0.839767] XXX: ps= PID USER VSZ STAT COMMAND [0.839768] 1 0 4456 S/bin/sh /init ro quiet [0.839769] 2 00 SW [kthreadd] [0.839770] 3 00 SW [ksoftirqd/0] [0.839771] 4 00 SW [kworker/0:0] [0.839771] 5 00 SW [kworker/u:0] [0.839772] 6 00 SW [migration/0] [0.839773] 7 00 SW [cpuset] [0.839773] 8 00 SW [khelper] [0.839774] 9 00 SW [netns] [0.839774]10 00 SW [sync_supers] [0.839775]11 00 SW [bdi-default] [0.839776]12 00 SW [kintegrityd] [0.839776]13 00 SW [kblockd] [0.839777]14 00 SW [ata_sff] [0.839778]15 00 SW [khubd] [0.839778]16 00 SW [md] [0.839779]17 00 SW [kworker/0:1] [0.839780]18 00 SW [khungtaskd] [0.839780]19 00 SW [kworker/u:1] [0.839781]20 00 SW [kswapd0] [0.839781]21 00 SWN [ksmd] [0.839782]22 00 SW [fsnotify_mark] [0.839783]23 00 SW [ecryptfs-kthrea] [0.839783]24 00 SW [crypto] [0.839784]32 00 SW [kthrotld] [0.839785]33 00 SW [aemon --resolve-names=never [0.839807]96 012752 S/sbin/udevd --daemon --resolve-names=never [0.839808]97 012752 S/sbin/udevd --daemon --resolve-names=never [0.839808]98 012752 S/sbin/udevd --daemon --resolve-names=never [0.839809]99 012752 S/sbin/udevd --daemon --resolve-names=never [0.839810] 100 012752 S/sbin/udevd --daemon --resolve-names=never [0.839811] 101 012752 S/sbin/udevd --daemon --resolve-names=never [0.839812] 102 012752 S/sbin/udevd --daemon --resolve-names=never [0.839813] 103 012752 S/sbin/udevd --daemon --resolve-names=never [0.839816] 104 012752 S/sbin/udevd --daemon --resolve-names=never [0.839817] 105 012752 S/sbin/udevd --daemon --resolve-names=never [0.839817] 106 012752 S/sbin/udevd --daemon --resolve-names=never [0.839818] 107 012752 S/sbin/udevd --daemon --resolve-names=never [0.839819] 108 012752 S/sbin/udevd --daemon --resolve-names=never [0.839820] 109 012752 S/sbin/udevd --daemon --resolve-names=never [0.839821] 110 012752 S/sb /sbin/udevd --daemon --resolve-names=never [0.839833] 115 012752 S/sbin/udevd --daemon --resolve-names=never [0.839834] 116 012752 S/sbin/udevd --daemon --resolve-names=never [0.839834] 120 012752 S/sbin/udevd --daemon --resolve-names=never [0.839835] 121 012884 S/sbin/udevd --daemon --resolve-names=never [0.839836] 122 012884 S/sbin/udevd --daemon --resolve-names=never [0.839837] 123 012884 S/sbin/udevd --daemon --resolve-names=never [0.839838] 124 012884 S/sbin/udevd --daemon --resolve-names=never [0.839839] 125 012884 S/sbin/udevd --daemon --resolve-names=never [0.839840] 126 012884 S/sbin/udevd --daemon --resolve-names=never [0.839840] 127 012884 S/sbin/udevd --daemon --resolve-names=never [0.839841] 128 012884 S/sbin/udevd --daemon --resolve-names=never [0.839842] 129 012884 S/sbin/udevd --daemon --resolve-names=never [0.839843] 130 012884 S/sbin/udevd --daemon --resolve-names=never [0.839844] 131 012884 S/sbin/udevd --daemon --resolve-names=never [0.839845] 132 0 012984 S/sbin/udevd --daemon --resolve-names=never [0.839854] 175 012988 S/sbin/udevd --daemon --resolve-names=never [0.839855] 176 012968 S/sbin/udevd --daemon --resolve-names=never [0.839856] 177 012964 S/sbin/udevd --daemon --resolve-names=never [0.839857] 178 012976 S/sbin/udevd --daemon --resolve-names=never [0.839858] 179 012796 S/sbin/udevd --daemon
[Bug 818177] Re: boot failures caused by udev race
Serge, I'm not sure what you were asking for then, but as I said in that comment, partition 1 is a simple ext3 filesystem. Oh, doh - I read the wrong comment. Sorry. :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
Ok. I have a machine here which triggers somethign similar to this pretty often. For me udev finds /dev is read-only and halts boot. This is presumably because our devtmpfs /dev has not made it into / when it starts. I was getting a failed boot about every 5-6 boots, 4 in 20 overall. I then used the work around in comment #54, with that in place I have 35 consecutive successful boots. The conjecture is that /dev in the initramfs is busy and cannot be moved when needed. From the ps output in comment #57 it seems udev has not actually quit when requested, which might well account for this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
After modifying initramfs similar to https://bugs.launchpad.net/ubuntu/oneiric/+source/udev/+bug/833783/comments/17 , ive managed to hit the bug. Attached is output from a failed boot and successful boot. ** Attachment added: 'udevadm monitor -e' log of a failed boot https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+attachment/2514534/+files/udev.failed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
** Attachment added: 'udevadm monitor -e log of a successful boot https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+attachment/2514537/+files/udev.success -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
analysis of the two logs shows the following events missing from udev in the failed case. UDEV [4.229209] add /devices/pci:00/:00:01.1/host2/target2:0:1/2:0:1:0/block/sdc (block) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/host2/target2:0:1/2:0:1:0/block/sdc SUBSYSTEM=block DEVNAME=/dev/sdc DEVTYPE=disk SEQNUM=770 ID_ATA=1 ID_TYPE=disk ID_BUS=ata ID_MODEL=QEMU_HARDDISK ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20 ID_REVISION=0.14.0 ID_SERIAL=QEMU_HARDDISK_QM4 ID_SERIAL_SHORT=QM4 ID_ATA_WRITE_CACHE=1 ID_ATA_WRITE_CACHE_ENABLED=0 ID_ATA_FEATURE_SET_SMART=1 ID_ATA_FEATURE_SET_SMART_ENABLED=1 ID_SCSI_COMPAT=SATA_QEMU_HARDDISK_QM4 ID_PATH=pci-:00:01.1-scsi-1:0:1:0 ID_PATH_TAG=pci-_00_01_1-scsi-1_0_1_0 ID_FS_UUID=4qMaMd-jD2v-vzpj-ScTe-Rsz7-CDw0-3DCuLn ID_FS_UUID_ENC=4qMaMd-jD2v-vzpj-ScTe-Rsz7-CDw0-3DCuLn ID_FS_VERSION=LVM2\x20001 ID_FS_TYPE=LVM2_member ID_FS_USAGE=raid MAJOR=8 MINOR=32 DEVLINKS=/dev/disk/by-id/ata-QEMU_HARDDISK_QM4 /dev/disk/by-id/scsi-SATA_QEMU_HARDDISK_QM4 /dev/disk/by-path/pci-:00:01.1-scsi-1:0:1:0 UDEV [4.229522] add /devices/pci:00/:00:01.1 (pci) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1 SUBSYSTEM=pci DRIVER=ata_piix PCI_CLASS=10180 PCI_ID=8086:7010 PCI_SUBSYS_ID=1AF4:1100 PCI_SLOT_NAME=:00:01.1 MODALIAS=pci:v8086d7010sv1AF4sd1100bc01sc01i80 SEQNUM=851 UDEV [4.230831] add /devices/pci:00/:00:01.1/ata1/ata_port/ata1 (ata_port) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata1/ata_port/ata1 SUBSYSTEM=ata_port SEQNUM=852 UDEV [4.230937] add /devices/pci:00/:00:01.1/ata1/link1/ata_link/link1 (ata_link) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata1/link1/ata_link/link1 SUBSYSTEM=ata_link SEQNUM=853 UDEV [4.229854] add /devices/pci:00/:00:01.1/ata1/link1/dev1.0/ata_device/dev1.0 (ata_device) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata1/link1/dev1.0/ata_device/dev1.0 SUBSYSTEM=ata_device SEQNUM=854 UDEV [4.229959] add /devices/pci:00/:00:01.1/ata1/link1/dev1.1/ata_device/dev1.1 (ata_device) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata1/link1/dev1.1/ata_device/dev1.1 SUBSYSTEM=ata_device SEQNUM=855 UDEV [4.230742] add /devices/pci:00/:00:01.1/ata2/ata_port/ata2 (ata_port) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata2/ata_port/ata2 SUBSYSTEM=ata_port SEQNUM=856 UDEV [4.230209] add /devices/pci:00/:00:01.1/ata2/link2/ata_link/l ink2 (ata_link) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata2/link2/ata_link/link2 SUBSYSTEM=ata_link SEQNUM=857 UDEV [4.230318] add /devices/pci:00/:00:01.1/ata2/link2/dev2.0/ata _device/dev2.0 (ata_device) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata2/link2/dev2.0/ata_device/dev2.0 SUBSYSTEM=ata_device SEQNUM=858 UDEV [4.230428] add /devices/pci:00/:00:01.1/ata2/link2/dev2.1/ata_device/dev2.1 (ata_device) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/ata2/link2/dev2.1/ata_device/dev2.1 SUBSYSTEM=ata_device SEQNUM=859 UDEV [4.230531] add /devices/pci:00/:00:01.1/host1 (scsi) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/host1 SUBSYSTEM=scsi DEVTYPE=scsi_host SEQNUM=860 UDEV [4.230638] add /devices/pci:00/:00:01.1/host2 (scsi) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/host2 SUBSYSTEM=scsi DEVTYPE=scsi_host SEQNUM=868 UDEV [4.231230] add /devices/pci:00/:00:01.1/host2/scsi_host/host2 (scsi_host) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/host2/scsi_host/host2 SUBSYSTEM=scsi_host SEQNUM=869 UDEV [4.231140] add /devices/pci:00/:00:01.1/host2/target2:0:1 (scsi) UDEV_LOG=3 ACTION=add DEVPATH=/devices/pci:00/:00:01.1/host2/target2:0:1 SUBSYSTEM=scsi DEVTYPE=scsi_target SEQNUM=870 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
The corresponding kernel events *are* present, so this seems to be definitively a udev bug and not a kernel bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
The last udev event shown in the failure case is this one: UDEV [3.735566] change /devices/virtual/block/dm-0 (block) UDEV_LOG=3 ACTION=change DEVPATH=/devices/virtual/block/dm-0 SUBSYSTEM=block DM_COOKIE=4228816 DEVNAME=/dev/dm-0 DEVTYPE=disk SEQNUM=1092 DM_UDEV_PRIMARY_SOURCE_FLAG=1 DM_NAME=test--vg-test--lv DM_UUID=LVM-4iWtHXttRCwYKe6cxdfiFUSSU9pFQ9Zelj82pgIWg6GreSTESCmwv43yf6Nrf5s2 DM_SUSPENDED=0 DM_UDEV_RULES=1 DM_VG_NAME=test-vg DM_LV_NAME=test-lv DEVLINKS=/dev/mapper/test--vg-test--lv /dev/disk/by-id/dm-name-test--vg-test--lv /dev/disk/by-id/dm-uuid-LVM-4iWtHXttRCwYKe6cxdfiFUSSU9pFQ9Zelj82pgIWg6GreSTESCmwv43yf6Nrf5s2 /dev/test-vg/test-lv MAJOR=252 MINOR=0 So that points to an lvm problem in this case, at least. I also notice a difference in this event - DM_SUSPENDED=0 vs. =1: UDEV [3.709786] add /devices/virtual/block/dm-0 (block) UDEV_LOG=3 ACTION=add DEVPATH=/devices/virtual/block/dm-0 SUBSYSTEM=block DEVNAME=/dev/dm-0 DEVTYPE=disk SEQNUM=1091 DM_SUSPENDED=0 DM_UDEV_RULES=1 MAJOR=252 MINOR=0 DEVLINKS=/dev/disk/by-id/dm-name- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
** Changed in: linux (Ubuntu Oneiric) Status: Incomplete = Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
Looking at bugs on lvm2 turns up this gem in bug #802626: Just a wild speculation, because I haven't yet digged into the interactions between kernel and udevd, but the semaphore decrementation event might be lost when transitioning from the initrd-udevd to the rootfs-udevd. In cases where people are seeing initrd problems and lvm is in use, are there always LVM VGs *other* than for the root fs? If there's a VG that's not a dependency of the rootfs, then indeed the vgchange call can be racing the main initramfs sequence. Twiddling the lvm2 package to insert a delay may help with reproducing the issue. None of this explains any issues with device nodes being absent from /dev on the root filesystem when lvm is *not* in use. The original bug report did not mention LVM at all. Can someone comment on the disk configuration used there? Maybe we need to split this into two bugs. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Steve, see comment #42 for one disk layout that reproduces it for me in a kvm VM. (the vm was created with 'vm-new oneiric amd64 clean' - well, technically with a customized vm-new using the mini iso which hasn't yet been merged into ubuntu-qa-tools) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Steve, also see bug 833891 as a udev bug specifically for the LVM case. In fact, Eduard toward the end speculated precisely the semaphore as a cause as you just did. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@JamesHunt, you mention LVs in comment #34, but assuming this is the same bug causing my hangs and read-only rootfs on VMs, it does not require LVs. Unfortunately, like you, whenever I've instrumented grub to print out the list of pending events, I can't reproduce it :) Perhaps kgdb is the way to go here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@JamesHunt, have you pursued trying to reproduce with a version of udevd which continues to process events when udev_exit==1? I'm unclear as to whether (1) we need to continue to process inotify events as well (so that udev workers don't get hung), and (2) whether that just means that kobject_uevent messages from the kernel will still get lost between the exit of the initramfs' udevd and the rootfs one. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@James: Im currently at a conference with limited wifi, but I checked yesterday and I can consistently reproduce on my thinkpad + kvm. Any chance you can publish those modded udev packages to a branch or PPA? I'm happy to test and see if I can get anything useful -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Adam: Not quite yet, but will work on that when I get a chance tomorrow if possible. I'm intrigued by your setup though as I have a thinkpad + kvm but only see the problem very infrequently (and I've tried setting up images as you specify). How exactly are you invoking kvm? To clarify, are you using the server image? @Serge: what was your storage configuration when you saw the problem without LVs? I think I've seen the problem a couple of times simply by providing 2 extra raw disks to the system but at that point my udevd debug wasn't helpful. I have tried modifying udevd to continue processing events but it then hangs. I really need to be able to recreate the problem more reliably on a system I've got control over before fathoming this one out. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
Hi James- I'm using the server image, kvm+libvirt for the VM, here is the corresponding XML config for the VM. http://paste.ubuntu.com/702405/ Should note that the root disk is a qcow2 image, the two additional images are raw dd'd images, each 100MB. I've also been sure to provide 2 CPUs to the VM, tho I'm not certain this affects reproducibility. The corresponding KVM process that gets spawned looks like this: http://paste.ubuntu.com/702411 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures caused by udev race
Quoting James Hunt (818...@bugs.launchpad.net): @Serge: what was your storage configuration when you saw the problem without LVs? I think I've seen the problem a couple of times simply by providing 2 extra raw disks to the system but at that point my udevd debug wasn't helpful. I have tried modifying udevd to continue processing events but it then hangs. I really need to be able to recreate the problem more reliably on a system I've got control over before fathoming this one out. I have simple vms, created with the vm-new from lp:ubuntu-qa-tools, whose root disk looks like: Device Boot Start End Blocks Id System /dev/vda1 *204815728639 7863296 83 Linux /dev/vda21573068616775167 5222415 Extended /dev/vda51573068816775167 522240 82 Linux swap / Solaris with vda1 being ext3. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures caused by udev race
Quoting James Hunt (818...@bugs.launchpad.net): @Serge: what was your storage configuration when you saw the problem without LVs? I think I've seen the problem a couple of times simply by providing 2 extra raw disks to the system but at that point my udevd debug wasn't helpful. I have tried modifying udevd to continue processing events but it then hangs. I really need to be able to recreate the problem more reliably on a system I've got control over before fathoming this one out. Here is the xml for that domain, for what it's worth. disk0.img is raw format: serge@sergelap:~/vms/kvm/c$ qemu-img info disk0.img image: disk0.img file format: raw virtual size: 8.0G (8589934592 bytes) disk size: 3.3G serge@sergelap:~/vms/kvm/c$ virsh dumpxml c domain type='kvm' id='2' namec/name uuid15405141-d078-c266-ae6f-8ca6b72a08e8/uuid memory524288/memory currentMemory524288/currentMemory vcpu1/vcpu os type arch='x86_64' machine='pc-0.14'hvm/type boot dev='hd'/ /os features acpi/ apic/ pae/ /features clock offset='utc'/ on_poweroffdestroy/on_poweroff on_rebootrestart/on_reboot on_crashrestart/on_crash devices emulator/usr/bin/kvm/emulator disk type='file' device='disk' driver name='qemu' type='raw'/ source file='/home/serge/vms/kvm/c/disk0.img'/ target dev='vda' bus='virtio'/ alias name='virtio-disk0'/ address type='pci' domain='0x' bus='0x00' slot='0x04' function='0x0'/ /disk disk type='block' device='cdrom' driver name='qemu' type='raw'/ target dev='hdc' bus='ide'/ readonly/ alias name='ide0-1-0'/ address type='drive' controller='0' bus='1' unit='0'/ /disk controller type='ide' index='0' alias name='ide0'/ address type='pci' domain='0x' bus='0x00' slot='0x01' function='0x1'/ /controller interface type='network' mac address='00:16:3e:26:26:74'/ source network='default'/ target dev='vnet0'/ model type='virtio'/ alias name='net0'/ address type='pci' domain='0x' bus='0x00' slot='0x03' function='0x0'/ /interface serial type='pty' source path='/dev/pts/4'/ target port='0'/ alias name='serial0'/ /serial console type='pty' tty='/dev/pts/4' source path='/dev/pts/4'/ target type='serial' port='0'/ alias name='serial0'/ /console input type='mouse' bus='ps2'/ graphics type='vnc' port='5900' autoport='yes'/ video model type='vmvga' vram='9216' heads='1'/ alias name='video0'/ address type='pci' domain='0x' bus='0x00' slot='0x02' function='0x0'/ /video memballoon model='virtio' alias name='balloon0'/ address type='pci' domain='0x' bus='0x00' slot='0x05' function='0x0'/ /memballoon /devices seclabel type='dynamic' model='apparmor' labellibvirt-15405141-d078-c266-ae6f-8ca6b72a08e8/label imagelabellibvirt-15405141-d078-c266-ae6f-8ca6b72a08e8/imagelabel /seclabel /domain The host is a core i7 laptop with 8G ram and showing 8 cpus. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Dave: I've modded udevd to display some internal details, but cannot now make the images I have fail to boot reliably. Currently working with @jamespage who has a machine that fails to boot most times. I've tried to force more frequent failures by installing with lots of LVs, but that doesn't seem to help either on virtual h/w (kvm) or the limited physical h/w I have access to. I've even gone as far as forcing particular LVs to respond *extremely* slowly (multi-second response time), but that hasn't helped trigger the problem either). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
BTW - I've also looked at how Fedora 15 stops udev with dracut and using Stefans terminology, they club it to death like we used to. They also pepper the code with frequent calls to settle and add a few sub-second sleeps here and there which feels horribly wrong IMHO. I have an off-beat idea as to how we could establish if messages are getting lost which I may be forced to pursue unless I can overcome the Heisenberg nature of debugging udevd. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@Adam: from comment #18, do you still have an image that fails to boot 1 in 5 times? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
@James, How did you get on with the debug version of udevd? When we tried to do the same, we were unable to reproduce the bug, as the debug statements seemed to slow down udev, hiding the race. Was this the same behaviour you encountered? Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
Andrew, I'm not sure you're experiencing the same issue; I would say in fact that you have some unrelated kernel bug, since there's no excuse for it taking 2 minutes to settle the kernel event queue. Getting a dump of 'udevadm monitor -e' from this initramfs (which would need to be started *before* this point to capture the actual events) should be instructive. Could you please open a new bug report for this and link it back here? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
It looks like udevd.c is rather aggressive when handling the exit scenario. I'm currently building a debug version of udevd + initramfs to try and see if and how messages are getting lost. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
** Summary changed: - HP DL380G5 root disk mounted read-only on boot and boot fails + boot failures caused by udev race -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures caused by udev race
I have a few machines throwing a kernel panic while netbooting oneiric with a similar error message. I have attached the boot message log with the panic. ** Attachment added: udev-race.log https://bugs.launchpad.net/ubuntu/+source/udev/+bug/818177/+attachment/2471068/+files/udev-race.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures caused by udev race To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
