[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
I can confirm that this is still an issue in Ubuntu 14.04.4. I ran into this problem trying to backup a server hosting lxc containers, some of them using overlayfs as the root fs. When the containers are running (and therefore overlayfs is mounted) lvcreate -s hangs forever. The debug output is: root@host:~# lvcreate -n foobar -L 5G -s -d -v /dev/vg0/var Setting logging type to disk Setting chunksize to 8 sectors. Finding volume group "vg0" Archiving volume group "vg0" metadata (seqno 5091). Creating logical volume foobar Creating volume group backup "/etc/lvm/backup/vg0" (seqno 5092). Found volume group "vg0" activation/volume_list configuration setting not defined: Checking only host tags for vg0/foobar Creating vg0-foobar Loading vg0-foobar table (252:5) Resuming vg0-foobar (252:5) Clearing start of logical volume "foobar" Creating logical volume snapshot0 Found volume group "vg0" Found volume group "vg0" Executing: /sbin/modprobe dm-snapshot Creating vg0-var-real Loading vg0-var-real table (252:6) Loading vg0-var table (252:3) Creating vg0-foobar-cow Loading vg0-foobar-cow table (252:7) Resuming vg0-foobar-cow (252:7) Loading vg0-foobar table (252:5) Suspending vg0-var (252:3) with filesystem sync with device flush When the containers are stopped (and therefore overlayfs is no longer mounted), the snapshot is created without issues. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
I did a little more testing. root@virt:~# uname -r 3.13.0-32-generic Actually the hanging seams to happen only when the overlayfs parts are on the logical volume being the snapshot source. Let's say /opt is a logical volume in the volume group virt. Case 1 (hanging): mkdir /opt/test/ cd /opt/test/ mkdir upper lower file_system touch lower/bla.dat mount -t overlayfs -o lowerdir=/opt/test/lower,upperdir=/opt/test/upper overlayfs /opt/test/file_system lvcreate -s -L 5G /dev/virt/opt -n vm_backup -> Hang (see previous posts) Case 2 (not hanging): mkdir /opt/test/ lvcreate -n test -L10G virt mkfs.xfs /dev/virt/test mount /dev/virt/test /opt/test cd /opt/test/ mkdir upper lower file_system touch lower/bla.dat mount -t overlayfs -o lowerdir=/opt/test/lower,upperdir=/opt/test/upper overlayfs /opt/test/file_system lvcreate -s -L 5G /dev/virt/opt -n vm_backup Snapshot created even through the overlayfs is in use (but on a different volume). So the work around for the moment is to use two types of logical volumes : 1 one type for use with overlayfs where one has to make sure that no one ever takes a snapshot of (at least until this bug is fixed.) 2. one type where overlayfs never gets used but where snapshots can be taken from. May be this helps someone running into this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
Update: I tried to find a work around by using aufs. To make a long story short: There seams to be the same issue. mkdir /tmp/dir1 /tmp/aufs-root mount -t aufs -o br=/tmp/dir1:/home/test,xino=/dev/shm/aufs.xino none /tmp/aufs-root lvcreate -s -L 5G /dev/virt/opt -n vm_backup -> Hang dmesg [1213440.776036] INFO: task ypbind:9957 blocked for more than 120 seconds. [1213440.776141] Not tainted 3.13.0-29-generic #53-Ubuntu [1213440.776233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1213440.776351] ypbind D 88042fc14440 0 9957 8265 0x2002 [1213440.776357] 880414fe9dd8 0002 880414c547d0 880414fe9fd8 [1213440.776362] 00014440 00014440 880414c547d0 880413016000 [1213440.776366] 0001 0001 8804130162b0 [1213440.776370] Call Trace: [1213440.776380] [] schedule+0x29/0x70 [1213440.776386] [] __sb_start_write+0x93/0xe0 [1213440.776393] [] ? prepare_to_wait_event+0x100/0x100 [1213440.776397] [] compat_do_readv_writev+0x20e/0x260 [1213440.776452] [] ? xfs_file_buffered_aio_write+0x1a0/0x1a0 [xfs] [1213440.776455] [] ? do_sync_read+0x90/0x90 [1213440.776461] [] ? acct_account_cputime+0x1c/0x20 [1213440.776465] [] ? account_user_time+0x8b/0xa0 [1213440.776468] [] ? vtime_account_user+0x54/0x60 [1213440.776472] [] compat_writev+0x37/0x70 [1213440.776476] [] compat_SyS_writev+0x49/0xa0 [1213440.776480] [] sysenter_dispatch+0x7/0x21 [1213440.776497] INFO: task lvcreate:28589 blocked for more than 120 seconds. [1213440.776603] Not tainted 3.13.0-29-generic #53-Ubuntu [1213440.776695] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1213440.776811] lvcreateD 88042fcd4440 0 28589 24487 0x [1213440.776815] 8802ce4c1bc8 0002 8802ce7b 8802ce4c1fd8 [1213440.776819] 00014440 00014440 8802ce7b 880413016290 [1213440.776823] 8802ce4c1c00 8802ce7b 880413016218 0001 [1213440.776827] Call Trace: [1213440.776831] [] schedule+0x29/0x70 [1213440.776835] [] sb_wait_write+0x9d/0xb0 [1213440.776839] [] ? prepare_to_wait_event+0x100/0x100 [1213440.776842] [] freeze_super+0x68/0x130 [1213440.776847] [] freeze_bdev+0x75/0xd0 [1213440.776853] [] dm_suspend+0x11a/0x1e0 [1213440.776857] [] dev_suspend+0x194/0x220 [1213440.776861] [] ? table_load+0x350/0x350 [1213440.776864] [] ctl_ioctl+0x255/0x500 [1213440.776868] [] dm_ctl_ioctl+0x13/0x20 [1213440.776873] [] do_vfs_ioctl+0x2e0/0x4c0 [1213440.776877] [] ? vtime_account_user+0x54/0x60 [1213440.776880] [] SyS_ioctl+0x81/0xa0 [1213440.776885] [] tracesys+0xe1/0xe6 I wonder how no one else can suffer from this problem. It should not be to uncommon to use LVM snapshots for backup purposes ... Greetings, Dirk -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
When I do the steps: sudo apt-get update sudo apt-get dist-upgrade I do not get a new kernel. Anyway, the Kernel version on the system seams to be more current then the one mentioned in the post above: df@feretti-virt:~$ uname -r 3.13.0-29-generic So either I am missing something or the bug does still exist. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
Thank you for taking the time to file a bug report on this issue. However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request. With the recent release of this Ubuntu release, would like to confirm if this bug is still present. Please test again with the newer kernel and indicate in the bug if this issue still exists or not. You can update to the latest development kernel by simply running the following commands in a terminal window: sudo apt-get update sudo apt-get dist-upgrade If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released. If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging. Thank you for your help, we really do appreciate it. ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete ** Tags added: kernel-request-3.13.0-24.46 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
As mentioned I already tested with a Kernel from 13.10 with the same result. To be sure I installed a minimal Ubuntu 13.10 in a virtual machine and did the steps to reproduce the issue. -> The Problem existed in 13.10 as well. df@ubuntu-bug-check:~$ cat /etc/issue Ubuntu 13.10 \n \l sudo mount -t overlayfs -o lowerdir=/usr,upperdir=/mnt/delta overlayfs /mnt/test sudo lvcreate -s -n backup -L512M /dev/ubuntu-vg/root -> Hang I will attach the dmesg output to this post. A test using a mainline kernel does not work, because overlayfs is not (yet) in mainline: df@feretti-virt:~$ uname -r 3.15.0-031500-generic df@feretti-virt:~$ sudo mount -t overlayfs -o lowerdir=/usr,upperdir=/mnt/delta overlayfs /mnt/test mount: unknown filesystem type 'overlayfs' So I will set the tag to 'kernel-unable-to-test-upstream'. Please change that in case there is a tag better fitting the situation. Thanks for looking into it. ** Attachment added: "dmesg output on the 13.10 installation" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+attachment/4130901/+files/dmesg-1310.txt ** Tags added: kernel-unable-to-test-upstream ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1328595] Re: lvcreate -s hangs when overlayfs is in use
Did this issue occur in a previous version of Ubuntu, or is this a new issue? Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.15 kernel[0]. If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'. If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'. If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'. Once testing of the upstream kernel is complete, please mark this bug as "Confirmed". Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.15-utopic/ ** Changed in: linux (Ubuntu) Importance: Undecided => High ** Tags added: kernel-da-key ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1328595 Title: lvcreate -s hangs when overlayfs is in use To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328595/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
