[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
*** This bug is a duplicate of bug 1781364 *** https://bugs.launchpad.net/bugs/1781364 ** This bug has been marked a duplicate of bug 1781364 Kernel error "task zfs:pid blocked for more than 120 seconds" -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in Linux: Fix Released Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
** Changed in: linux Status: New => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in Linux: Fix Released Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336] sync_filesystem+0x39/0x40 [ 1330.392340]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
** Changed in: linux Status: Unknown => New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in Linux: New Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336] sync_filesystem+0x39/0x40 [ 1330.392340]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
** Bug watch added: Github Issue Tracker for ZFS #7659 https://github.com/zfsonlinux/zfs/issues/7659 ** Changed in: linux Status: Fix Released => Unknown ** Changed in: linux Remote watch: Github Issue Tracker for ZFS #7691 => Github Issue Tracker for ZFS #7659 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in Linux: Unknown Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
** Changed in: linux Status: Unknown => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in Linux: Fix Released Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336] sync_filesystem+0x39/0x40 [ 1330.392340]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
Upstream bug report and pull request to try: "Kernel error "task zfs:pid blocked for more than 120 seconds" #7691" https://github.com/zfsonlinux/zfs/issues/7691 "Fix zpl_mount() deadlock #7693" https://github.com/zfsonlinux/zfs/pull/7693 ** Bug watch added: Github Issue Tracker for ZFS #7691 https://github.com/zfsonlinux/zfs/issues/7691 ** Also affects: linux via https://github.com/zfsonlinux/zfs/issues/7691 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in Linux: Unknown Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
It's an issue for my server too. LXD hangs when I try to restart containers at times. I'm switching to libvirt until this is resolved. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
vmlinuz-4.15.0-23-generic - it seems ok! I have upgraded normally and everything is fine, the machine boots up. It's all done for me. Thanks! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
4.15.0-23-generic - same bug: zfs still hangs. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336] sync_filesystem+0x39/0x40 [ 1330.392340] generic_shutdown_super+0x27/0x120 [ 1330.392343]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
And another little update: I started to downgrade kernels on my serves (ubuntu 18.04 lxd hw nodes) and also I had a problem: HP blade gen8 hangs on reboot/shutdown (system indicates blade failure and you need to pull it off and insert again), while gen7 works fine. With 4.13 kernel gen8 works fine and can be rebooted: so this issue was also fixed with kernel downgrade from 4.15... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ?
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
And another little update to previous experiment: Everything works fine with kernel 4.13.0-36 and zfs 0.7.5 IF you create new zpool with disabled unsupported features: feature@multi_vdev_crash_dump feature@large_dnode feature@sha512 feature@skein feature@edonr feature@userobj_accounting Problem is you can't disable them on current zpool, you need to recreate zpool. And this features is enabled by default: if you do "zpool create pool /dev/disk" - they are enabled on 4.15 or you get an error on 4.13. Also disabling this features don't fix bug: if you create zpool on 4.13 kernel and after switch back to 4.15 - zfs still hangs on container restart. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
Another experiment: 1) Setting new ubuntu 18.04 server with LXD on ZFS - zfs hangs on ct restart 2) Install 4.13.0-36 (from 16.04 HWE install disk) kernel + zfs 0.6.5 from xenial repo (because 0.7.5 not compatible with old kernel) 3) Setup grub to load old kernel and reboot 4) And... everything is fine :) I will use this method on my servers because I have no idea when this bug will be fixed... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
Little update here: I tried ubuntu server 16.04 with HWE (4.13.0-36-generic) and lxd 3.0 + zfs 0.7.5 (latest from bionic repo) - and it seems there is no such bug with 4.13 kernel. After that I updated kernel (from bionic repo again - 4.15.0-22-generic) - and but appear again: zfs hangs if my lxd ct restarts (on mount again). So now I am kind of sure that it is kernel bug. Also I tried 4.16.12 from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16.12/ Same behaviour - zfs hangs. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: Confirmed Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
apport information ** Tags added: apport-collected ** Description changed: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336] sync_filesystem+0x39/0x40 [ 1330.392340] generic_shutdown_super+0x27/0x120 [ 1330.392343] kill_anon_super+0x12/0x20 [ 1330.392419] zpl_kill_sb+0x1a/0x20 [zfs] [ 1330.392423] deactivate_locked_super+0x48/0x80 [ 1330.392427] deactivate_super+0x40/0x60 [ 1330.392430] cleanup_mnt+0x3f/0x80 [ 1330.392434]
[Kernel-packages] [Bug 1773392] Re: zfs hangs on mount/unmount
Little notice also: after that hang is happend you can't shutdown or reboot your server, you need to reset it or turn it off manually (or via ipmi). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1773392 Title: zfs hangs on mount/unmount Status in linux package in Ubuntu: New Bug description: I am running lxd 3.0 on ubuntu 18.04 with kernel 4.15.0-22-generic and 4.15.0-20-generic (same behaviour) with zfs backend (0.7.5-1ubuntu16; also tried 0.7.9). Sometimes lxd hangs when I try to stop / restart or "stop && move" some containers. Furhter investigation showed that problem is in zfs mount or unmount: it just hangs and lxd just wait it. Also commands like "zfs list" hangs to. It seems that it is not lxd or zfs issue, but kernel bug? https://github.com/lxc/lxd/issues/4104#issuecomment-392072939 I have one test ct that always hangs on restart, so here is info: dmesg: [ 1330.390938] INFO: task txg_sync:9944 blocked for more than 120 seconds. [ 1330.390994] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391101] txg_syncD0 9944 2 0x8000 [ 1330.391105] Call Trace: [ 1330.391117] __schedule+0x297/0x8b0 [ 1330.391122] schedule+0x2c/0x80 [ 1330.391136] cv_wait_common+0x11e/0x140 [spl] [ 1330.391141] ? wait_woken+0x80/0x80 [ 1330.391152] __cv_wait+0x15/0x20 [spl] [ 1330.391234] rrw_enter_write+0x3c/0xa0 [zfs] [ 1330.391306] rrw_enter+0x13/0x20 [zfs] [ 1330.391380] spa_sync+0x7c9/0xd80 [zfs] [ 1330.391457] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 1330.391534] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 1330.391543] thread_generic_wrapper+0x74/0x90 [spl] [ 1330.391549] kthread+0x121/0x140 [ 1330.391558] ? __thread_exit+0x20/0x20 [spl] [ 1330.391562] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391566] ? kthread_create_worker_on_cpu+0x70/0x70 [ 1330.391569] ret_from_fork+0x35/0x40 [ 1330.391582] INFO: task lxd:12419 blocked for more than 120 seconds. [ 1330.391630] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.391679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.391735] lxd D0 12419 1 0x [ 1330.391739] Call Trace: [ 1330.391745] __schedule+0x297/0x8b0 [ 1330.391749] schedule+0x2c/0x80 [ 1330.391752] rwsem_down_write_failed+0x162/0x360 [ 1330.391808] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 1330.391814] call_rwsem_down_write_failed+0x17/0x30 [ 1330.391817] ? call_rwsem_down_write_failed+0x17/0x30 [ 1330.391821] down_write+0x2d/0x40 [ 1330.391825] grab_super+0x30/0x90 [ 1330.391901] ? zpl_create+0x160/0x160 [zfs] [ 1330.391905] sget_userns+0x91/0x490 [ 1330.391908] ? get_anon_bdev+0x100/0x100 [ 1330.391983] ? zpl_create+0x160/0x160 [zfs] [ 1330.391987] sget+0x7d/0xa0 [ 1330.391990] ? get_anon_bdev+0x100/0x100 [ 1330.392066] zpl_mount+0xa8/0x160 [zfs] [ 1330.392071] mount_fs+0x37/0x150 [ 1330.392077] vfs_kern_mount.part.23+0x5d/0x110 [ 1330.392080] do_mount+0x5ed/0xce0 [ 1330.392083] ? copy_mount_options+0x2c/0x220 [ 1330.392086] SyS_mount+0x98/0xe0 [ 1330.392092] do_syscall_64+0x73/0x130 [ 1330.392096] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 1330.392099] RIP: 0033:0x4db36a [ 1330.392101] RSP: 002b:00c4207fa768 EFLAGS: 0216 ORIG_RAX: 00a5 [ 1330.392104] RAX: ffda RBX: RCX: 004db36a [ 1330.392106] RDX: 00c4205984cc RSI: 00c420a6ee00 RDI: 00c420a23b60 [ 1330.392108] RBP: 00c4207fa808 R08: 00c4209d4960 R09: [ 1330.392110] R10: R11: 0216 R12: [ 1330.392112] R13: 0039 R14: 0038 R15: 0080 [ 1330.392123] INFO: task lxd:16725 blocked for more than 120 seconds. [ 1330.392171] Tainted: P O 4.15.0-22-generic #24-Ubuntu [ 1330.392220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1330.392276] lxd D0 16725 1 0x0002 [ 1330.392279] Call Trace: [ 1330.392284] __schedule+0x297/0x8b0 [ 1330.392289] ? irq_work_queue+0x8d/0xa0 [ 1330.392293] schedule+0x2c/0x80 [ 1330.392297] io_schedule+0x16/0x40 [ 1330.392302] wait_on_page_bit_common+0xd8/0x160 [ 1330.392305] ? page_cache_tree_insert+0xe0/0xe0 [ 1330.392309] __filemap_fdatawait_range+0xfa/0x160 [ 1330.392313] ? _cond_resched+0x19/0x40 [ 1330.392317] ? bdi_split_work_to_wbs+0x45/0x2c0 [ 1330.392321] ? _cond_resched+0x19/0x40 [ 1330.392324] filemap_fdatawait_keep_errors+0x1e/0x40 [ 1330.392327] sync_inodes_sb+0x20d/0x2b0 [ 1330.392333] __sync_filesystem+0x1b/0x60 [ 1330.392336]
