Looks like your network deactivated before the rbd volume was unmounted. This is a known issue without a good programmatic workaround and you’ll need to adjust your configuration. On Tue, Jan 22, 2019 at 9:17 AM Gao, Wenjun <wenj...@ebay.com> wrote:
> I’m using krbd to map a rbd device to a VM, it appears when the device is > mounted, reboot OS will hung for more than 7min, in baremetal case, it > could be more than 15min, even using the latest kernel 5.0.0, the problem > still occurs. > > Here are the console logs with 4.15.18 kernel and mimic rbd client, reboot > seems to be stuck in umount rbd operation > > *[ OK ] Stopped Update UTMP about System Boot/Shutdown.* > > *[ OK ] Stopped Create Volatile Files and Directories.* > > *[ OK ] Stopped target Local File Systems.* > > * Unmounting /run/user/110281572...* > > * Unmounting /var/tmp...* > > * Unmounting /root/test...* > > * Unmounting /run/user/78402...* > > * Unmounting Configuration File System...* > > *[ OK ] Stopped Configure read-only root support.* > > *[ OK ] Unmounted /var/tmp.* > > *[ OK ] Unmounted /run/user/78402.* > > *[ OK ] Unmounted /run/user/110281572.* > > *[ OK ] Stopped target Swap.* > > *[ OK ] Unmounted Configuration File System.* > > *[ 189.919062] libceph: mon4 XX.XX.XX.XX:6789 session lost, hunting for > new mon* > > *[ 189.950085] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 189.950764] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 190.687090] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 190.694197] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 191.711080] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 191.745254] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 193.695065] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 193.727694] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 197.087076] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 197.121077] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 197.663082] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 197.680671] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 198.687122] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 198.719253] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 200.671136] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 200.702717] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 204.703115] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 204.736586] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 209.887141] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 209.918721] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 210.719078] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 210.750378] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 211.679118] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 211.712246] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 213.663116] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 213.696943] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 217.695062] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 217.728511] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 225.759109] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 225.775869] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 233.951062] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 233.951997] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 234.719114] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 234.720083] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 235.679112] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 235.680060] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 237.663088] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 237.664121] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 241.695082] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 241.696500] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 249.823095] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 249.824101] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 264.671119] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 264.672102] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 265.695109] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 265.696106] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 266.719145] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 266.720204] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 268.703121] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 268.704110] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 272.671115] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 272.672159] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 281.055087] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 281.056577] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 294.879098] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 294.880230] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 295.711107] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 295.712102] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 296.671090] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 296.672082] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 298.719086] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 298.720027] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 302.687077] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 302.688103] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 310.751132] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 310.763103] libceph: mon3 XX.XX.XX.XX:6789 connect error* > > *[ 325.087096] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 325.088045] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 325.663115] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 325.664046] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 326.687094] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 326.688120] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 328.671075] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 328.672157] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 332.703098] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 332.704027] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 340.959100] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 340.960055] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 355.807079] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 355.808025] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 356.703101] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 356.704094] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 357.663072] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 357.664025] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 359.711102] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 359.712063] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 363.679106] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 363.680063] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 369.631112] INFO: task umount:6489 blocked for more than 120 seconds.* > > *[ 369.639665] Not tainted 4.15.18 #1* > > *[ 369.640417] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables this message.* > > *[ 369.641909] umount D 0 6489 1 0x00000004* > > *[ 369.642842] Call Trace:* > > *[ 369.643479] ? __schedule+0x293/0x8a0* > > *[ 369.644183] ? out_of_line_wait_on_atomic_t+0x110/0x110* > > *[ 369.645063] schedule+0x32/0x80* > > *[ 369.645760] io_schedule+0x12/0x40* > > *[ 369.646477] bit_wait_io+0xd/0x50* > > *[ 369.647167] __wait_on_bit+0x5c/0x90* > > *[ 369.647846] out_of_line_wait_on_bit+0x8e/0xb0* > > *[ 369.648668] ? bit_waitqueue+0x30/0x30* > > *[ 369.649409] jbd2_write_superblock+0x11a/0x230 [jbd2]* > > *[ 369.650275] ? submit_bio+0x6e/0x140* > > *[ 369.650961] jbd2_journal_update_sb_log_tail+0x32/0x70 [jbd2]* > > *[ 369.651880] __jbd2_update_log_tail+0x35/0xf0 [jbd2]* > > *[ 369.652704] jbd2_cleanup_journal_tail+0x50/0xa0 [jbd2]* > > *[ 369.653562] jbd2_log_do_checkpoint+0x110/0x4b0 [jbd2]* > > *[ 369.654415] ? __schedule+0x29b/0x8a0* > > *[ 369.655106] ? prepare_to_wait_event+0x80/0x140* > > *[ 369.655879] jbd2_journal_destroy+0x116/0x280 [jbd2]* > > *[ 369.656703] ? remove_wait_queue+0x60/0x60* > > *[ 369.657564] ext4_put_super+0x76/0x3f0 [ext4]* > > *[ 369.658357] generic_shutdown_super+0x6c/0x120* > > *[ 369.659136] kill_block_super+0x21/0x50* > > *[ 369.659836] deactivate_locked_super+0x3f/0x70* > > *[ 369.660607] cleanup_mnt+0x3b/0x70* > > *[ 369.661286] task_work_run+0x88/0xa0* > > *[ 369.661983] exit_to_usermode_loop+0x6c/0xa3* > > *[ 369.662741] do_syscall_64+0x177/0x1a0* > > *[ 369.663444] entry_SYSCALL_64_after_hwframe+0x3d/0xa2* > > *[ 369.664285] RIP: 0033:0x7f39f4a41f47* > > *[ 369.664955] RSP: 002b:00007fff3fb03998 EFLAGS: 00000246 ORIG_RAX: > 00000000000000a6* > > *[ 369.666288] RAX: 0000000000000000 RBX: 0000563f3abfa040 RCX: > 00007f39f4a41f47* > > *[ 369.667335] RDX: 0000000000000001 RSI: 0000000000000000 RDI: > 0000563f3abffb80* > > *[ 369.668381] RBP: 0000563f3abffb80 R08: 0000563f3abffaf0 R09: > 0000000000000000* > > *[ 369.669425] R10: 00007fff3fb03420 R11: 0000000000000246 R12: > 00007f39f55bfd58* > > *[ 369.670521] R13: 0000000000000000 R14: 0000563f3abfa140 R15: > 0000563f3abfa040* > > *[ 371.679142] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 371.680098] libceph: mon4 XX.XX.XX.XX:6789 connect error* > > *[ 386.015126] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 386.016135] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 386.719109] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 386.720042] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 387.679128] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 387.680069] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 389.663096] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 389.664083] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 393.695118] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 393.696073] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 401.887064] libceph: connect XX.XX.XX.XX:6789 error -101* > > *[ 401.887995] libceph: mon0 XX.XX.XX.XX:6789 connect error* > > *[ 416.735084] libceph: connect XX.XX.XX.XX.10:6789 error -101* > > *[ 416.736165] libceph: mon1 XX.XX.XX.XX.10:6789 connect error* > > *[ 417.695092] libceph: connect XX.XX.XX.XX.10:6789 error -101* > > *[ 417.703178] libceph: mon1 XX.XX.XX.XX.10:6789 connect error* > > *[ 418.719156] libceph: connect XX.XX.XX.XX.10:6789 error -101* > > *[ 418.720055] libceph: mon1 XX.XX.XX.XX.10:6789 connect error* > > *[ 420.703086] libceph: connect XX.XX.XX.XX.10:6789 error -101* > > *[ 420.703940] libceph: mon1 XX.XX.XX.XX.10:6789 connect error* > > *[ 424.671082] libceph: connect XX.XX.XX.XX.10:6789 error -101* > > *[ 424.672037] libceph: mon1 XX.XX.XX.XX.10:6789 connect error* > > *[ OK ] Unmounted /root/test.* > > *[ OK ] Reached target Unmount All Filesystems.* > > *[ OK ] Stopped target Local File Systems (Pre).* > > *[ OK ] Stopped Create Static Device Nodes in /dev.* > > *[ OK ] Stopped Remount Root and Kernel File Systems.* > > *[ OK ] Reached target Shutdown.* > > *[ 431.016439] systemd-shutdown[1]: Syncing filesystems and block > devices.* > > *[ 433.119104] libceph: connect XX.XX.XX.XX.10:6789 error -101* > > *[ 433.120595] libceph: mon1 XX.XX.XX.XX.10:6789 connect error* > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
