Hi Sheng, this is a known issue (not related to nbd-client/nbd-server on the same box, but any termination of the nbd connection with mounted fs and I/O ongoing)
It was discussed at length in Nov 2013 in this thread (along with possible solutions): http://sourceforge.net/p/nbd/mailman/nbd-general/thread/20131114075827.GA13554%40quack.suse.cz/#msg31636917 Maybe Markus has some thoughts on how to fix it. I think there are a couple options. -- Paul On Wed, Sep 16, 2015 at 9:18 PM, Sheng Yang <[email protected]> wrote: > Hi, Markus, > > I've experienced kernel bug and system hang regarding using > nbd-client/server on the same machine. Kernel verison is v4.2. > > I know http://nbd.sourceforge.net/ said: > "Please note that read-write nbd with client and server on the same > machine is a bad idea: expect deadlock within seconds". But it's > mentioned at http://thread.gmane.org/gmane.linux.kernel/869784 that > NBD should be able to handle client/server on the same machine. > > The bug can be easily reproduce like this; > > truncate -s 1G nbd.vol > mkfs.ext4 nbd.vol > <add nbd.vol to nbd-server's config, named "nbd-vol"> > service nbd-server restart > > nbd-client localhost /dev/nbd0 -N nbd-vol > mount /dev/ndb0 /mnt/tmp > wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.2.tar.xz > --limit-rate=100k > > <then in another console> > pkill nbd-server > > Then the following would be triggered: > > [ 2886.348891] block nbd0: Attempted send on closed socket > [ 2886.350579] blk_update_request: I/O error, dev nbd0, sector 1016 > [ 2886.352210] EXT4-fs warning (device nbd0): > htree_dirblock_to_tree:959: inode #2: lblock 0: comm ls: error -5 > reading directory block > [ 2886.352236] block nbd0: Attempted send on closed socket > [ 2886.353646] blk_update_request: I/O error, dev nbd0, sector 524 > [ 2886.355267] block nbd0: Attempted send on closed socket > [ 2886.356665] blk_update_request: I/O error, dev nbd0, sector 526 > [ 2886.358271] block nbd0: Attempted send on closed socket > [ 2886.359654] blk_update_request: I/O error, dev nbd0, sector 528 > [ 2886.361253] block nbd0: Attempted send on closed socket > [ 2886.362597] blk_update_request: I/O error, dev nbd0, sector 530 > [ 2886.364106] block nbd0: Attempted send on closed socket > [ 2886.365400] blk_update_request: I/O error, dev nbd0, sector 532 > [ 2886.366905] block nbd0: Attempted send on closed socket > [ 2886.368205] blk_update_request: I/O error, dev nbd0, sector 534 > [ 2886.369714] block nbd0: Attempted send on closed socket > [ 2886.371025] blk_update_request: I/O error, dev nbd0, sector 536 > [ 2886.372560] block nbd0: Attempted send on closed socket > [ 2886.373871] blk_update_request: I/O error, dev nbd0, sector 538 > [ 2886.375283] block nbd0: Attempted send on closed socket > [ 2886.376504] blk_update_request: I/O error, dev nbd0, sector 540 > ... > [ 2886.405309] block nbd0: Attempted send on closed socket > [ 2886.406440] EXT4-fs error (device nbd0): __ext4_get_inode_loc:3927: > inode #2: block 261: comm ls: unable to read itable block > [ 2886.408816] ------------[ cut here ]------------ > [ 2886.409800] kernel BUG at fs/buffer.c:3025! > [ 2886.409800] invalid opcode: 0000 [#1] SMP > [ 2886.409800] Modules linked in: > [ 2886.409800] CPU: 3 PID: 17137 Comm: ls Not tainted 4.2.0 #11 > [ 2886.409800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.7.5-20150306_163512-brownie 04/01/2014 > [ 2886.409800] task: ffff8800b8a50000 ti: ffff8800b7b64000 task.ti: > ffff8800b7b64000 > [ 2886.409800] RIP: 0010:[<ffffffff811a778e>] [<ffffffff811a778e>] > submit_bh_wbc+0x16e/0x180 > [ 2886.409800] RSP: 0018:ffff8800b7b67b58 EFLAGS: 00010246 > [ 2886.409800] RAX: 0000000000000005 RBX: ffff8800adb849c0 RCX: > 0000000000000000 > [ 2886.409800] RDX: 0000000000000000 RSI: ffff8800adb849c0 RDI: > 0000000000000411 > [ 2886.409800] RBP: ffff8800b7b67b88 R08: fffffffffffffff0 R09: > 0000000000000000 > [ 2886.409800] R10: 0000000000000218 R11: 0000000000000006 R12: > 0000000000000411 > [ 2886.409800] R13: ffff8800b88fc400 R14: 000000000002facc R15: > ffff88013a390000 > [ 2886.409800] FS: 00007f83cf856840(0000) GS:ffff88013fd80000(0000) > knlGS:0000000000000000 > [ 2886.409800] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 2886.409800] CR2: 0000000000f75388 CR3: 0000000139896000 CR4: > 00000000000006e0 > [ 2886.409800] Stack: > [ 2886.409800] ffff88013a395358 ffff8800adb849c0 0000000000000411 > ffff8800b88fc400 > [ 2886.409800] 000000000002facc ffff88013a390000 ffff8800b7b67ba8 > ffffffff811a9449 > [ 2886.409800] ffffffff81f209e0 ffff8800adb849c0 ffff8800b7b67bb8 > ffffffff811a94ee > [ 2886.409800] Call Trace: > [ 2886.409800] [<ffffffff811a9449>] __sync_dirty_buffer+0x59/0xf0 > [ 2886.409800] [<ffffffff811a94ee>] sync_dirty_buffer+0xe/0x10 > [ 2886.409800] [<ffffffff8120de31>] ext4_commit_super+0x181/0x200 > [ 2886.409800] [<ffffffff8120e255>] __ext4_error_inode+0x85/0x140 > [ 2886.409800] [<ffffffff811f0fdd>] __ext4_get_inode_loc+0x1fd/0x3d0 > [ 2886.409800] [<ffffffff811f2a78>] ext4_get_inode_loc+0x18/0x20 > [ 2886.409800] [<ffffffff811f442f>] ext4_reserve_inode_write+0x1f/0x80 > [ 2886.409800] [<ffffffff811f7773>] ? ext4_dirty_inode+0x43/0x60 > [ 2886.409800] [<ffffffff811f44d8>] ext4_mark_inode_dirty+0x48/0x1e0 > [ 2886.409800] [<ffffffff811f7773>] ext4_dirty_inode+0x43/0x60 > [ 2886.409800] [<ffffffff8119e603>] __mark_inode_dirty+0x163/0x310 > [ 2886.409800] [<ffffffff8118d932>] generic_update_time+0x72/0xc0 > [ 2886.409800] [<ffffffff810a8bd1>] ? current_fs_time+0x11/0x60 > [ 2886.409800] [<ffffffff8118f143>] touch_atime+0x83/0xa0 > [ 2886.409800] [<ffffffff81187096>] iterate_dir+0xd6/0x120 > [ 2886.409800] [<ffffffff8118749c>] SyS_getdents+0x7c/0xe0 > [ 2886.409800] [<ffffffff811871a0>] ? fillonedir+0xc0/0xc0 > [ 2886.409800] [<ffffffff810485fc>] ? trace_do_page_fault+0x3c/0xb0 > [ 2886.409800] [<ffffffff818cc8ae>] entry_SYSCALL_64_fastpath+0x12/0x71 > [ 2886.409800] Code: 09 e7 14 00 48 83 c4 08 31 c0 5b 41 5c 41 5d 41 > 5e 41 5f 5d c3 40 f6 c7 01 0f 84 ff fe ff ff f0 80 63 01 f7 e9 f5 fe > ff ff 0f 0b <0f> 0b 0f 0b 0f 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 55 > 31 c9 > [ 2886.409800] RIP [<ffffffff811a778e>] submit_bh_wbc+0x16e/0x180 > [ 2886.409800] RSP <ffff8800b7b67b58> > [ 2886.461658] ---[ end trace 46f00e222737016e ]--- > > > I am trying to use NBD for some HA scenario, I got above BUG using > default nbd-server. After this, any access to /mnt/tmp or /dev/nbd0 > would hang. > > We also has a customerized nbd server, and in the case of killing the > connection would result in resursive fault and complete hang the > system. Something like this happened: > > [ 170.552688] block nbd0: Receive control failed (result -32) > [ 171.253234] block nbd0: Attempted send on closed socket > [ 171.254772] blk_update_request: I/O error, dev nbd0, sector 8464 > [ 171.256586] block nbd0: Attempted send on closed socket > [ 171.258044] blk_update_request: I/O error, dev nbd0, sector 8720 > [ 171.259953] block nbd0: Attempted send on closed socket > [ 171.261415] blk_update_request: I/O error, dev nbd0, sector 8976 > [ 171.263300] block nbd0: Attempted send on closed socket > [ 171.264719] blk_update_request: I/O error, dev nbd0, sector 9232 > [ 171.266570] block nbd0: Attempted send on closed socket > [ 171.268016] blk_update_request: I/O error, dev nbd0, sector 9488 > [ 171.269604] block nbd0: Attempted send on closed socket > [ 171.270903] blk_update_request: I/O error, dev nbd0, sector 9744 > [ 171.272470] block nbd0: Attempted send on closed socket > [ 171.273904] blk_update_request: I/O error, dev nbd0, sector 10000 > [ 171.275523] block nbd0: Attempted send on closed socket > [ 171.276815] blk_update_request: I/O error, dev nbd0, sector 10256 > [ 171.278499] block nbd0: Attempted send on closed socket > [ 171.279851] blk_update_request: I/O error, dev nbd0, sector 10512 > [ 171.281383] block nbd0: Attempted send on closed socket > [ 171.282592] blk_update_request: I/O error, dev nbd0, sector 10768 > [ 171.284021] block nbd0: Attempted send on closed socket > [ 171.285231] block nbd0: Attempted send on closed socket > [ 171.286568] block nbd0: Attempted send on closed socket > [ 171.287973] block nbd0: Attempted send on closed socket > [ 171.289283] block nbd0: Attempted send on closed socket > [ 171.290565] block nbd0: Attempted send on closed socket > [ 173.711085] ------------[ cut here ]------------ > [ 173.712036] kernel BUG at fs/buffer.c:3025! > [ 173.712036] invalid opcode: 0000 [#1] SMP > [ 173.712036] Modules linked in: > [ 173.712036] CPU: 0 PID: 2328 Comm: jbd2/nbd0-8 Not tainted 4.2.0 #10 > [ 173.712036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.7.5-0150306_163512-brownie 04/01/2014 > [ 173.712036] task: ffff8800bb8d0000 ti: ffff8800ad154000 task.ti: > ffff8800ad154000 > [ 173.712036] RIP: 0010:[<ffffffff811a778e>] [<ffffffff811a778e>] > submit_bh_wbc+0x16e/0x180 > [ 173.712036] RSP: 0018:ffff8800ad157bf8 EFLAGS: 00010246 > [ 173.712036] RAX: 0000000004000005 RBX: ffff880139eb6e38 RCX: > 0000000000000000 > [ 173.712036] RDX: 0000000000000000 RSI: ffff880139eb6e38 RDI: > 0000000000000411 > [ 173.712036] RBP: ffff8800ad157c28 R08: ffff8800ad154000 R09: > 0000000000000001 > [ 173.712036] R10: 00000000fffe12ab R11: 0000000000000020 R12: > ffff880135f04000 > [ 173.712036] R13: 0000000000000411 R14: ffff880139eb6e38 R15: > ffff8800b7855000 > [ 173.712036] FS: 0000000000000000(0000) GS:ffff88013fc00000(0000) > knlGS:0000000000000000 > [ 173.712036] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 173.712036] CR2: 00007fe561c38000 CR3: 0000000001e0c000 CR4: > 00000000000006f0 > [ 173.712036] Stack: > [ 173.712036] 0000000000000000 ffff880135f04000 ffff880135f04000 > 0000000000000411 > [ 173.712036] ffff880139eb6e38 ffff8800b7855000 ffff8800ad157c38 > ffffffff811a77bd > [ 173.712036] ffff8800ad157c78 ffffffff81241c17 0000000000000000 > ffff880135f04000 > [ 173.712036] Call Trace: > [ 173.712036] [<ffffffff811a77bd>] submit_bh+0xd/0x10 > [ 173.712036] [<ffffffff81241c17>] jbd2_write_superblock+0x87/0x170 > [ 173.712036] [<ffffffff812425f2>] > jbd2_journal_update_sb_log_tail+0x32/0x80 > [ 173.712036] [<ffffffff818cab91>] ? mutex_lock+0x11/0x40 > [ 173.712036] [<ffffffff8123b4ef>] > jbd2_journal_commit_transaction+0x5ff/0x1680 > [ 173.712036] [<ffffffff81080513>] ? dequeue_entity+0x163/0x710 > [ 173.712036] [<ffffffff81083b7d>] ? dequeue_task_fair+0x50d/0x7b0 > [ 173.712036] [<ffffffff810a9c33>] ? lock_timer_base.isra.35+0x53/0x70 > [ 173.712036] [<ffffffff810a9e0a>] ? try_to_del_timer_sync+0x4a/0x60 > [ 173.712036] [<ffffffff8123fa36>] kjournald2+0xb6/0x230 > [ 173.712036] [<ffffffff8108c3c0>] ? wait_woken+0x80/0x80 > [ 173.712036] [<ffffffff8123f980>] ? commit_timeout+0x10/0x10 > [ 173.712036] [<ffffffff8106ec44>] kthread+0xc4/0xe0 > [ 173.712036] [<ffffffff8106eb80>] ? kthread_create_on_node+0x180/0x180 > [ 173.712036] [<ffffffff818ccc9f>] ret_from_fork+0x3f/0x70 > [ 173.712036] [<ffffffff8106eb80>] ? kthread_create_on_node+0x180/0x180 > [ 173.712036] Code: 09 e7 14 00 48 83 c4 08 31 c0 5b 41 5c 41 5d 41 > 5e 41 5f 5d c3 40 f6 c7 01 0f 84 ff fe ff ff f0 80 63 01 f7 e9 f5 fe > ff ff 0f 0b <0f> 0b 0f 0b 0f 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 55 > 31 c9 > [ 173.712036] RIP [<ffffffff811a778e>] submit_bh_wbc+0x16e/0x180 > [ 173.712036] RSP <ffff8800ad157bf8> > [ 173.776031] ---[ end trace d41ec1552544a207 ]--- > [ 199.615574] block nbd0: Attempted send on closed socket > [ 199.617102] blk_update_request: 6 callbacks suppressed > [ 199.618090] blk_update_request: I/O error, dev nbd0, sector 8456 > [ 199.620187] EXT4-fs error (device nbd0): __ext4_get_inode_loc:3927: > inode #12: block 1057: comm kworker/u8:1: unable to read itable block > [ 199.623876] ------------[ cut here ]------------ > [ 199.624839] kernel BUG at fs/buffer.c:3025! > [ 199.624839] invalid opcode: 0000 [#2] SMP > [ 199.624839] Modules linked in: > [ 199.624839] CPU: 2 PID: 25 Comm: kworker/u8:1 Tainted: G D > 4.2.0 #10 > [ 199.624839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.7.5-20150306_163512-brownie 04/01/2014 > [ 199.624839] Workqueue: writeback wb_workfn (flush-43:0) > [ 199.624839] task: ffff88013abbd240 ti: ffff88013a424000 task.ti: > ffff88013a424000 > [ 199.624839] RIP: 0010:[<ffffffff811a778e>] [<ffffffff811a778e>] > submit_bh_wbc+0x16e/0x180 > [ 199.624839] RSP: 0018:ffff88013a4274e8 EFLAGS: 00010246 > [ 199.624839] RAX: 0000000000620005 RBX: ffff8800bb78af70 RCX: > 0000000000000000 > [ 199.624839] RDX: 0000000000000000 RSI: ffff8800bb78af70 RDI: > 0000000000000411 > [ 199.624839] RBP: ffff88013a427518 R08: fffffffffffffff0 R09: > 0000000000000000 > [ 199.624839] R10: 0000000000000226 R11: 0000000000000006 R12: > 0000000000000411 > [ 199.624839] R13: ffff8800b8d70400 R14: 0000000000043220 R15: > ffff880135f02800 > [ 199.624839] FS: 0000000000000000(0000) GS:ffff88013fd00000(0000) > knlGS:0000000000000000 > [ 199.624839] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 199.624839] CR2: 0000000002024808 CR3: 00000000b8630000 CR4: > 00000000000006e0 > [ 199.624839] Stack: > [ 199.624839] ffff880135f06358 ffff8800bb78af70 0000000000000411 > ffff8800b8d70400 > [ 199.624839] 0000000000043220 ffff880135f02800 ffff88013a427538 > ffffffff811a9449 > [ 199.624839] ffffffff81f209e0 ffff8800bb78af70 ffff88013a427548 > ffffffff811a94ee > [ 199.624839] Call Trace: > [ 199.624839] [<ffffffff811a9449>] __sync_dirty_buffer+0x59/0xf0 > [ 199.624839] [<ffffffff811a94ee>] sync_dirty_buffer+0xe/0x10 > [ 199.624839] [<ffffffff8120de31>] ext4_commit_super+0x181/0x200 > [ 199.624839] [<ffffffff8120e255>] __ext4_error_inode+0x85/0x140 > [ 199.624839] [<ffffffff8108c400>] ? autoremove_wake_function+0x40/0x40 > [ 199.624839] [<ffffffff811f0fdd>] __ext4_get_inode_loc+0x1fd/0x3d0 > [ 199.624839] [<ffffffff811f2a78>] ext4_get_inode_loc+0x18/0x20 > [ 199.624839] [<ffffffff811f442f>] ext4_reserve_inode_write+0x1f/0x80 > [ 199.624839] [<ffffffff81218fdd>] ? __ext4_ext_dirty+0x6d/0x90 > [ 199.624839] [<ffffffff811f44d8>] ext4_mark_inode_dirty+0x48/0x1e0 > [ 199.624839] [<ffffffff81218fdd>] __ext4_ext_dirty+0x6d/0x90 > [ 199.624839] [<ffffffff8121abdc>] ext4_ext_insert_extent+0x6ac/0x1310 > [ 199.624839] [<ffffffff8121e098>] ext4_ext_map_blocks+0x258/0xdc0 > [ 199.624839] [<ffffffff811f1ba4>] ext4_map_blocks+0x124/0x4f0 > [ 199.624839] [<ffffffff811f4b51>] ext4_writepages+0x4e1/0xc50 > [ 199.624839] [<ffffffff8132ab85>] ? find_next_bit+0x15/0x20 > [ 199.624839] [<ffffffff81084cbb>] ? find_busiest_group+0x11b/0x980 > [ 199.624839] [<ffffffff81123199>] do_writepages+0x19/0x30 > [ 199.624839] [<ffffffff8119e7f9>] __writeback_single_inode+0x49/0x2b0 > [ 199.624839] [<ffffffff8119ef64>] writeback_sb_inodes+0x274/0x4c0 > [ 199.624839] [<ffffffff8119f234>] __writeback_inodes_wb+0x84/0xb0 > [ 199.624839] [<ffffffff8119f47f>] wb_writeback+0x21f/0x290 > [ 199.624839] [<ffffffff8119fc9e>] wb_workfn+0x2fe/0x480 > [ 199.624839] [<ffffffff810698a6>] process_one_work+0x156/0x3e0 > [ 199.624839] [<ffffffff81069e59>] worker_thread+0x49/0x450 > [ 199.624839] [<ffffffff81069e10>] ? rescuer_thread+0x2e0/0x2e0 > [ 199.624839] [<ffffffff81069e10>] ? rescuer_thread+0x2e0/0x2e0 > [ 199.624839] [<ffffffff8106ec44>] kthread+0xc4/0xe0 > [ 199.624839] [<ffffffff8106eb80>] ? kthread_create_on_node+0x180/0x180 > [ 199.624839] [<ffffffff818ccc9f>] ret_from_fork+0x3f/0x70 > [ 199.624839] [<ffffffff8106eb80>] ? kthread_create_on_node+0x180/0x180 > [ 199.624839] Code: 09 e7 14 00 48 83 c4 08 31 c0 5b 41 5c 41 5d 41 > 5e 41 5f 5d c3 40 f6 c7 01 0f 84 ff fe ff ff f0 80 63 01 f7 e9 f5 fe > ff ff 0f 0b <0f> 0b 0f 0b 0f 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 55 > 31 c9 > [ 199.624839] RIP [<ffffffff811a778e>] submit_bh_wbc+0x16e/0x180 > [ 199.624839] RSP <ffff88013a4274e8> > [ 199.706502] ---[ end trace d41ec1552544a208 ]--- > [ 199.707614] kworker/u8:1 (25) used greatest stack depth: 11104 bytes > left > [ 199.708873] BUG: unable to handle kernel paging request at > ffffffffffffffd8 > [ 199.709861] IP: [<ffffffff8106f1cb>] kthread_data+0xb/0x20 > [ 199.709861] PGD 1e0d067 PUD 1e0f067 PMD 0 > [ 199.709861] Oops: 0000 [#3] SMP > [ 199.709861] Modules linked in: > [ 199.709861] CPU: 2 PID: 25 Comm: kworker/u8:1 Tainted: G D > 4.2.0 #10 > [ 199.709861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.7.5-0150306_163512-brownie 04/01/2014 > [ 199.709861] task: ffff88013abbd240 ti: ffff88013a424000 task.ti: > ffff88013a424000 > [ 199.709861] RIP: 0010:[<ffffffff8106f1cb>] [<ffffffff8106f1cb>] > kthread_data+0xb/0x20 > [ 199.709861] RSP: 0018:ffff88013a427188 EFLAGS: 00010092 > [ 199.709861] RAX: 0000000000000000 RBX: 0000000000000002 RCX: > 000000000000000d > [ 199.709861] RDX: 000000000000000d RSI: 0000000000000002 RDI: > ffff88013abbd240 > [ 199.709861] RBP: ffff88013a427188 R08: 0000000000000000 R09: > 0000000000000000 > [ 199.709861] R10: 0000000000000000 R11: 0000000000000006 R12: > 0000000000014fc0 > [ 199.709861] R13: ffff88013fd14fc0 R14: ffff88013abbd240 R15: > 0000000000000002 > [ 199.709861] FS: 0000000000000000(0000) GS:ffff88013fd00000(0000) > nlGS:0000000000000000 > [ 199.709861] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 199.709861] CR2: 0000000000000028 CR3: 00000000b8630000 CR4: > 00000000000006e0 > [ 199.709861] Stack: > [ 199.709861] ffff88013a4271a8 ffffffff8106a2d0 ffff88013a4271a8 > ffff88013fd14fc0 > [ 199.709861] ffff88013a4271f8 ffffffff818c8dac ffff880100000000 > ffff88013abbd240 > [ 199.709861] ffff88013a4271f8 ffff88013a428000 0000000000002b60 > ffff88013a426dd8 > [ 199.709861] Call Trace: > [ 199.709861] [<ffffffff8106a2d0>] wq_worker_sleeping+0x10/0x90 > [ 199.709861] [<ffffffff818c8dac>] __schedule+0x70c/0x900 > [ 199.709861] [<ffffffff818c8fd2>] schedule+0x32/0x80 > [ 199.709861] [<ffffffff810561b8>] do_exit+0x718/0xb00 > [ 199.709861] [<ffffffff81006c79>] oops_end+0x99/0xd0 > [ 199.709861] [<ffffffff81007036>] die+0x46/0x60 > [ 199.709861] [<ffffffff81003d38>] do_trap+0x138/0x140 > [ 199.709861] [<ffffffff810043a5>] do_error_trap+0x75/0xe0 > [ 199.709861] [<ffffffff811a778e>] ? submit_bh_wbc+0x16e/0x180 > [ 199.709861] [<ffffffff813d1e1f>] ? univ8250_console_write+0x19f/0x2e0 > [ 199.709861] [<ffffffff81099597>] ? print_time.part.13+0x67/0x90 > [ 199.709861] [<ffffffff81099625>] ? print_prefix+0x65/0xa0 > [ 199.709861] [<ffffffff8100451b>] do_invalid_op+0x1b/0x20 > [ 199.709861] [<ffffffff818ce19e>] invalid_op+0x1e/0x30 > [ 199.709861] [<ffffffff811a778e>] ? submit_bh_wbc+0x16e/0x180 > [ 199.709861] [<ffffffff810a9bcc>] ? internal_add_timer+0x6c/0x80 > [ 199.709861] [<ffffffff811a9449>] __sync_dirty_buffer+0x59/0xf0 > [ 199.709861] [<ffffffff811a94ee>] sync_dirty_buffer+0xe/0x10 > [ 199.709861] [<ffffffff8120de31>] ext4_commit_super+0x181/0x200 > [ 199.709861] [<ffffffff8120e255>] __ext4_error_inode+0x85/0x140 > [ 199.709861] [<ffffffff8108c400>] ? autoremove_wake_function+0x40/0x40 > [ 199.709861] [<ffffffff811f0fdd>] __ext4_get_inode_loc+0x1fd/0x3d0 > [ 199.709861] [<ffffffff811f2a78>] ext4_get_inode_loc+0x18/0x20 > [ 199.709861] [<ffffffff811f442f>] ext4_reserve_inode_write+0x1f/0x80 > [ 199.709861] [<ffffffff81218fdd>] ? __ext4_ext_dirty+0x6d/0x90 > [ 199.709861] [<ffffffff811f44d8>] ext4_mark_inode_dirty+0x48/0x1e0 > [ 199.709861] [<ffffffff81218fdd>] __ext4_ext_dirty+0x6d/0x90 > [ 199.709861] [<ffffffff8121abdc>] ext4_ext_insert_extent+0x6ac/0x1310 > [ 199.709861] [<ffffffff8121e098>] ext4_ext_map_blocks+0x258/0xdc0 > [ 199.709861] [<ffffffff811f1ba4>] ext4_map_blocks+0x124/0x4f0 > [ 199.709861] [<ffffffff811f4b51>] ext4_writepages+0x4e1/0xc50 > [ 199.709861] [<ffffffff8132ab85>] ? find_next_bit+0x15/0x20 > [ 199.709861] [<ffffffff81084cbb>] ? find_busiest_group+0x11b/0x980 > [ 199.709861] [<ffffffff81123199>] do_writepages+0x19/0x30 > [ 199.709861] [<ffffffff8119e7f9>] __writeback_single_inode+0x49/0x2b0 > [ 199.709861] [<ffffffff8119ef64>] writeback_sb_inodes+0x274/0x4c0 > [ 199.709861] [<ffffffff8119f234>] __writeback_inodes_wb+0x84/0xb0 > [ 199.709861] [<ffffffff8119f47f>] wb_writeback+0x21f/0x290 > [ 199.709861] [<ffffffff8119fc9e>] wb_workfn+0x2fe/0x480 > [ 199.709861] [<ffffffff810698a6>] process_one_work+0x156/0x3e0 > [ 199.709861] [<ffffffff81069e59>] worker_thread+0x49/0x450 > [ 199.709861] [<ffffffff81069e10>] ? rescuer_thread+0x2e0/0x2e0 > [ 199.709861] [<ffffffff81069e10>] ? rescuer_thread+0x2e0/0x2e0 > [ 199.709861] [<ffffffff8106ec44>] kthread+0xc4/0xe0 > [ 199.709861] [<ffffffff8106eb80>] ? kthread_create_on_node+0x180/0x180 > [ 199.709861] [<ffffffff818ccc9f>] ret_from_fork+0x3f/0x70 > [ 199.709861] [<ffffffff8106eb80>] ? kthread_create_on_node+0x180/0x180 > [ 199.709861] Code: 8b 03 48 8b 7b 08 48 83 c3 10 4c 89 e6 ff d0 48 > 8b 03 48 85 c0 75 eb e9 32 ff ff ff 0f 1f 40 00 48 8b 87 f8 04 00 00 > 55 48 89 e5 <48> 8b 40 d8 5d c3 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 > 00 00 > [ 199.709861] RIP [<ffffffff8106f1cb>] kthread_data+0xb/0x20 > [ 199.709861] RSP <ffff88013a427188> > [ 199.709861] CR2: ffffffffffffffd8 > [ 199.709861] ---[ end trace d41ec1552544a209 ]--- > [ 199.709861] Fixing recursive fault but reboot is needed! > > > It looks like when underlaying device is gone, filesystem still trying > to write to it, and wasn't handle gracefully. > > Is there anyway we can fix it or probably add a queue for pending > actions for a relative long timeout? Just in case later the connection > would come back. > > And I would be glad to help working on this issue if needed. > > Thanks! > > > --Sheng > > > ------------------------------------------------------------------------------ > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > _______________________________________________ > Nbd-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/nbd-general >
------------------------------------------------------------------------------
_______________________________________________ Nbd-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nbd-general
