That issue did seem to be load induced, nothing has changed on the system side since we did not update any packages. We will update the kernel to the latest trusty first and if it still persists to either the xenial kernel or newer
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1705349 Title: tgtd blocked with hitting hung_task_timeout_secs Status in linux package in Ubuntu: Incomplete Bug description: We see an issue where an IO thread of tgtd violates the hung task timeout and vmstat reports >20% wait on IO while we are not seeing a issue with the underlying hardware (iowait and svc time) inside IO stat. We already limited the kernel parameters vm.dirty_background_ratio=5 vm.dirty_ratio=10 to reduce the impact of massive dirty blocks hitting the storage which is still based on spinning rust in a RAID 10 array. This is based on kernel 3.13.0-96-generic with the deadline IO scheduler and we got the following stack trace: [25300644.048029] INFO: task tgtd:27827 blocked for more than 300 seconds. [25300644.048065] Not tainted 3.13.0-96-generic #143-Ubuntu [25300644.048100] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [25300644.048166] tgtd D ffff88103fc13180 0 27827 1 0x00000080 [25300644.048168] ffff88019cc61c78 0000000000000086 ffff880125791800 0000000000013180 [25300644.048171] ffff88019cc61fd8 0000000000013180 ffff880125791800 ffff88103fc13a18 [25300644.048173] ffff88207ffef6e8 ffff88019cc61d00 0000000000000002 ffffffff81151e90 [25300644.048176] Call Trace: [25300644.048178] [<ffffffff81151e90>] ? wait_on_page_read+0x60/0x60 [25300644.048190] [<ffffffff8172e71d>] io_schedule+0x9d/0x130 [25300644.048192] [<ffffffff81151e9e>] sleep_on_page+0xe/0x20 [25300644.048194] [<ffffffff8172ec98>] __wait_on_bit_lock+0x48/0xb0 [25300644.048196] [<ffffffff81151faa>] __lock_page+0x6a/0x70 [25300644.048198] [<ffffffff810adb10>] ? autoremove_wake_function+0x40/0x40 [25300644.048200] [<ffffffff8115d1c8>] write_cache_pages+0x2f8/0x4b0 [25300644.048202] [<ffffffff8115c740>] ? global_dirtyable_memory+0x50/0x50 [25300644.048205] [<ffffffff810dae48>] ? get_futex_key+0x1d8/0x2c0 [25300644.048207] [<ffffffff8107abfe>] ? send_signal+0x3e/0x80 [25300644.048209] [<ffffffff8115d3c0>] generic_writepages+0x40/0x60 [25300644.048212] [<ffffffff8115e58e>] do_writepages+0x1e/0x40 [25300644.048213] [<ffffffff81153559>] __filemap_fdatawrite_range+0x59/0x60 [25300644.048215] [<ffffffff8115365a>] filemap_write_and_wait_range+0x2a/0x70 [25300644.048217] [<ffffffff811fa79b>] blkdev_fsync+0x1b/0x50 [25300644.048219] [<ffffffff811f1841>] do_fsync+0x51/0x80 [25300644.048221] [<ffffffff811f1af3>] SyS_fdatasync+0x13/0x20 [25300644.048223] [<ffffffff8173aadd>] system_call_fastpath+0x1a/0x1f UPTIME report 16:30:01 up 292 days, 22:34, 0 users, load average: 9.89, 8.58, 5.33 FREE report total used free shared buffers cached Mem: 131777272 116935780 14841492 828 110629740 1123368 -/+ buffers/cache: 5182672 126594600 Swap: 7999484 9004 7990480 VMSTAT report procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 10 8 14493 108036 1097 0 0 7 227 0 0 0 0 100 0 0 0 10 8 14494 108036 1097 0 0 0 0 690 1149 0 0 78 22 0 0 10 8 14494 108036 1097 0 0 0 8 736 1442 0 0 78 22 0 IOSTAT report Linux 3.13.0-96-generic (cinder18) 07/19/2017 _x86_64_ (32 CPU) 07/19/2017 04:30:03 PM avg-cpu: %user %nice %system %iowait %steal %idle 0.35 0.00 0.08 0.07 0.00 99.50 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 3.97 0.40 10.89 1.75 63.16 11.51 0.00 0.16 0.13 0.16 0.01 0.01 sdb 51.40 107.25 2.20 15.89 215.29 7198.82 819.79 0.07 3.89 3.26 3.97 0.60 1.08 sdc 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.11 0.11 0.00 0.11 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 11.76 0.66 14.98 0.17 0.00 dm-1 0.00 0.00 0.01 14.86 0.19 63.13 8.52 0.00 0.15 3.10 0.15 0.00 0.01 dm-2 0.00 0.00 0.00 0.00 0.03 0.02 13.67 0.00 1.07 1.66 0.69 0.47 0.00 dm-18 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 3.51 3.51 4.00 3.51 0.00 dm-10 0.00 0.00 0.01 0.00 0.02 0.00 8.00 0.00 1.76 1.76 0.00 0.41 0.00 dm-28 0.00 0.00 0.00 0.02 0.00 0.06 8.00 0.00 1.39 5.76 1.20 0.06 0.00 dm-29 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 3.62 9.63 2.97 0.02 0.00 dm-30 0.00 0.00 0.00 0.01 0.01 0.03 8.00 0.00 2.99 2.84 3.03 0.03 0.00 dm-24 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.04 0.04 0.00 0.04 0.00 dm-25 0.00 0.00 0.00 0.25 0.00 0.99 8.00 0.00 6.27 3.23 6.27 0.00 0.00 dm-35 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 2.47 2.05 2.50 0.01 0.00 dm-36 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 1.98 2.69 1.71 0.09 0.00 dm-37 0.00 0.00 0.10 0.69 0.42 2.75 8.00 0.00 4.26 0.63 4.81 0.00 0.00 dm-38 0.00 0.00 0.00 0.01 0.01 0.85 188.30 0.00 2.40 4.21 1.99 0.89 0.00 dm-39 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 3.19 3.20 0.00 0.60 0.00 dm-40 0.00 0.00 0.02 0.00 0.08 0.00 8.00 0.00 0.20 0.20 0.00 0.01 0.00 dm-59 0.00 0.00 0.06 2.41 0.23 9.62 8.00 0.01 3.63 1.07 3.69 0.01 0.00 dm-60 0.00 0.00 0.00 0.02 0.01 0.09 8.00 0.00 3.72 16.50 1.75 0.16 0.00 dm-62 0.00 0.00 0.00 0.02 0.00 0.08 8.00 0.00 2.05 7.69 1.97 1.82 0.00 dm-63 0.00 0.00 0.03 1.29 0.10 5.16 8.00 0.01 3.96 10.41 3.84 0.20 0.03 dm-65 0.00 0.00 0.06 0.90 0.22 3.61 8.00 0.01 5.40 11.23 5.04 0.12 0.01 dm-104 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 2.86 8.67 2.20 0.02 0.00 dm-105 0.00 0.00 0.00 0.04 0.00 0.14 8.00 0.00 1.11 8.41 1.04 0.04 0.00 dm-106 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 3.85 7.98 3.41 0.02 0.00 dm-107 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 4.83 9.85 4.29 0.02 0.00 dm-108 0.00 0.00 0.00 0.03 0.00 0.12 8.00 0.00 1.95 15.02 1.79 0.08 0.00 dm-109 0.00 0.00 0.00 0.04 0.00 0.14 8.00 0.00 2.20 11.60 2.11 0.08 0.00 dm-116 0.00 0.00 0.00 0.01 0.00 0.04 8.00 0.00 9.27 3.84 9.34 0.01 0.00 dm-117 0.00 0.00 38.85 42.29 155.40 169.18 8.00 0.03 0.35 0.14 0.54 0.01 0.07 dm-118 0.00 0.00 0.00 25.91 0.00 103.63 8.00 0.09 3.31 10.21 3.31 0.00 0.00 dm-122 0.00 0.00 0.00 0.00 0.00 1.04 1006.91 0.00 0.29 2.25 0.26 0.16 0.00 dm-126 0.00 0.00 0.00 0.00 0.00 0.41 982.51 0.00 0.34 2.11 0.26 0.16 0.00 dm-130 0.00 0.00 0.00 0.00 0.00 2.07 1013.70 0.00 0.27 3.68 0.23 0.15 0.00 dm-134 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.61 2.61 0.00 0.44 0.00 dm-115 0.00 0.00 0.01 0.03 0.03 0.13 8.00 0.00 81.51 0.78 100.26 0.19 0.00 dm-119 0.00 0.00 0.79 1.82 3.15 7.28 8.00 0.16 60.81 1.02 86.69 0.05 0.01 dm-121 0.00 0.00 2.77 1.59 11.09 6.34 8.00 0.05 11.22 16.24 2.44 0.03 0.01 dm-149 0.00 0.00 0.00 0.01 0.00 0.03 8.00 0.00 36.92 12.62 38.03 0.02 0.00 dm-150 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 10.89 11.99 10.75 0.11 0.00 dm-151 0.00 0.00 0.00 0.01 0.00 0.04 8.00 0.00 78.01 12.63 80.68 0.08 0.00 dm-152 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.22 2.90 2.16 0.03 0.00 dm-153 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 71.05 11.97 77.41 1.03 0.00 dm-154 0.00 0.00 0.03 1.29 0.13 5.14 8.00 0.08 59.27 7.21 60.58 0.03 0.00 dm-55 0.00 0.00 4.23 0.87 16.93 3.49 8.00 0.01 1.71 1.68 1.86 0.01 0.00 dm-68 0.00 0.00 0.00 0.00 0.02 0.00 8.00 0.00 15.58 18.88 0.62 1.80 0.00 dm-96 0.00 0.00 0.00 4.94 0.00 19.74 8.00 0.02 3.18 6.69 3.18 0.00 0.00 dm-97 0.00 0.00 0.00 3.95 0.00 15.80 8.00 0.04 10.61 13.14 10.61 0.00 0.00 dm-98 0.00 0.00 0.00 0.08 0.00 0.32 8.00 0.00 51.66 1.12 51.67 0.01 0.00 dm-78 0.00 0.00 0.00 0.01 0.00 0.02 8.00 0.00 1.82 5.50 1.78 1.30 0.00 dm-79 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 3.78 32.34 2.20 0.15 0.00 dm-80 0.00 0.00 0.00 0.35 0.02 1.41 8.00 0.00 5.41 17.35 5.28 0.16 0.01 dm-81 0.00 0.00 0.00 0.00 0.00 0.02 8.00 0.00 7.98 8.75 7.90 0.03 0.00 dm-82 0.00 0.00 0.03 2.04 0.11 8.16 8.00 0.07 35.30 62.02 34.94 0.02 0.00 dm-83 0.00 0.00 0.60 4.38 2.39 17.52 8.00 0.04 8.01 0.38 9.05 0.00 0.00 dm-87 0.00 0.00 0.00 0.23 0.00 0.92 8.00 0.00 2.25 0.83 2.25 0.01 0.00 dm-20 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 1.75 1.75 0.00 1.75 0.00 dm-21 0.00 0.00 0.00 0.28 0.00 1.11 8.00 0.00 4.33 9.42 4.32 0.01 0.00 dm-26 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 10.76 10.76 0.00 10.76 0.00 dm-27 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 1.91 1.96 0.00 1.91 0.00 dm-75 0.00 0.00 0.00 0.16 0.00 0.65 8.00 0.00 3.74 4.58 3.73 0.14 0.00 dm-11 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 1.81 1.81 0.00 1.81 0.00 dm-49 0.00 0.00 0.00 0.23 0.00 0.90 8.00 0.00 4.34 11.38 4.34 0.01 0.00 dm-50 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 6.68 6.68 0.00 6.68 0.00 dm-52 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.61 2.63 0.00 2.61 0.00 dm-53 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.01 2.01 0.00 2.01 0.00 dm-61 0.00 0.00 0.00 0.03 0.00 0.12 8.00 0.00 2.20 1.83 2.20 0.02 0.00 dm-51 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 2.99 8.71 2.39 0.02 0.00 dm-54 0.00 0.00 0.00 0.01 0.01 0.03 8.00 0.00 3.42 3.13 3.47 0.02 0.00 dm-56 0.00 0.00 0.00 0.00 0.00 0.02 8.00 0.00 3.18 9.73 2.74 0.04 0.00 dm-72 0.00 0.00 0.01 0.07 0.03 0.29 8.00 0.00 12.51 0.40 13.57 0.00 0.00 dm-70 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.09 0.91 2.12 0.01 0.00 dm-71 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.46 3.10 2.43 0.02 0.00 dm-73 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 2.29 0.82 2.32 0.01 0.00 dm-47 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 1.17 1.17 0.00 1.17 0.00 dm-48 0.00 0.00 0.00 0.01 0.00 0.02 8.00 0.00 2.01 6.15 2.01 0.01 0.00 dm-88 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 4.75 4.76 0.00 4.75 0.00 dm-89 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.13 0.13 0.00 0.13 0.00 dm-90 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 3.96 3.97 0.00 3.96 0.00 dm-91 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 3.60 0.17 3.61 0.31 0.00 dm-92 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.54 0.54 0.00 0.54 0.00 dm-93 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.17 0.17 0.00 0.17 0.00 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.31 0.31 0.00 0.31 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.01 8.00 0.00 1.42 2.29 1.42 0.01 0.00 dm-6 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.13 0.13 0.00 0.13 0.00 dm-7 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.04 0.04 0.00 0.04 0.00 dm-8 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.15 0.15 0.00 0.15 0.00 dm-9 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.04 0.00 8.07 0.16 0.00 dm-17 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.29 0.29 0.00 0.29 0.00 dm-19 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.13 0.13 0.00 0.13 0.00 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.48 0.48 0.00 0.48 0.00 dm-12 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.99 0.22 0.99 0.00 0.00 dm-13 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.32 0.32 0.00 0.32 0.00 dm-14 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.26 0.26 0.00 0.26 0.00 dm-15 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.33 0.33 0.00 0.33 0.00 dm-16 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.28 0.04 0.28 0.01 0.00 dm-22 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.30 0.30 0.00 0.30 0.00 dm-23 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.09 0.09 0.00 0.09 0.00 dm-31 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.32 0.31 4.00 0.32 0.00 dm-32 0.00 0.00 0.05 0.00 0.21 1.66 66.75 0.00 0.39 0.10 4.16 0.09 0.00 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1705349/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
