On Sat, Dec 13, 2014 at 02:40:51PM -0800, Linus Torvalds wrote: > On Sat, Dec 13, 2014 at 2:36 PM, Dave Jones <da...@redhat.com> wrote: > > > > Ok, I think we can rule out preemption. I just checked on it, and > > found it wedged. > > Ok, one more. Mind checking what happens without CONFIG_DEBUG_PAGEALLOC?
Crap. Looks like it wedged. It's stuck that way until I get back to it on Wednesday. [ 6188.985536] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c175:14205] [ 6188.985612] CPU: 1 PID: 14205 Comm: trinity-c175 Not tainted 3.18.0+ #103 [loadavg: 200.63 151.07 150.40 179/407 17316] [ 6188.985652] task: ffff880056ac96d0 ti: ffff8800975d8000 task.ti: ffff8800975d8000 [ 6188.985680] RIP: 0010:[<ffffffff810c6430>] [<ffffffff810c6430>] lock_release+0xc0/0x240 [ 6188.985714] RSP: 0018:ffff8800975dbaa8 EFLAGS: 00000292 [ 6188.985734] RAX: ffff880056ac96d0 RBX: ffff8800975dbaf0 RCX: 00000000000003a0 [ 6188.985759] RDX: ffff88024500dd20 RSI: 0000000000000000 RDI: ffff880056ac9e40 [ 6188.985785] RBP: ffff8800975dbad8 R08: 0000000000000000 R09: 0000000000000000 [ 6188.985810] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000292 [ 6188.985835] R13: ffff8800975dba28 R14: 0000000000000292 R15: 0000000000000292 [ 6188.985861] FS: 00007f107fc69740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000 [ 6188.985890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6188.985912] CR2: 00007fff7457af40 CR3: 00000000145ed000 CR4: 00000000001407e0 [ 6188.985937] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6188.985963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [ 6188.985988] Stack: [ 6188.986000] ffff8800975dbb08 0000000000000000 0000000000000000 00000000001ce380 [ 6188.986034] ffff8800975dbd08 ffff8802451ce380 ffff8800975dbbd8 ffffffff8116f928 [ 6188.986067] ffffffff8116f842 ffff8800975dbaf0 ffff8800975dbaf0 0000000000000001 [ 6188.986101] Call Trace: [ 6188.986116] [<ffffffff8116f928>] __perf_sw_event+0x168/0x240 [ 6188.987079] [<ffffffff8116f842>] ? __perf_sw_event+0x82/0x240 [ 6188.988045] [<ffffffff81178ab2>] ? __lock_page_or_retry+0xb2/0xc0 [ 6188.989008] [<ffffffff811a68f8>] ? handle_mm_fault+0x458/0xe90 [ 6188.989986] [<ffffffff8104250e>] __do_page_fault+0x28e/0x5c0 [ 6188.990940] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6188.991884] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6188.992826] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6188.993773] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6188.994715] [<ffffffff8104284c>] do_page_fault+0xc/0x10 [ 6188.995658] [<ffffffff817d1a32>] page_fault+0x22/0x30 [ 6188.996590] [<ffffffff81375266>] ? __clear_user+0x36/0x60 [ 6188.997518] [<ffffffff81375247>] ? __clear_user+0x17/0x60 [ 6188.998440] [<ffffffff8100f3f1>] save_xstate_sig+0x81/0x220 [ 6188.999362] [<ffffffff817cf1cf>] ? _raw_spin_unlock_irqrestore+0x4f/0x60 [ 6189.000291] [<ffffffff810029e7>] do_signal+0x5c7/0x740 [ 6189.001220] [<ffffffff81209acf>] ? mnt_drop_write+0x2f/0x40 [ 6189.002164] [<ffffffff811e527e>] ? chmod_common+0xfe/0x150 [ 6189.003096] [<ffffffff81002bc5>] do_notify_resume+0x65/0x80 [ 6189.004038] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6189.004972] [<ffffffff817d00ff>] int_signal+0x12/0x17 [ 6189.005899] Code: ff 0f 85 7c 00 00 00 4c 89 ea 4c 89 e6 48 89 df e8 26 fc ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 41 56 9d <48> 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d f3 c3 65 ff 04 25 e0 [ 6189.007935] sending NMI to other CPUs: [ 6189.008904] NMI backtrace for cpu 2 [ 6189.009755] CPU: 2 PID: 14224 Comm: trinity-c194 Not tainted 3.18.0+ #103 [loadavg: 200.63 151.07 150.40 179/407 17316] [ 6189.010618] task: ffff880224af5b40 ti: ffff880225aec000 task.ti: ffff880225aec000 [ 6189.011555] RIP: 0010:[<ffffffff81176890>] [<ffffffff81176890>] pagecache_get_page+0x0/0x220 [ 6189.012501] RSP: 0018:ffff880225aefb50 EFLAGS: 00000282 [ 6189.013442] RAX: ffff88023f4b9d00 RBX: 00007fff7457b07f RCX: 0000000000000000 [ 6189.014396] RDX: 0000000000000000 RSI: 000000000001dda7 RDI: ffffffff81c6aa80 [ 6189.015357] RBP: ffff880225aefb58 R08: 0000000000000000 R09: 0000000007769c80 [ 6189.016317] R10: 0000000000000000 R11: 0000000000000029 R12: ffff88022a63be00 [ 6189.017283] R13: ffff8801c64afd10 R14: ffff880000000bd8 R15: ffff880187efea40 [ 6189.018253] FS: 00007f107fc69740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000 [ 6189.019247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6189.020245] CR2: 00007fff7457b07f CR3: 000000022789e000 CR4: 00000000001407e0 [ 6189.021230] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6189.022193] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [ 6189.023155] Stack: [ 6189.024104] ffffffff811b99da ffff880225aefbf8 ffffffff811a68a1 ffff880225aefbd8 [ 6189.025084] 0000000000000246 ffffffff81042418 ffffffff00000000 0000000000000000 [ 6189.026063] 0000000000000246 0000000100000000 ffff880187efeb58 0000000000000080 [ 6189.027025] Call Trace: [ 6189.027962] [<ffffffff811b99da>] ? lookup_swap_cache+0x2a/0x70 [ 6189.028897] [<ffffffff811a68a1>] handle_mm_fault+0x401/0xe90 [ 6189.029819] [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0 [ 6189.030731] [<ffffffff8104247c>] __do_page_fault+0x1fc/0x5c0 [ 6189.031635] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6189.032537] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6189.033432] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6189.034334] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6189.035238] [<ffffffff8104284c>] do_page_fault+0xc/0x10 [ 6189.036146] [<ffffffff817d1a32>] page_fault+0x22/0x30 [ 6189.037043] [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220 [ 6189.037934] [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220 [ 6189.038819] [<ffffffff810029e7>] do_signal+0x5c7/0x740 [ 6189.039699] [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40 [ 6189.040583] [<ffffffff81002bc5>] do_notify_resume+0x65/0x80 [ 6189.041464] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6189.042340] [<ffffffff817d00ff>] int_signal+0x12/0x17 [ 6189.043210] Code: f0 80 a6 81 48 89 df e8 7f a5 02 00 0f 0b 48 89 df e8 45 fd ff ff 48 89 df e8 8d e4 00 00 eb 83 66 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 41 57 45 89 c7 41 56 49 89 f6 41 55 [ 6189.045130] NMI backtrace for cpu 3 [ 6189.045244] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 36.225 msecs [ 6189.046980] CPU: 3 PID: 14076 Comm: trinity-c46 Not tainted 3.18.0+ #103 [loadavg: 200.63 151.07 150.40 181/407 17316] [ 6189.047934] task: ffff88008a6c4470 ti: ffff8801cdb58000 task.ti: ffff8801cdb58000 [ 6189.048893] RIP: 0010:[<ffffffff810c63a4>] [<ffffffff810c63a4>] lock_release+0x34/0x240 [ 6189.049867] RSP: 0000:ffff8801cdb5bad0 EFLAGS: 00000296 [ 6189.050834] RAX: ffff88008a6c4470 RBX: ffff88013f39e4a8 RCX: 00000000000003a0 [ 6189.051815] RDX: ffffffff81178a7f RSI: 0000000000000001 RDI: ffff88013f39e518 [ 6189.052799] RBP: ffff8801cdb5bb08 R08: 0000000000000000 R09: 00000000073e8480 [ 6189.053781] R10: ffffea0007927a80 R11: 0000000000000029 R12: ffff88013f39e518 [ 6189.054764] R13: ffffffff81178a7f R14: ffff880000000bd8 R15: 0000000000000001 [ 6189.055748] FS: 00007f107fc69740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000 [ 6189.056746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6189.057751] CR2: 00007fff7457b07f CR3: 0000000226f46000 CR4: 00000000001407e0 [ 6189.058770] DR0: 00007f322081b000 DR1: 0000000000[ 6216.969357] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c175:14205] [ 6216.970331] CPU: 1 PID: 14205 Comm: trinity-c175 Tainted: G L 3.18.0+ #103 [loadavg: 221.85 160.04 153.39 183/407 17316] [ 6216.971359] task: ffff880056ac96d0 ti: ffff8800975d8000 task.ti: ffff8800975d8000 [ 6216.972366] RIP: 0010:[<ffffffff817cf1b8>] [<ffffffff817cf1b8>] _raw_spin_unlock_irqrestore+0x38/0x60 [ 6216.973391] RSP: 0018:ffff8800975dba18 EFLAGS: 00000292 [ 6216.974423] RAX: 0000000000000001 RBX: ffff880056ac96d0 RCX: 0000000000005040 [ 6216.975459] RDX: ffff88024502f580 RSI: 0000000000000000 RDI: ffff88024e581e28 [ 6216.976507] RBP: ffff8800975dba28 R08: 0000000000000000 R09: ffff8800975dbaf0 [ 6216.977551] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000046 [ 6216.978594] R13: ffff8800975db9e8 R14: 0000000000000000 R15: 0000000000000000 [ 6216.979635] FS: 00007f107fc69740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000 [ 6216.980686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6216.981735] CR2: 00007fff7457af40 CR3: 00000000145ed000 CR4: 00000000001407e0 [ 6216.982774] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6216.983792] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [ 6216.984799] Stack: [ 6216.985785] ffff8800975dbad8 ffff8800975dbaf0 ffff8800975dba58 ffffffff810bcd56 [ 6216.986789] ffff8800975dbac0 ffff8800975dbad8 ffff88024e581e28 0000000000000082 [ 6216.987794] ffff8800975dbaa8 ffffffff817c9f4e ffff8800975dba78 000000010073e800 [ 6216.988770] Call Trace: [ 6216.989729] [<ffffffff810bcd56>] finish_wait+0x56/0x70 [ 6216.990693] [<ffffffff817c9f4e>] __wait_on_bit+0x7e/0x90 [ 6216.991661] [<ffffffff811789d7>] wait_on_page_bit_killable+0xc7/0xf0 [ 6216.992632] [<ffffffff810bd050>] ? autoremove_wake_function+0x40/0x40 [ 6216.993609] [<ffffffff81178ab2>] __lock_page_or_retry+0xb2/0xc0 [ 6216.994586] [<ffffffff811a6e5c>] handle_mm_fault+0x9bc/0xe90 [ 6216.995555] [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0 [ 6216.996516] [<ffffffff8104247c>] __do_page_fault+0x1fc/0x5c0 [ 6216.997468] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6216.998428] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6216.999386] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6217.000330] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6217.001269] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6217.002205] [<ffffffff8104284c>] do_page_fault+0xc/0x10 [ 6217.003136] [<ffffffff817d1a32>] page_fault+0x22/0x30 [ 6217.004055] [<ffffffff81375266>] ? __clear_user+0x36/0x60 [ 6217.004972] [<ffffffff81375247>] ? __clear_user+0x17/0x60 [ 6217.005882] [<ffffffff8100f3f1>] save_xstate_sig+0x81/0x220 [ 6217.006800] [<ffffffff817cf1cf>] ? _raw_spin_unlock_irqrestore+0x4f/0x60 [ 6217.007716] [<ffffffff810029e7>] do_signal+0x5c7/0x740 [ 6217.008635] [<ffffffff81209acf>] ? mnt_drop_write+0x2f/0x40 [ 6217.009555] [<ffffffff811e527e>] ? chmod_common+0xfe/0x150 [ 6217.010470] [<ffffffff81002bc5>] do_notify_resume+0x65/0x80 [ 6217.011382] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6217.012297] [<ffffffff817d00ff>] int_signal+0x12/0x17 [ 6217.013210] Code: fc 48 8b 55 08 53 48 8d 7f 18 48 89 f3 be 01 00 00 00 e8 cc 71 8f ff 4c 89 e7 e8 f4 a4 8f ff f6 c7 02 74 17 e8 0a b0 97 ff 53 9d <5b> 65 ff 0c 25 e0 a9 00 00 41 5c 5d c3 0f 1f 00 53 9d e8 f1 ae [ 6217.015229] sending NMI to other CPUs: [ 6217.016191] NMI backtrace for cpu 3 [ 6217.017110] CPU: 3 PID: 14076 Comm: trinity-c46 Tainted: G L 3.18.0+ #103 [loadavg: 221.85 160.04 153.39 183/407 17316] [ 6217.018066] task: ffff88008a6c4470 ti: ffff8801cdb58000 task.ti: ffff8801cdb58000 [ 6217.019021] RIP: 0010:[<ffffffff810c5071>] [<ffffffff810c5071>] __lock_acquire.isra.31+0x1b1/0x9f0 [ 6217.019997] RSP: 0000:ffff8801cdb5b9d8 EFLAGS: 00000083 [ 6217.020972] RAX: 000000000000001e RBX: ffff88008a6c4470 RCX: 0000000000000002 [ 6217.021966] RDX: 0000000000000157 RSI: 0000000000000008 RDI: 0000000000000000 [ 6217.022961] RBP: ffff8801cdb5ba48 R08: 0000000000000000 R09: 0000000000000000 [ 6217.023953] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001d [ 6217.024945] R13: 0000000000000001 R14: ffffffff81c50e60 R15: ffff88008a6c4c18 [ 6217.025939] FS: 00007f107fc69740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000 [ 6217.026943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6217.027943] CR2: 00007fff7457b07f CR3: 0000000226f46000 CR4: 00000000001407e0 [ 6217.028933] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6217.029896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [ 6217.030846] Stack: [ 6217.031772] ffff88008a6c4470 ffff88024e52e040 ffff8801cdb5ba28 0000000000000092 [ 6217.032708] ffff8801cdb5ba08 ffffffff810abab5 ffff8801cdb5ba88 ffffffff810c50ec [ 6217.033625] 0000000000000296 0000000000000246 0000000000000000 0000000000000000 [ 6217.034537] Call Trace: [ 6217.035425] [<ffffffff810abab5>] ? local_clock+0x25/0x30 [ 6217.036316] [<ffffffff810c50ec>] ? __lock_acquire.isra.31+0x22c/0x9f0 [ 6217.037210] [<ffffffff810c5fbf>] lock_acquire+0x9f/0x120 [ 6217.038103] [<ffffffff811766d5>] ? find_get_entry+0x5/0x120 [ 6217.038995] [<ffffffff81176717>] find_get_entry+0x47/0x120 [ 6217.039891] [<ffffffff811766d5>] ? find_get_entry+0x5/0x120 [ 6217.040776] [<ffffffff811768bf>] pagecache_get_page+0x2f/0x220 [ 6217.041653] [<ffffffff8116f842>] ? __perf_sw_event+0x82/0x240 [ 6217.042527] [<ffffffff811b99da>] lookup_swap_cache+0x2a/0x70 [ 6217.043399] [<ffffffff811a68a1>] handle_mm_fault+0x401/0xe90 [ 6217.044273] [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0 [ 6217.045140] [<ffffffff8104247c>] __do_page_fault+0x1fc/0x5c0 [ 6217.045999] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6217.046857] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6217.047713] [<ffffffff81042358>] ? __do_page_fault+0xd8/0x5c0 [ 6217.048562] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6217.049414] [<ffffffff8104284c>] do_page_fault+0xc/0x10 [ 6217.050263] [<ffffffff817d1a32>] page_fault+0x22/0x30 [ 6217.051108] [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220 [ 6217.051953] [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220 [ 6217.052787] [<ffffffff810029e7>] do_signal+0x5c7/0x740 [ 6217.053620] [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40 [ 6217.054457] [<ffffffff81002bc5>] do_notify_resume+0x65/0x80 [ 6217.055294] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6217.056133] [<ffffffff817d00ff>] int_signal+0x12/0x17 [ 6217.056977] Code: ea 48 8d 34 d5 00 00 00 00 48 c1 e2 06 48 29 f2 4c 8d bc 13 70 07 00 00 41 0f b7 57 f8 81 e2 ff 1f 00 00 39 d0 0f 84 3f 01 00 00 <41> 0f b7 57 30 66 25 ff 1f 4c 89 4d c8 41 c1 e2 07 83 e1 03 4d [ 6217.058855] NMI backtrace for cpu 2 [ 6217.059739] CPU: 2 PID: 14224 Comm: trinity-c194 Tainted: G L 3.18.0+ #103 [loadavg: 221.85 160.04 153.39 183/407 17316] [ 6217.060662] task: ffff880224af5b40 ti: ffff880225aec000 task.ti: ffff880225aec000 [ 6217.061589] RIP: 0010:[<ffffffff810c6396>] [<ffffffff810c6396>] lock_release+0x26/0x240 [ 6217.062533] RSP: 0018:ffff880225aefa10 EFLAGS: 00000046 [ 6217.063477] RAX: ffff880224af5b40 RBX: 0000000000000296 RCX: 0000000000000002 [ 6217.064435] RDX: ffffffff810bcd56 RSI: 0000000000000001 RDI: ffff88024e54da40 [ 6217.065392] RBP: ffff880225aefa28 R08: 0000000000000000 R09: ffff880225aefb10 [ 6217.066342] R[ 6217.158570] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 142.293 msecs [ 6225.814243] INFO: rcu_sched self-detected stall on CPU [ 6225.815127] 3: (5990 ticks this GP) idle=f83/140000000000001/0 softirq=390686/390686 [ 6225.816000] (t=6000 jiffies g=166553 c=166552 q=0) [ 6225.816870] Task dump for CPU 3: [ 6225.817736] trinity-c46 R running task 13568 14076 13551 0x1000000c [ 6225.818626] ffff88008a6c4470 00000000ef08e0c8 ffff880245403d68 ffffffff810a73fc [ 6225.819531] ffffffff810a7362 0000000000000003 0000000000000008 0000000000000003 [ 6225.820432] ffffffff81c523c0 0000000000000092 ffff880245403d88 ffffffff810ab4ad [ 6225.821340] Call Trace: [ 6225.822218] <IRQ> [<ffffffff810a73fc>] sched_show_task+0x11c/0x190 [ 6225.823122] [<ffffffff810a7362>] ? sched_show_task+0x82/0x190 [ 6225.824021] [<ffffffff810ab4ad>] dump_cpu_task+0x3d/0x50 [ 6225.824916] [<ffffffff810d9bf0>] rcu_dump_cpu_stacks+0x90/0xd0 [ 6225.825813] [<ffffffff810e0783>] rcu_check_callbacks+0x503/0x770 [ 6225.826697] [<ffffffff811304dc>] ? acct_account_cputime+0x1c/0x20 [ 6225.827581] [<ffffffff810abde7>] ? account_system_time+0x97/0x180 [ 6225.828464] [<ffffffff810e645b>] update_process_times+0x4b/0x80 [ 6225.829350] [<ffffffff810f6e13>] ? tick_sched_timer+0x23/0x1b0 [ 6225.830233] [<ffffffff810f6e3f>] tick_sched_timer+0x4f/0x1b0 [ 6225.831108] [<ffffffff810e72ff>] __run_hrtimer+0xaf/0x240 [ 6225.831977] [<ffffffff810e76eb>] ? hrtimer_interrupt+0x16b/0x260 [ 6225.832844] [<ffffffff810f6df0>] ? tick_init_highres+0x20/0x20 [ 6225.833709] [<ffffffff810e7687>] hrtimer_interrupt+0x107/0x260 [ 6225.834565] [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70 [ 6225.835384] [<ffffffff817d28c5>] smp_apic_timer_interrupt+0x45/0x60 [ 6225.836203] [<ffffffff817d0caf>] apic_timer_interrupt+0x6f/0x80 [ 6225.837023] <EOI> [<ffffffff810c50ec>] ? __lock_acquire.isra.31+0x22c/0x9f0 [ 6225.837858] [<ffffffff810c5fd4>] ? lock_acquire+0xb4/0x120 [ 6225.838688] [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0 [ 6225.839517] [<ffffffff810c225a>] down_read_trylock+0x5a/0x60 [ 6225.840345] [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0 [ 6225.841175] [<ffffffff81042418>] __do_page_fault+0x198/0x5c0 [ 6225.842004] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6225.842836] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6225.843672] [<ffffffff81042358>] ? __do_page_fault+0xd8/0x5c0 [ 6225.844506] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6225.845340] [<ffffffff8104284c>] do_page_fault+0xc/0x10 [ 6225.846174] [<ffffffff817d1a32>] page_fault+0x22/0x30 [ 6225.847002] [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220 [ 6225.847827] [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220 [ 6225.848648] [<ffffffff810029e7>] do_signal+0x5c7/0x740 [ 6225.849468] [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40 [ 6225.850287] [<ffffffff81002bc5>] do_notify_resume+0x65/0x80 [ 6225.851104] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6225.851927] [<ffffffff817d00ff>] int_signal+0x12/0x17 [ 6225.852746] INFO: rcu_sched detected stalls on CPUs/tasks: [ 6225.853609] 3: (5991 ticks this GP) idle=f83/140000000000000/0 softirq=390686/390686 [ 6225.854481] (detected by 1, t=6004 jiffies, g=166553, c=166552, q=0) [ 6225.855354] Task dump for CPU 3: [ 6225.856225] trinity-c46 R running task 13568 14076 13551 0x1000000c [ 6225.857127] ffffffff810bcd38 ffff88008a6c4470 ffff8801cdb5b9c8 ffffffff810abab5 [ 6225.858045] ffff88024e52e040 0000000000000046 ffff88008a6c4470 ffff88024e52e040 [ 6225.858951] ffff8801cdb5ba28 0000000000000092 ffffffff810bcb37 ffffffff810abab5 [ 6225.859860] Call Trace: [ 6225.860761] [<ffffffff810c35df>] ? lock_release_holdtime.part.24+0xf/0x190 [ 6225.861690] [<ffffffff810abab5>] ? local_clock+0x25/0x30 [ 6225.862612] [<ffffffff810c35df>] lock_release_holdtime.part.24+0xf/0x190 [ 6225.863543] [<ffffffff810abab5>] ? local_clock+0x25/0x30 [ 6225.864473] [<ffffffff810c50ec>] ? __lock_acquire.isra.31+0x22c/0x9f0 [ 6225.865402] [<ffffffff810bcd56>] ? finish_wait+0x56/0x70 [ 6225.866329] [<ffffffff817c9f4e>] ? __wait_on_bit+0x7e/0x90 [ 6225.867236] [<ffffffff811789d7>] ? wait_on_page_bit_killable+0xc7/0xf0 [ 6225.868122] [<ffffffff810bd050>] ? autoremove_wake_function+0x40/0x40 [ 6225.868996] [<ffffffff811b99da>] ? lookup_swap_cache+0x2a/0x70 [ 6225.869855] [<ffffffff811a68f8>] ? handle_mm_fault+0x458/0xe90 [ 6225.870706] [<ffffffff810c225a>] ? down_read_trylock+0x5a/0x60 [ 6225.871545] [<ffffffff8104247c>] ? __do_page_fault+0x1fc/0x5c0 [ 6225.872385] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6225.873219] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6225.874046] [<ffffffff81042358>] ? __do_page_fault+0xd8/0x5c0 [ 6225.874873] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6225.875692] [<ffffffff8104284c>] ? do_page_fault+0xc/0x10 [ 6225.876503] [<ffffffff817d1a32>] ? page_fault+0x22/0x30 [ 6225.877307] [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220 [ 6225.878107] [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220 [ 6225.878901] [<ffffffff810029e7>] ? do_signal+0x5c7/0x740 [ 6225.879695] [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40 [ 6225.880496] [<ffffffff81002bc5>] ? do_notify_resume+0x65/0x80 [ 6225.881292] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6225.882097] [<ffffffff817d00ff>] ? int_signal+0x12/0x17 [ 6244.953181] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [trinity-c194:14224] [ 6244.953995] CPU: 2 PID: 14224 Comm: trinity-c194 Tainted: G L 3.18.0+ #103 [loadavg: 238.82 170.06 156.93 185/407 17316] [ 6244.954854] task: ffff880224af5b40 ti: ffff880225aec000 task.ti: ffff880225aec000 [ 6244.955699] RIP: 0010:[<ffffffff810c5f60>] [<ffffffff810c5f60>] lock_acquire+0x40/0x120 [ 6244.956560] RSP: 0018:ffff880225aefb78 EFLAGS: 00000246 [ 6244.957418] RAX: ffff880224af5b40 RBX: ffff8802453ce380 RCX: 0000000000000001 [ 6244.958281] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000 [ 6244.959146] RBP: ffff880225aefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 6244.960008] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000001ce380 [ 6244.960867] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880225aefb28 [ 6244.961716] FS: 00007f107fc69740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000 [ 6244.962571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6244.963429] CR2: 00007fff7457b07f CR3: 000000022789e000 CR4: 00000000001407e0 [ 6244.964296] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6244.965160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [ 6244.966025] Stack: [ 6244.966881] 0000000005080021 ffffffff00000000 0000000000000000 0000000000000246 [ 6244.967776] 0000000100000000 ffff880187efeb58 0000000000000000 0000000000000029 [ 6244.968678] 00007fff7457b07f ffff880225aefd28 0000000000000002 ffff880187efea40 [ 6244.969564] Call Trace: [ 6244.970445] [<ffffffff810c225a>] down_read_trylock+0x5a/0x60 [ 6244.971336] [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0 [ 6244.972226] [<ffffffff81042418>] __do_page_fault+0x198/0x5c0 [ 6244.973125] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6244.974021] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6244.974913] [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310 [ 6244.975800] [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe [ 6244.976685] [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 6244.977573] [<ffffffff8104284c>] do_page_fault+0xc/0x10 [ 6244.978455] [<ffffffff817d1a32>] page_fault+0x22/0x30 [ 6244.979327] [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220 [ 6244.980190] [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220 [ 6244.981045] [<ffffffff810029e7>] do_signal+0x5c7/0x740 [ 6244.981892] [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40 [ 6244.982743] [<ffffffff81002bc5>] do_notify_resume+0x65/0x80 [ 6244.983582] [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 6244.984425] [<ffffffff817d00ff>] int_signal+0x12/0x17 [ 6244.985266] Code: 65 48 8b 04 25 00 aa 00 00 8b b8 6c 07 00 00 44 89 45 c4 85 ff 0f 85 84 00 00 00 41 89 f4 41 89 d5 41 89 ce 4d 89 cf 9c 8f 45 b8 <fa> c7 80 6c 07 00 00 01 00 00 00 0f 1f 44 00 00 65 ff 04 25 e0 [ 6244.987120] sending NMI to other CPUs: -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
