Hi,
we recently updated our kernel to 4.1.16 + patch for "unix: properly
account for FDs passed over unix sockets" and have since then
self-detected stalls triggered by the Samba daemon:
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] INFO: rcu_sched self-detected
> stall on CPU { 3} (t=162780 jiffies g=47565 c=47564 q=1055670)
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] Task dump for CPU 3:
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] smbd R running task
> 0 5938 1 0x0000000c
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] 0000000000000004
> ffffffff81851340 ffffffff810d3c84 000000000000b9cd
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] ffff8801bfd97100
> ffffffff81851340 ffffffff81851340 ffffffff818f6c60
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] ffffffff810d7659
> 0000000000000000 0000000000000000 00001e847fc2f700
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] Call Trace:
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] <IRQ> [<ffffffff810d3c84>] ?
> rcu_dump_cpu_stacks+0x84/0xc0
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810d7659>] ?
> rcu_check_callbacks+0x449/0x740
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ec7c0>] ?
> tick_sched_do_timer+0x40/0x40
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810dcc54>] ?
> update_process_times+0x34/0x70
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ec45c>] ?
> tick_sched_handle.isra.12+0x2c/0x70
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ec809>] ?
> tick_sched_timer+0x49/0x80
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810dd57d>] ?
> __run_hrtimer+0x6d/0x1b0
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ddd4d>] ?
> hrtimer_interrupt+0xed/0x210
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff815a0ed9>] ?
> smp_apic_timer_interrupt+0x39/0x50
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8159ef7e>] ?
> apic_timer_interrupt+0x6e/0x80
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] <EOI> [<ffffffff8159de85>] ?
> _raw_spin_lock+0x35/0x50
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8153b343>] ?
> unix_dgram_connect+0x93/0x200
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8147f248>] ?
> SYSC_connect+0xe8/0x100
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8159e0f2>] ?
> system_call_fast_compare_end+0xc/0x6b
> Feb 1 11:48:13 ucs22f kernel: [307999.162254] INFO: rcu_sched self-detected
> stall on CPU { 0} (t=5250 jiffies
> g=6586733 c=6586732 q=6757)
> Feb 1 11:48:13 ucs22f kernel: [307999.162264] Task dump for CPU 0:
> Feb 1 11:48:13 ucs22f kernel: [307999.162267] smbd R running
> 0 4615 4609 0x00000008
> Feb 1 11:48:13 ucs22f kernel: [307999.162272] 00200082 f5863b90 c10b3fe9
> c1682cc0 c1682cc0 c1682cc0 f79d2b00 f
> 5863bdc
> Feb 1 11:48:13 ucs22f kernel: [307999.162276] c10b722d c15bd400 00001482
> 0064816d 0064816c 00001a65 f79cd840 c
> 108b166
> Feb 1 11:48:13 ucs22f kernel: [307999.162280] 00000001 f5863bb8 f5863bb8
> 00000000 c1682cc0 f6808cf0 00001a65 f
> 6808cf0
> Feb 1 11:48:13 ucs22f kernel: [307999.162285] Call Trace:
> Feb 1 11:48:13 ucs22f kernel: [307999.162296] [<c10b3fe9>] ?
> rcu_dump_cpu_stacks+0x79/0xc0
> Feb 1 11:48:13 ucs22f kernel: [307999.162300] [<c10b722d>] ?
> rcu_check_callbacks+0x3cd/0x630
> Feb 1 11:48:13 ucs22f kernel: [307999.162304] [<c108b166>] ?
> account_process_tick+0x66/0x160
> Feb 1 11:48:13 ucs22f kernel: [307999.162307] [<c10bbe4f>] ?
> update_process_times+0x2f/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162310] [<c10cbf9d>] ?
> tick_sched_handle.isra.12+0x2d/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162328] [<c10cc210>] ?
> tick_sched_timer+0x40/0x80
> Feb 1 11:48:13 ucs22f kernel: [307999.162331] [<c10bc6b0>] ?
> __remove_hrtimer+0x40/0xa0
> Feb 1 11:48:13 ucs22f kernel: [307999.162334] [<c10bc97f>] ?
> __run_hrtimer+0x6f/0x190
> Feb 1 11:48:13 ucs22f kernel: [307999.162337] [<c10cc1d0>] ?
> tick_sched_do_timer+0x30/0x30
> Feb 1 11:48:13 ucs22f kernel: [307999.162339] [<c10bd15f>] ?
> hrtimer_interrupt+0xef/0x260
> Feb 1 11:48:13 ucs22f kernel: [307999.162343] [<c119ae3d>] ?
> getname_kernel+0x2d/0x100
> Feb 1 11:48:13 ucs22f kernel: [307999.162348] [<c1046f7f>] ?
> local_apic_timer_interrupt+0x2f/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162353] [<c14e4543>] ?
> smp_apic_timer_interrupt+0x33/0x50
> Feb 1 11:48:13 ucs22f kernel: [307999.162355] [<c14e3c7c>] ?
> apic_timer_interrupt+0x34/0x3c
> Feb 1 11:48:13 ucs22f kernel: [307999.162358] [<c14e2dc1>] ?
> _raw_spin_lock+0x51/0x70
> Feb 1 11:48:13 ucs22f kernel: [307999.162362] [<c148c075>] ?
> unix_state_double_lock+0x25/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162365] [<c148de10>] ?
> unix_dgram_connect+0x90/0x1f0
> Feb 1 11:48:13 ucs22f kernel: [307999.162369] [<c13e4267>] ?
> SYSC_connect+0xc7/0xe0
> Feb 1 11:48:13 ucs22f kernel: [307999.162371] [<c13e2931>] ?
> sock_map_fd+0x41/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162374] [<c13e5014>] ?
> SYSC_socketcall+0x1b4/0xa20
> Feb 1 11:48:13 ucs22f kernel: [307999.162376] [<c10c2940>] ?
> ktime_get+0x50/0x100
> Feb 1 11:48:13 ucs22f kernel: [307999.162379] [<c10466db>] ?
> lapic_next_event+0x1b/0x20
> Feb 1 11:48:13 ucs22f kernel: [307999.162381] [<c10ca0ed>] ?
> clockevents_program_event+0x9d/0x140
> Feb 1 11:48:13 ucs22f kernel: [307999.162385] [<c129e068>] ?
> list_del+0x8/0x20
> Feb 1 11:48:13 ucs22f kernel: [307999.162388] [<c1097ef7>] ?
> remove_wait_queue+0x27/0x40
> Feb 1 11:48:13 ucs22f kernel: [307999.162392] [<c11c8795>] ?
> inotify_read+0x295/0x340
> Feb 1 11:48:13 ucs22f kernel: [307999.162396] [<c10acc76>] ?
> handle_irq_event_percpu+0xa6/0x1a0
> Feb 1 11:48:13 ucs22f kernel: [307999.162399] [<c11a786f>] ?
> set_close_on_exec+0x2f/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162402] [<c119d084>] ?
> do_fcntl+0x2f4/0x4e0
> Feb 1 11:48:13 ucs22f kernel: [307999.162405] [<c107d6df>] ?
> commit_creds+0xff/0x1f0
> Feb 1 11:48:13 ucs22f kernel: [307999.162407] [<c119d380>] ?
> SyS_fcntl64+0x60/0x100
> Feb 1 11:48:13 ucs22f kernel: [307999.162409] [<c13e5953>] ?
> SyS_socketcall+0x13/0x20
> Feb 1 11:48:13 ucs22f kernel: [307999.162412] [<c14e30db>] ?
> sysenter_do_call+0x12/0x12
We have not yet been able to reproduce the hang, but going back to our
previous kernel 4.1.12 makes the problem go away.
Is this a known issue or do you have an idea where to look?
What information should I collect next time it happens?
(Can unix_diag.ko with `ss` help?)
What other kernel configs should I enable do debug this dead-lock?
Thanks in advance
Philipp
