Hello,
we've got problems with kernel 3.4. It hangs during stability tests.
Messages "INFO: rcu_sched self detected stall on CPU..." are still sent
to the console but network is down and system doesn't react on key press.
Test creates volume group with two logical volumes and 20 snapshots of
size five times smaller than LV. Snapshots are periodically started and
stopped while logical volumes are filled by dd processes. XFS filesystem
is used. It happens on different machines.
Already we have captured two different call traces (the second one
happened twice also on 3.4.69):
34519.493785] INFO: rcu_sched self-detected stall on CPU { 6} (t=6000 jiffies)
[34519.493791] Pid: 39642, comm: mount Tainted: G O
3.4.69-oe64-00000-g8e19b63 #1
[34519.493793] Call Trace:
[34519.493794]<IRQ> [<ffffffff8100fcca>] ? update_vsyscall+0xaa/0xd0
[34519.493803] [<ffffffff8109e34e>] ? __rcu_pending+0xbe/0x420
[34519.493805] [<ffffffff8109e6cf>] ? rcu_pending+0x1f/0x50
[34519.493807] [<ffffffff8109eea5>] ? rcu_check_callbacks+0x25/0x60
[34519.493810] [<ffffffff81048e1f>] ? update_process_times+0x3f/0x80
[34519.493814] [<ffffffff8107439b>] ? tick_sched_timer+0x5b/0xa0
[34519.493825] [<ffffffff8105b348>] ? __run_hrtimer+0x68/0x100
[34519.493827] [<ffffffff81074340>] ? tick_do_update_jiffies64+0xc0/0xc0
[34519.493829] [<ffffffff8105b4b7>] ? hrtimer_interrupt+0xd7/0x230
[34519.493833] [<ffffffff810269c3>] ? smp_apic_timer_interrupt+0x63/0xa0
[34519.493837] [<ffffffff815fbc0a>] ? apic_timer_interrupt+0x6a/0x70
[34519.493838]<EOI> [<ffffffff8102e1e4>] ? __ticket_spin_lock+0x14/0x20
[34519.493844] [<ffffffff815f51fb>] ? _raw_spin_lock_irqsave+0x2b/0x50
[34519.493846] [<ffffffff815f51c5>] ? _raw_spin_lock+0x5/0x10
[34519.493849] [<ffffffff812b76f8>] ? _atomic_dec_and_lock+0x48/0x70
[34519.493852] [<ffffffff8121b969>] ? xfs_buf_rele+0x39/0xc0
[34519.493855] [<ffffffff8121c8d1>] ? xfs_flush_buftarg+0x101/0x120
[34519.493857] [<ffffffff8121c91a>] ? xfs_free_buftarg+0x2a/0x60
[34519.493859] [<ffffffff81229b18>] ? xfs_fs_fill_super+0x178/0x280
[34519.493863] [<ffffffff810ee0a9>] ? mount_bdev+0x1b9/0x1f0
[34519.493864] [<ffffffff812299a0>] ? xfs_parseargs+0xbc0/0xbc0
[34519.493868] [<ffffffff810db46c>] ? alloc_pages_current+0xac/0x120
[34519.493870] [<ffffffff810edcde>] ? mount_fs+0x3e/0x180
[34519.493874] [<ffffffff81107c95>] ? vfs_kern_mount+0x65/0xf0
[34519.493877] [<ffffffff81107da3>] ? do_kern_mount+0x53/0x110
[34519.493879] [<ffffffff811083b4>] ? do_mount+0x554/0x790
[34519.493883] [<ffffffff8112ed39>] ? compat_sys_mount+0xa9/0x240
[34519.493885] [<ffffffff815fcbd9>] ? ia32_do_call+0x13/0x13
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913816] INFO: rcu_sched self-detected
stall on CPU { 4} (t=7647 jiffies)
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913819] Pid: 12889, comm: syslog-ng
Tainted: G O 3.4.58-oe64-00000-g46d0e40 #15
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913821] Call Trace:
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913822]<IRQ> [<ffffffff8100fcca>] ?
update_vsyscall+0xaa/0xd0
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913827] [<ffffffff8109e2fe>] ?
__rcu_pending+0xbe/0x420
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913829] [<ffffffff8109e67f>] ?
rcu_pending+0x1f/0x50
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913830] [<ffffffff8109ee55>] ?
rcu_check_callbacks+0x25/0x60
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913832] [<ffffffff81048e1f>] ?
update_process_times+0x3f/0x80
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913834] [<ffffffff8107434b>] ?
tick_sched_timer+0x5b/0xa0
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913836] [<ffffffff8105b328>] ?
__run_hrtimer+0x68/0x100
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913838] [<ffffffff810742f0>] ?
tick_do_update_jiffies64+0xc0/0xc0
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913839] [<ffffffff8105b497>] ?
hrtimer_interrupt+0xd7/0x230
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913841] [<ffffffff810269c3>] ?
smp_apic_timer_interrupt+0x63/0xa0
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913844] [<ffffffff815fb3ca>] ?
apic_timer_interrupt+0x6a/0x70
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913844]<EOI> [<ffffffff812c23f0>] ?
__const_udelay+0x40/0x40
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913848] [<ffffffff8103df0b>] ?
console_unlock+0x19b/0x280
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913850] [<ffffffff8136007d>] ?
do_con_write+0x63d/0x1d80
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913853] [<ffffffff810fca80>] ?
__pollwait+0x120/0x120
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913855] [<ffffffff810f00ff>] ?
copy_strings+0x19f/0x200
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913856] [<ffffffff810fca80>] ?
__pollwait+0x120/0x120
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913858] [<ffffffff81361816>] ?
con_write+0x16/0x40
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913860] [<ffffffff8134da2d>] ?
n_tty_write+0x1cd/0x3c0
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913862] [<ffffffff810630d0>] ?
try_to_wake_up+0x280/0x280
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913863] [<ffffffff81349dab>] ?
tty_write+0x19b/0x260
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913865] [<ffffffff8134d860>] ?
n_tty_ioctl+0xc0/0xc0
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913866] [<ffffffff810eac60>] ?
vfs_write+0xd0/0x170
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913868] [<ffffffff810eb413>] ?
sys_write+0x53/0x90
Dec 13 14:13:27 [kern.err] kernel: [ 5493.913870] [<ffffffff815fc399>] ?
ia32_do_call+0x13/0x13
Does anyone have idea what is the cause of this issues?
--
Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
