On Tue, Apr 01, 2014 at 06:16:16PM -0400, Dave Jones wrote: > On Tue, Apr 01, 2014 at 11:32:45AM -0700, Paul E. McKenney wrote: > > > > > Given that you have CONFIG_RCU_TRACE=y, could you please enable the > > > > following trace events and dump the trace before things hang? > > > > > > > > trace_event=rcu:rcu_grace_period,rcu:rcu_grace_period_init > > > > > > > > If it is not feasible to dump the trace before things hang, let me > > > > know, and I will work out some other diagnostic regime. > > > > > > I'll give that a shot when I get back in a few hours. > > ok, this is tricky, and I'm not sure how helpful the below is. > because I don't know when the hang is going to happen, in parallel to my > usual workload > I did this.. > while [ 1 ]; > do > cat trace > echo > trace > done > > and got this... > > > > # entries-in-buffer/entries-written: 7/7 #P:4 > # > # _-----=> irqs-off > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / delay > # TASK-PID CPU# |||| TIMESTAMP FUNCTION > # | | | |||| | | > <idle>-0 [000] dNs3 851.748475: rcu_grace_period: rcu_preempt > 21342 cpuend > <idle>-0 [000] dNs3 851.748477: rcu_grace_period: rcu_preempt > 21343 cpustart > <idle>-0 [000] dNs3 851.748477: rcu_grace_period: rcu_sched > 20080 cpuend > <idle>-0 [000] dN.2 851.748480: rcu_grace_period: rcu_preempt > 21343 cpuqs > rcu_preempt-9 [000] ...1 851.748485: rcu_grace_period: rcu_preempt > 21343 fqsstart > rcu_preempt-9 [000] ...1 851.748487: rcu_grace_period: rcu_preempt > 21343 fqsend > rcu_preempt-9 [000] ...1 851.748487: rcu_grace_period: rcu_preempt > 21343 fqswait > # tracer: nop > # > # entries-in-buffer/entries-written: 13/13 #P:4 > # > # _-----=> irqs-off > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / delay > # TASK-PID CPU# |||| TIMESTAMP FUNCTION > # | | | |||| | | > rcu_preempt-9 [003] d..2 851.878457: rcu_grace_period: rcu_preempt > 21349 end > rcu_preempt-9 [003] d..2 851.878459: rcu_grace_period: rcu_preempt > 21349 newreq > rcu_preempt-9 [003] ...1 851.878459: rcu_grace_period: rcu_preempt > 21349 reqwait > rcu_preempt-9 [003] d..2 851.878460: rcu_grace_period: rcu_preempt > 21350 start > rcu_preempt-9 [003] d..2 851.878461: rcu_grace_period: rcu_preempt > 21350 cpustart > rcu_preempt-9 [003] d..2 851.878462: rcu_grace_period_init: > rcu_preempt 21350 0 0 7 f > rcu_preempt-9 [003] ...1 851.878463: rcu_grace_period: rcu_preempt > 21350 fqswait > rcu_preempt-9 [003] d..2 851.878464: rcu_grace_period: rcu_preempt > 21350 cpuqs > <idle>-0 [000] dNs3 851.878482: rcu_grace_period: rcu_preempt > 21349 cpuend > <idle>-0 [000] dNs3 851.878483: rcu_grace_period: rcu_preempt > 21350 cpustart > <idle>-0 [000] dNs3 851.878484: rcu_grace_period: rcu_sched > 20086 cpuend > <idle>-0 [000] .N.2 851.878486: rcu_grace_period: rcu_sched > 20086 cpuqs > <idle>-0 [000] dN.2 851.878487: rcu_grace_period: rcu_preempt > 21350 cpuqs > > > followed by a half dozen 'empty' traces before it totally locked up.
Cool, thank you! Could you please also send the RCU CPU stall warning messages? Thanx, Paul -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/