On Thu, Sep 06, 2012 at 05:19:18PM +0200, Peter Zijlstra wrote:
> On Thu, 2012-09-06 at 11:07 -0400, Steven Rostedt wrote:
> > On Thu, 2012-09-06 at 16:56 +0200, Peter Zijlstra wrote:
> > > On Thu, 2012-08-30 at 11:56 -0700, Paul E. McKenney wrote:
> > > >
> > > > If a given CPU avoids the idle loop but also avoids starting a new
> > > > RCU grace period for a full minute, RCU can issue spurious RCU CPU
> > > > stall warnings. This commit fixes this issue by adding a check for
> > > > ongoing grace period to avoid these spurious stall warnings.
> > >
> > > How would it avoid starting a new period for over a minute? fqs should
> > > happen, right? And holding rcu_read_lock() for over a minute surely is a
> > > bug.
> >
> > I can see this happening in test cases, but it would seem weird on a
> > normal system. That is, for preempt rcu, having a process scheduled out
> > holding an rcu_read_lock() for over a minute could happen on a really
> > stressed out system. But for such a case, I don't think a warning is out
> > of question.
>
> One would hope that fqs would boost things.. but yeah, if your app is
> spinning above the rcu boost prio you're still toast. But in that case
> you're right, a warning is fully deserved.
Here are a few other ways that stalls can happen:
o A CPU looping in an RCU read-side critical section.
o A CPU looping with interrupts disabled. This condition can
result in RCU-sched and RCU-bh stalls.
o A CPU looping with preemption disabled. This condition can
result in RCU-sched stalls and, if ksoftirqd is in use, RCU-bh
stalls.
o A CPU looping with bottom halves disabled. This condition can
result in RCU-sched and RCU-bh stalls.
o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel
without invoking schedule().
o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
happen to preempt a low-priority task in the middle of an RCU
read-side critical section. This is especially damaging if
that low-priority task is not permitted to run on any other CPU,
in which case the next RCU grace period can never complete, which
will eventually cause the system to run out of memory and hang.
While the system is in the process of running itself out of
memory, you might see stall-warning messages.
o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
is running at a higher priority than the RCU softirq threads.
This will prevent RCU callbacks from ever being invoked,
and in a CONFIG_TREE_PREEMPT_RCU kernel will further prevent
RCU grace periods from ever completing. Either way, the
system will eventually run out of memory and hang. In the
CONFIG_TREE_PREEMPT_RCU case, you might see stall-warning
messages.
o A hardware or software issue shuts off the scheduler-clock
interrupt on a CPU that is not in dyntick-idle mode. This
problem really has happened, and seems to be most likely to
result in RCU CPU stall warnings for CONFIG_NO_HZ=n kernels.
o A bug in the RCU implementation.
o A hardware failure. This is quite unlikely, but has occurred
at least once in real life. A CPU failed in a running system,
becoming unresponsive, but not causing an immediate crash.
This resulted in a series of RCU CPU stall warnings, eventually
leading the realization that the CPU had failed.
Thanx, Paul
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
