On Fri, Mar 04, 2016 at 04:30:12PM +1100, Ross Green wrote: > On Fri, Feb 26, 2016 at 12:35 PM, Paul E. McKenney > <paul...@linux.vnet.ibm.com> wrote:
[ . . . ] > >> OK, so what wakeup path omits the sched_wakeup event? > >> > >> The sched_waking event looks to occur once in try_to_wake_up() and > >> once in try_to_wake_up_local(). Starting with try_to_wake_up(): > >> > >> o If the task is ->on_rq, ttwu_remote() is invoked: > >> > >> o This acquires the runqueue lock, then if > >> task_on_rq_queued() invokes ttwu_do_wakeup(). This > >> unconditionally does sched_wakeup, so we didn't go that > >> way. (And this path skips the bulk of try_to_wake_up() > >> on return.) > >> > >> o Otherwise, we release the runqueu lock and returns zero. > >> > >> o There is some ordering checking, runqueue selection, and then > >> p->state is set to TASK_WAKING. And we apparently are not getting > >> here, either. But I don't see any other way out. > >> > >> Ignoring this for the moment... > >> > >> We eventually reach to the call to ttwu_queue(). > >> > >> o Here the TTWU_QUEUE path seems to avoid doing a > >> sched_wakeup event -- and since we are trying to wake > >> CPU 0 from CPU 4, so they don't share cache (x86). > >> > >> o This invokes ttwu_queue_remote(), which sends an IPI > >> unless polling is in effect. I would need to enable > >> trace_sched_wake_idle_without_ipi() to see whether or > >> not the IPI was actually sent. > >> > >> If the target CPU was offline, we should have seen the > >> cpu_is_offline() WARN_ON(). I suppose that the CPU might > >> go offline between the check and the ->send_IPI_mask(), > >> but only once. And we are trying to wakeup on CPU 0 > >> quite a few times. > >> > >> Any thoughts on what to look for? > >> > >> Next, try_to_wake_up_local(): > >> > >> o After doing several checks, it does the sched_waking event. > >> > >> o If the task is already queued, it calls ttwu_activate(). > >> > >> o It then invokes ttwu_do_wakeup(), which unconditionally > >> does the sched_wakeup() event. > >> > >> So this path looks unlikely, even ignoring the fact that > >> the waking CPU in the traces above is always different than > >> the CPU to be awakened on. > >> > >> Any thoughts? > >> > >> Thanx, Paul > G'day, > > > Here is a series of rcu_preempt stall events(5) from linux-4.5-rc6 release. > > Again some testing procedure. boot, run series of brief benchmarks and > then leave idle. > The first stall event appeared quite quickly - within hours, the rest > at what appears to be random intervals after that. > > > I thought I might give Daniels patch set a try and see how that goes! Looks like the same issue from dmesg. For my part, I added more tracing, which seems to have further decreased the probability of occurrence. The sched_wake_idle_without_ipi event did not appear. My next step is to try writing a torture test focused specifically on this issue. We need a faster reproducer to make decent progress. Thanx, Paul