On Tue, 25 Jun 2013, Prarit Bhargava wrote: > On 06/24/2013 09:57 AM, Thomas Gleixner wrote: > > On Tue, 18 Jun 2013, Prarit Bhargava wrote: > > > >> Similar panics reported during bringup here: > >> > >> http://lists.infradead.org/pipermail/linux-arm-kernel/2013-May/166205.html > >> http://lkml.org/lkml/2013/5/8/342 > >> > >> I've seen this a few times on 3.10 based kernels. > >> > >> [ 175.842027] Disabling non-boot CPUs ... > >> [ 475.827017] BUG: unable to handle kernel NULL pointer dereference at > >> 0000000000000048 > > > > That looks like a stale bit in tick_broadcast_force_mask. > > > > Does the patch below fix it? > > > > Thomas, > > Thanks for the patch. > > The reproducibility appears to be quite low. I'm seeing this roughly 1 time > every six hours of continuous system reboots. I'm testing right now with your > patch. I'll update the thread in a couple of days...
I have a proper version of that patch now along with an explanation of the failure. --------------------> Subject: tick: Make oneshot broadcast robust vs. CPU offlining From: Thomas Gleixner <[email protected]> Date: Wed, 26 Jun 2013 12:17:32 +0200 In periodic mode we remove offline cpus from the broadcast propagation mask. In oneshot mode we fail to do so. This was not a problem so far, but the recent changes to the broadcast propagation introduced a constellation which can result in a NULL pointer dereference. What happens is: CPU0 CPU1 idle() arch_idle() tick_broadcast_oneshot_control(OFF); set cpu1 in tick_broadcast_force_mask if (cpu_offline()) arch_cpu_dead() cpu_dead_cleanup(cpu1) cpu1 tickdevice pointer = NULL broadcast interrupt dereference cpu1 tickdevice pointer -> OOPS We dereference the pointer because cpu1 is still set in tick_broadcast_force_mask and tick_do_broadcast() expects a valid cpumask and therefor lacks any further checks. Remove the cpu from the tick_broadcast_force_mask before we set the tick device pointer to NULL. Also add a sanity check to the oneshot broadcast function, so we can detect such issues w/o crashing the machine. Reported-by: Prarit Bhargava <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> --- kernel/time/tick-broadcast.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) Index: tip/kernel/time/tick-broadcast.c =================================================================== --- tip.orig/kernel/time/tick-broadcast.c +++ tip/kernel/time/tick-broadcast.c @@ -522,6 +522,13 @@ again: cpumask_clear(tick_broadcast_force_mask); /* + * Sanity check. Catch the case where we try to broadcast to + * offline cpus. + */ + if (WARN_ON_ONCE(!cpumask_subset(tmpmask, cpu_online_mask))) + cpumask_and(tmpmask, tmpmask, cpu_online_mask); + + /* * Wakeup the cpus which have an expired event. */ tick_do_broadcast(tmpmask); @@ -761,10 +768,12 @@ void tick_shutdown_broadcast_oneshot(uns raw_spin_lock_irqsave(&tick_broadcast_lock, flags); /* - * Clear the broadcast mask flag for the dead cpu, but do not - * stop the broadcast device! + * Clear the broadcast masks for the dead cpu, but do not stop + * the broadcast device! */ cpumask_clear_cpu(cpu, tick_broadcast_oneshot_mask); + cpumask_clear_cpu(cpu, tick_broadcast_pending_mask); + cpumask_clear_cpu(cpu, tick_broadcast_force_mask); raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags); } -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
