Re: BUG: scheduling while atomic: swapper/0/0x00000002
On Wed, Jun 09, 2010 at 06:08:54PM -0700, Paul E. McKenney wrote: On Thu, Jun 10, 2010 at 09:20:08AM +1000, Benjamin Herrenschmidt wrote: On Wed, 2010-06-09 at 14:52 -0700, Paul E. McKenney wrote: Hello! I get the following during boot on a 16 CPU Power box. Thoughts? (/proc/config attached) Wow... looks like the preempt count of the idle task got busted or something ... how reproduceable ? Something like a record of previous interrupts might be useful.. I have seen it only once, but it did get my attention. I will try running it again this evening when/if kernel-ml8 is free again. 2.6.35-rc2, I should have mentioned. And it does appear to be reproducible perhaps 50% of boots running CONFIG_PREEMPT on kernel-ml8. I have not yet seen it on any other system. Do you have a patch to instrument the interrupts so as to get the info you need, or should I improvise? Thanx, Paul ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: BUG: scheduling while atomic: swapper/0/0x00000002
On Mon, 2010-06-28 at 09:49 -0700, Paul E. McKenney wrote: And it does appear to be reproducible perhaps 50% of boots running CONFIG_PREEMPT on kernel-ml8. I have not yet seen it on any other system. Do you have a patch to instrument the interrupts so as to get the info you need, or should I improvise? ftrace friends ? I don't have anything existing in mind... Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
BUG: scheduling while atomic: swapper/0/0x00000002
Hello! I get the following during boot on a 16 CPU Power box. Thoughts? (/proc/config attached) Thanx, Paul UDP hash table entries: 2048 (order: 6, 262144 bytes) UDP-Lite hash table entries: 2048 (order: 6, 262144 bytes) NET: Registered protocol family 1 RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. Trying to unpack rootfs image as initramfs... Freeing initrd memory: 2455k freed BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 audit: initializing netlink socket (disabled) type=2000 audit(1276043741.388:1): initialized BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: # # Automatically generated make config: don't edit # Linux kernel version: 2.6.35-rc2-autokern1 # Wed Jun 9 00:29:21 2010 # CONFIG_PPC64=y # # Processor support # CONFIG_PPC_BOOK3S_64=y # CONFIG_PPC_BOOK3E_64 is not set CONFIG_PPC_BOOK3S=y # CONFIG_POWER4_ONLY is not set CONFIG_POWER3=y CONFIG_POWER4=y # CONFIG_TUNE_CELL is not set CONFIG_PPC_FPU=y CONFIG_ALTIVEC=y # CONFIG_VSX is not set CONFIG_PPC_STD_MMU=y CONFIG_PPC_STD_MMU_64=y CONFIG_PPC_MM_SLICES=y CONFIG_VIRT_CPU_ACCOUNTING=y CONFIG_PPC_HAVE_PMU_SUPPORT=y CONFIG_PPC_PERF_CTRS=y CONFIG_SMP=y CONFIG_NR_CPUS=8 CONFIG_64BIT=y CONFIG_WORD_SIZE=64 CONFIG_ARCH_PHYS_ADDR_T_64BIT=y CONFIG_MMU=y CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_GENERIC_TIME=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y CONFIG_HAVE_SETUP_PER_CPU_AREA=y CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y CONFIG_IRQ_PER_CPU=y CONFIG_NR_IRQS=512 CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_GENERIC_LOCKBREAK=y CONFIG_ARCH_HAS_ILOG2_U32=y CONFIG_ARCH_HAS_ILOG2_U64=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_FIND_NEXT_BIT=y CONFIG_ARCH_NO_VIRT_TO_BUS=y CONFIG_PPC=y CONFIG_EARLY_PRINTK=y CONFIG_COMPAT=y CONFIG_SYSVIPC_COMPAT=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_PPC_OF=y CONFIG_OF=y CONFIG_PPC_UDBG_16550=y # CONFIG_GENERIC_TBSYNC is not set CONFIG_AUDIT_ARCH=y CONFIG_GENERIC_BUG=y CONFIG_DTC=y # CONFIG_DEFAULT_UIMAGE is not set CONFIG_ARCH_HIBERNATION_POSSIBLE=y # CONFIG_PPC_DCR_NATIVE is not set # CONFIG_PPC_DCR_MMIO is not set # CONFIG_PPC_OF_PLATFORM_PCI is not set
Re: BUG: scheduling while atomic: swapper/0/0x00000002
On Wed, 2010-06-09 at 14:52 -0700, Paul E. McKenney wrote: Hello! I get the following during boot on a 16 CPU Power box. Thoughts? (/proc/config attached) Wow... looks like the preempt count of the idle task got busted or something ... how reproduceable ? Something like a record of previous interrupts might be useful.. Cheers, Ben. Thanx, Paul UDP hash table entries: 2048 (order: 6, 262144 bytes) UDP-Lite hash table entries: 2048 (order: 6, 262144 bytes) NET: Registered protocol family 1 RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. Trying to unpack rootfs image as initramfs... Freeing initrd memory: 2455k freed BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 audit: initializing netlink socket (disabled) type=2000 audit(1276043741.388:1): initialized BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: BUG: scheduling while atomic: swapper/0/0x00000002
On Thu, Jun 10, 2010 at 09:20:08AM +1000, Benjamin Herrenschmidt wrote: On Wed, 2010-06-09 at 14:52 -0700, Paul E. McKenney wrote: Hello! I get the following during boot on a 16 CPU Power box. Thoughts? (/proc/config attached) Wow... looks like the preempt count of the idle task got busted or something ... how reproduceable ? Something like a record of previous interrupts might be useful.. I have seen it only once, but it did get my attention. I will try running it again this evening when/if kernel-ml8 is free again. 2.6.35-rc2, I should have mentioned. Thanx, Paul Cheers, Ben. Thanx, Paul UDP hash table entries: 2048 (order: 6, 262144 bytes) UDP-Lite hash table entries: 2048 (order: 6, 262144 bytes) NET: Registered protocol family 1 RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. Trying to unpack rootfs image as initramfs... Freeing initrd memory: 2455k freed BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 audit: initializing netlink socket (disabled) type=2000 audit(1276043741.388:1): initialized BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: Call Trace: [c0008e1afc20] [c0011990] .show_stack+0x70/0x184 (unreliable) [c0008e1afcd0] [c005cb60] .__schedule_bug+0xac/0xd0 [c0008e1afd70] [c05f0c34] .schedule+0xc0/0x894 [c0008e1afe40] [c00142d0] .cpu_idle+0x1f8/0x20c [c0008e1afed0] [c0600b08] .start_secondary+0x3bc/0x3fc [c0008e1aff90] [c0008264] .start_secondary_prolog+0x10/0x14 BUG: scheduling while atomic: swapper/0/0x0002 no locks held by swapper/0. Modules linked in: ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
