RE: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot
Jeremy -- FYI, I think I've also seen this problem, or something similar, but ONLY on a Nehalem box (and only intermittently), not on my Core 2 Duo boxen. The Nehalem box is an SDP so I had assumed that it was something to do with that, but maybe not. Maybe some feature is "leaking through" to the guest. Hyperthreading? MSI? EPT? Anyway, if you have a newer box, you might try reproducing on that, rather than your usual development box (which IIRC was a Core 2 Duo laptop?) > -Original Message- > From: Jeremy Fitzhardinge [mailto:jer...@goop.org] > Sent: Thursday, September 23, 2010 6:02 PM > To: Pawel Zuzelski > Cc: virtualizat...@lists.osdl.org; Jeremy Fitzhardinge; xen- > de...@lists.xensource.com; lkml > Subject: Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics > just after the boot > > On 09/21/2010 12:05 PM, Paweł Zuzelski wrote: > > Hello, > > > > kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining > of the > > boot process. > > > > I have bisected it to a single commit, and the first bad commit is: > > [fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts > for IPIs and VIRQs > > > > kernel v2.6.35.5 with reverted this commit works for me. > > Thanks very much for doing that. I'll have to work out what's going on > (obviously it doesn't do it for me). > > J > > > Here are the kernel configs I was using: > > http://carme.pld-linux.org/~pawelz/kernel-2.6.35.5-domU-config > > http://carme.pld-linux.org/~pawelz/kernel-2.6.32.22-domU-config > > As you can see they are stripped down configs, intended to run in > domU only. > > > > I was testing it with the very simple domU configuration: > > > > kernel = '/srv/xen/bzImage' > > memory = '128' > > vcpus = 2 > > name= 'test' > > on_poweroff = 'destroy' > > on_reboot = 'restart' > > on_crash= 'restart' > > > > Here is the full output of kernel 2.6.35.5: > > > > Using config file "/etc/xen/test". > > Started domain test > > [0.00] Policy zone: DMA32 > > [0.00] Kernel command line: > > [0.00] PID hash table entries: 512 (order: 0, 4096 bytes) > > [0.00] Subtract (33 early reservations) > > [0.00] #1 [0001976000 - 0001987000] XEN PAGETABLES > > [0.00] #2 [000100 - 00019125f8] TEXT DATA BSS > > [0.00] #3 [0001933000 - 0001976000] XEN START INFO > > [0.00] #4 [01 - 012000] TRAMPOLINE > > [0.00] #5 [012000 - 04] PGTABLE > > [0.00] #6 [0001912600 - 0001917600] NODE_DATA > > [0.00] #7 [0001917600 - 0001918600] BOOTMEM > > [0.00] #8 [0001918600 - 0001918618] BOOTMEM > > [0.00] #9 [0001919000 - 000191a000] BOOTMEM > > [0.00] #10 [000191a000 - 000191b000] BOOTMEM > > [0.00] #11 [000191b000 - 000191c000] BOOTMEM > > [0.00] #12 [000220 - 00023c]MEMMAP 0 > > [0.00] #13 [0001918640 - 00019187c0] BOOTMEM > > [0.00] #14 [000191c000 - 000191cc00] BOOTMEM > > [0.00] #15 [000191d000 - 000191e000] BOOTMEM > > [0.00] #16 [000191e000 - 000191f000] BOOTMEM > > [0.00] #17 [000191f000 - 000192] BOOTMEM > > [0.00] #18 [00019187c0 - 00019188a0] BOOTMEM > > [0.00] #19 [00019188c0 - 0001918928] BOOTMEM > > [0.00] #20 [0001918940 - 00019189a8] BOOTMEM > > [0.00] #21 [00019189c0 - 0001918a28] BOOTMEM > > [0.00] #22 [0001918a40 - 0001918a41] BOOTMEM > > [0.00] #23 [0001918a80 - 0001918a81] BOOTMEM > > [0.00] #24 [0001987000 - 00019c1000] BOOTMEM > > [0.00] #25 [0001918ac0 - 0001918ac8] BOOTMEM > > [0.00] #26 [0001918b00 - 0001918b08] BOOTMEM > > [0.00] #27 [0001918b40 - 0001918b48] BOOTMEM > > [0.00] #28 [0001918b80 - 0001918b90] BOOTMEM > > [0.00] #29 [0001918bc0 - 0001918cc0] BOOTMEM > > [0.00] #30 [0001918cc0 - 0001918d08] BOOTMEM > > [0.00] #31 [0001918d40 - 0001918d88] BOOTMEM > > [0.00] #32 [000192 - 0001921000] BOOTMEM > > [0.00] Memory: 118724k/131072k available (3327k kernel code, > 448k absent, 11900k reserved, 3931k data, 440k init) >
Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot
On 09/23/2010 07:13 PM, Dan Magenheimer wrote: > Jeremy -- > > FYI, I think I've also seen this problem, or something similar, > but ONLY on a Nehalem box (and only intermittently), not on my > Core 2 Duo boxen. The > Nehalem box is an SDP so I had assumed that it was something > to do with that, but maybe not. Maybe some feature is > "leaking through" to the guest. Hyperthreading? MSI? EPT? > > Anyway, if you have a newer box, you might try reproducing > on that, rather than your usual development box (which IIRC > was a Core 2 Duo laptop?) No, this bug turned out to be a simple typo in the patch that got merged into the upstream stable kernels. J >> -Original Message- >> From: Jeremy Fitzhardinge [mailto:jer...@goop.org] >> Sent: Thursday, September 23, 2010 6:02 PM >> To: Pawel Zuzelski >> Cc: virtualizat...@lists.osdl.org; Jeremy Fitzhardinge; xen- >> de...@lists.xensource.com; lkml >> Subject: Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics >> just after the boot >> >> On 09/21/2010 12:05 PM, Paweł Zuzelski wrote: >>> Hello, >>> >>> kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining >> of the >>> boot process. >>> >>> I have bisected it to a single commit, and the first bad commit is: >>> [fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts >> for IPIs and VIRQs >>> kernel v2.6.35.5 with reverted this commit works for me. >> Thanks very much for doing that. I'll have to work out what's going on >> (obviously it doesn't do it for me). >> >> J >> >>> Here are the kernel configs I was using: >>> http://carme.pld-linux.org/~pawelz/kernel-2.6.35.5-domU-config >>> http://carme.pld-linux.org/~pawelz/kernel-2.6.32.22-domU-config >>> As you can see they are stripped down configs, intended to run in >> domU only. >>> I was testing it with the very simple domU configuration: >>> >>> kernel = '/srv/xen/bzImage' >>> memory = '128' >>> vcpus = 2 >>> name= 'test' >>> on_poweroff = 'destroy' >>> on_reboot = 'restart' >>> on_crash= 'restart' >>> >>> Here is the full output of kernel 2.6.35.5: >>> >>> Using config file "/etc/xen/test". >>> Started domain test >>> [0.00] Policy zone: DMA32 >>> [0.00] Kernel command line: >>> [0.00] PID hash table entries: 512 (order: 0, 4096 bytes) >>> [0.00] Subtract (33 early reservations) >>> [0.00] #1 [0001976000 - 0001987000] XEN PAGETABLES >>> [0.00] #2 [000100 - 00019125f8] TEXT DATA BSS >>> [0.00] #3 [0001933000 - 0001976000] XEN START INFO >>> [0.00] #4 [01 - 012000] TRAMPOLINE >>> [0.00] #5 [012000 - 04] PGTABLE >>> [0.00] #6 [0001912600 - 0001917600] NODE_DATA >>> [0.00] #7 [0001917600 - 0001918600] BOOTMEM >>> [0.00] #8 [0001918600 - 0001918618] BOOTMEM >>> [0.00] #9 [0001919000 - 000191a000] BOOTMEM >>> [0.00] #10 [000191a000 - 000191b000] BOOTMEM >>> [0.00] #11 [000191b000 - 000191c000] BOOTMEM >>> [0.00] #12 [000220 - 00023c]MEMMAP 0 >>> [0.00] #13 [0001918640 - 00019187c0] BOOTMEM >>> [0.00] #14 [000191c000 - 000191cc00] BOOTMEM >>> [0.00] #15 [000191d000 - 000191e000] BOOTMEM >>> [0.00] #16 [000191e000 - 000191f000] BOOTMEM >>> [0.00] #17 [000191f000 - 000192] BOOTMEM >>> [0.00] #18 [00019187c0 - 00019188a0] BOOTMEM >>> [0.00] #19 [00019188c0 - 0001918928] BOOTMEM >>> [0.00] #20 [0001918940 - 00019189a8] BOOTMEM >>> [0.00] #21 [00019189c0 - 0001918a28] BOOTMEM >>> [0.00] #22 [0001918a40 - 0001918a41] BOOTMEM >>> [0.00] #23 [0001918a80 - 0001918a81] BOOTMEM >>> [0.00] #24 [0001987000 - 00019c1000] BOOTMEM >>> [0.00] #25 [0001918ac0 - 0001918ac8] BOOTMEM >>> [0.00] #26 [0001918b00 - 0001918b08] BOOTMEM >>> [0.00] #27 [0001918b40 - 0001918b48] BOOTMEM >>> [0.00] #28 [0001918b80 - 0001918b90] BOOTM
Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot
On 09/23/2010 10:40 PM, Paweł Zuzelski wrote: > Have you seen the message from James Dingwall? > > He had found the typo in fb412a178502dc498430723b082a932f797e4763 > commit and sent the patch to lkml that fixes it. > > Please, see https://patchwork.kernel.org/patch/202282/ if you have > not received James' mail. That patch resolves the problem for me and > for James. > (!) Oh, well. Yes, that would explain it. I'd noticed the missing 'x', but assumed I'd just done it with an editor fat-finger... Looks like upstream is OK, so it crept in with the stable cherry-pick/backport to both stable-2.6.32/master and stable-2.6.35/master. J ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/virtualization
Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot
On Thu, 23 Sep 2010, Jeremy Fitzhardinge wrote: > On 09/21/2010 12:05 PM, Paweł Zuzelski wrote: > > Hello, > > > > kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining of the > > boot process. > > > > I have bisected it to a single commit, and the first bad commit is: > > [fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts for > > IPIs and VIRQs > > > > kernel v2.6.35.5 with reverted this commit works for me. > > Thanks very much for doing that. I'll have to work out what's going on > (obviously it doesn't do it for me). Have you seen the message from James Dingwall? He had found the typo in fb412a178502dc498430723b082a932f797e4763 commit and sent the patch to lkml that fixes it. Please, see https://patchwork.kernel.org/patch/202282/ if you have not received James' mail. That patch resolves the problem for me and for James. -- Regards, Paweł Zuzelski ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/virtualization
Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot
On 09/21/2010 12:05 PM, Paweł Zuzelski wrote: > Hello, > > kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining of the > boot process. > > I have bisected it to a single commit, and the first bad commit is: > [fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts for > IPIs and VIRQs > > kernel v2.6.35.5 with reverted this commit works for me. Thanks very much for doing that. I'll have to work out what's going on (obviously it doesn't do it for me). J > Here are the kernel configs I was using: > http://carme.pld-linux.org/~pawelz/kernel-2.6.35.5-domU-config > http://carme.pld-linux.org/~pawelz/kernel-2.6.32.22-domU-config > As you can see they are stripped down configs, intended to run in domU only. > > I was testing it with the very simple domU configuration: > > kernel = '/srv/xen/bzImage' > memory = '128' > vcpus = 2 > name= 'test' > on_poweroff = 'destroy' > on_reboot = 'restart' > on_crash= 'restart' > > Here is the full output of kernel 2.6.35.5: > > Using config file "/etc/xen/test". > Started domain test > [0.00] Policy zone: DMA32 > [0.00] Kernel command line: > [0.00] PID hash table entries: 512 (order: 0, 4096 bytes) > [0.00] Subtract (33 early reservations) > [0.00] #1 [0001976000 - 0001987000] XEN PAGETABLES > [0.00] #2 [000100 - 00019125f8] TEXT DATA BSS > [0.00] #3 [0001933000 - 0001976000] XEN START INFO > [0.00] #4 [01 - 012000] TRAMPOLINE > [0.00] #5 [012000 - 04] PGTABLE > [0.00] #6 [0001912600 - 0001917600] NODE_DATA > [0.00] #7 [0001917600 - 0001918600] BOOTMEM > [0.00] #8 [0001918600 - 0001918618] BOOTMEM > [0.00] #9 [0001919000 - 000191a000] BOOTMEM > [0.00] #10 [000191a000 - 000191b000] BOOTMEM > [0.00] #11 [000191b000 - 000191c000] BOOTMEM > [0.00] #12 [000220 - 00023c]MEMMAP 0 > [0.00] #13 [0001918640 - 00019187c0] BOOTMEM > [0.00] #14 [000191c000 - 000191cc00] BOOTMEM > [0.00] #15 [000191d000 - 000191e000] BOOTMEM > [0.00] #16 [000191e000 - 000191f000] BOOTMEM > [0.00] #17 [000191f000 - 000192] BOOTMEM > [0.00] #18 [00019187c0 - 00019188a0] BOOTMEM > [0.00] #19 [00019188c0 - 0001918928] BOOTMEM > [0.00] #20 [0001918940 - 00019189a8] BOOTMEM > [0.00] #21 [00019189c0 - 0001918a28] BOOTMEM > [0.00] #22 [0001918a40 - 0001918a41] BOOTMEM > [0.00] #23 [0001918a80 - 0001918a81] BOOTMEM > [0.00] #24 [0001987000 - 00019c1000] BOOTMEM > [0.00] #25 [0001918ac0 - 0001918ac8] BOOTMEM > [0.00] #26 [0001918b00 - 0001918b08] BOOTMEM > [0.00] #27 [0001918b40 - 0001918b48] BOOTMEM > [0.00] #28 [0001918b80 - 0001918b90] BOOTMEM > [0.00] #29 [0001918bc0 - 0001918cc0] BOOTMEM > [0.00] #30 [0001918cc0 - 0001918d08] BOOTMEM > [0.00] #31 [0001918d40 - 0001918d88] BOOTMEM > [0.00] #32 [000192 - 0001921000] BOOTMEM > [0.00] Memory: 118724k/131072k available (3327k kernel code, 448k > absent, 11900k reserved, 3931k data, 440k init) > [0.00] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, > CPUs=2, Nodes=1 > [0.00] Hierarchical RCU implementation. > [0.00]RCU-based detection of stalled CPUs is disabled. > [0.00]Verbose stalled-CPUs detection is disabled. > [0.00] NR_IRQS:2304 > [0.00] Console: colour dummy device 80x25 > [0.00] console [tty0] enabled > [0.00] console [hvc0] enabled > [0.00] installing Xen timer for CPU 0 > [0.00] BUG: unable to handle kernel NULL pointer dereference at (null) > [0.00] IP: [<(null)>] (null) > [0.00] PGD 0 > [0.00] Oops: 0010 [#1] SMP > [0.00] last sysfs file: > [0.00] CPU 0 > [0.00] Modules linked in: > [0.00] > [0.00] Pid: 0, comm: swapper Not tainted 2.6.35.5 #1 / > [0.00] RIP: e030:[<>] [<(null)>] (null) > [0.00] RSP: e02b:81601d70 EFLAGS: 00010082 > [0.00] RAX: 818fdb50 RBX: RCX: > > [0.00] RDX: RSI: 818c7958 RDI: > > [0.00] RBP: 81601d88 R08: ea1b22d8 R09: > 001a > [0.00] R10: R11: 6477 R12: > 81623280 > [0.00] R13: R14: ffea R15: > > [0.00] FS: () GS:880001987000() > knlGS: > [0.00] CS
