Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-08-22 Thread Avi Kivity
On 08/20/2011 04:16 PM, Brad Campbell wrote: Author: Alexander Duyck alexander.h.du...@intel.com Date: Thu Jul 1 13:28:27 2010 + x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN This patch removes the CONFIG_MCORE2 check from around NET_IP_ALIGN. It is based on

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-08-22 Thread Eric Dumazet
Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit : On 08/20/2011 04:16 PM, Brad Campbell wrote: Author: Alexander Duyck alexander.h.du...@intel.com Date: Thu Jul 1 13:28:27 2010 + x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN This patch removes the

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-08-20 Thread Brad Campbell
On 07/06/11 21:37, Eric Dumazet wrote: Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit : On 07/06/11 04:22, Eric Dumazet wrote: Could you please try latest linux-2.6 tree ? We fixed many networking bugs that could explain your crash. No good I'm afraid. [ 543.040056]

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-12 Thread Avi Kivity
On 06/10/2011 05:52 AM, Simon Horman wrote: At one point I would have need an 8000km long wire to the reset switch :-) Even more off-topic, there has been a case when a 200,000,000 km long wire to the reset button was needed. IIRC they got away with a watchdog. -- error compiling

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-10 Thread Mark Lord
On 11-06-09 10:52 PM, Simon Horman wrote: On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote: On 08/06/11 11:59, Eric Dumazet wrote: Well, a bisection definitely should help, but needs a lot of time in your case. Yes. compile, test, crash, walk out to the other building to press

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-10 Thread Henrique de Moraes Holschuh
On Fri, 10 Jun 2011, Mark Lord wrote: Something many of us don't realize is that nearly all Intel chipsets have a built-in hardware watchdog timer. This includes chipset for consumer desktop boards as well as the big iron server stuff. It's the i8xx_tco driver in the kernel enables use of

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-09 Thread Simon Horman
On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote: On 08/06/11 11:59, Eric Dumazet wrote: Well, a bisection definitely should help, but needs a lot of time in your case. Yes. compile, test, crash, walk out to the other building to press reset, lather, rinse, repeat. I need

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-08 Thread Brad Campbell
On 08/06/11 11:59, Eric Dumazet wrote: Well, a bisection definitely should help, but needs a lot of time in your case. Yes. compile, test, crash, walk out to the other building to press reset, lather, rinse, repeat. I need a reset button on the end of a 50M wire, or a hardware watchdog!

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-08 Thread Eric Dumazet
Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit : On 08/06/11 11:59, Eric Dumazet wrote: Well, a bisection definitely should help, but needs a lot of time in your case. Yes. compile, test, crash, walk out to the other building to press reset, lather, rinse, repeat. I need

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Brad Campbell
On 07/06/11 04:22, Eric Dumazet wrote: Could you please try latest linux-2.6 tree ? We fixed many networking bugs that could explain your crash. No good I'm afraid. [ 543.040056] = [ 543.040136] BUG

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Patrick McHardy
On 07.06.2011 05:33, Brad Campbell wrote: On 07/06/11 04:10, Bart De Schuymer wrote: Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't directly see an answer to in the threads I scanned... I'm assuming you actually use the

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Eric Dumazet
Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit : On 07/06/11 04:22, Eric Dumazet wrote: Could you please try latest linux-2.6 tree ? We fixed many networking bugs that could explain your crash. No good I'm afraid. [ 543.040056]

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Brad Campbell
On 07/06/11 21:30, Patrick McHardy wrote: On 07.06.2011 05:33, Brad Campbell wrote: On 07/06/11 04:10, Bart De Schuymer wrote: Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't directly see an answer to in the threads I

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Patrick McHardy
On 07.06.2011 16:40, Brad Campbell wrote: On 07/06/11 21:30, Patrick McHardy wrote: On 07.06.2011 05:33, Brad Campbell wrote: On 07/06/11 04:10, Bart De Schuymer wrote: Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Bart De Schuymer
Op 7/06/2011 16:40, Brad Campbell schreef: On 07/06/11 21:30, Patrick McHardy wrote: On 07.06.2011 05:33, Brad Campbell wrote: On 07/06/11 04:10, Bart De Schuymer wrote: Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Eric Dumazet
Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit : The main suspects would be NAT and TCPMSS. Did you also try whether the crash occurs with only one of these these rules? I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access the address the way I was doing

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Patrick McHardy
On 07.06.2011 20:31, Eric Dumazet wrote: Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit : The main suspects would be NAT and TCPMSS. Did you also try whether the crash occurs with only one of these these rules? I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Brad Campbell
On 08/06/11 02:04, Bart De Schuymer wrote: If the bug is easily triggered with your guest os, then you could try to capture the traffic with wireshark (or something else) in a configuration that doesn't crash your system. Save the traffic in a pcap file. Then you can see if resending that

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Brad Campbell
On 08/06/11 06:57, Patrick McHardy wrote: On 07.06.2011 20:31, Eric Dumazet wrote: Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit : The main suspects would be NAT and TCPMSS. Did you also try whether the crash occurs with only one of these these rules? I've just compiled out

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Brad Campbell
On 07/06/11 23:35, Patrick McHardy wrote: The main suspects would be NAT and TCPMSS. Did you also try whether the crash occurs with only one of these these rules? To be honest I'm actually having trouble finding where TCPMSS is actually set in that ruleset. This is a production machine so I

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-07 Thread Eric Dumazet
Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit : On 08/06/11 06:57, Patrick McHardy wrote: On 07.06.2011 20:31, Eric Dumazet wrote: Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit : The main suspects would be NAT and TCPMSS. Did you also try whether the crash

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-06 Thread Bart De Schuymer
Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't directly see an answer to in the threads I scanned... I'm assuming you actually use the bridging firewall functionality. So, what iptables modules do you use? Can you reduce

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-06 Thread Eric Dumazet
Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit : On 05/06/11 16:14, Avi Kivity wrote: On 06/03/2011 04:38 PM, Brad Campbell wrote: Is there anyone who can point me at the appropriate cage to rattle? I know it appears to be a netfilter issue, but I don't seem to be able to

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-06 Thread Eric Dumazet
Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit : Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't directly see an answer to in the threads I scanned... I'm assuming you actually use the bridging firewall

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-06 Thread Brad Campbell
On 07/06/11 04:10, Bart De Schuymer wrote: Hi Brad, This has probably nothing to do with ebtables, so please rmmod in case it's loaded. A few questions I didn't directly see an answer to in the threads I scanned... I'm assuming you actually use the bridging firewall functionality. So, what

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-05 Thread Avi Kivity
On 06/03/2011 04:38 PM, Brad Campbell wrote: Is there anyone who can point me at the appropriate cage to rattle? I know it appears to be a netfilter issue, but I don't seem to be able to get a message to the list (and I am subscribed to it and have been getting mail for months) and I'm not

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-05 Thread Brad Campbell
On 05/06/11 16:14, Avi Kivity wrote: On 06/03/2011 04:38 PM, Brad Campbell wrote: Is there anyone who can point me at the appropriate cage to rattle? I know it appears to be a netfilter issue, but I don't seem to be able to get a message to the list (and I am subscribed to it and have been

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-05 Thread Avi Kivity
On 06/05/2011 04:45 PM, Brad Campbell wrote: The mailing list might be set not to send your own mails back to you. Check the list archive. Yep, I did that first.. Given the response to previous issues along the same line, it looks a bit like I just remember not to actually use the system in

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-03 Thread Brad Campbell
On 02/06/11 07:03, CaT wrote: On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote: Unfortunately the only interface that is mentioned by name anywhere in my firewall is $DMZ (which is ppp0 and not part of any bridge). All of the nat/dnat and other horrible hacks are based on IP

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-03 Thread Bernhard Held
Am 03.06.2011 15:38, schrieb Brad Campbell: On 02/06/11 07:03, CaT wrote: On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote: Unfortunately the only interface that is mentioned by name anywhere in my firewall is $DMZ (which is ppp0 and not part of any bridge). All of the nat/dnat

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-03 Thread Brad Campbell
On 03/06/11 23:50, Bernhard Held wrote: Am 03.06.2011 15:38, schrieb Brad Campbell: On 02/06/11 07:03, CaT wrote: On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote: Unfortunately the only interface that is mentioned by name anywhere in my firewall is $DMZ (which is ppp0 and not

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Brad Campbell
On 01/06/11 12:52, Hugh Dickins wrote: I guess Brad could try SLUB debugging, boot with slub_debug=P for poisoning perhaps; though it might upset alignments and drive the problem underground. Or see if the same happens with SLAB instead of SLUB. Not much use I'm afraid. This is all I get in

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Avi Kivity
On 06/01/2011 09:31 AM, Brad Campbell wrote: On 01/06/11 12:52, Hugh Dickins wrote: I guess Brad could try SLUB debugging, boot with slub_debug=P for poisoning perhaps; though it might upset alignments and drive the problem underground. Or see if the same happens with SLAB instead of SLUB.

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Avi Kivity
On 06/01/2011 12:29 PM, Brad Campbell wrote: On 01/06/11 14:56, Avi Kivity wrote: On 06/01/2011 09:31 AM, Brad Campbell wrote: On 01/06/11 12:52, Hugh Dickins wrote: I guess Brad could try SLUB debugging, boot with slub_debug=P for poisoning perhaps; though it might upset alignments and

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Avi Kivity
On 06/01/2011 12:40 PM, Avi Kivity wrote: bridge and netfilter, IIRC this was also the problem last time. Do you have any ebtables loaded? Can you try building a kernel without ebtables? Without netfilter at all? Please run all tests with slub_debug=FZPU. -- error compiling

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Brad Campbell
On 01/06/11 17:41, Avi Kivity wrote: On 06/01/2011 12:40 PM, Avi Kivity wrote: bridge and netfilter, IIRC this was also the problem last time. Do you have any ebtables loaded? Never heard of them, but making a cursory check just in case.. brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Avi Kivity
On 06/01/2011 01:53 PM, Brad Campbell wrote: On 01/06/11 17:41, Avi Kivity wrote: On 06/01/2011 12:40 PM, Avi Kivity wrote: bridge and netfilter, IIRC this was also the problem last time. Do you have any ebtables loaded? Never heard of them, but making a cursory check just in case..

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread CaT
On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote: I rebooted into a netfilter kernel, and did all the steps I'd used on the no-netfilter kernel and it ticked along happily. So the result of the experiment is inconclusive. Having said that, the backtraces certainly smell

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread Brad Campbell
On 01/06/11 19:18, CaT wrote: On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote: I rebooted into a netfilter kernel, and did all the steps I'd used on the no-netfilter kernel and it ticked along happily. So the result of the experiment is inconclusive. Having said that, the

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-06-01 Thread CaT
On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote: Unfortunately the only interface that is mentioned by name anywhere in my firewall is $DMZ (which is ppp0 and not part of any bridge). All of the nat/dnat and other horrible hacks are based on IP addresses. Damn. Not referencing

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Brad Campbell
On 31/05/11 13:47, Borislav Petkov wrote: Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your machine from oopsing. Adding linux-mm. I initially thought that, so the second panic was produced with KSM disabled from boot. echo 0 /sys/kernel/mm/ksm/run If you still

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Borislav Petkov
On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote: On 31/05/11 13:47, Borislav Petkov wrote: Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your machine from oopsing. Adding linux-mm. I initially thought that, so the second panic was produced with KSM

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Brad Campbell
On 31/05/11 18:38, Borislav Petkov wrote: On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote: On 31/05/11 13:47, Borislav Petkov wrote: Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your machine from oopsing. Adding linux-mm. I initially thought that, so the

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Hugh Dickins
On Tue, 31 May 2011, Brad Campbell wrote: On 31/05/11 18:38, Borislav Petkov wrote: On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote: On 31/05/11 13:47, Borislav Petkov wrote: Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your machine from oopsing.

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Brad Campbell
On 01/06/11 06:31, Hugh Dickins wrote: Brad, my suspicion is that in each case the top 16 bits of RDX have been mysteriously corrupted from to , causing the general protection faults. I don't understand what that has to do with KSM. No, nor do I. The panic I reproduced with KSM off

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Brad Campbell
On 01/06/11 06:31, Hugh Dickins wrote: Brad, my suspicion is that in each case the top 16 bits of RDX have been mysteriously corrupted from to , causing the general protection faults. I don't understand what that has to do with KSM. But it's only a suspicion, because I can't make

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Andrea Arcangeli
Hello, On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote: On 01/06/11 06:31, Hugh Dickins wrote: Brad, my suspicion is that in each case the top 16 bits of RDX have been mysteriously corrupted from to , causing the general protection faults. I don't understand what

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Brad Campbell
On 01/06/11 09:15, Andrea Arcangeli wrote: Hello, On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote: On 01/06/11 06:31, Hugh Dickins wrote: Brad, my suspicion is that in each case the top 16 bits of RDX have been mysteriously corrupted from to , causing the general

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-31 Thread Hugh Dickins
On Wed, 1 Jun 2011, Andrea Arcangeli wrote: On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote: On 01/06/11 06:31, Hugh Dickins wrote: Brad, my suspicion is that in each case the top 16 bits of RDX have been mysteriously corrupted from to , causing the general

KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-30 Thread Brad Campbell
G'day all, I'm running a pretty standard home server x86_64 Phenom-II 6 Core 16GB DDR 3 I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32 Bit. These run at boot. When I fire up another XP 32 bit instance and play with it for more than about 2 minutes, I get the panics

Re: KVM induced panic on 2.6.38[2367] 2.6.39

2011-05-30 Thread Borislav Petkov
Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your machine from oopsing. Adding linux-mm. On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote: G'day all, I'm running a pretty standard home server x86_64 Phenom-II 6 Core 16GB DDR 3 I run some virtual