Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-10-01 Thread guy . helmer
On Wednesday, June 6, 2012 8:36:04 PM UTC-5, Mark Felder wrote: Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-10-01 Thread Mark Felder
On Mon, 01 Oct 2012 15:00:40 -0500, guy.hel...@gmail.com wrote: Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 5 ee 60 16 0 1 0 0 Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): SCSI

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-06-06 Thread Mark Felder
Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to my PC tonight, but this hopefully is enough to replicate the

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-31 Thread John Baldwin
On Wednesday, May 30, 2012 3:56:02 pm Mark Felder wrote: On Wed, 30 May 2012 12:17:07 -0500, John Baldwin j...@freebsd.org wrote: Humm, can you test it with 2 CPUs? We primarily only run with 1 CPU. We have seen it crash on multiple CPU VMs. Also, Dane Foster appeared to have been

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-31 Thread Mark Felder
So when this hang happens, there never is a real panic. It just sits in a state which I describe as like being in a deadlock. How would I go about getting a crashdump if it never panics? Is it possible to do the dump over a network or something because I don't believe it can write through

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-31 Thread John Baldwin
On Thursday, May 31, 2012 11:11:11 am Mark Felder wrote: So when this hang happens, there never is a real panic. It just sits in a state which I describe as like being in a deadlock. How would I go about getting a crashdump if it never panics? Is it possible to do the dump over a network

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-30 Thread John Baldwin
On Thursday, May 24, 2012 9:47:46 am Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-30 Thread Mark Felder
On Wed, 30 May 2012 10:06:13 -0500, John Baldwin j...@freebsd.org wrote: Do you only have one CPU in this VM? If not, do you know which threads the other CPUs were running (e.g. do you have ps7.png, etc.)? correct, only one CPU in the VM ___

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-30 Thread Mark Felder
On Wed, 30 May 2012 12:17:07 -0500, John Baldwin j...@freebsd.org wrote: Humm, can you test it with 2 CPUs? We primarily only run with 1 CPU. We have seen it crash on multiple CPU VMs. Also, Dane Foster appeared to have been using multiple CPUs in his video transcoding VMs.

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-24 Thread Mark Felder
On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-24 Thread dane foster
Hey all, On 25/05/2012, at 1:47 AM, Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-24 Thread Bjoern A. Zeeb
On 24. May 2012, at 13:47 , Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-24 Thread Adrian Chadd
Hi, You guys now absolutely, positively have enough information for a PR. It's still not clear whether it's a device/interrupt layer issue in FreeBSD, or whether vmware is doing something wrong with how it implements shared interrupts, or a bit of both.. Adrian On 24 May 2012 13:54, dane

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-23 Thread Adrian Chadd
Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? If this is indeed the problem then I really think we should root cause why the driver and/or interrupt handling code is getting angry with the shared interrupt. I'd also appreciate it if

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-21 Thread Mark Felder
OK guys I've been talking with another user who can recreate this crash and the last bit of information we've learned seems to be leaning towards interrupts/IRQ issues like someone (bz@ perhaps?) suggested. I'm still trying to test this myself, but the other user was able to recreate my

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-21 Thread Andrew Boyer
On May 21, 2012, at 12:41 PM, Mark Felder wrote: OK guys I've been talking with another user who can recreate this crash and the last bit of information we've learned seems to be leaning towards interrupts/IRQ issues like someone (bz@ perhaps?) suggested. I'm still trying to test this

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-21 Thread Mark Felder
On Mon, 21 May 2012 12:01:19 -0500, Andrew Boyer abo...@averesystems.com wrote: You could try switching mpt to MSI. MSI interrupts are never shared. Add this to /boot/device.hints: hint.mpt.0.msi_enable=1 Currently implementing this on the known crashy servers. I've been looking

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-21 Thread Michael Powell
Mark Felder wrote: OK guys I've been talking with another user who can recreate this crash and the last bit of information we've learned seems to be leaning towards interrupts/IRQ issues like someone (bz@ perhaps?) suggested. I'm still trying to test this myself, but the other user was able

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-21 Thread Mark Felder
On Mon, 21 May 2012 13:47:45 -0500, Michael Powell nightre...@hotmail.com wrote: Very curious how 'irq 22 at device 22.0' and 'dev.mpt.0.%location: slot=22' all match with a '22'. Strangely here in ESXi that doesn't work the same. Emulated BIOS must be considerably different... :/ $

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-05-10 Thread Mark Felder
Quick update: I have received word last night that this crash has been consistently happening to someone on FreeBSD 9 and they're looking for more ideas. I changed the following 41 days ago: - Video memory to auto if it wasn't already - SCSI controller changed from LSI Logic Parallel to

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-03 Thread Mark Felder
Guys, The crash on my machine with debugging has evaded me for a few days. I'm still looking for further suggestions of things I should grab from the DDB when it happens again. Thanks for the help everyone! ___ freebsd-questions@freebsd.org

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-03 Thread Doug Barton
On 4/2/2012 3:59 PM, Joe Greco wrote: On 4/2/2012 11:43 AM, Joe Greco wrote: As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-02 Thread Doug Barton
On 03/30/2012 07:41, Joe Greco wrote: On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-02 Thread Joe Greco
On 03/30/2012 07:41, Joe Greco wrote: On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-02 Thread Doug Barton
On 4/2/2012 11:43 AM, Joe Greco wrote: As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a workaround, you get criticized. If you

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-02 Thread Joe Greco
On 4/2/2012 11:43 AM, Joe Greco wrote: As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a workaround, you get criticized. If

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-30 Thread Joe Greco
On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-30 Thread Joe Greco
On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco jgr...@ns.sol.net wrote: It also doesn't explain the experience here, where one VM basically crapped out but only after a migration - and then stayed crapped out. It would be interesting to hear about your datastore, how busy it is, what

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Doug Barton
On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Michael Powell
Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Wed, 28 Mar 2012 18:31:38 -0500, Adrian Chadd adr...@freebsd.org wrote: * have you filed a PR? No * is the crash easily reproducable? Unfortunately not. It's totally random. Some servers will get the bug and crash daily, some will crash weekly, some might seem to be fine but 3

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 02:36:49 -0500, Doug Barton do...@freebsd.org wrote: As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
Thank you for the suggestion. We'll put it in our toolbox and see if it helps! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
Alright, new data. It happened to crash about 10 minutes after I came in this morning and I ran some stuff in the DDB. I have no idea what information is useful, but perhaps someone will see something out of the ordinary? http://feld.me/freebsd/esx_crash/ Thanks...

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
Hi, * have you filed a PR? * is the crash easily reproducable? * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create a ramdisk image using nanobsd?) and do some stress testing inside that? It sounds like you've established it's a storage issue, or at least interrupt

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Hans Petter Selasky
On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? --HPS ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 09:58:16 -0500, Hans Petter Selasky hsela...@c2i.net wrote: Do both 32- and 64-bit versions of FreeBSD crash? Correct, we see both i386 and amd64 flavors crash in the same way. ___ freebsd-questions@freebsd.org mailing list

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Eduardo Morras
At 16:03 29/03/2012, you wrote: Alright, new data. It happened to crash about 10 minutes after I came in this morning and I ran some stuff in the DDB. I have no idea what information is useful, but perhaps someone will see something out of the ordinary? http://feld.me/freebsd/esx_crash/

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? We've only seen it happen on one virtual machine. That was a 32-bit version. And it's not so much a crash as it is a disk I/O hang. The fact that it was happening regularly to that

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Hans Petter Selasky
On Thursday 29 March 2012 17:49:30 Joe Greco wrote: On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? We've only seen it happen on one virtual machine. That was a 32-bit version. And it's not so much a crash as it is a disk

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 10:31:24 -0500, Eduardo Morras nec...@retena.com wrote: Don't know about ESXi but on others VM Managers i can change the chipset emulation from ICH10 to ICH4. Can you change it to an older chipset too? Unfortunately there's no setting in the GUI for that but I'll keep

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 10:55:36 -0500, Hans Petter Selasky hsela...@c2i.net wrote: It almost sounds like the lost interrupt issue I've seen with USB EHCI devices, though disk I/O should have a retry timeout? What does wmstat -i output? --HPS Here's a server that has a week uptime and is due

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 10:49:30 -0500, Joe Greco jgr...@ns.sol.net wrote: I explained it at the time to one of my VMware friends: This is 100% identical to what we see, Joe! And we're so unlucky that we have this happen on probably a dozen servers, but a handful are the really bad ones.

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Jim Bryant
This sounds just like a race condition that happens under Windows 7 on this laptop. The race condition, as far as I can tell involves heavy disk access and heavy network access, and usually leaves the drive light on, while all activity monitors (alldisk, allcpu, allnetwork) are still active,

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Alan Cox
On Thu, Mar 29, 2012 at 11:27 AM, Mark Felder f...@feld.me wrote: On Thu, 29 Mar 2012 10:55:36 -0500, Hans Petter Selasky hsela...@c2i.net wrote: It almost sounds like the lost interrupt issue I've seen with USB EHCI devices, though disk I/O should have a retry timeout? What does wmstat

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Atkinson
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/29/2012 07:03, Mark Felder wrote: Alright, new data. It happened to crash about 10 minutes after I came in this morning and I ran some stuff in the DDB. I have no idea what information is useful, but perhaps someone will see something out of

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Jerry
On Thu, 29 Mar 2012 11:43:45 -0500 Jim Bryant articulated: Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 12:05:30 -0500, Mark Atkinson atkin...@gmail.com wrote: If this is an interrupt problem with disk i/o, then you might want to look into (DDB(4)) show intr show intrcount maybe show allrman Thank you! I really don't know what things we should be running in DDB to

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 12:24:30 -0500, je...@seibercom.net wrote: I just started reading this tread, but I am wondering if I missed something here. What does this have to do with Windows 7? I emailed him off-list but I'm guessing he thought this was on VMWare Workstation or another product

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 11:53:02 -0500, Alan Cox alan.l@gmail.com wrote: Not so long ago, VMware implemented a clever scheme for reducing the overhead of virtualized interrupts that must be delivered by at least some (if not all) of their emulated storage controllers:

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
On Thursday 29 March 2012 17:49:30 Joe Greco wrote: On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? We've only seen it happen on one virtual machine. That was a 32-bit version. And it's not so much a crash as it is

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Adam Vande More
On Thu, Mar 29, 2012 at 1:22 PM, Mark Felder f...@feld.me wrote: If we assume mpt is the culprit Doesn't VMWare offer different types of emulated disk controllers? If so, that might be the easiest way to narrow the field. Another thing maybe to try would be to backport the mpt Also, it's

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 15:53:52 -0500, Adam Vande More amvandem...@gmail.com wrote: Doesn't VMWare offer different types of emulated disk controllers? If so, that might be the easiest way to narrow the field. Another thing maybe to try would be to backport the mpt Yes, they offer

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Doug Barton
On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
And then there is this one with similar symptoms and a workaround: http://forums.freebsd.org/showthread.php?t=3D27899 I'm now investigating those loader.conf options. I have my crashy machine set to use them on next boot so we'll see if it crashes now that I'm using LSI SAS emulated

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Mark Felder
On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco jgr...@ns.sol.net wrote: It also doesn't explain the experience here, where one VM basically crapped out but only after a migration - and then stayed crapped out. It would be interesting to hear about your datastore, how busy it is, what technology,

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Jerome Herman
On 28/03/2012 22:59, Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Adrian Chadd
Again, it's starting to sound like an interrupt handling issue which may or may not be limited to the storage device. You'll have to engage someone who knows those device drivers and likely have them add some debugging to the driver which can be easily flipped on (via binaries in a ramdisk - very

Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-28 Thread Mark Felder
Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time,

Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-28 Thread Adrian Chadd
Hi, * have you filed a PR? * is the crash easily reproducable? * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create a ramdisk image using nanobsd?) and do some stress testing inside that? It sounds like you've established it's a storage issue, or at least interrupt handling for