Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wednesday, June 6, 2012 8:36:04 PM UTC-5, Mark Felder wrote: Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to my PC tonight, but this hopefully is enough to replicate the crash for any curious followers: ESXi 5 9 or 9-STABLE HAST 1 cpu is fine 1GB of ram UFS SUJ on HAST device No special loader.conf, sysctl, etc No need for VMWare tools Run Bonnie++ on the HAST device We can get the crash to happen on the first run of bonnie++ right now. I'll post the exact specs and precise command run in the PR. We found an old post from 2004 when we looked up the process state obtained from CTRL+T -- flswai -- which describes the symptoms nearly perfectly. http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2004-02/0250.html Hopefully this gets us closer to a fix... Is this a crash or a hang? Over the past couple of weeks, I've been working with a FreeBSD 9.1RC1 system under VMware ESXi 5.0 with a 64GB UFS root FS and 2TB ZFS filesystem mounted via a virtual LSI SAS interface. Sometimes during heavy I/O load (rsync from other servers) on the ZFS FS, this shows up in /var/log/messages: Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 5 ee 60 16 0 1 0 0 Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 42 51 0 1 0 0 Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 64 51 0 1 0 0 Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 66 51 0 1 0 0 Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy ... Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 41 f3 94 99 0 1 0 0 Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): Retrying command These have been happening roughly every other day. mpt0 and em0 were sharing int 18, so today I put hint.mpt.0.msi_enable=1 into /boot/devices.hints and rebooted; now mpt0 is using int 256. I'll see if it helps. Guy ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Mon, 01 Oct 2012 15:00:40 -0500, guy.hel...@gmail.com wrote: Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 5 ee 60 16 0 1 0 0 Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 42 51 0 1 0 0 Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 64 51 0 1 0 0 Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 66 51 0 1 0 0 Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy ... Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 41 f3 94 99 0 1 0 0 Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): Retrying command Sometimes you'll see this before a crash, but not every time. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to my PC tonight, but this hopefully is enough to replicate the crash for any curious followers: ESXi 5 9 or 9-STABLE HAST 1 cpu is fine 1GB of ram UFS SUJ on HAST device No special loader.conf, sysctl, etc No need for VMWare tools Run Bonnie++ on the HAST device We can get the crash to happen on the first run of bonnie++ right now. I'll post the exact specs and precise command run in the PR. We found an old post from 2004 when we looked up the process state obtained from CTRL+T -- flswai -- which describes the symptoms nearly perfectly. http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2004-02/0250.html Hopefully this gets us closer to a fix... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wednesday, May 30, 2012 3:56:02 pm Mark Felder wrote: On Wed, 30 May 2012 12:17:07 -0500, John Baldwin j...@freebsd.org wrote: Humm, can you test it with 2 CPUs? We primarily only run with 1 CPU. We have seen it crash on multiple CPU VMs. Also, Dane Foster appeared to have been using multiple CPUs in his video transcoding VMs. Unfortunately I can't give you more information at the moment. I'm working with Dane to compile easy to follow steps that recreate this failure. I have not been successful in getting this to crash on demand in my environment, but Dane has so we're trying to recreate his. Ok. It would be really helpful if we could get a crashdump, though I realize that may not be doable. Otherwise, full DDB ps output from a hang would be a good start. Primarily I would want to see what the system is doing and why it isn't running the threads on the run queue. It might also be useful to add KTR_SCHED tracing so we can get the output of that via 'show ktr' from DDB when it hangs. -- John Baldwin ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
So when this hang happens, there never is a real panic. It just sits in a state which I describe as like being in a deadlock. How would I go about getting a crashdump if it never panics? Is it possible to do the dump over a network or something because I don't believe it can write through the controller at all. Also, thank you for the KTR_SCHED tip. This is the type of info I was looking for. Unfortunately I've only ever seen this crash once on a kernel with debugging enabled. The machine which is currently prepared to do this work used to crash a few times a week and now it has 70 days uptime... however, it is an example of a machine with mpt0 and em0 sharing an IRQ so I might be able to trigger it using Dane's method. $ vmstat -i interrupt total rate irq1: atkbd0 392 0 irq6: fdc0 9 0 irq14: ata0 34 0 irq18: em0 mpt0 1189748491218 cpu0: timer 2174263198400 Total 3364012124619 I'm doing my best to get you guys the info you need, but this is one heck of a Heisenbug... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thursday, May 31, 2012 11:11:11 am Mark Felder wrote: So when this hang happens, there never is a real panic. It just sits in a state which I describe as like being in a deadlock. How would I go about getting a crashdump if it never panics? Is it possible to do the dump over a network or something because I don't believe it can write through the controller at all. You can break into ddb and run 'call doadump'. It should use polled IO, so there is a slight chance of it working. Also, thank you for the KTR_SCHED tip. This is the type of info I was looking for. Unfortunately I've only ever seen this crash once on a kernel with debugging enabled. The machine which is currently prepared to do this work used to crash a few times a week and now it has 70 days uptime... however, it is an example of a machine with mpt0 and em0 sharing an IRQ so I might be able to trigger it using Dane's method. $ vmstat -i interrupt total rate irq1: atkbd0 392 0 irq6: fdc0 9 0 irq14: ata0 34 0 irq18: em0 mpt0 1189748491218 cpu0: timer 2174263198400 Total 3364012124619 I'm doing my best to get you guys the info you need, but this is one heck of a Heisenbug... Thanks. -- John Baldwin ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thursday, May 24, 2012 9:47:46 am Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-) The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue. Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable=1 ? Hmm, so the set of ps output you have from DDB shows a lot of runnable processes and swi6 (Giant taskq) as the only running thread (all consistent with your hang). (And that is from your Ctrl-Alt-Esc) Do you only have one CPU in this VM? If not, do you know which threads the other CPUs were running (e.g. do you have ps7.png, etc.)? -- John Baldwin ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wed, 30 May 2012 10:06:13 -0500, John Baldwin j...@freebsd.org wrote: Do you only have one CPU in this VM? If not, do you know which threads the other CPUs were running (e.g. do you have ps7.png, etc.)? correct, only one CPU in the VM ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wed, 30 May 2012 12:17:07 -0500, John Baldwin j...@freebsd.org wrote: Humm, can you test it with 2 CPUs? We primarily only run with 1 CPU. We have seen it crash on multiple CPU VMs. Also, Dane Foster appeared to have been using multiple CPUs in his video transcoding VMs. Unfortunately I can't give you more information at the moment. I'm working with Dane to compile easy to follow steps that recreate this failure. I have not been successful in getting this to crash on demand in my environment, but Dane has so we're trying to recreate his. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-) The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue. Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable=1 ? Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hey all, On 25/05/2012, at 1:47 AM, Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-) The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue. Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable=1 ? The situation I've got that's stable now is: hw.pci.enable_msi=0 hw.pci.enable_msix=0 in /boot/loader.conf and: samael:~:% vmstat -i [ 6:31PM] interrupt total rate irq1: atkbd0 6 0 irq18: em0 mpt0 3061100 15 irq19: em1 6891706 35 cpu0: timer166383735868 cpu1: timer166382123868 cpu3: timer166382123868 cpu2: timer166382121868 Total 675482914 3525 Not using em0. This works for 8 (FreeBSD samael.slush.ca 8.3-STABLE FreeBSD 8.3-STABLE #1: Mon May 7 11:51:03 NZST 2012 r...@samael.slush.ca:/usr/obj/usr/src/sys/DENE amd64). Neither of those settings on their own seem to stop it from happening. The 9 box I've tried this on still hangs almost every time i run handbrake, no matter whether MSI/MSIX is enabled, or I have separate IRQs for mpt0 and em0/1 I can cause the hang mostly on demand, but not quite sure what information to provide from the hung system. If somebody can let me know what they need, including root access, I can make that happen. Cheers, Dane Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 24. May 2012, at 13:47 , Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-) The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?) Just for the public archives. Interrupts wasn't me. I might have mentioned disabling cdrom and fdc as good as possible but everything else I cannot remember... and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue. Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable=1 ? -- Bjoern A. Zeeb You have to have visions! It does not matter how good you are. It matters what good you do! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hi, You guys now absolutely, positively have enough information for a PR. It's still not clear whether it's a device/interrupt layer issue in FreeBSD, or whether vmware is doing something wrong with how it implements shared interrupts, or a bit of both.. Adrian On 24 May 2012 13:54, dane foster d...@ilovedene.com wrote: Hey all, On 25/05/2012, at 1:47 AM, Mark Felder wrote: On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd adr...@freebsd.org wrote: Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-) The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue. Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable=1 ? The situation I've got that's stable now is: hw.pci.enable_msi=0 hw.pci.enable_msix=0 in /boot/loader.conf and: samael:~:% vmstat -i [ 6:31PM] interrupt total rate irq1: atkbd0 6 0 irq18: em0 mpt0 3061100 15 irq19: em1 6891706 35 cpu0: timer 166383735 868 cpu1: timer 166382123 868 cpu3: timer 166382123 868 cpu2: timer 166382121 868 Total 675482914 3525 Not using em0. This works for 8 (FreeBSD samael.slush.ca 8.3-STABLE FreeBSD 8.3-STABLE #1: Mon May 7 11:51:03 NZST 2012 r...@samael.slush.ca:/usr/obj/usr/src/sys/DENE amd64). Neither of those settings on their own seem to stop it from happening. The 9 box I've tried this on still hangs almost every time i run handbrake, no matter whether MSI/MSIX is enabled, or I have separate IRQs for mpt0 and em0/1 I can cause the hang mostly on demand, but not quite sure what information to provide from the hung system. If somebody can let me know what they need, including root access, I can make that happen. Cheers, Dane Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hi, can you please, -please- file a PR? And place all of the above information in it so we don't lose it? If this is indeed the problem then I really think we should root cause why the driver and/or interrupt handling code is getting angry with the shared interrupt. I'd also appreciate it if you and the other people who can reproduce this could work with the em/mpt driver people and root cause why this is going. I think having FreeBSD on vmware work stable out of the box without these kinds of tweaks is the way to go - who knows what else is lurking here.. I'm very very glad you've persisted with this and if I had them, I'd send you a FreeBSD persistent bug reporter! t-shirt. Thanks, Adrian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
OK guys I've been talking with another user who can recreate this crash and the last bit of information we've learned seems to be leaning towards interrupts/IRQ issues like someone (bz@ perhaps?) suggested. I'm still trying to test this myself, but the other user was able to recreate my crash pretty much on demand. The fix was to not use the first NIC in the VM because it will always share an IRQ with mpt0. Once mpt0 is on its own the crash does not seem to be reproducible anymore. Before: $ vmstat -i interrupt total rate irq1: atkbd0 378 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em1687237 1 irq18: em0 mpt0319094024539 cpu0: timer236770821400 Total 556552503940 After: $ vmstat -i interrupt total rate irq1: atkbd0 38 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em1 2811 15 irq17: em2 5 0 cpu0: timer71013398 irq256: mpt0 12163 68 Total 86073483 Is there any other way we can make mpt0 get its own dedicated IRQ without having to do this? The problem is that it causes us to have to make rc.conf changes, pf.conf changes, and who knows what other software could be on these machines that is trying to bind to a specific NIC... Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On May 21, 2012, at 12:41 PM, Mark Felder wrote: OK guys I've been talking with another user who can recreate this crash and the last bit of information we've learned seems to be leaning towards interrupts/IRQ issues like someone (bz@ perhaps?) suggested. I'm still trying to test this myself, but the other user was able to recreate my crash pretty much on demand. The fix was to not use the first NIC in the VM because it will always share an IRQ with mpt0. Once mpt0 is on its own the crash does not seem to be reproducible anymore. Before: $ vmstat -i interrupt total rate irq1: atkbd0 378 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em1687237 1 irq18: em0 mpt0319094024539 cpu0: timer236770821400 Total 556552503940 After: $ vmstat -i interrupt total rate irq1: atkbd0 38 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em1 2811 15 irq17: em2 5 0 cpu0: timer71013398 irq256: mpt0 12163 68 Total 86073483 Is there any other way we can make mpt0 get its own dedicated IRQ without having to do this? The problem is that it causes us to have to make rc.conf changes, pf.conf changes, and who knows what other software could be on these machines that is trying to bind to a specific NIC... Thanks! You could try switching mpt to MSI. MSI interrupts are never shared. Add this to /boot/device.hints: hint.mpt.0.msi_enable=1 -Andrew -- Andrew Boyerabo...@averesystems.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Mon, 21 May 2012 12:01:19 -0500, Andrew Boyer abo...@averesystems.com wrote: You could try switching mpt to MSI. MSI interrupts are never shared. Add this to /boot/device.hints: hint.mpt.0.msi_enable=1 Currently implementing this on the known crashy servers. I've been looking around and all of our VM's that do NOT crash also do not share interrupts between em0/mpt0. Thank you very much if this is the fix we will be SO grateful. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Mark Felder wrote: OK guys I've been talking with another user who can recreate this crash and the last bit of information we've learned seems to be leaning towards interrupts/IRQ issues like someone (bz@ perhaps?) suggested. I'm still trying to test this myself, but the other user was able to recreate my crash pretty much on demand. The fix was to not use the first NIC in the VM because it will always share an IRQ with mpt0. Once mpt0 is on its own the crash does not seem to be reproducible anymore. [snip] I am not anywhere near your level in this subject area. My understanding is limited and do not have the in-depth experience. However, please allow me to possibly add an idea or two. I am shakedown testing FreeBSD 9 in a VirtualBox VM - so there is definitely a degree of 'apples vs oranges' present. VirtualBox (as I am using it) is a userland app and not a bare-metal hypervisor. When I set up the VM I chose to use the synthetic SAS controller as that would best represent actual server hardware in my workplace, along with the corresponding mpt driver in the FreeBSD 9 guest. Please note some of the following for comparative purposes only: [...] Event timer LAPIC quality 400 ACPI APIC Table: VBOX VBOXAPIC FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 Version 1.1 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: VBOX VBOXXSDT on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) Timecounter HPET frequency 14318180 Hz quality 950 Timecounter ACPI-fast frequency 3579545 Hz quality 900 acpi_timer0: 32-bit timer at 3.579545MHz port 0x4008-0x400b on acpi0 [...] em0: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 port 0xd000-0xd007 mem 0xf000-0xf001 irq 19 at device 3.0 on pci0 [...] mpt0: LSILogic SAS/SATA Adapter port 0xd100-0xd1ff mem 0xf082-0xf083,0xf084-0xf085 irq 22 at device 22.0 on pci0 mpt0: MPI Version=1.5.0.0 [...] The em0 is the first Intel NIC in Vbox and notice how it and mpt0 come up with distinctly different IRQs. A sysctl -a |grep mpt returns this: device mpt kern.sched.preemption: 1 kern.sched.preempt_thresh: 80 dev.mpt.0.%desc: LSILogic SAS/SATA Adapter dev.mpt.0.%driver: mpt dev.mpt.0.%location: slot=22 function=0 dev.mpt.0.%pnpinfo: vendor=0x1000 device=0x0054 subvendor=0x1000 subdevice=0x8000 class=0x01 dev.mpt.0.%parent: pci0 dev.mpt.0.debug: 3 dev.mpt.0.role: 1 Very curious how 'irq 22 at device 22.0' and 'dev.mpt.0.%location: slot=22' all match with a '22'. The obvious thing here is we are comparing a userland Vbox guest to a VMWare hypervisor. From what little I know concerning any of this, to me it sounds vaguely like an APIC, LAPIC, and IO/APIC bug. There are known bugs wrt to BIOS setting up IRQ routing incorrectly, and/or providing incorrect ACPI and/or IMS tables to operating systems. The parallel in this case would be the logical or synthetic so-called BIOS that the VMWare hypervisor presents to the FreeBSD guest at guest boot time. In this case the truest fix for the problem would fall to VMWare, e.g. if the hypervisor is setting up tables in such a way as to create the shared IRQ problem in the first place. If my idea/theory/potential hypothesis has any merit. I do not understand why any of this would be different depending upon which guest is installed, but I also know absolutely nothing about VMWare hypervisor internals. Is there any other way we can make mpt0 get its own dedicated IRQ without having to do this? The problem is that it causes us to have to make rc.conf changes, pf.conf changes, and who knows what other software could be on these machines that is trying to bind to a specific NIC... Very possibly Andrew's device.hints is probably your best shot at a workaround. Wish you the best of luck in any case. You have done quite a job in researching this problem even to arrive at this point. Thank-you for that, and for sharing it with the community. Even though I can't really offer the kind of assistance you require, I have followed along with interest for self edification. -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Mon, 21 May 2012 13:47:45 -0500, Michael Powell nightre...@hotmail.com wrote: Very curious how 'irq 22 at device 22.0' and 'dev.mpt.0.%location: slot=22' all match with a '22'. Strangely here in ESXi that doesn't work the same. Emulated BIOS must be considerably different... :/ $ vmstat -i interrupt total rate irq1: atkbd0 6 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em162 0 irq18: em0178079 17 cpu0: timer 4136470400 irq256: mpt0 112544 10 Total4427204428 $ sysctl -a | grep mpt kern.sched.preemption: 1 kern.sched.preempt_thresh: 64 dev.mpt.0.%desc: LSILogic SAS/SATA Adapter dev.mpt.0.%driver: mpt dev.mpt.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE40.S1F0 dev.mpt.0.%pnpinfo: vendor=0x1000 device=0x0054 subvendor=0x15ad subdevice=0x1976 class=0x010700 dev.mpt.0.%parent: pci3 dev.mpt.0.debug: 3 dev.mpt.0.role: 1 dev.mpt.0.wake: 0 irq256 and slot ... 0. Interesting. The obvious thing here is we are comparing a userland Vbox guest to a VMWare hypervisor. From what little I know concerning any of this, to me it sounds vaguely like an APIC, LAPIC, and IO/APIC bug. There are known bugs wrt to BIOS setting up IRQ routing incorrectly, and/or providing incorrect ACPI and/or IMS tables to operating systems. FWIW, VirtualBox and ESXi are nearly the same except ESXi just has as minimal an OS as possible for performance reasons. And what you're describing is exactly what I've been thinking for a long time but I just haven't had the proof. The parallel in this case would be the logical or synthetic so-called BIOS that the VMWare hypervisor presents to the FreeBSD guest at guest boot time. In this case the truest fix for the problem would fall to VMWare, e.g. if the hypervisor is setting up tables in such a way as to create the shared IRQ problem in the first place. If my idea/theory/potential hypothesis has any merit. I do not understand why any of this would be different depending upon which guest is installed, but I also know absolutely nothing about VMWare hypervisor internals. I don't know enough about how it's supposed to work but hopefully we're getting close to nailing down the real VMWare bug and we can finally tell their engineering to fix it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Quick update: I have received word last night that this crash has been consistently happening to someone on FreeBSD 9 and they're looking for more ideas. I changed the following 41 days ago: - Video memory to auto if it wasn't already - SCSI controller changed from LSI Logic Parallel to LSI Logic SAS It uses the same driver (da) but so far has been holding steady for us. As far as the video memory -- many of our servers somehow had video memory set to 1MB which seemed strange; newer builds of FreeBSD on ESXi do not show this option. Perhaps there was a build of ESXi in the past that had a different setting for video memory when you selected FreeBSD? Another change people might want to do as suggested to us by VMWare Support: - Change CPU/MMU Virtualization to the bottom option -- Use Intel VTx/AMD-V for instruction set virtualization and Intel EPT / AMD RVI for MMU virtualization Supposedly there are autodetection issues here with some OSes -- they named some BSDs and Netware. I'll provide further updates if anything changes, but this seems to be working well so far. We won't begin to trust it until we can hit at least 100 days of uptime, though. Unfortunately I was hoping to upgrade these servers to 8.3 before then... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Guys, The crash on my machine with debugging has evaded me for a few days. I'm still looking for further suggestions of things I should grab from the DDB when it happens again. Thanks for the help everyone! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 4/2/2012 3:59 PM, Joe Greco wrote: On 4/2/2012 11:43 AM, Joe Greco wrote: As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a workaround, you get criticized. If you submit a workaround but you don't submit a patch, you get criticized. If you submit a patch but it's not in the preferred format, you get criticized. I'm still not sure what you're taking as criticism. Nothing I've said was intended that way, nor should it be read that way. If you feel that you've been criticized by others in the manner you describe, you should probably take it up with them on an individual basis. It certainly seemed to me that As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. was an unwarranted criticism for reasons that I've already explained. Everything in that paragraph is a fact. If you feel criticized when people state facts, I'm not sure how much I can help you. Please note, I didn't say, You're an idiot for running old stuff. I even explicitly stated that I understood *why* the OP is running an old version. Nevertheless, the facts are what they are. The only way we can deal rationally with the world is to acknowledge reality and deal with it. Wishing it were otherwise isn't really useful. :) Or perhaps this: And since you can't reliably reproduce the problem, how do you expect us to? I understand that these sorts of bugs are difficult/annoying, etc. Been there, done that. Which would appear to be suggesting that either (or possibly both): 1) The reporter has a duty to be able to reliably reproduce the problem prior to reporting, and/or 2) That there was some unreasonable expectation on the reporter's part that you were expected to reproduce it. Quite the contrary, I was responding to your implication that there is some other answer that we should be able to give the OP, other than Try a newer version. Various people have chimed in on the thread, all have offered suggestions, none of which (AFAICS) have helped. I continue to maintain that the best course of action for the OP would be to try the latest 8-stable. And BTW, there are (at least) 2 reasons for that. First, the bug may actually be fixed. But second, we're in the middle of a release cycle for 8.3 right now. If the bug persists in the latest code it will be easier to get the right eyes onto the problem. That benefits both the OP and the community at large. Doug ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 03/30/2012 07:41, Joe Greco wrote: On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. Doug So you're saying that he should have been using 8.3-RELEASE, then. That isn't what I said at all, sorry if I wasn't clear. The OP mentioned 9.0-RELEASE, and in the context of his message (which I snipped) he mentioned 8-stable. That's what I was referring to. And since both the poster and I made it clear that this doesn't seem to be a case of it fails reliably on a machine of your choosing, just installing random other versions and hoping that it's going to cause a fail ... well, let's just say that doesn't make a whole lot of sense. Or at least it's a recipe for a hell of a lot of busywork, busywork not guaranteed to return any sort of useful result. And since you can't reliably reproduce the problem, how do you expect us to? I understand that these sorts of bugs are difficult/annoying, etc. Been there, done that. In the meantime, it's unrealistic to tell people to use supported releases, to wait fifteen months between releases, and then to criticize people complaining about problems with a supported release for using old code. Just to be clear, I didn't criticize anyone. And I share your frustration with the length of the 8.3 release cycle. I really wish I had a better answer, but as much as you and I may wish that things were different, Try a newer version is the best answer we have atm. Doug -- This .signature sanitized for your protection ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 03/30/2012 07:41, Joe Greco wrote: On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. Doug So you're saying that he should have been using 8.3-RELEASE, then. That isn't what I said at all, sorry if I wasn't clear. The OP mentioned 9.0-RELEASE, and in the context of his message (which I snipped) he mentioned 8-stable. That's what I was referring to. And since both the poster and I made it clear that this doesn't seem to be a case of it fails reliably on a machine of your choosing, just installing random other versions and hoping that it's going to cause a fail ... well, let's just say that doesn't make a whole lot of sense. Or at least it's a recipe for a hell of a lot of busywork, busywork not guaranteed to return any sort of useful result. And since you can't reliably reproduce the problem, how do you expect us to? I understand that these sorts of bugs are difficult/annoying, etc. Been there, done that. Nobody expected you to. We're trying to figure out any commonalities that might exist; these may serve to help shed light on where the problem lies. The interesting thing is that I took it and looked at it and came to a conclusion that might have been wrong, though I think the trail of reasoning I used was itself reasonable, given my exceedingly small (one example of problem) sample size. Mark's able to actually *reproduce* the problem on separate installs and with circumstances that are at least somewhat different than what my theory involved, though it is not quite possible to rule out some sort of corruption. Since I have to *assume* that many sites run some sort of FreeBSD on their VMware gear, given that VMware actually lists it as a supported version and VMware generally does things for profit, I am still kind of of the opinion that this is some sort of corruption bug, one that I triggered inadvertently, but one that Mark's environment reproduces rather more frequently. That just seems so unlikely, but more unlikely things have come to pass, so I'm holding onto it as my working theory ;-) I still plan to try to recover my broken VM from backups at some point if time permits. But in short, to answer your question: I don't *care* if you can reproduce the problem. As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a workaround, you get criticized. If you submit a workaround but you don't submit a patch, you get criticized. If you submit a patch but it's not in the preferred format, you get criticized. Hm. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 4/2/2012 11:43 AM, Joe Greco wrote: As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a workaround, you get criticized. If you submit a workaround but you don't submit a patch, you get criticized. If you submit a patch but it's not in the preferred format, you get criticized. I'm still not sure what you're taking as criticism. Nothing I've said was intended that way, nor should it be read that way. If you feel that you've been criticized by others in the manner you describe, you should probably take it up with them on an individual basis. My experience of FreeBSD as a community is that we tend to be both less critical of users, and less tolerant of it. Especially when compared to other communities that I've interacted with. Doug -- This .signature sanitized for your protection ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 4/2/2012 11:43 AM, Joe Greco wrote: As a user, you can't win. If you don't report a problem, you get criticized. If you report a problem but can't figure out how to reproduce it, you get criticized. If you can reproduce it but you don't submit a workaround, you get criticized. If you submit a workaround but you don't submit a patch, you get criticized. If you submit a patch but it's not in the preferred format, you get criticized. I'm still not sure what you're taking as criticism. Nothing I've said was intended that way, nor should it be read that way. If you feel that you've been criticized by others in the manner you describe, you should probably take it up with them on an individual basis. It certainly seemed to me that As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. was an unwarranted criticism for reasons that I've already explained. Or perhaps this: And since you can't reliably reproduce the problem, how do you expect us to? I understand that these sorts of bugs are difficult/annoying, etc. Been there, done that. Which would appear to be suggesting that either (or possibly both): 1) The reporter has a duty to be able to reliably reproduce the problem prior to reporting, and/or 2) That there was some unreasonable expectation on the reporter's part that you were expected to reproduce it. I consider 1) to be ridiculous, as long as the reporter is reasonably willing to work to resolve the issue, that should certainly be good enough, and he's certainly been interactive enough to _my_ comments, and 2) seems to be nowhere in sight in the reporter's comments, but is nonetheless present in your response. Please respect Reply-to. Thanks. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. Doug So you're saying that he should have been using 8.3-RELEASE, then. That isn't what I said at all, sorry if I wasn't clear. The OP mentioned 9.0-RELEASE, and in the context of his message (which I snipped) he mentioned 8-stable. That's what I was referring to. And since both the poster and I made it clear that this doesn't seem to be a case of it fails reliably on a machine of your choosing, just installing random other versions and hoping that it's going to cause a fail ... well, let's just say that doesn't make a whole lot of sense. Or at least it's a recipe for a hell of a lot of busywork, busywork not guaranteed to return any sort of useful result. What you suggest is a fine solution for My ASUS Sempron box fails when I do X! -- in such a case, Try a different version of FreeBSD makes lots of sense. The problem is, in a virtualization environment, theoretically the virtual hosts are all the same sort of hardware (modulo any specific configuration changes of course), so when someone presents a problem that afflicts only a percentage of their VM's, it is important to keep in mind that you are not interacting with physical hardware, and that reinstalling an OS on a problem VM...? Well, let's just say I like real hardware better for many reasons. In the meantime, it's unrealistic to tell people to use supported releases, to wait fifteen months between releases, and then to criticize people complaining about problems with a supported release for using old code. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco jgr...@ns.sol.net wrote: It also doesn't explain the experience here, where one VM basically crapped out but only after a migration - and then stayed crapped out. It would be interesting to hear about your datastore, how busy it is, what technology, whether you're using thin, etc. I just have this real strong feeling that it's some sort of corruption with the vmfs3 and thin provisioned disk format, but it'd be interesting to know if that's totally off-track. We've ruled out SAN, but we haven't ruled out VMFS. Even FreeBSD Guests on standalone ESXi servers with no SAN exhibit this crash. For the record, we only use thick provisioning and if it was corruption I'm not sure what layer the corruption could be at. The crashy servers show no abnormalities when I run either `freebsd-update IPS` or `pkg_libchk` to confirm checksums of all installed programs. Now the other data on there... it's not exactly verified, but our backups via rsnapshot seem to prove there is no issue there or we'd have lots of new files each run. Crud, there goes part of my theory :-) Have you migrated these hosts, or were they installed in-place and never moved? fwiw the apparent integrity of things on the VM is consistent with our experience too. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. Doug ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time, and we were hoping we could base some stuff on 8.3 for long term stability...) ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others. [snip] I think we've finally found enough data that this is definitely something in the FreeBSD world. I'm going to begin prepping some of the known crashy servers with more debugging. Any suggestions on what I should build the kernel with? They never do a proper panic, but I definitely want to at least *try* to get into the debugger the next time it crashes. And when it crashes, what the heck should I be running? I've never played with the KDB before... Thank you for any suggestions and help you can give me I am definitely out of my league here and this is way over my head, to be sure. Just a couple of shots in the dark for possibly covering a couple more data points for your research. And I am a tad fuzzy on both as I have never needed to dig into either because I've not had any trouble with either. IIRC there are three different timer subsystems one may choose from. You may want to look into expirementation with each of the three, just to see if this changes any observed behaviors. Or to possibly rule it out. Your situation sounds like a candidate for reverse logic - if I can't get any handle on what's wrong I start at the opposite end and try to make a list of what is right in an attempt to leave a smaller subset to probe. I also think this most likely has nothing to do with what's happening, but for some reason it just pops into my head. Try disabling msi in /boot/loader.conf like this: hw.pci.enable_msi=0 hw.pci.enable_msix=0 At least if it makes no difference maybe this will exclude it from being a 'possible'. Developers who are more in-depth aware of what the differences are between 7.x and 8.x/9.x in the development timeline can probably provide a better picture so as to narrow the field of what to look at. This is way over my head, just wish I could help - I know and have experienced the kind of quandary you have here (I feel for you). :-) -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wed, 28 Mar 2012 18:31:38 -0500, Adrian Chadd adr...@freebsd.org wrote: * have you filed a PR? No * is the crash easily reproducable? Unfortunately not. It's totally random. Some servers will get the bug and crash daily, some will crash weekly, some might seem to be fine but 3 months later hit this crash. * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create a ramdisk image using nanobsd?) and do some stress testing inside that? That's a plan I'd like to execute but my free time for building that environment is rather short at the moment :( I'm not that cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x and 8.0 which has caused this to surface. Was there a setting to revert ACPI behavior from 8.x to 7.x? I thought I read about that at one point or perhaps this was something available back in the dev cycle when 8 was -CURRENT. *shrug* I know 9.0 and onward has even more ACPI changes so assuming it truly is an ACPI bug I guess we could cross our fingers and hope that the bug has mysteriously vanished? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 02:36:49 -0500, Doug Barton do...@freebsd.org wrote: As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. The sad part is that VMWare's supported FreeBSD versions are a joke, and we've been trying to keep VMWare happy by only running supported versions. I honestly don't think they even test. It's so stupid. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Thank you for the suggestion. We'll put it in our toolbox and see if it helps! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Alright, new data. It happened to crash about 10 minutes after I came in this morning and I ran some stuff in the DDB. I have no idea what information is useful, but perhaps someone will see something out of the ordinary? http://feld.me/freebsd/esx_crash/ Thanks... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hi, * have you filed a PR? * is the crash easily reproducable? * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create a ramdisk image using nanobsd?) and do some stress testing inside that? It sounds like you've established it's a storage issue, or at least interrupt handling for storage issue. So I'd definitely try the ramdisk-only boot and thrash it using lighttpd/httperf or something. If that survives fine, I'd look at trying to establish whether there's something wrong in the disk driver(s) freebsd is using. I'm not that cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x and 8.0 which has caused this to surface. We've seen this. Or something that seems really like it. We run dozens of FreeBSD VM's, many of which are 8.mumble. We have a scripted build environment dating back many years, so generally servers come out in a fairly reproducible form. After several months of smooth running, we had need to shuffle some things around, and migrated some servers to a different datastore. Suddenly, one particular VM, our corp Jabber server, started randomly disconnecting people every morning. Some inspection showed that the machine was running, but disk I/O in the VM was freezing up. Subsequent inspection suggested that it was happening during the periodic daily, though we never managed to get it to happen by manually forcing periodic daily, so that's only a theory. Given that several times it appeared that one of the find commands was running, I was guessing that something in the thin provisioned disk image for the system had gone bad, but reading the entire disk with dd didn't cause a hang, running the periodic daily by hand didn't cause a hang, etc. Migrating the VM to a different host and datastore did not fix the issue. Migrating the VM from an Opteron to a Xeon host with all the latest ESXi 4 patches also didn't make any difference. Migrating the disk image from thin to full seemed to fix it, but I only gave it a day or two, then decided there were other good reasons to reload the VM, so I nuked the VM, which, of course, fixed it. In the meantime, a dozen other similar VM's alongside it run just fine. My conclusion was that it was something specific that had gone awry in the virtual machine, probably in the disk image, but I could not identify it without significant digging that I had no particular reason or inclination to do; since it appeared to be a VMware problem, the reload it and be done with it seemed the quickest path to resolution. That having been said, if anyone has any brilliant ideas about what would constitute useful further steps to isolate this, I can look at recovering the faulty VM from backup and seeing if it still exhibits the problem. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? --HPS ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. Doug So you're saying that he should have been using 8.3-RELEASE, then. If you'll kindly go over to http://www.freebsd.org and look under Latest Releases, please note that 8.2 is a production release. If you don't want it to be a production release, then find a way to make it so, but please don't snipe at people who are using the code that the FreeBSD project has indicated is a current production offering. There are many good reasons not to run arbitrary snapshots on your production gear. It's unrealistic to expect people to run non- RELEASE non-production code on their production gear. We can have that discussion if you don't understand that, drop me a note off- list and I'll be happy to explain it. Otherwise, you've told him to run a newer version, of which NONE IS AVAILABLE, unless you're thinking 9.0, but FreeBSD has a rather catastrophic history of point zero releases, and most clueful admins won't run those in production without carefully measuring the risks and benefits. So you've basically told him to run a newer version without any such version being realistically available. WTF? You want people not to use releases that came out over a year ago? The generally sensible solution to that is to release RELEASEs more than once every fourteen or fifteen months. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 09:58:16 -0500, Hans Petter Selasky hsela...@c2i.net wrote: Do both 32- and 64-bit versions of FreeBSD crash? Correct, we see both i386 and amd64 flavors crash in the same way. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
At 16:03 29/03/2012, you wrote: Alright, new data. It happened to crash about 10 minutes after I came in this morning and I ran some stuff in the DDB. I have no idea what information is useful, but perhaps someone will see something out of the ordinary? http://feld.me/freebsd/esx_crash/ Don't know about ESXi but on others VM Managers i can change the chipset emulation from ICH10 to ICH4. Can you change it to an older chipset too? Thanks... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? We've only seen it happen on one virtual machine. That was a 32-bit version. And it's not so much a crash as it is a disk I/O hang. The fact that it was happening regularly to that one VM, while a bunch of other similar VM's were running alongside it without any incident, along with the problem moving with the VM as it is moved from host to host and from Opteron to Xeon, strongly points at something being wrong with the VM itself. Our systems are built mostly by script; I rebuilt the VM a few months ago and the problem vanished. The rebuilt system should have been virtually identical to the original. I never actually compared them though. My working theory was that something bad had happened to the VM during a migration from one datastore to another. We have a really slow-writing iSCSI server that it had been migrated onto for a little bit, which was where the problem first appeared, I believe. At first I thought it was the nightly cron jobs just exceeding the iSCSI server's capacity to cope, so we migrated the VM onto a host with local datastores, and it remained broken thereafter. So my conclusion was that it seemed likely that somehow VMware's thin provisioned disk image had gotten fouled up, and under some unknown use case, it could be teased into locking up further I/O on the VM. I wasn't able to prove it. I tried a read-dd of the entire disk - passed, flying. I tried several things to duplicate the nightly periodic tasks where it seemed so prone to locking up. They all ran fine. But if I left the machine run, it'd do it again eventually. I explained it at the time to one of my VMware friends: But here's where it gets weird. Three times, now, one VM - our Jabber server - has gone wonky in the wee early AM hours. Disk I/O on the VM just locks up. You can type at the console until it does I/O, so you can put in root at the login: prompt but never get a pw prompt. My systems all run top from /etc/ttys and I can see that a whole bunch of processes are stopped in getblk. It's like the iSCSI disk has gone away, except it hasn't, since the other VM's are all happily churning away, on the same datastore, on the same VMware host. http://www.sol.net/tmp/freebsd/freebsd-esxi-lockup.gif Now it's *possible* that the problem actually happens after the 3AM cron run (note slight CPU/memory drop) but the Jabber implosion actually happens around 0530, see drop in memory%. But the root problem at the VM level seems to be that disk I/O has frozen. I can't tell for sure when that happens. All three instances are similar to this. I can't explain this or figure out how to debug it. Since it's locked up right now, thought I'd ping you for ideas before resetting it. Now that was actually before we migrated it back to local datastore, but when we did, the problem remained, suggesting that whatever has happened to the VM, it is contained within the VM's vmdk or other files. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thursday 29 March 2012 17:49:30 Joe Greco wrote: On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? We've only seen it happen on one virtual machine. That was a 32-bit version. And it's not so much a crash as it is a disk I/O hang. It almost sounds like the lost interrupt issue I've seen with USB EHCI devices, though disk I/O should have a retry timeout? What does wmstat -i output? --HPS ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 10:31:24 -0500, Eduardo Morras nec...@retena.com wrote: Don't know about ESXi but on others VM Managers i can change the chipset emulation from ICH10 to ICH4. Can you change it to an older chipset too? Unfortunately there's no setting in the GUI for that but I'll keep looking to see if there's a hidden option -- perhaps in the VM's config file. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 10:55:36 -0500, Hans Petter Selasky hsela...@c2i.net wrote: It almost sounds like the lost interrupt issue I've seen with USB EHCI devices, though disk I/O should have a retry timeout? What does wmstat -i output? --HPS Here's a server that has a week uptime and is due for a crash any hour now: root@server:/# vmstat -i interrupt total rate irq1: atkbd0 34 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em1778061 1 irq17: mpt0 19217711 31 irq18: em0 283674769460 cpu0: timer246571507400 Total 550242125892 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 10:49:30 -0500, Joe Greco jgr...@ns.sol.net wrote: I explained it at the time to one of my VMware friends: This is 100% identical to what we see, Joe! And we're so unlucky that we have this happen on probably a dozen servers, but a handful are the really bad ones. We've rebuilt them from scratch many times with no improvement. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
This sounds just like a race condition that happens under Windows 7 on this laptop. The race condition, as far as I can tell involves heavy disk access and heavy network access, and usually leaves the drive light on, while all activity monitors (alldisk, allcpu, allnetwork) are still active, although on this laptop disk takes priority, and network slows to a crawl. occasionally, the mouse will stop working, along with everything else, but usually not. keyboard is lower priority, and doesn't do anything. You might want to check with mickeysoft, this might just be their problem. This sounds so freaking similar to the issue I get, and I think it's a race condition (shared interrupts??). This laptop is a Compaq Presario C300 series, with the 945GM chipset and a T7600 Core2 Duo CPU, with 3G of RAM. Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time, and we were hoping we could base some stuff on 8.3 for long term stability...) ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others. History: Over the course of the last 2 years we've been banging our heads on the wall. VMWare is done debugging this. They claim it's not a VMWare issue. They can't identify what the heck happens. We had a glimmer of hope with ESXi 5.0 fixing it because we never saw any crashes in the handful of deployments, but our dreams were crushed today -- two days before an outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server and FreeBSD 8.2 guest crashed. Crash Details: The keyboard/mouse usually stops responding for input on the console; normally we can't type in a username or password. However, we can switch VTs. If there's a shell on the console and we can type, we can only run things in memory. Any time we try to access the disk it will hang indefinitely. The server still has network access. We can ping it without issue. SSH of course kicks you out because it can't do any I/O. If we were to serve a lightweight http server off a memory backed filesystem I'm confident it would run just fine as long as it wasn't logging or anything. On ESXi you see that there is a CPU spike of 100% that goes on indefinitely. No idea what the FreeBSD OS itself thinks it is doing because we can't run top during the crash. This crash can affect a server and happen multiple times a week. It can also not show up for 180 days or more. But it does happen. The server can be 100% idle and crash. We have servers that do more I/O than the ones that crash could ever attempt to do and these don't crash at all. Completely inexplicable. Things we've looked into: Nothing about the installed software matters. We've tried cross referencing the crashed servers by the programs they run but the base OS is the only common denominator due to the wide variety of servers it has affected. Storage doesn't matter. We've tried different iSCSI SANs, we've tried different switches, we've tried local datastores on the ESXi servers themselves. HP servers, Dell servers -- doesn't seem to matter either. (All with latest firmwares, BIOSes, etc) VMWare gave us a ton of debugging tasks, and we've given them gigabytes of debugging info and data; they can't find anything. VMWare tools -- with, without, using open-vm-tools makes no difference. I think we've done a fair job ruling out VMWare. I think we've finally found enough data that this is definitely something in the FreeBSD world. I'm going to begin prepping some of the known crashy servers with more debugging. Any suggestions on what I should build the kernel with? They never do a proper panic, but I definitely want to at least *try* to get into the debugger the next time it crashes. And when it crashes, what the heck should I be running? I've never played with the KDB before... Thank you for any suggestions and help you can give me ___ freebsd-hack...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, Mar 29, 2012 at 11:27 AM, Mark Felder f...@feld.me wrote: On Thu, 29 Mar 2012 10:55:36 -0500, Hans Petter Selasky hsela...@c2i.net wrote: It almost sounds like the lost interrupt issue I've seen with USB EHCI devices, though disk I/O should have a retry timeout? What does wmstat -i output? --HPS Here's a server that has a week uptime and is due for a crash any hour now: root@server:/# vmstat -i interrupt total rate irq1: atkbd0 34 0 irq6: fdc0 9 0 irq15: ata1 34 0 irq16: em1778061 1 irq17: mpt0 19217711 31 irq18: em0 283674769460 cpu0: timer246571507400 Total 550242125892 Not so long ago, VMware implemented a clever scheme for reducing the overhead of virtualized interrupts that must be delivered by at least some (if not all) of their emulated storage controllers: http://static.usenix.org/events/atc11/tech/techAbstracts.html#Ahmad Perhaps, there is a bad interaction between this scheme and FreeBSD's mpt driver. Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/29/2012 07:03, Mark Felder wrote: Alright, new data. It happened to crash about 10 minutes after I came in this morning and I ran some stuff in the DDB. I have no idea what information is useful, but perhaps someone will see something out of the ordinary? http://feld.me/freebsd/esx_crash/ If this is an interrupt problem with disk i/o, then you might want to look into (DDB(4)) show intr show intrcount maybe show allrman -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk90lloACgkQrDN5kXnx8yaCZACbBamQksNyWC26PUsOn5N9LJLV ql0AoJwYCFDfXhCpZIN735V9qg0VepFf =fCLN -END PGP SIGNATURE- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 11:43:45 -0500 Jim Bryant articulated: Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time, and we were hoping we could base some stuff on 8.3 for long term stability...) ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others. History: Over the course of the last 2 years we've been banging our heads on the wall. VMWare is done debugging this. They claim it's not a VMWare issue. They can't identify what the heck happens. We had a glimmer of hope with ESXi 5.0 fixing it because we never saw any crashes in the handful of deployments, but our dreams were crushed today -- two days before an outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server and FreeBSD 8.2 guest crashed. Crash Details: The keyboard/mouse usually stops responding for input on the console; normally we can't type in a username or password. However, we can switch VTs. If there's a shell on the console and we can type, we can only run things in memory. Any time we try to access the disk it will hang indefinitely. The server still has network access. We can ping it without issue. SSH of course kicks you out because it can't do any I/O. If we were to serve a lightweight http server off a memory backed filesystem I'm confident it would run just fine as long as it wasn't logging or anything. On ESXi you see that there is a CPU spike of 100% that goes on indefinitely. No idea what the FreeBSD OS itself thinks it is doing because we can't run top during the crash. This crash can affect a server and happen multiple times a week. It can also not show up for 180 days or more. But it does happen. The server can be 100% idle and crash. We have servers that do more I/O than the ones that crash could ever attempt to do and these don't crash at all. Completely inexplicable. Things we've looked into: Nothing about the installed software matters. We've tried cross referencing the crashed servers by the programs they run but the base OS is the only common denominator due to the wide variety of servers it has affected. Storage doesn't matter. We've tried different iSCSI SANs, we've tried different switches, we've tried local datastores on the ESXi servers themselves. HP servers, Dell servers -- doesn't seem to matter either. (All with latest firmwares, BIOSes, etc) VMWare gave us a ton of debugging tasks, and we've given them gigabytes of debugging info and data; they can't find anything. VMWare tools -- with, without, using open-vm-tools makes no difference. I think we've done a fair job ruling out VMWare. I think we've finally found enough data that this is definitely something in the FreeBSD world. I'm going to begin prepping some of the known crashy servers with more debugging. Any suggestions on what I should build the kernel with? They never do a proper panic, but I definitely want to at least *try* to get into the debugger the next time it crashes. And when it crashes, what the heck should I be running? I've never played with the KDB before... Thank you for any suggestions and help you can give me This sounds just like a race condition that happens under Windows 7 on this laptop. The race condition, as far as I can tell involves heavy disk access and heavy network access, and usually leaves the drive light on, while all activity monitors (alldisk, allcpu, allnetwork) are still active, although on this laptop disk takes priority, and network slows to a crawl. occasionally, the mouse will stop working, along with everything else, but usually not. keyboard is lower priority, and doesn't do anything. You might want to check with mickeysoft, this might just be their problem. This sounds so freaking similar to the issue I get, and I think it's a race condition (shared interrupts??). This laptop is a Compaq Presario C300 series, with the 945GM chipset and a T7600 Core2 Duo CPU, with 3G of RAM. {TOP POSTING CORRECTED} I just started reading this tread, but I am wondering if I missed something here. What does this have to do with Windows 7? -- Jerry ♔ Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 12:05:30 -0500, Mark Atkinson atkin...@gmail.com wrote: If this is an interrupt problem with disk i/o, then you might want to look into (DDB(4)) show intr show intrcount maybe show allrman Thank you! I really don't know what things we should be running in DDB to diagnose this and we will try this upon the next crash. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 12:24:30 -0500, je...@seibercom.net wrote: I just started reading this tread, but I am wondering if I missed something here. What does this have to do with Windows 7? I emailed him off-list but I'm guessing he thought this was on VMWare Workstation or another product that would virtualize FreeBSD on top of Windows as the host OS. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 11:53:02 -0500, Alan Cox alan.l@gmail.com wrote: Not so long ago, VMware implemented a clever scheme for reducing the overhead of virtualized interrupts that must be delivered by at least some (if not all) of their emulated storage controllers: http://static.usenix.org/events/atc11/tech/techAbstracts.html#Ahmad Perhaps, there is a bad interaction between this scheme and FreeBSD's mpt driver. Alan If we assume mpt is the culprit how can I go about diagnosing this more accurately? Is there something I should be looking for in vmstat -i? Too many interrupts? Not enough? Rate too high or too low? Or is this something that is much harder to track down because we're dealing with emulated hardware? If any BSD devs are interested in access to our environment I think we could comply. I might even be able to get authorization to give you an account on the most crash-prone server which doesn't have any sensitive customer data on it. I think at this point we'd even be willing to pay someone to look at a server in this state just so we (and hopefully others) can benefit and hopefully we end up with a more reliable FreeBSD-on-VMWare for everyone. I know Doug mentioned running newer OS versions and that is definitely tempting but because it's not 100% reproducible on demand it's hard to prove it fixes it without waiting 6 months. We're fighting internally here with trust 9.0 fixes it vs jump back to 7.4 because we KNOW it doesn't happen there. Having someone look at this and say oh, yes, that's a deficiency in mpt that appears to be fixed in the newer driver that was MFC'd to 8-STABLE and you'll find in 8.3-RELEASE and 9.0-RELEASE would be more comforting. Thanks to everyone for their time on this! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thursday 29 March 2012 17:49:30 Joe Greco wrote: On Thursday 29 March 2012 15:42:42 Joe Greco wrote: Hi, Do both 32- and 64-bit versions of FreeBSD crash? We've only seen it happen on one virtual machine. That was a 32-bit version. And it's not so much a crash as it is a disk I/O hang. It almost sounds like the lost interrupt issue I've seen with USB EHCI devices, though disk I/O should have a retry timeout? That doesn't seem to fit. Why would a perfectly functional VM suddenly develop this problem when given a slow underlying datastore (fits so far) but then the problem *remains* when returned to a fast local datastore, even on a different host and architecture? And why wouldn't the other VM's running alongside develop the same problem? What does wmstat -i output? No idea, we reloaded the VM months ago. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, Mar 29, 2012 at 1:22 PM, Mark Felder f...@feld.me wrote: If we assume mpt is the culprit Doesn't VMWare offer different types of emulated disk controllers? If so, that might be the easiest way to narrow the field. Another thing maybe to try would be to backport the mpt Also, it's not VMWare's place to claim not our problem when you are paying for support. If this doesn't happen on bare metal, it's a VMWare issue, or they need to demonstrate it's not their issue. At least that would be the expectation I have. There is also a comment on this post indicating someone else with the issue and who has received unofficial vmware feedback. http://www.hailang.me/tech/virtual/freebsd-vmware-esx-a-weird-error-with-san-storage/ And then there is this one with similar symptoms and a workaround: http://forums.freebsd.org/showthread.php?t=27899 -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 15:53:52 -0500, Adam Vande More amvandem...@gmail.com wrote: Doesn't VMWare offer different types of emulated disk controllers? If so, that might be the easiest way to narrow the field. Another thing maybe to try would be to backport the mpt Yes, they offer Paravirtual (not applicable for FreeBSD), LSI Parallel (default option), LSI SAS, and Buslogic (not available for 64bit). Both LSI SAS and LSI Parallel use the mpt driver. Also, it's not VMWare's place to claim not our problem when you are paying for support. If this doesn't happen on bare metal, it's a VMWare issue, or they need to demonstrate it's not their issue. At least that would be the expectation I have. You're right, but we've thrown a ton of money at their support and had direct phone access to their engineers. The best we can get out of them is no indication this is a VMWare problem. It's easy for them to blow people off when they're as big as they've grown to be. There is also a comment on this post indicating someone else with the issue and who has received unofficial vmware feedback. http://www.hailang.me/tech/virtual/freebsd-vmware-esx-a-weird-error-with-san-storage/ I found that post ages ago and that's me, mf, as the only person to comment on it. Unfortunately our problem does not align with what he's describing. And then there is this one with similar symptoms and a workaround: http://forums.freebsd.org/showthread.php?t=27899 I'm now investigating those loader.conf options. I have my crashy machine set to use them on next boot so we'll see if it crashes now that I'm using LSI SAS emulated controller. If it still crashes, we'll see what happens after that with those loader.conf options enabled. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 3/29/2012 7:01 AM, Joe Greco wrote: On 3/28/2012 1:59 PM, Mark Felder wrote: FreeBSD 8-STABLE, 8.3, and 9.0 are untested As much as I'm sensitive to your production requirements, realistically it's not likely that you'll get a helpful result without testing a newer version. 8.2 came out over a year ago, many many things have changed since then. Doug So you're saying that he should have been using 8.3-RELEASE, then. That isn't what I said at all, sorry if I wasn't clear. The OP mentioned 9.0-RELEASE, and in the context of his message (which I snipped) he mentioned 8-stable. That's what I was referring to. If you'll kindly go over to http://www.freebsd.org and look under Latest Releases, please note that 8.2 is a production release. If you don't want it to be a production release, then find a way to make it so, but please don't snipe at people who are using the code that the FreeBSD project has indicated is a current production offering. There are many good reasons not to run arbitrary snapshots on your production gear. It's unrealistic to expect people to run non- RELEASE non-production code on their production gear. We can have that discussion if you don't understand that, drop me a note off- list and I'll be happy to explain it. I can see that you're upset about something, sorry if my message caused you additional stress. I actually understand the realities of production environments quite well, and believe it or not I agree with some of your frustration about how we handle support for our supported releases. We've had various public threads about these issues, which have sparked some quite-lively private discussions amongst our committers, and I'm hoping that once the long-overdue 8.3-RELEASE is out we'll be able to buckle down and start putting some of those ideas into action. Meanwhile, this is still a volunteer project, and as a result sometimes the best way to get attention to a problem is to verify that it hasn't already been fixed. You've been around more than long enough to understand this Joe. We can spend time arguing about what *should* be (actually we can't ...) but my point was in trying to help the OP get the most/best help the fastest way possible. Doug ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
And then there is this one with similar symptoms and a workaround: http://forums.freebsd.org/showthread.php?t=3D27899 I'm now investigating those loader.conf options. I have my crashy machine set to use them on next boot so we'll see if it crashes now that I'm using LSI SAS emulated controller. If it still crashes, we'll see what happens after that with those loader.conf options enabled. Um, if I may, that's something completely different. VMDirectPath, or PCIe passthru, is making a hardware device on a VMware host available directly to a guest. It'll take your LSI controller, in the example cited, and make it unavailable to VMware ESXi, and present it instead inside the guest environment. You do this when you have an app whose performance would suffer greatly when made to operate through the indirection that a VM naturally lives in; for example, it is quite common for FreeNAS users to pass a disk controller through to a VM guest in order to allow a virtualized FreeNAS instance to directly manage the physical disks. In that case, there are some issues with ESXi and interrupt delivery to the guest VM; virtualization doesn't actually get rid of the possibility of ESXi problems, since the hypervisor is still ultimately involved. It is certainly possible that there's some common issue involving interrupt delivery somehow, but I wouldn't get my hopes up. It also doesn't explain the experience here, where one VM basically crapped out but only after a migration - and then stayed crapped out. It would be interesting to hear about your datastore, how busy it is, what technology, whether you're using thin, etc. I just have this real strong feeling that it's some sort of corruption with the vmfs3 and thin provisioned disk format, but it'd be interesting to know if that's totally off-track. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco jgr...@ns.sol.net wrote: It also doesn't explain the experience here, where one VM basically crapped out but only after a migration - and then stayed crapped out. It would be interesting to hear about your datastore, how busy it is, what technology, whether you're using thin, etc. I just have this real strong feeling that it's some sort of corruption with the vmfs3 and thin provisioned disk format, but it'd be interesting to know if that's totally off-track. We've ruled out SAN, but we haven't ruled out VMFS. Even FreeBSD Guests on standalone ESXi servers with no SAN exhibit this crash. For the record, we only use thick provisioning and if it was corruption I'm not sure what layer the corruption could be at. The crashy servers show no abnormalities when I run either `freebsd-update IPS` or `pkg_libchk` to confirm checksums of all installed programs. Now the other data on there... it's not exactly verified, but our backups via rsnapshot seem to prove there is no issue there or we'd have lots of new files each run. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On 28/03/2012 22:59, Mark Felder wrote: Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time, and we were hoping we could base some stuff on 8.3 for long term stability...) ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others. History: Over the course of the last 2 years we've been banging our heads on the wall. VMWare is done debugging this. They claim it's not a VMWare issue. They can't identify what the heck happens. We had a glimmer of hope with ESXi 5.0 fixing it because we never saw any crashes in the handful of deployments, but our dreams were crushed today -- two days before an outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server and FreeBSD 8.2 guest crashed. Crash Details: The keyboard/mouse usually stops responding for input on the console; normally we can't type in a username or password. However, we can switch VTs. If there's a shell on the console and we can type, we can only run things in memory. Any time we try to access the disk it will hang indefinitely. The server still has network access. We can ping it without issue. SSH of course kicks you out because it can't do any I/O. If we were to serve a lightweight http server off a memory backed filesystem I'm confident it would run just fine as long as it wasn't logging or anything. On ESXi you see that there is a CPU spike of 100% that goes on indefinitely. No idea what the FreeBSD OS itself thinks it is doing because we can't run top during the crash. This crash can affect a server and happen multiple times a week. It can also not show up for 180 days or more. But it does happen. The server can be 100% idle and crash. We have servers that do more I/O than the ones that crash could ever attempt to do and these don't crash at all. Completely inexplicable. Things we've looked into: Nothing about the installed software matters. We've tried cross referencing the crashed servers by the programs they run but the base OS is the only common denominator due to the wide variety of servers it has affected. Storage doesn't matter. We've tried different iSCSI SANs, we've tried different switches, we've tried local datastores on the ESXi servers themselves. HP servers, Dell servers -- doesn't seem to matter either. (All with latest firmwares, BIOSes, etc) VMWare gave us a ton of debugging tasks, and we've given them gigabytes of debugging info and data; they can't find anything. VMWare tools -- with, without, using open-vm-tools makes no difference. I think we've done a fair job ruling out VMWare. I think we've finally found enough data that this is definitely something in the FreeBSD world. I'm going to begin prepping some of the known crashy servers with more debugging. Any suggestions on what I should build the kernel with? They never do a proper panic, but I definitely want to at least *try* to get into the debugger the next time it crashes. And when it crashes, what the heck should I be running? I've never played with the KDB before... Thank you for any suggestions and help you can give me ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org Sorry, coming a bit late to the party, I have seen similar behavior on a few vm. All of them either Debian and FreeBSD. Even though CPU indication are not necessarily relevant in a VM, vi launched through crontab -e would take insane amount of CPU (up to 84%) and Apache was hanging around 350% 400% (quad CPU VM). Now the thing is that making a VM snapshot and deploying the snapshot a while later, or on a different (way less loaded) VMWare platform would basically make it perfectly usable again. Shutting down the VM and starting it again with only one CPU would also basically solve the problem. In a way Debian seemed to be able to survive the crisis but Disk I/O have latencies of many seconds, sometimes minutes. This would happen only on heavily loaded VMWare. In a quite similar way older version of Debian never shown the problem. Can you test whether you have similar behavior on your platform ? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Again, it's starting to sound like an interrupt handling issue which may or may not be limited to the storage device. You'll have to engage someone who knows those device drivers and likely have them add some debugging to the driver which can be easily flipped on (via binaries in a ramdisk - very important if you can't run sysctl because your disk IO has locked up!) to see what the current state of things. It's likely that the BSD mpt(4) and other storage drivers, and/or our interrupt handling code, is just slightly different enough to confuse the snot out of VMWare. I'd first look at the obvious - (eg, if you've just stopped receiving interrupts, even if new IO is scheduled). I'd also ask VMware if they have any tools that they can run on a VM to get the state of the internal emulated driver. For example, register dumps of the device to see if it's in a hung state, register dumps of the PIC/APIC to see what state they're in, etc. Maybe pull in someone like ixsystems and see if they can help debug this kind of stuff? If you're paying vmware for support, you could pull them into things with ixsystems and see if the two of them can help you sort this out? Thanks, Adrian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time, and we were hoping we could base some stuff on 8.3 for long term stability...) ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others. History: Over the course of the last 2 years we've been banging our heads on the wall. VMWare is done debugging this. They claim it's not a VMWare issue. They can't identify what the heck happens. We had a glimmer of hope with ESXi 5.0 fixing it because we never saw any crashes in the handful of deployments, but our dreams were crushed today -- two days before an outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server and FreeBSD 8.2 guest crashed. Crash Details: The keyboard/mouse usually stops responding for input on the console; normally we can't type in a username or password. However, we can switch VTs. If there's a shell on the console and we can type, we can only run things in memory. Any time we try to access the disk it will hang indefinitely. The server still has network access. We can ping it without issue. SSH of course kicks you out because it can't do any I/O. If we were to serve a lightweight http server off a memory backed filesystem I'm confident it would run just fine as long as it wasn't logging or anything. On ESXi you see that there is a CPU spike of 100% that goes on indefinitely. No idea what the FreeBSD OS itself thinks it is doing because we can't run top during the crash. This crash can affect a server and happen multiple times a week. It can also not show up for 180 days or more. But it does happen. The server can be 100% idle and crash. We have servers that do more I/O than the ones that crash could ever attempt to do and these don't crash at all. Completely inexplicable. Things we've looked into: Nothing about the installed software matters. We've tried cross referencing the crashed servers by the programs they run but the base OS is the only common denominator due to the wide variety of servers it has affected. Storage doesn't matter. We've tried different iSCSI SANs, we've tried different switches, we've tried local datastores on the ESXi servers themselves. HP servers, Dell servers -- doesn't seem to matter either. (All with latest firmwares, BIOSes, etc) VMWare gave us a ton of debugging tasks, and we've given them gigabytes of debugging info and data; they can't find anything. VMWare tools -- with, without, using open-vm-tools makes no difference. I think we've done a fair job ruling out VMWare. I think we've finally found enough data that this is definitely something in the FreeBSD world. I'm going to begin prepping some of the known crashy servers with more debugging. Any suggestions on what I should build the kernel with? They never do a proper panic, but I definitely want to at least *try* to get into the debugger the next time it crashes. And when it crashes, what the heck should I be running? I've never played with the KDB before... Thank you for any suggestions and help you can give me ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hi, * have you filed a PR? * is the crash easily reproducable? * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create a ramdisk image using nanobsd?) and do some stress testing inside that? It sounds like you've established it's a storage issue, or at least interrupt handling for storage issue. So I'd definitely try the ramdisk-only boot and thrash it using lighttpd/httperf or something. If that survives fine, I'd look at trying to establish whether there's something wrong in the disk driver(s) freebsd is using. I'm not that cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x and 8.0 which has caused this to surface. 2c, Adrian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Can someone please help me set up my tata-photon (huawei) modem ?
Hi all, My tata-photon (huawei) modem works painlessly on Win XP. Every setting is auto (including getting the nameserver). The auth protocol is CHAP. But when I boot into FreeBSD 8.0 (stable), and run 'ppp -auto pmdemand' followed by 'ping [1]www.google.com' (or yahoo or any other site, for that matter), I always get Hostname lookup failure. BTW, I don't have an /etc/resolv.conf. I only configured ppp.conf (please see below for details). Can someone please help me set up the modem so that I can dial out and connect to the internet from my FreeBSD box ? Thanks in advance. Manish Jain This is my ppp.conf : default: set log Phase Chat LCP IPCP CCP tun command allow users guru bourne # ident user-ppp VERSION (built COMPILATIONDATE) set device /dev/cuaU0 set speed 115200 set dial ABORT BUSY ABORT NO\\sCARRIER TIMEOUT 5 \ \\ AT OK-AT-OK ATFE0V1X1D2C1S0=0 OK \\dATDT\\T TIMEOUT 40 CONNECT set timeout 180 # 3 minute idle timer (the default) enable dns # request DNS info (for resolv.conf) pmdemand: set phone #777 set login set authname internet set authkey set timeout 0 disable ipv6cp set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.0 0.0.0.0 add default HISADDR accept CHAP References 1. http://www.google.com/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Can someone please help me set up my tata-photon (huawei) modem ?
El día Tuesday, June 28, 2011 a las 01:47:31PM +0530, Manish Jain escribió: Hi all, My tata-photon (huawei) modem works painlessly on Win XP. Every setting is auto (including getting the nameserver). The auth protocol is CHAP. But when I boot into FreeBSD 8.0 (stable), and run 'ppp -auto pmdemand' followed by 'ping [1]www.google.com' (or yahoo or any other site, for that matter), I always get Hostname lookup failure. BTW, I don't have an /etc/resolv.conf. I only configured ppp.conf (please see below for details). Can someone please help me set up the modem so that I can dial out and connect to the internet from my FreeBSD box ? Thanks in advance. Manish Jain This is my ppp.conf : ... Hello Manish, Before digging into the ppp details (there is a log about in /var/log/ppp.log), do you have after attaching the Huawei modem a device /dev/cuaU0 at all? What does the following command gives: ls -l /dev/cua* chat -vs /dev/cuaU0.0 /dev/cuaU0.0 '' AT OK 'ATI2' O printf \n HIH matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Can someone please help me set up my tata-photon (huawei) modem ?
Hello Matthias, Yes, I do have /dev/cuaU0.0 along with corresponding lock and init files, although I cannot give you the output you want right now since I am booted into XP with the FreeBSD 8.2 dvd image being downloaded. (This will hopefully finish in about 15 hours, as cygwin::ncftpget suggests). I also tried a few other things, most notably installing usb_modeswitch, and putting this into /etc/devd.conf : attach 100 { device-name ugen[0-9]+; match vendor 0x12d1; match product 0x1446; action /usr/local/bin/usb_modeswitch; }; Running kldload for u3g and ucom indicates they are both already loaded. Running usb_modeswitch seems to have also generated some /dev/ttyUn.n files, along with corresponding lock and init files - just like the cua entries. The only thing that seems out of place with the current setup is that usb_modeswitch's wrapper (usb_modeswitch_dispatcher) needs /usr/local/bin/tclsh, and for some reason the FreeBSD installer installed it as /usr/local/bin/tclsh8.2 I'll try fixing the above tomorrow and will try out anything you/someone else might have to suggest. BTW, I made a slight error in my original message. My system's release version is 8.1, not 8.0. Thank you Regards Manish Jain [1]invalid.poin...@gmail.com On 29-Jun-11 00:16, Matthias Apitz wrote: El día Tuesday, June 28, 2011 a las 01:47:31PM +0530, Manish Jain escribió: Hi all, My tata-photon (huawei) modem works painlessly on Win XP. Every setting is auto (including getting the nameserver). The auth protocol is CHAP. But when I boot into FreeBSD 8.0 (stable), and run 'ppp -auto pmdemand' followed by 'ping [1]www.google.com' (or yahoo or any other site, for that matter), I always get Hostname lookup failure. BTW, I don't have an /etc/resolv.conf. I only configured ppp.conf (please see below for details). Can someone please help me set up the modem so that I can dial out and connect to the internet from my FreeBSD box ? Thanks in advance. Manish Jain This is my ppp.conf : ... Hello Manish, Before digging into the ppp details (there is a log about in /var/log/ppp.log), do you have after attaching the Huawei modem a device /dev/cuaU0 at all? What does the following command gives: ls -l /dev/cua* chat -vs /dev/cuaU0.0 /dev/cuaU0.0 '' AT OK 'ATI2' O printf \n HIH matthias References 1. mailto:invalid.poin...@gmail.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Can someone please help me set up my tata-photon (huawei) modem ?
Hello Matthias, I got it to work ! I symlinked /usr/local/bin/tclsh8.2 as tclsh, but that was probably not the only factor involved. In a streak of unintended brilliance, I had set the device in ppp.conf as /dev/cuaU0, whereas there is no such device - as /var/log/ppp.log complained. (Thanks for your suggestion for looking up the log). I changed /dev/cuaU0 to /dev/cuaU0.0, as a matter of courtesy. Now my FreeBSD box is connected to the internet and I don't have to boot into XP again (as long as I can hope). Thanks again Manish Jain On 29-Jun-11 00:55, Manish Jain wrote: Hello Matthias, Yes, I do have /dev/cuaU0.0 along with corresponding lock and init files, although I cannot give you the output you want right now since I am booted into XP with the FreeBSD 8.2 dvd image being downloaded. (This will hopefully finish in about 15 hours, as cygwin::ncftpget suggests). I also tried a few other things, most notably installing usb_modeswitch, and putting this into /etc/devd.conf : attach 100 { device-name ugen[0-9]+; match vendor 0x12d1; match product 0x1446; action /usr/local/bin/usb_modeswitch; }; Running kldload for u3g and ucom indicates they are both already loaded. Running usb_modeswitch seems to have also generated some /dev/ttyUn.n files, along with corresponding lock and init files - just like the cua entries. The only thing that seems out of place with the current setup is that usb_modeswitch's wrapper (usb_modeswitch_dispatcher) needs /usr/local/bin/tclsh, and for some reason the FreeBSD installer installed it as /usr/local/bin/tclsh8.2 I'll try fixing the above tomorrow and will try out anything you/someone else might have to suggest. BTW, I made a slight error in my original message. My system's release version is 8.1, not 8.0. Thank you Regards Manish Jain [1]invalid.poin...@gmail.com On 29-Jun-11 00:16, Matthias Apitz wrote: El día Tuesday, June 28, 2011 a las 01:47:31PM +0530, Manish Jain escribió: Hi all, My tata-photon (huawei) modem works painlessly on Win XP. Every setting is auto (including getting the nameserver). The auth protocol is CHAP. But when I boot into FreeBSD 8.0 (stable), and run 'ppp -auto pmdemand' followed by 'ping [1]www.google.com' (or yahoo or any other site, for that matter), I always get Hostname lookup failure. BTW, I don't have an /etc/resolv.conf. I only configured ppp.conf (please see below for details). Can someone please help me set up the modem so that I can dial out and connect to the internet from my FreeBSD box ? Thanks in advance. Manish Jain This is my ppp.conf : ... Hello Manish, Before digging into the ppp details (there is a log about in /var/log/ppp.log), do you have after attaching the Huawei modem a device /dev/cuaU0 at all? What does the following command gives: ls -l /dev/cua* chat -vs /dev/cuaU0.0 /dev/cuaU0.0 '' AT OK 'ATI2' O printf \n HIH matthias References 1. mailto:invalid.poin...@gmail.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Please help me
hi, yesterday i have modify squid.conf file.i have use vi editior.but i cannot delete in text message. following error ^? appear.How to do it?. please help me. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me
On Wed, Jan 20, 2010 at 01:10:48PM +0630, komyo gyi wrote: hi, yesterday i have modify squid.conf file.i have use vi editior.but i cannot delete in text message. following error ^? appear.How to do it?. please help me. Looks like you're hitting the Delete key. That's not a valid vi command. You need to be in command mode and hit the 'x' key. -- Jonathan Chen j...@chen.org.nz -- The human mind ordinarily operates at only ten percent of its capacity -- the rest is overhead for the operating system. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: please help me make sense of top's CPU output
Dan Nelson wrote: Junior Hacker Project: add an instantaneous-CPU value (calculated by subtracting successive ki_runtime values) to the list of things top calculates and toggle it and weighted-CPU when pressing C. The toggling code is already there; it just toggles between two different weighted-cpu values at the moment. Makes sense, thank you. If I want to hack a port program, I go to the work directory, edit the source, and rebuild. How do I hack a non-port program like top? Chris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: please help me make sense of top's CPU output
2009/11/3 Chris Stankevitz cstankev...@toyon.com: Dan Nelson wrote: Junior Hacker Project: add an instantaneous-CPU value (calculated by subtracting successive ki_runtime values) to the list of things top calculates and toggle it and weighted-CPU when pressing C. The toggling code is already there; it just toggles between two different weighted-cpu values at the moment. Makes sense, thank you. If I want to hack a port program, I go to the work directory, edit the source, and rebuild. How do I hack a non-port program like top? Chris Look in the Makefile for /usr/src/usr.bin/top, and you'll see the source is in /usr/src/contrib/top Hack away! Chris -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in a mailing list? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
please help me make sense of top's CPU output
Hello, I recently performed a CPU intensive task with Xorg. When I completed the task and Xorg no longer was using the CPU, I got this result from top: === last pid: 1201; load averages: 0.24, 0.10, 0.09up 0+00:29:42 63 processes: 1 running, 62 sleeping CPU: 1.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.0% idle Mem: 161M Active, 67M Inact, 68M Wired, 1240K Cache, 41M Buf, 1676M Free Swap: 4060M Total, 4060M Free PID USERNAME THR PRI NICE SIZERES STATE C TIMECPU 1017 cstankevitz1 1040 366M 331M select 0 3:25 35.89% Xorg === Note that the CPU row reports 99% idle. Note that the CPU column reports 36% Xorg I have two questions: 1. Why do these two numbers seem to not agree? One reports the CPU is not being used, the other reports Xorg is using the CPU. 2. How can I change my system so that these two numbers seem to agree? Thank you, Chris PS: conky does the same thing -- I assume this means the seemingly disagreeing numbers are coming from the FreeBSD kernel. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: please help me make sense of top's CPU output
In the last episode (Nov 02), Chris Stankevitz said: I recently performed a CPU intensive task with Xorg. When I completed the task and Xorg no longer was using the CPU, I got this result from top: === last pid: 1201; load averages: 0.24, 0.10, 0.09up 0+00:29:42 63 processes: 1 running, 62 sleeping CPU: 1.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.0% idle Mem: 161M Active, 67M Inact, 68M Wired, 1240K Cache, 41M Buf, 1676M Free Swap: 4060M Total, 4060M Free PID USERNAME THR PRI NICE SIZERES STATE C TIMECPU 1017 cstankevitz1 1040 366M 331M select 0 3:25 35.89% Xorg The CPU column in the process list is a decaying average (more useful to the kernel scheduler than an instantaneous value). You'll see it slowly drop to 0 over 10-15 seconds. Junior Hacker Project: add an instantaneous-CPU value (calculated by subtracting successive ki_runtime values) to the list of things top calculates and toggle it and weighted-CPU when pressing C. The toggling code is already there; it just toggles between two different weighted-cpu values at the moment. -- Dan Nelson dnel...@allantgroup.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Please help me win Ipod nano :)
Hello friends! I want to ask you for help in winning ipod nano, if you register on biggest social network here: http://vkontakte.ru/reg632660 i'm can win :) Thank you! :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Some Questions About Freebsd Please Help Me !!
Hi , i want to ask some questions about freebsd , one of my friend have freebsd in his server , he is using it , he have ips issu on his server and he is converting ips in proxies ( Socks 4/5 ) , i want to know how i can do that , how i can set firewall that or what i need to do? , like let me show you what he have did , he have give me SSH access , in that when i go i need to put commands , i will give you some commands , to make ip as proxy i need to give this command in ( Putty ) socks -d -p14344 -i204.18.245.9 -e204.18.245.9 , it will convert ip in proxy , but i dun know how to do that in freebsd , i will show you 1 screenshot as well , here is screenshot ( http://i36.tinypic.com/wuoro6.png ) , you can have a look on that as well , please help me if you can , like this i am going to buy may be 20 freebsd for that , i want to know how can i set all this in that , if you can help me in that , please send me steps how i can make ip in proxy with the help of freebsd. Here is Some More Commands. To Stop Socks Here is Command : killall -9 socks To Start Socks Here is Command : socks -d -p14344 -i204.18.245.9 -e204.18.245.9 Regrads Bravo italy 00393888992300 Alice Messenger ;-) chatti anche con gli amici di Windows Live Messenger e tutti i telefonini TIM! Vai su http://maileservizi.alice.it/alice_messenger/index.html?pmk=footer ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please Help Me ...
On Mon, Feb 16, 2009 at 08:15:41AM +0330, Ali Reza wrote: Hi i am Install FreeBsd 7.1 in to Detacetad Server where Are Support Cpanel Whm .. ? Please Answer Me ... cPanel and WHM are commercial products: http://www.cpanel.net/products/cpwhm/cpanel11/index.html Apparently, they also support FreeBSD, since many FreeBSD web hosters do provide cPanel to their customers. Just have a look at their website and ask there. ;-) My Blog http://weblog.teamnet.ir Regards, -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Please Help Me ...
Hi i am Install FreeBsd 7.1 in to Detacetad Server where Are Support Cpanel Whm .. ? Please Answer Me ... -- My Blog http://weblog.teamnet.ir Free Download From Rapidshare ... http://www.libertyreserve.com/?ref=U123456 http://rapidbaz.com/from/51289 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Please help me with my PF config
2008/3/7, Preston Hagar [EMAIL PROTECTED]: It looks like you already have your problem solved. One utility you might want to look at is pftop. With it, you can see pretty much in real time what is going through pf and what is being blocked. This has helped me a lot to find out which rule in blocking something I need to let through. If you run pftop, then hit the right arrow key, it will have the rule that is being applied for a given connection on the far right-hand side. If you hit SHIFT+S it will order the connections by source ip to help you find what you are looking for. Once you have the rule number that is blocking the connection you need to let through, hit the right arrow 4 more times to see the list of rules and their corresponding numbers. It may or may not be of much use to you, but I have found pftop to be indispensable when setting up pf firewalls. Preston It looks great. I was looking for something like this because I would like to see what is happening in the network. I'm gonna install this on monday. Thankz for the tip. Hugs, Alaor Neto. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me with my PF config
On Thu, Mar 6, 2008 at 12:02 PM, Alaor Barroso de Carvalho Neto [EMAIL PROTECTED] wrote: 2008/3/6, Erik Norgaard [EMAIL PROTECTED]: I know my config is far away from a good config but it's the first time I configure an firewall, and I have only basic english knowledge, I'm not totally sure about I can and I can not do, even since I read the tutorials, because my english skills aren't good enough. The IN and OUT stuff is very confusing for me yet. But thankz a lot, it's working now. Hugs, Alaor Neto It looks like you already have your problem solved. One utility you might want to look at is pftop. With it, you can see pretty much in real time what is going through pf and what is being blocked. This has helped me a lot to find out which rule in blocking something I need to let through. If you run pftop, then hit the right arrow key, it will have the rule that is being applied for a given connection on the far right-hand side. If you hit SHIFT+S it will order the connections by source ip to help you find what you are looking for. Once you have the rule number that is blocking the connection you need to let through, hit the right arrow 4 more times to see the list of rules and their corresponding numbers. It may or may not be of much use to you, but I have found pftop to be indispensable when setting up pf firewalls. Preston ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Please help me with my PF config
Hi guyz, let me explain what I have. I work in a school, we have access to the internet, two internal networks (academic and administrative) and we have to connect to some servers in another school because we share databases and to video-conference. I have a FreeBSD box with PF and squid, i want all my web traffic to pass through the squid, it's working. I want to academic net don't be able to communicate with administrative net, and the inverse, it's working. But I would like to my adm net to communicate with some servers in the other school network, and only this servers, no other ip would be accessible, it's NOT working. I can ping to the servers but I can't connect to the services ports (SQL Server, and so on). Here's my pf.conf: BEGIN OF CONFIG ext_if=em0 adm_if=xl0 acad_if=xl1 cefet_if=xl2 all_if={ em0, xl0, xl1, xl2 } ext_net=XXX.XXX.XXX.XXX/XX adm_net=192.168.1.0/24 acad_net=192.168.2.0/24 cefet_net=10.10.0.0/16 cefet_servers={ 10.10.0.10, 10.10.0.15, 10.10.0.213 } internal_nets={ 192.168.1.0/24, 192.168.2.0/24 } tcp_services={ ssh, smtp, domain, http, https, ftp, ftp-data, nntp, pop3, pop3s, auth, 3128 } } udp_services={ domain, ntp } proxy_ports={ 80, 8000, 8080, 3128 } martians={ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, 0.0.0.0/8, 240.0.0.0/4 } set block-policy return scrub in all nat on $ext_if from $internal_nets to any - ($ext_if) nat on $cefet_if from $adm_net to any - ($cefet_if) rdr on $all_if proto tcp from any to any port $proxy_ports - 127.0.0.1 port 3128 block all block drop in quick on $ext_if from $martians to any block drop out quick on $ext_if from any to $martians block drop quick from $acad_net to $adm_net block drop quick from $adm_net to $acad_net pass quick proto icmp from any to any keep state pass quick from $adm_net to $cefet_servers keep state pass quick from $cefet_servers to $adm_net keep state block quick from any to $cefet_net block quick from $cefet_net to any pass proto tcp to any port $tcp_services keep state pass proto udp to any port $udp_services keep state antispoof for $all_if END OF CONFIG cefet_net is the network of the other school, and cefet_servers are the servers I want to communicate with, I want all ports and protocols to these servers, but it's not working. I need a light guyz. Thankz, and sorry my poor english. Alaor Neto ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me with my PF config
Alaor Barroso de Carvalho Neto wrote: nat on $ext_if from $internal_nets to any - ($ext_if) nat on $cefet_if from $adm_net to any - ($cefet_if) rdr on $all_if proto tcp from any to any port $proxy_ports \ - 127.0.0.1 port 3128 OK, so do these nat rules actually take effect? Which one? Why do you need nat from adm_net to cefet_net? It appears there is no such need, but then could require a change on cefet_net to tell these hosts the route to adm_net. It is possible that the rdr rule applies even though it appears under the nat rule because rdr is applied on the way in while nat is applied on the way out. rdr is only applied to tcp, the nat rule then is applied to udp and icmp - this will explain why you can ping but not connect with tcp. You can add log statements to your nat rules to see which is applied. pass quick proto icmp from any to any keep state pass quick from $adm_net to $cefet_servers keep state pass quick from $cefet_servers to $adm_net keep state It appears that ping is passed by the first rule, but other protocols are not matched in the second/third rule. block quick from any to $cefet_net block quick from $cefet_net to any Then it is probably blocked here. Some general considerations: When writing your ruleset, make sure to add log in any block statement. That will show you which rule is applied when a packet is blocked. Once things are working remove log statements. Make your rules as specific as possible. State direction and interfaces and avoid the use of any. It is easier to avoid that some other rule take effect than the one you intended. any and lack of direction/interface is fine in policy rules - that is when you explicitly state block all. I usually build my rules like this: block all # default policy block in all block in on $ext_if all ... block in quick on $ext_if all ... block in quick all block out all ... block out quick all block quick all # catch up just in case I messed up It makes it easy to locate any error, and it actually also follows the rules of the pf skip ahead optimizer. Cheers, Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me with my PF config
2008/3/6, Erik Norgaard [EMAIL PROTECTED]: You can add log statements to your nat rules to see which is applied. pass quick proto icmp from any to any keep state pass quick from $adm_net to $cefet_servers keep state pass quick from $cefet_servers to $adm_net keep state It appears that ping is passed by the first rule, but other protocols are not matched in the second/third rule. block quick from any to $cefet_net block quick from $cefet_net to any Then it is probably blocked here. Thankz, brother, it worked. I need the nat to work with the firewall config of the other school. Then, I saw in the log that the traffic going through the 10.10.0.50 (my if) to the servers was being blocked. For me saying that adm_net should communicate with cefet_server would be enough to the firewall understand that it should pass trough any if on the way. I know my config is far away from a good config but it's the first time I configure an firewall, and I have only basic english knowledge, I'm not totally sure about I can and I can not do, even since I read the tutorials, because my english skills aren't good enough. The IN and OUT stuff is very confusing for me yet. But thankz a lot, it's working now. Hugs, Alaor Neto ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Can someone please help me get gdmchooser running, again?
I'm running FreeBSD 6.2-STABLE #1, compiled locally on Tue May 8 16:39:25 CDT 2007 I was able to use gdmchooser on X.org 7.2, but after upgrade to 7.3 (and GDM 2.20.1) she says no serving hosts were found after scanning the local network When I first started this email, I got this from sockstat -l46 | grep 177 lholcombessh-agent 30601 8 udp4 *:177 *:* lholcombegnome-sess 30587 8 udp4 *:177 *:* root Xorg 30570 8 udp4 *:177 *:* root gdm-binary 30569 8 udp4 *:177 *:* root gdm-binary 29955 8 udp4 *:177 *:* Now, after monkeying with it (including a few reboots), I get root gdm-binary 33486 8 udp4 *:177 *:* so it seems that gdm is listening on 177, but the chooser doesn't list the local system as an X server. Instead the chooser reports no serving hosts were found I can log in using the greeter, but I'd really like to use the chooser, as I've got several machines running X, and I'm too cheap to buy extra keyboards, and too lazy to move around the office to do my work :) gdm was compiled WITHOUT IPv6 support, and I've got ipv6_enable=NO in /etc/rc.conf It's gotta be something simple, 'cause I didn't spend much time setting up the chooser originally. But I think I need an extra set of eyes to see what I'm not seeing :) Here is the uncommented part of /usr/local/etc/gdm/custom.conf: [daemon] Greeter=/usr/local/libexec/gdmgreeter RemoteGreeter=/usr/local/libexec/gdmgreeter [security] AllowRemoteRoot=true DisallowTCP=false NeverPlaceCookiesOnNFS=false CheckDirOwner=false [xdmcp] Enable=true [gui] [greeter] IncludeAll=true GraphicalTheme=happygnome-list GraphicalThemeRand=true GraphicalThemes=circles/:happygnome-list/:happygnome [chooser] [debug] Enable=true [servers] 0=Standard [server-Standard] name=Standard server command=/usr/local/bin/X -audit 0 chooser=false handled=true flexible=true priority=0 Thanks for being an extra set of eyes! lane ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Can someone please help me get gdmchooser running, again?
On Fri, Nov 09, 2007 at 11:12:20PM -0600, Lane Holcombe wrote: I'm running FreeBSD 6.2-STABLE #1, compiled locally on Tue May 8 16:39:25 CDT 2007 I was able to use gdmchooser on X.org 7.2, but after upgrade to 7.3 (and GDM 2.20.1) she says no serving hosts were found after scanning the local network When I first started this email, I got this from sockstat -l46 | grep 177 lholcombessh-agent 30601 8 udp4 *:177 *:* lholcombegnome-sess 30587 8 udp4 *:177 *:* root Xorg 30570 8 udp4 *:177 *:* root gdm-binary 30569 8 udp4 *:177 *:* root gdm-binary 29955 8 udp4 *:177 *:* Now, after monkeying with it (including a few reboots), I get root gdm-binary 33486 8 udp4 *:177 *:* so it seems that gdm is listening on 177, but the chooser doesn't list the local system as an X server. Instead the chooser reports no serving hosts were found I can log in using the greeter, but I'd really like to use the chooser, as I've got several machines running X, and I'm too cheap to buy extra keyboards, and too lazy to move around the office to do my work :) gdm was compiled WITHOUT IPv6 support, and I've got ipv6_enable=NO in /etc/rc.conf It's gotta be something simple, 'cause I didn't spend much time setting up the chooser originally. But I think I need an extra set of eyes to see what I'm not seeing :) Here is the uncommented part of /usr/local/etc/gdm/custom.conf: [daemon] Greeter=/usr/local/libexec/gdmgreeter RemoteGreeter=/usr/local/libexec/gdmgreeter [security] AllowRemoteRoot=true DisallowTCP=false NeverPlaceCookiesOnNFS=false CheckDirOwner=false [xdmcp] Enable=true [gui] [greeter] IncludeAll=true GraphicalTheme=happygnome-list GraphicalThemeRand=true GraphicalThemes=circles/:happygnome-list/:happygnome [chooser] [debug] Enable=true [servers] 0=Standard [server-Standard] name=Standard server command=/usr/local/bin/X -audit 0 chooser=false handled=true flexible=true priority=0 Thanks for being an extra set of eyes! lane Hi, Lane, please check gdm related issues (IPv6 only) at http://www.nabble.com/remaining-issues-with-gnome-2.20-t4721430.html (taken from thread on gnome@). And sorry if it's not related to your problem. Yuri ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Can someone please help me get gdmchooser running, again?
On Sat, 2007-11-10 at 08:28 +0300, Yuri Pankov wrote: On Fri, Nov 09, 2007 at 11:12:20PM -0600, Lane Holcombe wrote: I'm running FreeBSD 6.2-STABLE #1, compiled locally on Tue May 8 16:39:25 CDT 2007 snip gdm was compiled WITHOUT IPv6 support, and I've got ipv6_enable=NO in /etc/rc.conf snip Thanks for being an extra set of eyes! lane Hi, Lane, please check gdm related issues (IPv6 only) at http://www.nabble.com/remaining-issues-with-gnome-2.20-t4721430.html (taken from thread on gnome@). And sorry if it's not related to your problem. Yuri Yuri, Thanks for your response. I've suspected this was an issue with IPV6, but poking around sysctl oids is like spelunking without a rope or a partner. As soon as the current CD finishes (Janet Jackson - Rhythm Nation [cause I'm retro, like that]) I'm gonna reboot with these additions to /etc/sysctl.conf: net.inet6.ip6.v6only=0 net.inet6.ip6.forwarding=1 I thought there was a map sysctl oid that explicitly maps ipv6 to ipv4, but I can't find it now. Anyway, if I get anywhere with this I'll post the results. Otherwise ... I'll just whine some more :) lane ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Please Help me...
Hello: Try going to http://www.freebsd.org/releases/6.2R/announce.html in your browser and select from any of the links there for ftp sites with the ISO's. Regards, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:owner-freebsd- [EMAIL PROTECTED] On Behalf Of I am ws:ion Sent: Thursday, September 13, 2007 7:19 PM To: [EMAIL PROTECTED] Subject: Please Help me... I want to download Free BSD This Link http://www.freebsd.org/where.html i386 [Distribution] [ISO] but I can't download because I don't know User and Password please help me Thank you. KaeW r. 34 - Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions- [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Problem fetching iso vi ftp (was Re: Please Help me...)
In response to I am ws:ion [EMAIL PROTECTED]: I want to download Free BSD This Link http://www.freebsd.org/where.html i386 [Distribution] [ISO] but I can't download because I don't know User and Password please help me The user is anonymous or ftp and the password can be anything. This is typical of open FTP services. And please use a descriptive subject in your emails so people know what they're about. -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Please Help me...
I want to download Free BSD This Link http://www.freebsd.org/where.html i386 [Distribution] [ISO] but I can't download because I don't know User and Password please help me Thank you. KaeW r. 34 - Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
please Help ME!
Hi! I'm meisam . i want Book or PDF about kernel freebsd Data Structures, Algorithms please fast Reply tanks for you'r HELP. bye Send instant messages to your online friends http://uk.messenger.yahoo.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: please Help ME!
On 08/04/07, Noname Noname [EMAIL PROTECTED] wrote: Hi! I'm meisam . i want Book or PDF about kernel freebsd Data Structures, Algorithms please fast Reply tanks for you'r HELP. bye Why don't you start with the Documentation on the FreeBSD Website, e.g: http://www.freebsd.org/docs/books.html And you might find Comments in the kernel source, too. Just make sure that /usr/src on your system is populated and go to /usr/src/sys, which is the location of the kernel source. HTH Christian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: please Help ME!
In response to Noname Noname [EMAIL PROTECTED]: Hi! I'm meisam . i want Book or PDF about kernel freebsd Data Structures, Algorithms I recommend _The_Design_and_Implementation_of_FreeBSD_ by Dr. McKusick and others. please fast Reply I hope that was fast enough for you. -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: please Help ME!
On Sun, 8 Apr 2007, Noname Noname wrote: Hi! I'm meisam . i want Book or PDF about kernel freebsd Data Structures, Algorithms please fast Reply tanks for you'r HELP. bye It's called go to Amazon.com and search for FreeBSD operating system design. And please use a better subject line the next time you have a question. -Garrett PS A gimme would be http://www.amazon.com/Design-Implementation-FreeBSD-Operating-System/dp/0201702452;. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Please help me !
Dear sir I'm Mohammad Ajorlou I have MCP,MCSA,MCSE 2000 2003 certification , but very like learn FreeBSD Operating System i know FreeBSD very very stable , reliable and secure but unfortunately i don't know where must learn it Please help me that which resource and document must use for learning FreeBSD Thank you _ [1]Find just what you're after with the new, more precise MSN Search - try it now! References 1. http://g.msn.com/8HMBENUS/2731??PS=47575 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me !
On 10/19/05, mohammad ajorlou [EMAIL PROTECTED] wrote: Dear sir I'm Mohammad Ajorlou I have MCP,MCSA,MCSE 2000 2003 certification , but very like learn FreeBSD Operating System i know FreeBSD very very stable , reliable and secure but unfortunately i don't know where must learn it Please help me that which resource and document must use for learning FreeBSD Thank you _ [1]Find just what you're after with the new, more precise MSN Search - try it now! References 1. http://g.msn.com/8HMBENUS/2731??PS=47575 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Start with reading this: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me !
On Wed, 19 Oct 2005 06:21:30 + mohammad ajorlou [EMAIL PROTECTED] wrote: Dear sir I'm Mohammad Ajorlou I have MCP,MCSA,MCSE 2000 2003 certification , but very like learn FreeBSD Operating System i know FreeBSD very very stable , reliable and secure but unfortunately i don't know where must learn it Please help me that which resource and document must use for learning FreeBSD Thank you You can find links to several online documents, including the FreeBSD Handbook, at the url below: http://www.freebsd.org/docs.html You can also find books regarding FreeBSD at online bookstores such as Amazon.com. Best of luck, Andrew Gould ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Please help me !
Hi, Please try: 1. The FreeBSD handbook (available on www.freebsd.org or from FreeBSD Mall www.freebsdmall.com). It has an excellent chapter on the installation process 2. Then you can check Afnog's website at ws.afnog.org They have interesting and clear, step-by-step tutorials on building web servers, mail servers, clustering servers, etc. 3. Last but not least, try Greg Lehey's book The complete FreeBSD. There also Absolute FreeBSD. These are good guides to several topics, including system administration. Hope this helps, Simon mohammad ajorlou [EMAIL PROTECTED] a écrit : Dear sir I'm Mohammad Ajorlou I have MCP,MCSA,MCSE 2000 2003 certification , but very like learn FreeBSD Operating System i know FreeBSD very very stable , reliable and secure but unfortunately i don't know where must learn it Please help me that which resource and document must use for learning FreeBSD Thank you _ [1]Find just what you're after with the new, more precise MSN Search - try it now! References 1. http://g.msn.com/8HMBENUS/2731??PS=47575 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] - Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger Téléchargez le ici ! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me booooot!
I'd just boot off of disc 1 of the install set. That way you have a known (Bgood boot environment. Then you can back things up as per normal. You (Bcan also get a little bit of a sense of what is where. (B (B (BOn Mon, 23 May 2005, Joel wrote: (B (B [...] (B when i boot either ad6 or ad4 alone, everything is cool. (B (B when i plug both ad4 and ad6 into, (B (B Usually, when you set up for multi-boot, you plug all your drives in (B before you set up. (B (B At least, you do this until you understand what't going on. (B (B Now, if you understood what happens when the machine boots, you might be (B able to edit your fstab to get FreeBSD to boot. Or maybe not. (B (B i am brought to the freebsd (B bootloader. however, half-way down the boot i get: (B (B Mounting root from ufs:/dev/ad6s1a (B setrootbyname failed (B ffs_mountroot: can't find rootvp (B Root mount failed: 6 (B (B when i switch ad6's cable over to ad4 and vice versa, my computer (B doesn't boot past its bios at all. what's wrong here? (B (B Drive name changed at the low level, maybe? (B (B It happens. (B (B i just need to (B mount ad4 so i can back up some stuff, that's all. thank you so much! (B (B (B -- (B Joel Rees [EMAIL PROTECTED] (B digitcom, inc. ³ô¼°²ñ¼Ò¥Ç¥¸¥³¥à (B Kobe, Japan +81-78-672-8800 (B ** http://www.ddcom.co.jp ** (B (B ___ (B freebsd-questions@freebsd.org mailing list (B http://lists.freebsd.org/mailman/listinfo/freebsd-questions (B To unsubscribe, send any mail to "[EMAIL PROTECTED]" (B ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Please help me booooot!
hey! i have two s-ata harddrives on this computer: ad6, with FreeBSD 5.4-STABLE and ad4, with Windows XP when i boot either ad6 or ad4 alone, everything is cool. when i plug both ad4 and ad6 into, i am brought to the freebsd bootloader. however, half-way down the boot i get: Mounting root from ufs:/dev/ad6s1a setrootbyname failed ffs_mountroot: can't find rootvp Root mount failed: 6 when i switch ad6's cable over to ad4 and vice versa, my computer doesn't boot past its bios at all. what's wrong here? i just need to mount ad4 so i can back up some stuff, that's all. thank you so much! -- Fafa Hafiz Krantz Research Designer @ http://www.home.no/barbershop Enlightened @ http://www.home.no/barbershop/smart/sharon.pdf -- ___ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me booooot!
Fafa Hafiz Krantz wrote: hey! Dot, dot, dot when i switch ad6's cable over to ad4 and vice versa, my computer doesn't boot past its bios at all. what's wrong here? i just need to mount ad4 so i can back up some stuff, that's all. thank you so much! First and formost - pick up a good A+ book. Learn why drives boot in a certain order, learn about boot sectors. Learn why drives on seperate controllers work the way they do. That's the problem with youngsters these days - most have no idea about the basics of PC's. Why they work the way they do. I won't even speak of the old DOS days and using fdisk... Tsk, tsk, tsk ... Next - read the handbook on dual booting, then - search the list archives. It's been covered more time then you might expect. However, it does take a little work on your part to find the answer. -- Best regards, Chris How long a minute is depends on which side of the bathroom door you're on. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Please help me booooot!
[...] (B when i boot either ad6 or ad4 alone, everything is cool. (B (B when i plug both ad4 and ad6 into, (B (BUsually, when you set up for multi-boot, you plug all your drives in (Bbefore you set up. (B (BAt least, you do this until you understand what't going on. (B (BNow, if you understood what happens when the machine boots, you might be (Bable to edit your fstab to get FreeBSD to boot. Or maybe not. (B (B i am brought to the freebsd (B bootloader. however, half-way down the boot i get: (B (B Mounting root from ufs:/dev/ad6s1a (B setrootbyname failed (B ffs_mountroot: can't find rootvp (B Root mount failed: 6 (B (B when i switch ad6's cable over to ad4 and vice versa, my computer (B doesn't boot past its bios at all. what's wrong here? (B (BDrive name changed at the low level, maybe? (B (BIt happens. (B (B i just need to (B mount ad4 so i can back up some stuff, that's all. thank you so much! (B (B (B-- (BJoel Rees [EMAIL PROTECTED] (Bdigitcom, inc. $B3t<02q
Please help me with PF (thanks)
Hello! Problems: 1. BIND stops workin after a while 2. I cannot establish SSH connections 3. I cannot FTP to certain sites 4. PF crashes my computer on FTP uploads So I'm seeking solutions to these problems. And maybe ways to SIMPLIFY my pf.conf: - int_if=ep0 ext_if=lnc0 # *** Options # set block-policy drop # *** Scrub incoming packets # scrub in all # *** NAT # nat on $ext_if from $int_if:network to any - ($ext_if) rdr on $int_if proto tcp from any to any \ port 21 - 127.0.0.1 port 8021 # *** Default deny policy # block drop log all # *** Pass loopback traffic # passquick on { lo0 $int_if } # *** Outgoing # passout on $ext_if inet proto { tcp, udp, icmp } \ from ($ext_if) to any keep state # *** Bootstrap # passout on $ext_if inet proto udp \ from any port 68 to any port 67 keep state # *** DNS and NTP # passout on $ext_if inet proto udp \ from ($ext_if) to any port { 53, 123 } keep state # *** SSH, HTTP and Ident # passin on $ext_if inet proto tcp \ from any to ($ext_if) port { 22, 80, 113 } flags S/SA keep state # *** Active FTP # passin on $ext_if inet proto tcp \ from port 20 to ($ext_if) user proxy flags S/SA keep state - Thank you all so much! -- Fafa -- ___ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Please help me with PF (thanks)
Simple question, do problems 1, 2, 3 still happen when you comment out the pf statements in rc.conf and run with out a firewall? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Fafa Diliha Romanova Sent: Saturday, April 30, 2005 7:51 AM To: [EMAIL PROTECTED] Subject: Please help me with PF (thanks) Hello! Problems: 1. BIND stops workin after a while 2. I cannot establish SSH connections 3. I cannot FTP to certain sites 4. PF crashes my computer on FTP uploads So I'm seeking solutions to these problems. And maybe ways to SIMPLIFY my pf.conf: - int_if=ep0 ext_if=lnc0 # *** Options # set block-policy drop # *** Scrub incoming packets # scrub in all # *** NAT # nat on $ext_if from $int_if:network to any - ($ext_if) rdr on $int_if proto tcp from any to any \ port 21 - 127.0.0.1 port 8021 # *** Default deny policy # block drop log all # *** Pass loopback traffic # passquick on { lo0 $int_if } # *** Outgoing # passout on $ext_if inet proto { tcp, udp, icmp } \ from ($ext_if) to any keep state # *** Bootstrap # passout on $ext_if inet proto udp \ from any port 68 to any port 67 keep state # *** DNS and NTP # passout on $ext_if inet proto udp \ from ($ext_if) to any port { 53, 123 } keep state # *** SSH, HTTP and Ident # passin on $ext_if inet proto tcp \ from any to ($ext_if) port { 22, 80, 113 } flags S/SA keep state # *** Active FTP # passin on $ext_if inet proto tcp \ from port 20 to ($ext_if) user proxy flags S/SA keep state - Thank you all so much! -- Fafa -- ___ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]