Re: FreeBSD 5.1-R kernel panic
Hi Bosko, Well a couple weeks ago I re-cvsup'd to current since I had missed one of your updates it would seem by a day. I still had the panic occur... so I did as suggested and upped my KVA. I did the following: In the kernel options KVA_PAGES=400 options NMBCLUSTERS=8192 In the loader.conf kern.vm.kmem.size=4 This seems to have helped, since I am no longer panic'ng. I have identified a php script that slowly builds to 100% cpu usage in a couple days which we believe was a culprit in our problems. The guys who made the php script that runs on this box is having a close look at it to figure out why it's behaving this way... however I did have some questions. 1. Should the panic occur in the first place? 2. Is there a way to monitor how much kva is currently being used so that I can monitor this value so I can have something alert me if I'm running out of this memory... I'm sure we would of caught this months ago if I was monitoring for this? Thanks again for everything you have done to assist me thru this problem. Stephane Raimbault. [EMAIL PROTECTED] (Bosko Milekic) wrote in message news:[EMAIL PROTECTED]... On Thu, Aug 14, 2003 at 10:57:07AM -0600, Stephane Raimbault wrote: Hi Bosko, This is the output of sysctl vm.zone about 2 minutes before the crash occured. let me know if there is anything else I can provide you for this crashing problem. H. I don't know, maybe you really do have a machine too loaded for the KVA you have configured... I have to re-iterate that it's extremely important that you double-check that you are in fact in sync with the latest -current and _NOT_ RELENG_5_1. Make sure you're building at least version 1.73 of src/sys/vm/uma_core.c (grep FBSDID src/sys/vm/uma_core.c). With that said, you can try the following: options KVA_PAGES=400 in your kernel configuration file. Following that, you can do this: kern.vm.kmem.size=4 In your /boot/loader.conf Make sure to not set NMBCLUSTERS too high. Around 8K is probably more than enough, but you should look at how much you're using on average with `netstat -m' and then set the number to roughly 3 times that. If following this your crash persists, even if after a longer time, then I would suspect (another?) race. Again, I have to re-iterate that you really need to make sure you're supping to HEAD: *default release=cvs tag=. Regards, ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Thu, Aug 14, 2003 at 10:57:07AM -0600, Stephane Raimbault wrote: Hi Bosko, This is the output of sysctl vm.zone about 2 minutes before the crash occured. let me know if there is anything else I can provide you for this crashing problem. H. I don't know, maybe you really do have a machine too loaded for the KVA you have configured... I have to re-iterate that it's extremely important that you double-check that you are in fact in sync with the latest -current and _NOT_ RELENG_5_1. Make sure you're building at least version 1.73 of src/sys/vm/uma_core.c (grep FBSDID src/sys/vm/uma_core.c). With that said, you can try the following: options KVA_PAGES=400 in your kernel configuration file. Following that, you can do this: kern.vm.kmem.size=4 In your /boot/loader.conf Make sure to not set NMBCLUSTERS too high. Around 8K is probably more than enough, but you should look at how much you're using on average with `netstat -m' and then set the number to roughly 3 times that. If following this your crash persists, even if after a longer time, then I would suspect (another?) race. Again, I have to re-iterate that you really need to make sure you're supping to HEAD: *default release=cvs tag=. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
I updated to 5.1-CURRENT this weekend and my kernel panic'd this morning. Here is what I have from the trace panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 0; lapic.id = Debugger(panic) Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0 db trace Debugger(c0538a46,0,c054e317,e91c4b6c,100) at Debugger+0x55 panic(c054e317,1000,1068,e91c4b98,0) at panic+0x15f kmem_malloc(c082f0b0,1000,2,e91c4bf0,c0490f22) at kmem_malloc+0x100 page_alloc(c083ad80,1000,e91c4be3,2,c053b88d) at page_alloc+0x27 slab_zalloc(c083ad80,102,c058f30c,60b,c0630c80) at slab_zalloc+0xc2 uma_zone_slab(c083ad80,102,167,c058f30c,c083ae44) at uma_zone_slab+0xe8 uma_zalloc_bucket(c083ad80,102,c054fb50,599,167) at uma_zalloc_bucket+0x185 uma_zalloc_arg(c083ad80,0,102,fe,d70cb400) at uma_zalloc_arg+0x369 malloc(80,c058f960,102,0,e91c4ce0) at malloc+0xd3 crget(bfbffa70,d70cb414,8,cb0017fc,1) at crget+0x25 setgroups(cafff130,e91c4d10,c0554758,3ee,2) at setgroups+0x58 syscall(2f,2f,bfbf002f,0,bfbffa70) at syscall+0x273 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (80, FreeBSD ELF32, setgroups), eip = 0x282d208f, esp = 0xbfbffa4c, ebp = 0xbfbffab8 --- Let me know if there is other data you need from me. Thanks again, Stephane. - Original Message - From: Stephane Raimbault [EMAIL PROTECTED] To: Bosko Milekic [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Wednesday, July 30, 2003 11:02 AM Subject: Re: FreeBSD 5.1-R kernel panic Hi Bosko, Thank you for your suggestion. I am a little un-easy about upgrading my system from -RELEASE to -CURRENT. Did you mean, simply upgrading the kernel to -CURRENT, or the entire system? I'll think of a way I can continue this testing with you so that we can make sure this problem is resolved. It might take me a couple days for me to safeguard my data and come up with a backup plan for my system should upgrading to -current blows up my system :) If it's just a question of upgrading the kernel, then I can probably implement that sooner rather then later, so let me know if it's just the kernel, or the entire /usr/src system you would like me to upgrade. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 30, 2003 9:35 AM Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 30, 2003 at 09:25:34AM -0600, Stephane Raimbault wrote: Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. Hmmm. For what it's worth, I've seen both of these panics before. Have you tried to cvsup to the latest -current (with all of those UMA fixes?) Also, try grabbing http://people.freebsd.org/~bmilekic/uma_races.patch and applying that to a fresh -current. I'm not sure that it's related, but at this point it's worth a shot. It's worth noting that a small chunk of the above patch may fail as I just committed a part of it. You should be able to safely ignore that. Let me know how it works out. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
--- syscall (80, FreeBSD ELF32, setgroups), eip = 0x282d208f, esp = 0xbfbffa4c, ebp = 0xbfbffab8 --- Let me know if there is other data you need from me. Thanks again, Stephane. - Original Message - From: Stephane Raimbault [EMAIL PROTECTED] To: Bosko Milekic [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Wednesday, July 30, 2003 11:02 AM Subject: Re: FreeBSD 5.1-R kernel panic Hi Bosko, Thank you for your suggestion. I am a little un-easy about upgrading my system from -RELEASE to -CURRENT. Did you mean, simply upgrading the kernel to -CURRENT, or the entire system? I'll think of a way I can continue this testing with you so that we can make sure this problem is resolved. It might take me a couple days for me to safeguard my data and come up with a backup plan for my system should upgrading to -current blows up my system :) If it's just a question of upgrading the kernel, then I can probably implement that sooner rather then later, so let me know if it's just the kernel, or the entire /usr/src system you would like me to upgrade. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 30, 2003 9:35 AM Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 30, 2003 at 09:25:34AM -0600, Stephane Raimbault wrote: Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. Hmmm. For what it's worth, I've seen both of these panics before. Have you tried to cvsup to the latest -current (with all of those UMA fixes?) Also, try grabbing http://people.freebsd.org/~bmilekic/uma_races.patch and applying that to a fresh -current. I'm not sure that it's related, but at this point it's worth a shot. It's worth noting that a small chunk of the above patch may fail as I just committed a part of it. You should be able to safely ignore that. Let me know how it works out. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Bosko, I've setup a cronjob every 5 minutes to e-mail me the requested data sysctl vm.zone. The crashing is re-occuring, it seems to happen every day and a half or so. We did not fix the problem, but we did leave it at about a week and half ago when you suggested to update to -CURRENT from -RELEASE. This past weekend, I finally got to updating to -CURRENT (I deleted my /usr/src then cvsup'd src-all from HEAD (.). Also, System Load is way higher then it was in -RELEASE around 10.00 - 20.00. I'm assuming this is due to all the debug options enabled in -CURRENT. Anything I can disable to lower the average load but continue to troubleshoot this problem? Some of the debug options which may not be helpful at this point perhaps? If I reboot the server on a nightly bassis, I can get away from the crashing, but rebooting every night is quickly becoming old and would rather have this fixed up. Again the problem seems to occur every 1.5 - 2 days. Let me know if there is anything else I can provide to help find a solution to this problem. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Monday, August 11, 2003 6:15 PM Subject: Re: FreeBSD 5.1-R kernel panic Please run 'sysctl vm.zone' periodically and capture the output. Then wait for this to happen and feel free to send me the last captured output before the crash. Is this a re-occuring crash for you? I thought we fixed this already. Are you sure you have the absolute latest -current? (or are you supping RELENG_5_1, which still has this problem?) -Bosko On Mon, Aug 11, 2003 at 03:45:14PM -0600, Stephane Raimbault wrote: I updated to 5.1-CURRENT this weekend and my kernel panic'd this morning. Here is what I have from the trace panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 0; lapic.id = Debugger(panic) Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0 db trace Debugger(c0538a46,0,c054e317,e91c4b6c,100) at Debugger+0x55 panic(c054e317,1000,1068,e91c4b98,0) at panic+0x15f kmem_malloc(c082f0b0,1000,2,e91c4bf0,c0490f22) at kmem_malloc+0x100 page_alloc(c083ad80,1000,e91c4be3,2,c053b88d) at page_alloc+0x27 slab_zalloc(c083ad80,102,c058f30c,60b,c0630c80) at slab_zalloc+0xc2 uma_zone_slab(c083ad80,102,167,c058f30c,c083ae44) at uma_zone_slab+0xe8 uma_zalloc_bucket(c083ad80,102,c054fb50,599,167) at uma_zalloc_bucket+0x185 uma_zalloc_arg(c083ad80,0,102,fe,d70cb400) at uma_zalloc_arg+0x369 malloc(80,c058f960,102,0,e91c4ce0) at malloc+0xd3 crget(bfbffa70,d70cb414,8,cb0017fc,1) at crget+0x25 setgroups(cafff130,e91c4d10,c0554758,3ee,2) at setgroups+0x58 syscall(2f,2f,bfbf002f,0,bfbffa70) at syscall+0x273 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (80, FreeBSD ELF32, setgroups), eip = 0x282d208f, esp = 0xbfbffa4c, ebp = 0xbfbffab8 --- Let me know if there is other data you need from me. Thanks again, Stephane. - Original Message - From: Stephane Raimbault [EMAIL PROTECTED] To: Bosko Milekic [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Wednesday, July 30, 2003 11:02 AM Subject: Re: FreeBSD 5.1-R kernel panic Hi Bosko, Thank you for your suggestion. I am a little un-easy about upgrading my system from -RELEASE to -CURRENT. Did you mean, simply upgrading the kernel to -CURRENT, or the entire system? I'll think of a way I can continue this testing with you so that we can make sure this problem is resolved. It might take me a couple days for me to safeguard my data and come up with a backup plan for my system should upgrading to -current blows up my system :) If it's just a question of upgrading the kernel, then I can probably implement that sooner rather then later, so let me know if it's just the kernel, or the entire /usr/src system you would like me to upgrade. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 30, 2003 9:35 AM Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 30, 2003 at 09:25:34AM -0600, Stephane Raimbault wrote: Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. Hmmm. For what it's worth, I've seen both of these panics before. Have you tried to cvsup to the latest -current (with all of those UMA fixes?) Also, try grabbing http://people.freebsd.org/~bmilekic/uma_races.patch and applying that to a fresh
Re: FreeBSD 5.1-R kernel panic
Please run 'sysctl vm.zone' periodically and capture the output. Then wait for this to happen and feel free to send me the last captured output before the crash. Is this a re-occuring crash for you? I thought we fixed this already. Are you sure you have the absolute latest -current? (or are you supping RELENG_5_1, which still has this problem?) -Bosko On Mon, Aug 11, 2003 at 03:45:14PM -0600, Stephane Raimbault wrote: I updated to 5.1-CURRENT this weekend and my kernel panic'd this morning. Here is what I have from the trace panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 0; lapic.id = Debugger(panic) Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0 db trace Debugger(c0538a46,0,c054e317,e91c4b6c,100) at Debugger+0x55 panic(c054e317,1000,1068,e91c4b98,0) at panic+0x15f kmem_malloc(c082f0b0,1000,2,e91c4bf0,c0490f22) at kmem_malloc+0x100 page_alloc(c083ad80,1000,e91c4be3,2,c053b88d) at page_alloc+0x27 slab_zalloc(c083ad80,102,c058f30c,60b,c0630c80) at slab_zalloc+0xc2 uma_zone_slab(c083ad80,102,167,c058f30c,c083ae44) at uma_zone_slab+0xe8 uma_zalloc_bucket(c083ad80,102,c054fb50,599,167) at uma_zalloc_bucket+0x185 uma_zalloc_arg(c083ad80,0,102,fe,d70cb400) at uma_zalloc_arg+0x369 malloc(80,c058f960,102,0,e91c4ce0) at malloc+0xd3 crget(bfbffa70,d70cb414,8,cb0017fc,1) at crget+0x25 setgroups(cafff130,e91c4d10,c0554758,3ee,2) at setgroups+0x58 syscall(2f,2f,bfbf002f,0,bfbffa70) at syscall+0x273 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (80, FreeBSD ELF32, setgroups), eip = 0x282d208f, esp = 0xbfbffa4c, ebp = 0xbfbffab8 --- Let me know if there is other data you need from me. Thanks again, Stephane. - Original Message - From: Stephane Raimbault [EMAIL PROTECTED] To: Bosko Milekic [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Wednesday, July 30, 2003 11:02 AM Subject: Re: FreeBSD 5.1-R kernel panic Hi Bosko, Thank you for your suggestion. I am a little un-easy about upgrading my system from -RELEASE to -CURRENT. Did you mean, simply upgrading the kernel to -CURRENT, or the entire system? I'll think of a way I can continue this testing with you so that we can make sure this problem is resolved. It might take me a couple days for me to safeguard my data and come up with a backup plan for my system should upgrading to -current blows up my system :) If it's just a question of upgrading the kernel, then I can probably implement that sooner rather then later, so let me know if it's just the kernel, or the entire /usr/src system you would like me to upgrade. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 30, 2003 9:35 AM Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 30, 2003 at 09:25:34AM -0600, Stephane Raimbault wrote: Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. Hmmm. For what it's worth, I've seen both of these panics before. Have you tried to cvsup to the latest -current (with all of those UMA fixes?) Also, try grabbing http://people.freebsd.org/~bmilekic/uma_races.patch and applying that to a fresh -current. I'm not sure that it's related, but at this point it's worth a shot. It's worth noting that a small chunk of the above patch may fail as I just committed a part of it. You should be able to safely ignore that. Let me know how it works out. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Sunday, July 27, 2003 8:56 Subject: Re: FreeBSD 5.1-R kernel panic On Sat, Jul 26, 2003 at 10:48:21PM -0600, Stephane Raimbault wrote: Well, I had compiled options DDB into the kernel and today the kernel panic'd... here is what I got. I ran the following in the db prompt. trace, show reg, ps. Let me know if this is the kind of information you need, and if there is anything else I need to provide... can I re-compile the kernel without the options DDB now, or should I provide the same info next time in panic's to confirm it's the same problem? Thanks, Stephane. I've attached the file debug.txt which contains the panic info. Let me know if you need it in a different format. Thanks again, Stephane Raimbault. Are you using USB? Approximately 2 weeks ago, some changes were introduced into the USB code which could in some scenarios mean a depletion of kmem_map. I've glanced at usb_mem.c and it appears that the USB code 'caches' everything that it allocates into a couple of lists it maintains locally. This prevents UMA from seeing the freed memory and, from the point of view of the VM, the memory is never reclaimed. I am unsure as to whether or not the memory allocated by the USB stuff needs to be type stable so I can't really fix this. I urge the USB authors to glance there. If you are using USB, try turning it off to see if the panic persists. If not, then we'll have to look elsewhere. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.502 / Virus Database: 300 - Release Date: 7/18/2003 panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 2; lapic.id = 0100 Debugger(panic) Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0 db trace Debugger(c04d1139,100,c04e2d72,e97f18a0,1) at Debugger+0x55 panic(c04e2d72,1000,1068,e97f18cc,6b6) at panic+0x11f kmem_malloc(c082f0b0,1000,2,e97f1920,c0437b86) at kmem_malloc+0xf7 page_alloc(c082ab40,1000,e97f1913,2,d2042be0) at page_alloc+0x27 slab_zalloc(c082ab40,2,caf2a124,0,e97f1954) at slab_zalloc+0x106 uma_zone_slab(c082ab40,2,c0412888,caf2a124,c082aca0) at uma_zone_slab+0xd8 uma_zalloc_bucket(c082ab40,2,c04e3fcd,586,98000) at uma_zalloc_bucket+0x15d uma_zalloc_arg(c082ab40,0,2,cb87cab0,c082ab40) at uma_zalloc_arg+0x230 malloc(48,c0524fe0,2,e97f1a58,0) at malloc+0x58 cache_enter(cbce47fc,d3e665b4,e97f1c4c,e97f1a48,e97f1a4c) at cache_enter+0x93 ufs_lookup(e97f1b0c,e97f1b48,c0330ba0,e97f1b0c,e97f1c38) at ufs_lookup+0xdf8 ufs_vnoperate(e97f1b0c,e97f1c38,e97f1c4c,1030002,d2042be0) at ufs_vnoperate+0x18 vfs_cache_lookup(e97f1b8c,e97f1ba8,c03356a2,e97f1b8c,20002) at vfs_cache_lookup+0x2f0 ufs_vnoperate(e97f1b8c,20002,d2042be0,d2042be0,d2042be0) at ufs_vnoperate+0x18 lookup(e97f1c24,0,c04d5482,a4,d2042be0) at lookup+0x302 namei(e97f1c24,8344e48,60,0,d2042be0) at namei+0x1ee lstat(d2042be0,e97f1d10,c04e84ac,3fb,2) at lstat+0x52 syscall(82d002f,82d002f,bfbf002f,8344f00,8344f48) at syscall+0x24e Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (190, FreeBSD ELF32, lstat), eip = 0x280bba23, esp = 0xbfbffaec, ebp = 0xbfbffb88 --- db show reg cs 0x8 ds0x10 es 0xd2040010 fs0x18 ss0x10 eax 0x12 ecx 0x3fd edx 0 ebx 0 esp 0xe97f1858 ebp 0xe97f1864 esi 0x100 edi 0xd2042be0 eip 0xc045d4c5 Debugger+0x55 efl 0x286 dr0 0 dr1 0 dr2 0 dr3 0 dr4 0x0ff0 dr5 0x400 dr6 0x0ff0 dr7 0x400 Debugger+0x55: xchgl %ebx,in_Debugger.0 db ps pid proc addruid ppid pgrp flag stat wmesgwchan cmd 56961 cbef53c0 e9371000 80 13415 13415 100 norm[SLPQ lockf d4294740][SLP] httpd 56960 d277d000 e98ed0000 56952 56808 0004000 norm[CPU 2] find 56958 d27353c0 e94e20000 56956 56808 0004000 norm[SLPQ piperd caee7e70][SLP] cat 56956 cb5f2b40 e927e0000 56946 56808 000 norm[SLPQwait
Re: FreeBSD 5.1-R kernel panic
On Wed, Jul 30, 2003 at 09:25:34AM -0600, Stephane Raimbault wrote: Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. Hmmm. For what it's worth, I've seen both of these panics before. Have you tried to cvsup to the latest -current (with all of those UMA fixes?) Also, try grabbing http://people.freebsd.org/~bmilekic/uma_races.patch and applying that to a fresh -current. I'm not sure that it's related, but at this point it's worth a shot. It's worth noting that a small chunk of the above patch may fail as I just committed a part of it. You should be able to safely ignore that. Let me know how it works out. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Bosko, Thank you for your suggestion. I am a little un-easy about upgrading my system from -RELEASE to -CURRENT. Did you mean, simply upgrading the kernel to -CURRENT, or the entire system? I'll think of a way I can continue this testing with you so that we can make sure this problem is resolved. It might take me a couple days for me to safeguard my data and come up with a backup plan for my system should upgrading to -current blows up my system :) If it's just a question of upgrading the kernel, then I can probably implement that sooner rather then later, so let me know if it's just the kernel, or the entire /usr/src system you would like me to upgrade. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 30, 2003 9:35 AM Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 30, 2003 at 09:25:34AM -0600, Stephane Raimbault wrote: Hi Bosko, My kernel panic'd again this morning. I had removed all the USB devices from my kernel and had set my /etc/rc.conf to usbd_enable=NO and the kernel panic'd again. I have attached both my kernel config file and the trace of the panic. Let me know if I can provide any further information to help analyze this problem. Thanks, Stephane Raimbault. Hmmm. For what it's worth, I've seen both of these panics before. Have you tried to cvsup to the latest -current (with all of those UMA fixes?) Also, try grabbing http://people.freebsd.org/~bmilekic/uma_races.patch and applying that to a fresh -current. I'm not sure that it's related, but at this point it's worth a shot. It's worth noting that a small chunk of the above patch may fail as I just committed a part of it. You should be able to safely ignore that. Let me know how it works out. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Wed, Jul 30, 2003 at 11:02:05AM -0600, Stephane Raimbault wrote: Hi Bosko, Thank you for your suggestion. I am a little un-easy about upgrading my system from -RELEASE to -CURRENT. Did you mean, simply upgrading the kernel to -CURRENT, or the entire system? Due to potential sync issues between userland and kernel, if you're going to do this, it would probably be a good idea to rebuild everything. This is especially true if you're using something like the linuxulator, or whatever we call linux emulation these days. :-) I'll think of a way I can continue this testing with you so that we can make sure this problem is resolved. It might take me a couple days for me to safeguard my data and come up with a backup plan for my system should upgrading to -current blows up my system :) If it's just a question of upgrading the kernel, then I can probably implement that sooner rather then later, so let me know if it's just the kernel, or the entire /usr/src system you would like me to upgrade. Unfortunately, unless you're ready to do some rather complicated manual merging, I would recommend making sure that you have backups, then cvsupping to latest -current, doing cd /usr/src make buildworld make buildkernel make installkernel make installworld mergemaster. Tread carefully, as always. Alternately, you can wait to see if the recent changes fix the same panic for other people who have been seeing it. Bonne chance. :-) Thanks, Stephane. -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Sat, Jul 26, 2003 at 10:48:21PM -0600, Stephane Raimbault wrote: Well, I had compiled options DDB into the kernel and today the kernel panic'd... here is what I got. I ran the following in the db prompt. trace, show reg, ps. Let me know if this is the kind of information you need, and if there is anything else I need to provide... can I re-compile the kernel without the options DDB now, or should I provide the same info next time in panic's to confirm it's the same problem? Thanks, Stephane. I've attached the file debug.txt which contains the panic info. Let me know if you need it in a different format. Thanks again, Stephane Raimbault. Are you using USB? Approximately 2 weeks ago, some changes were introduced into the USB code which could in some scenarios mean a depletion of kmem_map. I've glanced at usb_mem.c and it appears that the USB code 'caches' everything that it allocates into a couple of lists it maintains locally. This prevents UMA from seeing the freed memory and, from the point of view of the VM, the memory is never reclaimed. I am unsure as to whether or not the memory allocated by the USB stuff needs to be type stable so I can't really fix this. I urge the USB authors to glance there. If you are using USB, try turning it off to see if the panic persists. If not, then we'll have to look elsewhere. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Bosko, Thanks for your response. I do not use USB on the system... I'll try removing those devices from the kernel and see if the problem continues. I will let you know. Thanks, Stephane Raimbault. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Sunday, July 27, 2003 8:56 Subject: Re: FreeBSD 5.1-R kernel panic On Sat, Jul 26, 2003 at 10:48:21PM -0600, Stephane Raimbault wrote: Well, I had compiled options DDB into the kernel and today the kernel panic'd... here is what I got. I ran the following in the db prompt. trace, show reg, ps. Let me know if this is the kind of information you need, and if there is anything else I need to provide... can I re-compile the kernel without the options DDB now, or should I provide the same info next time in panic's to confirm it's the same problem? Thanks, Stephane. I've attached the file debug.txt which contains the panic info. Let me know if you need it in a different format. Thanks again, Stephane Raimbault. Are you using USB? Approximately 2 weeks ago, some changes were introduced into the USB code which could in some scenarios mean a depletion of kmem_map. I've glanced at usb_mem.c and it appears that the USB code 'caches' everything that it allocates into a couple of lists it maintains locally. This prevents UMA from seeing the freed memory and, from the point of view of the VM, the memory is never reclaimed. I am unsure as to whether or not the memory allocated by the USB stuff needs to be type stable so I can't really fix this. I urge the USB authors to glance there. If you are using USB, try turning it off to see if the panic persists. If not, then we'll have to look elsewhere. Regards, -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Well, I had compiled options DDB into the kernel and today the kernel panic'd... here is what I got. I ran the following in the db prompt. trace, show reg, ps. Let me know if this is the kind of information you need, and if there is anything else I need to provide... can I re-compile the kernel without the options DDB now, or should I provide the same info next time in panic's to confirm it's the same problem? Thanks, Stephane. I've attached the file debug.txt which contains the panic info. Let me know if you need it in a different format. Thanks again, Stephane Raimbault. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 23, 2003 10:14 Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 23, 2003 at 09:56:32AM -0600, Stephane Raimbault wrote: Hi Bosko, Looking at netstat -m, the value I'd probably be interested in is the following: 3% of cluster map consumed knowing that the Maximum possible is 25600 I can deduce that ~768 are being used? Is that correct. I'm not much of a programmer, but I did recognize the printf(); statements from a C class I didn't do well in half a decade ago... as you can tell, I'm not much of a programmer :). If it's not the 3% I should be paying attention too... then let me know :) Look at the in pool values for all the pcpu and GEN caches and add them up. Do this for clusters (since there are fewer). Compare to the Maximum Possible value. With a machine that goes into spike-load periods, you may want to have the Maximum Possible stay about 4-6 times what you have as your average in pool value (remember to sum the in pool values for the pcpu and GEN caches). The 3% is not what you think it is. It's the percentage of the allocated wired-down memory that is NOT in any of the caches but is allocated and circulating in the system. If you have a high number of cached items but the percentage is relatively low for most of the time, it's a sign that you were probably in a heavy-usage scenario some time ago, but that your current usage is relatively low. As for using the option DDB in my kernel, I do have one question. I do have remote console access that I use to go into single user mode on the box remotely. I'm suspecting I could use the debugger mode over the comconsole... I just want to make sure there is some kind of reboot command from the debugger so that I can tell the box to reboot once I've captured the stack trace? If so, I'll enable the DDB tonight and get you the info as soon as I can. Yes, you can do DDB over serial console. Take a look at the handbook for more information on using DDB. If you have the space in /var and enough swap, you may want to try getting a crashdump so that you can reboot and use GDB to debug. Again, take a look at the handbook. thanks again, Stephane. -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.502 / Virus Database: 300 - Release Date: 7/18/2003 panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 0; lapic.id = Debugger(panic) Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0 db trace Debugger(c04f954d,0,c050b17e,e959eb04,1) at Debugger+0x55 panic(c050b17e,1000,1068,e959eb30,e959eb3c) at panic+0x11f kmem_malloc(c082f0b0,1000,2,e959eb84,c0457486) at kmem_malloc+0xf7 page_alloc(c082ab40,1000,e959eb77,2,d47486d8) at page_alloc+0x27 slab_zalloc(c082ab40,102,d47486d8,e959ebcc,c036644d) at slab_zalloc+0x106 uma_zone_slab(c082ab40,102,0,d5069960,c082ac18) at uma_zone_slab+0xd8 uma_zalloc_bucket(c082ab40,102,c050c3d9,586,c02f0651) at uma_zalloc_bucket+0x15d uma_zalloc_arg(c082ab40,0,102,2,c082ab40) at uma_zalloc_arg+0x230 malloc(80,c054bf40,102,d7080680,e959ec50) at malloc+0x58 crget(c06f1140,e959ec50,d7080680,e959ecc8,c0361457) at crget+0x25 crdup(d7080680,8e1ad280,af77e,dd313180,d5069960) at crdup+0xe kern_access(d17b85f0,84536cc,0,0,e959ed40) at kern_access+0x17 access(d17b85f0,e959ed10,c05108b8,3fb,2) at access+0x29 syscall(bfbf002f,bfbf002f,bfbf002f,bfbfc6a4,284f15d4) at syscall+0x24e Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (33, FreeBSD ELF32, access), eip = 0x282cc463, esp = 0xbfbfc5fc, ebp = 0xbfbfc6c8 --- db show reg cs 0x8 ds0x10 es 0xd2e00010 fs 0xcada0018 ss0x10 eax 0x12 ecx0x4 edx 0 ebx 0 esp 0xe959eabc ebp 0xe959eac8 esi 0x100 edi
Re: FreeBSD 5.1-R kernel panic
Stephane Raimbault wrote: I recently realized that I was miss-understanding how much free memory I had on the system, and I doubt I even need the full 4Gig's. Perhaps I can re-confirm how to check how much free real memory is available on the system. For 4G of physical RAM, with 3G of KVA and 1G of UVA, you have enough memory to hold the kernel and the page mappings for all of KVA + the pages dedicated to the kernel, plust the defaults for buffers (if you *do not* autotune) for ~2G of mbufs and tuning for approximately 280,000 socket connections and open files, *IF* you disable IPSEC, AND you intend to run one or two processes in user space and use mode of the 1G of address space there. If you autotune, the numbers of connections you can run drops sharply. Also note that the default UVA/KVA plist is *not* 1G/3G, it's 2G/2G, and it used to be 3G/1G. For 4G server systems, the very first thing I always recommend, if you want the highest load capacity possible, is to reduce the UVA and increse the KVA so you have enough space in the KVA to map and store the resources you'll need for a high load. The most common thing to run out of, in everything but database servers, is KVA space (panic: kmem_map too small). For databases with large data sets, you have to artificially restrict the amount of total KVA that everything added together can consume to keep it under the 2G ceiling. This usually means a tradeoff between number of connections and database size. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Stephane Raimbault wrote: Well I went to go change my /boot/loader.conf options to reflect the following: kern.vm.kmem.size=35 Assuming this is in pages, it is 1/3 of the total physical RAM in the system. This is way too large, unless you have recompiled your kernel to have 3G KVA vs. 1G UVA, instead of the default. Kernel recompilation is required because the load address is a fixed virtual address based on the KVA size, subtracted from the top of memory. This is so that user programs do not have to be recompiled when you move the ratio of UVA to KVA around (with the exception of the Linux emulator, if you are using Linux threads, since it seems to want its threads mailboxes at a known location in memory). kern.ipc.nmbclusters=8192 This is large, as well. You need to turn off automatic tuning, and act like you are on a memory budget equal to the amount of KVA space in the system, which you can use for different things, but not at the same time. If you leave the system at the defaults, it will (mostly) be able to keep itself under the budget ceiling for an assumed 2G KVA space, but as soon as you start tuning some things up, you will very quickly blow your budget, since you tuning something you want to use up will not necessarily tune something you aren't going to use down, to avoid blowing your total budget. and enabled options DDB in my kernel. Unfortunately, I ran into a problem on the reboot, the SMP kernel would fail to load due to some of my /boot/loader.conf changes perhaps I miss-understood something in your instructions... anyhow, this is what was displayed on the console when I set those above settings in the /boot/loader.conf... and since I had the debugger running, I did a trace and included in the paste below... is that the kind of stuff you will want if the box crashes again as originally mentioned in this thread. This particular panic is an initialization of a region with a header structure, where the memory allocation has failed because you have already blown your budget, and so the allocation returns NULL. This happens early enough in the system startup that there is no error checking to ensure that the allocation has succeeded. But even if you added error checking in all these places, the best the system is going to be able to do is pnaic with a slightly more informative message. When it goes to fill in this structure, it tries to fill in a field containing something 8 bytes into the structure, and explodes. If you look at the function where the traceback says it crashed, this should be visibly obvious to you. A good rule of thumb for tuning is to start with the autotuned defaults (though they can screw you on occasion, since RAM is installed in discrete amounts, and the autotuning tends to use a continuous function of the amount of physical RAM, so you get a stair function, and not all possible steps in the stair have actually been tested with all possible resource allocations having been made, to see if a problem occurs). Once you know those, you hard-code them so that they are no longer autotuned. Then you tweak the resource allocations for resources you don't care about using down. Then you tweak what you do care about using up, until you crash. Then you back off on your last tweak to give you some safety margin. This will get you within 85%-90% of where you can get without modifying code. The only other alternative is to know where every byte of memory is going. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Bosko, Well I went to go change my /boot/loader.conf options to reflect the following: kern.vm.kmem.size=35 kern.ipc.nmbclusters=8192 and enabled options DDB in my kernel. Unfortunately, I ran into a problem on the reboot, the SMP kernel would fail to load due to some of my /boot/loader.conf changes perhaps I miss-understood something in your instructions... anyhow, this is what was displayed on the console when I set those above settings in the /boot/loader.conf... and since I had the debugger running, I did a trace and included in the paste below... is that the kind of stuff you will want if the box crashes again as originally mentioned in this thread. --- /boot/kernel/acpi.ko text=0x39b54 data=0x1638+0xf68 syms=[0x4+0x6200+0x4+0x7b62] 2097152K of memory above 4GB ignored Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.1-RELEASE #0: Thu Jul 24 00:19:10 MDT 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SMP Preloaded elf kernel /boot/kernel/kernel at 0xc0702000. Preloaded elf module /boot/kernel/ipfw.ko at 0xc07022e4. Preloaded elf module /boot/kernel/acpi.ko at 0xc0702390. Timecounter i8254 frequency 1193182 Hz Timecounter TSC frequency 2399328136 Hz CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) Origin = GenuineIntel Id = 0xf27 Stepping = 7 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA ,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Hyperthreading: 2 logical CPUs real memory = 4160225280 (3967 MB) avail memory = 4045996032 (3858 MB) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 - irq 0 Programming 24 pins in IOAPIC #1 Programming 24 pins in IOAPIC #2 FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): apic id: 0, version: 0x00050014, at 0xfee0 cpu1 (AP): apic id: 6, version: 0x00050014, at 0xfee0 cpu2 (AP): apic id: 1, version: 0x00050014, at 0xfee0 cpu3 (AP): apic id: 7, version: 0x00050014, at 0xfee0 io0 (APIC): apic id: 2, version: 0x00178020, at 0xfec0 io1 (APIC): apic id: 3, version: 0x00178020, at 0xfec8 io2 (APIC): apic id: 4, version: 0x00178020, at 0xfec80400 Fatal trap 12: page fault while in kernel mode cpuid = 0; lapic.id = fault virtual address = 0x8 fault code = supervisor write, page not present instruction pointer = 0x8:0xc034678e stack pointer = 0x10:0xc0724d60 frame pointer = 0x10:0xc0724d80 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) kernel: type 12 trap, code=0 Stopped at sf_buf_init+0xce: movl%eax,0x8(%ebx,%edx,1) db trace sf_buf_init(0,721000,721c00,721000,0) at sf_buf_init+0xce mi_startup() at mi_startup+0xb5 begin() at begin+0x2c db --- Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 23, 2003 10:14 Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 23, 2003 at 09:56:32AM -0600, Stephane Raimbault wrote: Hi Bosko, Looking at netstat -m, the value I'd probably be interested in is the following: 3% of cluster map consumed knowing that the Maximum possible is 25600 I can deduce that ~768 are being used? Is that correct. I'm not much of a programmer, but I did recognize the printf(); statements from a C class I didn't do well in half a decade ago... as you can tell, I'm not much of a programmer :). If it's not the 3% I should be paying attention too... then let me know :) Look at the in pool values for all the pcpu and GEN caches and add them up. Do this for clusters (since there are fewer). Compare to the Maximum Possible value. With a machine that goes into spike-load periods, you may want to have the Maximum Possible stay about 4-6 times what you have as your average in pool value (remember to sum the in pool values for the pcpu and GEN caches). The 3% is not what you think it is. It's the percentage of the allocated wired-down memory that is NOT in any of the caches but is allocated and circulating in the system. If you have a high number of cached items but the percentage is relatively low for most of the time, it's a sign that you were probably in a heavy-usage scenario some time ago, but that your current usage is relatively low. As for using the option DDB in my kernel, I do have one question. I do have remote console access that I use to go into single user mode on the box remotely. I'm suspecting I could use the debugger mode over the comconsole... I just want to make sure there is some kind of reboot command from the debugger so that I
Re: FreeBSD 5.1-R kernel panic
Thanks for getting back to me regarding that point Scott. I recently realized that I was miss-understanding how much free memory I had on the system, and I doubt I even need the full 4Gig's. Perhaps I can re-confirm how to check how much free real memory is available on the system. Looking at the top command, add the Inact and Free memory? Mem: 180M Active, 499M Inact, 213M Wired, 76K Cache, 112M Buf, 2999M Free so in this case, 3498 actual free memory? Or is there a more accurate way in determining how much memory is left in the system to be used... basically I'm trying to find the indicator I should be monitoring that should tell me if I need to add more ram to the system then I currently have. Thanks, Stephane. - Original Message - From: Scott Long [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 23, 2003 14:09 Subject: Re: FreeBSD 5.1-R kernel panic Stephane Raimbault wrote: Hi Thanks for your response, I do not have PAE enabled... I've been hesitant of turning it on, I'm not sure if it's too stable, I noticed that the asr driver is in the nodriver list in the PAE kernel config file and I use the asr driver for my Adaptec 2015S raid card. If you think its safe to enable PAE with my current setup, please let me know, I do have 2 more gig's of ram in this particular box that I'd like to enable if it doesn't compromise system stability. The asr(4) driver will likely not operate correctly in a PAE environment. There is no estimated time frame for fixing this either. Scott ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 2; lapic.id = 0100 boot() called on cpu#2 syncing disks, buffers remaining... 7149 7148 7148 7148 7143 7151 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 7143 kernel paniced again... this time with the re-compiled 5.1-R kernel with the 1.126 kern_malloc.c file. I was looking at uping the kern.vm.kmem.size as suggested to do as well, but I cannot find that value in sysctl -a, so I'm not sure where to set that specifically. I have found the value for nmbclusters and it is set to the following: kern.ipc.nmbclusters: 25600 I guess short of setting the kern.vm.kmem.size I need to get someone here the stack trace leading to the crash... is there a URL that someone can point me to for me to set the box up for this? I'll dig around in the developers handbook, I seem to remember seeing something about it in there. Thanks, Stephane. - Original Message - From: Stephane Raimbault [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Tuesday, July 22, 2003 0:33 Subject: Re: FreeBSD 5.1-R kernel panic Hi Thanks for your response, I do not have PAE enabled... I've been hesitant of turning it on, I'm not sure if it's too stable, I noticed that the asr driver is in the nodriver list in the PAE kernel config file and I use the asr driver for my Adaptec 2015S raid card. If you think its safe to enable PAE with my current setup, please let me know, I do have 2 more gig's of ram in this particular box that I'd like to enable if it doesn't compromise system stability. I will re-compile the kernel with the kern_malloc.c as you suggested, however I see that 1.126 fixes a PAE issue, so I'm not sure if that's going to help me out much. Where can I get information on how to get my kernel to provide a stack trace? I have done little of this type of troubleshooting, but would like to provide as much info before I'm forced to revert to 4.8-R in hopes to return stability to our system. If the new kernel with the updated kern_malloc.c doesn't help, I'll look at increasing the values you suggested. Thanks for the info, and I'll keep you posted with what I find. Thanks again, Stephane Raimbault - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Monday, July 21, 2003 16:36 Subject: Re: FreeBSD 5.1-R kernel panic On Mon, Jul 21, 2003 at 03:01:24PM -0600, Stephane Raimbault wrote: I'm running FreeBSD 5.1-RELEASE with the SMP kernel and ran across the following kernel panic. panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated I'm trying to figure out what could be causing this, what kind of information that I could provide to this group (or other group?) to see if this is a bug in FreeBSD that needs to be looked into? The box is basically a busy apache server... the kernel panic seemed to occur during the periodic daily was running. It seems to complete the 440.status-mailq part of periodic daily , but doesn't do 450.status-security. This isn't the first time the box has crashed at aprox. 3:01 am (when daily runs)... however this is the first time I've seend the kernel panic message quoted above in the /var/run/dmesg.boot file. I have attached the entire /var/run/dmesg.boot file to this message. What can I do to assist in identifiying and resolving this problem? Thanks, Stephane Raimbault. Just a few things: 1) Do you have PAE enabled? 2) Can you upgrade to src/sys/kern/kern_malloc.c version 1.126 or upgrade to HEAD? If you have PAE enabled and (2) does not fix your problem, then please post a stack trace that resulted in the panic. You also have a lot of RAM so if (2) above does not fix your problem, try setting the kern.vm.kmem.size bootable to approximately 350,000,000 or so (even 400,000,000 is OK as long as NMBCLUSTERS is not too-too high). -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Wed, Jul 23, 2003 at 09:08:18AM -0600, Stephane Raimbault wrote: ... I was looking at uping the kern.vm.kmem.size as suggested to do as well, but I cannot find that value in sysctl -a, so I'm not sure where to set that specifically. I have found the value for nmbclusters and it is set to the following: It's a boot-time tunable. Look at /boot/defaults/loader.conf and copy over the relevant line with a setting such as, for example, 35 into your /boot/loader.conf. Increasing nmbclusters will not help you here. In fact, if you're not running out, I would recommend keeping the value reasonable (e.g., 8192). kern.ipc.nmbclusters: 25600 I guess short of setting the kern.vm.kmem.size I need to get someone here the stack trace leading to the crash... is there a URL that someone can point me to for me to set the box up for this? I'll dig around in the developers handbook, I seem to remember seeing something about it in there. Thanks, Stephane. ... -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Thanks Bosko, I've changed my /boot/loader.conf to reflect the kern.vm.kmem.size option. kern.vm.kmem.size=35 As far as changing the nmbclusters, I'm not sure how many I use now. Do you know where I could get some values as what the total vs. how much is being used for the above values? I'll setup some graphs to monitor those values for me and get an idea of how much the system is using and when if I can. Also, I took a quick look at the developers handbook and couldn't find just yet what I needed to change to the kernel to provide a stack trace... do you know what options I should be adding to my kernel? Also, should I try not to use an SMP kernel and just run GENERIC to see if continues to have the problem? I can probably run on one CPU for a few days, especially over the weekend. Thanks, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 23, 2003 9:14 Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 23, 2003 at 09:08:18AM -0600, Stephane Raimbault wrote: ... I was looking at uping the kern.vm.kmem.size as suggested to do as well, but I cannot find that value in sysctl -a, so I'm not sure where to set that specifically. I have found the value for nmbclusters and it is set to the following: It's a boot-time tunable. Look at /boot/defaults/loader.conf and copy over the relevant line with a setting such as, for example, 35 into your /boot/loader.conf. Increasing nmbclusters will not help you here. In fact, if you're not running out, I would recommend keeping the value reasonable (e.g., 8192). kern.ipc.nmbclusters: 25600 I guess short of setting the kern.vm.kmem.size I need to get someone here the stack trace leading to the crash... is there a URL that someone can point me to for me to set the box up for this? I'll dig around in the developers handbook, I seem to remember seeing something about it in there. Thanks, Stephane. ... -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Wed, Jul 23, 2003 at 09:24:24AM -0600, Stephane Raimbault wrote: Thanks Bosko, I've changed my /boot/loader.conf to reflect the kern.vm.kmem.size option. kern.vm.kmem.size=35 As far as changing the nmbclusters, I'm not sure how many I use now. Do you know where I could get some values as what the total vs. how much is being used for the above values? I'll setup some graphs to monitor those values for me and get an idea of how much the system is using and when if I can. 'netstat -m' You can access the relevant sysctls directly; take a look at the way netstat does it in src/usr.bin/netstat/mbuf.c Also, I took a quick look at the developers handbook and couldn't find just yet what I needed to change to the kernel to provide a stack trace... do you know what options I should be adding to my kernel? Also, should I try not to use an SMP kernel and just run GENERIC to see if continues to have the problem? I can probably run on one CPU for a few days, especially over the weekend. At the very least, you need options DDB. This will drop you into the debugger on a kernel panic, at which point you can just issue 'tr' to get a stack trace. Be careful, if you only have remote access to the machine, this is generally not a good idea. Thanks, Stephane. -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Bosko, Looking at netstat -m, the value I'd probably be interested in is the following: 3% of cluster map consumed knowing that the Maximum possible is 25600 I can deduce that ~768 are being used? Is that correct. I'm not much of a programmer, but I did recognize the printf(); statements from a C class I didn't do well in half a decade ago... as you can tell, I'm not much of a programmer :). If it's not the 3% I should be paying attention too... then let me know :) As for using the option DDB in my kernel, I do have one question. I do have remote console access that I use to go into single user mode on the box remotely. I'm suspecting I could use the debugger mode over the comconsole... I just want to make sure there is some kind of reboot command from the debugger so that I can tell the box to reboot once I've captured the stack trace? If so, I'll enable the DDB tonight and get you the info as soon as I can. thanks again, Stephane. - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Wednesday, July 23, 2003 9:28 Subject: Re: FreeBSD 5.1-R kernel panic On Wed, Jul 23, 2003 at 09:24:24AM -0600, Stephane Raimbault wrote: Thanks Bosko, I've changed my /boot/loader.conf to reflect the kern.vm.kmem.size option. kern.vm.kmem.size=35 As far as changing the nmbclusters, I'm not sure how many I use now. Do you know where I could get some values as what the total vs. how much is being used for the above values? I'll setup some graphs to monitor those values for me and get an idea of how much the system is using and when if I can. 'netstat -m' You can access the relevant sysctls directly; take a look at the way netstat does it in src/usr.bin/netstat/mbuf.c Also, I took a quick look at the developers handbook and couldn't find just yet what I needed to change to the kernel to provide a stack trace... do you know what options I should be adding to my kernel? Also, should I try not to use an SMP kernel and just run GENERIC to see if continues to have the problem? I can probably run on one CPU for a few days, especially over the weekend. At the very least, you need options DDB. This will drop you into the debugger on a kernel panic, at which point you can just issue 'tr' to get a stack trace. Be careful, if you only have remote access to the machine, this is generally not a good idea. Thanks, Stephane. -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Wed, Jul 23, 2003 at 09:56:32AM -0600, Stephane Raimbault wrote: Hi Bosko, Looking at netstat -m, the value I'd probably be interested in is the following: 3% of cluster map consumed knowing that the Maximum possible is 25600 I can deduce that ~768 are being used? Is that correct. I'm not much of a programmer, but I did recognize the printf(); statements from a C class I didn't do well in half a decade ago... as you can tell, I'm not much of a programmer :). If it's not the 3% I should be paying attention too... then let me know :) Look at the in pool values for all the pcpu and GEN caches and add them up. Do this for clusters (since there are fewer). Compare to the Maximum Possible value. With a machine that goes into spike-load periods, you may want to have the Maximum Possible stay about 4-6 times what you have as your average in pool value (remember to sum the in pool values for the pcpu and GEN caches). The 3% is not what you think it is. It's the percentage of the allocated wired-down memory that is NOT in any of the caches but is allocated and circulating in the system. If you have a high number of cached items but the percentage is relatively low for most of the time, it's a sign that you were probably in a heavy-usage scenario some time ago, but that your current usage is relatively low. As for using the option DDB in my kernel, I do have one question. I do have remote console access that I use to go into single user mode on the box remotely. I'm suspecting I could use the debugger mode over the comconsole... I just want to make sure there is some kind of reboot command from the debugger so that I can tell the box to reboot once I've captured the stack trace? If so, I'll enable the DDB tonight and get you the info as soon as I can. Yes, you can do DDB over serial console. Take a look at the handbook for more information on using DDB. If you have the space in /var and enough swap, you may want to try getting a crashdump so that you can reboot and use GDB to debug. Again, take a look at the handbook. thanks again, Stephane. -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Stephane Raimbault wrote: Hi Thanks for your response, I do not have PAE enabled... I've been hesitant of turning it on, I'm not sure if it's too stable, I noticed that the asr driver is in the nodriver list in the PAE kernel config file and I use the asr driver for my Adaptec 2015S raid card. If you think its safe to enable PAE with my current setup, please let me know, I do have 2 more gig's of ram in this particular box that I'd like to enable if it doesn't compromise system stability. The asr(4) driver will likely not operate correctly in a PAE environment. There is no estimated time frame for fixing this either. Scott ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
Hi Thanks for your response, I do not have PAE enabled... I've been hesitant of turning it on, I'm not sure if it's too stable, I noticed that the asr driver is in the nodriver list in the PAE kernel config file and I use the asr driver for my Adaptec 2015S raid card. If you think its safe to enable PAE with my current setup, please let me know, I do have 2 more gig's of ram in this particular box that I'd like to enable if it doesn't compromise system stability. I will re-compile the kernel with the kern_malloc.c as you suggested, however I see that 1.126 fixes a PAE issue, so I'm not sure if that's going to help me out much. Where can I get information on how to get my kernel to provide a stack trace? I have done little of this type of troubleshooting, but would like to provide as much info before I'm forced to revert to 4.8-R in hopes to return stability to our system. If the new kernel with the updated kern_malloc.c doesn't help, I'll look at increasing the values you suggested. Thanks for the info, and I'll keep you posted with what I find. Thanks again, Stephane Raimbault - Original Message - From: Bosko Milekic [EMAIL PROTECTED] Newsgroups: mailing.freebsd.current Sent: Monday, July 21, 2003 16:36 Subject: Re: FreeBSD 5.1-R kernel panic On Mon, Jul 21, 2003 at 03:01:24PM -0600, Stephane Raimbault wrote: I'm running FreeBSD 5.1-RELEASE with the SMP kernel and ran across the following kernel panic. panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated I'm trying to figure out what could be causing this, what kind of information that I could provide to this group (or other group?) to see if this is a bug in FreeBSD that needs to be looked into? The box is basically a busy apache server... the kernel panic seemed to occur during the periodic daily was running. It seems to complete the 440.status-mailq part of periodic daily , but doesn't do 450.status-security. This isn't the first time the box has crashed at aprox. 3:01 am (when daily runs)... however this is the first time I've seend the kernel panic message quoted above in the /var/run/dmesg.boot file. I have attached the entire /var/run/dmesg.boot file to this message. What can I do to assist in identifiying and resolving this problem? Thanks, Stephane Raimbault. Just a few things: 1) Do you have PAE enabled? 2) Can you upgrade to src/sys/kern/kern_malloc.c version 1.126 or upgrade to HEAD? If you have PAE enabled and (2) does not fix your problem, then please post a stack trace that resulted in the panic. You also have a lot of RAM so if (2) above does not fix your problem, try setting the kern.vm.kmem.size bootable to approximately 350,000,000 or so (even 400,000,000 is OK as long as NMBCLUSTERS is not too-too high). -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Tue, Jul 22, 2003 at 12:18:31AM +0100, Bruce Cran wrote: It sounds like the same or similar problem reported in the 'USB crappiness' thread - the system slows down, and then any command crashes the system with the error about kmem. I posted a backtrace to the problem in usb_mem.c, and the developer has posted a temporary fix - it was a problem with bus_dma, it was allocating too much memory and running out of kernel memory. That temporary fix was committed to the tree yesterday, so a cvsup/rebuild should alleviate the symptoms. Joe -- Josef Karthauser ([EMAIL PROTECTED]) http://www.josef-k.net/ FreeBSD (cvs meister, admin and hacker) http://www.uk.FreeBSD.org/ Physics Particle Theory (student) http://www.pact.cpes.sussex.ac.uk/ An eclectic mix of fact and theory. = ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
For what it is worth, I am having the exact same problem. I cvsup'd and builtworld on Sunday, July 20, and my machine has been crashing about every half-hour since. It starts slowing down, the load average begins to climb until it eventually grinds to a halt. If I wait long enough, I will see the same panic Stephane saw. Doing a ps -auxw shows that usb0 is using the vast majority of the CPU time before the grinding halt. I have tried leaving the machine in multi-user and sigle-user mode with the same results. I have attempted to compile a new kernel that does not have the USB stuff compiled in, but my machine won't stay running long enough. I do have the debugger compiled in, but I do not know enough of what I am doing to provide reasonable information to the list. If somebody can tell me the commands to run in the debugger, I will let my machine panic again and grab that data. Mik On Mon, Jul 21, 2003 at 03:01:24PM -0600, Stephane Raimbault wrote: I'm running FreeBSD 5.1-RELEASE with the SMP kernel and ran across the following kernel panic. panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated I'm trying to figure out what could be causing this, what kind of information that I could provide to this group (or other group?) to see if this is a bug in FreeBSD that needs to be looked into? The box is basically a busy apache server... the kernel panic seemed to occur during the periodic daily was running. It seems to complete the 440.status-mailq part of periodic daily , but doesn't do 450.status-security. This isn't the first time the box has crashed at aprox. 3:01 am (when daily runs)... however this is the first time I've seend the kernel panic message quoted above in the /var/run/dmesg.boot file. I have attached the entire /var/run/dmesg.boot file to this message. What can I do to assist in identifiying and resolving this problem? Thanks, Stephane Raimbault. %cat /var/run/dmesg.boot 76.232.138 in via em0 panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated cpuid = 0; lapic.id = boot() called on cpu#0 syncing disks, buffers remaining... 7154 7154 7153 7153 7153 7153 7154 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 7152 Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.1-RELEASE #0: Fri Jun 20 18:36:54 MDT 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SRV2 Preloaded elf kernel /boot/kernel/kernel at 0xc06ef000. Preloaded elf module /boot/kernel/ipfw.ko at 0xc06ef2e4. Preloaded elf module /boot/kernel/acpi.ko at 0xc06ef390. Timecounter i8254 frequency 1193182 Hz Timecounter TSC frequency 2399328676 Hz CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) Origin = GenuineIntel Id = 0xf27 Stepping = 7 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Hyperthreading: 2 logical CPUs real memory = 4160225280 (3967 MB) avail memory = 4045828096 (3858 MB) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 - irq 0 Programming 24 pins in IOAPIC #1 Programming 24 pins in IOAPIC #2 FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): apic id: 0, version: 0x00050014, at 0xfee0 cpu1 (AP): apic id: 6, version: 0x00050014, at 0xfee0 cpu2 (AP): apic id: 1, version: 0x00050014, at 0xfee0 cpu3 (AP): apic id: 7, version: 0x00050014, at 0xfee0 io0 (APIC): apic id: 2, version: 0x00178020, at 0xfec0 io1 (APIC): apic id: 3, version: 0x00178020, at 0xfec8 io2 (APIC): apic id: 4, version: 0x00178020, at 0xfec80400 Pentium Pro MTRR support enabled npx0: math processor on motherboard npx0: INT 16 interface acpi0: PTLTDRSDT on motherboard pcibios: BIOS version 2.10 Using $PIR table, 24 entries at 0xc00fde40 acpi0: power button is handled as a fixed feature programming model. Timecounter ACPI-fast frequency 3579545 Hz acpi_timer0: 24-bit timer at 3.579545MHz port 0x1008-0x100b on acpi0 acpi_cpu0: CPU on acpi0 acpi_cpu1: CPU on acpi0 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 IOAPIC #0 intpin 16 - irq 2 IOAPIC #0 intpin 19 - irq 10 IOAPIC #0 intpin 18 - irq 11 pcib1: ACPI PCI-PCI bridge at device 2.0 on pci0 pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.HLB_ - AE_NOT_FOUND pci1: ACPI PCI bus on pcib1 pci1: base peripheral, interrupt controller at device 28.0 (no driver attached) pcib2: ACPI PCI-PCI bridge at device 29.0 on pci1 pci2: ACPI PCI bus on pcib2 IOAPIC #2 intpin 6 - irq 16 IOAPIC #2 intpin 7 - irq 17 em0: Intel(R) PRO/1000 Network Connection, Version - 1.5.31 port 0x3000-0x303f mem 0xf820-0xf821 irq 16 at device 3.0 on pci2 em0:
Re: FreeBSD 5.1-R kernel panic
On Mon, Jul 21, 2003 at 03:01:24PM -0600, Stephane Raimbault wrote: I'm running FreeBSD 5.1-RELEASE with the SMP kernel and ran across the following kernel panic. panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated I'm trying to figure out what could be causing this, what kind of information that I could provide to this group (or other group?) to see if this is a bug in FreeBSD that needs to be looked into? The box is basically a busy apache server... the kernel panic seemed to occur during the periodic daily was running. It seems to complete the 440.status-mailq part of periodic daily , but doesn't do 450.status-security. This isn't the first time the box has crashed at aprox. 3:01 am (when daily runs)... however this is the first time I've seend the kernel panic message quoted above in the /var/run/dmesg.boot file. I have attached the entire /var/run/dmesg.boot file to this message. What can I do to assist in identifiying and resolving this problem? Thanks, Stephane Raimbault. Just a few things: 1) Do you have PAE enabled? 2) Can you upgrade to src/sys/kern/kern_malloc.c version 1.126 or upgrade to HEAD? If you have PAE enabled and (2) does not fix your problem, then please post a stack trace that resulted in the panic. You also have a lot of RAM so if (2) above does not fix your problem, try setting the kern.vm.kmem.size bootable to approximately 350,000,000 or so (even 400,000,000 is OK as long as NMBCLUSTERS is not too-too high). -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Mon, Jul 21, 2003 at 06:24:07PM -0400, Mik Firestone wrote: For what it is worth, I am having the exact same problem. I cvsup'd and builtworld on Sunday, July 20, and my machine has been crashing about every half-hour since. It starts slowing down, the load average begins to climb until it eventually grinds to a halt. If I wait long enough, I will see the same panic Stephane saw. Doing a ps -auxw shows that usb0 is using the vast majority of the CPU time before the grinding halt. I have tried leaving the machine in multi-user and sigle-user mode with the same results. I have attempted to compile a new kernel that does not have the USB stuff compiled in, but my machine won't stay running long enough. I do have the debugger compiled in, but I do not know enough of what I am doing to provide reasonable information to the list. If somebody can tell me the commands to run in the debugger, I will let my machine panic again and grab that data. Mik Does reverting to pre-July 20 get rid of your problem? Note that the originator of the first Email mentionned that he is running what appeared to be stock 5.1-RELEASE, which may or may not be related to what you're seeing. If reverting to pre-July 20 gets rid of your problem, perhaps we can figure out what commit triggered this behavior for you. Also, do you have PAE enabled? -- Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED] TECHNOkRATIS Consulting Services * http://www.technokratis.com/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-R kernel panic
On Mon, Jul 21, 2003 at 06:43:11PM +, Bosko Milekic wrote: On Mon, Jul 21, 2003 at 06:24:07PM -0400, Mik Firestone wrote: For what it is worth, I am having the exact same problem. I cvsup'd and builtworld on Sunday, July 20, and my machine has been crashing about every half-hour since. It starts slowing down, the load average begins to climb until it eventually grinds to a halt. If I wait long enough, I will see the same panic Stephane saw. Doing a ps -auxw shows that usb0 is using the vast majority of the CPU time before the grinding halt. I have tried leaving the machine in multi-user and sigle-user mode with the same results. I have attempted to compile a new kernel that does not have the USB stuff compiled in, but my machine won't stay running long enough. I do have the debugger compiled in, but I do not know enough of what I am doing to provide reasonable information to the list. If somebody can tell me the commands to run in the debugger, I will let my machine panic again and grab that data. Mik Does reverting to pre-July 20 get rid of your problem? Note that the originator of the first Email mentionned that he is running what appeared to be stock 5.1-RELEASE, which may or may not be related to what you're seeing. If reverting to pre-July 20 gets rid of your problem, perhaps we can figure out what commit triggered this behavior for you. Also, do you have PAE enabled? It sounds like the same or similar problem reported in the 'USB crappiness' thread - the system slows down, and then any command crashes the system with the error about kmem. I posted a backtrace to the problem in usb_mem.c, and the developer has posted a temporary fix - it was a problem with bus_dma, it was allocating too much memory and running out of kernel memory. If the kernel debugger is compiled in, the system should drop to the db prompt when it panics. Type 'tr' and post the results it gives. This panic appears to stop memory dumps working, otherwise you could set the dumpdev variable in rc.conf also get the source listing of where the problem occurs: the FreeBSD Developers Handbook has more information. -- Bruce Cran ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
