Re: Increasing KVM on amd64
By the way, this part of Alan's patch fixes a bug in RELENG7 where
mapbase is passed to vm_map_find uninitialized. -CURRENT already has
this change applied. Perhaps it's worth committing in RELENG7, too.
--- ./kern/link_elf_obj.c.orig 2008-09-01 11:06:44.0 -0700
+++ ./kern/link_elf_obj.c 2008-09-10 13:07:54.793310216 -0700
@@ -667,6 +667,7 @@
goto out;
}
ef-address = (caddr_t) vm_map_min(kernel_map);
+ mapbase = KERNBASE;
error = vm_map_find(kernel_map, ef-object, 0, mapbase,
round_page(mapsize), TRUE, VM_PROT_ALL, VM_PROT_ALL, FALSE);
if (error) {
--Artem
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
On Wed, Sep 10, 2008 at 04:12:25PM -0700, Artem Belevich wrote: Alan, Thanks a lot for the patch. I've applied it to RELENG_7 and it seems to work great - make -j8 buildworld succeeds, linux emulation seems to work well enough to run linux-sun-jdk14 binaries, ZFS ARC size is bigger, too. So far I didn't see any ZFS-related KVM shortages either. The only problem is that everything is fine as long as vm.kmem_size is set to less or equal to 4096M. As soon as I set it to 4100M or anything larger, kernel crashes on startup. I'm unable to capture exact crash messages as they keep scrolling really fast on the screen for a few seconds until the box reboots. Unfortunately the box does not have built-in serial ports, so the messages are gone before I can see them. :-( Is there a way to bump up KVM size even further - beyond 6GB? I've got a box with 8GB or RAM and would like let ZFS ARC use most of it which would require pretty large vm.kmem_max to fit it in. I was told fairly recently (a few days ago) that the 6GB limit was increased to 512GB on HEAD/CURRENT. The 6GB limit was during a transitional phase of addressing the problem. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
SVN rev 180308 on 2008-07-05 19:34:33Z by alc Enable the creation of a kmem map larger than 4GB. Submitted by: Tz-Huan Huang Make several variables related to kmem map auto-sizing static. Found by: CScout I did apply Tz-Huan Huang's patch that he pointed to shortly after you've announced your patch. As far as I can tell it's identical to SVN rev 180308 changes. Second, there is no room for a kmem map greater than 4GB unless the overall KVM size is greater than 6GB. Specifically, a 4GB kmem map isn't possible with 6GB KVM because the kmem map would overlap the kernel's code, data, and bss segment. If you're able to apply the above kern_malloc.c change to your kernel, then I should be able to describe how to increase your KVM beyond 6GB. I'd be glad to give it a try. -- --Artem ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
Artem Belevich wrote: Alan, Thanks a lot for the patch. I've applied it to RELENG_7 and it seems to work great - make -j8 buildworld succeeds, linux emulation seems to work well enough to run linux-sun-jdk14 binaries, ZFS ARC size is bigger, too. So far I didn't see any ZFS-related KVM shortages either. The only problem is that everything is fine as long as vm.kmem_size is set to less or equal to 4096M. As soon as I set it to 4100M or anything larger, kernel crashes on startup. I'm unable to capture exact crash messages as they keep scrolling really fast on the screen for a few seconds until the box reboots. Unfortunately the box does not have built-in serial ports, so the messages are gone before I can see them. :-( There are two underlying causes. First, the size of the kmem map, which holds the kernel's heap, is recorded in a 32-bit int. So, setting vm.kmem_size to 4100M is leading to integer overflow. The following change addresses this issue: sys/kern/kern_malloc.c Revision *1.167*: download http://www.freebsd.org/cgi/cvsweb.cgi/%7Echeckout%7E/src/sys/kern/kern_malloc.c?rev=1.167;content-type=text%2Fplain - view: text http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?rev=1.167;content-type=text%2Fplain, markup http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?rev=1.167;content-type=text%2Fx-cvsweb-markup, annotated http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?annotate=1.167 - select for diffs http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?r1=1.167#rev1.167 /Sat Jul 5 19:34:33 2008 UTC/ (2 months ago) by /alc/ Branches: MAIN http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?only_with_tag=MAIN CVS tags: HEAD http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?only_with_tag=HEAD Diff to: previous 1.166: preferred http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c.diff?r1=1.166;r2=1.167, colored http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c.diff?r1=1.166;r2=1.167;f=h Changes since revision 1.166: +11 -11 lines SVN rev 180308 on 2008-07-05 19:34:33Z by alc Enable the creation of a kmem map larger than 4GB. Submitted by: Tz-Huan Huang Make several variables related to kmem map auto-sizing static. Found by: CScout Second, there is no room for a kmem map greater than 4GB unless the overall KVM size is greater than 6GB. Specifically, a 4GB kmem map isn't possible with 6GB KVM because the kmem map would overlap the kernel's code, data, and bss segment. If you're able to apply the above kern_malloc.c change to your kernel, then I should be able to describe how to increase your KVM beyond 6GB. Regards, Alan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
Tz-Huan Huang wrote: On Sun, Jun 8, 2008 at 7:59 AM, Alan Cox [EMAIL PROTECTED] wrote: You can download a patch from http://www.cs.rice.edu/~alc/amd64_kvm_6GB.patch that increases amd64's kernel virtual address space to 6GB. This patch also increases the default for the kmem map to almost 2GB. I believe that kernel loadable modules still work. However, I suspect that mini-dumps are broken. I don't plan on committing this patch in its current form. Some of the changes are done in a hackish way. I am, however, curious to hear whether or not it works for you. Thanks for the patch. I applied it on 7-stable but got failed on pmap.c. [snip] We have no machine running 8-current with more than 6G memory now... Sorry, at this point the patch is only applicable to HEAD. That said, the failed chunk is probably easily applied by hand to RELENG_7. Thanks, Alan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
On Sun, Jun 8, 2008 at 2:02 PM, Alan Cox [EMAIL PROTECTED] wrote: Tz-Huan Huang wrote: On Sun, Jun 8, 2008 at 7:59 AM, Alan Cox [EMAIL PROTECTED] wrote: You can download a patch from http://www.cs.rice.edu/~alc/amd64_kvm_6GB.patch that increases amd64's kernel virtual address space to 6GB. This patch also increases the default for the kmem map to almost 2GB. I believe that kernel loadable modules still work. However, I suspect that mini-dumps are broken. I don't plan on committing this patch in its current form. Some of the changes are done in a hackish way. I am, however, curious to hear whether or not it works for you. Thanks for the patch. I applied it on 7-stable but got failed on pmap.c. [snip] We have no machine running 8-current with more than 6G memory now... Sorry, at this point the patch is only applicable to HEAD. That said, the failed chunk is probably easily applied by hand to RELENG_7. Ok, I applied the patch and fixed the failed chunk by hand. I also modified the type of vm_kmem_size from u_int to u_long. Here is the diff: http://w.csie.org/~tzhuan/tmp/vm_kmem_size.diff The patched kernel is compiled and boots without problem. I set the vm.kmem_size/vm.kmem_size_max to 3G and vfs.zfs.arc_max to 2G, everything looks fine. Some stress tests are running now, I'll report if any problem. Thanks, Tz-Huan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
On Sun, Jun 8, 2008 at 7:59 AM, Alan Cox [EMAIL PROTECTED] wrote:
You can download a patch from
http://www.cs.rice.edu/~alc/amd64_kvm_6GB.patch that increases amd64's
kernel virtual address space to 6GB. This patch also increases the default
for the kmem map to almost 2GB. I believe that kernel loadable modules
still work. However, I suspect that mini-dumps are broken.
I don't plan on committing this patch in its current form. Some of the
changes are done in a hackish way. I am, however, curious to hear whether
or not it works for you.
Thanks for the patch. I applied it on 7-stable but got failed on pmap.c.
Patching file amd64/amd64/pmap.c using Plan A...
Hunk #1 succeeded at 429 (offset -12 lines).
Hunk #2 failed at 442.
Hunk #3 succeeded at 1505 (offset -168 lines).
Hunk #4 succeeded at 1691 (offset -12 lines).
amd64/amd64/pmap.c.rej:
***
*** 442,456
/* Map from zero to end of allocations under 2M pages */
/* This replaces some of the KPTphys entries above */
for (i = 0; (i PDRSHIFT) *firstaddr; i++) {
- ((pd_entry_t *)KPDphys)[i] = i PDRSHIFT;
- ((pd_entry_t *)KPDphys)[i] |= PG_RW | PG_V | PG_PS | PG_G;
}
/* And connect up the PD to the PDP */
for (i = 0; i NKPDPE; i++) {
- ((pdp_entry_t *)KPDPphys)[i + KPDPI] = KPDphys +
(i PAGE_SHIFT);
- ((pdp_entry_t *)KPDPphys)[i + KPDPI] |= PG_RW | PG_V | PG_U;
}
/* Now set up the direct map space using either 2MB or 1GB pages */
--- 442,456
/* Map from zero to end of allocations under 2M pages */
/* This replaces some of the KPTphys entries above */
for (i = 0; (i PDRSHIFT) *firstaddr; i++) {
+ ((pd_entry_t *)KPDphys)[2048 + i] = i PDRSHIFT;
+ ((pd_entry_t *)KPDphys)[2048 + i] |= PG_RW | PG_V | PG_PS |
PG_G;
}
/* And connect up the PD to the PDP */
for (i = 0; i NKPDPE; i++) {
+ ((pdp_entry_t *)KPDPphys)[i + KPDPI - 4] = KPDphys +
(i PAGE_SHIFT);
+ ((pdp_entry_t *)KPDPphys)[i + KPDPI - 4] |= PG_RW | PG_V | PG_U;
}
/* Now set up the direct map space using either 2MB or 1GB pages */
We have no machine running 8-current with more than 6G memory now...
Thanks,
Tz-Huan
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
Kostik Belousov wrote: On Sat, Jun 07, 2008 at 06:59:35PM -0500, Alan Cox wrote: You can download a patch from http://www.cs.rice.edu/~alc/amd64_kvm_6GB.patch that increases amd64's kernel virtual address space to 6GB. This patch also increases the default for the kmem map to almost 2GB. I believe that kernel loadable modules still work. However, I suspect that mini-dumps are broken. The amd64 modules text/data/bss virtual addresses are allocated from the kernel_map (link_elf_obj.c, link_elf_load_file). Now, the lower end of the kernel_map is top-6Gb. Kernel code (both kernel proper and modules) is compiled for kernel memory model, according to the gcc info: `-mcmodel=kernel' Generate code for the kernel code model. The kernel runs in the negative 2 GB of the address space. This model has to be used for Linux kernel code. I suspect we have a problem there. The change to link_elf_obj.c is supposed to ensure allocation of an address in the upper 2GB of the kernel map for the module. Alan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Increasing KVM on amd64
On Sun, Jun 08, 2008 at 12:39:43AM -0500, Alan Cox wrote: Kostik Belousov wrote: On Sat, Jun 07, 2008 at 06:59:35PM -0500, Alan Cox wrote: You can download a patch from http://www.cs.rice.edu/~alc/amd64_kvm_6GB.patch that increases amd64's kernel virtual address space to 6GB. This patch also increases the default for the kmem map to almost 2GB. I believe that kernel loadable modules still work. However, I suspect that mini-dumps are broken. The amd64 modules text/data/bss virtual addresses are allocated from the kernel_map (link_elf_obj.c, link_elf_load_file). Now, the lower end of the kernel_map is top-6Gb. Kernel code (both kernel proper and modules) is compiled for kernel memory model, according to the gcc info: `-mcmodel=kernel' Generate code for the kernel code model. The kernel runs in the negative 2 GB of the address space. This model has to be used for Linux kernel code. I suspect we have a problem there. The change to link_elf_obj.c is supposed to ensure allocation of an address in the upper 2GB of the kernel map for the module. Ah, I overlooked it, sorry. pgpXaOmf6yTKR.pgp Description: PGP signature
Re: Increasing KVM on amd64
On Sat, Jun 07, 2008 at 06:59:35PM -0500, Alan Cox wrote: You can download a patch from http://www.cs.rice.edu/~alc/amd64_kvm_6GB.patch that increases amd64's kernel virtual address space to 6GB. This patch also increases the default for the kmem map to almost 2GB. I believe that kernel loadable modules still work. However, I suspect that mini-dumps are broken. The amd64 modules text/data/bss virtual addresses are allocated from the kernel_map (link_elf_obj.c, link_elf_load_file). Now, the lower end of the kernel_map is top-6Gb. Kernel code (both kernel proper and modules) is compiled for kernel memory model, according to the gcc info: `-mcmodel=kernel' Generate code for the kernel code model. The kernel runs in the negative 2 GB of the address space. This model has to be used for Linux kernel code. I suspect we have a problem there. I don't plan on committing this patch in its current form. Some of the changes are done in a hackish way. I am, however, curious to hear whether or not it works for you. Alan pgpigo0s4evDL.pgp Description: PGP signature
Re: Increasing KVM
Ian Campbell [EMAIL PROTECTED] writes: How exactly would I go about increasing KVM? Read the FAQ. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Increasing KVM
Ian Campbell wrote: How exactly would I go about increasing KVM? I tried bumping up KVA_PAGES from 256 to 260, but all that did was cause page faults whenever apache (or fatboy, like in this case) was started... here's the error from /var/log/messages [ ... ] I remember reading somewhere that NKPT should also be increased if you're going to fiddle with KVA_PAGES, is that true? Clearly there's something I'm not doing, I remember reading about people cranking KVA_PAGES up to 768. You only need to make additional changes if you are running a version of FreeBSD prior to 4.5; otherwise, just changing the KVA_PAGES should be enough. If you have a kernel where changing NKPT is relevent, then you are running a version before 4.5-RELEASE, and, and KVA_PAGES should have no effect whatsoever on the KVA size (it will be ignored, much as if you had used: option FRITZ_THE_CAT=260 (i.e. you can set any variable you want, and if no one looks at it, no one cares what you set it to). Prior to 4.6, Matt Dillon's autosizing patches had not yet been committed for KVA usage. On systems with large amounts of RAM, this meant that the amount of KVA space for page tables grew too large too quickly. Subsequent to the 4.6 release, there were additional changes for autotuning. Therefore you should be running at least 4.7-RELEASE, if you are messing with this at all. You haven't said whether you were running -current; everyone will assume that you are not, since you posted to the wrong mailing list for -current kernel issues (-hackers). One thing that would help is a traceback of why the fault was happening, which you haven't provided, since you only provided the fact that you were having a fault, and a meaningless address and other information, which is useless without the kernel and source code you are using. So first thing, you should get a system dump, run the debugger against a debug kernel image, and get a traceback. It will give you information which you will need to have for anyone to answer your question. My guess is that you are using up all available physical memory for backing KVA, and then crashing in a page fault attempting to trap to allocate physical backing pages for KVA space you've allocated for some reason, but have not yet used. This happens a lot these days in -current, because the wrong map is being used for preallocation of space with the new allocator (see -current list archived for recent patches for this, if you are running -current). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
