On Mon, Nov 04, 2013 at 07:49:08AM -0800, David Wolfskill wrote: > I am supporting a performance-sensitive software development > environment that: > * Is presently contrained to operate in a FreeBSD/i386 (32-bit) > environment. > * Primarily uses svn, gcc (with various target architectures), and bmake. > * Has more main memory than FreeBSD appears to be able to make > constructive use of for this workload, but not enough to just put the > entire set of files involved in a malloc-based memory disk. > > Our current implementation involves running the processes involved > in a "full system image" jail on FreeBSD/amd64 8.x hosts.(We presently > have but one jail on each host; the developers are, in general, only > permitted to login to the jailed environment, not the host itself. > Several develeopers are usually logged in for each jail, and concurrent > builds are not uncommon.) > > My "test" environment (which I intend to provide for deployment ... > soon) is similar, but switches the host to FreeBSD/amd64 9.2-S while > leaving the jailed image unchanged; the performance is about 18% > better (using elapsed time of the software build as the salient > metric), and system CPU is significantly reduced as well (which is > a bug issue when we have multiple such builds running concurrently). > > I have (also) started experimenting with increasing compat.ia32.maxdsiz > beyond its default of 512MB: Initially, I kicked it to 2GB; more > recently, I tried setting it to 3GB. > > While I have not noticed any negative issues from either of the above > non-default settings, I have done but limited testing with concurrent > software builds in my test environment. Our swap usage is neglible, and > given that there is a more-than-adequate amount of RAM on the machines, > I wouldn't expect that increasing the compat.ia32.maxdsiz value to > be a constraint even with concurrent builds (as they are in separate > processes, and thus, separate address spaces), but one of the reasons > for this note is to ask for a reality check on that perception. > > Oddly enough, I have not noted any statistically significant > performance change from setting compat.ia32.maxdsiz. On the other > hand, I have noticed one significant behavioral change: some svn > merge operations that had been failing (for alck of available memory) > with the default (512MB) compat.ia32.maxdsiz no longer fail when > it is set to 2GB. (I have not actually tested that for 3GB, though > I expect that it would also work.) Does this make sense to anyone > else? (I have a vague memory of gcc altering its behavior to take > advantage of additional memory if it finds "enough" memory available > -- but it's been a couple of decades since I poked around in the > internals of gcc, and that's not actually an experience I'd prefer > to repeat.) > > Thanks for any insights you're willing to share!
For 32 bit kernel, the user VA layout is empty - text - data - bss ->|<- mmap area, including dso -->|<- stack ->| 0 maxdsiz 3G-68M 3G-4M For the 64bit kernel executing 32bit process, the layout is the same, but the top of the user VA is at 4G, and bottom of stack is at 4G-64M. By increasing maxdsiz to 3G, you allow for the larger heap in exchange for the limiting the space for mmap(2) to use, and shared libraries to load. In your load, where toolchain binaries are linked statically, the 'dso' is put out of the scope for memory-hungry processes. This would not work on the 32bit kernel, because 3G is the total address space available to the user process.
pgp8FhGVkXWPz.pgp
Description: PGP signature
