Re: Huge pages and druntime parameters
On 09/27/2013 04:01 AM, luminousone wrote: I have played around with huge pages a bit, my understanding from reading about them, and use(ROAM diamond tree nodes in huge page memory), is that they need to be used carefully, many processors only have a small number of entries for huge pages in their lookup tables. When out of huge pages it would fallback to normal mmap. The best use of huge pages seems to be from code built around them, not sure if it entirely makes sense to put them in gc, The main benefit should come from heavily reducing the amount of TLB entries in the cache. So if your program suffers from many cache misses due to TLB lookups that would help. And the freed cache can also be used for other stuff instead.
Re: Huge pages and druntime parameters
On 09/27/2013 03:33 PM, Rainer Schuetze wrote: An application that needs to allocate ore than a few MB is currently often slowed down considerably by early collections of the GC. Being able to use larger values for initial memory and pool size increments would help a lot here. Yep, that's the main use case. For the precise GC I moved the configuration variable(s) to its own module: https://github.com/rainers/druntime/blob/gcx_precise/src/gc/config.d This module allows configuration via environment variables, but you can change the defaults by adding a file with module declaration gc.config and other settings. Env variables make sense.
Re: Huge pages and druntime parameters
On 09/27/2013 02:54 AM, Jonathan M Davis wrote: IIRC, Leandro made his concurrent GC use environment variables to set stuff like this. - Jonathan M Davis Speaking about the concurrent GC, huge pages incur a huge penalty for COW because the virtual memory granularity goes from 4Kb to 2Mb, i.e. 500x more copying when the parent process modifies a page while the forked process performs GC.
Re: Huge pages and druntime parameters
On 27.09.2013 02:31, Martin Nowak wrote: Running dmd with huge page backed malloc resulted in about 10% faster compilations. This looks very promising though other tests with some D GC microbenchmarks did not benefit from huge pages. You can read more about this here. http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/ I'm interested in adding a runtime parameter that would advise the GC to preferably use huge pages as backing memory. There are a number of other interesting runtime parameters, like max heap size, GC grow policies, turning on statistics, so I wonders how to make them available. An application that needs to allocate ore than a few MB is currently often slowed down considerably by early collections of the GC. Being able to use larger values for initial memory and pool size increments would help a lot here. Java seems to use -XX:+UseLargePages and -Xmx for the maximum heap size. GHC uses a nice scheme where runtime parameters passed via command line are embraced by +RTS arg1 arg2 -RTS. They also require to link-in support for this. http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html Any good ideas for druntime? For the precise GC I moved the configuration variable(s) to its own module: https://github.com/rainers/druntime/blob/gcx_precise/src/gc/config.d This module allows configuration via environment variables, but you can change the defaults by adding a file with module declaration gc.config and other settings.
Huge pages and druntime parameters
Running dmd with huge page backed malloc resulted in about 10% faster compilations. This looks very promising though other tests with some D GC microbenchmarks did not benefit from huge pages. You can read more about this here. http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/ I'm interested in adding a runtime parameter that would advise the GC to preferably use huge pages as backing memory. There are a number of other interesting runtime parameters, like max heap size, GC grow policies, turning on statistics, so I wonders how to make them available. Java seems to use -XX:+UseLargePages and -Xmx for the maximum heap size. GHC uses a nice scheme where runtime parameters passed via command line are embraced by +RTS arg1 arg2 -RTS. They also require to link-in support for this. http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html Any good ideas for druntime?
Re: Huge pages and druntime parameters
On Friday, September 27, 2013 02:31:21 Martin Nowak wrote: I'm interested in adding a runtime parameter that would advise the GC to preferably use huge pages as backing memory. [snip] Any good ideas for druntime? IIRC, Leandro made his concurrent GC use environment variables to set stuff like this. - Jonathan M Davis
Re: Huge pages and druntime parameters
On Friday, 27 September 2013 at 00:31:23 UTC, Martin Nowak wrote: Running dmd with huge page backed malloc resulted in about 10% faster compilations. This looks very promising though other tests with some D GC microbenchmarks did not benefit from huge pages. You can read more about this here. http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/ I'm interested in adding a runtime parameter that would advise the GC to preferably use huge pages as backing memory. There are a number of other interesting runtime parameters, like max heap size, GC grow policies, turning on statistics, so I wonders how to make them available. Java seems to use -XX:+UseLargePages and -Xmx for the maximum heap size. GHC uses a nice scheme where runtime parameters passed via command line are embraced by +RTS arg1 arg2 -RTS. They also require to link-in support for this. http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html Any good ideas for druntime? I have played around with huge pages a bit, my understanding from reading about them, and use(ROAM diamond tree nodes in huge page memory), is that they need to be used carefully, many processors only have a small number of entries for huge pages in their lookup tables. The best use of huge pages seems to be from code built around them, not sure if it entirely makes sense to put them in gc,
