Re: Building GHC for performance testing
On 08/03/14 07:54, Herbert Valerio Riedel wrote: Hi Simon, On 2014-03-08 at 08:18:20 +0100, Simon Marlow wrote: On 06/03/14 09:50, Johan Tibell wrote: [...] * Are there any tweaks to mk/build.mk http://build.mk we can do to make the build faster without compromising the results? Turn down the stage2 optimisation, and turn off the docs: GhcStage2HcOpts = -O ...but doesn't that compromise the compile-time metrics collected by nofib about GHC's performance? I think Johan is interested primarily in runtime performance, but yes if you care about compile time then it's better to leave GhcStage2HcOpts at the default. Cheers, Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Building GHC for performance testing
one idea that I think Ben Gamari (or was it Reid?) pointed out is that another (potentially valuable) metric is how long does it take for ghc to build itself On Sat, Mar 8, 2014 at 5:20 AM, Simon Marlow marlo...@gmail.com wrote: On 08/03/14 07:54, Herbert Valerio Riedel wrote: Hi Simon, On 2014-03-08 at 08:18:20 +0100, Simon Marlow wrote: On 06/03/14 09:50, Johan Tibell wrote: [...] * Are there any tweaks to mk/build.mk http://build.mk we can do to make the build faster without compromising the results? Turn down the stage2 optimisation, and turn off the docs: GhcStage2HcOpts = -O ...but doesn't that compromise the compile-time metrics collected by nofib about GHC's performance? I think Johan is interested primarily in runtime performance, but yes if you care about compile time then it's better to leave GhcStage2HcOpts at the default. Cheers, Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Building GHC for performance testing
Hi Simon, On 2014-03-08 at 08:18:20 +0100, Simon Marlow wrote: On 06/03/14 09:50, Johan Tibell wrote: [...] * Are there any tweaks to mk/build.mk http://build.mk we can do to make the build faster without compromising the results? Turn down the stage2 optimisation, and turn off the docs: GhcStage2HcOpts = -O ...but doesn't that compromise the compile-time metrics collected by nofib about GHC's performance? cheers, hvr ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Building GHC for performance testing
Hi, I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one: cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow Questions: * Does this look sensible? * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results? * Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic? ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Building GHC for performance testing
On 2014-03-06 at 10:50:09 +0100, Johan Tibell wrote: I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one: cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow Questions: * Does this look sensible? If the script is supposed to start in an empty environment, then it looks good to me * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results? You should disable LaTeX generation, and maybe also haddock generation; and possibly remove the libraries/dph folder (unless that's becomes part of the benchmarks) how often would you run the benchmarks? once a day? * Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic? Btw, does benchmarking inside a VM provide good enough measurements? E.g. I'd be worried from noise due to vSMP scheduling (as GHC's GC is rather sensible to proper SMP scheduling) and maybe from noise coming from other VMs trashing the caches. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Building GHC for performance testing
On 06/03/14 09:50, Johan Tibell wrote: Hi, I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one: cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow Questions: * Does this look sensible? * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results? * Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic? ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs I think this ties in with the whole issue of not having nightlies/test bots running. I think if the infrastructure for that was in place then this would simply be a part of the functionality. re: subset of benchmarks, I believe they are just regular tests, in which case you can explicitly pass in which test cases should be ran. Correct me if I'm wrong. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Building GHC for performance testing
There's also the issue that certain vms disable some simd instructions(I think), and while we don't have many libs that are simd heavy as yet this could be an issue in certain future benchmarks? (I could be completely wrong though. ) On Thursday, March 6, 2014, Herbert Valerio Riedel hvrie...@gmail.com wrote: On 2014-03-06 at 10:50:09 +0100, Johan Tibell wrote: I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one: cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow Questions: * Does this look sensible? If the script is supposed to start in an empty environment, then it looks good to me * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results? You should disable LaTeX generation, and maybe also haddock generation; and possibly remove the libraries/dph folder (unless that's becomes part of the benchmarks) how often would you run the benchmarks? once a day? * Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic? Btw, does benchmarking inside a VM provide good enough measurements? E.g. I'd be worried from noise due to vSMP scheduling (as GHC's GC is rather sensible to proper SMP scheduling) and maybe from noise coming from other VMs trashing the caches. ___ ghc-devs mailing list ghc-devs@haskell.org javascript:; http://www.haskell.org/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Building GHC for performance testing
Hello Johan, Off the top of my head: Technically speaking, the compiler does not have to be optimized to do build tests; just the libraries. So you could probably make the builds run a bit faster this way. Your command for nofib will only run the standard benchmarks; to run, for example, the GC benchmarks, you will have to cd into the gc directory and run make there. You may also need to vary the tests parameters in other ways, such as threaded and un-threaded. Cheers, Edward Excerpts from Johan Tibell's message of 2014-03-06 01:50:09 -0800: Hi, I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one: cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow Questions: * Does this look sensible? * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results? * Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic? ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
