On Tue, Feb 5, 2013 at 2:54 AM, Simon Peyton-Jones <simo...@microsoft.com> wrote: > Austin, a 25% performance regression moving to 7.6 is not AT ALL what I > expect. I generally expect modest performance improvements? Can you > characterise more precisely what is happening? The place I always start is > to compile the entire thing with -ticky and see where allocation is changing. > (Using -prof affects the optimiser too much.)
I have CC'd Eyal just in case. The discussion was informal but he can hopefully provide more context and rigor. I think off hand, this occurred in a rather large-ish application of his (Lamdu?,) and so tracking down precise reasons may prove difficult. I think the most likely case is just those few 'small cuts' accumulate quickly and are reflecting poorly for this particular case - and that's really the worse 'bug report' of all! Hashable/lens alone for example could certainly make a sizable impact here when added up, e.g. [1] is a recent example of an alleged perf anomaly as of late. And the OS could certainly be relevant.[2] All the more reason to expand nofib and get those bots up! [1] https://github.com/tibbe/hashable/issues/57 [2] Just thinking out loud, but, whenever this happens we really need to characterize results on a per-OS/hardware basis if possible in the future, with some relatively detailed hardware info, to be unambiguous. In terms of raw CPU speed, a lot of benchmarks probably won't stand out due to the OS. But OS X is scheduled to get worse in the SMP case soon[3] for example, and if we inevitably try and start doing things like latency or I/O benchmarks, I'm more than certain things will pop up here. [3] See this ticket: http://hackage.haskell.org/trac/ghc/ticket/7602 > Simon > > | -----Original Message----- > | From: ghc-devs-boun...@haskell.org [mailto:ghc-devs-boun...@haskell.org] > | On Behalf Of Austin Seipp > | Sent: 05 February 2013 04:22 > | To: Johan Tibell > | Cc: ghc-devs@haskell.org > | Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 > | > | I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a few > | minutes ago, and he suffered a rather large (~25%) performance hit when > | upgrading to 7.6.1, which is unfortunate. > | > | Committers are typically very good about recording nofib results in > | their commit and being performance-courteous, but I'm not sure there's > | ever been a longer-scale view of GHC performance over multiple releases > | like this - or even a few months. At least not recently. On top of that, > | his application was a type checker, which may certainly stress different > | performance points than what nofib might. Once we get performance bots > | set up, I've got a small set of machines I'm willing to throw at it. > | > | Thanks for the results, Johan! > | > | On Mon, Feb 4, 2013 at 4:33 PM, Johan Tibell <johan.tib...@gmail.com> > | wrote: > | > Hi all, > | > > | > I haven't had much time to do performance tzar work yet, but I did run > | > nofib on the last few GHC releases to see the current trend. The > | > benchmarks where run on my 64-bit Core i7-3770 @ 3.40GHz Linux > | machine. Here are the results: > | > > | > 7.0.4 to 7.4.2: > | > > | > ---------------------------------------------------------------------- > | ---------- > | > Program Size Allocs Runtime Elapsed TotalMem > | > ---------------------------------------------------------------------- > | ---------- > | > Min -1.6% -57.3% -39.1% -36.4% -25.0% > | > Max +21.5% +121.5% +24.5% +25.4% +300.0% > | > Geometric Mean +8.5% -0.7% -7.1% -5.2% +2.0% > | > > | > The big loser here in terms of runtime is "kahan", which I added to > | > test tight loops involving unboxed arrays and floating point > | > arithmetic. I believe there was a regression in fromIntegral RULES > | > during this release, which meant that some conversions between > | > fixed-width types went via Integer, causing unnecessary allocation. > | > > | > 7.4.2 to 7.6.1: > | > > | > ---------------------------------------------------------------------- > | ---------- > | > Program Size Allocs Runtime Elapsed TotalMem > | > ---------------------------------------------------------------------- > | ---------- > | > Min -5.1% -23.8% -11.8% -12.9% -50.0% > | > Max +5.3% +225.5% +7.2% +8.8% +200.0% > | > Geometric Mean -0.4% +2.1% +0.3% +0.2% +0.7% > | > > | > The biggest loser here in terms of runtime is "integrate". I haven't > | > looked into why yet. > | > > | > 7.6.1 to 7.6.2: > | > > | > ---------------------------------------------------------------------- > | ---------- > | > Program Size Allocs Runtime Elapsed TotalMem > | > ---------------------------------------------------------------------- > | ---------- > | > Min -2.9% +0.0% -4.8% -4.4% -1.9% > | > Max +0.0% +1.0% +4.5% +6.4% +20.8% > | > Geometric Mean -1.7% +0.0% +0.1% +0.3% +0.2% > | > > | > I have two takeaways: > | > > | > * It's worthwhile running nofib before releases as it does find some > | > programs that regressed. > | > * There are some other regressions out there (i.e. in code on > | > Hackage) that aren't reflected here, suggesting that we need to add > | > more programs to nofib. > | > > | > Cheers, > | > Johan > | > > | > > | > _______________________________________________ > | > ghc-devs mailing list > | > ghc-devs@haskell.org > | > http://www.haskell.org/mailman/listinfo/ghc-devs > | > > | > | > | > | -- > | Regards, > | Austin > | > | _______________________________________________ > | ghc-devs mailing list > | ghc-devs@haskell.org > | http://www.haskell.org/mailman/listinfo/ghc-devs -- Regards, Austin _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs