Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-07 Thread Andy Georges
Hi all, On 07 Feb 2013, at 10:44, Simon Marlow wrote: > On 06/02/13 22:26, Andy Georges wrote: >> Quantifying performance changes with effect size confidence intervals - >> Tomas Kalibera and Richard Jones, 2012 (tech report) > > This is a good one - it was actually a talk by Richard Jones tha

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-07 Thread Simon Marlow
On 06/02/13 22:26, Andy Georges wrote: Quantifying performance changes with effect size confidence intervals - Tomas Kalibera and Richard Jones, 2012 (tech report) This is a good one - it was actually a talk by Richard Jones that highlighted to me the problems with averaging over benchmarks (

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-06 Thread Andy Georges
Hi Johan, On 06 Feb 2013, at 17:04, Johan Tibell wrote: > On Wed, Feb 6, 2013 at 2:09 AM, Simon Marlow wrote: > This is slightly off topic, but I wanted to plant this thought in people's > brains: we shouldn't place much significance in the average of a bunch of > benchmarks (even the geometr

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-06 Thread Simon Marlow
On 06/02/13 16:04, Johan Tibell wrote: On Wed, Feb 6, 2013 at 2:09 AM, Simon Marlow mailto:marlo...@gmail.com>> wrote: This is slightly off topic, but I wanted to plant this thought in people's brains: we shouldn't place much significance in the average of a bunch of benchmarks (even

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-06 Thread Johan Tibell
On Wed, Feb 6, 2013 at 2:09 AM, Simon Marlow wrote: > This is slightly off topic, but I wanted to plant this thought in people's > brains: we shouldn't place much significance in the average of a bunch of > benchmarks (even the geometric mean), because it assumes that the > benchmarks have a sens

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-06 Thread Simon Marlow
On 05/02/13 23:48, Johan Tibell wrote: I've now added the shootout programs that could be added without modifying the programs itself. I described why some programs weren't added in nofib/shootout/README. For the curious, here's the change in these benchmarks from 7.0.4 to 7.6.2: --

RE: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-06 Thread Simon Peyton-Jones
d Terei Cc: Simon Peyton-Jones; Nicolas Frisby; ghc-devs@haskell.org Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 On Tue, Feb 5, 2013 at 3:19 AM, David Terei mailto:davidte...@gmail.com>> wrote: On 5 February 2013 02:13, Simon Peyton-Jones mailto:simo...@microsoft.

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Johan Tibell
I've now added the shootout programs that could be added without modifying the programs itself. I described why some programs weren't added in nofib/shootout/README. For the curious, here's the change in these benchmarks from 7.0.4 to 7.6.2: ---

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Johan Tibell
On Tue, Feb 5, 2013 at 2:11 PM, David Terei wrote: > Why are you creating new shootout benchmarks though rather than simply > move the exiting Shootout folder from fibon/Shootout to the top level > and fixing the makefile? > I discussed this with David offline. The summary is that the shootout b

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread David Terei
On 5 February 2013 09:34, Johan Tibell wrote: > On Tue, Feb 5, 2013 at 3:19 AM, David Terei wrote: >> >> On 5 February 2013 02:13, Simon Peyton-Jones >> wrote: >> > I believe fibon/ was helpfully added by someone, but never integrated >> > into >> > the nofib build system. Just needs doing, I t

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Johan Tibell
On Tue, Feb 5, 2013 at 3:19 AM, David Terei wrote: > On 5 February 2013 02:13, Simon Peyton-Jones > wrote: > > I believe fibon/ was helpfully added by someone, but never integrated > into > > the nofib build system. Just needs doing, I think > > No I spent a fair amount of effort fixing this up

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Austin Seipp
n > > | -Original Message- > | From: ghc-devs-boun...@haskell.org [mailto:ghc-devs-boun...@haskell.org] > | On Behalf Of Austin Seipp > | Sent: 05 February 2013 04:22 > | To: Johan Tibell > | Cc: ghc-devs@haskell.org > | Subject: Re: nofib comparisons between 7.0.4, 7.4.

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Tim Watson
We have some benchmarks for Cloud Haskell and its underlying network-transport infrastructure that I'm in the process of trying to automate. I'd be very interested to see how these fare against various GHC releases, though I suspect we'll have to tweak the dependencies considerably in order to make

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Nicolas Frisby
I'd like to investigate the "other regressions out there". Do you have more info? Perhaps a list? Maybe even benchmarking code? Thanks. On Tue, Feb 5, 2013 at 4:22 AM, Austin Seipp wrote: > I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a > few minutes ago, and he suffered

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread David Terei
; > To: Johan Tibell > Cc: ghc-devs@haskell.org > Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 > > > > Is anyone familiar with the "fibon" directory within the nofib.git > repository? > > > > http://darcs.haskell.org/nofib/fibon/ > >

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread David Terei
On 5 February 2013 01:24, Nicolas Frisby wrote: > Is anyone familiar with the "fibon" directory within the nofib.git > repository? > > http://darcs.haskell.org/nofib/fibon/ Yes. They are from here: https://github.com/dmpots/fibon Fibon is a newer, alternative benchmarking suite for Haskell done

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Simon Marlow
t:* Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 Is anyone familiar with the "fibon" directory within the nofib.git repository? http://darcs.haskell.org/nofib/fibon/ Johan, this at least seems like an potential home for the additional programs you suggested adding. In par

RE: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Simon Peyton-Jones
@haskell.org Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 Is anyone familiar with the "fibon" directory within the nofib.git repository? http://darcs.haskell.org/nofib/fibon/ Johan, this at least seems like an potential home for the additional programs you sugges

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Nicolas Frisby
Is anyone familiar with the "fibon" directory within the nofib.git repository? http://darcs.haskell.org/nofib/fibon/ Johan, this at least seems like an potential home for the additional programs you suggested adding. In particular, it has Repa, Dph, Shootout, and Hackage subdirectories. I'm doin

RE: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-05 Thread Simon Peyton-Jones
...@haskell.org] | On Behalf Of Austin Seipp | Sent: 05 February 2013 04:22 | To: Johan Tibell | Cc: ghc-devs@haskell.org | Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 | | I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a few | minutes ago, and he suf

Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

2013-02-04 Thread Austin Seipp
I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a few minutes ago, and he suffered a rather large (~25%) performance hit when upgrading to 7.6.1, which is unfortunate. Committers are typically very good about recording nofib results in their commit and being performance-courteou