Re: Compilation time

[Karel Gardas]

On 07/ 8/17 01:33 AM, Ben Gamari wrote:

8.2 will prefer both gold and lld over bfd ld. However two conditions
must hold for these to be used,

  * The ld.lld/ld.gold executable must be in $PATH (or explicitly named
by passing the LD variable to configure)

  * $CC must understand the `-fuse-ld={gold,lld}` option. For (IMHO quite
silly) political reasons, gcc doesn't support `-fuse-ld=lld`. Debian
happens to patch gcc to add support but I don't know how common this
is in other distributions.

Unfortunately, some earlier `gcc` versions didn't fail if given a
`-fuse-ld` option that they didn't understand. Sadly we have no reliable
way to detect this, so in this case we may end up passing a `-fuse-ld`
option that gcc simply ignores.


I've run into this issue too, but it looks like the issue is not in gcc, 
but in ghc's aclocal.m4 (one '$' missing). Attached patch solves this on 
OpenBSD 6.1-current where HEAD fails building on linker error. It passes 
build-id option to the gcc's linker (bfd ld) which does not support it 
as configure detects lld is presented on this system too.


Sorry for not being able to push that through usual arc.

Thanks,
Karel

diff --git a/aclocal.m4 b/aclocal.m4
index 677c0e77bc..921f137b95 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -2278,6 +2278,7 @@ AC_DEFUN([FIND_LD],[
   [enable_ld_override=yes])
 
 if test "x$enable_ld_override" = "xyes"; then
+BACKUP_LD="$LD"
 AC_CHECK_TARGET_TOOLS([LD], [ld.gold ld.lld ld])
 UseLd=''
 
@@ -2288,8 +2289,13 @@ AC_DEFUN([FIND_LD],[
   "LLD"*)  FP_CC_LINKER_FLAG_TRY(lld, $2) ;;
   *) AC_MSG_NOTICE([unknown linker version $out]) ;;
 esac
-if test "z$2" = "z"; then
+if test "z$$2" = "z"; then
 AC_MSG_NOTICE([unable to convince '$CC' to use linker '$LD'])
+# disable ld override and recheck with just ld
+enable_ld_override=no
+LD="$BACKUP_LD"
+unset ac_cv_prog_ac_ct_LD
+AC_CHECK_TARGET_TOOL([LD], [ld])
 fi
else
 AC_CHECK_TARGET_TOOL([LD], [ld])
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Ben Gamari]

Moritz Angermann  writes:

> For those of us who are on macOS,

...

> Thus, if you end up having the llvm tools (and ld.lld) in your PATH, you
> will need to set `--disable-ld-override` during configure on macOS, or your
> ghc build will fail, because you end up trying to link MachO object files
> with an ELF only linker.
>
Yikes, this is quite bad. We'll need to teach autoconf to recognize this
before the release. Thanks for bringing this up!

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Moritz Angermann]

For those of us who are on macOS,

you usually do not have any gcc, it’s all clang behind the curtains. E.g.
see

$ /usr/bin/gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

ld, is apples ld64 (see https://opensource.apple.com/source/ld64/)

$ /usr/bin/ld -v
@(#)PROGRAM:ld  PROJECT:ld64-278.4
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h 
armv6m armv7k armv7m armv7em (tvOS)
LTO support using: LLVM version 8.1.0, (clang-802.0.42)
TAPI support using: Apple TAPI version 1.33.11

if you end up having clang+llvm-4.0.0-x86_64-apple-darwin/bin or similar
in your path, you WILL have `ld.lld`, but that one is an ELF linker,
which on a MachO platform won’t be very useful (and will therefore fail
to link MachO objects).

Llvm’s `lld` will provide you with the following semi helpful note:
$ lld
lld is a generic driver.
Invoke ld.lld (Unix), ld (Mac) or lld-link (Windows) instead.

however, my copy of clang+llvm, does only ship ld.lld, and lld-link, but
not `ld`.

Thus, if you end up having the llvm tools (and ld.lld) in your PATH, you
will need to set `--disable-ld-override` during configure on macOS, or your
ghc build will fail, because you end up trying to link MachO object files
with an ELF only linker.

Cheers,
 Moritz

> On Jul 8, 2017, at 7:33 AM, Ben Gamari  wrote:
> 
> Evan Laforge  writes:
> 
>> On Fri, Jul 7, 2017 at 9:28 AM, Ben Gamari  wrote:
>>> 
>>> In short, ./configure will now choose to use ld.gold or ld.lld if
>>> available (although this can be disabled using the --disable-ld-override
>>> configure flag).
>> 
>> Just from curiosity, does this apply to OS X?  Of course, gold is
>> ELF-only, so it surely doesn't apply, but there's still lld.  OS X
>> uses clang to compile so I thought it might already use lld, but the
>> 'ld -v' output looks different, and lld.llvm.org implies it's a
>> replacement for OS X ld, so maybe not.  But it doesn't look like GNU
>> ld either, so maybe it's not affected by the BFD bug?
>> 
>> I'll try 8.2 on OS X and see if the link time changes.
> 
> 8.2 will prefer both gold and lld over bfd ld. However two conditions
> must hold for these to be used,
> 
> * The ld.lld/ld.gold executable must be in $PATH (or explicitly named
>   by passing the LD variable to configure)
> 
> * $CC must understand the `-fuse-ld={gold,lld}` option. For (IMHO quite
>   silly) political reasons, gcc doesn't support `-fuse-ld=lld`. Debian
>   happens to patch gcc to add support but I don't know how common this
>   is in other distributions.
> 
> Unfortunately, some earlier `gcc` versions didn't fail if given a
> `-fuse-ld` option that they didn't understand. Sadly we have no reliable
> way to detect this, so in this case we may end up passing a `-fuse-ld`
> option that gcc simply ignores.
> 
> In the case of OS X we use Apple's own home-grown linker. I'm not sure
> whether/how OS X's gcc wrapper treats `-fuse-ld` (beyond that it doesn't
> error if the flag is given). I also don't know whether lld is currently a
> capable replacement for OS X ld.
> 
> Cheers,
> 
> - Ben
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Ben Gamari]

Evan Laforge  writes:

> On Fri, Jul 7, 2017 at 9:28 AM, Ben Gamari  wrote:
>>
>> In short, ./configure will now choose to use ld.gold or ld.lld if
>> available (although this can be disabled using the --disable-ld-override
>> configure flag).
>
> Just from curiosity, does this apply to OS X?  Of course, gold is
> ELF-only, so it surely doesn't apply, but there's still lld.  OS X
> uses clang to compile so I thought it might already use lld, but the
> 'ld -v' output looks different, and lld.llvm.org implies it's a
> replacement for OS X ld, so maybe not.  But it doesn't look like GNU
> ld either, so maybe it's not affected by the BFD bug?
>
> I'll try 8.2 on OS X and see if the link time changes.

8.2 will prefer both gold and lld over bfd ld. However two conditions
must hold for these to be used,

 * The ld.lld/ld.gold executable must be in $PATH (or explicitly named
   by passing the LD variable to configure)

 * $CC must understand the `-fuse-ld={gold,lld}` option. For (IMHO quite
   silly) political reasons, gcc doesn't support `-fuse-ld=lld`. Debian
   happens to patch gcc to add support but I don't know how common this
   is in other distributions.

Unfortunately, some earlier `gcc` versions didn't fail if given a
`-fuse-ld` option that they didn't understand. Sadly we have no reliable
way to detect this, so in this case we may end up passing a `-fuse-ld`
option that gcc simply ignores.

In the case of OS X we use Apple's own home-grown linker. I'm not sure
whether/how OS X's gcc wrapper treats `-fuse-ld` (beyond that it doesn't
error if the flag is given). I also don't know whether lld is currently a
capable replacement for OS X ld.

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Evan Laforge]

On Fri, Jul 7, 2017 at 9:28 AM, Ben Gamari  wrote:
> Ryan Scott  writes:
>
>> Another things to note about 8.2 build times is that a bulk of the time is
>> probably being spent during the linking phase. This is because GHC 8.2 has
>> switched over to split-sections by default, but unfortunately, old versions
>> of the BFD linker take a long time to link code that uses split-sections
>> (see the discussion at [1]). There is ongoing work (which should hopefully
>> land before the final 8.2 release) to use gold instead of BFD as the
>> default linker when available, which will reduce linking times dramatically.
>>
> Indeed it has already landed. See #13541.
>
> In short, ./configure will now choose to use ld.gold or ld.lld if
> available (although this can be disabled using the --disable-ld-override
> configure flag).

Just from curiosity, does this apply to OS X?  Of course, gold is
ELF-only, so it surely doesn't apply, but there's still lld.  OS X
uses clang to compile so I thought it might already use lld, but the
'ld -v' output looks different, and lld.llvm.org implies it's a
replacement for OS X ld, so maybe not.  But it doesn't look like GNU
ld either, so maybe it's not affected by the BFD bug?

I'll try 8.2 on OS X and see if the link time changes.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Ben Gamari]

Ryan Scott  writes:

> Another things to note about 8.2 build times is that a bulk of the time is
> probably being spent during the linking phase. This is because GHC 8.2 has
> switched over to split-sections by default, but unfortunately, old versions
> of the BFD linker take a long time to link code that uses split-sections
> (see the discussion at [1]). There is ongoing work (which should hopefully
> land before the final 8.2 release) to use gold instead of BFD as the
> default linker when available, which will reduce linking times dramatically.
>
Indeed it has already landed. See #13541.

In short, ./configure will now choose to use ld.gold or ld.lld if
available (although this can be disabled using the --disable-ld-override
configure flag).

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: Compilation time

[Simon Peyton Jones via ghc-devs]

Thanks Gael.

As you may know, we’ve focused a lot on compile time in the last year or so, 
and 8.4 is a lot faster.  I think there are still plenty of bad cases, but 
things are better.

Our compile-time perf page is here: 
https://ghc.haskell.org/trac/ghc/wiki/Performance/Compiler

It links to a list of open perf tickets: 
https://ghc.haskell.org/trac/ghc/query?status=!closed=Compile-time+performance+bug

What helps above all are reproducible test cases.  Package X compiles much more 
slowly with 8.2 than with 7.10, say.   You did exactly that, thank you – can 
you open a ticket explaining exactly how to reproduce it, and what measurements 
you took.  (Matthew’s numbers seemed two orders of magnitude different; I 
wonder why?)

Another thing that is really helpful is people who characterise what the 
problem is:

  *   Is GHC generating a lot more code?   -dshow-passes can reveal this.
  *   Is it GHC at all, or is it link-times, or some other part of the tool 
chain?
  *   If it is GHC, and the output of compilation is no bigger than before, 
where is the time going?  -dshow-passes gives a rough idea of that too.
  *   Is there a particular characteristic of the program that seems to trigger 
the poor behaviour?  Can you accentuate that characteristic to make it behave 
wildly badly?

Everyone: GHC is your compiler!  We need your help.

Simon


From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Deest, Gaël
Sent: 04 July 2017 08:17
To: ghc-devs@haskell.org
Cc: Manuel Chakravarty 
Subject: Compilation time

Hi all,
As you are probably well aware, GHC performance has been a growing concern over 
the last few years. Many Haskell programmers complain that build time has 
significantly increased over the last few releases. However, to our knowledge, 
there isn't much data available to substantiate this claim and the severity of 
these problems is not well known.
That's why we would like to bring some anecdotal evidence to your attention 
that seems to indicate really major performance regressions. We stumbled upon 
the CI of the data-reify package, which is built against all GHC releases since 
2010 :

https://travis-ci.org/ku-fpg/data-reify
tl;dr: Build time has gone from 1 min 32s for GHC 7.0 to 4 min 35s for GHC 8.2. 
The 8.2 release alone seems to have increased compilation time by almost 2 
minutes, with the current development branch bringing only minor performance 
improvements.
Of course, this single data point is not sufficient to establish how severe and 
widespread these problems are. More data could probably be gathered from other 
packages. However, it certainly matches our (subjective) experience and we felt 
important to report it to you.
Regards,

--
Gaël Deest
Tweag I/O
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Matthew Pickering]

I tried the package with 8.2.1 and 8.0.2 and both build in less than 5
seconds with quite a large amount of variability. As Robin points out,
there are varying factors which can account for longer build times on
Travis.

On Tue, Jul 4, 2017 at 8:16 AM, Deest, Gaël  wrote:
> Hi all,
>
> As you are probably well aware, GHC performance has been a growing concern
> over the last few years. Many Haskell programmers complain that build time
> has significantly increased over the last few releases. However, to our
> knowledge, there isn't much data available to substantiate this claim and
> the severity of these problems is not well known.
>
> That's why we would like to bring some anecdotal evidence to your attention
> that seems to indicate really major performance regressions. We stumbled
> upon the CI of the data-reify package, which is built against all GHC
> releases since 2010 :
>
> https://travis-ci.org/ku-fpg/data-reify
>
> tl;dr: Build time has gone from 1 min 32s for GHC 7.0 to 4 min 35s for GHC
> 8.2. The 8.2 release alone seems to have increased compilation time by
> almost 2 minutes, with the current development branch bringing only minor
> performance improvements.
>
> Of course, this single data point is not sufficient to establish how severe
> and widespread these problems are. More data could probably be gathered from
> other packages. However, it certainly matches our (subjective) experience
> and we felt important to report it to you.
>
> Regards,
>
> --
> Gaël Deest
> Tweag I/O
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Compilation time

[Robin Palotai]

Note that Travis CI time is a sum of all operations, including
fetching/saving caches etc.

Opening the build log and looking for the compilation might be more
revealing. For example, 7.10 seems ~10 seconds, 8.2.1 seems ~50 seconds.

There are two issues I can see with this, that should be addressed by
benchmarking:

1) The Travis CI build hosts might show varying performance across reruns
due to shared machines.

2) Tuning the RTS params when invoking GHC as a compiler can yield
significant benefits, at least on larger compilations (such as -A128m).

It would probably be doable to fetch a set of packages from *ackage and
build with various compilers under controlled circumstances?

Robin

2017-07-04 9:16 GMT+02:00 Deest, Gaël :

> Hi all,
>
> As you are probably well aware, GHC performance has been a growing concern
> over the last few years. Many Haskell programmers complain that build time
> has significantly increased over the last few releases. However, to our
> knowledge, there isn't much data available to substantiate this claim and
> the severity of these problems is not well known.
>
> That's why we would like to bring some anecdotal evidence to your
> attention that seems to indicate really major performance regressions. We
> stumbled upon the CI of the data-reify package, which is built against all
> GHC releases since 2010 :
>
> https://travis-ci.org/ku-fpg/data-reify
>
> tl;dr: Build time has gone from 1 min 32s for GHC 7.0 to 4 min 35s for GHC
> 8.2. The 8.2 release alone seems to have increased compilation time by
> almost 2 minutes, with the current development branch bringing only minor
> performance improvements.
>
> Of course, this single data point is not sufficient to establish how
> severe and widespread these problems are. More data could probably be
> gathered from other packages. However, it certainly matches our
> (subjective) experience and we felt important to report it to you.
>
> Regards,
>
> --
> Gaël Deest
> Tweag I/O
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs