Re: Building on limited memory environments (or does -M really work?)

[Ben Gamari]

Saurabh Nanda  writes:

> Hi,
>
> We're struggling to get our build pipeline working on CircleCI, which has a
> 4GB RAM limit. Here are some project stats to set the context:
>
> - 1,200+ modules
> - 36,315 LoC of Haskell
> - On the local machine with -O1 -j the build takes approx 5.2 GB of RAM [1]
>
> Trying to build on CircleCI with -O1 -j fails consistently, as expected.
> However, the behaviour of -M -j -O1 is erratic, at best. Sometimes it
> magically works, sometimes it fails. A number of times we've seen GHC take
> 3.5 GB (as reported by `top`), even though the limit was set to 2.5GB. Here
> are flags that we've tried in various combinations we've tried. None of
> them is consistent in building / failing:
>
> * -O1
> * -j
> * +RTS -M262144 -RTS
> * +RTS -A32m -RTS
> * +RTS -n2m -RTS
>
> What is the **real** behaviour of the -M option? How does it interact with
> -j and -A?
>
Did you ever make any progress on this, Saurabh?

The summary is this:

 * -j just tells GHC to parallelise compilation across modules. This can
increase the maximum heap size needed by the compiler.

 * -A sets the nursery size; to first order the doesn't affect the
maximum heap size, but rather is helpful when running parallel
programs (e.g. ghc with -j) to minimize the frequency with which we
must garbage-collect.

 * -M is a bit tricky to define. For one, it defines the maximum heap
   size beyond which we will terminate. However, we also use it in
   garbage collector to make various decisions about GC scheduling. I'll
   admit that I'm not terribly familiar with the details here.

Note that -M does not guarantee that GHC will find a way to keep your
program under the limit that you provide. It merely ensures that the
program doesn't exceed the given size, aborting if necessary.

At least this is my understanding.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2

[Alan & Kim Zimmerman]

To me this is just further evidence that we need a first class API for tool
makers.  Which is a (slow) work in progress.

Alan

On 6 November 2017 at 19:46, Sven Panne  wrote:

> 2017-11-06 17:54 GMT+01:00 Ben Gamari :
>
>> Next time something like this arises please do open a ticket.
>>
>
> Yep, will do...
>
>
>> Yes, I have opened a differential adding such a flag. See D4164 [1].
>> Please bikeshed to taste.
>>
>
> Thanks for the quick fix!
>
>
>> In general I would really prefer that we *not* consider GHCi's REPL to be
>> a stable programmatic interface.
>
>
> I fully understand that, and that's definitely the way to go.
> Nevertheless, parsing tool/compiler output is still one of the most used
> hacks^H^H^H techniques for lots of Emacs modes (and probably other IDEs).
> Not every project is as open to suggestions and changes as GHC is, so this
> is often the only way out.
>
>
>> That being said, we cannot always preemptively add complexity to the
>> project out of fear that a given change might break a hypothetical
>> mechanical consumer.
>
>
> That's of course not what was proposed. :-)
>
>
>> GHCi is first-and-foremost a REPL for users.
>> When evaluating a change, if we feel it is likely that we will break a
>> mechanical user then we will likely guard the change with a flag.
>> However, if not, we won't.
>>
>
> I think the main problem here was communication. I can't speak for the
> haskell-mode maintainers, but for my part I didn't notice the problems
> because I mainly use LTS Stackage and that is still(!) at 8.0.2 (Why? This
> is definitely part of the whole problem.). I tried the 8.2 series only
> sparingly and only via the command line, so this is perhaps what others
> did, too, so the interaction bug went unnoticed for such a long time.
>
> Cheers,
>Sven
>
> ___
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GHC build broken

[Ben Gamari]

Simon Peyton Jones via ghc-devs  writes:

> Ben: I'm getting this from a clean build on Linux.  Any ideas?
> I'm up to date on master
>
> git branch -v
>
>   data-kind-syntax 4174458178 Add IfacePromotionInfo
>
> * master   436b3ef01e Clean up comments about match algorithm a 
> bit.
>
>   tc-untouchables  b737a45391 More simplifications to the constraint 
> solver
>
>   type-nats87d2a63b8d [behind 1491] Fix uses of (<=?) as a 
> function to reduce properly.
>
>   wip/pattern-synonyms e54d396752 [ahead 6, behind 4569] Comments and tracing 
> only
>
>   wip/spj-T13397   43540c8c6b Improve code generation for conditionals
>
>   wip/spj-early-inline 8515fcfa91 Mark non-recursive join lambdas as one-shot
>
> simonpj@cam-05-unx:~/code/HEAD/compiler$
>
> Simon
>
>
>
> libraries/ghc-prim/cbits/atomic.c:311:54: error:
>
>  error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)
>
This is due to Peter Trommler's D4009
(bd765f4b1332b3d2a7908de3f9ff1d50da0e0b1d). This changes the atomic
primops to use new GCC builtins available only in GCC 4.7 and later.
This version was released in 2012, so I thought we would be fairly safe
requiring it. This is another case where upgrading your Ubuntu
installation will help. That being said, we should certainly provide a
better error message, perhaps catching this in `configure` instead of
failing during the build.

We could also provide a fallback in the event that we are compiling with
an older gcc, although I'm not sure that this is worth the effort.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2

[Sven Panne]

2017-11-06 17:54 GMT+01:00 Ben Gamari :

> Next time something like this arises please do open a ticket.
>

Yep, will do...


> Yes, I have opened a differential adding such a flag. See D4164 [1].
> Please bikeshed to taste.
>

Thanks for the quick fix!


> In general I would really prefer that we *not* consider GHCi's REPL to be
> a stable programmatic interface.


I fully understand that, and that's definitely the way to go. Nevertheless,
parsing tool/compiler output is still one of the most used hacks^H^H^H
techniques for lots of Emacs modes (and probably other IDEs). Not every
project is as open to suggestions and changes as GHC is, so this is often
the only way out.


> That being said, we cannot always preemptively add complexity to the
> project out of fear that a given change might break a hypothetical
> mechanical consumer.


That's of course not what was proposed. :-)


> GHCi is first-and-foremost a REPL for users.
> When evaluating a change, if we feel it is likely that we will break a
> mechanical user then we will likely guard the change with a flag.
> However, if not, we won't.
>

I think the main problem here was communication. I can't speak for the
haskell-mode maintainers, but for my part I didn't notice the problems
because I mainly use LTS Stackage and that is still(!) at 8.0.2 (Why? This
is definitely part of the whole problem.). I tried the 8.2 series only
sparingly and only via the command line, so this is perhaps what others
did, too, so the interaction bug went unnoticed for such a long time.

Cheers,
   Sven
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2

[Ben Gamari]

Next time something like this arises please do open a ticket.


Sven Panne  writes:

> 2017-11-05 15:37 GMT+01:00 :
>
>> A better approach might be to develop a "machine-readable" output format
>> which then is kept stable, and can be enabled with a flag. Git has a
>> similar solution.
>>
>
> Without doubt, this is definitely the better approach, but this is hardly
> what can be achieved for 8.2.2. Adding some flag to get the old behavior
> back when wanted *is* achievable.
>
Yes, I have opened a differential adding such a flag. See D4164 [1].
Please bikeshed to taste.

In general I would really prefer that we *not* consider GHCi's REPL to be
a stable programmatic interface. If haskell-mode needs access to some
bit of information from GHC, then please let's discuss how to provide it
in a more robust manner. This shouldn't be hard; we just need to have
feedback on what the interfaces required should look like.

[1] https://phabricator.haskell.org/D4164

>
>> It would be a shame to avoid changes which make the user experience better
>> simply because other projects cannot sync their development cycle,
>>
>
> Don't get me wrong: I'm all for improving user experience, but making ad
> hoc changes without enough thought or even a chance to get the old behavior
> back is probably not the right way to proceed. All SW lives in some kind of
> ecosystem, so it should behave well in that. And for Emacs users, the user
> experience has been made much worse.
>
The trouble is that GHC has an extremely large surface area, any bit of
which might have mechanical consumers. Those who reviewed the patch
indeed considered making this change behind a flag (as Francesco's
original patch did) but we concluded that this would be pointless as it
didn't seem likely that anyone relied on this behavior. It seems that in
this case we were wrong

That being said, we cannot always preemptively add complexity to the
project out of fear that a given change might break a hypothetical
mechanical consumer. GHCi is first-and-foremost a REPL for users. 
When evaluating a change, if we feel it is likely that we will break a
mechanical user then we will likely guard the change with a flag.
However, if not, we won't.

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: GHC build broken

[Simon Peyton Jones via ghc-devs]

ps this Linux box is way behind

gcc --version

gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

Copyright (C) 2011 Free Software Foundation, Inc.

This is free software; see the source for copying conditions.  There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.



From: Simon Peyton Jones
Sent: 06 November 2017 16:23
To: ghc-devs@haskell.org
Subject: GHC build broken

Ben: I'm getting this from a clean build on Linux.  Any ideas?
I'm up to date on master

git branch -v

  data-kind-syntax 4174458178 Add IfacePromotionInfo

* master   436b3ef01e Clean up comments about match algorithm a bit.

  tc-untouchables  b737a45391 More simplifications to the constraint solver

  type-nats87d2a63b8d [behind 1491] Fix uses of (<=?) as a function 
to reduce properly.

  wip/pattern-synonyms e54d396752 [ahead 6, behind 4569] Comments and tracing 
only

  wip/spj-T13397   43540c8c6b Improve code generation for conditionals

  wip/spj-early-inline 8515fcfa91 Mark non-recursive join lambdas as one-shot

simonpj@cam-05-unx:~/code/HEAD/compiler$

Simon



libraries/ghc-prim/cbits/atomic.c:311:54: error:

 error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)

|

311 |   __atomic_store_n((StgWord16 *) x, (StgWord16) val, __ATOMIC_SEQ_CST);

|  ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite32':



make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.o] Error 1

make[1]: *** Waiting for unfinished jobs

libraries/ghc-prim/cbits/atomic.c:318:54: error:

 error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)

|

318 |   __atomic_store_n((StgWord32 *) x, (StgWord32) val, __ATOMIC_SEQ_CST);

|  ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite64':



libraries/ghc-prim/cbits/atomic.c:325:54: error:

 error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)

|

325 |   __atomic_store_n((StgWord64 *) x, (StgWord64) val, __ATOMIC_SEQ_CST);

|  ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread64':



libraries/ghc-prim/cbits/atomic.c:294:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

294 | }

| ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread32':



libraries/ghc-prim/cbits/atomic.c:287:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

287 | }

| ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread16':



libraries/ghc-prim/cbits/atomic.c:280:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

280 | }

| ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread8':



libraries/ghc-prim/cbits/atomic.c:273:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

273 | }

| ^

cc1: all warnings being treated as errors

`gcc' failed in phase `C Compiler'. (Exit code: 1)

make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.dyn_o] Error 1

<>

make: *** [all] Error 2

simonpj@cam-05-unx:~/5builds/HEAD$ dirs

~/5builds/HEAD ~/5builds/HEAD/testsuite/tests/typecheck/should_fail

simonpj@cam-05-unx:~/5builds/HEAD$
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

GHC build broken

[Simon Peyton Jones via ghc-devs]

Ben: I'm getting this from a clean build on Linux.  Any ideas?
I'm up to date on master

git branch -v

  data-kind-syntax 4174458178 Add IfacePromotionInfo

* master   436b3ef01e Clean up comments about match algorithm a bit.

  tc-untouchables  b737a45391 More simplifications to the constraint solver

  type-nats87d2a63b8d [behind 1491] Fix uses of (<=?) as a function 
to reduce properly.

  wip/pattern-synonyms e54d396752 [ahead 6, behind 4569] Comments and tracing 
only

  wip/spj-T13397   43540c8c6b Improve code generation for conditionals

  wip/spj-early-inline 8515fcfa91 Mark non-recursive join lambdas as one-shot

simonpj@cam-05-unx:~/code/HEAD/compiler$

Simon



libraries/ghc-prim/cbits/atomic.c:311:54: error:

 error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)

|

311 |   __atomic_store_n((StgWord16 *) x, (StgWord16) val, __ATOMIC_SEQ_CST);

|  ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite32':



make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.o] Error 1

make[1]: *** Waiting for unfinished jobs

libraries/ghc-prim/cbits/atomic.c:318:54: error:

 error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)

|

318 |   __atomic_store_n((StgWord32 *) x, (StgWord32) val, __ATOMIC_SEQ_CST);

|  ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite64':



libraries/ghc-prim/cbits/atomic.c:325:54: error:

 error: '__ATOMIC_SEQ_CST' undeclared (first use in this function)

|

325 |   __atomic_store_n((StgWord64 *) x, (StgWord64) val, __ATOMIC_SEQ_CST);

|  ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread64':



libraries/ghc-prim/cbits/atomic.c:294:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

294 | }

| ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread32':



libraries/ghc-prim/cbits/atomic.c:287:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

287 | }

| ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread16':



libraries/ghc-prim/cbits/atomic.c:280:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

280 | }

| ^

libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread8':



libraries/ghc-prim/cbits/atomic.c:273:1: error:

 error: control reaches end of non-void function [-Werror=return-type]

|

273 | }

| ^

cc1: all warnings being treated as errors

`gcc' failed in phase `C Compiler'. (Exit code: 1)

make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.dyn_o] Error 1

<>

make: *** [all] Error 2

simonpj@cam-05-unx:~/5builds/HEAD$ dirs

~/5builds/HEAD ~/5builds/HEAD/testsuite/tests/typecheck/should_fail

simonpj@cam-05-unx:~/5builds/HEAD$
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs