RE: Adding a "release" setting in build.mk.sample (and some other build system questions)

[Harry .]

> Date: Fri, 29 Jan 2016 17:44:56 +0100 
> Subject: Re: Adding a "release" setting in build.mk.sample (and some  
> other build system questions) 
> From: thomasmied...@gmail.com 
> To: volderm...@hotmail.com 
> CC: ghc-devs@haskell.org 
>  
> The default (and thus release) `SRC_HC_OPTS` uses `-H32m` (see  
> mk/config.mk.in). 
>  
> Maybe you want to run some tests to see if it makes a difference on the  
> total build time? 
>  
> I suggest also measuring without any `-H` flag, with just `-H` (see  
> commit below), and with `-H1G` or some other large value. 

I'm building in a container on a busy host at work (i.e. which I don't 
control), so unfortunately there's too much background noise to make meaningful 
measurements.

Does SRC_HC_OPTS control memory usage while building GHC, or when it's running? 
And either way, why does this need setting for building GHC, but isn't normally 
used when compiling regular Haskell programs?
  
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: [ANNOUNCE] Shaking up GHC

[Andrey Mokhov]

Simon & all,

> Why not just make the 'validate' script invoke the shake build system
> (instead of 'make') when you say 'sh validate --shake'?  That would be
> simple, wouldn't it? So shake builds GHC, and then 'validate' invokes
> 'make test' (or whatever) to run the testsuite.
>   
> Little things like this would significantly increase uptake I think!

We can now run:

shake-build/build.sh validate

This will rebuild stage2 GHC, ghc-pkg and hpc if need be, and will then run GHC 
tests by executing `make fast` in `testsuite/tests` folder 
(https://github.com/snowleopard/shaking-up-ghc#testing). 

There is still a lot of work to add proper support for various useful command 
line settings like `TEST=abc` or `TEST_HC=xyz`.

It would be great if someone could add support for `--shake` command line flag 
to the `validate` script, as Simon suggested. This would be a better solution, 
but I'm not confident enough to edit the validate script myself.

Thanks to Thomas Miedema for his patient guidance on how GHC test suite works.

Cheers,
Andrey

> -Original Message-
> From: Simon Peyton Jones [mailto:simo...@microsoft.com]
> Sent: 26 January 2016 08:55
> To: Andrey Mokhov; Ben Gamari
> Cc: ghc-devs@haskell.org
> Subject: RE: [ANNOUNCE] Shaking up GHC
> 
> |  That's correct. Note though that the two build systems put (some)
> |  build results in the same directories, e.g. inplace/bin/ghc-stage1,
> so
> |  there is some interaction between them. In future it would be
> possible
> |  to decouple them completely (if need be).
> 
> Documenting this side-by-side possibility on the home page would be
> very helpful to up-take.
> 
> |  > How can I do that using Shake to build?  Maybe
> |  >  sh validate --shake --fast --no-clean
> |  > or something?  Just modifying the validate script to make it
> |  >convenient to use shake for the build part would be great.
> |
> |  This may be possible precisely because binaries are where validate
> |  expects them to be. I tried to pull this off, but unsuccessfully so
> |  far (validate starts rebuilding everything from scratch with make).
> I
> |  think the reason is that we changed some naming conventions for
> |  directories (using stageN instead of dist, dist-boot, dist-install),
> |  so validate can't find some utils. I'll let you know if I find a
> |  workaround.
> 
> Why not just make the 'validate' script invoke the shake build system
> (instead of 'make') when you say 'sh validate --shake'?  That would be
> simple, wouldn't it?  So shake builds GHC, and then 'validate' invokes
> 'make test' (or whatever) to run the testsuite.
>   
> Little things like this would significantly increase uptake I think!
> 
> Simon
> 
> 
> 
> |
> |  Ben,
> |
> |  > > But User.hs is a source file, which we shouldn't normally modify
> |  > > lest we accidentally commit it. Could we have a non-source file
> to
> |  > > modify (rather like
> |  > >
> |  https://na01.safelinks.protection.outlook.com/?url=build.mk=01%
> |  > >
> |  7c01%7csimonpj%40064d.mgd.microsoft.com%7c7f0edcff3c78490bd0e808d325
> |  > >
> |  be4774%7c72f988bf86f141af91ab2d7cd011db47%7c1=8L%2b7QWlhbAwUnY
> |  > > fAfCEwTXTC2PUMd17m1ceCcD2TE%2fU%3d being based on
> build.mk.sample)
> |  > >
> |  > One way around this would be to add User.hs to .gitignore. That
> way
> |  > git will ignore changes to this files when you `git commit -a`.
> You
> |  > can still, however, commit changes to it if you ask git
> explicitly.
> |
> |  Yes, I was thinking about the same idea.
> |
> |  Cheers,
> |  Andrey
> |
> |  -Original Message-
> |  From: Simon Peyton Jones [mailto:simo...@microsoft.com]
> |  Sent: 25 January 2016 12:14
> |  To: Andrey Mokhov ; ghc-
> |  d...@haskell.org
> |  Cc: Neil Mitchell ; Simon Marlow
> |  
> |  Subject: RE: [ANNOUNCE] Shaking up GHC
> |
> |  Very good.
> |
> |  As I understand it, it can work side-by-side with the existing build
> |  system, correct?  That means we don't need to make an either/or
> |  choice, which is very helpful.
> |
> |  Every day I do
> | sh validate --fast --no-clean
> |  How can I do that using Shake to build?  Maybe
> | sh validate --shake --fast --no-clean
> |  or something?  Just modifying the validate script to make it
> |  convenient to use shake for the build part would be great.
> |
> |  You say: The make-based build system uses mk/build.mk to specify
> user
> |  build settings. We use src/Settings/User.hs for the same purpose.
> Feel
> |  free to experiment following the Haddock comments.
> |
> |  But User.hs is a source file, which we shouldn't normally modify
> lest
> |  we accidentally commit it.  Could we have a non-source file to
> modify
> |  (rather like
> |
> https://na01.safelinks.protection.outlook.com/?url=build.mk=01%7c
> |
> 01%7csimonpj%40064d.mgd.microsoft.com%7c7f0edcff3c78490bd0e808d325be47
> |
> 74%7c72f988bf86f141af91ab2d7cd011db47%7c1=8L%2b7QWlhbAwUnYfAfCEw

Build broken

[Geoffrey Mainland]

I attempted to rebase my fix for #11487, but I can no longer validate.
The culprit is 6c7760b2. I am seeing the errors below under linux,
building with 7.10.3.

Herbert, I hope there is a quick fix?

Thanks,
Geoff

libraries/unix/System/Posix/IO/Common.hsc:316:61: error:
• Couldn't match type ‘()’ with ‘Base.CFLock’
  Expected type: Ptr Base.CFLock
Actual type: Ptr CFLock
• In the third argument of ‘Base.c_fcntl_lock’, namely ‘p_flock’
  In the second argument of ‘throwErrnoIfMinus1_’, namely
‘(Base.c_fcntl_lock fd (5) p_flock)’
  In a stmt of a 'do' block:
throwErrnoIfMinus1_ "getLock" (Base.c_fcntl_lock fd (5) p_flock)

libraries/unix/System/Posix/IO/Common.hsc:364:61: error:
• Couldn't match type ‘()’ with ‘Base.CFLock’
  Expected type: Ptr Base.CFLock
Actual type: Ptr CFLock
• In the third argument of ‘Base.c_fcntl_lock’, namely ‘p_flock’
  In the second argument of ‘throwErrnoIfMinus1_’, namely
‘(Base.c_fcntl_lock fd (6) p_flock)’
  In the expression:
throwErrnoIfMinus1_ "setLock" (Base.c_fcntl_lock fd (6) p_flock)

libraries/unix/System/Posix/IO/Common.hsc:371:35: error:
• Couldn't match type ‘()’ with ‘Base.CFLock’
  Expected type: Ptr Base.CFLock
Actual type: Ptr CFLock
• In the third argument of ‘Base.c_fcntl_lock’, namely ‘p_flock’
  In the second argument of ‘throwErrnoIfMinus1_’, namely
‘(Base.c_fcntl_lock fd (7) p_flock)’
  In the expression:
throwErrnoIfMinus1_
  "waitToSetLock" (Base.c_fcntl_lock fd (7) p_flock)

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Nailing down what we expect IO to do and not do - and why

[Christopher Allen]

I'm writing a book, I'd like to get this nailed down and to get it right.
If anyone on here that's familiar with the various ways in which
IO/State#/realWorld# work in GHC and you have time to reply, anything at
all would be welcome. Any pointers, links, references, details, anecdotes,
or faint memories of GHC bugs will be greatly appreciated! Getting this
written up (possibly for addition to Michael Snoyman's wiki article?) would
make me, and I imagine others, a lot happier with trying to understand how
the different bits and bobs fit together.

I will be dumping my notes as I don't want to get linked to stuff that can
be googled because I've already lost 10-15 hours to just that in the past
3-4 days. Digging it up in the compiler is hard because compiler behavior
that influences how IO actions are treated don't necessarily have "IO" or
"realWorld" mentioned in the relevant parts of the compiler, optimizations,
etc.

What I'm hoping for is answers on what specifically preserves the listed
properties we want from IO in the compiler, prims, or structure of how we
write IO actions.


What we expect IO to do:

- Disable sharing of results, even when it's not a lambda and is evaluated
multiple times by the same name. ie, getCurrentTime :: IO UTCTime   should
get evaluated more than once.

- Not reorder sequential IO actions, such as in a do-block. Called
"linearity" below

- Not duplicate the effects of IO actions. Effects shouldn't be spuriously
duplicated during optimization passes.

- Effects should not be discarded separately of the value returned by an IO
action, merged, or elided.


# Sharing

A friend suggested that perhaps one-shot semantics via the state hack for
State# in the IO type is responsible for disabling sharing, I don't believe
so, but here are my notes.

>-fno-state-hack
>Turn off the "state hack" whereby any lambda with a State# token as
argument is considered to be single-entry, hence it is considered OK to
inline things inside it. This can improve performance of IO and ST monad
code, but it runs the risk of reducing sharing.

>A one shot lambda
>State hack, makes the lambda over State# assume it's one-shot universally
by default.
>one-shot/state hack is an anti-inlining heuristic, suggesting that
inlining is costly.

Also I found this on Trac, does anyone know the answer to this? Is the
summary above accurate?

>Can the IO state hack be avoided if oneShot is used in the right places in
library code, e.g. in IO’s definition of >>=?

This seems related how the state token works, for differentiating which IO
action is which and how many times an IO action should run, when it should
run, etc.

>From the prims:

>data State# s

>State# is the primitive, unlifted type of states. It has one type
parameter, thus State# RealWorld, or State# s, where s is a type variable.
The only purpose of the type parameter is to keep different state threads
separate. It is represented by nothing at all.

>data RealWorld

>RealWorld is deeply magical. It is primitive, but it is not unlifted
(hence ptrArg). We never manipulate values of type RealWorld; it's only
used in the type system, to parameterise State#.


# Linearity

Is this from the nesting of lambdas? It doesn't seem like that's enough
based on the various examples using State/State# in GHC Trac bug tickets.
The RealWorld token seems to be what's driving this but precisely how that
works hasn't been easy to find.


# Discarding, not inlining effects

I believe these are addressed by has_side_effects in the prim ops. I could
very well be wrong.

can_fail has_side_effects
DiscardNONO
Float in   YES   YES
Float out  NONO
Duplicate  YES   NO

* Duplication.  You cannot duplicate a has_side_effect primop.  You
  might wonder how this can occur given the state token threading, but
  just look at Control.Monad.ST.Lazy.Imp.strictToLazy!  We get
  something like this
p = case readMutVar# s v of
  (# s', r #) -> (S# s', r)
s' = case p of (s', r) -> s'
r  = case p of (s', r) -> r

I believe duplication addresses inlining IO actions more generally but I
could be wrong. Here's a note I found regarding elision/merging:

  * Use the compiler flag @-fno-cse@ to prevent common sub-expression
elimination being performed on the module, which might combine
two side effects that were meant to be separate.  A good example
is using multiple global variables (like @test@ in the example
below).



Any help or pointers for nailing down and documenting this would be greatly
appreciated. Also if there's a more detailed explanation of what behavior
is expected out of each unsafe function, that would help as well. There are
bits and pieces I've been able to aggregate from the GHC trac tickets.


References used (not exhaustive):

- Referential Transparency; Haskell Wiki
https://wiki.haskell.org/Referential_transparency

- IO Inside; Haskell Wiki