from:"John Ericson"

Re: Reinstallable - base

2023-10-20 Thread John Ericson


On 10/20/23 04:00, Simon Peyton Jones wrote:


A very large proportion of libraries, and virtually all end-user
applications, transitively depend on Template Haskell. Whether
they use Template Haskell directly or not. So if we're saying
“base is reinstallable, except when you have Template Haskell
somewhere”, we're effectively saying “base is not reinstallable”.
Now, it could be a good stepping-stone, from an engineering
standpoint, but I don't think we could deliver this and be
satisfied that we've accomplished anything.


No one has yet answered my naive question (from 3 days ago) asking why 
Template Haskell stops base being reinstallable. I'll quote it here 
for completeness.


Let's say that

  * An old library mylib (which uses TH) depends on base-4.7.
  * A new GHC, say GHC 9.10, depends on a newer version of
base-4.9, which in turn depends on ghc-internal-9.10.
  * At the same time, though, we release base-4.7.1, which depends
on ghc-internal-9.10, and exposes the base-4.7 API.

At this point we use ghc-9.10 to compile L, against base-4.7.1.  
(Note that the ghc-9.10 binary includes a compiled form of
`base-4.9`.)

  * That produces compiled object files, such as, mylib:M.o.
  * To run TH we need to link them with the running binary
  * So we need to link the compiled `base-4.7.1` as well.  No
problem: it contains very little code; it is mostly a shim
for ghc-internal-9.10

So the only thing we need is the ability to have a single linked
binary that includes (the compiled form for) two different
versions/instantiations of `base`.   I think that's already
supported: each has a distinct "installed package id".

(End of quote)

What am I missing?

Simon


Simon I think you are right on the level of GHC itself: GHC can indeed 
cope with multiple versions of libraries just fine. However if we do 
this we run the risk of the user getting "base.x.y FooType is not the 
same as base.x.(y+1) FooType" errors, and I don't think that is good. 
cabal-install and stack current enforce build plans such that those 
sorts of errors are not possible, and I think that is a good thing we 
should not revisit at this time.


I don't think this should stop anything about reinstallable base, 
however. We just need to make a nice error when someone tries to splice 
in code using the wrong version of template haskell (or otherwise 
interact with GHC) using ABI hashes. Then we have the following situation:


1. Users that use stack/cabal-install plans only get one version of
   base, and if they have template-haskell using the the "wrong"
   version of base (identified via API hashes) they will get a nice error.
2. Users that "go rogue" and manually set up their project to mix base
   versions and avoid other errors will have their TH work as you
   describe./

So the thing Simon talks about is not /encouraged/, but it is a "free 
feature" we shouldn't go out of our way to prevent either.


John
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

2023-10-20 Thread John Ericson

I think it is making progress. It's not just an engineering milestone 
but also messaging. Right now a huge issue that every Haskeller faces is 
new GHCs having new major versions of base. If we make clear that "it's 
not base's fault, it's Template Haskell's fault", then we will have an 
easier time coordinating people and eventually fundraising to get issued 
fixed with Template Haskell.

John

On 10/20/23 03:56, Arnaud Spiwack wrote:
A very large proportion of libraries, and virtually all end-user 
applications, transitively depend on Template Haskell. Whether they 
use Template Haskell directly or not. So if we're saying “base is 
reinstallable, except when you have Template Haskell somewhere”, we're 
effectively saying “base is not reinstallable”. Now, it could be a 
good stepping-stone, from an engineering standpoint, but I don't think 
we could deliver this and be satisfied that we've accomplished anything.

On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus  wrote:

For what it worth, `template-haskell` itself depends on a `base`.
So if
`base` if different base is used, different `template-haskell` is
to be
used.

In my opinion is not *too unfair* to require that if you actually
splice
in (i.e. the code not only provides template-haskell combinators to
create/modify splices) then you must have base and template-haskell
versions aligned with host GHC used versions.

The same restriction is GHC plugins, isn't it, except
`template-haskell`
is replaced with `ghc`?

- Oleg

On 17.10.2023 18.54, Adam Gundry wrote:
> Hi Simon,
>
> Thanks for starting this discussion, it would be good to see
progress
> in this direction. As it happens I was discussing this question
with
> Ben and Matt over dinner last night, and unfortunately they
explained
> to me that it is more difficult than I naively hoped, even once
> wired-in and known-key things are moved to ghc-internal.
>
> The difficulty is that, as a normal Haskell library, ghc itself
will
> be compiled against a particular version of base. Then when
Template
> Haskell is used (with the internal interpreter), code will be
> dynamically loaded into a process that already has symbols for
ghc's
> version of base, which means it is not safe for the code to
depend on
> a different version of base. This is rather like the situation
with TH
> and cross-compilers.
>
> Adam
>
>
>
> On 17/10/2023 11:08, Simon Peyton Jones wrote:
>> Dear GHC devs
>>
>> Given the now-agreed split between ghc-internal and base
>> ,
what
>> stands in the way of a "reinstallable base"?
>>
>> Specifically, suppose that
>>
>>   * GHC 9.8 comes out with base-4.9
>>   * The CLC decides to make some change to `base`, so we get
base-4.10
>>   * Then GHC 9.10 comes out with base-4.10
>>
>> I think we'd all like it if someone could use GHC 9.10 to
compile a
>> library L that depends on base-4.9 and either L doesn't work at
all
>> with base-4.10, or L's dependency bounds have not yet been
adjusted
>> to allow base-4.10.
>>
>> We'd like to have a version of `base`, say `base-4.9.1` that
has the
>> exact same API as `base-4.9` but works with GHC 9.10.
>>
>> Today, GHC 9.10 comes with a specific version of base, /and you
can't
>> change it/. The original reason for that was, I recall, that GHC
>> knows the precise place where (say) the type Int is declared, and
>> it'll get very confused if that data type definition moves around.
>>
>> But now we have `ghc-internal`, all these "things that GHC
magically
>> knows" are in `ghc-internal`, not `base`.
>>
>> *Hence my question: what (now) stops us making `base` behave
like any
>> other library*?  That would be a big step forward, because it
would
>> mean that a newer GHC could compile old libraries against their
old
>> dependencies.
>>
>> (Some changes would still be difficult.  If, for example, we
removed
>> Monad and replaced it with classes Mo1 and Mo2, it might be
hard to
>> simulate the old `base` with a shim.  But getting 99% of the way
>> there would still be fantastic.)
>>
>> Simon
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

--
Arnaud Spiwack
Director, Research at https://moduscreate.com and https://tweag.io.

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs___
ghc-devs mailing list
ghc-devs@haskell.org

Re: Reinstallable - base

2023-10-17 Thread John Ericson

I think that's exactly right. Furthermore in 
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11400 the approach Ben has 
started going with is moving everything to ghc-internal to start, so this 
should be even easier --- surely we can reinstall an empty (except for 
reexports) library without the world falling down?! :)

I would love us to commit, day 0 of the split, to the principle that that base 
is now and forever more reinstallable. Then, as we move things back to base 
(untangling base's and ghc-internals's implementations), we'll be committing to 
not regressing on this. For example, if we want to move something back that GHC 
does in fact refer to, then we have to create the facility to make this not 
interfere with reinstallablilty before we move it.

(I would eventually like things like NonEmpty that GHC only barely knows about, 
and which are perfectly standard bits of Haskell, to live in base not 
ghc-internals. Agda's and Rust's putting annotations on things to let the 
compiler know it should use them, rather than baking in names/locations into 
the compiler, is very good prior art that this can work.)

John

On Tue, Oct 17, 2023, at 6:08 AM, Simon Peyton Jones wrote:
> Dear GHC devs
> 
> Given the now-agreed split between ghc-internal and base 
> , what stands in 
> the way of a "reinstallable base"?
> 
> Specifically, suppose that
>  • GHC 9.8 comes out with base-4.9
>  • The CLC decides to make some change to `base`, so we get base-4.10
>  • Then GHC 9.10 comes out with base-4.10
> I think we'd all like it if someone could use GHC 9.10 to compile a library L 
> that depends on base-4.9 and either L doesn't work at all with base-4.10, or 
> L's dependency bounds have not yet been adjusted to allow base-4.10.
> 
> We'd like to have a version of `base`, say `base-4.9.1` that has the exact 
> same API as `base-4.9` but works with GHC 9.10.
> 
> Today, GHC 9.10 comes with a specific version of base, *and you can't change 
> it*. The original reason for that was, I recall, that GHC knows the precise 
> place where (say) the type Int is declared, and it'll get very confused if 
> that data type definition moves around.
> 
> But now we have `ghc-internal`, all these "things that GHC magically knows" 
> are in `ghc-internal`, not `base`.
> 
> *Hence my question: what (now) stops us making `base` behave like any other 
> library*?  That would be a big step forward, because it would mean that a 
> newer GHC could compile old libraries against their old dependencies.
> 
> (Some changes would still be difficult.  If, for example, we removed Monad 
> and replaced it with classes Mo1 and Mo2, it might be hard to simulate the 
> old `base` with a shim.  But getting 99% of the way there would still be 
> fantastic.)
> 
> Simon
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: New implementation for `ImpredicativeTypes`

2021-09-07 Thread John Ericson



On 9/7/21 8:41 PM, Richard Eisenberg wrote:



On Sep 6, 2021, at 11:21 AM, John Ericson 
<mailto:john.ericson@obsidian.systems>> wrote:


On 9/2/21 11:04 PM, Richard Eisenberg wrote:

On Sep 2, 2021, at 2:56 PM, john.ericson 
<mailto:john.ericson@obsidian.systems>> wrote:


Does the most basic e.g.

newtype Some f where
  MkSome :: forall a. f a -> Some f

Have one of those problematic equalities?


No. That's not a GADT -- the constructor doesn't restrict anything 
about `f`.


Morally, sure, but GHC doesn't know about this.



Sure it does -- GHC doesn't include an equality constraint as one of 
the fields of MkSome. This isn't about extensions -- it's about the 
way the data constructor is interpreted.


Oops, agreed. I guess meant extensions didn't seem to check for this in 
a really syntax-driven way. But yes deciding the cases apart is not hard 
once the data definition is compiled.




I think you're after newtype existentials. I think these should 
indeed be possible, because what you propose appears to be the same as


newtype Some f = MkSome (exists a. f a)

We can probably support the syntax you wrote, too, but I don't want 
to commit to that right now.


The syntax I wrote is already basically valid?

data Some f = forall a. Some (f a)
data Some f where MkSome :: forall a f. f a -> Some f

Is accepted

newtype Some f = forall a. Some (f a)
newtype Some f where MkSome :: forall a f. f a -> Some f

Is not with "A newtype constructor cannot have existential type 
variables"


I propose we teach GHC how these "GADTs" in fact merely have 
existential variables, and not the FC constraints that require the 
extra evaluation for soundness. Than we can get the 
operational/runtime benefits of what you propose for cheap. Don't get 
me wrong -- the other aspects in the paper this doesn't address are 
still quite valuable, but I think this is a useful stepping stone / 
removal of artificial restrictions we should do first.


This sort of thing is brought up in #1965, where it is alleged this 
is in fact more difficult than it sounds. All more reason it is a 
good stepping stone, I say!




This is more difficult than it sounds. :) Newtypes are implemented via 
coercions in Core, and coercions are inherently bidirectional. The 
appearance of an existential in this way requires one-way conversions, 
which are currently not supported. So, to get what you want, we'd have 
to modify the core language as in the existentials paper, along with 
some extra work to automatically add `pack` and `open` -- rather 
similar to the type inference described in the existentials paper. The 
bottom line for me is that this seems just as hard as implementing the 
whole thing, so I see no value in having the stepping stone. If we 
always wrote out the newtype constructor, then maybe we could use its 
appearance to guide the `pack` and `open` but we don't: sometimes, we 
just use `coerce`. So I really don't think this is any easier than 
implementing the paper as written. Once that's done, we can come back 
and add this new feature relatively easily (I think).

Makes sense. That route is indeed harder than I was thinking.

There was a time before coercions when I gather newtypes were mostly 
implemented with a special-case lowering to STG. I imagine it would be 
quite easy to implement this that way, but that door is now mostly shut.


I had a few more thoughts, but I am not sure if I should trouble the 
list with them as they are still a bit rough around the edges. Suffice 
to say, I hope we could sort of "smooth out" the feature space a bit so 
that these and GADTs are not completely different, e.g. I think all 
existential variables can be projected like this (I opened #20337 for 
this), and perhaps connect GADTs to the existentials with evidence you 
mention in the "future work" section, so "true" GADTs also have 
structural counterparts in a similar way.


Happy to just continue mulling on it until a proposal or something, 
however. :)


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: New implementation for `ImpredicativeTypes`

2021-09-06 Thread John Ericson


On 9/2/21 11:04 PM, Richard Eisenberg wrote:

On Sep 2, 2021, at 2:56 PM, john.ericson 
mailto:john.ericson@obsidian.systems>> 
wrote:


Does the most basic e.g.

newtype Some f where
  MkSome :: forall a. f a -> Some f

Have one of those problematic equalities?


No. That's not a GADT -- the constructor doesn't restrict anything 
about `f`.


Morally, sure, but GHC doesn't know about this.

I tried, and -XGADTSyntax + -XExistenialTypes = -XGADTs it seems.



I think you're after newtype existentials. I think these should indeed 
be possible, because what you propose appears to be the same as


newtype Some f = MkSome (exists a. f a)

We can probably support the syntax you wrote, too, but I don't want to 
commit to that right now.


The syntax I wrote is already basically valid?

data Some f = forall a. Some (f a)

data Some f where MkSome :: forall a f. f a -> Some f

Is accepted

newtype Some f = forall a. Some (f a)

newtype Some f where MkSome :: forall a f. f a -> Some f

Is not with "A newtype constructor cannot have existential type variables"

I propose we teach GHC how these "GADTs" in fact merely have existential 
variables, and not the FC constraints that require the extra evaluation 
for soundness. Than we can get the operational/runtime benefits of what 
you propose for cheap. Don't get me wrong -- the other aspects in the 
paper this doesn't address are still quite valuable, but I think this is 
a useful stepping stone / removal of artificial restrictions we should 
do first.


This sort of thing is brought up in #1965, where it is alleged this is 
in fact more difficult than it sounds. All more reason it is a good 
stepping stone, I say!


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is simplified subsumption really necessary?

2021-06-20 Thread John Ericson

I'm sorry, I misunderstood the paper and thought the depth of the 
instantiation and subsumption could be varied more independently.


That said, what about the seq example below? Does forcing any function 
that is eta expanded like that sketchy to you? There is still a runtime 
cost to the eta expansion, but think with more elbow grease that could 
also be addressed (post-type-erasure optimization or new coercions).


John

On 6/18/21 3:56 PM, Simon Peyton Jones wrote:


Richard’s paper argues for lazy rather than eager instantiation.

It does *not* argue for deep rather than shallow subsumption and 
instantiation; on the contrary, it argues for shallow.  (That is, for 
“simple subsumption”.)   And it is simple subsumption that is the 
focus of this conversation.


Simon

*From:*John Ericson 
*Sent:* 18 June 2021 16:56
*To:* ghc-devs 
*Cc:* Simon Peyton Jones 
*Subject:* Re: Is simplified subsumption really necessary?

On 6/16/21 12:00 PM, Simon Peyton Jones via ghc-devs wrote:

 I’m sorry to hear that Chris.   It’s exactly backwards from what
I would expect – the typing rules with simple subsumption are,
well, simpler than those for complicated subsumption, and so one
might hope that your intuition had fewer complexities to grapple
with.

In https://richarde.dev/papers/2021/stability/stability.pdf 
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fricharde.dev%2Fpapers%2F2021%2Fstability%2Fstability.pdf=04%7C01%7Csimonpj%40microsoft.com%7C7655b09d06a54a4af03508d9327193cd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637596286139778988%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=R1quXKjsnEjusvX%2BSxhPT25t%2B%2Bwqwo7mTPhnulvctQ0%3D=0> 
it is written


The analysis around stability in this paper strongly suggests that
GHC should use the lazy, shallow approach to instantiation. Yet
the struggles with lazy instantiation above remain. In order to
simplify the implementation, GHC has recently (for GHC 9.0)
switched to use exclusively eager instantiation.This choice
sacrifices stability for convenience in implementation.

I think the principles outlined in the paper are very good, and 
explain the queasiness some users may feel in 9.0


But wouldn't it be possible to choose a desugaring with seq that
doesn't do so?

I just don’t know how to do that.  Maybe someone else does.

Is it not

  f `seq` \x -> f x

and similar? I haven't thought about the issue in a while or in very 
much depth, but when I first discussed the proposal years back with 
some other people at work, they spit-balled the same counter-proposal.




Having little "skin in the game" as I haven't yet ported any serious 
programs over to 9.0, I suppose I am glad the experimentation with 
QuickLook is happening, and OK that our accepting on-par fewer 
programs now opens design space for later (i.e. we got the breakage 
out of the way.) But I certainly think there are improvements in the 
spirit outlined in Richard's paper to be done down the road.


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Is simplified subsumption really necessary?

2021-06-18 Thread John Ericson


On 6/16/21 12:00 PM, Simon Peyton Jones via ghc-devs wrote:

I’m sorry to hear that Chris. It’s exactly backwards from what I would 
expect – the typing rules with simple subsumption are, well, simpler 
than those for complicated subsumption, and so one might hope that 
your intuition had fewer complexities to grapple with.



In https://richarde.dev/papers/2021/stability/stability.pdf it is written

   The analysis around stability in this paper strongly suggests that
   GHC should use the lazy, shallow approach to instantiation. Yet the
   struggles with lazy instantiation above remain. In order to simplify
   the implementation, GHC has recently (for GHC 9.0) switched to use
   exclusively eager instantiation.This choice sacrifices stability for
   convenience in implementation.

I think the principles outlined in the paper are very good, and explain 
the queasiness some users may feel in 9.0


But wouldn't it be possible to choose a desugaring with seq that 
doesn't do so?


I just don’t know how to do that.  Maybe someone else does.


Is it not

  f `seq` \x -> f x

and similar? I haven't thought about the issue in a while or in very 
much depth, but when I first discussed the proposal years back with some 
other people at work, they spit-balled the same counter-proposal.




Having little "skin in the game" as I haven't yet ported any serious 
programs over to 9.0, I suppose I am glad the experimentation with 
QuickLook is happening, and OK that our accepting on-par fewer programs 
now opens design space for later (i.e. we got the breakage out of the 
way.) But I certainly think there are improvements in the spirit 
outlined in Richard's paper to be done down the road.


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GHC and the future of Freenode

2021-06-07 Thread John Ericson

And this is a good example of how both using libera.chat and having an 
official matrix bridge with a nice name can help more people use IRC too!


On 6/6/21 2:10 PM, Brandon Allbery wrote:
Pasting directly into the channel is generally a no-no on IRC. Things 
like Matrix or IRCCloud convert to pastebins automatically.


On Sun, Jun 6, 2021 at 2:07 PM Viktor Dukhovni > wrote:


On 19 May 2021, at 11:48 am, Carter Schonwald
mailto:carter.schonw...@gmail.com>>
wrote:

> I personally vote for irc.  Perhaps via Libera.

Perhaps I'm too much of an IRC noob, but I still found it it rather
surprising to be banned from libera.chat (my IP is blacklisted) for
pasting a 25-line build script for building GHC via hadrian on FreeBSD
into the #ghc channel.

This was in response to a discussion about issues with the bindist,
how the port is built, ... and while perhaps I'm expected to use a
paste bin, the abrupt ban was rather a harsh response.

The ban appears to have been "temporary", an hour or so later I am
able to reconnect, but this does not leave a good impression.

-- 
        Viktor.

___
ghc-devs mailing list
ghc-devs@haskell.org 
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs




--
brandon s allbery kf8nh
allber...@gmail.com 

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GHC and the future of Freenode

2021-06-03 Thread John Ericson

The libera.chat IRC bridge is up, so one can go to #ghc:libera.chat to
give the bridging a spin.

To make my recommendation from before more concrete, we would *not* use
that "portal room" (one spawned on demand by the bridge), but instead
make a #ghc:haskell.org. One should be able to then "tombstone" the
portal room telling people to go to the manually-bridged instead, which
can preserve history across bridging changes like e.g. libera.chat and
Freenode are reunited later, adding slack/discord bridges, etc.

John

On 6/3/21 3:52 AM, Matthew Pickering wrote:

I have been trying out Matrix a bit recently. It seems the best of the
options in my opinion and has the advantage of being able to bridge to
IRC (and other platforms).

The NixOS community rapidly moved over without any ill effects after
the demise of freenode.

As with all these things, who feels willing to make a decision for us?

Cheers,

Matt

On Sat, May 22, 2021 at 7:52 PM John Ericson
wrote:

As Ben and others say, Matrix provides many modern features new users will
expect, while preserving the spirit of IRC. Without wading into the details,
the design of Matrix I find impressive and to my liking, and it has seemed to
get steadily better over time for quite a while now.

Re Zulip, in https://news.ycombinator.com/item?id=27202838 one of the lead
Matrix devs says their up-and-coming threading model aims to support what Zulip
does and they've been discussing deeper integration with Zulip. Granted, It
would be better to hear about those discussions from the Zulip side as Matrix
aims to assimilate everything and Zulip could have some reservations, but I
remain hopeful. (I certainly would like to see culled the current explosion of
mutually-incompatible chat applications, leaving us with fewer protocols but as
many competing implementations.)

What I recommend for now that we make some official Matrix channels, but also bridge them with the
libera.chat ones once the bridge is up (should be a few days). Creating a matrix room and bridging
it is a bit different underneath the hood than using a channel generated by the bridge on demand.
We can give them nice names on the matrix side, and basically keep both options open of being
"IRC-first" or "Matrix-first" down the road.

For reference, see https://matrix.to/#/#community:nixos.org?via=nixos.org which is the Matrix
"Space" (room that is a directory of sub-rooms, filling the role of a Discord
"server") that Nix community created while they debate what to do next. See also
https://github.com/NixOS/rfcs/pull/94 where this same discussion is playing out.

John

On 5/21/21 4:00 PM, Iavor Diatchki wrote:

As I said, I am not a heavy IRC user, for my online chatting needs I mostly use
Mattermost, Discord, and Slack.So I don't have an informed opinion on the
technical merits of the various platforms---mostly I've heard that the Matrix
clients and servers are quite a bit less robust than IRC ones but I've never
personally used them.

If there is a feeling that GHC wants to use a new chatting platform, by all means we should try it
out. I just don't think that the unfortunate situation with free-node is a good reason to drop IRC
entirely. Despite its flows, I think it has served our community well, and while it may look
"old" to "young" users it does have the benefit of being pretty stable, unlike
the myriad of chatting services that seem to be popping up all the time.

-Iavor

On Fri, May 21, 2021 at 10:41 AM Ben Gamari wrote:

Iavor Diatchki writes:

Hello,

I am not a heavy IRC user, but I'd say it makes most sense to just use
Libera. It is essentially the same people that were running free-node
running pretty much the exact same service, and I believe they are trying
to make it extra easy to just switch, so this should be the least effort
transition.

I believe IRC has served the GHC community quite well so far, and there is
a reddit post by Ed Kmett that the normal Haskell channels have already
been transitioned over, so I think it makes sense for GHC to stick with the
rest of the Haskell community.

The problem is that, in order to grow (or even merely not to shrink),
the community also needs to adapt to the preferences of younger users.

The fact of the matter is the younger users tend to be, at best,
unfamiliar with IRC. In the worst case, the need to leave a browser/sign
up for a new account means that they simply won't participate. Of the
new contributors I have had approach me in the past year, less than half
have had any familiarity with IRC.

Matrix has the advantage of being accessible to "web-native" community
members while being open enough to (at least in principle) allow
community members who are accustomed to IRC to continue to participate
via a bridge.

Cheers,

- Ben

___
ghc-devs m

Re: GHC and the future of Freenode

2021-05-22 Thread John Ericson

As Ben and others say, Matrix provides many modern features new users 
will expect, while preserving the spirit of IRC. Without wading into the 
details, the design of Matrix I find impressive and to my liking, and it 
has seemed to get steadily better over time for quite a while now.

Re Zulip, in https://news.ycombinator.com/item?id=27202838 one of the 
lead Matrix devs says their up-and-coming threading model aims to 
support what Zulip does and they've been discussing deeper integration 
with Zulip. Granted, It would be better to hear about those discussions 
from the Zulip side as Matrix aims to assimilate everything and Zulip 
could have some reservations, but I remain hopeful. (I certainly would 
like to see culled the current explosion of mutually-incompatible chat 
applications, leaving us with fewer protocols but as many competing 
implementations.)

What I recommend for now that we make some official Matrix channels, but 
also bridge them with the libera.chat ones once the bridge is up (should 
be a few days). Creating a matrix room and bridging it is a bit 
different underneath the hood than using a channel generated by the 
bridge on demand. We can give them nice names on the matrix side, and 
basically keep both options open of being "IRC-first" or "Matrix-first" 
down the road.

For reference, see 
https://matrix.to/#/#community:nixos.org?via=nixos.org which is the 
Matrix "Space" (room that is a directory of sub-rooms, filling the role 
of a Discord "server") that Nix community created while they debate what 
to do next. See also https://github.com/NixOS/rfcs/pull/94 where this 
same discussion is playing out.

John

On 5/21/21 4:00 PM, Iavor Diatchki wrote:
As I said, I am not a heavy IRC user, for my online chatting needs I 
mostly use Mattermost, Discord, and Slack. So I don't have an informed 
opinion on the technical merits of the various platforms---mostly I've 
heard that the Matrix clients and servers are quite a bit less robust 
than IRC ones but I've never personally used them.

If there is a feeling that GHC wants to use a new chatting platform, 
by all means we should try it out.  I just don't think that the 
unfortunate situation with free-node is a good reason to drop IRC 
entirely.   Despite its flows, I think it has served our community 
well, and while it may look "old" to "young" users it does have the 
benefit of being pretty stable, unlike the myriad of chatting services 
that seem to be popping up all the time.

-Iavor

On Fri, May 21, 2021 at 10:41 AM Ben Gamari > wrote:

Iavor Diatchki mailto:iavor.diatc...@gmail.com>> writes:

> Hello,
>
> I am not a heavy IRC user, but I'd say it makes most sense to
just use
> Libera.  It is essentially the same people that were running
free-node
> running pretty much the exact same service, and I believe they
are trying
> to make it extra easy to just switch, so this should be the
least effort
> transition.
>
> I believe IRC has served the GHC community quite well so far,
and there is
> a reddit post by Ed Kmett that the normal Haskell channels have
already
> been transitioned over, so I think it makes sense for GHC to
stick with the
> rest of the Haskell community.
>
The problem is that, in order to grow (or even merely not to shrink),
the community also needs to adapt to the preferences of younger users.

The fact of the matter is the younger users tend to be, at best,
unfamiliar with IRC. In the worst case, the need to leave a
browser/sign
up for a new account means that they simply won't participate. Of the
new contributors I have had approach me in the past year, less
than half
have had any familiarity with IRC.

Matrix has the advantage of being accessible to "web-native" community
members while being open enough to (at least in principle) allow
community members who are accustomed to IRC to continue to participate
via a bridge.

Cheers,

- Ben

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: structured error messages

2021-05-03 Thread John Ericson



Ultimately, if reading error messages did require linking with GHC, it 
wouldn't be the end of the world.  I would just write my own 
translation layer to JSON.  So consider this a weak preference.


I think this is the key. Ultimately, all other uses can be done on top 
of just linking GHC the library. Just as we have ghc-lib to paper over 
the differences between GHC versions and save individual projects from 
developing their own comparability layers, so we could also have some 
ghc-json library and tweaked frontend executable or something.


The GHC repo has stringent CI and is somewhat a bottleneck already, so 
the more we can do out-of-band, the better, I say.


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: External STG Interpreter blog post

2021-04-27 Thread John Ericson

Hi Csaba,

Thanks for your interest. I'm not sure whether it would be better to do
a call sooner or later, but certainly want to get to this at some point.
Fendor and Sylvain (and occasionally I too) have been chipping away at
multiple home packages, which I hope could help with these sorts of
whole program things. I'm sort of thinking milestones like:

1. Proper Int64 => Single sdist for GHC the library that is truely
target/configures-script-choice agnostic, but sdist contains vendored
generated code

2. Multiple home packages => Maybe easier to rig up what you have with
GHC's driver, though I suppose not needed for Hadrian since it uses
one-shot mode.

3. Extensible interface directory => Can better upstream your work to
save STG for later, needed for Hadrian / one-shot mode.

4. Matthew's proposal => can get whole program closure for just for TH,
pruning away runtime stuff.

5. Put it all together => single sdist that is much more minimally
processed, with TH instead of including generated code (no vendoring or
Setup.hs evils necessary). Easier to work on IDE stuff with either sdist
or fresh checkout of GHC.

So it's quite a while before GHC is ready to fully leverage your work on
the TH front, I think, but we could still talk through things in advance
of that if people are interested.

Also, included Sylvain Henry in the thread who's now back from paternity
leave,

John

On 4/20/21 5:38 PM, Csaba Hruska wrote:

Hello,

In theory the ext-stg interpereter could be used to run TH, but it is
not done yet.
I'm interested in optimization much much more, but I deliberately
designed the ext-stg ir/tooling to be simple and easy to use and
understand for others so that an average Haskell programmer could
implement his ideas or do his experiment.
I'm also open for discussion about the possibilities. We can have a
video meeting about this if you'd like to.

Cheers,
Csaba

On Mon, Apr 12, 2021 at 7:57 PM John Ericson
wrote:

Indeed it is!

I also am very much hoping this can be the missing piece we need
to allow Template Haskell to work with a stage 0 GHC, so that we
can safely use TH in GHC itself even when the ABI is changing. See
https://github.com/ghc-proposals/ghc-proposals/issues/162
<https://github.com/ghc-proposals/ghc-proposals/issues/162> for
details, where I've also linked Csaba's blog post. (And maybe also
https://gitlab.haskell.org/ghc/ghc/-/wikis/cross-compilation/roadmap
<https://gitlab.haskell.org/ghc/ghc/-/wikis/cross-compilation/roadmap>
for more background, and related things like
https://github.com/ghc-proposals/ghc-proposals/pull/412
<https://github.com/ghc-proposals/ghc-proposals/pull/412>.)

TH could perhaps be used to replace some of the ad-hoc code gen we
have for e.g. RTS constants and primops. This in turn would
simplify the build system, allowing for `cabal install ghc` to
work without shipping generated Haskell code in the sdist.

John

On 4/12/21 4:51 AM, Simon Peyton Jones via ghc-devs wrote:

That’s extremely impressive, Csaba.

Maybe of interest to folk who have been thinking about “fat
interface files”.

Simon

*From:*ghc-devs
<mailto:ghc-devs-boun...@haskell.org> *On Behalf Of *Csaba Hruska
*Sent:* 10 April 2021 11:49
*To:* GHC developers
<mailto:ghc-devs@haskell.org>
*Subject:* External STG Interpreter blog post

Hello,

I've written a blog post about GHC-WPC's external stg interpreter.
https://www.patreon.com/posts/external-stg-49857800

<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.patreon.com%2Fposts%2Fexternal-stg-49857800=04%7C01%7Csimonpj%40microsoft.com%7C1535f88608f549dabb8108d8fc0e4c3f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637536486264337310%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000=VXa%2FhDs1ePRS5NTu04buv42yj%2BFKw%2FJzBLQJSwp8qTs%3D=0>

Feedback is welcome.

Cheers,

Csaba

___
ghc-devs mailing list
ghc-devs@haskell.org <mailto:ghc-devs@haskell.org>
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
<http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs>

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: External STG Interpreter blog post

2021-04-12 Thread John Ericson


Indeed it is!

I also am very much hoping this can be the missing piece we need to 
allow Template Haskell to work with a stage 0 GHC, so that we can safely 
use TH in GHC itself even when the ABI is changing. See 
https://github.com/ghc-proposals/ghc-proposals/issues/162 for details, 
where I've also linked Csaba's blog post. (And maybe also 
https://gitlab.haskell.org/ghc/ghc/-/wikis/cross-compilation/roadmap for 
more background, and related things like 
https://github.com/ghc-proposals/ghc-proposals/pull/412.)


TH could perhaps be used to replace some of the ad-hoc code gen we have 
for e.g. RTS constants and primops. This in turn would simplify the 
build system, allowing for `cabal install ghc` to work without shipping 
generated Haskell code in the sdist.


John

On 4/12/21 4:51 AM, Simon Peyton Jones via ghc-devs wrote:


That’s extremely impressive, Csaba.

Maybe of interest to folk who have been thinking about “fat interface 
files”.



Simon

*From:*ghc-devs  *On Behalf Of *Csaba Hruska
*Sent:* 10 April 2021 11:49
*To:* GHC developers 
*Subject:* External STG Interpreter blog post

Hello,

I've written a blog post about GHC-WPC's external stg interpreter.
https://www.patreon.com/posts/external-stg-49857800 



Feedback is welcome.

Cheers,

Csaba


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Why TcLclEnv and DsGblEnv need to store the same IORef for errors?

2021-04-01 Thread John Ericson

Yeah good point. Ultimately, I hope we can abstract over things like the 
IORef itself so that while TcM can kick off DsM and vice-versa, each 
monad can only log messages of the right type, but that can come later.


Good luck, very excited to see this work happen!

John

On 4/1/21 2:00 AM, Alfredo Di Napoli wrote:

Hello all,

John: right, I am not opposed to what you describe, but at the end of 
the day we need to add all these messages to a single IORef (unless we 
go with the two IORef idea that Richard is not fond of), and for that 
we need a single monomorphic type, which could be, initially, even 
something like:


type TcRnDsMessage = Either DsMessage TcRnMessage

I guess I'll have to iterate on this until we get something meaningful 
and that passes the full testsuite :)


A.



On Wed, 31 Mar 2021 at 16:36, John Ericson 
 wrote:


I might still be tempted to do:

data DsMessage =
    ...
  | DsLiftedTcRnMessage !TcRnMessage
  -- ^ A diagnostic coming straight from the Typecheck-renamer.

data TcRnMessage =
    ...
  | TcRnLiftedDsMessage !DsMessage
  -- ^ A diagnostic coming straight from the Desugarer.

tying them together with hs-boot. Yes, that means one can do some
silly `TcRnLiftedDsMessage . DsLiftedTcRnMessage .
TcRnLiftedDsMessage ...`, but that could even show up in a render
as "while desugaring a splice during type checking, while
typechecking during desguaring, ..." so arguably the information
the wrapping isn't purely superfluous.

I think this would pose no practical problem today, while still
"soft enforcing" the abstraction boundaries we want.

On 3/31/21 3:45 AM, Alfredo Di Napoli wrote:

Follow up:

Argh! I have just seen that I have a bunch of test failures
related to my MR (which, needless to say, it's still WIP).

For example:

run/T9140.run.stdout.normalised 2021-03-31 09:35:48.0 +0200
@@ -1,12 +1,4 @@
-:2:5:
-    You can't mix polymorphic and unlifted bindings: a = (# 1 #)
-    Probable fix: add a type signature
-
-:3:5:
-    You can't mix polymorphic and unlifted bindings: a = (# 1, 3 #)
-    Probable fix: add a type signature
-

So it looks like some diagnostic is now not being reported and,
surprise surprise, this was emitted from the DsM monad.

I have the suspect that indeed Richard was right (like he always
is :) ) -- when we go from a DsM to a TcM monad (See `initDsTc`)
for example, I think we also need to carry into the new monad all
the diagnostics we collected so far.

This implies indeed a mutual dependency (as Simon pointed out, heh).


So I think my cunning plan of embedding is crumbling -- I suspect
we would end up with a type `TcRnDsMessage` which captures the
dependency.

Sorry for not seeing it sooner!








On Wed, 31 Mar 2021 at 08:05, Alfredo Di Napoli
mailto:alfredo.dinap...@gmail.com>>
wrote:

Morning all,

*Richard*: sorry! Unfortunately MR !4798 is the cornerstone
of this refactoring work but it's also gargantuan. Let's
discuss a plan to attack it, but fundamentally there is a
critical mass of changes that needs to happen atomically or
it wouldn't make much sense, and alas this doesn't play in
our favour when it comes to MR size and ease of review.
However, to quickly reply to your remak: currently (for the
sake of the "minimum-viable-product") I am trying to
stabilise the external interfaces, by which I mean giving
functions their final type signature while I do what's
easiest to make things typecheck. In this phase what I think
is the easiest is to wrap the majority of diagnostics into
the `xxUnknownxx` constructor, and change them gradually
later. A fair warning, though: you say "I would think that a
DsMessage would later be wrapped in an envelope." This might
be true for Ds messages (didn't actually invest any brain
cycles to check that) but in general we have to turn a
message into an envelope as soon as we have a chance to do
so, because we need to grab the `SrcSpan` and the `DynFlags`
*at the point of creation* of the diagnostics. Carrying
around a message and make it bubble up at some random point
won't be a good plan (even for Ds messages). Having said
that, I clearly have very little knowledge about this area of
GHC, so feel free to disagree :)

*John*: Although it's a bit hard to predict how well this is
going to evolve, my current embedding, to refresh everyone's
memory, is the following:

data DsMessage =

  DsUnknownMessage !DiagnosticMessage

-- ^ Stop-gap constructor to ease the migration.

| DsLiftedTcRnMessage !TcRnMessage

-- ^ A

Re: Why TcLclEnv and DsGblEnv need to store the same IORef for errors?

2021-03-31 Thread John Ericson


I might still be tempted to do:

data DsMessage =
    ...
  | DsLiftedTcRnMessage !TcRnMessage
  -- ^ A diagnostic coming straight from the Typecheck-renamer.

data TcRnMessage =
    ...
  | TcRnLiftedDsMessage !DsMessage
  -- ^ A diagnostic coming straight from the Desugarer.

tying them together with hs-boot. Yes, that means one can do some silly 
`TcRnLiftedDsMessage . DsLiftedTcRnMessage . TcRnLiftedDsMessage ...`, 
but that could even show up in a render as "while desugaring a splice 
during type checking, while typechecking during desguaring, ..." so 
arguably the information the wrapping isn't purely superfluous.


I think this would pose no practical problem today, while still "soft 
enforcing" the abstraction boundaries we want.


On 3/31/21 3:45 AM, Alfredo Di Napoli wrote:

Follow up:

Argh! I have just seen that I have a bunch of test failures related to 
my MR (which, needless to say, it's still WIP).


For example:

run/T9140.run.stdout.normalised 2021-03-31 09:35:48.0 +0200
@@ -1,12 +1,4 @@
-:2:5:
-    You can't mix polymorphic and unlifted bindings: a = (# 1 #)
-    Probable fix: add a type signature
-
-:3:5:
-    You can't mix polymorphic and unlifted bindings: a = (# 1, 3 #)
-    Probable fix: add a type signature
-

So it looks like some diagnostic is now not being reported and, 
surprise surprise, this was emitted from the DsM monad.


I have the suspect that indeed Richard was right (like he always is :) 
) -- when we go from a DsM to a TcM monad (See `initDsTc`) for 
example, I think we also need to carry into the new monad all the 
diagnostics we collected so far.


This implies indeed a mutual dependency (as Simon pointed out, heh).


So I think my cunning plan of embedding is crumbling -- I suspect we 
would end up with a type `TcRnDsMessage` which captures the dependency.


Sorry for not seeing it sooner!








On Wed, 31 Mar 2021 at 08:05, Alfredo Di Napoli 
mailto:alfredo.dinap...@gmail.com>> wrote:


Morning all,

*Richard*: sorry! Unfortunately MR !4798 is the cornerstone of
this refactoring work but it's also gargantuan. Let's discuss a
plan to attack it, but fundamentally there is a critical mass of
changes that needs to happen atomically or it wouldn't make much
sense, and alas this doesn't play in our favour when it comes to
MR size and ease of review. However, to quickly reply to your
remak: currently (for the sake of the "minimum-viable-product") I
am trying to stabilise the external interfaces, by which I mean
giving functions their final type signature while I do what's
easiest to make things typecheck. In this phase what I think is
the easiest is to wrap the majority of diagnostics into the
`xxUnknownxx` constructor, and change them gradually later. A fair
warning, though: you say "I would think that a DsMessage would
later be wrapped in an envelope." This might be true for Ds
messages (didn't actually invest any brain cycles to check that)
but in general we have to turn a message into an envelope as soon
as we have a chance to do so, because we need to grab the
`SrcSpan` and the `DynFlags` *at the point of creation* of the
diagnostics. Carrying around a message and make it bubble up at
some random point won't be a good plan (even for Ds messages).
Having said that, I clearly have very little knowledge about this
area of GHC, so feel free to disagree :)

*John*: Although it's a bit hard to predict how well this is going
to evolve, my current embedding, to refresh everyone's memory, is
the following:

data DsMessage =

  DsUnknownMessage !DiagnosticMessage

-- ^ Stop-gap constructor to ease the migration.

| DsLiftedTcRnMessage !TcRnMessage

-- ^ A diagnostic coming straight from the Typecheck-renamer.

-- More messages added in the future, of course


At first I thought this was the wrong way around, due to Simon's
comment, but this actually creates pleasant external interfaces.
To give you a bunch of examples from MR !4798:


deSugar :: HscEnv -> ModLocation -> TcGblEnv -> IO (Messages
DsMessage, Maybe ModGuts)

deSugarExpr :: HscEnv -> LHsExpr GhcTc -> IO (Messages DsMessage,
Maybe CoreExpr)

Note something interesting: the second function actually calls
`runTcInteractive` inside the body, but thanks to the
`DsLiftedTcRnMessage` we can still expose to the consumer an
opaque `DsMessage` , which is what I would expect to see from a
function called "deSugarExpr". Conversely, I would be puzzled to
find those functions returning a `TcRnDsMessage`.


Having said all of that, I am not advocating this design is "the
best". I am sure we will iterate on it. I am just reporting that
even this baseline seems to be decent from an API perspective :)


On Wed, 31 Mar 2021 at 05:45, John Ericson

Re: Why TcLclEnv and DsGblEnv need to store the same IORef for errors?

2021-03-30 Thread John Ericson

Alfredo also replied to this pointing his embedding plan. I also prefer
that, because I really wish TH didn't smear together the phases so much.
Moreover, I hope with

- GHC proposals
https://github.com/ghc-proposals/ghc-proposals/pull/412 /
https://github.com/ghc-proposals/ghc-proposals/pull/243

- The parallelism work currently be planned in
https://gitlab.haskell.org/ghc/ghc/-/wikis/Plan-for-increased-parallelism-and-more-detailed-intermediate-output

we might actually have an opportunity/extra motivation to do that.
Splices and quotes will still induce intricate inter-phase dependencies,
but I hope that could be mediated by the driver rather than just baked
into each phase.

(One final step would be the "stuck macros" technique of
https://www.youtube.com/watch?v=nUvKoG_V_U0 /
https://github.com/gelisam/klister, where TH splices would be able to
making "blocking queries" of the the compiler in ways that induce more
of these fine-grained dependencies.)

Anyways, while we could also do a "RnTsDsError" and split later, I hope
Alfredo's alternative of embedding won't be too much harder and prepare
us for these exciting areas of exploration.

John

On 3/30/21 10:14 AM, Richard Eisenberg wrote:

On Mar 30, 2021, at 4:57 AM, Alfredo Di Napoli
mailto:alfredo.dinap...@gmail.com>> wrote:

I'll explore the idea of adding a second IORef.

Renaming/type-checking is already mutually recursive. (The renamer
must call the type-checker in order to rename -- that is, evaluate --
untyped splices. I actually can't recall why the type-checker needs to
call the renamer.) So we will have a TcRnError. Now we see that the
desugarer ends up mixed in, too. We could proceed how Alfredo
suggests, by adding a second IORef. Or we could just make TcRnDsError
(maybe renaming that).

What's the disadvantage? Clients will have to potentially know about
all the different error forms with either approach (that is, using my
combined type or using multiple IORefs). The big advantage to
separating is maybe module dependencies? But my guess is that the
dependencies won't be an issue here, due to the fact that these
components are already leaning on each other. Maybe the advantage is
just in having smaller types? Maybe.

I don't have a great sense as to what to do here, but I would want a
clear reason that e.g. the TcRn monad would have two IORefs, while
other monads will work with GhcMessage (instead of a whole bunch of
IORefs).

Richard

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: On CI

2021-03-18 Thread John Ericson

My guess is most of the "noise" is not run time, but the compiled code 
changing in hard to predict ways.

https://gitlab.haskell.org/ghc/ghc/-/merge_requests/1776/diffs for 
example was a very small PR that took *months* of on-off work to get 
passing metrics tests. In the end, binding `is_boot` twice helped a bit, 
and dumb luck helped a little bit more. No matter how you analyze that, 
that's a lot of pain for what's manifestly a performance-irrelevant MR 
--- no one is writing 10,000 default methods or whatever could possibly 
make this the micro-optimizing worth it!

Perhaps this is an extreme example, but my rough sense is that it's not 
an isolated outlier.

John

On 3/18/21 1:39 PM, davean wrote:
I left the wiggle room for things like longer wall time causing more 
time events in the IO Manager/RTS which can be a thermal/HW issue.

They're small and indirect though

-davean

On Thu, Mar 18, 2021 at 1:37 PM Sebastian Graf > wrote:

To be clear: All performance tests that run as part of CI measure
allocations only. No wall clock time.
Those measurements are (mostly) deterministic and reproducible
between compiles of the same worktree and not impacted by thermal
issues/hardware at all.

Am Do., 18. März 2021 um 18:09 Uhr schrieb davean mailto:dav...@xkcd.com>>:

That really shouldn't be near system noise for a well
constructed performance test. You might be seeing things like
thermal issues, etc though - good benchmarking is a serious
subject.
Also we're not talking wall clock tests, we're talking
specific metrics. The machines do tend to be bare metal, but
many of these are entirely CPU performance independent, memory
timing independent, etc. Well not quite but that's a longer
discussion.

The investigation of Haskell code performance is a very good
thing to do BTW, but you'd still want to avoid regressions in
the improvements you made. How well we can do that and the
cost of it is the primary issue here.

-davean

On Wed, Mar 17, 2021 at 6:22 PM Karel Gardas
mailto:karel.gar...@centrum.cz>> wrote:

On 3/17/21 4:16 PM, Andreas Klebinger wrote:
> Now that isn't really an issue anyway I think. The
question is rather is
> 2% a large enough regression to worry about? 5%? 10%?

5-10% is still around system noise even on lightly loaded
workstation.
Not sure if CI is not run on some shared cloud resources
where it may be
even higher.

I've done simple experiment of pining ghc compiling
ghc-cabal and I've
been able to "speed" it up by 5-10% on W-2265.

Also following this CI/performance regs discussion I'm not
entirely sure
if  this is not just a witch-hunt hurting/beating mostly
most active GHC
developers. Another idea may be to give up on CI doing
perf reg testing
at all and invest saved resources into proper investigation of
GHC/Haskell programs performance. Not sure, if this would
not be more
beneficial longer term.

Just one random number thrown to the ring. Linux's perf
claims that
nearly every second L3 cache access on the example above
ends with cache
miss. Is it a good number or bad number? See stats below
(perf stat -d
on ghc with +RTS -T -s -RTS').

Good luck to anybody working on that!

Karel

Linking utils/ghc-cabal/dist/build/tmp/ghc-cabal ...
  61,020,836,136 bytes allocated in the heap
   5,229,185,608 bytes copied during GC
     301,742,768 bytes maximum residency (19 sample(s))
       3,533,000 bytes maximum slop
             840 MiB total memory in use (0 MB lost due to
fragmentation)

                                     Tot time (elapsed) 
Avg pause  Max
pause
  Gen  0      2012 colls,     0 par    5.725s  5.731s   
 0.0028s
0.1267s
  Gen  1        19 colls,     0 par    1.695s  1.696s   
 0.0893s
0.2636s

  TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0
fizzled)

  INIT    time    0.000s  (  0.000s elapsed)
  MUT     time   27.849s  ( 32.163s elapsed)
  GC      time    7.419s  (  7.427s elapsed)
  EXIT    time    0.000s  (  0.010s elapsed)
  Total   time   35.269s  ( 39.601s elapsed)

  Alloc rate    2,191,122,004 bytes per MUT second

  Productivity  79.0% of total

Re: On CI

2021-03-17 Thread John Ericson

Yes, I think the counter point of "automating what Ben does" so people 
besides Ben can do it is very important. In this case, I think a good 
thing we could do is asynchronously build more of master post-merge, 
such as use the perf stats to automatically bisect anything that is 
fishy, including within marge bot roll-ups which wouldn't be built by 
the regular workflow anyways.


I also agree with Sebastian that the overfit/overly-synthetic nature of 
our current tests + the sketchy way we ignored drift makes the current 
approach worth abandoning in any event. The fact that the gold standard 
must include tests of larger, "real world" code, which unfortunately 
takes longer to build, I also think is a point towards this asynchronous 
approach: We trade MR latency for stat latency, but better utilize our 
build machines and get better stats, and when a human is to fix 
something a few days later, they have a much better foundation to start 
their investigation.


Finally I agree with SPJ that for fairness and sustainability's sake, 
the person investigating issues after the fact should ideally be the MR 
authors, and definitely definitely not Ben. But I hope that better 
stats, nice looking graphs, and maybe a system to automatically ping MR 
authors, will make the perf debugging much more accessible enabling that 
goal.


John

On 3/17/21 9:47 AM, Sebastian Graf wrote:
Re: Performance drift: I opened 
https://gitlab.haskell.org/ghc/ghc/-/issues/17658 
 a while ago with 
an idea of how to measure drift a bit better.
It's basically an automatically checked version of "Ben stares at 
performance reports every two weeks and sees that T9872 has regressed 
by 10% since 9.0"


Maybe we can have Marge check for drift and each individual MR for 
incremental perf regressions?


Sebastian

Am Mi., 17. März 2021 um 14:40 Uhr schrieb Richard Eisenberg 
mailto:r...@richarde.dev>>:





On Mar 17, 2021, at 6:18 AM, Moritz Angermann
mailto:moritz.angerm...@gmail.com>>
wrote:

But what do we expect of patch authors? Right now if five people
write patches to GHC, and each of them eventually manage to get
their MRs green, after a long review, they finally see it
assigned to marge, and then it starts failing? Their patch on its
own was fine, but their aggregate with other people's code leads
to regressions? So we now expect all patch authors together to
try to figure out what happened? Figuring out why something
regressed is hard enough, and we only have a very few people who
are actually capable of debugging this. Thus I believe it would
end up with Ben, Andreas, Matthiew, Simon, ... or someone else
from GHC HQ anyway to figure out why it regressed, be it in the
Review Stage, or dissecting a marge aggregate, or on master.


I have previously posted against the idea of allowing Marge to
accept regressions... but the paragraph above is sadly convincing.
Maybe Simon is right about opening up the windows to, say, be 100%
(which would catch a 10x regression) instead of infinite, but I'm
now convinced that Marge should be very generous in allowing
regressions -- provided we also have some way of monitoring drift
over time.

Separately, I've been concerned for some time about the
peculiarity of our perf tests. For example, I'd be quite happy to
accept a 25% regression on T9872c if it yielded a 1% improvement
on compiling Cabal. T9872 is very very very strange! (Maybe if
*all* the T9872 tests regressed, I'd be more worried.) I would be
very happy to learn that some more general, representative tests
are included in our examinations.

Richard
___
ghc-devs mailing list
ghc-devs@haskell.org 
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GSOC Idea: Bytecode serialization and/or Fat Interface files

2021-03-12 Thread John Ericson

Yes, see 
https://gitlab.haskell.org/ghc/ghc/-/wikis/Plan-for-increased-parallelism-and-more-detailed-intermediate-output 
where we (Obsidian) and IOHK have been planning together.

I must saw, I am a bit skeptical about a GSOC being able to take this on 
successfully. I thought Fendor did a great job with multiple home units, 
for example, but we have still to finish merging all his work! The 
driver is perhaps the biggest cesspool of technical debt in GHC, and it 
will take a while to untangle let alone implement new features.

I forget what the rules are for more incremental or multifaceted 
projects, but I would prefer an approach of trying to untangle things 
with no singular large goal. Or maybe we can involve a student with 
efforts to improve CI, attacking the root cause for why it's so hard to 
land things in the first place .

John

On 3/12/21 7:11 PM, Moritz Angermann wrote:
Yes there is also John resumable compilation ideas. And the current 
performance work obsidian systems does.

On Sat, 13 Mar 2021 at 6:21 AM, Cheng Shao > wrote:

I believe Josh has already been working on 2 some time ago? cc'ing him
to this thread.

I'm personally in favor of 2 since it's also super useful for
prototyping whole-program ghc backends, where one can just read all
the CgGuts from the .hi files, and get all codegen-related Core for
free.

Cheers,
Cheng

On Fri, Mar 12, 2021 at 10:32 PM Zubin Duggal
mailto:zubin.dug...@gmail.com>> wrote:
>
> Hi all,
>
> This is following up on this recent discussion on the list
concerning fat
> interface files:
https://mail.haskell.org/pipermail/ghc-devs/2020-October/019324.html

>
> Now that we have been accepted as a GSOC organisation, I think
> it would be a good project idea for a sufficiently motivated and
> advanced student. This is a call for mentors (and students as
> well!) who would be interested in this project
>
> The problem is the following:
>
> Haskell Language Server (and ghci with `-fno-code`) have very
> fast startup times for codebases which don't make use of Template
> Haskell, and thus don't require any code-gen to typecheck. This
> is because they can simply read the cached iface files generated
by a
> previous compile and don't need to re-invoke the typechecker.
>
> However, as soon as TH is involved, we are forced to retypecheck and
> compile files, since it is not possible to restart the code-gen
process
> starting with only a iface file. I can think of two ways to
address this
> problem:
>
> 1. Allow bytecode to be serialized
>
> 2. Serialize desugared Core into iface files (fat interfaces),
so that
> (byte)code-gen can be restarted from this point and doesn't need
>
> (1) might be challenging, but offers a few more advantages over (2),
> in that we can reduce the work done to load TH-heavy codebases
to just
> a load of the cached bytecode objects from disk, and could make the
> load process (and times) for these codebases directly comparable to
> their TH-free cousins.
>
> It would also make ghci startup a lot faster with a warm cache of
> bytecode objects, bringing ghci startup times in line with those of
> -fno-code
>
> However (2) might be much easier to achieve and offers many
> of the same advantages, in that we would not need to re-run
> the compiler frontend or core-to-core optimisation phases.
> There is also already a (slightly bitrotted) implementation
> of (2) thanks to the work of Edward Yang.
>
> If any of this sounds exciting to you as a student or a mentor,
please
> get in touch.
>
> In particular, I think (2) is a feasible project that can be
completed
> with minimal mentoring effort. However, I'm only vaguely
familiar with
> the details of the byte code generator, so if (1) is a direction
we want
> to pursue, we would need a mentor familiar with the details of
this part
> of GHC.
>
> Cheers,
> Zubin
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org 
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org 
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Requesting help with unregisterized backend failure of !4717

2021-03-08 Thread John Ericson

The problem occurs earlier in the pipeline than that. The generated C 
doesn't have the proper branching present in the original Haskell.


On 3/8/21 7:25 PM, Carter Schonwald wrote:
Isn’t the unregisterized backend a c compiler? You should check what 
gcc and clang or whoever we use handles those issues


On Mon, Mar 8, 2021 at 6:56 PM John Ericson 
 wrote:


Hi everyone,

https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4717
<https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4717> fails some
numerical boundary checking tests with the unregisterized backend. In
particular, `minBound / (-1)` and `pred minBound` are *not*
diverging as
expected. This stumped me a few days ago, and no new ideas have
struct
me since. I would very much appreciate some help.

This does seem like something that is very likely to be
backend-dependent, as different instructions/arches handle such edge
cases in different ways. What makes this so peculiar, however, is
that
the boundary condition checking/branching is done *in regular
Haskell*.
I'm thus quite baffled as to what could be going wrong.

If anyone wants to dive in, see my last comment
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4717#note_329840
<https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4717#note_329840>,

which has a minimization to declutter the code without doing enough
constant folding that the problem is avoided entirely. (I went with a
compilation unit trick, but in hindsight NOINLINE should also work
and
be simpler.)

Finally, let my provide some context for why I am hoping to get this
merged soon. To be clear, !4492 was the main MR relating to the
numerics
primops critical to get in for 9.2 and it thankfully already
landed. But
landing this MR too would be nice: It shrinks the intermediate
representations of numerical code probably smaller than it was
originally, whereas now it is perhaps larger than it was before
due to
more size<->native conversions. Also, I think this MR will help get
!3658 over the finish line, which, while again not as critical for
9.2
as !4492 was, would be awfully nice to do to fully realize the new
simplicity of the plan set forth in
https://gitlab.haskell.org/ghc/ghc/-/wikis/Unboxed-Numerics
<https://gitlab.haskell.org/ghc/ghc/-/wikis/Unboxed-Numerics>.

Thanks,

John

N.B. I just rebased and repushed the MR, so CI might be still
running or
failing due to something else, but based on local testing this is
still
the an issue. https://gitlab.haskell.org/ghc/ghc/-/pipelines/31002
<https://gitlab.haskell.org/ghc/ghc/-/pipelines/31002> is an
earlier pipeline run that one can look at until CI finishes again.

___
ghc-devs mailing list
ghc-devs@haskell.org <mailto:ghc-devs@haskell.org>
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
<http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs>

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Requesting help with unregisterized backend failure of !4717

2021-03-08 Thread John Ericson


Hi everyone,

https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4717 fails some 
numerical boundary checking tests with the unregisterized backend. In 
particular, `minBound / (-1)` and `pred minBound` are *not* diverging as 
expected. This stumped me a few days ago, and no new ideas have struct 
me since. I would very much appreciate some help.


This does seem like something that is very likely to be 
backend-dependent, as different instructions/arches handle such edge 
cases in different ways. What makes this so peculiar, however, is that 
the boundary condition checking/branching is done *in regular Haskell*. 
I'm thus quite baffled as to what could be going wrong.


If anyone wants to dive in, see my last comment 
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4717#note_329840, 
which has a minimization to declutter the code without doing enough 
constant folding that the problem is avoided entirely. (I went with a 
compilation unit trick, but in hindsight NOINLINE should also work and 
be simpler.)


Finally, let my provide some context for why I am hoping to get this 
merged soon. To be clear, !4492 was the main MR relating to the numerics 
primops critical to get in for 9.2 and it thankfully already landed. But 
landing this MR too would be nice: It shrinks the intermediate 
representations of numerical code probably smaller than it was 
originally, whereas now it is perhaps larger than it was before due to 
more size<->native conversions. Also, I think this MR will help get 
!3658 over the finish line, which, while again not as critical for 9.2 
as !4492 was, would be awfully nice to do to fully realize the new 
simplicity of the plan set forth in 
https://gitlab.haskell.org/ghc/ghc/-/wikis/Unboxed-Numerics.


Thanks,

John

N.B. I just rebased and repushed the MR, so CI might be still running or 
failing due to something else, but based on local testing this is still 
the an issue. https://gitlab.haskell.org/ghc/ghc/-/pipelines/31002 is an 
earlier pipeline run that one can look at until CI finishes again.


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes to performance testing?

2021-02-23 Thread John Ericson

If we make the changes proposed in the "On CI" thread, I think we solve 
most of this for free. If a perf test fails, resubmitting should not 
need to rebuild GHC because it is cached. I'd say at that point, it's 
not even worth making Marge bot auto-accept extra improvements because 
restarting with new perf windows should be so easy.


John

On 2/22/21 8:46 AM, Andreas Klebinger wrote:


This seems quite reasonable to me.
Not sure about the cost of implementing it (and the feasability of it 
if/when merge-trains arrive).


Andreas

Am 21/02/2021 um 21:31 schrieb Richard Eisenberg:



On Feb 21, 2021, at 11:24 AM, Ben Gamari > wrote:


To mitigate this I would suggest that we allow performance test failures
in marge-bot pipelines. A slightly weaker variant of this idea would
instead only allow performance *improvements*. I suspect the latter
would get most of the benefit, while eliminating the possibility that a
large regression goes unnoticed.


The value in making performance improvements a test failure is so 
that patch authors can be informed of what they have done, to make 
sure it matches expectations. This need can reasonably be satisfied 
without stopping merging. That is, if Marge can accept performance 
improvements, while (say) posting to each MR involved that it may 
have contributed to a performance improvement, then I think we've 
done our job here.


On the other hand, a performance degradation is a bug, just like, 
say, an error message regression. Even if it's a combination of 
commits that cause the problem (an actual possibility even for error 
message regressions), it's still a bug that we need to either fix or 
accept (balanced out by other improvements). The pain of debugging 
this scenario might be mitigated if there were a collation of the 
performance wibbles for each individual commit. This information is, 
in general, available: each commit passed CI on its own, and so it 
should be possible to create a little report with its rows being perf 
tests and its columns being commits or MR #s; each cell in the table 
would be a percentage regression. If we're lucky, the regression 
Marge sees will be the sum(*) of the entries in one of the rows -- 
this means that we have a simple agglomeration of performance 
degradation. If we're less lucky, the whole will not equal the sum of 
the parts, and some of the patches interfere. In either case, the 
table would suggest a likely place to look next.


(*) I suppose if we're recording percentages, it wouldn't necessarily 
be the actual sum, because percentages are a bit funny. But you get 
my meaning.


Pulling this all together:
* I'm against the initial proposal of allowing all performance 
failures by Marge. This will allow bugs to accumulate (in my opinion).
* I'm in favor of allowing performance improvements to be accepted by 
Marge.
* To mitigate against the information loss of Marge accepting 
performance improvements, it would be great if Marge could alert MR 
authors that a cumulative performance improvement took place.
* To mitigate against the annoyance of finding a performance 
regression in a merge commit that does not appear in any component 
commit, it would be great if there were a tool to collect performance 
numbers from a set of commits and present them in a table for further 
analysis.


These "mitigations" might take work. If labor is impossible to 
produce to complete this work, I'm in favor of simply allowing the 
performance improvements, maybe also filing a ticket about these 
potential improvements to the process.


Richard

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: On CI

2021-02-22 Thread John Ericson

I agree one should be able to get most of the testing value from stage1.
And the tooling team at IOHK has done some work in
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3652 to allow a
stage 1 compiler to be tested. That's a very important first step!

But TH and GHCi require either iserv (external interpreter) or an
compiler whose own ABI and the outputted ABI match for the internal
interpreter, and ideally we should test both. I think doing a --freeze1
stage2 build *in addition* to the stage1 build would work in the
majority of cases, and that would allow us to incrementally build and
test both. Remember that iserv uses the ghc library, and needs to be ABI
comparable with the stage1 compiler that is using it, so it is less a
panacea than it might seem like for ABI changes vs mere cross compilation.

I opened https://github.com/ghc-proposals/ghc-proposals/issues/162 for
an ABI-agnostic interpreter that would allow stage1 alone to do GHCi and
TH a third away unconditionally. This would also allow TH to safely be
used in GHC itself, but for the purposes of this discussion, it's nice
to make testing more reliable without the --freeze1 stage 2 gamble.

Bottom line is, yes, building stage 2 from a freshly-built stage 1 will
invalidate any cache, and so we should avoid that.

John

On 2/22/21 8:42 AM, Spiwack, Arnaud wrote:
Let me know if I'm talking nonsense, but I believe that we are
building both stages for each architecture and flavour. Do we need to
build two stages everywhere? What stops us from building a single
stage? And if anything, what can we change to get into a situation
where we can?

Quite better than reusing build incrementally, is not building at all.

On Mon, Feb 22, 2021 at 10:09 AM Simon Peyton Jones via ghc-devs
mailto:ghc-devs@haskell.org>> wrote:

Incremental CI can cut multiple hours to < mere minutes,
especially with the test suite being embarrassingly parallel.
There simply no way optimizations to the compiler independent from
sharing a cache between CI runs can get anywhere close to that
return on investment.

I rather agree with this. I don’t think there is much low-hanging
fruit on compile times, aside from coercion-zapping which we are
working on anyway. If we got a 10% reduction in compile time we’d
be over the moon, but our users would barely notice.

To get truly substantial improvements (a factor of 2 or 10) I
think we need to do less compiling – hence incremental CI.

Simon

*From:*ghc-devs mailto:ghc-devs-boun...@haskell.org>> *On Behalf Of *John Ericson
*Sent:* 22 February 2021 05:53
*To:* ghc-devs mailto:ghc-devs@haskell.org>>
*Subject:* Re: On CI

I'm not opposed to some effort going into this, but I would
strongly opposite putting all our effort there. Incremental CI can
cut multiple hours to < mere minutes, especially with the test
suite being embarrassingly parallel. There simply no way
optimizations to the compiler independent from sharing a cache
between CI runs can get anywhere close to that return on investment.

(FWIW, I'm also skeptical that the people complaining about GHC
performance know what's hurting them most. For example, after
non-incrementality, the next slowest thing is linking, which
is...not done by GHC! But all that is a separate conversation.)

John

On 2/19/21 2:42 PM, Richard Eisenberg wrote:

There are some good ideas here, but I want to throw out
another one: put all our effort into reducing compile times.
There is a loud plea to do this on Discourse

<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdiscourse.haskell.org%2Ft%2Fcall-for-ideas-forming-a-technical-agenda%2F1901%2F24=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691120329%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=1CV0MEVUZpbAbmKAWTIiqLgjft7IbN%2BCSnvB3W3iX%2FU%3D=0>,
and it would both solve these CI problems and also help
everyone else.

This isn't to say to stop exploring the ideas here. But since
time is mostly fixed, tackling compilation times in general
may be the best way out of this. Ben's survey of other
projects (thanks!) shows that we're way, way behind in how
long our CI takes to run.

Richard

On Feb 19, 2021, at 7:20 AM, Sebastian Graf
mailto:sgraf1...@gmail.com>> wrote:

Recompilation avoidance

I think in order to cache more in CI, we first have to
invest some time in fixing recompilation avoidance in our
bootstrapped build system.

I just tested on a hadrian perf ticky build: Adding one
line of *comment* in the compiler causes

* a (

Re: On CI

2021-02-21 Thread John Ericson

language with staged meta-programming.

Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via
ghc-devs mailto:ghc-devs@haskell.org>>:

Doing "optimistic caching" like you suggest sounds very
promising. A way to regain more robustness would be as follows.
If the build fails while building the libraries or the stage2
compiler, this might be a false negative due to the optimistic
caching. Therefore, evict the "optimistic caches" and restart
building the libraries. That way we can validate that the build
failure was a true build failure and not just due to the
aggressive caching scheme.

Just my 2p

Josef

*From:* ghc-devs mailto:ghc-devs-boun...@haskell.org>> on behalf of Simon Peyton
Jones via ghc-devs mailto:ghc-devs@haskell.org>>
*Sent:* Friday, February 19, 2021 8:57 AM
*To:* John Ericson mailto:john.ericson@obsidian.systems>>; ghc-devs
mailto:ghc-devs@haskell.org>>
*Subject:* RE: On CI

1. Building and testing happen together. When tests failure
spuriously, we also have to rebuild GHC in addition to
re-running the tests. That's pure waste.
https://gitlab.haskell.org/ghc/ghc/-/issues/13897

I don’t get this. We have to build GHC before we can test it,
don’t we?
2 . We don't cache between jobs.
This is, I think, the big one. We endlessly build the exact
same binaries.
There is a problem, though. If we make **any** change in GHC,
even a trivial refactoring, its binary will change slightly. So
now any caching build system will assume that anything built by
that GHC must be rebuilt – we can’t use the cached version. That
includes all the libraries and the stage2 compiler. So caching
can save all the preliminaries (building the initial Cabal, and
large chunk of stage1, since they are built with the same
bootstrap compiler) but after that we are dead.
I don’t know any robust way out of this. That small change in
the source code of GHC might be trivial refactoring, or it might
introduce a critical mis-compilation which we really want to see
in its build products.
However, for smoke-testing MRs, on every architecture, we could
perhaps cut corners. (Leaving Marge to do full diligence.) For
example, we could declare that if we have the result of compiling
library module X.hs with the stage1 GHC in the last full commit
in master, then we can re-use that build product rather than
compiling X.hs with the MR’s slightly modified stage1 GHC. That
**might** be wrong; but it’s usually right.
Anyway, there are big wins to be had here.
Simon

*From:*ghc-devs mailto:ghc-devs-boun...@haskell.org>> *On Behalf Of *John Ericson
*Sent:* 19 February 2021 03:19
*To:* ghc-devs mailto:ghc-devs@haskell.org>>
*Subject:* Re: On CI

I am also wary of us to deferring checking whole platforms and
what not. I think that's just kicking the can down the road, and
will result in more variance and uncertainty. It might be alright
for those authoring PRs, but it will make Ben's job keeping the
system running even more grueling.

Before getting into these complex trade-offs, I think we should
focus on the cornerstone issue that CI isn't incremental.

<https://nam06.safelinks.protection.outlook.com/?url=https://gitlab.haskell.org/ghc/ghc/-/issues/13897=04%7c01%7csimo...@microsoft.com%7C3d503922473f4cd0543f08d8d48522b2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637493018301253098%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0=%7C3000=FG2fyYCXbacp69Q8Il6GE0aX+7ZLNkH1u84NA/VMjQc==0>
tracks this more or less.
2. We don't cache between jobs. Shake and Make do not enforce
dependency soundness, nor cache-correctness when the build
plan itself changes, and this had made this hard/impossible
to do safely. Naively this only helps with stage 1 and not
stage 2, but if we have separate stage 1 and --freeze1 stage
2 builds, both can be incremental. Yes, this is also lossy,
but I only see it lea

Re: On CI

2021-02-18 Thread John Ericson

I am also wary of us to deferring checking whole platforms and what not. 
I think that's just kicking the can down the road, and will result in 
more variance and uncertainty. It might be alright for those authoring 
PRs, but it will make Ben's job keeping the system running even more 
grueling.


Before getting into these complex trade-offs, I think we should focus on 
the cornerstone issue that CI isn't incremental.


1. Building and testing happen together. When tests failure spuriously,
   we also have to rebuild GHC in addition to re-running the tests.
   That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897
   tracks this more or less.
2. We don't cache between jobs. Shake and Make do not enforce
   dependency soundness, nor cache-correctness when the build plan
   itself changes, and this had made this hard/impossible to do safely.
   Naively this only helps with stage 1 and not stage 2, but if we have
   separate stage 1 and --freeze1 stage 2 builds, both can be
   incremental. Yes, this is also lossy, but I only see it leading to
   false failures not false acceptances (if we can also test the stage
   1 one), so I consider it safe. MRs that only work with a slow full
   build because ABI can so indicate.

The second, main part is quite hard to tackle, but I strongly believe 
incrementality is what we need most, and what we should remain focused on.


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: presentation: Next-gen Haskell Compilation Techniques

2021-01-11 Thread John Ericson


Great presentation, Csaba!

I definitely strongly agree in broad terms that these are the overlooked 
big questions we should be asking.


On the is last point, not only can we find a middle ground we like best, 
we can also do the full spectrum. I know something that many people I've 
talked to would like is a "compilation server" of sorts that just keeps 
on optimizing, building up a bigger database of knowledge of spitting 
out better binaries the longer you keep it running. If I understand 
correctly, a datalog-based method with very good properties re 
incrementality and monotonicity should be the perfect architecture for this.


Cheers,

John

On 1/11/21 8:17 AM, Csaba Hruska wrote:
Sure, some require whole-program analysis. But I really do not worry 
about it, because what I'd like to build is an engineering vehicle. 
Where a single optimization idea could be built in several ways with 
different tradeoffs. Then the sweet spot could be found after an in 
depth investigation of the problem domain.
I.e. removing all indirect calls surely require whole program 
defunctionalization, but a significant reduction of indirect calls 
could be achieved with other techniques that does not require whole 
program analysis. But it is totally valuable to compare the two 
approaches just to know the tradeoffs even if only one of them is 
applicable in practice.


Csaba

On Mon, Jan 11, 2021 at 1:51 PM Simon Peyton Jones 
mailto:simo...@microsoft.com>> wrote:


I may not emphasize in the talk, but the goal of the grin compiler
project is to build a compiler pipeline that allows easy
experimentation of different compilation techniques. Anything
between whole program compilation to per module incremental
codegen. So the whole program compilation is not really a
requirement but an option.

Right – but /some/ optimisations absolutely require whole-program
analysis, don’t they?  I’m thinking of flow analyses that support
defunctionalisation, when you must know all the lambdas that could
be bound to `f` in the definition of `map` for example.

Such optimisations are powerful, but brittle because they are
simply inapplicable without whole-program analysis.  Or maybe you
can find ways to make them more resilient.

Simon

*From:*ghc-devs mailto:ghc-devs-boun...@haskell.org>> *On Behalf Of *Csaba Hruska
*Sent:* 11 January 2021 12:19
*To:* Sebastian Graf mailto:sgraf1...@gmail.com>>
*Cc:* GHC developers mailto:ghc-devs@haskell.org>>
*Subject:* Re: presentation: Next-gen Haskell Compilation Techniques

Hi Sebastian,

Thanks for your feedback.

I know that CIB and Perceus have issues with cycles, but these
systems are still in development so who knows what will be the
conclusion.

I may not emphasize in the talk, but the goal of the grin compiler
project is to build a compiler pipeline that allows easy
experimentation of different compilation techniques. Anything
between whole program compilation to per module incremental
codegen. So the whole program compilation is not really a
requirement but an option.

Cheers,

Csaba

On Sun, Jan 10, 2021 at 1:58 PM Sebastian Graf
mailto:sgraf1...@gmail.com>> wrote:

Hi Csaba,

Thanks for your presentation, that's a nice high-level
overview of what you're up to.

A few thoughts:

  * Whole-program optimization sounds great, but also very
ambitious, given the amount of code GHC generates today.
I'd be amazed to see advances in that area, though, and
your >100-module CFA performance incites hope!
  * I wonder if going through GRIN results in a more efficient
mapping to hardware. I recently found that the code GHC
generates is dominated by administrative traffic from and
to the heap [1]. I suspect that you can have big wins here
if you manage to convey better call stack, heap and alias
information to LLVM.
  * The Control Analysis+specialisation approach sounds pretty
similar to doing Constructor Specialisation [2] for
Lambdas (cf. 6.2) if you also inline the function for
which you specialise afterwards. I sunk many hours into
making that work reliably, fast and without code bloat in
the past, to no avail. Frankly, if you can do it in GRIN,
I don't see why we couldn't do it in Core. But maybe we
can learn from the GRIN implementation afterwards and
maybe rethink SpecConstr. Maybe the key is not to inline
the function for which we specialise? But then you don't
gain that much...
  * I follow the Counting Immutable Beans [3] stuff quite
closely (Sebastian is a colleague of mine) and hope that
it is applicable to Haskell some day. But I think using

Re: Troubles when building ghc

2020-12-23 Thread John Ericson

Actually, your alex is /too/ new. See 
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4657. If you take 
the latest master, it will contain that, and your problem will be fixed.


On 12/23/20 2:10 PM, Richard Eisenberg wrote:
Without having particular knowledge about this specific error: try 
upgrading `alex`, the tool that GHC uses to produce its lexer. It's 
possible a newer version of that tool will fix this problem.


Richard

On Dec 23, 2020, at 1:37 PM, Neophytos Michael > wrote:


I am having some trouble building ghc on ubuntu.  I have ghc 8.8.4 
installed on the machine.  I pulled from the main repository and 
followed the instructions here: 
https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian 



Building with:
./hadrian/build -j

it builds for a long time until it finally fails with the message 
below.  I would appreciate some advice on how to fix this.


Thank you,
Neo

===
Exit code: 1
Stderr and Stdout:
_build/stage1/compiler/build/GHC/Parser/Lexer.hs:3495:3: error:
    Variable not in scope: int16ToInt# :: Int# -> Int#
     |
3495 |   int16ToInt#
     |   ^^^

_build/stage1/compiler/build/GHC/Parser/Lexer.hs:3520:3: error:
    Variable not in scope: int32ToInt# :: Int# -> Int#
     |
3520 |   int32ToInt#
     |   ^^^
===
On that line (in Lexer.hs) we see:

#if __GLASGOW_HASKELL__ >= 901
  int16ToInt#
#endif
    (indexInt16OffAddr# arr off)
#endif

so we are in the "then" clause and somehow "int16ToInt#" is nowhere 
to be found.


--
Neo

___
ghc-devs mailing list
ghc-devs@haskell.org 
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Using a development snapshot of happy

2020-12-04 Thread John Ericson

I'll take the bait :) and offer co-maintain both Happy and Alex, on the 
condition that we agree to make them submodules in GHC.


John

On 12/4/20 7:00 AM, Oleg Grenrus wrote:

Would you be happy to be a maintainer of happy?

Sorry, couldn't resist, but I ask seriously.

- Oleg

On 2.8.2020 10.43, Vladislav Zavialov wrote:

Hi ghc-devs,

I’m working on the unification of parsers for terms and types, and one of the 
things I’d really like to make use of is a feature I implemented in ‘happy’ in 
October 2019 (9 months ago):

   https://github.com/simonmar/happy/pull/153

It’s been merged upstream, but there has been no release of ‘happy’, despite 
repeated requests:

   1. I asked for a release in December: 
https://github.com/simonmar/happy/issues/164
   2. Ben asked for a release a month ago: 
https://github.com/simonmar/happy/issues/168

I see two solutions here:

   a) Find a co-maintainer for ‘happy’ who could make releases more frequently 
(I understand the current maintainers probably don’t have the time to do it).
   b) Use a development snapshot of ‘happy’ in GHC

Maybe we need to do both, but one reason I’d like to see (b) in particular 
happen is that I can imagine introducing more features to ‘happy’ for use in 
GHC, and it’d be nice not to wait for a release every time. For instance, there 
are some changes I’d like to make to happy/alex in order to implement #17750

So here are two questions I have:

   1. Are there any objections to this idea?
   2. If not, could someone more familiar with the build process guide me as to 
how this should be implemented? Do I add ‘happy’ as a submodule and change 
something in the ./configure script, or is there more to it? Do I need to 
modify make/hadrian, and if so, then how?

Thanks,
- Vlad
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Using a development snapshot of happy

2020-12-03 Thread John Ericson

Seeing https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4560 stuck on 
needing a new version of Alex reminded me of this.


Ben raises a good point on Happy bootstrapping from itself making this a 
pain, but I'd hope we could just get around this by vendoring the 
generated happy parser in the happy repo. In fact their two ways to do this:


- as a permanent change, in which case we'd want to write a script to 
update the vendor instead of the custom sdist that is is the point of 
the Makefile.


- in a separate "master-sdist" branch of generated sdists, which GHC 
would track with the submodule instead of master.


While I'm rarely for vendoring generated code, breaking a bootstrap 
cycle is a pretty darn good reason. Also this is rather benign case of 
bootstrap artifact vendoring as:


 - Generated happy code is a lot easier to understand than machine code 
at scale


 - Ken Thompson attacks on the parser scare me less than elsewhere in 
the compiler


Finally If all that is still too ugly, well, it would be nice to have a 
parser-combinator implementation of the same functionality that can 
double as a oracle for testing.


Cheers,

John

On 8/4/20 1:21 PM, Ben Gamari wrote:

Vladislav Zavialov  writes:


Hi ghc-devs,

I’m working on the unification of parsers for terms and types, and one
of the things I’d really like to make use of is a feature I
implemented in ‘happy’ in October 2019 (9 months ago):

   https://github.com/simonmar/happy/pull/153

It’s been merged upstream, but there has been no release of ‘happy’,
despite repeated requests:

   1. I asked for a release in December: 
https://github.com/simonmar/happy/issues/164
   2. Ben asked for a release a month ago: 
https://github.com/simonmar/happy/issues/168

I see two solutions here:

   a) Find a co-maintainer for ‘happy’ who could make releases more
   frequently (I understand the current maintainers probably don’t have
   the time to do it).
   b) Use a development snapshot of ‘happy’ in GHC

Maybe we need to do both, but one reason I’d like to see (b) in
particular happen is that I can imagine introducing more features to
‘happy’ for use in GHC, and it’d be nice not to wait for a release
every time. For instance, there are some changes I’d like to make to
happy/alex in order to implement #17750

So here are two questions I have:

   1. Are there any objections to this idea?

I'm not entirely keen on the idea: while the cost of the submodule
itself is pretty low (happy is a small package which takes little time
to build), I am skeptical of addressing social issues like happy's lack
of maintenance with technical solutions. Ultimately, shipping happy as a
submodule would merely kick the current problem down the road:
eventually (when we release GHC) we will need a happy release. Unless
the underlying maintainership problem is addressed we will end up right
back where we are today.

Moreover, note that happy requires happy as a build dependency so we won't be
able to drop it as a build dependency of GHC even if we do include it as
a submodule.

For this reason, I would weakly prefer that we first find a maintainer
and try to get a release out before jumping straight to adding happy as
a submodule. I will try to bring up the matter with Simon Marlow to see
if we can't find a solution here.


   2. If not, could someone more familiar with the build process guide
   me as to how this should be implemented? Do I add ‘happy’ as a
   submodule and change something in the ./configure script, or is
   there more to it? Do I need to modify make/hadrian, and if so, then
   how?


It will be a tad more involved than this. We will need to teach the
build systems to build Happy, use the configure executable, and update
the source distribution packaging rules to include the new submodule.
Moreover, happy (unfortunately) has a make-based build system which will
need to be used to generate its parser.

Updating the build systems likely won't be difficult, but there isn't clear
documentation on what it will involve. This will really be a matter of
finding a similar existing case (e.g. genprimops, perhaps?), following
it as a model, and figuring out how to fill any gaps.

Moreover, build system logic is inevitably a bug-nest; adding the same
logic twice greatly increases the chance that we will have (two sets of)
bugs that will only be caught post-release. For me, this underscores the
need to first try to make the existing decoupled situation work before
moving to a vendored solution.

Cheers,

- Ben

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Use of forall as a sigil

2020-11-22 Thread John Ericson

I have thought about this too, and don't believe it has been widely 
discussed.


- We are already getting `forall {a}.`, so it fits nicely with that.

- However, it would have to be `forall @a ->`, because `forall a.` is 
already an invisible quantification, unless one wants to just change the 
meaning of `forall a.`!


John

On 11/22/20 6:23 AM, Andrey Mokhov wrote:

Hi Richard,


In the end, I've never loved the forall ... -> syntax, but I've never seen
anything better.

What about the forall @a. syntax?

For example:

   sizeOf :: forall @a. Sized a => Int

We already use @ to explicitly specify types, so it seems natural mark type 
parameters that must be explicitly specified with @ too.

Here's how one would read it: "for all explicitly specified a, ..."

Apologies if this has been discussed and I missed it. It doesn't seem to be 
mentioned in the Alternatives section of the proposal but perhaps it will just 
never work for some reason.

Cheers,
Andrey

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GHC's internal confusion about Ints and Words

2020-10-22 Thread John Ericson

That sounds like a great change, and beautifully dovetails with !3658. 
(In fact an earlier version of that PR also attempted the Int32 change.)

I would just try to finish that and/or reuse the techniques. Sylvain has 
been doing all the work lately as I've been starved for time/ideas, so 
talk to him.

John

On 10/22/20 9:45 AM, Moritz Angermann wrote:

Hi *,

so, after some discussion with Simon and Simon, as well as Ben, we are 
all in agreement that using sized hints
is a band-aid solution for the real underlying problem. Where the 
underlying problem is that we have CInt ~ Int32,
and we represent Int32 as I32# Int#.  And the proper solution would 
not likely be to represent Int32 as I32# Int32#.

After some trial and error (mostly be being too aggressive on changing 
Ints to sized ones, unnecessarily -- thanks
Ben for helping me stay on course!), I've produce what mostly amounts 
to this patch[1].

It also requires some additional narrow/extend calls to a few 
Data.Array.Base signatures to make them typecheck.

However I've got plenty of failures in the testsuite now. Hooray!

Most of them are of this form:

*** Core Lint errors : in result of Desugar (before optimization) ***
T12010.hsc:34:1: warning:
    Argument value doesn't match argument type:
    Fun type: Int# -> Int#
    Arg type: Int32#
    Arg: ds_d1B3
    In the RHS of c_socket :: CInt -> CInt -> CInt -> IO CInt
    In the body of lambda with binder ds_d1AU :: Int32
    In the body of lambda with binder ds_d1AV :: Int32
    In the body of lambda with binder ds_d1AW :: Int32
    In a case alternative: (I32# ds_d1AY :: Int32#)
    In a case alternative: (I32# ds_d1B0 :: Int32#)
    In a case alternative: (I32# ds_d1B2 :: Int32#)
    In the body of lambda with binder ds_d1B5 :: State# RealWorld
    In a case alternative: ((#,#) ds_d1B4 :: State# RealWorld,
                                  ds_d1B3 :: Int32#)
    Substitution: [TCvSubst
                     In scope: InScope {}
                     Type env: []
                     Co env: []]

(full log at 
https://gist.github.com/angerman/3d6e1e3da5299b9365125ee9e0a2c40f)

Some other minor ones are test that now need explicit narrow/extending 
where it didn't need before.

As well as this beauty:

-- RHS size: {terms: 16, types: 0, coercions: 0, joins: 0/0}
i32 :: Int32
[GblId,
 Cpr=m1,
 Unf=Unf{Src=, TopLvl=True, Value=True, ConLike=True,
         WorkFree=False, Expandable=False, Guidance=IF_ARGS [] 23 10}]
i32
  = GHC.Int.I32#
      (GHC.Prim.narrowInt32#
         (GHC.Prim.andI#
            (GHC.Prim.extendInt32#
               (GHC.Prim.narrowInt32#
                  (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# 1#
            (GHC.Prim.extendInt32#
               (GHC.Prim.narrowInt32#
                  (GHC.Prim.notI#
                     (GHC.Prim.extendInt32#
                        (GHC.Prim.narrowInt32#
                           (GHC.Prim.extendInt32# 
(GHC.Prim.narrowInt32# 1#)

This clearly needs some clean up.

Apart from that the rest seems to be mostly working. Any input would 
be appreciated. I'll need to do the same for

Word as well I'm afraid.

Cheers,
 Moritz
--
[1]: 
https://gitlab.haskell.org/ghc/ghc/-/commit/acb5ce792806bc3c1e1730c6bdae853d2755de16?merge_request_iid=3641

On Tue, Oct 20, 2020 at 10:34 PM Cheng Shao > wrote:

Indeed STG to Cmm lowering drops the correct size information for
ccall arguments, there's even a TODO comment that has been around for
quite a few years:

https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/StgToCmm/Foreign.hs#L83

This has been an annoyance for Asterius as well. When we try to
translate a CmmUnsafeForeignCall node to a wasm function call, a CInt
argument (which should be i32 in wasm) can be mistyped as i64 which
causes a validation error. We have to insert wrap/extend opcodes based
on the callee function signature, but if we preserve correct argument
size in Cmm (or at least enrich the hints to include it), we won't
need such a hack.

On Tue, Oct 20, 2020 at 4:05 PM Moritz Angermann
mailto:moritz.angerm...@gmail.com>>
wrote:
>
> Yes, that's right. I'm not sure it's in core though, as the
width information still seems to be available in Stg. However the
lowering from
> stg into cmm widens it.
>
> On Tue, Oct 20, 2020 at 9:57 PM Carter Schonwald
mailto:carter.schonw...@gmail.com>>
wrote:
>>
>> ... are you talking about Haskell Int and word? Those are
always the same size in bits and should match native point size.
That is definitely an assumption of ghc
>>
>> It sounds like some information that is dropped after core is
needed to correctly do something in stg/cmm in the context of the
ARM64 ncg that was recently added to handle cint being 32bit in
this context ?
>>
>>
>> On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann

Re: How should we treat changes to the GHC API?

2020-07-28 Thread John Ericson


On 7/27/20 10:57 PM, Richard Eisenberg wrote:
I guess I'm arguing that ghc-lib (or something like it) should be the 
permanent solution. GHC will always be in flux internally.


So to be clear I don't disagree that GHC will always be in flux. But I 
think things like ghc-lib largely work today because GHC is so imperative.


Once it (hopefully!) is more functional, we'll have the problem less 
that functions change[1], and more that data structures change. I think 
that will be harder to abstract over---often to migrate an interface 
properly, data structure will need migrations in both directions *and* 
that those migrations form an isomorphism, which can be an impossible 
requirement. It's just a fact of life that rich data structures make 
migrations harder, but that's no reason we shouldn't have rich data 
structures!


I'm probably too deeply exploring what's only one possible future at 
this point, so let me rewind out of that exact hypothesis and just make 
the broader point that if the interfaces change a lot, a lot of 
organizational stuff can/should too. If the interface changes *don't* 
engender cultural shifts around working with GHC, we're probably doing 
them wrong and they are vapid churn after all!


John

[1]: there should be way fewer functions than today. we currently have a 
gratuitous combinatorial explosion of little helpers which obscures what 
the underlying orthogonal concepts are.


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: How should we treat changes to the GHC API?

2020-07-27 Thread John Ericson


On 7/27/20 4:57 PM, Richard Eisenberg wrote
On the other hand, we could imagine a dedicated GHC API volunteer who 
maintains the API on top of the shifting sands of GHC. Having a 
central project to deal with the API would satisfy clients by 
projecting a stable interface, and it wouldn't hinder GHC development, 
as the API volunteer would take on the compatibility burden. Of 
course, success here would require a dedicated, competent volunteer.


I think that basically exists: 
https://hackage.haskell.org/package/ghc-lib ? As permanent solution I 
don't like it, but as a stop gap to relieve any pressure for stability 
until we have sane interfaces, I *love* it.


John

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: How should we treat changes to the GHC API?

2020-07-27 Thread John Ericson

I would say the current API isabsolutely terrible and so we should 
have no qualms about breaking it. Talking to the people who work on the 
IDE---i.e. the people who are the most adversely impacted by 
churn---they also don't like it and not want it stabilized in its 
current form.


I think the only good long term solution is to modularize GHC into 
multiple packages, giving us conceptually distinct interfaces which 
break at different rates. This is similar to modularizing base as e.g. 
GHC.* breaks far more often than e.g. Data.List.


How we evaluate changes is less important to me as my main priority is 
to get the pace of development much higher so we have a chance on fixing 
all the technical debt to get us out of the current situation. 
Basically, you could count me in "team heat bath"; I think that should 
have the right connotations.


John

On 7/27/20 2:07 PM, Eric Seidel wrote:

As another former user of the GHC API, I'd say my two biggest complaints were 
the relative instability of the API and the lack of documentation. I haven't 
used the API in at least three years though, so it's possible much has changed 
since my experience.

I remember often having to do significant work to adapt LiquidHaskell to new 
versions of GHC due to small changes in the API. We often felt like we had to 
import internal modules (ie other than 'GHC') to get key bits of functionality 
we needed, which might explain the churn. But it also points to Iavor's point 
that the public API grew organically and might benefit from a bit of top-down 
design to make sure it's complete enough for typical use cases.

For documentation, the issue was less the API docs but a lack of "How do I do 
X?" docs and examples. One problem that I remember being particularly vexing was 
resolving names in a particular scope (in my case it was always module-level scopes, but 
I can easily imagine clients that would want to resolve names in local scopes).

I don't know if the API needs to go through something like the Steering 
Committee, but a stronger focus on API stability and perhaps a broader view of 
what constitutes (or should be included in) the public-facing API would be 
welcome!

On Mon, Jul 27, 2020, at 11:45, Iavor Diatchki wrote:

In principle, I think we should treat the GHC API like any other
library, and try not to break code unnecessarily.  However, my
impression is that the GHC API grew somewhat organically, so we may
want to put some additional work before we stabilize things too much.
It's been a while since I used it, so I might be out of date, but last
I looked the GHC API was a module exporting some high-level functions
from GHC.   I think that a single module is too small of an API for a
project as large as GHC.  In fact, it probably makes sense to define
more than one API.  For example, each plugin point should probably have
its own API, and that's likely different to the GHC API that exposes
functionality such as "load and type check this module here", or "parse
and evaluate this string".

-Iavor



On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs
 wrote:

What I’m after is a clear opportunity for informed debate, and a clear yes/no 
decision.  That need not be high overhead.
__ __
It means paying some upfront cost for design changes.  But that’s better than 
the legacy cost of dealing with things that turn out, in retrospect, to be less 
well designed than they could be.
__ __
We tend to think of APIs as implementation details.  But they are deeply 
significant, and express key abstractions, just like language designs do.  I 
think we should treat them just as seriously.
__ __
Simon
__ __
*From:* Mathieu Boespflug 
*Sent:* 27 July 2020 11:11
*To:* Simon Peyton Jones 
*Cc:* ghc-devs@haskell.org Devs 
*Subject:* Re: How should we treat changes to the GHC API?
__ __
I would just point out that decision by committee, and in particular the GHC 
Proposals process, has a high cost in terms of both total human brain cycles 
and latency. The cost is entirely justified when it comes to things that are a) 
hard to revert and b) extremely hard to get right the first time, like new 
extensions to the language, or c) very sensitive (like governance, say). For 
things like breaking changes to API's, it's worth writing out what the current 
problems are. Are users complaining that the API churn is too high? Are they 
concerned about endemic quality problems with the API?
__ __
It may be enough to make sure to know who the main users of the API are and tag 
them as reviewers on these types of changes in GitLab. Or to avoid extra 
process but enshrine principles that might be necessary to adopt, like saying 
that existing API functions should always be kept as-is during some deprecation 
period and new functionality should be exposed in new additions to the API. 
Principles to be upheld by reviewers.
__ __
On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones  
wrote:

A recent MR

Re: GHCs dependencies (libraries) and maintenance

2020-06-01 Thread John Ericson

I completely agree. I would like GHC be able to use *more* of the 
ecosystem than it does today, and there's no way we can can do that 
unless we dramatically decrease the pain per dependency.


John

On 6/1/20 5:23 AM, Moritz Angermann wrote:

Hi there!

so this comes up periodically, and I think we need to discuss this.  This is not
related to anything right now, so if you wonder if I'm writing this because of
something that just happened that I'm involved and you might have missed
something, you probably did not.  It came up on the #ghc IRC channel a
few day ago.

GHC depends on quite a set of libraries, and ships those in releases. When ever
a new GHC release is cut, all these dependencies need to be on hackage and
have release versions.  We do not want to produce a GHC release which depends
on in-flight packages.  In-flight might happen for example due to GHC having to
patch dependencies to make them work with HEAD.

Everyone who maintains any kind of software online, knows that maintenance can
be a drag, and then life happens, and what not.  There are many very responsive
maintainers and we all owe them a grate amount of gratitude towards their
relentless work, keeping those libraries up to date and responding to questions,
patches, ...

I therefore would like to float the following idea to make the GHC
release processes
a bit more reliable.  GHCHQ (that is those in charge of producing GHC
releases for
us all), are made co-maintainers on each library GHC depends on, to guarantee
that GHC can move forward in the worst of circumstances.  Now I would
hope that in
almost all cases GHCHQ would never have to maintain any of the dependencies
actively, they deal with GHC already, so let's try to keep it that
way.  However GHCHQ
can, after a 14day notification period, exercise the co-maintainance
and cut releases
(and upload them to hackage), should the maintainer not be able to do
so on his own
for various reasons.

I'd like to see this as an insurance policy for GHC continuous
development.  The only
alternative that I see would be that GHCHQ starts forking dependencies
and initiates
the hackage maintainer takeover protocol, which will cause additional
delays, and
incurs an extra burden on the GHC maintainers.

I hope we can all agree that libraries that end up being dependencies
of GHC should
be held in high regards and form the very foundation GHC is built
upon.  As such it
should be an honour to have GHCHQ being a co-maintainer for ones library, as it
signifies that importances of the library for the continuous development of GHC.

Again I don't expect much to change, except for GHCHQ becoming co-maintainers
for libraries GHC depends on. The baseline expectation will remain as
it is.  However we
will have ensured the frictionless development of GHC going forward.

Cheers,
  Moritz
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: 8.12 plans

2020-05-24 Thread John Ericson

In that case I should perhaps try to get in !1102, which makes all the 
fixed-sized prim types always available, in too?


I'm fine either way, but figure it's nice for any CPP to deal with the 
fixed and bignum numeric representation changing across a single 
release, and the schedule gives us time to patch important packages that 
need it between mid-June and the end of September.


Cheers,

John

On 5/23/20 6:49 AM, Sylvain Henry wrote:


Hi Ben,

ghc-bignum (!2231) is ready to be merged for 8.12. We are just waiting 
for bytestring/text maintainers to merge 2 simple patches.


Thanks,
Sylvain


On 05/05/2020 20:12, Ben Gamari wrote:

Hi everyone,

The time is again upon us to start thinking about release planning for
the next major release: GHC 8.12. In keeping with our 6-month release
schedule, I propose the following schedule:

  * Mid-June 2020: Aim to have all major features in the tree
  * Late-June 2020: Cut the ghc-8.12 branch
  * June - August 2020: 3 alpha releases
  * 1 September 2020: beta release
  * 25 September 2020: Final 8.12.1 release

So, if you have any major features which you would like to merge for
8.12, now is the time to start planning how to wrap them up in the next
month or so. As always, do let me know if you think this may be
problematic and we can discuss options.

Cheers,

- Ben


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Can someone help my test my MR on *BSD?

2019-09-12 Thread John Ericson

MR !1699 deletes some code in the compiler about passing -l flags for 
the RTS. This is a layer violation and hopefully a useless one; the RTS 
config file in the package db lists external library dependencies just 
like any other package so the -l flags should be included because that 
alone.


CI passes, so everything works on Linux, Darwin, and Windows. Ben Gamari 
wanted to double check the behavior on the *BSDs, since the HAVE_* 
configure results might reasonably vary there. I got a FreeBSD VM and 
started installing the requisite ports, but I've never used any BSD 
before and don't have much time to learn it---say how to stop to the TUI 
pop-ups that prevent me building everything overnight. If someone with 
an up-and-running development environment could quickly give this change 
a spin, that would be much appreciated.


Fingers crossed, and thanks in advanced,

John

P.S. If said BSD angel wants to contribute a machine for the official 
CI, so neither me or anyone else is tempted to bug them again, that 
would also be great :)


!1699: https://gitlab.haskell.org/ghc/ghc/merge_requests/1699/diffs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Installation failure building GHC targeting 64-bit Android

2017-02-09 Thread John Ericson

There error seems to be on the Nix side of things, after make install.
I'll report back once I know for sure.

On Thu, Feb 9, 2017 at 1:26 PM, John Ericson
<john_eric...@alumni.brown.edu> wrote:
> Hello all,
>
> I am working on Nixpkgs support for cross-compiling Haskell. GHC
> finishes building but fails during installation. As far as I can tell,
> the error seems to be that `make install` fails after it thinks it has
> installed haddock, looking for the installed one and not finding it.
>
> I've attached a build log and the top-level config.log. In a moment
> I'll push all relevant Nix expressions in case anyone wants to
> reproduce exactly.
>
> Thanks,
>
> John
>
> (aka Ericson2314 on GitHub; aka Sonarpulse on Freenode)
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

38 matches

Mail list logo