Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Shayne Fletcher via ghc-devs]

Excellent work Sebastian. I confirm that the situation is resolved by
https://gitlab.haskell.org/ghc/ghc/merge_requests/1743.

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf  wrote:

> Hi Shayne,
>
> Sorry to hear that! We didn't consider modularity at all and I would be
> happy to try to refactor in a way that would allow `ghc-lib-parser` to be
> properly separated again.
> I'm fairly certain that I didn't directly touch anything parser related,
> but apparently the new cyclic import of PmOracle within TcRnTypes (which is
> also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
> I'll see if I can fix that tomorrow, if only by extracting a separate
> `Types`-style module.
>
> Cheers,
> Sebastian
>
> Am Mo., 16. Sept. 2019 um 22:04 Uhr schrieb Shayne Fletcher via ghc-devs <
> ghc-devs@haskell.org>:
>
>> Some time back, the `ghc-lib` project split into two targets :
>> `ghc-lib-parser` for those projects that just need to produce syntax trees
>> and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining
>> modules for projects that need to go on and distill parse trees to Core.
>> The idea of course was to reduce build times for tools like hlint that only
>> need parse trees.
>>
>> Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today
>> after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape
>> information in `PmOracle`", `ghc-lib-parser` now needs 543 files and
>> `ghc-lib` gets just 25.
>>
>> That may be just bad luck for `ghc-lib-parser` and the way it has to be
>> but I thought I should at least mention the knock-on effect of this change
>> on the modularity of the GHC API in case this consequence hasn't been
>> considered?
>>
>> --
>> *Shayne Fletcher*
>> Language Engineer */* +1 917 699 7663
>> *Digital Asset* , creators of *DAML
>> *
>>
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.digitalasset.com/emaildisclaimer.html. If you are not the
>> intended recipient, please delete this message.
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>

-- 
*Shayne Fletcher*
Language Engineer */* +1 917 699 7663
*Digital Asset* , creators of *DAML
*

-- 
This message, and any attachments, is for the intended recipient(s) only, 
may contain information that is privileged, confidential and/or proprietary 
and subject to important terms and conditions available at 
http://www.digitalasset.com/emaildisclaimer.html 
. If you are not the 
intended recipient, please delete this message.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Shayne Fletcher via ghc-devs]

Getting you some raw data to look at but first results indicate at least
that `DsMonad` is pulling in `nativeGen`, `llvmGen`, `stgToCmm`,
`simplCore`, `hieFile` and so on.

On Tue, Sep 17, 2019 at 12:05 PM Shayne Fletcher 
wrote:

> Seeing what I can do.
>
> On Tue, Sep 17, 2019 at 10:31 AM Sebastian Graf 
> wrote:
>
>> Shayne,
>>
>> out of curiosity, could you find out which of the three modules DsMonad,
>> FamInst and TcSimplify lead to the blowup? If it's not too much of a
>> hassle, that is.
>> These are the only imports of PmOracle that aren't already exported from
>> ghc-lib-parser.
>>
>> Cheers,
>> Sebastian
>>
>> Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher <
>> shayne.fletc...@daml.com>:
>>
>>> Hi Sebastian,
>>>
>>> On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf 
>>> wrote:
>>>
 Hi Shayne,

 Sorry to hear that! We didn't consider modularity at all and I would be
 happy to try to refactor in a way that would allow `ghc-lib-parser` to be
 properly separated again.
 I'm fairly certain that I didn't directly touch anything parser
 related, but apparently the new cyclic import of PmOracle within TcRnTypes
 (which is also exposed from `ghc-lib-parser`) pulled in the other half of
 GHC.
 I'll see if I can fix that tomorrow, if only by extracting a separate
 `Types`-style module.


>>> That sounds awesome. Tremendous. Thank-you! Please feel free to reach
>>> out to me if there's anything I can do to help your analysis[*]!
>>>
>>> [*] For the record, the procedure for calculating the `ghc-lib-parser`
>>> modules is a little complicated by there needing to be some generated
>>> equivalents of `.hsc` files present for this to work but the procedure is
>>> at the end of the day just `ghc -M` invoked over `Parser.hs`.
>>>
>>>
 Cheers,
 Sebastian

>>>
>>> Fingers crossed and all the best!
>>>
>>> --
>>> *Shayne Fletcher*
>>> Language Engineer */* +1 917 699 7663
>>> *Digital Asset* , creators of *DAML
>>> *
>>>
>>> This message, and any attachments, is for the intended recipient(s)
>>> only, may contain information that is privileged, confidential and/or
>>> proprietary and subject to important terms and conditions available at
>>> http://www.digitalasset.com/emaildisclaimer.html. If you are not the
>>> intended recipient, please delete this message.
>>
>>
>
> --
> *Shayne Fletcher*
> Language Engineer */* +1 917 699 7663
> *Digital Asset* , creators of *DAML
> *
>


-- 
*Shayne Fletcher*
Language Engineer */* +1 917 699 7663
*Digital Asset* , creators of *DAML
*

-- 
This message, and any attachments, is for the intended recipient(s) only, 
may contain information that is privileged, confidential and/or proprietary 
and subject to important terms and conditions available at 
http://www.digitalasset.com/emaildisclaimer.html 
. If you are not the 
intended recipient, please delete this message.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Shayne Fletcher via ghc-devs]

Seeing what I can do.

On Tue, Sep 17, 2019 at 10:31 AM Sebastian Graf  wrote:

> Shayne,
>
> out of curiosity, could you find out which of the three modules DsMonad,
> FamInst and TcSimplify lead to the blowup? If it's not too much of a
> hassle, that is.
> These are the only imports of PmOracle that aren't already exported from
> ghc-lib-parser.
>
> Cheers,
> Sebastian
>
> Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher <
> shayne.fletc...@daml.com>:
>
>> Hi Sebastian,
>>
>> On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf 
>> wrote:
>>
>>> Hi Shayne,
>>>
>>> Sorry to hear that! We didn't consider modularity at all and I would be
>>> happy to try to refactor in a way that would allow `ghc-lib-parser` to be
>>> properly separated again.
>>> I'm fairly certain that I didn't directly touch anything parser related,
>>> but apparently the new cyclic import of PmOracle within TcRnTypes (which is
>>> also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
>>> I'll see if I can fix that tomorrow, if only by extracting a separate
>>> `Types`-style module.
>>>
>>>
>> That sounds awesome. Tremendous. Thank-you! Please feel free to reach out
>> to me if there's anything I can do to help your analysis[*]!
>>
>> [*] For the record, the procedure for calculating the `ghc-lib-parser`
>> modules is a little complicated by there needing to be some generated
>> equivalents of `.hsc` files present for this to work but the procedure is
>> at the end of the day just `ghc -M` invoked over `Parser.hs`.
>>
>>
>>> Cheers,
>>> Sebastian
>>>
>>
>> Fingers crossed and all the best!
>>
>> --
>> *Shayne Fletcher*
>> Language Engineer */* +1 917 699 7663
>> *Digital Asset* , creators of *DAML
>> *
>>
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.digitalasset.com/emaildisclaimer.html. If you are not the
>> intended recipient, please delete this message.
>
>

-- 
*Shayne Fletcher*
Language Engineer */* +1 917 699 7663
*Digital Asset* , creators of *DAML
*

-- 
This message, and any attachments, is for the intended recipient(s) only, 
may contain information that is privileged, confidential and/or proprietary 
and subject to important terms and conditions available at 
http://www.digitalasset.com/emaildisclaimer.html 
. If you are not the 
intended recipient, please delete this message.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Sebastian Graf]

Shayne,

out of curiosity, could you find out which of the three modules DsMonad,
FamInst and TcSimplify lead to the blowup? If it's not too much of a
hassle, that is.
These are the only imports of PmOracle that aren't already exported from
ghc-lib-parser.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher <
shayne.fletc...@daml.com>:

> Hi Sebastian,
>
> On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf 
> wrote:
>
>> Hi Shayne,
>>
>> Sorry to hear that! We didn't consider modularity at all and I would be
>> happy to try to refactor in a way that would allow `ghc-lib-parser` to be
>> properly separated again.
>> I'm fairly certain that I didn't directly touch anything parser related,
>> but apparently the new cyclic import of PmOracle within TcRnTypes (which is
>> also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
>> I'll see if I can fix that tomorrow, if only by extracting a separate
>> `Types`-style module.
>>
>>
> That sounds awesome. Tremendous. Thank-you! Please feel free to reach out
> to me if there's anything I can do to help your analysis[*]!
>
> [*] For the record, the procedure for calculating the `ghc-lib-parser`
> modules is a little complicated by there needing to be some generated
> equivalents of `.hsc` files present for this to work but the procedure is
> at the end of the day just `ghc -M` invoked over `Parser.hs`.
>
>
>> Cheers,
>> Sebastian
>>
>
> Fingers crossed and all the best!
>
> --
> *Shayne Fletcher*
> Language Engineer */* +1 917 699 7663
> *Digital Asset* , creators of *DAML
> *
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.digitalasset.com/emaildisclaimer.html. If you are not the
> intended recipient, please delete this message.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Sebastian Graf]

Hmm. The issue of build parallelism is kind-of orthogonal to .hs-boot
files/circular imports.

The impact of .hs-boot files on build parallelism depends on their
announced imports; at least in my semantic model of them (which might be
incorrect) they just break cycles.
The degree of parallelism is visible as a property of this DAG where cycles
are broken through insertion of .hs-boot nodes.

The issue here is one of transitive dependencies, which is a property on
the original cyclic dependency graph.

So while we have the dependency graph

HsExpr <--> TcRnTypes <--> PmOracle

This is of no issue for build parallelism, because the .hs-boot files
remove "feedback vertices" from the graph, rendering it acyclic:

TcRnTypes.hs-boot <- HsExpr <- TcRnTypes <- PmOracle
PmOracle.hs-boot <- TcRnTypes

Yet a package including HsExpr must also include PmOracle to compile.

The parallelism issue is best tackled by hadrian based profiles, on which
we could compute try to come up with the optimal schedule and compare it
across multiple CI runs.
But the dependency issue here is rather specific to the set of modules that
should be included in ghc-lib-parser, I'm afraid, and I don't see how to
solve it in a general way other than pre-computing the set of transitive
dependencies for specific blessed modules. Which might be a thing we
want... One set for each major entry point to the compiler pipeline, for
example.

Am Di., 17. Sept. 2019 um 10:19 Uhr schrieb Matthew Pickering <
matthewtpicker...@gmail.com>:

> This is precisely the problem I was worried about in April.
>
> https://mail.haskell.org/pipermail/ghc-devs/2019-April/017493.html
>
> Any fix should ensure adding a test to make sure it doesn't happen again.
>
> Cheers,
>
> Matt
>
> On Mon, Sep 16, 2019 at 10:38 PM Shayne Fletcher via ghc-devs
>  wrote:
> >
> > Hi Sebastian,
> >
> > On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf 
> wrote:
> >>
> >> Hi Shayne,
> >>
> >> Sorry to hear that! We didn't consider modularity at all and I would be
> happy to try to refactor in a way that would allow `ghc-lib-parser` to be
> properly separated again.
> >> I'm fairly certain that I didn't directly touch anything parser
> related, but apparently the new cyclic import of PmOracle within TcRnTypes
> (which is also exposed from `ghc-lib-parser`) pulled in the other half of
> GHC.
> >> I'll see if I can fix that tomorrow, if only by extracting a separate
> `Types`-style module.
> >>
> >
> > That sounds awesome. Tremendous. Thank-you! Please feel free to reach
> out to me if there's anything I can do to help your analysis[*]!
> >
> > [*] For the record, the procedure for calculating the `ghc-lib-parser`
> modules is a little complicated by there needing to be some generated
> equivalents of `.hsc` files present for this to work but the procedure is
> at the end of the day just `ghc -M` invoked over `Parser.hs`.
> >
> >>
> >> Cheers,
> >> Sebastian
> >
> >
> > Fingers crossed and all the best!
> >
> > --
> > Shayne Fletcher
> > Language Engineer / +1 917 699 7663
> > Digital Asset, creators of DAML
> >
> > This message, and any attachments, is for the intended recipient(s)
> only, may contain information that is privileged, confidential and/or
> proprietary and subject to important terms and conditions available at
> http://www.digitalasset.com/emaildisclaimer.html. If you are not the
> intended recipient, please delete this
> message.___
> > ghc-devs mailing list
> > ghc-devs@haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Matthew Pickering]

This is precisely the problem I was worried about in April.

https://mail.haskell.org/pipermail/ghc-devs/2019-April/017493.html

Any fix should ensure adding a test to make sure it doesn't happen again.

Cheers,

Matt

On Mon, Sep 16, 2019 at 10:38 PM Shayne Fletcher via ghc-devs
 wrote:
>
> Hi Sebastian,
>
> On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf  wrote:
>>
>> Hi Shayne,
>>
>> Sorry to hear that! We didn't consider modularity at all and I would be 
>> happy to try to refactor in a way that would allow `ghc-lib-parser` to be 
>> properly separated again.
>> I'm fairly certain that I didn't directly touch anything parser related, but 
>> apparently the new cyclic import of PmOracle within TcRnTypes (which is also 
>> exposed from `ghc-lib-parser`) pulled in the other half of GHC.
>> I'll see if I can fix that tomorrow, if only by extracting a separate 
>> `Types`-style module.
>>
>
> That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to 
> me if there's anything I can do to help your analysis[*]!
>
> [*] For the record, the procedure for calculating the `ghc-lib-parser` 
> modules is a little complicated by there needing to be some generated 
> equivalents of `.hsc` files present for this to work but the procedure is at 
> the end of the day just `ghc -M` invoked over `Parser.hs`.
>
>>
>> Cheers,
>> Sebastian
>
>
> Fingers crossed and all the best!
>
> --
> Shayne Fletcher
> Language Engineer / +1 917 699 7663
> Digital Asset, creators of DAML
>
> This message, and any attachments, is for the intended recipient(s) only, may 
> contain information that is privileged, confidential and/or proprietary and 
> subject to important terms and conditions available at 
> http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended 
> recipient, please delete this 
> message.___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Shayne Fletcher via ghc-devs]

Hi Sebastian,

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf  wrote:

> Hi Shayne,
>
> Sorry to hear that! We didn't consider modularity at all and I would be
> happy to try to refactor in a way that would allow `ghc-lib-parser` to be
> properly separated again.
> I'm fairly certain that I didn't directly touch anything parser related,
> but apparently the new cyclic import of PmOracle within TcRnTypes (which is
> also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
> I'll see if I can fix that tomorrow, if only by extracting a separate
> `Types`-style module.
>
>
That sounds awesome. Tremendous. Thank-you! Please feel free to reach out
to me if there's anything I can do to help your analysis[*]!

[*] For the record, the procedure for calculating the `ghc-lib-parser`
modules is a little complicated by there needing to be some generated
equivalents of `.hsc` files present for this to work but the procedure is
at the end of the day just `ghc -M` invoked over `Parser.hs`.


> Cheers,
> Sebastian
>

Fingers crossed and all the best!

-- 
*Shayne Fletcher*
Language Engineer */* +1 917 699 7663
*Digital Asset* , creators of *DAML
*

-- 
This message, and any attachments, is for the intended recipient(s) only, 
may contain information that is privileged, confidential and/or proprietary 
and subject to important terms and conditions available at 
http://www.digitalasset.com/emaildisclaimer.html 
. If you are not the 
intended recipient, please delete this message.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

[Sebastian Graf]

Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be
happy to try to refactor in a way that would allow `ghc-lib-parser` to be
properly separated again.
I'm fairly certain that I didn't directly touch anything parser related,
but apparently the new cyclic import of PmOracle within TcRnTypes (which is
also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate
`Types`-style module.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:04 Uhr schrieb Shayne Fletcher via ghc-devs <
ghc-devs@haskell.org>:

> Some time back, the `ghc-lib` project split into two targets :
> `ghc-lib-parser` for those projects that just need to produce syntax trees
> and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining
> modules for projects that need to go on and distill parse trees to Core.
> The idea of course was to reduce build times for tools like hlint that only
> need parse trees.
>
> Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today
> after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape
> information in `PmOracle`", `ghc-lib-parser` now needs 543 files and
> `ghc-lib` gets just 25.
>
> That may be just bad luck for `ghc-lib-parser` and the way it has to be
> but I thought I should at least mention the knock-on effect of this change
> on the modularity of the GHC API in case this consequence hasn't been
> considered?
>
> --
> *Shayne Fletcher*
> Language Engineer */* +1 917 699 7663
> *Digital Asset* , creators of *DAML
> *
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.digitalasset.com/emaildisclaimer.html. If you are not the
> intended recipient, please delete this message.
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs