Re: Require -fexternal-interpreter support for future TH changes?

[Simon Marlow]

I've answered all these questions (I hope) in the form of new Notes and
signposts, please feel free to comment on this diff:
https://phabricator.haskell.org/D2358.  I'm happy to add more if needs be.

On 22 June 2016 at 16:39, Richard Eisenberg  wrote:

>
> On Jun 22, 2016, at 10:38 AM, Simon Marlow  wrote:
>
> > Thanks for the feedback Richard.  It's helpful to know which parts of
> the implementation you found non-obvious.  I will write some more Notes,
> but first:
>
> I think what would help most is a place that describes the little state
> machine that's going on here: the startTH / runTH / finishTH cycle, with
> possible callbacks in between. All RPC calls are blocking, yes? That
> certainly simplifies the matter. In the state machine, I think it would
> help to see clearly what causes the state transition (e.g. splice
> encountered during renaming, reify call made during splicing) and where the
> code is located (both source file/function and client vs server). There's
> just a bunch of moving parts here spread across a number of files and two
> processes.
>
> >
> > * On why we need a RemoteRef of an IORef - because the IORef lives on
> the server, so GHC has a RemoteRef to it.   Did you see Note [External GHCi
> pointers] in compiler/ghci/GHCi.hs?  Does that help?
>
> Somewhat. So the server is sending a pointer (the IORef) to the client?
> Presumably this pointer is only usable on the server, so the client just
> ends up sending it back at some point. From the client's point of view,
> it's just getting an opaque key that only the server can decode. Perhaps
> this would all become clearer with the overall map.
>
> >   Should I xref it from somewhere else?
>
> Perhaps from libraries/ghci/GHCi/RemoteTypes.hs, on the declaration of
> RemoteRef.
>
> Thanks!
> Richard
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Require -fexternal-interpreter support for future TH changes?

[Richard Eisenberg]

On Jun 22, 2016, at 10:38 AM, Simon Marlow  wrote:

> Thanks for the feedback Richard.  It's helpful to know which parts of the 
> implementation you found non-obvious.  I will write some more Notes, but 
> first:

I think what would help most is a place that describes the little state machine 
that's going on here: the startTH / runTH / finishTH cycle, with possible 
callbacks in between. All RPC calls are blocking, yes? That certainly 
simplifies the matter. In the state machine, I think it would help to see 
clearly what causes the state transition (e.g. splice encountered during 
renaming, reify call made during splicing) and where the code is located (both 
source file/function and client vs server). There's just a bunch of moving 
parts here spread across a number of files and two processes.

> 
> * On why we need a RemoteRef of an IORef - because the IORef lives on the 
> server, so GHC has a RemoteRef to it.   Did you see Note [External GHCi 
> pointers] in compiler/ghci/GHCi.hs?  Does that help?

Somewhat. So the server is sending a pointer (the IORef) to the client? 
Presumably this pointer is only usable on the server, so the client just ends 
up sending it back at some point. From the client's point of view, it's just 
getting an opaque key that only the server can decode. Perhaps this would all 
become clearer with the overall map.

>   Should I xref it from somewhere else?

Perhaps from libraries/ghci/GHCi/RemoteTypes.hs, on the declaration of 
RemoteRef.

Thanks!
Richard
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Require -fexternal-interpreter support for future TH changes?

[Simon Marlow]

One other thing:

On 22 June 2016 at 14:36, Richard Eisenberg <e...@cis.upenn.edu> wrote:

> It seems there are nice advantages to this approach, and so I'm fine
> requiring support from TH. But here is a comment I just posted on D2286
> requesting more comments in the code:
>
> ---
> I've just spent some time staring at GHCi/TH.hs and friends, and it's
> starting to make sense. Would it be possible to add some comments/Notes
> there? It would be great to know what is run on the client (GHC, right?)
> and what on the server (TH, right?). Also, why do we need a RemoteRef of
> an IORef? This, at first, strikes me as odd. I'm sure it makes great
> sense once I understand all the other moving parts, but it would be helpful
> to have a primer. I also find it a bit confusing that the Message type
> contains both messages to the server and messages from the server. For
> example, I would expect StartTH and Reify to live in different datatypes,
> because the two should never mix. Or am I horribly mistaken?
> ---
>
> Before we require -fexternal-interpreter (which seems to be the endgoal
> here), has anyone looked at performance implications? All the marshaling
> and unmarshaling has to take a toll.
>

Yes, I've looked at performance and done a fair bit of optimisation of the
current implementation, but mainly for GHCi, not so much for TH yet.  For
GHCi I think it was <10% of compile time, mainly in shipping the byte code
over the pipe.  I'll need to do some measurements of TH performance before
making it the default.  We can probably claw back quite a lot of perf by
switching from binary to something else (binary-cbor or store).

Cheers
Simon


>
> Richard
>
> On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs <
> ghc-devs@haskell.org> wrote:
>
> It’s a great start, thanks
>
> Simon
>
> *From:* Simon Marlow [mailto:marlo...@gmail.com]
> *Sent:* 22 June 2016 12:32
> *To:* Simon Peyton Jones <simo...@microsoft.com>
> *Cc:* ghc-devs@haskell.org
> *Subject:* Re: Require -fexternal-interpreter support for future TH
> changes?
>
>
> How's this?
> https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
>
> I don't want to go into too much detail in the wiki, because details are
> more likely to stay current if they're in Notes in the code.  There are
> already a few Notes (e.g.
> https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
> but if anything is confusing I'll happily add more Notes.
>
> Cheers
>
> Simon
>
>
>
> On 22 June 2016 at 11:50, Simon Marlow <marlo...@gmail.com> wrote:
>
> On 22 June 2016 at 11:37, Simon Peyton Jones <simo...@microsoft.com>
> wrote:
>
> I’m ok with this.   *It would certainly be great not to support TWO
> mechanisms indefinitely*.
>
> What are the disadvantages to committing to this path?  Would anyone even
> notice?
>
>
> Yes, people who are making changes to TH will need to ensure that their
> changes work with -fexternal-interpreter.  In some cases that might mean
> extra work, e.g. if we do TemplateHaskell/Introspective
> <https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective> then
> potentially the whole of HsSyn needs to be binary serializable.  That
> worries me quite a lot - THSyn is big but tractable, Generic deriving
> handled the generation of the binary instances easily enough, but HsSyn is
> another matter entirely.
>
>
> There are a lot of moving parts to the implementation, and I for one am
> utterly ignorant of how it all works.  I would love to see an
> implementation overview, either somewhere in the code or on a wiki page.
> Things like:
>
> · How, when, and where in the compiler is the separate process
> started?
>
> · How do the compiler and server communicate?  Unix pipes? Is it
> the same on Windows and Unix?
>
> · What is serialised, when, and how?  For example, GHC has some
> TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send
> that?  Or what?
>
> · How are external references managed.  E.g. if the code to be
> run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers
> to “a library implementing a message type…” but I don’t know what that
> library is called or where it lives.
>
>
>
> Yes, we should really have a page in the commentary with an overview of
> the pieces and the main implementation strategy. I'll write one.
> Cheers
> Simon
>
>
>
> Thanks
>
> Simon
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Simo

Re: Require -fexternal-interpreter support for future TH changes?

[Simon Marlow]

Yes - I wasn't planning on dropping support for the in-process interpreter,
for that reason.

I haven't thought about plugins very much, but aren't they always compiled
code?  We don't need an in-process interpreter for those, but we do need
in-process linking.  Maybe that doesn't simplify things much.

Cheers
Simon

On 22 June 2016 at 15:40, Edward Z. Yang  wrote:

> Hello Simon,
>
> I have no exception to having it be default and dropping the special
> case support for building profiled/dynamic so that TH works.  But
> I don't think support for loading code in-process for GHC should be
> dropped,
> c.f. Manuel's email
> https://mail.haskell.org/pipermail/ghc-devs/2015-November/010491.html
> and also the necessity to run code in-process for typechecking plugins,
> etc.
>
> Edward
>
> Excerpts from Simon Marlow's message of 2016-06-22 04:51:12 -0400:
> > *Background*
> >
> > A few months ago I added -fexternal-interpreter to GHC:
> >
> >- docs:
> >
> http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
> >- wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
> >
> > When -fexternal-interpreter is used, GHC runs interpreted code in a
> > separate subprocess, and communicates with it using binary messages over
> a
> > pipe.
> >
> > -fexternal-interpreter currently implements all of TH, quasi-quoting,
> > annotations, and all the GHCi features except for some features of the
> > debugger.  It is also now implemented on Windows, thanks to Tamar
> Christina.
> >
> > *Proposal*
> >
> > I'd like to propose that going forward we commit to maintaining full
> > support for -fexternal-interpreter, with a view to making it the default.
> >
> > Why?
> >
> >- -fexternal-interpreter will be a prerequisite for GHCJS support, so
> >maintaining full support for TH in -fexternal-interpreter will ensure
> that
> >everything that works with GHC works with GHCJS.
> >- We will be able to make simplifications in GHC and the build system
> >once -fexternal-interpreter is the default, because when compiling
> with
> >-prof or -dynamic we won't have to compile things twice any more.
> >- Ultimately we don't want to have two ways of doing everything,
> because
> >that's harder to maintain.
> >
> > How?
> >
> >- I'll make all the TH and quasi-quoting tests run with and without
> >-fexternal-interpreter, so it will break validate if one of these
> fails.
> >
> > *Why now?*
> >
> > There are some TH changes in the pipeline that will need special
> attention
> > to work with -fexternal-interpreter.  e.g.
> > https://phabricator.haskell.org/D2286 and
> > https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so
> I'd
> > like to raise it now so we can keep the issue in mind.
> >
> >
> > Cheers
> >
> > Simon
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Require -fexternal-interpreter support for future TH changes?

[Edward Z. Yang]

Hello Simon,

I have no exception to having it be default and dropping the special
case support for building profiled/dynamic so that TH works.  But
I don't think support for loading code in-process for GHC should be dropped,
c.f. Manuel's email
https://mail.haskell.org/pipermail/ghc-devs/2015-November/010491.html
and also the necessity to run code in-process for typechecking plugins,
etc.

Edward

Excerpts from Simon Marlow's message of 2016-06-22 04:51:12 -0400:
> *Background*
> 
> A few months ago I added -fexternal-interpreter to GHC:
> 
>- docs:
>
> http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
>- wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
> 
> When -fexternal-interpreter is used, GHC runs interpreted code in a
> separate subprocess, and communicates with it using binary messages over a
> pipe.
> 
> -fexternal-interpreter currently implements all of TH, quasi-quoting,
> annotations, and all the GHCi features except for some features of the
> debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
> 
> *Proposal*
> 
> I'd like to propose that going forward we commit to maintaining full
> support for -fexternal-interpreter, with a view to making it the default.
> 
> Why?
> 
>- -fexternal-interpreter will be a prerequisite for GHCJS support, so
>maintaining full support for TH in -fexternal-interpreter will ensure that
>everything that works with GHC works with GHCJS.
>- We will be able to make simplifications in GHC and the build system
>once -fexternal-interpreter is the default, because when compiling with
>-prof or -dynamic we won't have to compile things twice any more.
>- Ultimately we don't want to have two ways of doing everything, because
>that's harder to maintain.
> 
> How?
> 
>- I'll make all the TH and quasi-quoting tests run with and without
>-fexternal-interpreter, so it will break validate if one of these fails.
> 
> *Why now?*
> 
> There are some TH changes in the pipeline that will need special attention
> to work with -fexternal-interpreter.  e.g.
> https://phabricator.haskell.org/D2286 and
> https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd
> like to raise it now so we can keep the issue in mind.
> 
> 
> Cheers
> 
> Simon
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Require -fexternal-interpreter support for future TH changes?

[Simon Marlow]

Thanks for the feedback Richard.  It's helpful to know which parts of the
implementation you found non-obvious.  I will write some more Notes, but
first:

* On why we need a RemoteRef of an IORef - because the IORef lives on the
server, so GHC has a RemoteRef to it.   Did you see Note [External GHCi
pointers] in compiler/ghci/GHCi.hs?  Does that help?  Should I xref it from
somewhere else?

* We probably could separate Message into two types, it's a good idea.
I'll look into doing that.

Cheers
Simon

On 22 June 2016 at 14:36, Richard Eisenberg <e...@cis.upenn.edu> wrote:

> It seems there are nice advantages to this approach, and so I'm fine
> requiring support from TH. But here is a comment I just posted on D2286
> requesting more comments in the code:
>
> ---
> I've just spent some time staring at GHCi/TH.hs and friends, and it's
> starting to make sense. Would it be possible to add some comments/Notes
> there? It would be great to know what is run on the client (GHC, right?)
> and what on the server (TH, right?). Also, why do we need a RemoteRef of
> an IORef? This, at first, strikes me as odd. I'm sure it makes great
> sense once I understand all the other moving parts, but it would be helpful
> to have a primer. I also find it a bit confusing that the Message type
> contains both messages to the server and messages from the server. For
> example, I would expect StartTH and Reify to live in different datatypes,
> because the two should never mix. Or am I horribly mistaken?
> ---
>
> Before we require -fexternal-interpreter (which seems to be the endgoal
> here), has anyone looked at performance implications? All the marshaling
> and unmarshaling has to take a toll.
>
> Richard
>
> On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs <
> ghc-devs@haskell.org> wrote:
>
> It’s a great start, thanks
>
> Simon
>
> *From:* Simon Marlow [mailto:marlo...@gmail.com]
> *Sent:* 22 June 2016 12:32
> *To:* Simon Peyton Jones <simo...@microsoft.com>
> *Cc:* ghc-devs@haskell.org
> *Subject:* Re: Require -fexternal-interpreter support for future TH
> changes?
>
>
> How's this?
> https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
>
> I don't want to go into too much detail in the wiki, because details are
> more likely to stay current if they're in Notes in the code.  There are
> already a few Notes (e.g.
> https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
> but if anything is confusing I'll happily add more Notes.
>
> Cheers
>
> Simon
>
>
>
> On 22 June 2016 at 11:50, Simon Marlow <marlo...@gmail.com> wrote:
>
> On 22 June 2016 at 11:37, Simon Peyton Jones <simo...@microsoft.com>
> wrote:
>
> I’m ok with this.   *It would certainly be great not to support TWO
> mechanisms indefinitely*.
>
> What are the disadvantages to committing to this path?  Would anyone even
> notice?
>
>
> Yes, people who are making changes to TH will need to ensure that their
> changes work with -fexternal-interpreter.  In some cases that might mean
> extra work, e.g. if we do TemplateHaskell/Introspective
> <https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective> then
> potentially the whole of HsSyn needs to be binary serializable.  That
> worries me quite a lot - THSyn is big but tractable, Generic deriving
> handled the generation of the binary instances easily enough, but HsSyn is
> another matter entirely.
>
>
> There are a lot of moving parts to the implementation, and I for one am
> utterly ignorant of how it all works.  I would love to see an
> implementation overview, either somewhere in the code or on a wiki page.
> Things like:
>
> · How, when, and where in the compiler is the separate process
> started?
>
> · How do the compiler and server communicate?  Unix pipes? Is it
> the same on Windows and Unix?
>
> · What is serialised, when, and how?  For example, GHC has some
> TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send
> that?  Or what?
>
> · How are external references managed.  E.g. if the code to be
> run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers
> to “a library implementing a message type…” but I don’t know what that
> library is called or where it lives.
>
>
>
> Yes, we should really have a page in the commentary with an overview of
> the pieces and the main implementation strategy. I'll write one.
> Cheers
> Simon
>
>
>
> Thanks
>
> Simon
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@hask

Re: Require -fexternal-interpreter support for future TH changes?

[Richard Eisenberg]

It seems there are nice advantages to this approach, and so I'm fine requiring 
support from TH. But here is a comment I just posted on D2286 requesting more 
comments in the code:

---
I've just spent some time staring at GHCi/TH.hs and friends, and it's starting 
to make sense. Would it be possible to add some comments/Notes there? It would 
be great to know what is run on the client (GHC, right?) and what on the server 
(TH, right?). Also, why do we need a RemoteRef of an IORef? This, at first, 
strikes me as odd. I'm sure it makes great sense once I understand all the 
other moving parts, but it would be helpful to have a primer. I also find it a 
bit confusing that the Message type contains both messages to the server and 
messages from the server. For example, I would expect StartTH and Reify to live 
in different datatypes, because the two should never mix. Or am I horribly 
mistaken?
---

Before we require -fexternal-interpreter (which seems to be the endgoal here), 
has anyone looked at performance implications? All the marshaling and 
unmarshaling has to take a toll.

Richard

On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs 
<ghc-devs@haskell.org> wrote:

> It’s a great start, thanks
>  
> Simon
>  
> From: Simon Marlow [mailto:marlo...@gmail.com] 
> Sent: 22 June 2016 12:32
> To: Simon Peyton Jones <simo...@microsoft.com>
> Cc: ghc-devs@haskell.org
> Subject: Re: Require -fexternal-interpreter support for future TH changes?
>  
> How's this? 
> https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
> 
> I don't want to go into too much detail in the wiki, because details are more 
> likely to stay current if they're in Notes in the code.  There are already a 
> few Notes 
> (e.g.https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
>  but if anything is confusing I'll happily add more Notes.
> 
> Cheers
> 
> Simon
> 
>  
> 
> On 22 June 2016 at 11:50, Simon Marlow <marlo...@gmail.com> wrote:
> 
> On 22 June 2016 at 11:37, Simon Peyton Jones <simo...@microsoft.com> wrote:
> 
> I’m ok with this.   It would certainly be great not to support TWO mechanisms 
> indefinitely.
>  
> What are the disadvantages to committing to this path?  Would anyone even 
> notice?
>  
> Yes, people who are making changes to TH will need to ensure that their 
> changes work with -fexternal-interpreter.  In some cases that might mean 
> extra work, e.g. if we do TemplateHaskell/Introspective then potentially the 
> whole of HsSyn needs to be binary serializable.  That worries me quite a lot 
> - THSyn is big but tractable, Generic deriving handled the generation of the 
> binary instances easily enough, but HsSyn is another matter entirely.
>  
> There are a lot of moving parts to the implementation, and I for one am 
> utterly ignorant of how it all works.  I would love to see an implementation 
> overview, either somewhere in the code or on a wiki page.  Things like:
> · How, when, and where in the compiler is the separate process 
> started?
> 
> · How do the compiler and server communicate?  Unix pipes? Is it the 
> same on Windows and Unix?
> 
> · What is serialised, when, and how?  For example, GHC has some TH 
> code to run.  Do we send a syntax tree?  Or compile to bytecode and send 
> that?  Or what?
> 
> · How are external references managed.  E.g. if the code to be run 
> refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
> 
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to 
> “a library implementing a message type…” but I don’t know what that library 
> is called or where it lives.
>  
> Yes, we should really have a page in the commentary with an overview of the 
> pieces and the main implementation strategy. I'll write one.
> 
> Cheers
> Simon
>  
>  
> Thanks
>  
> Simon
>  
> From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Simon Marlow
> Sent: 22 June 2016 09:51
> To: ghc-devs@haskell.org
> Subject: Require -fexternal-interpreter support for future TH changes?
>  
> Background
> 
> A few months ago I added -fexternal-interpreter to GHC:
> 
> docs: 
> http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
> wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
> When -fexternal-interpreter is used, GHC runs interpreted code in a separate 
> subprocess, and communicates with it using binary messages over a pipe.
> 
> -fexternal-interpreter currently implements all of TH, quasi-quoting, 
> annotations, and all the GHCi features except for some f

RE: Require -fexternal-interpreter support for future TH changes?

[Simon Peyton Jones via ghc-devs]

It’s a great start, thanks

Simon

From: Simon Marlow [mailto:marlo...@gmail.com]
Sent: 22 June 2016 12:32
To: Simon Peyton Jones <simo...@microsoft.com>
Cc: ghc-devs@haskell.org
Subject: Re: Require -fexternal-interpreter support for future TH changes?

How's this? 
https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
I don't want to go into too much detail in the wiki, because details are more 
likely to stay current if they're in Notes in the code.  There are already a 
few Notes (e.g. 
https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
 but if anything is confusing I'll happily add more Notes.
Cheers
Simon

On 22 June 2016 at 11:50, Simon Marlow 
<marlo...@gmail.com<mailto:marlo...@gmail.com>> wrote:
On 22 June 2016 at 11:37, Simon Peyton Jones 
<simo...@microsoft.com<mailto:simo...@microsoft.com>> wrote:
I’m ok with this.   It would certainly be great not to support TWO mechanisms 
indefinitely.

What are the disadvantages to committing to this path?  Would anyone even 
notice?

Yes, people who are making changes to TH will need to ensure that their changes 
work with -fexternal-interpreter.  In some cases that might mean extra work, 
e.g. if we do 
TemplateHaskell/Introspective<https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective>
 then potentially the whole of HsSyn needs to be binary serializable.  That 
worries me quite a lot - THSyn is big but tractable, Generic deriving handled 
the generation of the binary instances easily enough, but HsSyn is another 
matter entirely.

There are a lot of moving parts to the implementation, and I for one am utterly 
ignorant of how it all works.  I would love to see an implementation overview, 
either somewhere in the code or on a wiki page.  Things like:

• How, when, and where in the compiler is the separate process started?

• How do the compiler and server communicate?  Unix pipes? Is it the 
same on Windows and Unix?

• What is serialised, when, and how?  For example, GHC has some TH code 
to run.  Do we send a syntax tree?  Or compile to bytecode and send that?  Or 
what?

• How are external references managed.  E.g. if the code to be run 
refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a 
library implementing a message type…” but I don’t know what that library is 
called or where it lives.

Yes, we should really have a page in the commentary with an overview of the 
pieces and the main implementation strategy. I'll write one.
Cheers
Simon


Thanks

Simon

From: ghc-devs 
[mailto:ghc-devs-boun...@haskell.org<mailto:ghc-devs-boun...@haskell.org>] On 
Behalf Of Simon Marlow
Sent: 22 June 2016 09:51
To: ghc-devs@haskell.org<mailto:ghc-devs@haskell.org>
Subject: Require -fexternal-interpreter support for future TH changes?

Background
A few months ago I added -fexternal-interpreter to GHC:

  *   docs: 
http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter<https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
  *   wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
When -fexternal-interpreter is used, GHC runs interpreted code in a separate 
subprocess, and communicates with it using binary messages over a pipe.
-fexternal-interpreter currently implements all of TH, quasi-quoting, 
annotations, and all the GHCi features except for some features of the 
debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
Proposal
I'd like to propose that going forward we commit to maintaining full support 
for -fexternal-interpreter, with a view to making it the default.
Why?

  *   -fexternal-interpreter will be a prerequisite for GHCJS support, so 
maintaining full support for TH in -fexternal-interpreter will ensure that 
everything that works with GHC works with GHCJS.
  *   We will be able to make simplifications in GHC and the build system once 
-fexternal-interpreter is the default, because when compiling with -prof or 
-dynamic we won't have to compile things twice any more.
  *   Ultimately we don't want to have two ways of doing everything, because 
that's harder to maintain.
How?

  *   I'll make all the TH and quasi-quoting tests run with and without 
-fexternal-interpreter, so it will break validate if one of these fails.

Why now?

There are some TH changes in the pipeline that will need special attention to 
work with -fexternal-interpreter.  e.g. https://phabricator.haskell.org/D2286 
and

Re: Require -fexternal-interpreter support for future TH changes?

[Simon Marlow]

How's this?
https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter

I don't want to go into too much detail in the wiki, because details are
more likely to stay current if they're in Notes in the code.  There are
already a few Notes (e.g.
https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
but if anything is confusing I'll happily add more Notes.

Cheers
Simon

On 22 June 2016 at 11:50, Simon Marlow <marlo...@gmail.com> wrote:

> On 22 June 2016 at 11:37, Simon Peyton Jones <simo...@microsoft.com>
> wrote:
>
>> I’m ok with this.   *It would certainly be great not to support TWO
>> mechanisms indefinitely*.
>>
>>
>>
>> What are the disadvantages to committing to this path?  Would anyone even
>> notice?
>>
>
> Yes, people who are making changes to TH will need to ensure that their
> changes work with -fexternal-interpreter.  In some cases that might mean
> extra work, e.g. if we do TemplateHaskell/Introspective
> <https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective>
> then potentially the whole of HsSyn needs to be binary serializable.  That
> worries me quite a lot - THSyn is big but tractable, Generic deriving
> handled the generation of the binary instances easily enough, but HsSyn is
> another matter entirely.
>
>
>> There are a lot of moving parts to the implementation, and I for one am
>> utterly ignorant of how it all works.  I would love to see an
>> implementation overview, either somewhere in the code or on a wiki page.
>> Things like:
>>
>> · How, when, and where in the compiler is the separate process
>> started?
>>
>> · How do the compiler and server communicate?  Unix pipes? Is it
>> the same on Windows and Unix?
>>
>> · What is serialised, when, and how?  For example, GHC has some
>> TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send
>> that?  Or what?
>>
>> · How are external references managed.  E.g. if the code to be
>> run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
>>
>> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers
>> to “a library implementing a message type…” but I don’t know what that
>> library is called or where it lives.
>>
>
> Yes, we should really have a page in the commentary with an overview of
> the pieces and the main implementation strategy. I'll write one.
>
> Cheers
> Simon
>
>
>>
>>
>> Thanks
>>
>>
>>
>> Simon
>>
>>
>>
>> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Simon
>> Marlow
>> *Sent:* 22 June 2016 09:51
>> *To:* ghc-devs@haskell.org
>> *Subject:* Require -fexternal-interpreter support for future TH changes?
>>
>>
>>
>> *Background*
>>
>> A few months ago I added -fexternal-interpreter to GHC:
>>
>>- docs:
>>
>> http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
>>
>> <https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
>>- wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
>>
>> When -fexternal-interpreter is used, GHC runs interpreted code in a
>> separate subprocess, and communicates with it using binary messages over a
>> pipe.
>>
>> -fexternal-interpreter currently implements all of TH, quasi-quoting,
>> annotations, and all the GHCi features except for some features of the
>> debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
>>
>> *Proposal*
>>
>> I'd like to propose that going forward we commit to maintaining full
>> support for -fexternal-interpreter, with a view to making it the default.
>>
>> Why?
>>
>>- -fexternal-interpreter will be a prerequisite for GHCJS support, so
>>maintaining full support for TH in -fexternal-interpreter will ensure that
>>everything that works with GHC works with GHCJS.
>>- We will be able to make simplifications in GHC and the build system
>>once -fexternal-interpreter is the default, because when compiling with
>>-prof or -dynamic we won't have to compile things twice any more.
>>- Ultimately we don't

Re: Require -fexternal-interpreter support for future TH changes?

[Simon Marlow]

On 22 June 2016 at 11:37, Simon Peyton Jones <simo...@microsoft.com> wrote:

> I’m ok with this.   *It would certainly be great not to support TWO
> mechanisms indefinitely*.
>
>
>
> What are the disadvantages to committing to this path?  Would anyone even
> notice?
>

Yes, people who are making changes to TH will need to ensure that their
changes work with -fexternal-interpreter.  In some cases that might mean
extra work, e.g. if we do TemplateHaskell/Introspective
<https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective> then
potentially the whole of HsSyn needs to be binary serializable.  That
worries me quite a lot - THSyn is big but tractable, Generic deriving
handled the generation of the binary instances easily enough, but HsSyn is
another matter entirely.


> There are a lot of moving parts to the implementation, and I for one am
> utterly ignorant of how it all works.  I would love to see an
> implementation overview, either somewhere in the code or on a wiki page.
> Things like:
>
> · How, when, and where in the compiler is the separate process
> started?
>
> · How do the compiler and server communicate?  Unix pipes? Is it
> the same on Windows and Unix?
>
> · What is serialised, when, and how?  For example, GHC has some
> TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send
> that?  Or what?
>
> · How are external references managed.  E.g. if the code to be
> run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
>
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers
> to “a library implementing a message type…” but I don’t know what that
> library is called or where it lives.
>

Yes, we should really have a page in the commentary with an overview of the
pieces and the main implementation strategy. I'll write one.

Cheers
Simon


>
>
> Thanks
>
>
>
> Simon
>
>
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Simon
> Marlow
> *Sent:* 22 June 2016 09:51
> *To:* ghc-devs@haskell.org
> *Subject:* Require -fexternal-interpreter support for future TH changes?
>
>
>
> *Background*
>
> A few months ago I added -fexternal-interpreter to GHC:
>
>- docs:
>
> http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
>
> <https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
>- wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
>
> When -fexternal-interpreter is used, GHC runs interpreted code in a
> separate subprocess, and communicates with it using binary messages over a
> pipe.
>
> -fexternal-interpreter currently implements all of TH, quasi-quoting,
> annotations, and all the GHCi features except for some features of the
> debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
>
> *Proposal*
>
> I'd like to propose that going forward we commit to maintaining full
> support for -fexternal-interpreter, with a view to making it the default.
>
> Why?
>
>- -fexternal-interpreter will be a prerequisite for GHCJS support, so
>maintaining full support for TH in -fexternal-interpreter will ensure that
>everything that works with GHC works with GHCJS.
>- We will be able to make simplifications in GHC and the build system
>once -fexternal-interpreter is the default, because when compiling with
>-prof or -dynamic we won't have to compile things twice any more.
>- Ultimately we don't want to have two ways of doing everything,
>because that's harder to maintain.
>
> How?
>
>- I'll make all the TH and quasi-quoting tests run with and without
>-fexternal-interpreter, so it will break validate if one of these fails.
>
> *Why now?*
>
> There are some TH changes in the pipeline that will need special attention
> to work with -fexternal-interpreter.  e.g.
> https://phabricator.haskell.org/D2286 and
> https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so
> I'd like to raise it now so we can keep the issue in mind.
>
>
>
> Cheers
>
> Simon
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: Require -fexternal-interpreter support for future TH changes?

[Simon Peyton Jones via ghc-devs]

I’m ok with this.   It would certainly be great not to support TWO mechanisms 
indefinitely.

What are the disadvantages to committing to this path?  Would anyone even 
notice?

There are a lot of moving parts to the implementation, and I for one am utterly 
ignorant of how it all works.  I would love to see an implementation overview, 
either somewhere in the code or on a wiki page.  Things like:

· How, when, and where in the compiler is the separate process started?

· How do the compiler and server communicate?  Unix pipes? Is it the 
same on Windows and Unix?

· What is serialised, when, and how?  For example, GHC has some TH code 
to run.  Do we send a syntax tree?  Or compile to bytecode and send that?  Or 
what?

· How are external references managed.  E.g. if the code to be run 
refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a 
library implementing a message type…” but I don’t know what that library is 
called or where it lives.

Thanks

Simon

From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Simon Marlow
Sent: 22 June 2016 09:51
To: ghc-devs@haskell.org
Subject: Require -fexternal-interpreter support for future TH changes?

Background
A few months ago I added -fexternal-interpreter to GHC:

  *   docs: 
http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter<https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
  *   wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
When -fexternal-interpreter is used, GHC runs interpreted code in a separate 
subprocess, and communicates with it using binary messages over a pipe.
-fexternal-interpreter currently implements all of TH, quasi-quoting, 
annotations, and all the GHCi features except for some features of the 
debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
Proposal
I'd like to propose that going forward we commit to maintaining full support 
for -fexternal-interpreter, with a view to making it the default.
Why?

  *   -fexternal-interpreter will be a prerequisite for GHCJS support, so 
maintaining full support for TH in -fexternal-interpreter will ensure that 
everything that works with GHC works with GHCJS.
  *   We will be able to make simplifications in GHC and the build system once 
-fexternal-interpreter is the default, because when compiling with -prof or 
-dynamic we won't have to compile things twice any more.
  *   Ultimately we don't want to have two ways of doing everything, because 
that's harder to maintain.
How?

  *   I'll make all the TH and quasi-quoting tests run with and without 
-fexternal-interpreter, so it will break validate if one of these fails.

Why now?

There are some TH changes in the pipeline that will need special attention to 
work with -fexternal-interpreter.  e.g. https://phabricator.haskell.org/D2286 
and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd 
like to raise it now so we can keep the issue in mind.



Cheers

Simon
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs