Re: Reinstallable - base

[Moritz Angermann]

Something I haven’t gotten around to but only preliminary experiments with
is dynamically built iserv binaries.

Using -fexternal-interpreter can decouple the symbols the interpreter sees
and those the compiler sees (They can even be of different architectures).
iserv could be linked against the base the project wants to use, whereas
GHC itself could use a different base. I’m not sure this covers everything,
but it covers at least the case where we don’t need to load two different
packages into the same process.

Wrt to TH, I’m a bit behind on reading all the prior work to solve this,
but conceptually I still believe template-haskell itself should not expose
the internal ast, but only a combinator API to it.

Regarding DSO’s: let’s please not make the existence of DSO a hard
dependency. There are platforms for which we don’t have DSO capabilities,
and where we are forced to use the in-memory loader and linker.

On Wed, 18 Oct 2023 at 4:17 AM, Simon Peyton Jones <
simon.peytonjo...@gmail.com> wrote:

> (Meta-question: on reflection, would this discussion perhaps be better on
> a ticket? But where?  GHC's repo?  Or HF's?)
>
> The difficulty is that, as a normal Haskell library, ghc itself will be
>> compiled against a particular verson of base. Then when Template Haskell is
>> used (with the internal interpreter), code will be dynamically loaded into
>> a process that already has symbols for ghc's version of base, which means
>> it is not safe for the code to depend on a different version of base.
>
>
> I'm not understanding the difficulty yet.
>
> Let's say that
>
>- An old library mylib (which uses TH) depends on base-4.7.
>- A new GHC, say GHC 9.10, depends on a newer version of base-4.9,
>which in turn depends on ghc-internal-9.10.
>- At the same time, though, we release base-4.7.1, which depends on
>ghc-internal-9.10, and exposes the base-4.7 API.
>
> At this point we use ghc-9.10 to compile L, against base-4.7.1.   (Note
> the the ghc-9.10 binary includes a compiled form of `base-4.9`.
>
>- That produces compiled object files, such as, mylib:M.o.
>- To run TH we need to link them with the running binary
>- So we need to link the compiled `base-4.7.1` as well.  No problem:
>it contains very little code; it is mostly a shim for ghc-internal-9.10
>
> So the only thing we need is the ability to have a single linked binary
> that includes (the compiled form for) two different versions/instantiations
> of `base`.   I think that's already supported: each has a distinct
> "installed package id".
>
> What am I missing?
>
> Simon
>
>
>
> On Tue, 17 Oct 2023 at 16:54, Adam Gundry  wrote:
>
>> Hi Simon,
>>
>> Thanks for starting this discussion, it would be good to see progress in
>> this direction. As it happens I was discussing this question with Ben
>> and Matt over dinner last night, and unfortunately they explained to me
>> that it is more difficult than I naively hoped, even once wired-in and
>> known-key things are moved to ghc-internal.
>>
>> The difficulty is that, as a normal Haskell library, ghc itself will be
>> compiled against a particular version of base. Then when Template
>> Haskell is used (with the internal interpreter), code will be
>> dynamically loaded into a process that already has symbols for ghc's
>> version of base, which means it is not safe for the code to depend on a
>> different version of base. This is rather like the situation with TH and
>> cross-compilers.
>>
>> Adam
>>
>>
>>
>> On 17/10/2023 11:08, Simon Peyton Jones wrote:
>> > Dear GHC devs
>> >
>> > Given the now-agreed split between ghc-internal and base
>> > , what
>> > stands in the way of a "reinstallable base"?
>> >
>> > Specifically, suppose that
>> >
>> >   * GHC 9.8 comes out with base-4.9
>> >   * The CLC decides to make some change to `base`, so we get base-4.10
>> >   * Then GHC 9.10 comes out with base-4.10
>> >
>> > I think we'd all like it if someone could use GHC 9.10 to compile a
>> > library L that depends on base-4.9 and either L doesn't work at all
>> with
>> > base-4.10, or L's dependency bounds have not yet been adjusted to allow
>> > base-4.10.
>> >
>> > We'd like to have a version of `base`, say `base-4.9.1` that has the
>> > exact same API as `base-4.9` but works with GHC 9.10.
>> >
>> > Today, GHC 9.10 comes with a specific version of base, /and you can't
>> > change it/. The original reason for that was, I recall, that GHC knows
>> > the precise place where (say) the type Int is declared, and it'll get
>> > very confused if that data type definition moves around.
>> >
>> > But now we have `ghc-internal`, all these "things that GHC magically
>> > knows" are in `ghc-internal`, not `base`.
>> >
>> > *Hence my question: what (now) stops us making `base` behave like any
>> > other library*?  That would be a big step forward, because it would
>> mean
>> > that a newer GHC could compile old libraries against the

Re: Reinstallable - base

[Simon Peyton Jones]

(Meta-question: on reflection, would this discussion perhaps be better on a
ticket? But where?  GHC's repo?  Or HF's?)

The difficulty is that, as a normal Haskell library, ghc itself will be
> compiled against a particular verson of base. Then when Template Haskell is
> used (with the internal interpreter), code will be dynamically loaded into
> a process that already has symbols for ghc's version of base, which means
> it is not safe for the code to depend on a different version of base.


I'm not understanding the difficulty yet.

Let's say that

   - An old library mylib (which uses TH) depends on base-4.7.
   - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which
   in turn depends on ghc-internal-9.10.
   - At the same time, though, we release base-4.7.1, which depends on
   ghc-internal-9.10, and exposes the base-4.7 API.

At this point we use ghc-9.10 to compile L, against base-4.7.1.   (Note the
the ghc-9.10 binary includes a compiled form of `base-4.9`.

   - That produces compiled object files, such as, mylib:M.o.
   - To run TH we need to link them with the running binary
   - So we need to link the compiled `base-4.7.1` as well.  No problem: it
   contains very little code; it is mostly a shim for ghc-internal-9.10

So the only thing we need is the ability to have a single linked binary
that includes (the compiled form for) two different versions/instantiations
of `base`.   I think that's already supported: each has a distinct
"installed package id".

What am I missing?

Simon



On Tue, 17 Oct 2023 at 16:54, Adam Gundry  wrote:

> Hi Simon,
>
> Thanks for starting this discussion, it would be good to see progress in
> this direction. As it happens I was discussing this question with Ben
> and Matt over dinner last night, and unfortunately they explained to me
> that it is more difficult than I naively hoped, even once wired-in and
> known-key things are moved to ghc-internal.
>
> The difficulty is that, as a normal Haskell library, ghc itself will be
> compiled against a particular version of base. Then when Template
> Haskell is used (with the internal interpreter), code will be
> dynamically loaded into a process that already has symbols for ghc's
> version of base, which means it is not safe for the code to depend on a
> different version of base. This is rather like the situation with TH and
> cross-compilers.
>
> Adam
>
>
>
> On 17/10/2023 11:08, Simon Peyton Jones wrote:
> > Dear GHC devs
> >
> > Given the now-agreed split between ghc-internal and base
> > , what
> > stands in the way of a "reinstallable base"?
> >
> > Specifically, suppose that
> >
> >   * GHC 9.8 comes out with base-4.9
> >   * The CLC decides to make some change to `base`, so we get base-4.10
> >   * Then GHC 9.10 comes out with base-4.10
> >
> > I think we'd all like it if someone could use GHC 9.10 to compile a
> > library L that depends on base-4.9 and either L doesn't work at all with
> > base-4.10, or L's dependency bounds have not yet been adjusted to allow
> > base-4.10.
> >
> > We'd like to have a version of `base`, say `base-4.9.1` that has the
> > exact same API as `base-4.9` but works with GHC 9.10.
> >
> > Today, GHC 9.10 comes with a specific version of base, /and you can't
> > change it/. The original reason for that was, I recall, that GHC knows
> > the precise place where (say) the type Int is declared, and it'll get
> > very confused if that data type definition moves around.
> >
> > But now we have `ghc-internal`, all these "things that GHC magically
> > knows" are in `ghc-internal`, not `base`.
> >
> > *Hence my question: what (now) stops us making `base` behave like any
> > other library*?  That would be a big step forward, because it would mean
> > that a newer GHC could compile old libraries against their old
> dependencies.
> >
> > (Some changes would still be difficult.  If, for example, we removed
> > Monad and replaced it with classes Mo1 and Mo2, it might be hard to
> > simulate the old `base` with a shim.  But getting 99% of the way there
> > would still be fantastic.)
> >
> > Simon
>
> --
> Adam Gundry, Haskell Consultant
> Well-Typed LLP, https://www.well-typed.com/
>
> Registered in England & Wales, OC335890
> 27 Old Gloucester Street, London WC1N 3AX, England
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Howard B. Golden]

My _very_ naive question is whether Template Haskell should have all the 
features of the latest language? Perhaps it could instead be limited to a 
simpler subset so that it could become a bolt-on to the rest of the compiler 
and it could be compiled and implemented independently using an older compiler 
and libraries. 

Howard

> On Oct 17, 2023, at 9:54 AM, Viktor Dukhovni  wrote:
> 
> On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:
> 
>> Thanks for starting this discussion, it would be good to see progress in
>> this direction. As it happens I was discussing this question with Ben and
>> Matt over dinner last night, and unfortunately they explained to me that it
>> is more difficult than I naively hoped, even once wired-in and known-key
>> things are moved to ghc-internal.
>> 
>> The difficulty is that, as a normal Haskell library, ghc itself will be
>> compiled against a particular version of base. Then when Template Haskell is
>> used (with the internal interpreter), code will be dynamically loaded into a
>> process that already has symbols for ghc's version of base, which means it
>> is not safe for the code to depend on a different version of base. This is
>> rather like the situation with TH and cross-compilers.
> 
> To avoid that problem, GHC's own dependency on "base" could be indirect
> via a shared object with versioned symbol names and a version-specific
> SONAME (possibly even a private to GHC SONAME and private symbol version
> names).  Say "libbase.so.4.19.1".
> 
> The dependency on "base" in the TemplatHaskell generated code would then
> also need to be dynamic, allowing the two versions of base to coexist
> without conflict, both in turn depdent on a common version of the GHC
> internal libraries.
> 
> This would of course somewhat complicate binary distributions, but that
> should be manageable.  Perhaps there are less invasive (more clever)
> solutions?
> 
> -- 
>Viktor
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Viktor Dukhovni]

On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:

> Thanks for starting this discussion, it would be good to see progress in
> this direction. As it happens I was discussing this question with Ben and
> Matt over dinner last night, and unfortunately they explained to me that it
> is more difficult than I naively hoped, even once wired-in and known-key
> things are moved to ghc-internal.
> 
> The difficulty is that, as a normal Haskell library, ghc itself will be
> compiled against a particular version of base. Then when Template Haskell is
> used (with the internal interpreter), code will be dynamically loaded into a
> process that already has symbols for ghc's version of base, which means it
> is not safe for the code to depend on a different version of base. This is
> rather like the situation with TH and cross-compilers.

To avoid that problem, GHC's own dependency on "base" could be indirect
via a shared object with versioned symbol names and a version-specific
SONAME (possibly even a private to GHC SONAME and private symbol version
names).  Say "libbase.so.4.19.1".

The dependency on "base" in the TemplatHaskell generated code would then
also need to be dynamic, allowing the two versions of base to coexist
without conflict, both in turn depdent on a common version of the GHC
internal libraries.

This would of course somewhat complicate binary distributions, but that
should be manageable.  Perhaps there are less invasive (more clever)
solutions?

-- 
Viktor
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Adam Gundry]

Hi Simon,

Thanks for starting this discussion, it would be good to see progress in 
this direction. As it happens I was discussing this question with Ben 
and Matt over dinner last night, and unfortunately they explained to me 
that it is more difficult than I naively hoped, even once wired-in and 
known-key things are moved to ghc-internal.


The difficulty is that, as a normal Haskell library, ghc itself will be 
compiled against a particular version of base. Then when Template 
Haskell is used (with the internal interpreter), code will be 
dynamically loaded into a process that already has symbols for ghc's 
version of base, which means it is not safe for the code to depend on a 
different version of base. This is rather like the situation with TH and 
cross-compilers.


Adam



On 17/10/2023 11:08, Simon Peyton Jones wrote:

Dear GHC devs

Given the now-agreed split between ghc-internal and base 
, what 
stands in the way of a "reinstallable base"?


Specifically, suppose that

  * GHC 9.8 comes out with base-4.9
  * The CLC decides to make some change to `base`, so we get base-4.10
  * Then GHC 9.10 comes out with base-4.10

I think we'd all like it if someone could use GHC 9.10 to compile a 
library L that depends on base-4.9 and either L doesn't work at all with 
base-4.10, or L's dependency bounds have not yet been adjusted to allow 
base-4.10.


We'd like to have a version of `base`, say `base-4.9.1` that has the 
exact same API as `base-4.9` but works with GHC 9.10.


Today, GHC 9.10 comes with a specific version of base, /and you can't 
change it/. The original reason for that was, I recall, that GHC knows 
the precise place where (say) the type Int is declared, and it'll get 
very confused if that data type definition moves around.


But now we have `ghc-internal`, all these "things that GHC magically 
knows" are in `ghc-internal`, not `base`.


*Hence my question: what (now) stops us making `base` behave like any 
other library*?  That would be a big step forward, because it would mean 
that a newer GHC could compile old libraries against their old dependencies.


(Some changes would still be difficult.  If, for example, we removed 
Monad and replaced it with classes Mo1 and Mo2, it might be hard to 
simulate the old `base` with a shim.  But getting 99% of the way there 
would still be fantastic.)


Simon


--
Adam Gundry, Haskell Consultant
Well-Typed LLP, https://www.well-typed.com/

Registered in England & Wales, OC335890
27 Old Gloucester Street, London WC1N 3AX, England

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[John Ericson]

I think that's exactly right. Furthermore in 
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11400 the approach Ben has 
started going with is moving everything to ghc-internal to start, so this 
should be even easier --- surely we can reinstall an empty (except for 
reexports) library without the world falling down?! :)

I would love us to commit, day 0 of the split, to the principle that that base 
is now and forever more reinstallable. Then, as we move things back to base 
(untangling base's and ghc-internals's implementations), we'll be committing to 
not regressing on this. For example, if we want to move something back that GHC 
does in fact refer to, then we have to create the facility to make this not 
interfere with reinstallablilty before we move it.

(I would eventually like things like NonEmpty that GHC only barely knows about, 
and which are perfectly standard bits of Haskell, to live in base not 
ghc-internals. Agda's and Rust's putting annotations on things to let the 
compiler know it should use them, rather than baking in names/locations into 
the compiler, is very good prior art that this can work.)

John

On Tue, Oct 17, 2023, at 6:08 AM, Simon Peyton Jones wrote:
> Dear GHC devs
> 
> Given the now-agreed split between ghc-internal and base 
> , what stands in 
> the way of a "reinstallable base"?
> 
> Specifically, suppose that
>  • GHC 9.8 comes out with base-4.9
>  • The CLC decides to make some change to `base`, so we get base-4.10
>  • Then GHC 9.10 comes out with base-4.10
> I think we'd all like it if someone could use GHC 9.10 to compile a library L 
> that depends on base-4.9 and either L doesn't work at all with base-4.10, or 
> L's dependency bounds have not yet been adjusted to allow base-4.10.
> 
> We'd like to have a version of `base`, say `base-4.9.1` that has the exact 
> same API as `base-4.9` but works with GHC 9.10.
> 
> Today, GHC 9.10 comes with a specific version of base, *and you can't change 
> it*. The original reason for that was, I recall, that GHC knows the precise 
> place where (say) the type Int is declared, and it'll get very confused if 
> that data type definition moves around.
> 
> But now we have `ghc-internal`, all these "things that GHC magically knows" 
> are in `ghc-internal`, not `base`.
> 
> *Hence my question: what (now) stops us making `base` behave like any other 
> library*?  That would be a big step forward, because it would mean that a 
> newer GHC could compile old libraries against their old dependencies.
> 
> (Some changes would still be difficult.  If, for example, we removed Monad 
> and replaced it with classes Mo1 and Mo2, it might be hard to simulate the 
> old `base` with a shim.  But getting 99% of the way there would still be 
> fantastic.)
> 
> Simon
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Reinstallable - base

[Simon Peyton Jones]

Dear GHC devs

Given the now-agreed split between ghc-internal and base
, what stands
in the way of a "reinstallable base"?

Specifically, suppose that

   - GHC 9.8 comes out with base-4.9
   - The CLC decides to make some change to `base`, so we get base-4.10
   - Then GHC 9.10 comes out with base-4.10

I think we'd all like it if someone could use GHC 9.10 to compile a library
L that depends on base-4.9 and either L doesn't work at all with base-4.10,
or L's dependency bounds have not yet been adjusted to allow base-4.10.

We'd like to have a version of `base`, say `base-4.9.1` that has the exact
same API as `base-4.9` but works with GHC 9.10.

Today, GHC 9.10 comes with a specific version of base, *and you can't
change it*. The original reason for that was, I recall, that GHC knows the
precise place where (say) the type Int is declared, and it'll get very
confused if that data type definition moves around.

But now we have `ghc-internal`, all these "things that GHC magically knows"
are in `ghc-internal`, not `base`.

*Hence my question: what (now) stops us making `base` behave like any other
library*?  That would be a big step forward, because it would mean that a
newer GHC could compile old libraries against their old dependencies.

(Some changes would still be difficult.  If, for example, we removed Monad
and replaced it with classes Mo1 and Mo2, it might be hard to simulate the
old `base` with a shim.  But getting 99% of the way there would still be
fantastic.)

Simon
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs