Re: Reinstallable - base

[Arnaud Spiwack]

A very large proportion of libraries, and virtually all end-user
applications, transitively depend on Template Haskell. Whether they use
Template Haskell directly or not. So if we're saying “base is
reinstallable, except when you have Template Haskell somewhere”, we're
effectively saying “base is not reinstallable”. Now, it could be a good
stepping-stone, from an engineering standpoint, but I don't think we could
deliver this and be satisfied that we've accomplished anything.

On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus  wrote:

> For what it worth, `template-haskell` itself depends on a `base`. So if
> `base` if different base is used, different `template-haskell` is to be
> used.
>
> In my opinion is not *too unfair* to require that if you actually splice
> in (i.e. the code not only provides template-haskell combinators to
> create/modify splices) then you must have base and template-haskell
> versions aligned with host GHC used versions.
>
> The same restriction is GHC plugins, isn't it, except `template-haskell`
> is replaced with `ghc`?
>
> - Oleg
>
> On 17.10.2023 18.54, Adam Gundry wrote:
> > Hi Simon,
> >
> > Thanks for starting this discussion, it would be good to see progress
> > in this direction. As it happens I was discussing this question with
> > Ben and Matt over dinner last night, and unfortunately they explained
> > to me that it is more difficult than I naively hoped, even once
> > wired-in and known-key things are moved to ghc-internal.
> >
> > The difficulty is that, as a normal Haskell library, ghc itself will
> > be compiled against a particular version of base. Then when Template
> > Haskell is used (with the internal interpreter), code will be
> > dynamically loaded into a process that already has symbols for ghc's
> > version of base, which means it is not safe for the code to depend on
> > a different version of base. This is rather like the situation with TH
> > and cross-compilers.
> >
> > Adam
> >
> >
> >
> > On 17/10/2023 11:08, Simon Peyton Jones wrote:
> >> Dear GHC devs
> >>
> >> Given the now-agreed split between ghc-internal and base
> >> , what
> >> stands in the way of a "reinstallable base"?
> >>
> >> Specifically, suppose that
> >>
> >>   * GHC 9.8 comes out with base-4.9
> >>   * The CLC decides to make some change to `base`, so we get base-4.10
> >>   * Then GHC 9.10 comes out with base-4.10
> >>
> >> I think we'd all like it if someone could use GHC 9.10 to compile a
> >> library L that depends on base-4.9 and either L doesn't work at all
> >> with base-4.10, or L's dependency bounds have not yet been adjusted
> >> to allow base-4.10.
> >>
> >> We'd like to have a version of `base`, say `base-4.9.1` that has the
> >> exact same API as `base-4.9` but works with GHC 9.10.
> >>
> >> Today, GHC 9.10 comes with a specific version of base, /and you can't
> >> change it/. The original reason for that was, I recall, that GHC
> >> knows the precise place where (say) the type Int is declared, and
> >> it'll get very confused if that data type definition moves around.
> >>
> >> But now we have `ghc-internal`, all these "things that GHC magically
> >> knows" are in `ghc-internal`, not `base`.
> >>
> >> *Hence my question: what (now) stops us making `base` behave like any
> >> other library*?  That would be a big step forward, because it would
> >> mean that a newer GHC could compile old libraries against their old
> >> dependencies.
> >>
> >> (Some changes would still be difficult.  If, for example, we removed
> >> Monad and replaced it with classes Mo1 and Mo2, it might be hard to
> >> simulate the old `base` with a shim.  But getting 99% of the way
> >> there would still be fantastic.)
> >>
> >> Simon
> >
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>


-- 
Arnaud Spiwack
Director, Research at https://moduscreate.com and https://tweag.io.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Simon Peyton Jones]

>
> A very large proportion of libraries, and virtually all end-user
> applications, transitively depend on Template Haskell. Whether they use
> Template Haskell directly or not. So if we're saying “base is
> reinstallable, except when you have Template Haskell somewhere”, we're
> effectively saying “base is not reinstallable”. Now, it could be a good
> stepping-stone, from an engineering standpoint, but I don't think we could
> deliver this and be satisfied that we've accomplished anything.
>

No one has yet answered my naive question (from 3 days ago) asking why
Template Haskell stops base being reinstallable.  I'll quote it here for
completeness.

Let's say that
>

   - An old library mylib (which uses TH) depends on base-4.7.
   - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which
   in turn depends on ghc-internal-9.10.
   - At the same time, though, we release base-4.7.1, which depends on
   ghc-internal-9.10, and exposes the base-4.7 API.

At this point we use ghc-9.10 to compile L, against base-4.7.1.   (Note
that the ghc-9.10 binary includes a compiled form of `base-4.9`.)


   - That produces compiled object files, such as, mylib:M.o.
   - To run TH we need to link them with the running binary
   - So we need to link the compiled `base-4.7.1` as well.  No problem: it
   contains very little code; it is mostly a shim for ghc-internal-9.10

So the only thing we need is the ability to have a single linked binary
that includes (the compiled form for) two different versions/instantiations
of `base`.   I think that's already supported: each has a distinct
"installed package id".

(End of quote)

What am I missing?

Simon

On Fri, 20 Oct 2023 at 08:57, Arnaud Spiwack 
wrote:

> A very large proportion of libraries, and virtually all end-user
> applications, transitively depend on Template Haskell. Whether they use
> Template Haskell directly or not. So if we're saying “base is
> reinstallable, except when you have Template Haskell somewhere”, we're
> effectively saying “base is not reinstallable”. Now, it could be a good
> stepping-stone, from an engineering standpoint, but I don't think we could
> deliver this and be satisfied that we've accomplished anything.
>
> On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus  wrote:
>
>> For what it worth, `template-haskell` itself depends on a `base`. So if
>> `base` if different base is used, different `template-haskell` is to be
>> used.
>>
>> In my opinion is not *too unfair* to require that if you actually splice
>> in (i.e. the code not only provides template-haskell combinators to
>> create/modify splices) then you must have base and template-haskell
>> versions aligned with host GHC used versions.
>>
>> The same restriction is GHC plugins, isn't it, except `template-haskell`
>> is replaced with `ghc`?
>>
>> - Oleg
>>
>> On 17.10.2023 18.54, Adam Gundry wrote:
>> > Hi Simon,
>> >
>> > Thanks for starting this discussion, it would be good to see progress
>> > in this direction. As it happens I was discussing this question with
>> > Ben and Matt over dinner last night, and unfortunately they explained
>> > to me that it is more difficult than I naively hoped, even once
>> > wired-in and known-key things are moved to ghc-internal.
>> >
>> > The difficulty is that, as a normal Haskell library, ghc itself will
>> > be compiled against a particular version of base. Then when Template
>> > Haskell is used (with the internal interpreter), code will be
>> > dynamically loaded into a process that already has symbols for ghc's
>> > version of base, which means it is not safe for the code to depend on
>> > a different version of base. This is rather like the situation with TH
>> > and cross-compilers.
>> >
>> > Adam
>> >
>> >
>> >
>> > On 17/10/2023 11:08, Simon Peyton Jones wrote:
>> >> Dear GHC devs
>> >>
>> >> Given the now-agreed split between ghc-internal and base
>> >> , what
>> >> stands in the way of a "reinstallable base"?
>> >>
>> >> Specifically, suppose that
>> >>
>> >>   * GHC 9.8 comes out with base-4.9
>> >>   * The CLC decides to make some change to `base`, so we get base-4.10
>> >>   * Then GHC 9.10 comes out with base-4.10
>> >>
>> >> I think we'd all like it if someone could use GHC 9.10 to compile a
>> >> library L that depends on base-4.9 and either L doesn't work at all
>> >> with base-4.10, or L's dependency bounds have not yet been adjusted
>> >> to allow base-4.10.
>> >>
>> >> We'd like to have a version of `base`, say `base-4.9.1` that has the
>> >> exact same API as `base-4.9` but works with GHC 9.10.
>> >>
>> >> Today, GHC 9.10 comes with a specific version of base, /and you can't
>> >> change it/. The original reason for that was, I recall, that GHC
>> >> knows the precise place where (say) the type Int is declared, and
>> >> it'll get very confused if that data type definition moves around.
>> >>
>> >> But now we have `ghc-internal`, all these "things that GHC magica

Re: Reinstallable - base

[Ben Gamari]

Simon Peyton Jones  writes:

> Dear GHC devs
>
> Given the now-agreed split between ghc-internal and base
> , what stands
> in the way of a "reinstallable base"?
>
> Specifically, suppose that
>
>- GHC 9.8 comes out with base-4.9
>- The CLC decides to make some change to `base`, so we get base-4.10
>- Then GHC 9.10 comes out with base-4.10
>
We thought about this quite a bit at the Well-Typed meeting this week.
In short, I suspect that after the `ghc-internal` split is merged and
some further work we will be able to use multiple `base` versions with
a single compiler version.

I imagine this would be done in roughly three phases. The first two
phases can happen in the short term; the last is a bit further off.
I'll describe these in turn below.


# Phase 1: Reinstallable `base`

Specifically, (in the absence of TH) we need to...

 * Give `base` a proper unit ID. This is necessary as GHC may allow
   multiple versions of a single package in a single build; under the
   status quo this would result in overlapping symbol names.

 * Ensure that all known-key and wired-in declarations are placed in
   `ghc-internal`, not `base`. This is necessary as wired-in and
   known-key declarations are detected by their `Name`, which we cannot
   know a priori if `base` has a proper unit ID.

 * Teach `Cabal` that `base` is a reinstallable package (`ghc-internal`
   will remain non-reinstallable)

 * With each GHC version, provide minor releases of `base` which can be
   built against the new GHC's `ghc-internal`

With this done, we can offer (limited) compatibility across GHC/`base`
version pairs. Naturally, changes in semantics of things defined in
`ghc-internal` will make general compatibility difficult or impossible.
However, Simon suggests, limited compatibility is much better than the
status quo.


# Phase 2: Template Haskell

The above story is made considerably messier by `template-haskell`.
Specifically, `template-haskell` depends upon `base`. Typically (i.e.
when not using the external interpreter) values (e.g., splice results)
are communicated between the splice and the compiler as heap objects;
since these heap objects may be of types defined in `base` we must
guarantee that `template-haskell` and `ghc` are built against the same
`base` ABI. Since GHC doesn't have a notion of ABI stability, this would
render reinstallable `base` unusable for a majority of programs.

The first step to fixing this would be to eliminate the ABI dependency
between `template-haskell` and `ghc`. We can do this by either:

 * using the external interpreter (`-fexternal-interpreter`), in which
   case splice results are communicated by serialisation

 * teaching the internal interpreter to communicate using the same
   serialisation mechanism used in the external interpreter

However, `template-haskell` suffers from the same problem as `base` in
that it contains definitions of known-key declarations. For this reason,
we must apply the same approach that we used in `base` above, moving all
known-key definitions into a new `template-haskell-internal` package
against which `template-haskell` can link.

At this point we can teach `Cabal` that `template-haskell` can be
reinstalled and users will be able to build packages using
`TemplateHaskell` while linking against arbitrary `base` versions.


# Phase 3: Compatibility across `template-haskell` versions

One of the major compatibility headaches of moving between GHC versions
is accommodating changes in the `template-haskell` package. While there
are various interface-level changes which we can use to alleviate some
of this pressure (#20828, #24021), being able to provide compatibility
with multiple `template-haskell` versions would be very convenient.

Once we have split `template-haskell-internal` and `template-haskell`,
we can potentially use a similar approach to that proposed above:
support for multiple `template-haskell` major versions can be provided
for each compiler.


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Ben Gamari]

Viktor Dukhovni  writes:

> On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:
>
>> Thanks for starting this discussion, it would be good to see progress in
>> this direction. As it happens I was discussing this question with Ben and
>> Matt over dinner last night, and unfortunately they explained to me that it
>> is more difficult than I naively hoped, even once wired-in and known-key
>> things are moved to ghc-internal.
>> 
>> The difficulty is that, as a normal Haskell library, ghc itself will be
>> compiled against a particular version of base. Then when Template Haskell is
>> used (with the internal interpreter), code will be dynamically loaded into a
>> process that already has symbols for ghc's version of base, which means it
>> is not safe for the code to depend on a different version of base. This is
>> rather like the situation with TH and cross-compilers.
>
> To avoid that problem, GHC's own dependency on "base" could be indirect
> via a shared object with versioned symbol names and a version-specific
> SONAME (possibly even a private to GHC SONAME and private symbol version
> names).  Say "libbase.so.4.19.1".
>
The problem here is deeper than simply the symbol names. For instance,
the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ.
Since lists are used in the `template-haskell` AST, we would be unable
to share lists between `template-haskell` and `ghc`.

As noted in my recent reply elsewhere in the thread, this can be avoided
by communicating via serialisation instead of heap objects.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Ben Gamari]

"Howard B. Golden"  writes:

> My _very_ naive question is whether Template Haskell should have all
> the features of the latest language? Perhaps it could instead be
> limited to a simpler subset so that it could become a bolt-on to the
> rest of the compiler and it could be compiled and implemented
> independently using an older compiler and libraries.
>
While I think users should be able to access all of the features of the
language from TemplateHaskell, I agree that most users only need a small
subset. We have explored some ideas exploiting this property in #24021.
While it's not trivial per se, I think there is a real opportunity here.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Ben Gamari]

Simon Peyton Jones  writes:

> (Meta-question: on reflection, would this discussion perhaps be better on a
> ticket? But where?  GHC's repo?  Or HF's?)
>
Given the current state of the discussion and the fact that the
necessary steps are mostly internal to GHC, I think it is better to keep
the discussion here or the issue tracker.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Andrei Borzenkov]

> the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may 
differ. This is not actually true, `List` will obviously live in 
`ghc-internal` and `ghc-internal` would be the same for `base-4.9` and 
for `base-4.7.1`. This raises the question about do we really need to 
depend on `base` in `template-haskell`? Perhaps we may be satisfied by 
small set of things that will contain `ghc-internal`?


20.10.2023 14:23, Ben Gamari writes:

Viktor Dukhovni  writes:


On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:


Thanks for starting this discussion, it would be good to see progress in
this direction. As it happens I was discussing this question with Ben and
Matt over dinner last night, and unfortunately they explained to me that it
is more difficult than I naively hoped, even once wired-in and known-key
things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be
compiled against a particular version of base. Then when Template Haskell is
used (with the internal interpreter), code will be dynamically loaded into a
process that already has symbols for ghc's version of base, which means it
is not safe for the code to depend on a different version of base. This is
rather like the situation with TH and cross-compilers.

To avoid that problem, GHC's own dependency on "base" could be indirect
via a shared object with versioned symbol names and a version-specific
SONAME (possibly even a private to GHC SONAME and private symbol version
names).  Say "libbase.so.4.19.1".


The problem here is deeper than simply the symbol names. For instance,
the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ.
Since lists are used in the `template-haskell` AST, we would be unable
to share lists between `template-haskell` and `ghc`.

As noted in my recent reply elsewhere in the thread, this can be avoided
by communicating via serialisation instead of heap objects.

Cheers,

- Ben


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Sebastian Graf]

Hi,

Thanks, Ben, that sounds very interesting. Allow me to provide the
following perspective.
It seems that those `*-internal` packages take the role of a static
library's symbol table in the absence of a fixed ABI:
It allows clients (such as `base` and `template-haskell`) to link against
fixed, internal implementations provided by GHC.
"Reinstallable" then just means that a library can be linked against
multiple different GHC versions providing the same set of internal APIs.

But perhaps this analogy is not all to useful, given the clash with the
traditional use of the term "linking" and "static library".

Cheers,
Sebastian

Am Fr., 20. Okt. 2023 um 12:35 Uhr schrieb Andrei Borzenkov <
andreyborzenkov2...@gmail.com>:

> >  the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ.
> This is not actually true, `List` will obviously live in `ghc-internal` and
> `ghc-internal` would be the same for `base-4.9` and for `base-4.7.1`. This
> raises the question about do we really need to depend on `base` in
> `template-haskell`? Perhaps we may be satisfied by small set of things that
> will contain `ghc-internal`?
> 20.10.2023 14:23, Ben Gamari writes:
>
> Viktor Dukhovni   writes:
>
>
> On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:
>
>
> Thanks for starting this discussion, it would be good to see progress in
> this direction. As it happens I was discussing this question with Ben and
> Matt over dinner last night, and unfortunately they explained to me that it
> is more difficult than I naively hoped, even once wired-in and known-key
> things are moved to ghc-internal.
>
> The difficulty is that, as a normal Haskell library, ghc itself will be
> compiled against a particular version of base. Then when Template Haskell is
> used (with the internal interpreter), code will be dynamically loaded into a
> process that already has symbols for ghc's version of base, which means it
> is not safe for the code to depend on a different version of base. This is
> rather like the situation with TH and cross-compilers.
>
> To avoid that problem, GHC's own dependency on "base" could be indirect
> via a shared object with versioned symbol names and a version-specific
> SONAME (possibly even a private to GHC SONAME and private symbol version
> names).  Say "libbase.so.4.19.1".
>
>
> The problem here is deeper than simply the symbol names. For instance,
> the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ.
> Since lists are used in the `template-haskell` AST, we would be unable
> to share lists between `template-haskell` and `ghc`.
>
> As noted in my recent reply elsewhere in the thread, this can be avoided
> by communicating via serialisation instead of heap objects.
>
> Cheers,
>
> - Ben
>
>
>
> ___
> ghc-devs mailing 
> listghc-devs@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[John Ericson]

I think it is making progress. It's not just an engineering milestone 
but also messaging. Right now a huge issue that every Haskeller faces is 
new GHCs having new major versions of base. If we make clear that "it's 
not base's fault, it's Template Haskell's fault", then we will have an 
easier time coordinating people and eventually fundraising to get issued 
fixed with Template Haskell.


John

On 10/20/23 03:56, Arnaud Spiwack wrote:
A very large proportion of libraries, and virtually all end-user 
applications, transitively depend on Template Haskell. Whether they 
use Template Haskell directly or not. So if we're saying “base is 
reinstallable, except when you have Template Haskell somewhere”, we're 
effectively saying “base is not reinstallable”. Now, it could be a 
good stepping-stone, from an engineering standpoint, but I don't think 
we could deliver this and be satisfied that we've accomplished anything.


On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus  wrote:

For what it worth, `template-haskell` itself depends on a `base`.
So if
`base` if different base is used, different `template-haskell` is
to be
used.

In my opinion is not *too unfair* to require that if you actually
splice
in (i.e. the code not only provides template-haskell combinators to
create/modify splices) then you must have base and template-haskell
versions aligned with host GHC used versions.

The same restriction is GHC plugins, isn't it, except
`template-haskell`
is replaced with `ghc`?

- Oleg

On 17.10.2023 18.54, Adam Gundry wrote:
> Hi Simon,
>
> Thanks for starting this discussion, it would be good to see
progress
> in this direction. As it happens I was discussing this question
with
> Ben and Matt over dinner last night, and unfortunately they
explained
> to me that it is more difficult than I naively hoped, even once
> wired-in and known-key things are moved to ghc-internal.
>
> The difficulty is that, as a normal Haskell library, ghc itself
will
> be compiled against a particular version of base. Then when
Template
> Haskell is used (with the internal interpreter), code will be
> dynamically loaded into a process that already has symbols for
ghc's
> version of base, which means it is not safe for the code to
depend on
> a different version of base. This is rather like the situation
with TH
> and cross-compilers.
>
> Adam
>
>
>
> On 17/10/2023 11:08, Simon Peyton Jones wrote:
>> Dear GHC devs
>>
>> Given the now-agreed split between ghc-internal and base
>> ,
what
>> stands in the way of a "reinstallable base"?
>>
>> Specifically, suppose that
>>
>>   * GHC 9.8 comes out with base-4.9
>>   * The CLC decides to make some change to `base`, so we get
base-4.10
>>   * Then GHC 9.10 comes out with base-4.10
>>
>> I think we'd all like it if someone could use GHC 9.10 to
compile a
>> library L that depends on base-4.9 and either L doesn't work at
all
>> with base-4.10, or L's dependency bounds have not yet been
adjusted
>> to allow base-4.10.
>>
>> We'd like to have a version of `base`, say `base-4.9.1` that
has the
>> exact same API as `base-4.9` but works with GHC 9.10.
>>
>> Today, GHC 9.10 comes with a specific version of base, /and you
can't
>> change it/. The original reason for that was, I recall, that GHC
>> knows the precise place where (say) the type Int is declared, and
>> it'll get very confused if that data type definition moves around.
>>
>> But now we have `ghc-internal`, all these "things that GHC
magically
>> knows" are in `ghc-internal`, not `base`.
>>
>> *Hence my question: what (now) stops us making `base` behave
like any
>> other library*?  That would be a big step forward, because it
would
>> mean that a newer GHC could compile old libraries against their
old
>> dependencies.
>>
>> (Some changes would still be difficult.  If, for example, we
removed
>> Monad and replaced it with classes Mo1 and Mo2, it might be
hard to
>> simulate the old `base` with a shim.  But getting 99% of the way
>> there would still be fantastic.)
>>
>> Simon
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



--
Arnaud Spiwack
Director, Research at https://moduscreate.com and https://tweag.io.

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/

Re: Reinstallable - base

[John Ericson]

On 10/20/23 04:00, Simon Peyton Jones wrote:


A very large proportion of libraries, and virtually all end-user
applications, transitively depend on Template Haskell. Whether
they use Template Haskell directly or not. So if we're saying
“base is reinstallable, except when you have Template Haskell
somewhere”, we're effectively saying “base is not reinstallable”.
Now, it could be a good stepping-stone, from an engineering
standpoint, but I don't think we could deliver this and be
satisfied that we've accomplished anything.


No one has yet answered my naive question (from 3 days ago) asking why 
Template Haskell stops base being reinstallable. I'll quote it here 
for completeness.


Let's say that

  * An old library mylib (which uses TH) depends on base-4.7.
  * A new GHC, say GHC 9.10, depends on a newer version of
base-4.9, which in turn depends on ghc-internal-9.10.
  * At the same time, though, we release base-4.7.1, which depends
on ghc-internal-9.10, and exposes the base-4.7 API.

At this point we use ghc-9.10 to compile L, against base-4.7.1.  
(Note that the ghc-9.10 binary includes a compiled form of
`base-4.9`.)

  * That produces compiled object files, such as, mylib:M.o.
  * To run TH we need to link them with the running binary
  * So we need to link the compiled `base-4.7.1` as well.  No
problem: it contains very little code; it is mostly a shim
for ghc-internal-9.10

So the only thing we need is the ability to have a single linked
binary that includes (the compiled form for) two different
versions/instantiations of `base`.   I think that's already
supported: each has a distinct "installed package id".

(End of quote)

What am I missing?

Simon


Simon I think you are right on the level of GHC itself: GHC can indeed 
cope with multiple versions of libraries just fine. However if we do 
this we run the risk of the user getting "base.x.y FooType is not the 
same as base.x.(y+1) FooType" errors, and I don't think that is good. 
cabal-install and stack current enforce build plans such that those 
sorts of errors are not possible, and I think that is a good thing we 
should not revisit at this time.


I don't think this should stop anything about reinstallable base, 
however. We just need to make a nice error when someone tries to splice 
in code using the wrong version of template haskell (or otherwise 
interact with GHC) using ABI hashes. Then we have the following situation:


1. Users that use stack/cabal-install plans only get one version of
   base, and if they have template-haskell using the the "wrong"
   version of base (identified via API hashes) they will get a nice error.
2. Users that "go rogue" and manually set up their project to mix base
   versions and avoid other errors will have their TH work as you
   describe./

So the thing Simon talks about is not /encouraged/, but it is a "free 
feature" we shouldn't go out of our way to prevent either.


John
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Ben Gamari]

Ben Gamari  writes:

> Simon Peyton Jones  writes:
>
>> Dear GHC devs
>>
>> Given the now-agreed split between ghc-internal and base
>> , what stands
>> in the way of a "reinstallable base"?
>>
>> Specifically, suppose that
>>
>>- GHC 9.8 comes out with base-4.9
>>- The CLC decides to make some change to `base`, so we get base-4.10
>>- Then GHC 9.10 comes out with base-4.10
>>
> We thought about this quite a bit at the Well-Typed meeting this week.
> In short, I suspect that after the `ghc-internal` split is merged and
> some further work we will be able to use multiple `base` versions with
> a single compiler version.
>
> I imagine this would be done in roughly three phases. The first two
> phases can happen in the short term; the last is a bit further off.
> I'll describe these in turn below.
>
One final point that I forgot to mention: None of this addresses the
problem of compiler plugins. This is problematic as `Cabal`'s
one-version-per-install-plan restriction means means that any package
using a plugin will be forced to use the precise `base` that `ghc`
itself is linked against.

I can think of three ways to address this:

 * Teach `Cabal` to distinguish between plugin dependencies (which are
   only needed at build time and needn't be linked into final build
   results) and normal runtime dependencies. This is not a complete
   solution as many plugins have both compile-time and runtime
   components, but it would help in some cases.

 * Make `ghc` reinstallable. This allows `Cabal` to rebuild the compiler
   when we need to link against a different `base`. We have started this
   process but in general it is quite tricky to get right (see #20742)
   and will require cooperation from `Cabal` .

 * Serialise structures used by plugins, just as we do for the TH AST.
   Unfortunately, the surface area of the plugin interface is
   significantly larger than that of TH.

None of these options are easy. For this reason, I think it would be
wise to leave plugins as future work (if we decide to address it at
all).


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Howard B. Golden]

Cut the Gordian Knot by serializing all the interfaces. This may be a good 
first step. 

Howard

> On Oct 20, 2023, at 10:26 AM, Ben Gamari  wrote:
> 
> Ben Gamari  writes:
> 
>> Simon Peyton Jones  writes:
>> 
>>> Dear GHC devs
>>> 
>>> Given the now-agreed split between ghc-internal and base
>>> , what stands
>>> in the way of a "reinstallable base"?
>>> 
>>> Specifically, suppose that
>>> 
>>>   - GHC 9.8 comes out with base-4.9
>>>   - The CLC decides to make some change to `base`, so we get base-4.10
>>>   - Then GHC 9.10 comes out with base-4.10
>>> 
>> We thought about this quite a bit at the Well-Typed meeting this week.
>> In short, I suspect that after the `ghc-internal` split is merged and
>> some further work we will be able to use multiple `base` versions with
>> a single compiler version.
>> 
>> I imagine this would be done in roughly three phases. The first two
>> phases can happen in the short term; the last is a bit further off.
>> I'll describe these in turn below.
>> 
> One final point that I forgot to mention: None of this addresses the
> problem of compiler plugins. This is problematic as `Cabal`'s
> one-version-per-install-plan restriction means means that any package
> using a plugin will be forced to use the precise `base` that `ghc`
> itself is linked against.
> 
> I can think of three ways to address this:
> 
> * Teach `Cabal` to distinguish between plugin dependencies (which are
>   only needed at build time and needn't be linked into final build
>   results) and normal runtime dependencies. This is not a complete
>   solution as many plugins have both compile-time and runtime
>   components, but it would help in some cases.
> 
> * Make `ghc` reinstallable. This allows `Cabal` to rebuild the compiler
>   when we need to link against a different `base`. We have started this
>   process but in general it is quite tricky to get right (see #20742)
>   and will require cooperation from `Cabal` .
> 
> * Serialise structures used by plugins, just as we do for the TH AST.
>   Unfortunately, the surface area of the plugin interface is
>   significantly larger than that of TH.
> 
> None of these options are easy. For this reason, I think it would be
> wise to leave plugins as future work (if we decide to address it at
> all).
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Ben Gamari]

Ben Gamari  writes:

> ...
>
> # Phase 2: Template Haskell
>
> The above story is made considerably messier by `template-haskell`.
> ...

One further clarification as I fear this may be unclear:

Phase 2 does *not* allow multiple versions of `template-haskell` to
be used with a single GHC. That is, each GHC version will have precisely
one corresponding `template-haskell` version; consequently, programs
using TemplateHaskell's combinator's will still need to adapt to new
`template-haskell` releases when moving to a new GHC.

Phase 3 describes how we could eliminate this need by allowing multiple
`template-haskell` versions to be used with a single GHC release.


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Reinstallable - base

[Howard B. Golden]

Perhaps there may be useful ideas in 
https://draugus.github.io/pdf/Software%20Design%20for%20Flexibility%20-%20Chris%20Hanson.pdf
 .

Howard

> On Oct 20, 2023, at 1:23 PM, Ben Gamari  wrote:
> 
> Ben Gamari  writes:
> 
>> ...
>> 
>> # Phase 2: Template Haskell
>> 
>> The above story is made considerably messier by `template-haskell`.
>> ...
> 
> One further clarification as I fear this may be unclear:
> 
> Phase 2 does *not* allow multiple versions of `template-haskell` to
> be used with a single GHC. That is, each GHC version will have precisely
> one corresponding `template-haskell` version; consequently, programs
> using TemplateHaskell's combinator's will still need to adapt to new
> `template-haskell` releases when moving to a new GHC.
> 
> Phase 3 describes how we could eliminate this need by allowing multiple
> `template-haskell` versions to be used with a single GHC release.
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs