Re: Multiple GHC sessions

[Simon Marlow]

On 18 January 2016 at 11:18, Alan & Kim Zimmerman 
wrote:

> From a tool writer perspective I would be happy to have GHC/Linker
> sessions be completely separate from each other. But I guess in
> practical terms a shared address space needs to be managed, making
> this impossible.
>
> The external interpreter for GHCI effectively solves the actual
> loading problem, the other things needing a GHC session are loading a
> module as far as type checking, for error reporting. I know this gets
> complicated if there is FFI involved and the loading has to go via
> object file generation.
>

Yes, I think -fexternal-interpreter is the basis of the right solution
here.  You do also need to move the PersistentLinkerState to HscEnv, but
that should be a fairly straightforward refactoring.  (actually I almost
did this as part of the remote GHCi work, but some of the comments next to
PersistentLinkerState suggested that there might be some real use cases for
having the linker state be shared between multiple sessions, so I decided
not to attack that problem right away.)

Cheers,
Simon

Alan
>
> On Mon, Jan 18, 2016 at 12:53 PM, Ben Gamari  wrote:
> > Simon Peyton Jones  writes:
> >
> >> I don't think the state-hack tail should wag the dog here. The nub of
> >> the problem in practice is the shared linker state isn't it?
> >>
> > I'm not sure; it's not clear to me what would break if the linker were
> > used by more than one session at a time, even with the current
> > GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all
> > sessions to use the same linker state given that they are also sharing
> > an address space.
> >
> > The only issue that may require a bit of care is code unloading.
> > Consider that you have two sessions A and B,
> >
> >   1. A loads MyModule
> >   2. B loads MyModule
> >   3. They both use MyModule for a while
> >   4. A unloads MyModule
> >   5. B continues
> >   6. B unloads MyModule
> >
> > I suspect that things will work fine up to step 6 since we don't
> > actually unload code until the garbage collector has marked the it as
> > dead. That being said, I suspect you would see an error from unloadObj
> > upon attempting to unload the object the second time (and even then, the
> > error is merely a message on stderr). Note, however, that the above
> > conclusions have not been tested.
> >
> > Otherwise, it appears that things should behave fair well as the
> > PersistentLinkerState is already wrapped in an MVar which should
> > serve to serialize linker calls.
> >
> > Cheers,
> >
> > - Ben
> >
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: Multiple GHC sessions

[Ben Gamari]

Simon Peyton Jones  writes:

> I don't think the state-hack tail should wag the dog here. The nub of
> the problem in practice is the shared linker state isn't it?
>
I'm not sure; it's not clear to me what would break if the linker were
used by more than one session at a time, even with the current
GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all
sessions to use the same linker state given that they are also sharing
an address space.

The only issue that may require a bit of care is code unloading.
Consider that you have two sessions A and B,

  1. A loads MyModule 
  2. B loads MyModule
  3. They both use MyModule for a while
  4. A unloads MyModule
  5. B continues 
  6. B unloads MyModule

I suspect that things will work fine up to step 6 since we don't
actually unload code until the garbage collector has marked the it as
dead. That being said, I suspect you would see an error from unloadObj
upon attempting to unload the object the second time (and even then, the
error is merely a message on stderr). Note, however, that the above
conclusions have not been tested.

Otherwise, it appears that things should behave fair well as the
PersistentLinkerState is already wrapped in an MVar which should
serve to serialize linker calls.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: Multiple GHC sessions

[Simon Peyton Jones]

I don't think the state-hack tail should wag the dog here.  The nub of the 
problem in practice is the shared linker state isn't it?

Simon

|  -Original Message-
|  From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Ben
|  Gamari
|  Sent: 17 January 2016 19:55
|  To: Alan & Kim Zimmerman <alan.z...@gmail.com>; ghc-devs@haskell.org
|  Subject: Re: Multiple GHC sessions
|  
|  Alan & Kim Zimmerman <alan.z...@gmail.com> writes:
|  
|  > At the moment there are issues with having multiple GHC API sessions
|  > in a single executable, which boil down to GHC having global
|  > variables.
|  >
|  > A quick grep over the GHC sources shows the following instances
|  >
|  > compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ...
|  > compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, 
|  >
|  This isn't the only global state had by the linker; have a look at
|  rts/Linker.c. Unfortunately this would take a fair amount of work to
|  resolve. The easiest solution here would probably be to put a lock
|  around the linker and have all sessions in a process use the same
|  linker.
|  
|  > compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, 
|  > compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, 
|  >
|  We've been meaning to get rid of the remaining static flags for quite
|  some time.
|  
|  I was a bit surprised to find that opt_NoStateHack is static. This is
|  a bit of an ugly one: the only user is a test in Id.isStateHackType
|  which is buried inside several layers [1] of pure predicates that have
|  no DynFlags. It would be a shame to pass a DynFlags through all of
|  these calls just to support the state hack (which we threaten drop
|  about once every six months).
|  
|  One option, of course, would be to merge all of the static flags into
|  DynFlags and rework their users to use unsafeGlobalDynFlags.
|  
|  [1] One particular callpath is,
|  
|  Id.isStateHackType
|  Id.typeOneShot
|  CoreArity.typeArity
|  CoreArity.exprArity
|  CorePrep.rhsToBody
|  CorePrep.cpeBody
|  
|  
|  >
|  compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,
|  >
|  The principle user of this is Outputable, which uses it to provide
|  DynFlags for pprTrace and friends. As you say, these are rather
|  special cases and it's probably fine if they behave a bit funky in the
|  case of more than one session in a process.
|  
|  > ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,
|  >
|  It is completely unclear why this is global at all. It seems like it
|  would fit just fine in the GHCi monad. I've opened D1789 doing exactly
|  this.
|  
|  There may be other global state that I'm not thinking of, but if this
|  is everything then it seems quite possible to fix this up for 8.2.
|  
|  Cheers,
|  
|  - Ben
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Multiple GHC sessions

[Alan & Kim Zimmerman]

>From a tool writer perspective I would be happy to have GHC/Linker
sessions be completely separate from each other. But I guess in
practical terms a shared address space needs to be managed, making
this impossible.

The external interpreter for GHCI effectively solves the actual
loading problem, the other things needing a GHC session are loading a
module as far as type checking, for error reporting. I know this gets
complicated if there is FFI involved and the loading has to go via
object file generation.

Alan

On Mon, Jan 18, 2016 at 12:53 PM, Ben Gamari  wrote:
> Simon Peyton Jones  writes:
>
>> I don't think the state-hack tail should wag the dog here. The nub of
>> the problem in practice is the shared linker state isn't it?
>>
> I'm not sure; it's not clear to me what would break if the linker were
> used by more than one session at a time, even with the current
> GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all
> sessions to use the same linker state given that they are also sharing
> an address space.
>
> The only issue that may require a bit of care is code unloading.
> Consider that you have two sessions A and B,
>
>   1. A loads MyModule
>   2. B loads MyModule
>   3. They both use MyModule for a while
>   4. A unloads MyModule
>   5. B continues
>   6. B unloads MyModule
>
> I suspect that things will work fine up to step 6 since we don't
> actually unload code until the garbage collector has marked the it as
> dead. That being said, I suspect you would see an error from unloadObj
> upon attempting to unload the object the second time (and even then, the
> error is merely a message on stderr). Note, however, that the above
> conclusions have not been tested.
>
> Otherwise, it appears that things should behave fair well as the
> PersistentLinkerState is already wrapped in an MVar which should
> serve to serialize linker calls.
>
> Cheers,
>
> - Ben
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Multiple GHC sessions

[Ben Gamari]

Alan & Kim Zimmerman  writes:

> At the moment there are issues with having multiple GHC API sessions
> in a single executable, which boil down to GHC having global
> variables.
>
> A quick grep over the GHC sources shows the following instances
>
> compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ...
> compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, 
>
This isn't the only global state had by the linker; have a look at
rts/Linker.c. Unfortunately this would take a fair amount of work to
resolve. The easiest solution here would probably be to put a lock
around the linker and have all sessions in a process use the same
linker.

> compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, 
> compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, 
>
We've been meaning to get rid of the remaining static flags for quite
some time.

I was a bit surprised to find that opt_NoStateHack is static. This is a
bit of an ugly one: the only user is a test in Id.isStateHackType which
is buried inside several layers [1] of pure predicates that have no
DynFlags. It would be a shame to pass a DynFlags through all of these
calls just to support the state hack (which we threaten drop about once
every six months).

One option, of course, would be to merge all of the static flags into
DynFlags and rework their users to use unsafeGlobalDynFlags.

[1] One particular callpath is,

Id.isStateHackType
Id.typeOneShot
CoreArity.typeArity
CoreArity.exprArity
CorePrep.rhsToBody
CorePrep.cpeBody


> compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,
>
The principle user of this is Outputable, which uses it to provide
DynFlags for pprTrace and friends. As you say, these are rather special
cases and it's probably fine if they behave a bit funky in the case of
more than one session in a process.

> ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,
>
It is completely unclear why this is global at all. It seems like it
would fit just fine in the GHCi monad. I've opened D1789 doing exactly
this.

There may be other global state that I'm not thinking of, but if this is
everything then it seems quite possible to fix this up for 8.2.

Cheers,

- Ben


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Multiple GHC sessions

[Alan & Kim Zimmerman]

At the moment there are issues with having multiple GHC API sessions
in a single executable, which boil down to GHC having global
variables.

A quick grep over the GHC sources shows the following instances

compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ...
compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, 

compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, 
compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, 

compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,

ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,

Of these the v_unsafeGlobalDynFlags can probably be ignored, as they
are intended to be used in very specific, limited circumstances only.

So, would it be possible to create a structure housing the remaining
ones, and somehow make it possible to access these in some kind of
session? Or preferably get rid of them, or move them into existing
structures such as DynFlags.

This is an exploratory email, more to get a handle on what has been
done/considered already in this space.

The issue does have a major impact on tooling developers though.

Regards
  Alan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Multiple GHC sessions

[Bartosz Nitka]

>From https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi:
"We can have multiple instances of a GHC Session on the GHC API, each
running TH and/or interpreted code. Right now this is not possible because
the linker's state is global."

Regards,
Bartosz

2016-01-17 14:49 GMT+00:00 Alan & Kim Zimmerman :

> At the moment there are issues with having multiple GHC API sessions
> in a single executable, which boil down to GHC having global
> variables.
>
> A quick grep over the GHC sources shows the following instances
>
> compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ...
> compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, 
>
> compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, 
> compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, 
>
> compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,
>
> ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,
>
> Of these the v_unsafeGlobalDynFlags can probably be ignored, as they
> are intended to be used in very specific, limited circumstances only.
>
> So, would it be possible to create a structure housing the remaining
> ones, and somehow make it possible to access these in some kind of
> session? Or preferably get rid of them, or move them into existing
> structures such as DynFlags.
>
> This is an exploratory email, more to get a handle on what has been
> done/considered already in this space.
>
> The issue does have a major impact on tooling developers though.
>
> Regards
>   Alan
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Multiple GHC sessions

[Alan & Kim Zimmerman]

Yes, but this solution is bypassing the problem by using remote execution.

My question is what could be done to make the remote execution
unnecessary, i.e addressing the "Right now" part.

Alan

On Sun, Jan 17, 2016 at 4:58 PM, Bartosz Nitka  wrote:
> From https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi:
> "We can have multiple instances of a GHC Session on the GHC API, each
> running TH and/or interpreted code. Right now this is not possible because
> the linker's state is global."
>
> Regards,
> Bartosz
>
> 2016-01-17 14:49 GMT+00:00 Alan & Kim Zimmerman :
>>
>> At the moment there are issues with having multiple GHC API sessions
>> in a single executable, which boil down to GHC having global
>> variables.
>>
>> A quick grep over the GHC sources shows the following instances
>>
>> compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ...
>> compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, 
>>
>> compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, 
>> compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, 
>>
>> compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,
>>
>> ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,
>>
>> Of these the v_unsafeGlobalDynFlags can probably be ignored, as they
>> are intended to be used in very specific, limited circumstances only.
>>
>> So, would it be possible to create a structure housing the remaining
>> ones, and somehow make it possible to access these in some kind of
>> session? Or preferably get rid of them, or move them into existing
>> structures such as DynFlags.
>>
>> This is an exploratory email, more to get a handle on what has been
>> done/considered already in this space.
>>
>> The issue does have a major impact on tooling developers though.
>>
>> Regards
>>   Alan
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs