Re: Multiple GHC sessions
On 18 January 2016 at 11:18, Alan & Kim Zimmerman wrote: > From a tool writer perspective I would be happy to have GHC/Linker > sessions be completely separate from each other. But I guess in > practical terms a shared address space needs to be managed, making > this impossible. > > The external interpreter for GHCI effectively solves the actual > loading problem, the other things needing a GHC session are loading a > module as far as type checking, for error reporting. I know this gets > complicated if there is FFI involved and the loading has to go via > object file generation. > Yes, I think -fexternal-interpreter is the basis of the right solution here. You do also need to move the PersistentLinkerState to HscEnv, but that should be a fairly straightforward refactoring. (actually I almost did this as part of the remote GHCi work, but some of the comments next to PersistentLinkerState suggested that there might be some real use cases for having the linker state be shared between multiple sessions, so I decided not to attack that problem right away.) Cheers, Simon Alan > > On Mon, Jan 18, 2016 at 12:53 PM, Ben Gamari wrote: > > Simon Peyton Jones writes: > > > >> I don't think the state-hack tail should wag the dog here. The nub of > >> the problem in practice is the shared linker state isn't it? > >> > > I'm not sure; it's not clear to me what would break if the linker were > > used by more than one session at a time, even with the current > > GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all > > sessions to use the same linker state given that they are also sharing > > an address space. > > > > The only issue that may require a bit of care is code unloading. > > Consider that you have two sessions A and B, > > > > 1. A loads MyModule > > 2. B loads MyModule > > 3. They both use MyModule for a while > > 4. A unloads MyModule > > 5. B continues > > 6. B unloads MyModule > > > > I suspect that things will work fine up to step 6 since we don't > > actually unload code until the garbage collector has marked the it as > > dead. That being said, I suspect you would see an error from unloadObj > > upon attempting to unload the object the second time (and even then, the > > error is merely a message on stderr). Note, however, that the above > > conclusions have not been tested. > > > > Otherwise, it appears that things should behave fair well as the > > PersistentLinkerState is already wrapped in an MVar which should > > serve to serialize linker calls. > > > > Cheers, > > > > - Ben > > > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Multiple GHC sessions
>From a tool writer perspective I would be happy to have GHC/Linker sessions be completely separate from each other. But I guess in practical terms a shared address space needs to be managed, making this impossible. The external interpreter for GHCI effectively solves the actual loading problem, the other things needing a GHC session are loading a module as far as type checking, for error reporting. I know this gets complicated if there is FFI involved and the loading has to go via object file generation. Alan On Mon, Jan 18, 2016 at 12:53 PM, Ben Gamari wrote: > Simon Peyton Jones writes: > >> I don't think the state-hack tail should wag the dog here. The nub of >> the problem in practice is the shared linker state isn't it? >> > I'm not sure; it's not clear to me what would break if the linker were > used by more than one session at a time, even with the current > GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all > sessions to use the same linker state given that they are also sharing > an address space. > > The only issue that may require a bit of care is code unloading. > Consider that you have two sessions A and B, > > 1. A loads MyModule > 2. B loads MyModule > 3. They both use MyModule for a while > 4. A unloads MyModule > 5. B continues > 6. B unloads MyModule > > I suspect that things will work fine up to step 6 since we don't > actually unload code until the garbage collector has marked the it as > dead. That being said, I suspect you would see an error from unloadObj > upon attempting to unload the object the second time (and even then, the > error is merely a message on stderr). Note, however, that the above > conclusions have not been tested. > > Otherwise, it appears that things should behave fair well as the > PersistentLinkerState is already wrapped in an MVar which should > serve to serialize linker calls. > > Cheers, > > - Ben > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
RE: Multiple GHC sessions
Simon Peyton Jones writes: > I don't think the state-hack tail should wag the dog here. The nub of > the problem in practice is the shared linker state isn't it? > I'm not sure; it's not clear to me what would break if the linker were used by more than one session at a time, even with the current GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all sessions to use the same linker state given that they are also sharing an address space. The only issue that may require a bit of care is code unloading. Consider that you have two sessions A and B, 1. A loads MyModule 2. B loads MyModule 3. They both use MyModule for a while 4. A unloads MyModule 5. B continues 6. B unloads MyModule I suspect that things will work fine up to step 6 since we don't actually unload code until the garbage collector has marked the it as dead. That being said, I suspect you would see an error from unloadObj upon attempting to unload the object the second time (and even then, the error is merely a message on stderr). Note, however, that the above conclusions have not been tested. Otherwise, it appears that things should behave fair well as the PersistentLinkerState is already wrapped in an MVar which should serve to serialize linker calls. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
RE: Multiple GHC sessions
I don't think the state-hack tail should wag the dog here. The nub of the problem in practice is the shared linker state isn't it? Simon | -Original Message- | From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Ben | Gamari | Sent: 17 January 2016 19:55 | To: Alan & Kim Zimmerman ; ghc-devs@haskell.org | Subject: Re: Multiple GHC sessions | | Alan & Kim Zimmerman writes: | | > At the moment there are issues with having multiple GHC API sessions | > in a single executable, which boil down to GHC having global | > variables. | > | > A quick grep over the GHC sources shows the following instances | > | > compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ... | > compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, | > | This isn't the only global state had by the linker; have a look at | rts/Linker.c. Unfortunately this would take a fair amount of work to | resolve. The easiest solution here would probably be to put a lock | around the linker and have all sessions in a process use the same | linker. | | > compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, | > compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, | > | We've been meaning to get rid of the remaining static flags for quite | some time. | | I was a bit surprised to find that opt_NoStateHack is static. This is | a bit of an ugly one: the only user is a test in Id.isStateHackType | which is buried inside several layers [1] of pure predicates that have | no DynFlags. It would be a shame to pass a DynFlags through all of | these calls just to support the state hack (which we threaten drop | about once every six months). | | One option, of course, would be to merge all of the static flags into | DynFlags and rework their users to use unsafeGlobalDynFlags. | | [1] One particular callpath is, | | Id.isStateHackType | Id.typeOneShot | CoreArity.typeArity | CoreArity.exprArity | CorePrep.rhsToBody | CorePrep.cpeBody | | | > | compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags, | > | The principle user of this is Outputable, which uses it to provide | DynFlags for pprTrace and friends. As you say, these are rather | special cases and it's probably fine if they behave a bit funky in the | case of more than one session in a process. | | > ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref, | > | It is completely unclear why this is global at all. It seems like it | would fit just fine in the GHCi monad. I've opened D1789 doing exactly | this. | | There may be other global state that I'm not thinking of, but if this | is everything then it seems quite possible to fix this up for 8.2. | | Cheers, | | - Ben ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Multiple GHC sessions
Alan & Kim Zimmerman writes: > At the moment there are issues with having multiple GHC API sessions > in a single executable, which boil down to GHC having global > variables. > > A quick grep over the GHC sources shows the following instances > > compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ... > compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, > This isn't the only global state had by the linker; have a look at rts/Linker.c. Unfortunately this would take a fair amount of work to resolve. The easiest solution here would probably be to put a lock around the linker and have all sessions in a process use the same linker. > compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, > compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, > We've been meaning to get rid of the remaining static flags for quite some time. I was a bit surprised to find that opt_NoStateHack is static. This is a bit of an ugly one: the only user is a test in Id.isStateHackType which is buried inside several layers [1] of pure predicates that have no DynFlags. It would be a shame to pass a DynFlags through all of these calls just to support the state hack (which we threaten drop about once every six months). One option, of course, would be to merge all of the static flags into DynFlags and rework their users to use unsafeGlobalDynFlags. [1] One particular callpath is, Id.isStateHackType Id.typeOneShot CoreArity.typeArity CoreArity.exprArity CorePrep.rhsToBody CorePrep.cpeBody > compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags, > The principle user of this is Outputable, which uses it to provide DynFlags for pprTrace and friends. As you say, these are rather special cases and it's probably fine if they behave a bit funky in the case of more than one session in a process. > ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref, > It is completely unclear why this is global at all. It seems like it would fit just fine in the GHCi monad. I've opened D1789 doing exactly this. There may be other global state that I'm not thinking of, but if this is everything then it seems quite possible to fix this up for 8.2. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Multiple GHC sessions
Yes, but this solution is bypassing the problem by using remote execution. My question is what could be done to make the remote execution unnecessary, i.e addressing the "Right now" part. Alan On Sun, Jan 17, 2016 at 4:58 PM, Bartosz Nitka wrote: > From https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi: > "We can have multiple instances of a GHC Session on the GHC API, each > running TH and/or interpreted code. Right now this is not possible because > the linker's state is global." > > Regards, > Bartosz > > 2016-01-17 14:49 GMT+00:00 Alan & Kim Zimmerman : >> >> At the moment there are issues with having multiple GHC API sessions >> in a single executable, which boil down to GHC having global >> variables. >> >> A quick grep over the GHC sources shows the following instances >> >> compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ... >> compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, >> >> compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, >> compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, >> >> compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags, >> >> ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref, >> >> Of these the v_unsafeGlobalDynFlags can probably be ignored, as they >> are intended to be used in very specific, limited circumstances only. >> >> So, would it be possible to create a structure housing the remaining >> ones, and somehow make it possible to access these in some kind of >> session? Or preferably get rid of them, or move them into existing >> structures such as DynFlags. >> >> This is an exploratory email, more to get a handle on what has been >> done/considered already in this space. >> >> The issue does have a major impact on tooling developers though. >> >> Regards >> Alan >> ___ >> ghc-devs mailing list >> ghc-devs@haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Multiple GHC sessions
>From https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi: "We can have multiple instances of a GHC Session on the GHC API, each running TH and/or interpreted code. Right now this is not possible because the linker's state is global." Regards, Bartosz 2016-01-17 14:49 GMT+00:00 Alan & Kim Zimmerman : > At the moment there are issues with having multiple GHC API sessions > in a single executable, which boil down to GHC having global > variables. > > A quick grep over the GHC sources shows the following instances > > compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ... > compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, > > compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, > compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, > > compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags, > > ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref, > > Of these the v_unsafeGlobalDynFlags can probably be ignored, as they > are intended to be used in very specific, limited circumstances only. > > So, would it be possible to create a structure housing the remaining > ones, and somehow make it possible to access these in some kind of > session? Or preferably get rid of them, or move them into existing > structures such as DynFlags. > > This is an exploratory email, more to get a handle on what has been > done/considered already in this space. > > The issue does have a major impact on tooling developers though. > > Regards > Alan > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs