Hi Simon, Sounds great!
This may very well what you have got in mind anyway, but I could imagine to run the interpreter on a different thread in the -fno-external-interpreter case and arrange communication through the same messaging API that you outlined for the seperate-process interpreter. Then, the essential difference between the two modes would be whether memory is shared or not (i.e., multithreading vs multi-process). Cheers, Manuel > Simon Marlow <[email protected]>: > > Hi Manuel, > > Thanks for the detailed reply, I have a much better understanding of your > requirements now. > > I'm going to support both models of running interpreted code. The current > plan is to have a flag, -fexternal-interpreter, which GHC will use by default > when running Template Haskell during compilation, and perhaps for GHCi, but > for compatibility with applications like yours I'll probably leave it off for > GHC API users. > > There's really no downside to doing this, it's not much more complicated than > implementing the separate-process model. > > Cheers, > Simon > > On 21/11/2015 03:38, Manuel M T Chakravarty wrote: >>> Simon Marlow <[email protected]>: >>> On 18/11/2015 01:41, Manuel M T Chakravarty wrote: >>>> Hi Simon, >>>> >>>> While this is an interesting proposal, Haskell for Mac strongly >>>> relies on running interpreted code in the same process. I’m using >>>> ’dynCompileExpr’ as well as ’hscStmtWithLocation’ and some other >>>> stuff. >>> >>> Let me say first of all that I'm not going to remove anything, so there's >>> no need to worry. But I'd like to explore exactly what you need, so that >>> we can see whether there's a way to accommodate it with a separate-process >>> implementation. >>> >>> hscStmtWithLocation is part of the core GHCi functionality, it is >>> definitely supported. It has a slightly different signature: >>> >>> hscStmtWithLocation :: HscEnv >>> -> String -- ^ The statement >>> -> String -- ^ The source >>> -> Int -- ^ Starting line >>> -> IO ( Maybe ([Id] >>> , RemoteHValue {- IO [HValue] -} >>> , FixityEnv)) >>> >>> RemoteHValue is a reference to a value in the interpreter's context. These >>> have to be evaluated via an explicit API, rather than just unsafeCoercing >>> Value as we do now. (this is not strictly speaking part of the GHC API, so >>> a separate but interesting question is: why did you need to use this >>> directly, and what should we add to the GHC API?) >> >> The GHC API basically assumes that the ”result” of statement execution is >> the *side-effect* of printing the result to stdout. This is not sufficient >> for an interactive graphical environment as >> >> (1) I want to have the result (even if it is a string) separate from >> anything else interpreted code execution writes to stdout. (In Haskell for >> Mac, these things are displayed in different places.) >> >> (2) I want results that are not just strings. For example, a result (of >> running Haskell code) may be a ForeignPtr to a C-land data structure >> representing an image (e.g., an in-memory representation of a PNG image >> rendered by Diagrams). >> >> For the latter, I’m actually using `compileExpr`, then `unsafeCoerce` the >> `hValue` into `IO (ForeignPtr ())` and `try` that (to also catch any >> exceptions). When this code runs, in some cases, it calls back and forth >> between interpreted Haskell code and the host application using the FFI. >> >>> I believe that many uses of dynCompileExpr can be changed so that the code >>> using the resulting value is moved into the interpreter’s context, and then >>> there’s no problem. >> >> This is difficult in my case, because the resulting value is used in the GUI >> code written in Swift. Code running in a different process cannot call the >> Cocoa framework methods for the GUI of the main process. >> >>>> This is quite crucial for some of the interactive >>>> functionality. Imagine a game where the game engine is in Swift >>>> linked into the main application and the game logic is in >>>> *interpreted* Haskell code. The engine calls into the Haskell code >>>> multiple times per frame of the animation and for all >>>> keyboard/mouse/etc input (using StablePtr and ForeignPtr to construct >>>> the scene graph across the Swift and Haskell heap). >>> >>> So my question is, why wouldn't you run the whole game engine in the >>> interpreter's context? That’s what would happen if you were to load the >>> program into GHCi and run it. >> >> On a fundamental level: The game engine runs on OpenGL. If it is in a >> different process, it cannot access the OpenGL context of the main process >> (which it needs to do to render into a specific view of a specific window of >> the main process). >> >> In practice, it is not just an OpenGL problem as I’m using a framework >> called SpriteKit with its own event and rendering loop that in turn uses >> OpenGL for the actual rendering. It does a lot of things behind the scenes >> (which makes it convenient to use), which requires you to be careful which >> threads you use to execute some operations. Running in an entire different >> process is surely going to break things. >> >>> Directly calling back and forth between the client of the GHC API and the >>> program being interpreted is arguably a strange thing to do, and it’s kind >>> of accidental that we allow it. >> >> I understand that, but I also think that it is an artefact of Haskell mostly >> being used in a command line program set up. I don’t think, it is just by >> chance that the IHaskell people do some quite similar things to at least >> some of what I’m doing. Once you want a more interactive experience, call >> patterns get more complicated. >> >>>> I actually also might have a use for the architecture that you are >>>> proposing. However, I really would like to keep the ability to, at >>>> least, optionally run interpreted code in the same process (without >>>> profiling etc). Do you think we could have both? >>> >>> We can certainly have both, it's straightforward to implement, but I don't >>> get to throw away some of the hacks we have to support same-process >>> execution, which would be a shame. We just add more code rather than >> >> Yes, I understand that and, as I wrote, I do like the idea of running in a >> separate process. However, it would also be a shame to prevent richer and >> more interactive experiences than CLI applications. >> >> I have thought a bit more about what the fundamental obstacle is. I think, >> it is two things: >> >> (1) I have interpreted Haskell code that (via a compiled Haskell library) >> uses FFI calls to call Cocoa system framework methods to create Cocoa >> objects. In Haskell, these Cocoa objects are referenced via a ForeignPtr and >> I need the interpreter to be able to return these foreign pointers. The >> ForeignPtr’s need to refer to memory of the main host process; hence, the >> FFI calls need to run the Cocoa framework code in the host process. >> >> (2) The Cocoa objects from (1) include both StablePtrs as well as C function >> pointers created via foreign dynamic wrapper. At least some of the >> StablePtrs refer to Haskell heap structures that need to be accessed by >> interpreted Haskell code. And calling the dynamic wrapper code from Swift in >> the main process needs to execute Haskell code that may refer to closures >> created by interpreted code. >> >> So, the issue really is that I would need FFI calls in the interpreter >> process that call Cocoa code in the main process and dynamic wrapper entry >> code in the main process that needs to call Haskell code in the interpreter >> process. (Crossing the FFI language chasm corresponds to cross-process >> calls.) >> >> I cannot move the Cocoa code from the main process to the interpreter >> process, as Cocoa requires that it runs on the *main* thread of the main >> process (to interact with the GUI and also to render via OpenGL). >> >> Does that make sense? >> >> Cheers, >> Manuel >> >>>>> Simon Marlow <[email protected]>: >>>>> >>>>> Hi folks - I've been thinking about changing the way we run interpreted >>>>> code so that it would be run in a separate process. It turns out this >>>>> has quite a few benefits, and would let us kill some of the really >>>>> awkward hacks we have in GHC to work around problems that arise because >>>>> we're running interpreted code and the compiler on the same runtime. >>>>> >>>>> I summarised the idea here: >>>>> https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi >>>>> >>>>> I'd be interested to hear if anyone has any thoughts around this, >>>>> particularly if doing this would make your life difficult in some way. >>>>> Are people relying on dynCompileExpr for anything? >>>>> >>>>> Cheers, >>>>> Simon >>>>> _______________________________________________ >>>>> ghc-devs mailing list >>>>> [email protected] >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>> >> > _______________________________________________ > ghc-devs mailing list > [email protected] > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs _______________________________________________ ghc-devs mailing list [email protected] http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
