RE: Dynamic loading problem
On 30 April 2004 18:01, Duncan Coutts wrote: > On Fri, 2004-04-30 at 17:36, Simon Marlow wrote: >> On 30 April 2004 11:26, Duncan Coutts wrote: >> >>> Could someone remind me what the remaining issues are in getting ghc >>> to build Haskell modules/packaged into unix .so shared libraries? > >> It might be possible to do this, but only if you want to do >> dlopen()-style linking at runtime. Linking a haskell binary against >> Haskell .so libraries still isn't possible - I had another look at >> this recently, and although one of the obstacles has been removed, >> there are others (details available on request). > > Details would be interesting. I can't guarantee that everything in here is correct, but here are the notes I made last time I looked into the problem: http://www.haskell.org/~simonmar/shared-libs if anyone has any further thoughts I'd be glad to collect them into a document. >> Any remaining issues to do with dlopen()ing a Haskell .so library are >> probably to do with CAFs. The dynamically loaded code needs to call >> newDynCAF() instead of newCAF() in a CAF's entry code - this is >> normally arranged by the RTS linker, but for a .so you'll have to do >> it some other way, like getting the compiler to emit the different >> call, or doing some CPP magic (but only for -fvia-C). Any >> reasonable patches to implement this will be incorporated. > > Does that involve changing what reference get linked to which symbols > or would it be a matter of finding a bunch of symbols & calling them > after doing a dlopen() ? Another way of asking the same thing is can > the calling newDynCAF be done in a separate phase (ie after dlopen()) > or is it intimately part of the linking process? It is done when a CAF is entered for the first time, so it would be difficult to do at link time. The real problem is that the garbage collector has no idea which CAFs are still reachable in dynamically linked code. For GHCi, we retain *all* CAFs in dynamically linked code for this reason. This is done by arranging that all dynamically loaded CAFs get placed on a list when they are first entered - this is what newDynCAF() does. If you load code using GHC's linker then this is the behaviour you get. A workaround for the problem, if you're not using GHC's linker, is to make StablePtrs for each of the functions you're going to call in the dynamically loaded code. This will ensure that the GC retains all CAFs reachable from those functions. Perhaps the dynamic linking library should do this? Cheers, Simon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: Dynamic loading problem
On Fri, 2004-04-30 at 17:36, Simon Marlow wrote: > On 30 April 2004 11:26, Duncan Coutts wrote: > > > Could someone remind me what the remaining issues are in getting ghc > > to build Haskell modules/packaged into unix .so shared libraries? > It might be possible to do this, but only if you want to do > dlopen()-style linking at runtime. Linking a haskell binary against > Haskell .so libraries still isn't possible - I had another look at this > recently, and although one of the obstacles has been removed, there are > others (details available on request). Details would be interesting. > Any remaining issues to do with dlopen()ing a Haskell .so library are > probably to do with CAFs. The dynamically loaded code needs to call > newDynCAF() instead of newCAF() in a CAF's entry code - this is normally > arranged by the RTS linker, but for a .so you'll have to do it some > other way, like getting the compiler to emit the different call, or > doing some CPP magic (but only for -fvia-C). Any reasonable patches to > implement this will be incorporated. Does that involve changing what reference get linked to which symbols or would it be a matter of finding a bunch of symbols & calling them after doing a dlopen() ? Another way of asking the same thing is can the calling newDynCAF be done in a separate phase (ie after dlopen()) or is it intimately part of the linking process? There seem to be several people interested in doing plug-in systems with dynamically loaded modules. I just came across this report: http://www.cs.chalmers.se/Cs/Grundutb/Kurser/afp/distinguished/dynamiclinker.pdf Duncan ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: Dynamic loading problem
On 30 April 2004 11:26, Duncan Coutts wrote: > Could someone remind me what the remaining issues are in getting ghc > to build Haskell modules/packaged into unix .so shared libraries? > > As I recall it seems to nearly work. Andre Pang got them to load and > function but it was a bit fragile and the rts would complain after a > short while: > http://www.mail-archive.com/[EMAIL PROTECTED]/msg03923.h tml It might be possible to do this, but only if you want to do dlopen()-style linking at runtime. Linking a haskell binary against Haskell .so libraries still isn't possible - I had another look at this recently, and although one of the obstacles has been removed, there are others (details available on request). Any remaining issues to do with dlopen()ing a Haskell .so library are probably to do with CAFs. The dynamically loaded code needs to call newDynCAF() instead of newCAF() in a CAF's entry code - this is normally arranged by the RTS linker, but for a .so you'll have to do it some other way, like getting the compiler to emit the different call, or doing some CPP magic (but only for -fvia-C). Any reasonable patches to implement this will be incorporated. Oh, and strictly speaking we should run the __stginit_Foo() function from each dynamically loaded module. But the RTS linker still doesn't do this. Cheers, Simon ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: Dynamic loading problem
Could someone remind me what the remaining issues are in getting ghc to build Haskell modules/packaged into unix .so shared libraries? As I recall it seems to nearly work. Andre Pang got them to load and function but it was a bit fragile and the rts would complain after a short while: http://www.mail-archive.com/[EMAIL PROTECTED]/msg03923.html Duncan ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: Dynamic loading problem
On 29 April 2004 16:55, Hampus Ram wrote:
> by using the RTS I've made it possible to dynamically load modules for
> quite some time now. However, recently I've stumbled inte quite a
> problem.
>
> I've created a bunch of modules to handle all dynamic loading for me
> and things are good for simple programs. But now I want to load my
> dynamic loading modules dynamically (yes, it sounds weird) and that
> can't be done - not on linux anyways.
>
> The functions from the rts are ffi:ed into my program but when
> dynamically loading the module, these functions can't be found
> ("unknown symbol `lookupSymbol`") and neither can I load the
> HSrts.o package to make them visible due to duplicate functions
> ("I found a duplicate definition for symbol makeStableNamezh_fast").
> How can I proceed? Can I pass some flag to the linker or such to force
> them to behave as I want?
The linker (ghc/rts/Linker.c) has an explicit symbol table for symbols
in the RTS that need to be visible in dynamically loaded code. Not
pretty, but it works and is cross-platform.
You need to add some more symbols to this table, and recompile the RTS.
Alternatively, you might be able to re-link HSrts.o with
--export-dynamic (something like 'ld -r --export-dynamic HSrts.o -o
my-HSrts.o') and then link this object into your program. This works by
making all the symbols in HSrts.o visible to the system's dynamic
linker, which is used to resolve symbols that the RTS dynamic linker
can't find.
> I only have this problem with linux (tested on Debian and Redhat)
> using ghc 6.2.1. On Solaris everything works just great (one doesn't
> even have to load any extra packages there) with ghc 6.0.1.
Not sure why it works on Solaris.
Cheers,
Simon
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Dynamic loading problem
On Thu, Apr 29 2004, Gregory Wright wrote:
> It has nothing to do with shared objects.
No, it hasn't and I have never claimed that it is so.
> the main program (the one that loads the loadable modules) to explicitly
> link all of the object files that the set {, modules>}
> depends on.
No. The problem isn't this and you need not do that in most cases as
long as you dynamically load the libraries too (which I think you really
should to allow truly dynamic loading not depending on which libraries
you compiled in). The problem is the functions in the RTS that are ffi:ed
into the dynamic loader. They are not "global" symbols on some platforms.
The rts is linked into alls programs by default (so specifying it separatly
is a no-op and waste of time).
I have no problem what so ever with anything _but_ dynamically loading
a module that requires functions from the RTS (i.e. loadObj,
lookupSymbol etc.).
However, I've now found the solution. All that was required was
to give the flag -optl-export-dynamic to ghc, not
-optl--export-dynamic, stupid mistake, but there it is...
Thanks to all that tried to help me!
/Hampus
--
Homepage: http://www.dtek.chalmers.se/~d00ram
E-mail: [EMAIL PROTECTED]
"Det är aldrig försent att ge upp"
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Dynamic loading problem
Hi,
I was playing with dynamically loaded modules and had the same problem.
It has nothing to do with shared objects. What you have to do is to
force
the main program (the one that loads the loadable modules) to explicitly
link all of the object files that the set {, }
depends on.
For example, if a module depends on symbols in
/HShaskell98.o,
(but the main program doesn't), you should explicit link
/HShaskell98.o
into the main program.
This happens because when you build your loadable modules the linker
assumes that they will later be linked into a main program, at which
time
all of the undefined references will be resolved. Since you never link
the modules
and main together, some references remain unresolved.
It shouldn't be hard to automate this; a script could run nm on the
module
and main program, make a list of unresolved references, search the files
in to know which object to include, and then relink the
main
program.
If you look at the makefiles in
http://www.algorithm.com.au/wiki/hacking/haskell.ghc_runtime_loading
you'll see some examples of how the main program has to be linked.
Best Wishes,
Greg
On Apr 29, 2004, at 3:24 PM, Hampus Ram wrote:
On Thu, Apr 29 2004, Alastair Reid wrote:
Does passing the flag RTLD_GLOBAL to dlopen help? (man dlopen for
info about
this flag)
No, since none of the objects involved are shared objects (I use
neither
dlopen or addDLL but the loadObj function). They are just compiled
Haskell
modules and the problem lies in the RTS symbols which for some reason
can't
be loaded dynamically (why it doesn't work when the base package etc.
can
be loaded is beyond me).
However when loading shared objects your flag is the one to use. That
works without flaw and is really a nicer way to load shared objects
than
the RTS function addDLL.
/Hampus
--
Homepage: http://www.dtek.chalmers.se/~d00ram
E-mail: [EMAIL PROTECTED]
"Det är aldrig försent att ge upp"
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Dynamic loading problem
On Thu, Apr 29 2004, Alastair Reid wrote: > Does passing the flag RTLD_GLOBAL to dlopen help? (man dlopen for info about > this flag) No, since none of the objects involved are shared objects (I use neither dlopen or addDLL but the loadObj function). They are just compiled Haskell modules and the problem lies in the RTS symbols which for some reason can't be loaded dynamically (why it doesn't work when the base package etc. can be loaded is beyond me). However when loading shared objects your flag is the one to use. That works without flaw and is really a nicer way to load shared objects than the RTS function addDLL. /Hampus -- Homepage: http://www.dtek.chalmers.se/~d00ram E-mail: [EMAIL PROTECTED] "Det är aldrig försent att ge upp" ___ Glasgow-haskell-users mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Dynamic loading problem
> The functions from the rts are ffi:ed into my program but when
> dynamically loading the module, these functions can't be found
> ("unknown symbol `lookupSymbol`") and neither can I load the
> HSrts.o package to make them visible due to duplicate functions
> ("I found a duplicate definition for symbol makeStableNamezh_fast").
> How can I proceed? Can I pass some flag to the linker or such to force
> them to behave as I want?
Does passing the flag RTLD_GLOBAL to dlopen help? (man dlopen for info about
this flag)
--
Alastair Reid
___
Glasgow-haskell-users mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
