Re: shared libs for OSX
On 2015-05-30 01:03, bitwise wrote: No, it opens the image for the main program, even if called from a shared lib. And that void* which is returned is actually a mach_header*, if I recall correctly. Not sure if this is true. I see, it was mostly a guess. -- /Jacob Carlborg
Re: shared libs for OSX
On Sat, 30 May 2015 05:06:34 -0400, Jacob Carlborg d...@me.com wrote: On 2015-05-30 01:03, bitwise wrote: No, it opens the image for the main program, even if called from a shared lib. And that void* which is returned is actually a mach_header*, if I recall correctly. Not sure if this is true. I see, it was mostly a guess. Not sure how accurate this code[1] is, but it seems dlopen returns a pointer to the image itself, which has a function machHeader(); http://www.opensource.apple.com/source/dyld/dyld-353.2.1/src/dyldAPIs.cpp dlopen returns magic numbers(RTLD_MAIN_ONLY or RTLD_DEFAULT) if you pass null, and sometimes it |s a 1 with the image pointer. So we can't really use it to get at the header. Bit
Re: shared libs for OSX
On Wednesday, 27 May 2015 at 21:24:25 UTC, bitwise wrote: Basically, I've gone back to the idea of using the dylib ctor/dtors. I don't think we really even need the image-added callback, at least not for dylibs. Looks good. The compiler could add a simple ctor/dtor to any D object.
Re: shared libs for OSX
On 2015-05-27 23:24, bitwise wrote: Good point. I've come up another solution and posted it here: http://dpaste.com/0DXBSNQ Will dladdr return different values (mach_header) for different dynamic libraries? Won't there only be one init function, as you said earlier. Or does it work somehow anyway? -- /Jacob Carlborg
Re: shared libs for OSX
On Friday, 29 May 2015 at 12:46:43 UTC, Jacob Carlborg wrote: On 2015-05-27 23:24, bitwise wrote: Good point. I've come up another solution and posted it here: http://dpaste.com/0DXBSNQ Will dladdr return different values (mach_header) for different dynamic libraries? Won't there only be one init function, as you said earlier. Or does it work somehow anyway? I'm going to test it out this weekend, but if I do get the wrong address, I think I can add a symbol with __attribute__ ((visibility (hidden))) and use dladdr on that instead. I don't think hidden symbols should collide.
Re: shared libs for OSX
On Fri, 29 May 2015 14:05:10 -0400, Jacob Carlborg d...@me.com wrote: On 2015-05-29 18:45, bitwise wrote: I'm going to test it out this weekend, but if I do get the wrong address, I think I can add a symbol with __attribute__ ((visibility (hidden))) and use dladdr on that instead. I don't think hidden symbols should collide. Doesn't dlopen open the current image if null is passed? No, it opens the image for the main program, even if called from a shared lib. And that void* which is returned is actually a mach_header*, if I recall correctly. Not sure if this is true. This works for sure in terms of getting a mach_header*, but I still have to check if visibility(hidden) allows each shared lib to have it's own copy of _symbol: __attribute__ ((visibility(hidden))) static int _symbol = 1; Dl_info info; if(dladdr(symbol, info)) const struct mach_header* myHeader = cast(const struct mach_header*)info.dli_fbase https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dladdr.3.html Bit
Re: shared libs for OSX
On 2015-05-29 18:45, bitwise wrote: I'm going to test it out this weekend, but if I do get the wrong address, I think I can add a symbol with __attribute__ ((visibility (hidden))) and use dladdr on that instead. I don't think hidden symbols should collide. Doesn't dlopen open the current image if null is passed? And that void* which is returned is actually a mach_header*, if I recall correctly. -- /Jacob Carlborg
Re: shared libs for OSX
On 2015-05-27 23:24, bitwise wrote: Good point. I've come up another solution and posted it here: http://dpaste.com/0DXBSNQ BTW, I'm not sure what's the best solution but you can calculate the slide without iterating all images. Look at the first statements in tlv_allocate_and_initialize_for_key in [1]. Specifically where it sets the slide variable. Basically, I've gone back to the idea of using the dylib ctor/dtors. I don't think we really even need the image-added callback, at least not for dylibs. I callback could still be used when the main application is written in D, but we could also replace it with my solution in the dpaste(the getThisImageInfo function). I think it would be simpler to only have one way to do this. As far as TLS goes, I haven't looking into the details of actually initializing the images yet. Once we have a solution for when and how to initialize the dylibs, I'll move on to the specifics... pun intended ;) Fair enough. I think the solution above may be a winner though. I agree, but I think ctor/dtors can be used for the executable as well. [1] http://opensource.apple.com/source/dyld/dyld-353.2.1/src/threadLocalVariables.c -- /Jacob Carlborg
Re: shared libs for OSX
On Tuesday, 26 May 2015 at 16:25:52 UTC, bitwise wrote: Isn't it better to avoid private undocumented functions? Not only better, but mandatory, otherwise Apple will reject the app from the app store. Calling back into an unloaded image without proving a mean to deregister the callback is a bug, so you might fix it. Looking at the code [1] again there is also an interesting sImagesToNotifyAboutOtherImages. [1]: http://opensource.apple.com/source/dyld/dyld-95.3/src/dyld.cpp
Re: shared libs for OSX
On Wednesday, 27 May 2015 at 06:45:49 UTC, Jacob Carlborg wrote: I'm not sure. The ___tls_get_addr function [1] is used when accessing a TLS variable on OS X. In all native implementations, both on OS X and Linux, the parameter is not just a void* but struct containing the image header as well. On Linux you call it with an dso index and an offset. The DSO index is assigned by the runtime linker (there is a special relocation for that). Something similar could be done manually if not natively supported on OSX.
Re: shared libs for OSX
On Tuesday, 26 May 2015 at 16:25:52 UTC, bitwise wrote: Since all global functions and symbols are shared between images anyways, receiving the callback in the main image would be fine. So in this case, unregistering the callbacks is no longer needed. That only works when the host executable is linked against druntime, but falls when you load runtime dynamically, e.g. a D plugin for a C host.
Re: shared libs for OSX
On 2015-05-27 15:38, Martin Nowak wrote: On Linux you call it with an dso index and an offset. The DSO index is assigned by the runtime linker (there is a special relocation for that). Something similar could be done manually if not natively supported on OSX. It is natively support. -- /Jacob Carlborg
Re: shared libs for OSX
On Wed, 27 May 2015 02:45:57 -0400, Jacob Carlborg d...@me.com wrote: What about using a D dynamic library in a C application? The C application would initialize the runtime which would register the callback. Then it would be undefined to unload druntime? Good point. I've come up another solution and posted it here: http://dpaste.com/0DXBSNQ Basically, I've gone back to the idea of using the dylib ctor/dtors. I don't think we really even need the image-added callback, at least not for dylibs. I callback could still be used when the main application is written in D, but we could also replace it with my solution in the dpaste(the getThisImageInfo function). As far as TLS goes, I haven't looking into the details of actually initializing the images yet. Once we have a solution for when and how to initialize the dylibs, I'll move on to the specifics... pun intended ;) I think the solution above may be a winner though. Bit
Re: shared libs for OSX
On 2015-05-26 18:25, bitwise wrote: I think Martin is right. We don't need ctors/dtors or any compiler fanciness. All we need is the two callbacks, which can be registered when druntime is initialized. _dyld_register_func_for_add_image _dyld_register_func_for_remove_image At this point, we would only be registering the callbacks once in the main image, and not from the shared library. Since all global functions and symbols are shared between images anyways, receiving the callback in the main image would be fine. So in this case, unregistering the callbacks is no longer needed. What about using a D dynamic library in a C application? The C application would initialize the runtime which would register the callback. Then it would be undefined to unload druntime? How would loading shared libraries change this? Couldn't TLS, however it's implemented now, be applied to shared libraries as well? I'm not sure. The ___tls_get_addr function [1] is used when accessing a TLS variable on OS X. In all native implementations, both on OS X and Linux, the parameter is not just a void* but struct containing the image header as well. Looking at SectionGroup [2] and how its data is initialized [3] you can see that there's only one set of TLS data (__tls_data and __tlscoal_nt) in SectionGroup. It would be straight forward to change that to an array but then you would not know which index to access in getTLSBlockAlloc [4] which is used by ___tls_get_addr. [1] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L115 [2] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L26 [3] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L227-L239 [4] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L172 -- /Jacob Carlborg
Re: shared libs for OSX
On 2015-05-25 21:40, bitwise wrote: So then I think I have a full solution: 1) _dyld_register_func_for_add_image should be taken care of with the above two fixes 2) __attribute__((constructor/destructor)) can be added to druntime when building for osx like in the file dylib_fixes.c [1] 3) copy paste rt_init/rt_term, rename them to dylib_init/dylib_term and remove everything except whats needed to initialize a shared lib's image. Does this make sense? You plan to use __attribute__((constructor)) instead of _dyld_register_func_for_add_image should? As Marin said, you need to look at sections_osx.d and _d_dso_registry. What do you plan to do about TLS? -- /Jacob Carlborg
Re: shared libs for OSX
On 2015-05-25 22:58, Martin Nowak wrote: On Monday, 25 May 2015 at 19:40:52 UTC, bitwise wrote: 1) _dyld_register_func_for_add_image should be taken care of with the above two fixes You still cannot unregister the callback, so it can't be used for dynamically loading druntime. Last time we talked about this problem, we found some undocumented function that could be deregistered. 2) __attribute__((constructor/destructor)) can be added to druntime when building for osx like in the file dylib_fixes.c [1] For linux we let the compiler emit comdat constructors into every D object, so you'll end up with exactly a single function for any binary containing D code. I don't think you need ctors/dtors on OSX if you already have the dylib callback. I'm not sure if there is any other solution. There is one private function, dyld_register_image_state_change_handler, that should work. I think it works because the image is never unloaded. I have not seen any function for deregistering a callback, private or public. Isn't it better to avoid private undocumented functions? -- /Jacob Carlborg
Re: shared libs for OSX
On Tue, 26 May 2015 02:28:14 -0400, Jacob Carlborg d...@me.com wrote: On 2015-05-25 22:58, Martin Nowak wrote: On Monday, 25 May 2015 at 19:40:52 UTC, bitwise wrote: 1) _dyld_register_func_for_add_image should be taken care of with the above two fixes You still cannot unregister the callback, so it can't be used for dynamically loading druntime. Last time we talked about this problem, we found some undocumented function that could be deregistered. 2) __attribute__((constructor/destructor)) can be added to druntime when building for osx like in the file dylib_fixes.c [1] For linux we let the compiler emit comdat constructors into every D object, so you'll end up with exactly a single function for any binary containing D code. I don't think you need ctors/dtors on OSX if you already have the dylib callback. I'm not sure if there is any other solution. There is one private function, dyld_register_image_state_change_handler, that should work. I think it works because the image is never unloaded. I have not seen any function for deregistering a callback, private or public. The I think Martin is right. We don't need ctors/dtors or any compiler fanciness. All we need is the two callbacks, which can be registered when druntime is initialized. _dyld_register_func_for_add_image _dyld_register_func_for_remove_image At this point, we would only be registering the callbacks once in the main image, and not from the shared library. Since all global functions and symbols are shared between images anyways, receiving the callback in the main image would be fine. So in this case, unregistering the callbacks is no longer needed. Isn't it better to avoid private undocumented functions? Not only better, but mandatory, otherwise Apple will reject the app from the app store. I am certain this is the case for iOS, and I assume it would be the same for desktop. On Tue, 26 May 2015 02:30:51 -0400, Jacob Carlborg d...@me.com wrote: What do you plan to do about TLS? How would loading shared libraries change this? Couldn't TLS, however it's implemented now, be applied to shared libraries as well? Bit
Re: shared libs for OSX
I've been reading through the Mach-O docs[1], and it seems that dynamic libs are treated the same as static libs in that exported symbols can only be defined once, even across dynamically loaded libraries. This is why calling rt_init from my dylib ended up calling the one that was already defined in the application image. I was able to call the rt_init from the dylib by specifically requesting it through dlsym, but then, all the global variables it used(_isRuntimeInitialized, etc) still ended up resolving to the ones in the application image. At this point, my impression is that it would be very impractical, if not impossible to have separate druntimes for each shared library. Even when you do link separate runtimes, dyld still treats all the exported symbols as shared. @Martin So correct me if I'm wrong, but it seems the _only_ choice is a shared druntime for osx. Could you elaborate a bit on how you've managed to do this for linux? [1] https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/MachOTopics/0-Introduction/introduction.html Thanks, Bit
Re: shared libs for OSX
On 2015-05-25 16:33, bitwise wrote: I've been reading through the Mach-O docs[1], and it seems that dynamic libs are treated the same as static libs in that exported symbols can only be defined once, even across dynamically loaded libraries. This is why calling rt_init from my dylib ended up calling the one that was already defined in the application image. I was able to call the rt_init from the dylib by specifically requesting it through dlsym, but then, all the global variables it used(_isRuntimeInitialized, etc) still ended up resolving to the ones in the application image. At this point, my impression is that it would be very impractical, if not impossible to have separate druntimes for each shared library. Even when you do link separate runtimes, dyld still treats all the exported symbols as shared. @Martin So correct me if I'm wrong, but it seems the _only_ choice is a shared druntime for osx. Could you elaborate a bit on how you've managed to do this for linux? If I recall correctly, this is how it works: On Linux the compiler will generate a function which is placed in a special section in the binary, same as annotating a C function with __attribute__((constructor)). This function calls a function in the druntime. This will give similar properties as _dyld_register_func_for_add_image on OS X but without the issues with registered callbacks Each image then becomes responsible to initialize itself. The image updates the shared data structure containing the necessary data (TLS, exception handling tables, ...). As you noticed yourself, all global symbols are shared across all images. -- /Jacob Carlborg
Re: shared libs for OSX
On Monday, 25 May 2015 at 14:33:43 UTC, bitwise wrote: At this point, my impression is that it would be very impractical, if not impossible to have separate druntimes for each shared library. Even when you do link separate runtimes, dyld still treats all the exported symbols as shared. Yes, you can't mix them with a D executable.
Re: shared libs for OSX
On Monday, 25 May 2015 at 19:40:52 UTC, bitwise wrote: 1) _dyld_register_func_for_add_image should be taken care of with the above two fixes You still cannot unregister the callback, so it can't be used for dynamically loading druntime. Last time we talked about this problem, we found some undocumented function that could be deregistered. 2) __attribute__((constructor/destructor)) can be added to druntime when building for osx like in the file dylib_fixes.c [1] For linux we let the compiler emit comdat constructors into every D object, so you'll end up with exactly a single function for any binary containing D code. I don't think you need ctors/dtors on OSX if you already have the dylib callback. 3) copy paste rt_init/rt_term, rename them to dylib_init/dylib_term and remove everything except whats needed to initialize a shared lib's image. Not sure what you want to copy, but for sure you need to extend sections_osx to handle multiple dylibs. It should be very similar to _d_dso_registry for ELF, except that you iterate over the sections of a dylib image to get EH tables and ModuleInfo.
Re: shared libs for OSX
On Mon, 25 May 2015 14:39:37 -0400, Jacob Carlborg d...@me.com wrote: On 2015-05-25 16:33, bitwise wrote: I've been reading through the Mach-O docs[1], and it seems that dynamic libs are treated the same as static libs in that exported symbols can only be defined once, even across dynamically loaded libraries. This is why calling rt_init from my dylib ended up calling the one that was already defined in the application image. I was able to call the rt_init from the dylib by specifically requesting it through dlsym, but then, all the global variables it used(_isRuntimeInitialized, etc) still ended up resolving to the ones in the application image. At this point, my impression is that it would be very impractical, if not impossible to have separate druntimes for each shared library. Even when you do link separate runtimes, dyld still treats all the exported symbols as shared. @Martin So correct me if I'm wrong, but it seems the _only_ choice is a shared druntime for osx. Could you elaborate a bit on how you've managed to do this for linux? If I recall correctly, this is how it works: On Linux the compiler will generate a function which is placed in a special section in the binary, same as annotating a C function with __attribute__((constructor)). This function calls a function in the druntime. This will give similar properties as _dyld_register_func_for_add_image on OS X but without the issues with registered callbacks Each image then becomes responsible to initialize itself. The image updates the shared data structure containing the necessary data (TLS, exception handling tables, ...). As you noticed yourself, all global symbols are shared across all images. So then I think I have a full solution: 1) _dyld_register_func_for_add_image should be taken care of with the above two fixes 2) __attribute__((constructor/destructor)) can be added to druntime when building for osx like in the file dylib_fixes.c [1] 3) copy paste rt_init/rt_term, rename them to dylib_init/dylib_term and remove everything except whats needed to initialize a shared lib's image. Does this make sense? Thanks, Bit [1] https://github.com/D-Programming-Language/druntime/blob/61ba4b8d3c0052065c17ffc8eef4f11496f3db3e/src/rt/dylib_fixes.c
Re: shared libs for OSX
I've read through these now, which I missed the first time around, so
sorry for making you guys repeat yourselves ;)
Runtime issue on Mac OS X
http://comments.gmane.org/gmane.comp.lang.d.runtime/1214
ideas for runtime loading of shared libraries.
http://forum.dlang.org/thread/mailman.2052.1325532031.24802.digitalmar...@puremagic.com
So in terms of a shared lib having it's own runtime, we have these
problems:
[1] problem
On Fri, 22 May 2015 12:04:24 -0400, Martin Nowak c...@dawg.eu wrote:
Yes separate shared libraries (with multiple runtimes) work on every
other platform.
The problem for OSX is that onAddImage gets called for the executable
and every shared library. It would be trivial to only register the image
containing the current runtime, by comparing the address of a private
symbol with the memory region of the images.
https://github.com/D-Programming-Language/druntime/blob/6331ab1ae19f3ff82449a5734b59d81b128685f4/src/rt/sections_osx.d#L186
[2] problem
On Thu, 21 May 2015 15:34:56 -0400, Jacob Carlborg d...@me.com wrote:
The runtime uses the _dyld_register_func_for_add_image function
provided by the dynamic linker. This function is used to register a
callback. The callback will be called for each currently loaded image
(executable/dynamic library) in the executable. The callback will also
be called for every newly loaded image, i.e. using dlopen. You need to
keep track of which image is yourself and which are other images you
should ignore.
Then, the other problem with _dyld_register_func_for_add_image is that
it's not possible to unregister the callback. Which means, if you
register a callback using this function from a dynamic library loaded
with dlopen and then unload it. Next time dlopen is called it will
crash, because the callback points to an address that doesn't exist
anymore.
I think I have found solutions for these problems.
[1] solution
I've modified sections_osx.d as follows:
extern (C) void _sections_osx_onAddImage_STUB(in mach_header* h, intptr_t
slide) {
// empty
}
extern (C) void sections_osx_onAddImage(in mach_header* h, intptr_t slide)
{
// on mac osx, Dl_info.dli_fbase is a pointer to the mach_header for the
library. [I]
// here we return unless onAddImage is being called for the current library
mach_header* myHeader = null;
Dl_info info;
// this line also makes sure that the stub isn't
// removed by the linker(it's needed for [2])
if(dladdr(_sections_osx_onAddImage_STUB, info))
{
mach_header* mh = cast(mach_header*)info.dli_fbase;
if(mh == cast(mach_header*)h)
myHeader = mh;
}
if(!myHeader)
return;
// initialize sections.
}
[2] solution
Although the callback passed to _dyld_register_func_for_add_image cannot
be removed, it can be replaced, so I've replaced it with a pointer to the
stub located in the main program.
I've modified initSections() in sections_osx.d as follows:
void initSections()
{
pthread_key_create(_tlsKey, null);
// register the callback as usual. This will call the callback
// for every library currently loaded before returning, so
// once it has returned, it should be safe to set the callback
// to something else(the empty stub)
_dyld_register_func_for_add_image(sections_osx_onAddImage);
// dlopen(null, ..) will retrieve a handle to the main program [II]
// OSX docs says it returns the first symbol found using
// RTLD_DEFAULT or the default library search order which
// should(and does as far as I can tell) return the handle
// to the main program
void *main = dlopen(null, RTLD_NOW);
assert(main);
alias typeof(_sections_osx_onAddImage_STUB) addImgFn;
addImgFn func = cast(addImgFn)dlsym(main,
_sections_osx_onAddImage_STUB);
assert(func);
// set the callback to the empty stub in the main program
_dyld_register_func_for_add_image(func);
_isRuntimeInitialized = true;
}
[I]
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dladdr.3.html
[II]
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html
http://linux.die.net/man/3/dlopen
So at this point, it seems like these two fixes work as expected, but now,
I'm having some new and very strange problems.
I have a simple shared library and program I've been using to test this:
[main.d]
module main;
import std.stdio;
import std.conv;
import std.string;
import core.sys.posix.dlfcn;
void main(string[] args)
{
alias void function() fnType;
void *handle = dlopen(myShared.dylib, RTLD_NOW);
assert(handle);
fnType init = cast(fnType)dlsym(handle, initLib);
assert(init);
init();
fnType term = cast(fnType)dlsym(handle, termLib);
assert(term);
term();
dlclose(handle);
writeln(done);
}
[myShared.d]
module myShared;
import core.runtime;
import
Re: shared libs for OSX
On Sun, 24 May 2015 15:40:04 -0400, bitwise bitwise@gmail.com wrote:
[snip]
So at this point, it seems like these two fixes work as expected, but
now, I'm having some new and very strange problems.
I have a simple shared library and program I've been using to test this:
[main.d]
module main;
import std.stdio;
import std.conv;
import std.string;
import core.sys.posix.dlfcn;
void main(string[] args)
{
alias void function() fnType;
void *handle = dlopen(myShared.dylib, RTLD_NOW);
assert(handle);
fnType init = cast(fnType)dlsym(handle, initLib);
assert(init);
init();
fnType term = cast(fnType)dlsym(handle, termLib);
assert(term);
term();
dlclose(handle);
writeln(done);
}
[myShared.d]
module myShared;
import core.runtime;
import std.stdio;
extern(C) void initLib() {
writeln(Initializing Runtime);
Runtime.initialize();
}
extern(C) void termLib() {
writeln(Terminating Runtime);
Runtime.terminate();
}
So, when I run the above program, rt_init() should be called once for
the main program, and once for the shared library. However, when I run
the above program, rt_init() from the main program seems to get called
twice. To clarify, I mean that when I retrieve initLib() with dlsym()
and call it, rt_init() from the main module gets called.
This seems to prove the above:
In dmain2.d, I have modified rt_init() as follows:
extern (C) int rt_init()
{
import core.sys.posix.dlfcn;
Dl_info info;
if(dladdr(rt_init, info))
fprintf(stdout, RT INIT: %s\n, info.dli_fname); // this
prints main for both calls
if (atomicOp!+=(_initCount, 1) 1)
{
fprintf(stdout, RT ALREADY INITIALIZED\n);
return 1;
}
// ...
fprintf(stdout, RT INIT COMPLETE\n);
}
When the main program calls rt_init(), the output correctly reads RT
INIT COMPLETE.
When I load the dynamic library however, I get the output RT ALREADY
INITIALIZED
How is this possible? I am not using a shared druntime afaik..
I just found that gcc has -fvisibility=hidden and
-fvisibility-ms-compat which are equivalent. The lack of this
functionality for dmd seems to explain the above problem.
Is this correct?
Any ideas?
Bit
Re: shared libs for OSX
On 2015-05-22 18:12, Martin Nowak wrote: One could also make emulated TLS work with shared libraries. Yeah, but then you would need to implement something similar to the TLS code already present in the dynamic linker in druntime. Both alternatives would require changes to both the compiler and runtime. Hopefully implementing native TLS in the compiler would not be any more difficult than adding support for shared libraries to the emulated TLS. -- /Jacob Carlborg
Re: shared libs for OSX
On 2015-05-22 03:13, bitwise wrote: So does that mean that _dyld_register_func_for_add_image shouldn't be called at all from shared libs at all then? The fact that there is no unregister function seems to suggest that it's meant to be called once from the main executable to add a callbacck that survives the entire life of the program. Yes. Can't whatever the callback does be done from the runtime linked to the main executable? The runtime extracts some data from the loaded image and stores it in some global variable. If the executable would do this it needs to store the data in the global variable in the dynamic library. -- /Jacob Carlborg
Re: shared libs for OSX
On Wednesday, 20 May 2015 at 21:35:38 UTC, bitwise wrote: Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least). Have you tried LDC? They recently got shared library support, maybe also on OSX.
Re: shared libs for OSX
On Thursday, 21 May 2015 at 08:07:49 UTC, Jacob Carlborg wrote: I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1]. One could also make emulated TLS work with shared libraries.
Re: shared libs for OSX
On Thursday, 21 May 2015 at 01:31:37 UTC, bitwise wrote: I've been reading over all the bugs and conversations I can find, but I'm still confused about exactly what's going on here. What I need: -load a dynamic library on OSX using dlopen or Runtime.loadLibrary -use dlsym to retrieve my functions and set up a gc proxy Forget about the GC proxy, it's a hack to work around ODR issues. What you really need is a shared phobos library as we have on linux or freebsd. -pull interfaces of classes defined in the dynamic library into my program for use. If you want to exchange data and code across multiple shared libraries and your executable the runtime must be fully aware of all shared libraries to support casting ,exceptions and finalizers.
Re: shared libs for OSX
On Thursday, 21 May 2015 at 15:51:38 UTC, bitwise wrote: I'm not sure exactly what you mean about integrated runtime. I think he means Phobos/Druntime as shared library. Looking in /usr/share/dmd/lib, I see phobos as a static library (which I'm assuming includes druntime, which I don't see anywhere). If the runtime is linked as a static library, shouldn't it be able to link to a dynamic library and just work? I think a separate runtime is what I was expecting. Yes separate shared libraries (with multiple runtimes) work on every other platform. The problem for OSX is that onAddImage gets called for the executable and every shared library. It would be trivial to only register the image containing the current runtime, by comparing the address of a private symbol with the memory region of the images. https://github.com/D-Programming-Language/druntime/blob/6331ab1ae19f3ff82449a5734b59d81b128685f4/src/rt/sections_osx.d#L186
Re: shared libs for OSX
On 2015-05-20 23:35, bitwise wrote: Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least). I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1]. [1] https://issues.dlang.org/show_bug.cgi?id=9476 -- /Jacob Carlborg
Re: shared libs for OSX
On 2015-05-21 07:06, Joakim wrote: Actually, TLS works on OS X: dmd emulates it, while ldc has native TLS using undocumented OS X APIs. The emulated TLS won't work with dynamic libraries. The runtime can only fetch the TLS data from one source, which will be the executable itself. I'm unsure of how full-fledged the support is on those other issues, but I believe the big issue is that loading multiple shared libraries is not supported, as that will take more work: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L198 No, that issue is loading dynamic libraries regardless if it's one or several. As you can see in the comment, it will pick the sections from the executable. If you want to talk to someone about getting more of shared libraries working, I believe Martin is your man. Yeah, but I'm not sure how familiar he is with the OS X specific things. And native TLS is a requirement, which requires changes in the compiler and the runtime. Alternatively we need to reimplement the TLS handling from the dynamic linker. -- /Jacob Carlborg
Re: shared libs for OSX
dlopen() of a D shared library works when called from a C++ file, and that a C++ file can call multiple D shared libraries simultaneously when using RTLD_LAZY option. Before waiting for a full solution with an integrated druntime, is there at least a way to have a separate runtime in each shared library, so that dlopen() can be called from a D file? On Thu, May 21, 2015 at 1:07 AM, Jacob Carlborg via Digitalmars-d digitalmars-d@puremagic.com wrote: On 2015-05-20 23:35, bitwise wrote: Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least). I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1]. [1] https://issues.dlang.org/show_bug.cgi?id=9476 -- /Jacob Carlborg
Re: shared libs for OSX
On Thu, 21 May 2015 15:34:56 -0400, Jacob Carlborg d...@me.com wrote: On 2015-05-21 10:12, Timothee Cour via Digitalmars-d wrote: dlopen() of a D shared library works when called from a C++ file, and that a C++ file can call multiple D shared libraries simultaneously when using RTLD_LAZY option. Before waiting for a full solution with an integrated druntime, is there at least a way to have a separate runtime in each shared library, so that dlopen() can be called from a D file? No, not out of the box. I'm not sure if I remember all the details correctly, but this is basically how it works: The runtime uses the _dyld_register_func_for_add_image function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore. Then, the other problem with _dyld_register_func_for_add_image is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore. There's an other, undocumented, function that does basically the same but prevents the issue with the callback by not allowing the dynamic library to be unloaded. So does that mean that _dyld_register_func_for_add_image shouldn't be called at all from shared libs at all then? The fact that there is no unregister function seems to suggest that it's meant to be called once from the main executable to add a callbacck that survives the entire life of the program. Can't whatever the callback does be done from the runtime linked to the main executable? Bit
Re: shared libs for OSX
On Thu, 21 May 2015 21:13:56 -0400, bitwise bitwise@gmail.com wrote: Can't whatever the callback does be done from the runtime linked to the main executable? Or simply use dlsym from the main executable to call back into the dylib to set things up? Bit
Re: shared libs for OSX
On 2015-05-21 17:51, bitwise wrote: I'm not sure exactly what you mean about integrated runtime. Looking in /usr/share/dmd/lib, I see phobos as a static library (which I'm assuming includes druntime, which I don't see anywhere). If the runtime is linked as a static library, shouldn't it be able to link to a dynamic library and just work? I think a separate runtime is what I was expecting. No, see my answer to Timothee [1]. [1] http://forum.dlang.org/post/mjlc10$2pi5$1...@digitalmars.com -- /Jacob Carlborg
Re: shared libs for OSX
On 2015-05-21 10:12, Timothee Cour via Digitalmars-d wrote: dlopen() of a D shared library works when called from a C++ file, and that a C++ file can call multiple D shared libraries simultaneously when using RTLD_LAZY option. Before waiting for a full solution with an integrated druntime, is there at least a way to have a separate runtime in each shared library, so that dlopen() can be called from a D file? No, not out of the box. I'm not sure if I remember all the details correctly, but this is basically how it works: The runtime uses the _dyld_register_func_for_add_image function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore. Then, the other problem with _dyld_register_func_for_add_image is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore. There's an other, undocumented, function that does basically the same but prevents the issue with the callback by not allowing the dynamic library to be unloaded. -- /Jacob Carlborg
Re: shared libs for OSX
On Thu, 21 May 2015 04:12:19 -0400, Timothee Cour via Digitalmars-d digitalmars-d@puremagic.com wrote: dlopen() of a D shared library works when called from a C++ file, and that a C++ file can call multiple D shared libraries simultaneously when using RTLD_LAZY option. Before waiting for a full solution with an integrated druntime, is there at least a way to have a separate runtime in each shared library, so that dlopen() can be called from a D file? I'm not sure exactly what you mean about integrated runtime. Looking in /usr/share/dmd/lib, I see phobos as a static library (which I'm assuming includes druntime, which I don't see anywhere). If the runtime is linked as a static library, shouldn't it be able to link to a dynamic library and just work? I think a separate runtime is what I was expecting. On Thu, May 21, 2015 at 1:07 AM, Jacob Carlborg via Digitalmars-d digitalmars-d@puremagic.com wrote: I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1]. [1] https://issues.dlang.org/show_bug.cgi?id=9476 -- /Jacob Carlborg (In reply to John Colvin from comment #11) I think the time has come that 10.6 support could be dropped, no? It's no longer supported by Apple. +1 Bit
Re: shared libs for OSX
On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote: On 2015-05-20 16:44, bitwise wrote: I tried using a shared library for OSX yesterday. I opened it with dlopen, retrieved my extern(C) function, and called it. All was well, and it seemed to work(wrote to the console with writeln). But, I got a message in the console saying shared libraries were not yet implemented for OSX. What exactly is it that's not working? TLS, module constructors, module/class info, perhaps the GC, similar things. Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).
Re: shared libs for OSX
On 2015-05-20 16:44, bitwise wrote: I tried using a shared library for OSX yesterday. I opened it with dlopen, retrieved my extern(C) function, and called it. All was well, and it seemed to work(wrote to the console with writeln). But, I got a message in the console saying shared libraries were not yet implemented for OSX. What exactly is it that's not working? TLS, module constructors, module/class info, perhaps the GC, similar things. -- /Jacob Carlborg
Re: shared libs for OSX
On Wed, 20 May 2015 17:35:37 -0400, bitwise bitwise@gmail.com wrote: On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote: On 2015-05-20 16:44, bitwise wrote: I tried using a shared library for OSX yesterday. I opened it with dlopen, retrieved my extern(C) function, and called it. All was well, and it seemed to work(wrote to the console with writeln). But, I got a message in the console saying shared libraries were not yet implemented for OSX. What exactly is it that's not working? TLS, module constructors, module/class info, perhaps the GC, similar things. Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least). I've been reading over all the bugs and conversations I can find, but I'm still confused about exactly what's going on here. What I need: -load a dynamic library on OSX using dlopen or Runtime.loadLibrary -use dlsym to retrieve my functions and set up a gc proxy -pull interfaces of classes defined in the dynamic library into my program for use. Not sure how much of this is working properly or not. I'd be willing to throw a couple of bucks at this for a bounty, but I'm not sure how much it's worth though. Any takers for $25? ;) Bit
Re: shared libs for OSX
On Wednesday, 20 May 2015 at 21:35:38 UTC, bitwise wrote: On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote: On 2015-05-20 16:44, bitwise wrote: I tried using a shared library for OSX yesterday. I opened it with dlopen, retrieved my extern(C) function, and called it. All was well, and it seemed to work(wrote to the console with writeln). But, I got a message in the console saying shared libraries were not yet implemented for OSX. What exactly is it that's not working? TLS, module constructors, module/class info, perhaps the GC, similar things. Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least). Actually, TLS works on OS X: dmd emulates it, while ldc has native TLS using undocumented OS X APIs. I'm unsure of how full-fledged the support is on those other issues, but I believe the big issue is that loading multiple shared libraries is not supported, as that will take more work: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L198 If you want to talk to someone about getting more of shared libraries working, I believe Martin is your man.
