Re: dynamic library building and loading
On Tuesday, 18 June 2013 at 20:23:42 UTC, 1100110 wrote: On 06/18/2013 10:14 AM, Bottled Gin wrote: Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up Hello D Experts I am replying to an old thread since I want to know if and how the situation has improved over the past few months. In particular I want to link and call fairly complex D functions from a C++ application. I am exclusively using Linux. Can somebody please guide me if I should expect things to work with current DMD master from github? Regards - Puneet There's a DConf talk about shared libraries in D. Perhaps there's useful information there? http://www.youtube.com/watch?v=i63VeudjZM4 (I'm watching it now so I don't know how detailed it is.) Hello all, I'm wondering if this has been resolved to date. I get this error on master: /usr/bin/ld: generated/linux/release/64/libphobos2.so.0.66.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC generated/linux/release/64/libphobos2.so.0.66.o: could not read symbols: Bad value collect2: error: ld returned 1 exit status even though I've built with fPIC: cc -c -m64 -fPIC -O3 etc/c/zlib/trees.c -ogenerated/linux/release/64/etc/c/zlib/trees.o cc -c -m64 -fPIC -O3 etc/c/zlib/uncompr.c -ogenerated/linux/release/64/etc/c/zlib/uncompr.o cc -c -m64 -fPIC -O3 etc/c/zlib/zutil.c -ogenerated/linux/release/64/etc/c/zlib/zutil.o etc...
Re: dynamic library building and loading
Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up Hello D Experts I am replying to an old thread since I want to know if and how the situation has improved over the past few months. In particular I want to link and call fairly complex D functions from a C++ application. I am exclusively using Linux. Can somebody please guide me if I should expect things to work with current DMD master from github? Regards - Puneet
Re: dynamic library building and loading
On 06/18/2013 10:14 AM, Bottled Gin wrote: Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up Hello D Experts I am replying to an old thread since I want to know if and how the situation has improved over the past few months. In particular I want to link and call fairly complex D functions from a C++ application. I am exclusively using Linux. Can somebody please guide me if I should expect things to work with current DMD master from github? Regards - Puneet There's a DConf talk about shared libraries in D. Perhaps there's useful information there? http://www.youtube.com/watch?v=i63VeudjZM4 (I'm watching it now so I don't know how detailed it is.)
Re: dynamic library building and loading
Am Mon, 01 Oct 2012 19:18:46 +0200 schrieb Jacob Carlborg d...@me.com: On 2012-10-01 17:06, Johannes Pfau wrote: the problem is that we don't want the C main function in a shared libgdruntime.so, because you might want to use libgdruntime.so in a C/C++ app which has it's own main function. So we currently don't link in dmain2.o into the shared library and it must be included manually when linking an application. (But dmain2 also contains some stuff that really should be in libdruntime.so, so this source file should probably be split up at some time.) I'm not sure if I follow this correctly or not, but why is this needed to be handled manually? If you pass -shared to the compiler just skip linking dmain2.o, otherwise link with it. Would that work? Yes something similar would work, it's just not yet implemented. GDC should detect if we're linking against a shared druntime and then it should automatically add dmain2.o to the linker command.
Re: dynamic library building and loading
On 2012-10-01 01:42, Rob T wrote: It seems that an attempt to make the runtime shared is well under way. Did anything get into the main dmd branch or has the effort been stalled or ...? Seems pretty stalled. I will look at this too. Thanks for the pointers. No problem. -- /Jacob Carlborg
Re: dynamic library building and loading
Am Sat, 29 Sep 2012 15:19:30 +0200 schrieb Jacob Carlborg d...@me.com: On 2012-09-28 20:25, Maxim Fomin wrote: I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are working and here is some kind of test: https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't executed when it is situated in a shared library and which calls some throwing function from another shared library. Unittests aren't working either. Regarding other parts - they seem to work. That's a fairly uninteresting test. You are linking to the dynamic library. What's interesting is loading a dynamic library using dlopen, or similar. What's the point of using dynamic libraries if you're linking with them? There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test It should also be enhanced to test multiple shared libraries. The good news: * Exception handling is working * ModuleInfos are working * unit tests are working * Static variables, gshared variables, tls variables are working * Object.factory is working * Calling functions, passing function pointers, passing classes between dso/app is working The bad news: * The GC doesn't scan TLS/__gshared/static data in dynamic libraries, it only scans the main app.
Re: dynamic library building and loading
On 2012-10-01 12:42, Johannes Pfau wrote: There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test It should also be enhanced to test multiple shared libraries. The good news: * Exception handling is working * ModuleInfos are working * unit tests are working * Static variables, gshared variables, tls variables are working * Object.factory is working * Calling functions, passing function pointers, passing classes between dso/app is working That's good. Have you tested this with DMD? The bad news: * The GC doesn't scan TLS/__gshared/static data in dynamic libraries, it only scans the main app. That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux. BTW, is the runtime and phobos statically linked both with the dynamic library and the executable? -- /Jacob Carlborg
Re: dynamic library building and loading
On 1 October 2012 13:34, Jacob Carlborg d...@me.com wrote: On 2012-10-01 12:42, Johannes Pfau wrote: There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test It should also be enhanced to test multiple shared libraries. The good news: * Exception handling is working * ModuleInfos are working * unit tests are working * Static variables, gshared variables, tls variables are working * Object.factory is working * Calling functions, passing function pointers, passing classes between dso/app is working That's good. Have you tested this with DMD? The bad news: * The GC doesn't scan TLS/__gshared/static data in dynamic libraries, it only scans the main app. That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux. BTW, is the runtime and phobos statically linked both with the dynamic library and the executable? -- /Jacob Carlborg On Linux, there has already been an runtime implementation written that scans /proc/self/maps and adds all data sections to the GC that way. Whether or not DMD wishes to go down that route is their own decision. I am looking into a solution that doesn't have any bearing on what platform it's running on... Regards -- Iain Buclaw *(p e ? p++ : p) = (c 0x0f) + '0';
Re: dynamic library building and loading
On 2012-10-01 14:40, Iain Buclaw wrote: On Linux, there has already been an runtime implementation written that scans /proc/self/maps and adds all data sections to the GC that way. Whether or not DMD wishes to go down that route is their own decision. I am looking into a solution that doesn't have any bearing on what platform it's running on... Well, /proc isn't available on Mac OS X so I think you have to continue looking. -- /Jacob Carlborg
Re: dynamic library building and loading
On 2012-10-01 12:42, Johannes Pfau wrote: There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test Isn't dmain2 used when building shared libraries using GDC? That's where the implementation of rt_init is located. -- /Jacob Carlborg
Re: dynamic library building and loading
On 2012-10-01 14:40, Iain Buclaw wrote: On Linux, there has already been an runtime implementation written that scans /proc/self/maps and adds all data sections to the GC that way. Whether or not DMD wishes to go down that route is their own decision. I am looking into a solution that doesn't have any bearing on what platform it's running on... I think this is already working on Mac OS X. It's handled by: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L82 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L133 -- /Jacob Carlborg
Re: dynamic library building and loading
Am Mon, 01 Oct 2012 14:34:21 +0200 schrieb Jacob Carlborg d...@me.com: On 2012-10-01 12:42, Johannes Pfau wrote: There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test It should also be enhanced to test multiple shared libraries. The good news: * Exception handling is working * ModuleInfos are working * unit tests are working * Static variables, gshared variables, tls variables are working * Object.factory is working * Calling functions, passing function pointers, passing classes between dso/app is working That's good. Have you tested this with DMD? Not yet. The bad news: * The GC doesn't scan TLS/__gshared/static data in dynamic libraries, it only scans the main app. That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux. BTW, is the runtime and phobos statically linked both with the dynamic library and the executable? I tested two different configurations: druntime and phobos are shared libraries as well (this is the correct solution, it's mostly working except for the GC issues) druntime and phobos static linking: phobos and druntime are statically linked into the app, libdso.so is not linked against druntime/phobos at all. (this is a hack to get better test results: With a shared druntime, you can't call GC.collect twice, because the first call frees important objects in druntime and then the second call segfaults)
Re: dynamic library building and loading
Am Mon, 01 Oct 2012 15:11:49 +0200 schrieb Jacob Carlborg d...@me.com: On 2012-10-01 12:42, Johannes Pfau wrote: There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test Isn't dmain2 used when building shared libraries using GDC? That's where the implementation of rt_init is located. the problem is that we don't want the C main function in a shared libgdruntime.so, because you might want to use libgdruntime.so in a C/C++ app which has it's own main function. So we currently don't link in dmain2.o into the shared library and it must be included manually when linking an application. (But dmain2 also contains some stuff that really should be in libdruntime.so, so this source file should probably be split up at some time.)
Re: dynamic library building and loading
On 1 October 2012 16:06, Johannes Pfau nos...@example.com wrote: Am Mon, 01 Oct 2012 15:11:49 +0200 schrieb Jacob Carlborg d...@me.com: On 2012-10-01 12:42, Johannes Pfau wrote: There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test Isn't dmain2 used when building shared libraries using GDC? That's where the implementation of rt_init is located. the problem is that we don't want the C main function in a shared libgdruntime.so, because you might want to use libgdruntime.so in a C/C++ app which has it's own main function. That is at least one of the theories behind it anyway. :-) The more I think about it, the less I think I would want a C++ app to link against a D shared library though. With C, there will need to be defined a common interface header - similar to what used to be mars.h in rt/ but also contain some useful runtime functions. Regards -- Iain Buclaw *(p e ? p++ : p) = (c 0x0f) + '0';
Re: dynamic library building and loading
On 2012-10-01 17:03, Johannes Pfau wrote: I tested two different configurations: druntime and phobos are shared libraries as well (this is the correct solution, it's mostly working except for the GC issues) I agree. druntime and phobos static linking: phobos and druntime are statically linked into the app, libdso.so is not linked against druntime/phobos at all. (this is a hack to get better test results: With a shared druntime, you can't call GC.collect twice, because the first call frees important objects in druntime and then the second call segfaults) Ok. -- /Jacob Carlborg
Re: dynamic library building and loading
On 2012-10-01 17:06, Johannes Pfau wrote: the problem is that we don't want the C main function in a shared libgdruntime.so, because you might want to use libgdruntime.so in a C/C++ app which has it's own main function. So we currently don't link in dmain2.o into the shared library and it must be included manually when linking an application. (But dmain2 also contains some stuff that really should be in libdruntime.so, so this source file should probably be split up at some time.) I'm not sure if I follow this correctly or not, but why is this needed to be handled manually? If you pass -shared to the compiler just skip linking dmain2.o, otherwise link with it. Would that work? -- /Jacob Carlborg
Re: dynamic library building and loading
On Saturday, 29 September 2012 at 16:09:14 UTC, Jacob Carlborg wrote: There's no point in doing that. But what I'm saying is if you know beforehand the libraries you will use you can get quite far with static libraries. There are plenty of cases where you have to use a dynamically loaded lib even if you know before hand what will be loaded. I think you understand this if I read your posts correctly. In my case I have a pre-existing C++ app that is designed to load user defined C++ plugins. I wanted to interface D to one of my own C++ plugins, but I apparently cannot do it reliably because of initialization issues with the GC and possibly some other obscure items. If I can figure out what needs to be done to resolve the problem in enough detail, then maybe I can hack the runtime source and roll out a solution. The GC always seems to pop up as a source of problems. For long term solution, the GC should be made 100% optional (in practice as opposed to in theory), the standard library ought to be made to work with or wothout a GC (or simply without), and the GC itself should be easily replaceable with alternate versions. I think this idea has already been discussed elsewhere, and is on the TODO list (I hope!). --rt
Re: dynamic library building and loading
On Friday, 28 September 2012 at 21:26:47 UTC, Iain Buclaw wrote: The way I intend to address it is to have each module handle it's own gshared/thread data and pass the information to the D runtime during the module construction stage (.ctor) - there is already something done this way for _Dmodule_ref - so it may end up being that two new fields will be tacked onto it; void[] tlsdata, void[] gsharedata; I re-built libgphobos and libgdruntime with -fPIC and I can now successfully create dynamically loaded D libs. I have successfully linked a dynamic D lib to a C++ app, executing some test code successfully. I have not yet tried to dlopen a dynamic D lib from C++, but I will try maybe tomorrow. My simple dynamic lib test seems to run fine, but I understand that there may be problems, such as the GC failing to work properly, and perhaps more can/will go wrong as per Jacob's post http://forum.dlang.org/post/k4219f$uft$1...@digitalmars.com Any idea when/if you will get around to implementing a fix? Wish I could help but I've only just started looking at the source code, so whatever I try to fix will probably cause more harm than good for a while (but it's a start). --rt
Re: dynamic library building and loading
On 2012-09-30 08:41, Rob T wrote: There are plenty of cases where you have to use a dynamically loaded lib even if you know before hand what will be loaded. I think you understand this if I read your posts correctly. In my case I have a pre-existing C++ app that is designed to load user defined C++ plugins. I wanted to interface D to one of my own C++ plugins, but I apparently cannot do it reliably because of initialization issues with the GC and possibly some other obscure items. But now we're back at plugins. I think this part of the discussion is starting to run in circles and become quite pointless. I think most of us know that we need to support all types of libraries. Static and dynamic, both for link time and runtime. If I can figure out what needs to be done to resolve the problem in enough detail, then maybe I can hack the runtime source and roll out a solution. You can have a look at the work done by Martin Nowak: https://github.com/dawgfoto/druntime/commits/SharedRuntime He has a couple of other useful branches as well, for example: https://github.com/dawgfoto/druntime/commits/dmain2Refactoring On Mac OS X there's also the problem with TLS. There is no native support for TLS in Mac OS X prior to Lion (10.7). DMD has rolled its own implementation that needs to be adapted to support dynamic libraries. The GC always seems to pop up as a source of problems. For long term solution, the GC should be made 100% optional (in practice as opposed to in theory), the standard library ought to be made to work with or wothout a GC (or simply without), and the GC itself should be easily replaceable with alternate versions. I think this idea has already been discussed elsewhere, and is on the TODO list (I hope!). You can replace the GC at link time. Here's an example of a stubbed implemented of the GC: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub This is for Tango but the druntime is based on the Tango runtime so I would guess most of this would be the same. -- /Jacob Carlborg
Re: dynamic library building and loading
On Sunday, 30 September 2012 at 10:57:24 UTC, Jacob Carlborg wrote: On 2012-09-30 08:41, Rob T wrote: I think most of us know that we need to support all types of libraries. Static and dynamic, both for link time and runtime. OK we're all in agreement on this point. You can have a look at the work done by Martin Nowak: https://github.com/dawgfoto/druntime/commits/SharedRuntime It seems that an attempt to make the runtime shared is well under way. Did anything get into the main dmd branch or has the effort been stalled or ...? I've only recently been learning what's going on with D and why it is what it is, so a breif history lesson may be needed once in a while. You can replace the GC at link time. Here's an example of a stubbed implemented of the GC: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub I will look at this too. Thanks for the pointers. --rt
Re: dynamic library building and loading
On 2012-09-28 20:25, Maxim Fomin wrote: I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are working and here is some kind of test: https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't executed when it is situated in a shared library and which calls some throwing function from another shared library. Unittests aren't working either. Regarding other parts - they seem to work. That's a fairly uninteresting test. You are linking to the dynamic library. What's interesting is loading a dynamic library using dlopen, or similar. What's the point of using dynamic libraries if you're linking with them? -- /Jacob Carlborg
Re: dynamic library building and loading
On 2012-09-28 19:54, Rob T wrote: I suppose the answer is very complicated, but why can't the runtime execute as a shared lib? Is it a design limitation of the runtime model? This sort of problem should have been dealt with from the ground up, not now. What can be done to fix it, or is it a major redesign + re-write effort? See one of my other posts: http://forum.dlang.org/thread/k3vfm9$1tq$1...@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Re: dynamic library building and loading
On Saturday, 29 September 2012 at 13:19:01 UTC, Jacob Carlborg wrote: That's a fairly uninteresting test. I am not a D developer which means I have no incentive in blindly portraying D as a language having shared libraries support when in fact it has some issues. I am a D user which has incentive to gladly report which part of D does work (or seems to work) and which doesn't when making shared libraries. You are linking to the dynamic library. What's interesting is loading a dynamic library using dlopen, or similar. What's the point of using dynamic libraries if you're linking with them? I was not taking about dynamic loading, but about dynamic linking. If I understand topic right, the issue is the former, not the latter. BTW, in majority cases dynamic loading gives no advantages over dynamic linking (just the opposite: doing unnecessary job which can be done by linker and loader). In most cases, when an app is being written, it is known which functional from which third-party libraries would be used. The only exceptions I can name are pluggin support and hacker's binary ELF/PE tools. I can count few apps in windows and linux which are used often, which dynamically load something, and if they do, they likely load pluggins. This was made for cases when app developers by definition cannot know beforehand full list of used shared libraries. So, the question is just opposite: What's the point of using dynamic loading if you know beforehand which libraries you use, which happens almost in all cases?. But I certainly agree that dynamic loading is essential for some applications, if not irreplaceable, and support of it also needed to be inspected and improved.
Re: dynamic library building and loading
On 2012-09-29 17:40, Maxim Fomin wrote: I was not taking about dynamic loading, but about dynamic linking. If I understand topic right, the issue is the former, not the latter. The title of the thread says ... and loading. BTW, in majority cases dynamic loading gives no advantages over dynamic linking (just the opposite: doing unnecessary job which can be done by linker and loader). In most cases, when an app is being written, it is known which functional from which third-party libraries would be used. I agree, in these cases their no point in using dynamic loading. The only exceptions I can name are pluggin support and hacker's binary ELF/PE tools. I can count few apps in windows and linux which are used often, which dynamically load something, and if they do, they likely load pluggins. This was made for cases when app developers by definition cannot know beforehand full list of used shared libraries. Exactly, pluggins are one of the main reasons for using dynamic loading. An other reason is license issues. There's a D library called Derelict which uses dynamic loading to load C libraries (OpenGL, SDL, OpenAL and others). One of the many reason for Derelict using dynamic loading is due to license issues/complications with these libraries. http://dsource.org/projects/derelict/ So, the question is just opposite: What's the point of using dynamic loading if you know beforehand which libraries you use, which happens almost in all cases?. There's no point in doing that. But what I'm saying is if you know beforehand the libraries you will use you can get quite far with static libraries. -- /Jacob Carlborg
Re: dynamic library building and loading
On 2012-09-28 02:04, Rob T wrote: I'm using C++ main app that dynamically loads shared libs at runtime, but I cannot load D shared libs because they cannot be linked to the non-fPIC built phobos2/druntime lib. It seems that the only way to get things working is to re-build the C++ code so that it is linked with non-fPIC phobos2, but this is not something I want to be doing. So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number? No, it will not work ok. Depending on what you do in the D code it might work, but everything will not work. Perhaps if you limit yourself to a C subset. See one of my other posts: http://forum.dlang.org/thread/k3vfm9$1tq$1...@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Re: dynamic library building and loading
On 2012-09-28 01:21, Rob T wrote:
For me to get C or C++ to run a D function, I had to do the following:
//
// C/C++ library source
//
// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization shutdown
void init();
void done();
void bar()
{
foo(6, 7, 8);
}
int main(int argc, char **argv)
{
init();
bar();
done();
return 0;
}
//
// D library source
//
import std.stdio;
static import core.runtime;
// sample D function for test
extern (C++) //int foo(int i, int j, int k)
void foo(int i, int j, int k)
{
writefln(i = %s, i);
writefln(j = %s, j);
writefln(k = %s, k);
int t = i + j + k;
writefln(Total = %s, t);
}
// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after loading
shared lib
core.runtime.Runtime.initialize();
}
extern (C++) export void done() { // to be called before unloading
shared lib
core.runtime.Runtime.terminate();
}
// I had to include main even though this is a library.
int main()
{
return 0;
}
That is not sufficient to have everything work. It might be ok if you
limit yourself to a C subset of D. On some platforms, calling
Runtime.initialize, will miss initializing some parts.
The implementation of the runtime initialization contains a lot of code
duplication and some functions/branches are missing functionality. Example:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L346
The C main function handles the runtime initialization for a regular
statically linked executable. If you initialize the runtime via
rt_init, which is called by Runtime.initialize, it will miss to
initialize some parts on some platforms. The C main function should
really call rt_init to remove the code duplication.
Also see this post why everything will not properly work:
http://forum.dlang.org/thread/k3vfm9$1tq$1...@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com
--
/Jacob Carlborg
Re: dynamic library building and loading
Am Fri, 28 Sep 2012 02:04:11 +0200 schrieb Rob T r...@ucora.com: So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number? --rt It shouldn't compile on x86 (32bit) with -fPIC because there's some incompatible asm code in druntime and phobos. If dmd doesn't warn in these cases, this code will fail as soon as it's called. GDC used to warn about these errors, but as we dropped the D inline assembler that code doesn't affect us anymore and there are no more warnings. The fix for this issue is simple: Do not modify the EBX register, or at least save restore it. Old patch, which shows what needs to be done: https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329 Example asm code which probably doesn't work: https://github.com/D-Programming-Language/druntime/blob/master/src/core/cpuid.d#L432 I just tried building a shared druntime phobos with gdc: Druntime seems to be working, Phobos doesn't link here because something's wrong with the fstat64 symbol. I haven't done any real testing and we probably need some additional support in the runtime, but the compiler part seems to be OK with gdc. http://gdcproject.org/wiki/SharedRuntime https://github.com/D-Programming-GDC/gdc/pull/32
Re: dynamic library building and loading
On 28 September 2012 12:14, Johannes Pfau nos...@example.com wrote: Am Fri, 28 Sep 2012 02:04:11 +0200 schrieb Rob T r...@ucora.com: So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number? --rt It shouldn't compile on x86 (32bit) with -fPIC because there's some incompatible asm code in druntime and phobos. If dmd doesn't warn in these cases, this code will fail as soon as it's called. GDC used to warn about these errors, but as we dropped the D inline assembler that code doesn't affect us anymore and there are no more warnings. The fix for this issue is simple: Do not modify the EBX register, or at least save restore it. In GDC D1 phobos, one hack around it was to use db to manually write out the bytecode instruction (thus bypassing GCC 'ebx' clobbered checks). Which was a fiendish workaround to say the least. :~) -- Iain Buclaw *(p e ? p++ : p) = (c 0x0f) + '0';
Re: dynamic library building and loading
I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that even with a successulf PIC build I will still not be able to reliably execute D functions directly from C/C++ code. Old patch, which shows what needs to be done: https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329 Does anyone know why patches like this are taking so [bleeping] long to get recognized and implemented? -rt
Re: dynamic library building and loading
On 28 September 2012 17:43, Rob T r...@ucora.com wrote: I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that even with a successulf PIC build I will still not be able to reliably execute D functions directly from C/C++ code. Old patch, which shows what needs to be done: https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329 Does anyone know why patches like this are taking so [bleeping] long to get recognized and implemented? -rt Because the [bleeping] runtime does not support running from a shared library. -- Iain Buclaw *(p e ? p++ : p) = (c 0x0f) + '0';
Re: dynamic library building and loading
On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote: Because the [bleeping] runtime does not support running from a shared library. I suppose the answer is very complicated, but why can't the runtime execute as a shared lib? Is it a design limitation of the runtime model? This sort of problem should have been dealt with from the ground up, not now. What can be done to fix it, or is it a major redesign + re-write effort? -rt
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg wrote: Actually, I seriously doubt everything is working as expected. I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are working and here is some kind of test: https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't executed when it is situated in a shared library and which calls some throwing function from another shared library. Unittests aren't working either. Regarding other parts - they seem to work.
Re: dynamic library building and loading
On 28 September 2012 18:54, Rob T r...@ucora.com wrote: On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote: Because the [bleeping] runtime does not support running from a shared library. I suppose the answer is very complicated, but why can't the runtime execute as a shared lib? Is it a design limitation of the runtime model? This sort of problem should have been dealt with from the ground up, not now. What can be done to fix it, or is it a major redesign + re-write effort? -rt The big implementation issue is that the runtime only passes two data ranges to the GC to scan. Global data segment and TLS data segment of the resultant binary/executable. So any static data stored in shared libraries aren't scanned / recognised as data we want to keep. So the use of them may cause sudden unwanted collections. The way I intend to address it is to have each module handle it's own gshared/thread data and pass the information to the D runtime during the module construction stage (.ctor) - there is already something done this way for _Dmodule_ref - so it may end up being that two new fields will be tacked onto it; void[] tlsdata, void[] gsharedata; Regards -- Iain Buclaw *(p e ? p++ : p) = (c 0x0f) + '0';
Re: dynamic library building and loading
Hi,
I have same issue, but it is possible make shared library,
first of all you have to make shared variant of druntime and
phobos library, than it should work ok.
Now I am at work, when I come back home I will post some more
details about this.
Daniel Kozak
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei
Alexandrescu wrote:
Haven't done any dynamic linking with D and I need to. I'm
using dmd 2.058/Linux at work to build and use dynamic
libraries. Here's my attempt:
*** Makefile
all: main lib.so
main: main.d
dmd main
lib.so: lib.d
dmd -fPIC -shared lib.d -of./lib.so
*** lib.d
extern(C) int fun(string s)
{
return 42;
}
*** main.d
import std.stdio;
void main()
{
}
Running make prints:
dmd -fPIC -shared lib.d -of./lib.so
/usr/bin/ld:
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o):
relocation R_X86_64_32 against `a local symbol' can not be used
when making a shared object; recompile with -fPIC
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a:
could not read symbols: Bad value
collect2: ld returned 1 exit status
What steps do I need to take to get off the ground?
Thanks,
Andrei
Re: dynamic library building and loading
Now I try it, and it is not required to build shared variant of
druntime and phobos, only rebuild it with -fPIC
On Thursday, 27 September 2012 at 06:12:38 UTC, Daniel Kozak
wrote:
Hi,
I have same issue, but it is possible make shared library,
first of all you have to make shared variant of druntime and
phobos library, than it should work ok.
Now I am at work, when I come back home I will post some more
details about this.
Daniel Kozak
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei
Alexandrescu wrote:
Haven't done any dynamic linking with D and I need to. I'm
using dmd 2.058/Linux at work to build and use dynamic
libraries. Here's my attempt:
*** Makefile
all: main lib.so
main: main.d
dmd main
lib.so: lib.d
dmd -fPIC -shared lib.d -of./lib.so
*** lib.d
extern(C) int fun(string s)
{
return 42;
}
*** main.d
import std.stdio;
void main()
{
}
Running make prints:
dmd -fPIC -shared lib.d -of./lib.so
/usr/bin/ld:
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o):
relocation R_X86_64_32 against `a local symbol' can not be
used when making a shared object; recompile with -fPIC
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a:
could not read symbols: Bad value
collect2: ld returned 1 exit status
What steps do I need to take to get off the ground?
Thanks,
Andrei
Re: dynamic library building and loading
On Wednesday, 26 September 2012 at 20:15:35 UTC, nazriel wrote: On Wednesday, 26 September 2012 at 20:10:47 UTC, Michael wrote: Thanks. The loading part is very useful, but I'm still lost when it comes to build the shared library itself. Andrei Program loads dll at runtime using loader which is configured to load concrete dll file(s). Like in gtkD http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d Loading Shared lib isn't big issues here. The bigger one is building Shared library (written in D) and running it in host application without issues (EH, shared GC etc). Andrei, if you find out how to make those things work, please share your findings. I'm also in need of using shared libraries. And yeah, probably Martin Nowak will be the best bet to get information from. Native Oberon/BlueBottle OS really a lot in dynamic modules, as everything is dynamically loaded. As far as I know both operating systems have a global GC. Not sure how Spin does it. Singularity, uses local GC per process and does not support dynamic loading. Bringing this into D's context, I imagine a solution for the GC would be to have it shared across all modules. The biggest problem I see, is that all shared libraries have to use the same D compiler, or a standard GC ABI needs to be defined. -- Paulo
Re: dynamic library building and loading
Am Thu, 27 Sep 2012 08:26:36 +0200 schrieb Daniel Kozak kozz...@gmail.com: Now I try it, and it is not required to build shared variant of druntime and phobos, only rebuild it with -fPIC In the end you'll probably need a shared druntime phobos: Let's say your main app doesn't use std.datetime and you statically link libphobos. Then the linker might strip std.datetime from your executable. If your shared library now needs std.datetime it won't work. The funny part is that on ARM+GDC we can at least build druntime phobos as a shared library, but on x86/64 last time I checked it didn't work because of non-PIC assembly. https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:
Maxim Fomin wrote:
You can build shared libraries on linux by manually compiling
object
files and linking them. On windows last time I tries it was not
possible.
Can you give detailed steps for doing this on Linux? Because
nobody as
far as I know has made this work yet?
Jens
Dpaste seems not working, so, sorry for code
lib.d---
import std.stdio;
static this()
{
writeln(module ctor);
}
static ~this()
{
writeln(module dtor);
}
class A
{
private string text;;
this(string text)
{
writeln(class ctor);
this.text = text;
}
void tell()
{
writeln(this.text);
}
~this()
{
writeln(this.text);
writeln(dtor);
}
static this()
{
writeln(static ctor);
}
static ~this()
{
writeln(static dtor);
}
}
---
-main.d
import lib;
void main()
{
auto a = new A(some text);
a.tell();
}
---
dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
linux-vdso.so.1 (0x7fff703ff000)
liblib.so = ./liblib.so (0x7f48158f1000)
librt.so.1 = /lib64/librt.so.1 (0x7f48156cd000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x7f48154b1000)
libc.so.6 = /lib64/libc.so.6 (0x7f481510c000)
/lib64/ld-linux-x86-64.so.2 (0x7f4815af4000)
Re: dynamic library building and loading
On 2012-09-27 10:04, Maxim Fomin wrote:
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:
Maxim Fomin wrote:
You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.
Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?
Jens
Dpaste seems not working, so, sorry for code
lib.d---
import std.stdio;
static this()
{
writeln(module ctor);
}
static ~this()
{
writeln(module dtor);
}
class A
{
private string text;;
this(string text)
{
writeln(class ctor);
this.text = text;
}
void tell()
{
writeln(this.text);
}
~this()
{
writeln(this.text);
writeln(dtor);
}
static this()
{
writeln(static ctor);
}
static ~this()
{
writeln(static dtor);
}
}
---
-main.d
import lib;
void main()
{
auto a = new A(some text);
a.tell();
}
---
dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
linux-vdso.so.1 (0x7fff703ff000)
liblib.so = ./liblib.so (0x7f48158f1000)
librt.so.1 = /lib64/librt.so.1 (0x7f48156cd000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x7f48154b1000)
libc.so.6 = /lib64/libc.so.6 (0x7f481510c000)
/lib64/ld-linux-x86-64.so.2 (0x7f4815af4000)
1. Does this actually run?
2. This is statically linked with druntime and Phobos. What happens when
you create an executable that links with the D dynamic library?
Last time I tried this (on Mac OS X) I got several symbols missing. This
was all symbols that are usually pointing to the executable, inserted by
the compiler. One of them would be main and symbols like these:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27
--
/Jacob Carlborg
Re: dynamic library building and loading
Jacob Carlborg wrote:
On 2012-09-27 10:04, Maxim Fomin wrote:
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:
Maxim Fomin wrote:
You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.
Can you give detailed steps for doing this on Linux? Because nobody as
far as I know has made this work yet?
Jens
Dpaste seems not working, so, sorry for code
lib.d---
import std.stdio;
static this()
{
writeln(module ctor);
}
static ~this()
{
writeln(module dtor);
}
class A
{
private string text;;
this(string text)
{
writeln(class ctor);
this.text = text;
}
void tell()
{
writeln(this.text);
}
~this()
{
writeln(this.text);
writeln(dtor);
}
static this()
{
writeln(static ctor);
}
static ~this()
{
writeln(static dtor);
}
}
---
-main.d
import lib;
void main()
{
auto a = new A(some text);
a.tell();
}
---
dmd -c -fPIC lib.d
gcc -shared lib.o -o liblib.so
dmd -c main.d
gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=.
./a.out
ldd a.out
linux-vdso.so.1 (0x7fff703ff000)
liblib.so = ./liblib.so (0x7f48158f1000)
librt.so.1 = /lib64/librt.so.1 (0x7f48156cd000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x7f48154b1000)
libc.so.6 = /lib64/libc.so.6 (0x7f481510c000)
/lib64/ld-linux-x86-64.so.2 (0x7f4815af4000)
1. Does this actually run?
I just tried.
$ ./a.out
module ctor
static ctor
class ctor
some text
static dtor
module dtor
some text
dtor
2. This is statically linked with druntime and Phobos. What happens
when you create an executable that links with the D dynamic library?
a.out is linked dynamically against liblib.so.
Last time I tried this (on Mac OS X) I got several symbols missing.
This was all symbols that are usually pointing to the executable,
inserted by the compiler. One of them would be main and symbols
like these:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27
I'm running Linux. You can test on Mac OS X.
I'm also astonished that this works. This is great.
Jens
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg wrote: 1. Does this actually run? If it were non-runnable, I wouldn't posted it. 2. This is statically linked with druntime and Phobos. What happens when you create an executable that links with the D dynamic library? Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first. Last time I tried this (on Mac OS X) I got several symbols missing. This was all symbols that are usually pointing to the executable, inserted by the compiler. One of them would be main and symbols like these: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27 I have no idea about D support of shared libraries on Mac OS.
Re: dynamic library building and loading
On 2012-09-27 10:55, Maxim Fomin wrote: If it were non-runnable, I wouldn't posted it. Ok, I see. Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first. Obviously he wants to use the shared library, otherwise it would be rather useless and pointless. Last I checked there was a problem with: * Using a shared library written in D with a D application * Making druntime and Phobos shared libraries * Possibly also compiling shared libraries -- /Jacob Carlborg
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 09:16:48 UTC, Jacob Carlborg wrote: On 2012-09-27 10:55, Maxim Fomin wrote: If it were non-runnable, I wouldn't posted it. Ok, I see. Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first. Obviously he wants to use the shared library, otherwise it would be rather useless and pointless. Last I checked there was a problem with: * Using a shared library written in D with a D application * Making druntime and Phobos shared libraries * Possibly also compiling shared libraries With DMD 2.060 last two points are possible, only first point (Using a shared ...) I dont try
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg
wrote:
Last time I tried this (on Mac OS X) I got several symbols
missing. This was all symbols that are usually pointing to the
executable, inserted by the compiler. One of them would be
main and symbols like these:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27
_deh_ problem AFAIK arise when non-D and D code is merged and
can be avoided. For example, assume C/C++ application using D
library. This can be done by writing D main function which
forwards call to C/C++ main function which uses D shared library
code. Again, sorry for code post, dpaste isn't working:
---main.cpp---
extern C void foo(void);
#include stdio.h
class CPP
{
public:
CPP() { printf(CPP ctor\n); }
};
CPP cpp;
extern C int c_main_func()
{
printf(C main function reached\n);
foo();
// C/C++ application
return 0;
}
---dmain.d---
import std.stdio;
extern(C) int c_main_func();
static this()
{
writeln(main module ctor);
}
static ~this()
{
writeln(main module dtor);
}
int main()
{
return c_main_func();
}
---test.d---
import std.stdio;
static this()
{
writeln(lib module ctor);
}
static ~this()
{
writeln(lib module dtor);
}
extern(C) void foo()
{
auto a = new A(data);
writeln(in foo);
}
class A
{
string x;
static this() { writeln(static ctor); }
static ~this() { writeln(static dtor); }
this(string x) { writeln(ctor); this.x = x; }
~this() { writeln(dtor); writeln(x); }
}
--
# g++ main.cpp -c
# dmd -c dmain.d
# dmd -c test.d -fPIC
# gcc -shared test.o -o libtest.so
# gcc dmain.o main.o -ltest -lphobos2 -lpthread -lrt -L.
-Wl,-rpath=.
# ./a.out
CPP ctor
main module ctor
lib module ctor
static ctor
C main function reached
ctor
in foo
static dtor
lib module dtor
main module dtor
dtor
data
# ldd ./a.out
linux-vdso.so.1 (0x7fff4cd55000)
libtest.so = ./libtest.so (0x7f970520b000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x7f9704fd3000)
librt.so.1 = /lib64/librt.so.1 (0x7f9704dcb000)
libc.so.6 = /lib64/libc.so.6 (0x7f9704a26000)
/lib64/ld-linux-x86-64.so.2 (0x7f970540e000)
#
Re: dynamic library building and loading
On 9/27/12 2:26 AM, Daniel Kozak wrote: Now I try it, and it is not required to build shared variant of druntime and phobos, only rebuild it with -fPIC Could you please send a troika composed of one dynlib, one loader using it, and a makefile that puts the all together? Thanks much! Andrei
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 08:03:34 UTC, Maxim Fomin wrote: On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller wrote: Maxim Fomin wrote: You can build shared libraries on linux by manually compiling object files and linking them. On windows last time I tries it was not possible. Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens Dpaste seems not working, so, sorry for code [snip] Hmm, that looks nice, seems that stuff improved from last time I checked. Gonna test this out today. I have project that would gain from Hot swapping of Shared libraries. Temporarily solution I made was just writing plugins in C++. Btw, sorry for OT. What exactly doesn't work in Dpaste? It seems to work fine for me(TM). If those are UI glitches, try pressing F5 2-3 times ;D
Re: dynamic library building and loading
On 2012-09-27 12:02, Daniel Kozak wrote: With DMD 2.060 last two points are possible, only first point (Using a shared ...) I dont try I really need to try this when I get home. -- /Jacob Carlborg
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 13:00:24 UTC, nazriel wrote: Btw, sorry for OT. What exactly doesn't work in Dpaste? It seems to work fine for me(TM). If those are UI glitches, try pressing F5 2-3 times ;D Dpaste was completely down when I tried to post code. It was in the middle of today.
Re: dynamic library building and loading
On 2012-09-27 10:55, Maxim Fomin wrote: On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg wrote: 1. Does this actually run? If it were non-runnable, I wouldn't posted it. 2. This is statically linked with druntime and Phobos. What happens when you create an executable that links with the D dynamic library? Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first. Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up A quick look in rt.minfo in druntime shows the it uses a single array or linked list for all module infos. It expect the module infos to be in just one place, i.e. in the application. There's no handling when a dynamic library is loaded. Have a look at this: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184 That symbol points to the start of the linked list containing module infos. That's only a single linked list, no handling of module infos from multiple places. rt.minfo needs to be changed to use an associative array with the keys pointing to images (executables and loaded dynamic libraries) and the values mapped to module infos of the given image. It also needs to intercept when a library is dynamically loaded, extract the module infos and register it with the runtime. I would think that the same is true for exception handling tables, TLS and GC sections. -- /Jacob Carlborg
Re: dynamic library building and loading
On 9/27/12 6:26 AM, Maxim Fomin wrote:
[snip]
Thanks! I adapted your code as follows and it works with 2.058 on Centos.
*** lib.d
import std.stdio;
extern(C) int fun()
{
writeln(, world!);
return 42;
}
*** main.d
import std.stdio;
extern(C) int fun();
void main()
{
write(Hello);
fun();
}
*** Makefile
PHOBOS_PATH=/path/to/phobos/
main: main.o liblib.so
gcc main.o -llib -lphobos2 -lpthread -lrt -L. -L${PHOBOS_PATH}
-Wl,-rpath=. -o main
liblib.so: lib.d
dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so
%.o: %.d
dmd -c $
clean:
rm -f main main.o lib.o liblib.so
Building the makefile and running ./main prints the classic message.
Thanks all for answering, this will get us going.
Andrei
Re: dynamic library building and loading
On 2012-09-27 20:25, Andrei Alexandrescu wrote: On 9/27/12 6:26 AM, Maxim Fomin wrote: [snip] Thanks! I adapted your code as follows and it works with 2.058 on Centos. I seriously doubt that everything is working properly, have a look at my reply to Maxim Fomin: http://forum.dlang.org/thread/k3vfm9$1tq$1...@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg wrote: On 2012-09-27 10:55, Maxim Fomin wrote: On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg wrote: 1. Does this actually run? If it were non-runnable, I wouldn't posted it. 2. This is statically linked with druntime and Phobos. What happens when you create an executable that links with the D dynamic library? Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first. Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up A quick look in rt.minfo in druntime shows the it uses a single array or linked list for all module infos. It expect the module infos to be in just one place, i.e. in the application. There's no handling when a dynamic library is loaded. Have a look at this: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184 That symbol points to the start of the linked list containing module infos. That's only a single linked list, no handling of module infos from multiple places. rt.minfo needs to be changed to use an associative array with the keys pointing to images (executables and loaded dynamic libraries) and the values mapped to module infos of the given image. It also needs to intercept when a library is dynamically loaded, extract the module infos and register it with the runtime. I would think that the same is true for exception handling tables, TLS and GC sections. Posted code doesn't load libraries at runtime, it is just linked to shared libraries.
Re: dynamic library building and loading
On 9/27/12 3:37 PM, Maxim Fomin wrote: Posted code doesn't load libraries at runtime, it is just linked to shared libraries. Exactly! (I can't believe I'm starting to get the hang of this...) But what we ultimately need is true dynamic loading of never-seen modules. After the initial test I tried to load symbols with dlopen(). Indeed this works: auto p = dlopen(liblib.so, RTLD_LAZY); And it does work, all fine. To my dismay, as soon as I removed the flag -llib from the linker command line in the makefile, dlopen() didn't want to work anymore. So I think in order to enable true dynamic loading, I'll need to generate PIC for druntime and phobos, and then link liblib.so like this: dmd -fPIC -c lib.d gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2 Is that correct? At that point, of course, a variety of issues will need to be resolved as people pointed out. Andrei
Re: dynamic library building and loading
For me to get C or C++ to run a D function, I had to do the
following:
//
// C/C++ library source
//
// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization shutdown
void init();
void done();
void bar()
{
foo(6, 7, 8);
}
int main(int argc, char **argv)
{
init();
bar();
done();
return 0;
}
//
// D library source
//
import std.stdio;
static import core.runtime;
// sample D function for test
extern (C++) // int foo(int i, int j, int k)
void foo(int i, int j, int k)
{
writefln(i = %s, i);
writefln(j = %s, j);
writefln(k = %s, k);
int t = i + j + k;
writefln(Total = %s, t);
}
// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after
loading shared lib
core.runtime.Runtime.initialize();
}
extern (C++) export void done() { // to be called before
unloading shared lib
core.runtime.Runtime.terminate();
}
// I had to include main even though this is a library.
int main()
{
return 0;
}
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 07:54:29 UTC, Johannes Pfau wrote: Am Thu, 27 Sep 2012 08:26:36 +0200 schrieb Daniel Kozak kozz...@gmail.com: Now I try it, and it is not required to build shared variant of druntime and phobos, only rebuild it with -fPIC In the end you'll probably need a shared druntime phobos: Let's say your main app doesn't use std.datetime and you statically link libphobos. Then the linker might strip std.datetime from your executable. If your shared library now needs std.datetime it won't work. I'm using C++ main app that dynamically loads shared libs at runtime, but I cannot load D shared libs because they cannot be linked to the non-fPIC built phobos2/druntime lib. It seems that the only way to get things working is to re-build the C++ code so that it is linked with non-fPIC phobos2, but this is not something I want to be doing. So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number? --rt
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 19:57:13 UTC, Andrei Alexandrescu wrote: So I think in order to enable true dynamic loading, I'll need to generate PIC for druntime and phobos, and then link liblib.so like this: dmd -fPIC -c lib.d gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2 Is that correct? Yes, that is correct. The other thing missing is a phobos2.so for dynamic runtime linking. -rt
dynamic library building and loading
Haven't done any dynamic linking with D and I need to. I'm using dmd
2.058/Linux at work to build and use dynamic libraries. Here's my attempt:
*** Makefile
all: main lib.so
main: main.d
dmd main
lib.so: lib.d
dmd -fPIC -shared lib.d -of./lib.so
*** lib.d
extern(C) int fun(string s)
{
return 42;
}
*** main.d
import std.stdio;
void main()
{
}
Running make prints:
dmd -fPIC -shared lib.d -of./lib.so
/usr/bin/ld:
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o):
relocation R_X86_64_32 against `a local symbol' can not be used when
making a shared object; recompile with -fPIC
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a:
could not read symbols: Bad value
collect2: ld returned 1 exit status
What steps do I need to take to get off the ground?
Thanks,
Andrei
[OT: Windows note] Re: dynamic library building and loading
26.09.2012 21:58, Andrei Alexandrescu пишет: Haven't done any dynamic linking with D and I need to. I'm using dmd 2.058/Linux at work to build and use dynamic libraries. Sorry for the OT (as you are a Linux user), but if you will ever do it on Windows, be aware of this nasty trap: http://d.puremagic.com/issues/show_bug.cgi?id=8130 -- Денис В. Шеломовский Denis V. Shelomovskij
Re: dynamic library building and loading
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei Alexandrescu wrote: Haven't done any dynamic linking with D and I need to. I'm using dmd 2.058/Linux at work to build and use dynamic libraries. Here's my attempt: Maybe it will help you D: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d Fortran: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95 Dmd 2.060 used. DLL on OpenSUSE 64bit compiled with command: gfortran -shared -fPIC -O3 -o simple.so simple.f95
Re: dynamic library building and loading
On 2012-09-26 19:58, Andrei Alexandrescu wrote: Haven't done any dynamic linking with D and I need to. I'm using dmd 2.058/Linux at work to build and use dynamic libraries. Here's my attempt: Running make prints: dmd -fPIC -shared lib.d -of./lib.so /usr/bin/ld: /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a: could not read symbols: Bad value collect2: ld returned 1 exit status What steps do I need to take to get off the ground? You need to properly implement support for dynamic libraries in druntime and DMD. I'm not entirely sure if there are some problems left in DMD, but druntime is definitely not ready yet. In general what needs to be done is to properly handle: * Module infos * TLS * Exception handling tables Martin Nowak has a branch for this: https://github.com/dawgfoto/druntime/tree/SharedRuntime -- /Jacob Carlborg
Re: [OT: Windows note] Re: dynamic library building and loading
On 9/26/12 2:25 PM, Denis Shelomovskij wrote: 26.09.2012 21:58, Andrei Alexandrescu пишет: Haven't done any dynamic linking with D and I need to. I'm using dmd 2.058/Linux at work to build and use dynamic libraries. Sorry for the OT (as you are a Linux user), but if you will ever do it on Windows, be aware of this nasty trap: http://d.puremagic.com/issues/show_bug.cgi?id=8130 Thanks! Andrei
Re: dynamic library building and loading
On 9/26/12 2:59 PM, Michael wrote: On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei Alexandrescu wrote: Haven't done any dynamic linking with D and I need to. I'm using dmd 2.058/Linux at work to build and use dynamic libraries. Here's my attempt: Maybe it will help you D: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d Fortran: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95 Dmd 2.060 used. DLL on OpenSUSE 64bit compiled with command: gfortran -shared -fPIC -O3 -o simple.so simple.f95 Thanks. The loading part is very useful, but I'm still lost when it comes to build the shared library itself. Andrei
Re: dynamic library building and loading
On 9/26/12 3:18 PM, Jacob Carlborg wrote: Martin Nowak has a branch for this: https://github.com/dawgfoto/druntime/tree/SharedRuntime Thanks, I'll follow up with him. Andrei
Re: dynamic library building and loading
Thanks. The loading part is very useful, but I'm still lost when it comes to build the shared library itself. Andrei Program loads dll at runtime using loader which is configured to load concrete dll file(s). Like in gtkD http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d
Re: dynamic library building and loading
On Wednesday, 26 September 2012 at 20:10:47 UTC, Michael wrote: Thanks. The loading part is very useful, but I'm still lost when it comes to build the shared library itself. Andrei Program loads dll at runtime using loader which is configured to load concrete dll file(s). Like in gtkD http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d Loading Shared lib isn't big issues here. The bigger one is building Shared library (written in D) and running it in host application without issues (EH, shared GC etc). Andrei, if you find out how to make those things work, please share your findings. I'm also in need of using shared libraries. And yeah, probably Martin Nowak will be the best bet to get information from.
Re: dynamic library building and loading
Loading Shared lib isn't big issues here. The bigger one is building Shared library (written in D) and running it in host application without issues (EH, shared GC etc). Andrei, if you find out how to make those things work, please share your findings. I'm also in need of using shared libraries. And yeah, probably Martin Nowak will be the best bet to get information from. I understand. As I remember on forum were topic with same approach for dll written in D. But I can't find a link.
Re: dynamic library building and loading
You can build shared libraries on linux by manually compiling object files and linking them. On windows last time I tries it was not possible.
Re: dynamic library building and loading
Maxim Fomin wrote: You can build shared libraries on linux by manually compiling object files and linking them. On windows last time I tries it was not possible. Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens
