[Issue 15566] New: [Writing Shared Libraries With D On Linux]
https://issues.dlang.org/show_bug.cgi?id=15566 Issue ID: 15566 Summary: [Writing Shared Libraries With D On Linux] Product: D Version: D2 Hardware: All URL: http://dlang.org/ OS: All Status: NEW Severity: normal Priority: P3 Component: dlang.org Assignee: nob...@puremagic.com Reporter: john.loughran.col...@gmail.com It's super confusing using "DLL" (somewhat inconsistently as well...) here, AFAIK it's a windows-only abbreviation. --
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
I'm waiting for that too! On Wed, May 29, 2013 at 5:47 PM, Ettienne Gilbert ettienne.gilb...@gmail.com wrote: Hi, What happened with the next video/slide release? I was under the impression that there was agreement that the releases will be done every 3 days or so. So far: Opening Keynote by Walter Bright: video and slides available: May 08 Day 1 Talk 2: Copy and Move Semantics in D by Ali Cehreli: May 10 Day 1 Talk 3: Distributed Caching Compiler for D by Robert Schadek: May 13 Day 1 Talk 4: Writing Testable Code in D by Ben Gertzfield: May 15 Day 1 Talk 5: Using D Alongside a Game Engine by Manu Evans: May 17 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella: May 20 Day 1 Talk 7: Panel with Walter Bright and Andrei Alexandrescu: May 23 Day 2 Talk 1: GDC by Iain Buclaw: May 24 So we have been waiting with bated breath and counting the days for the last 5 days! For those of us on the far side of the world that could not attend (I'm in South Africa) we are really looking forward to this. Of course, if work (or personal) pressures do not allow for this at the moment, that will be perfectly understandable. But some indication on the forum then (even if May XX +/-yy days) will be really nice! Rgds
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
On Wednesday, 29 May 2013 at 12:17:59 UTC, Ettienne Gilbert wrote: Hi, What happened with the next video/slide release? I was under the impression that there was agreement that the releases will be done every 3 days or so. So far: Opening Keynote by Walter Bright: video and slides available: May 08 Day 1 Talk 2: Copy and Move Semantics in D by Ali Cehreli: May 10 Day 1 Talk 3: Distributed Caching Compiler for D by Robert Schadek: May 13 Day 1 Talk 4: Writing Testable Code in D by Ben Gertzfield: May 15 Day 1 Talk 5: Using D Alongside a Game Engine by Manu Evans: May 17 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella: May 20 Day 1 Talk 7: Panel with Walter Bright and Andrei Alexandrescu: May 23 Day 2 Talk 1: GDC by Iain Buclaw: May 24 So we have been waiting with bated breath and counting the days for the last 5 days! For those of us on the far side of the world that could not attend (I'm in South Africa) we are really looking forward to this. Of course, if work (or personal) pressures do not allow for this at the moment, that will be perfectly understandable. But some indication on the forum then (even if May XX +/-yy days) will be really nice! Rgds 27th of may was a national holiday in the USA. I believe the next talk will be published later today (29th GMT). If Andrei times it similar to previously, it'll be in the next few hours.
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
I'd understand postponing a Game of trones episode, but this?! J/K great conference, take your time with uploading.
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
On 5/29/13 9:42 AM, John Colvin wrote: 27th of may was a national holiday in the USA. I believe the next talk will be published later today (29th GMT). If Andrei times it similar to previously, it'll be in the next few hours. Indeed that is the reason. Andrei
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
Indeed that is the reason. Andrei Perfectly understandable then. Shared libraries is actually a big issue for me, so I am looking forward to this one. So yeah, let the music begin! BTW, thanks for all the hard work in getting it all out! Rgds
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
On Wednesday, 29 May 2013 at 15:33:36 UTC, Ettienne Gilbert wrote: BTW, thanks for all the hard work in getting it all out! Rgds +1
Re: DConf 2013 Day 2 Talk 2: Shared Libraries in D by Martin Nowak
With regard to the last point in the talk where Walter was suggesting not calling finalizers on objects whose code has been unloaded - would it not make more sense to simply call all the finalizers before unloading the library? If the finalizer is not called you will potentially get resource leaks - although there's no guarantee currently that it will be called, isn't there the assumption that if it's not called the object still exists, even if it's not referenced anywhere? ie. you still have some guarantee that if you keep allocating objects and then unreferencing them that the number of unfinalized objects will never exceed some large but fixed value.
Re: DConf 2013 Day 2 Talk 2: Shared Libraries in D by Martin Nowak
On 5/29/13 10:05 PM, Meta wrote: On Wednesday, 29 May 2013 at 14:44:38 UTC, Andrei Alexandrescu wrote: http://www.reddit.com/r/programming/comments/1f9qq3/dconf_2013_day_2_talk_2_shared_libraries_in_d_by/ Apologies for the delay. Enjoy and vote up! Andrei The first couple talks were posted to Hacker News as well. Have you stopped posting these there now? Walter used to do that. I'll do it from now on. https://news.ycombinator.com/item?id=5790649 Andrei
Re: DConf 2013 Day 2 Talk 2: Shared Libraries in D by Martin Nowak
On Wed, 29 May 2013 22:12:54 -0400, Diggory digg...@googlemail.com wrote: With regard to the last point in the talk where Walter was suggesting not calling finalizers on objects whose code has been unloaded - would it not make more sense to simply call all the finalizers before unloading the library? In fact, I think in discussions after the talk (not recorded on video), we came to the same conclusion. If you are unloading the library and have any pointers to classes from that library, that is a programming error. So it should be safe to destroy all known objects from that library. -Steve
Re: DConf 2013 Day 2 Talk 2: Shared Libraries in D by Martin Nowak
On Thursday, 30 May 2013 at 04:00:18 UTC, Steven Schveighoffer wrote: On Wed, 29 May 2013 22:12:54 -0400, Diggory digg...@googlemail.com wrote: With regard to the last point in the talk where Walter was suggesting not calling finalizers on objects whose code has been unloaded - would it not make more sense to simply call all the finalizers before unloading the library? In fact, I think in discussions after the talk (not recorded on video), we came to the same conclusion. If you are unloading the library and have any pointers to classes from that library, that is a programming error. So it should be safe to destroy all known objects from that library. -Steve Ah, good to know :)
Re: shared libraries in D
== Quote from Iain Buclaw (ibuc...@ubuntu.com)'s article == Quote from Christian Kamm (kamm-incasoftw...@removethis.de)'s article Iain Buclaw wrote: == Quote from Christian Kamm (kamm-incasoftw...@removethis.de)'s article Iain Buclaw wrote: Will be making shared libraries default in GDC pretty soon now... Did you adjust the GC to check the shared libraries' data sections for references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian When adding more ranges in the GC to check, slowdowns are inevitable. So far in my personal testing, the slowdowns seem pretty negligible in comparison (never more than 0.300 seconds, though that could be a sign that I'm not pushing it in the right way). To prevent the GC from scanning C libraries, I've added an extra check to only add ranges that have a D module inside them. That sounds good. I was asking because I didn't see any code like this in http://bitbucket.org/goshawk/gdc - which repository is this in? Christian It's currently on my local workstation, all mashed together with the D2 beta merge. I have faith that my workstation won't crash and die in the meantime touchwood and will sift through it all (and break it down into sizeable commits) once I'm satisfied that there's nothing more I can do. Regards Iain Module/Routine is now up here: https://bitbucket.org/goshawk/gdc/src/a402fef51a6a/d/druntime/rt/gccmemory.d
Re: shared libraries in D
== Quote from Christian Kamm (kamm-incasoftw...@removethis.de)'s article Iain Buclaw wrote: == Quote from Christian Kamm (kamm-incasoftw...@removethis.de)'s article Iain Buclaw wrote: Will be making shared libraries default in GDC pretty soon now... Did you adjust the GC to check the shared libraries' data sections for references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian When adding more ranges in the GC to check, slowdowns are inevitable. So far in my personal testing, the slowdowns seem pretty negligible in comparison (never more than 0.300 seconds, though that could be a sign that I'm not pushing it in the right way). To prevent the GC from scanning C libraries, I've added an extra check to only add ranges that have a D module inside them. That sounds good. I was asking because I didn't see any code like this in http://bitbucket.org/goshawk/gdc - which repository is this in? Christian It's currently on my local workstation, all mashed together with the D2 beta merge. I have faith that my workstation won't crash and die in the meantime touchwood and will sift through it all (and break it down into sizeable commits) once I'm satisfied that there's nothing more I can do. Regards Iain
Re: shared libraries in D
Iain Buclaw wrote: Came across this obscure documentation in the tldp. Libraries should export initialization and cleanup routines using the gcc __attribute__((constructor)) and __attribute__((destructor)) function attributes. This is what gdc was doing anyway. Constructor routines are executed before dlopen returns (or before main() is started if the library is loaded at load time). Destructor routines are executed before dlclose returns (or after exit() or completion of main() if the library is loaded at load time). This is what should have been happening, but wasn't earlier because: Shared libraries must not be compiled with the gcc arguments `-nostartfiles' or `-nostdlib'. If those arguments are used, the constructor/destructor routines will not be executed (unless special measures are taken). Whoops! Fixed and it now works. :~) Will be making shared libraries default in GDC pretty soon now... Great, now I've got 2 questions: 1.) Are shared libraries compiled with gdc abi compitible with dmd? (if dmd did support shared libraries) 2.) Is it now possible to load a shared d library into a C program? Will jut loading the library initialize the runtime or is it necessary to call some runtime setup functions manually? If the C program loads 2 D libraries, will those share the gc/runtime? -- Johannes Pfau signature.asc Description: PGP signature
Re: shared libraries in D
On 2011-02-15 16:02, Johannes Pfau wrote: Iain Buclaw wrote: Came across this obscure documentation in the tldp. Libraries should export initialization and cleanup routines using the gcc __attribute__((constructor)) and __attribute__((destructor)) function attributes. This is what gdc was doing anyway. Constructor routines are executed before dlopen returns (or before main() is started if the library is loaded at load time). Destructor routines are executed before dlclose returns (or after exit() or completion of main() if the library is loaded at load time). This is what should have been happening, but wasn't earlier because: Shared libraries must not be compiled with the gcc arguments `-nostartfiles' or `-nostdlib'. If those arguments are used, the constructor/destructor routines will not be executed (unless special measures are taken). Whoops! Fixed and it now works. :~) Will be making shared libraries default in GDC pretty soon now... Great, now I've got 2 questions: 1.) Are shared libraries compiled with gdc abi compitible with dmd? (if dmd did support shared libraries) I'm not completely sure but the runtimes are different. 2.) Is it now possible to load a shared d library into a C program? Will jut loading the library initialize the runtime or is it necessary to call some runtime setup functions manually? If the C program loads 2 D libraries, will those share the gc/runtime? I don't know how Iain has implemented this but it should be easy to do. Just add, to the runtime, a C function with __attribute__((constructor)) that initializes the runtime. -- /Jacob Carlborg
Re: shared libraries in D
== Quote from Christian Kamm (kamm-incasoftw...@removethis.de)'s article
Iain Buclaw wrote:
Will be making shared libraries default in GDC pretty soon now...
Did you adjust the GC to check the shared libraries' data sections for
references? When we looked at this for LDC that turned out to slow down GC
runs significantly.
I'm pretty sure bearophile benchmarked it at the time.
As far as I remember the main problem was that we were scanning all data
sections - even of plain C libraries.
Regards,
Christian
When adding more ranges in the GC to check, slowdowns are inevitable. So far in
my
personal testing, the slowdowns seem pretty negligible in comparison (never more
than 0.300 seconds, though that could be a sign that I'm not pushing it in the
right way).
To prevent the GC from scanning C libraries, I've added an extra check to only
add
ranges that have a D module inside them.
Think:
ModuleReference *mr;
for ( mr = _Dmodule_ref; mr; mr = mr.next )
{
if ( mr = start mr = end )
{
addrange = 1;
break;
}
}
This is viable for GDC, as all platforms share the same implementation. Note,
this
also adds a variable overhead at startup, but I think the unskewed runtime
justifies it to remain.
Regards
shared libraries in D
So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment). Just for
sake of visual proof.
[iain@natty gdc]$ cat hello.d
import std.stdio;
void main()
{
writeln(Hello World);
}
[iain@natty gdc]$ gdc hello.d -lrt -ldl # Never needed these before...
[iain@natty gdc]$ du -hs a.out
8.0Ka.out
[iain@natty gdc]$ ldd a.out
linux-gate.so.1 = (0x00a32000)
librt.so.1 = /lib/librt.so.1 (0x00c32000)
libdl.so.2 = /lib/libdl.so.2 (0x00914000)
libgphobos2 = /usr/local/lib/libgphobos2 (0x0050d000)
libgdruntime = /usr/local/lib/libgdruntime (0x00a9)
libm.so.6 = /lib/libm.so.6 (0x00921000)
libgcc_s.so.1 = /lib/libgcc_s.so.1 (0x00e8f000)
libpthread.so.0 = /lib/libpthread.so.0 (0x007b9000)
libc.so.6 = /lib/libc.so.6 (0x0011)
/lib/ld-linux.so.2 (0x00f07000)
Now, what's the catch?
The curent catch is that *none* of the external static ctors/dtors are called,
because the linked module list (_Dmodule_ref) is generated before the program
enters C main, thus before the shared druntime and phobos libraries have been
loaded into memory (don't know if this is the same for DMD, but I'll assume
that will likely be the case too).
Breakpoint 1, hello.__modinit() () at hello.d:1
(gdb) bt
#0 hello.__modinit() () at hello.d:1
#1 0x0804889d in __do_global_ctors_aux ()
#2 0x080485d0 in _init ()
#3 0x08048829 in __libc_csu_init ()
#4 0x003a8c84 in __libc_start_main () from /lib/libc.so.6
#5 0x08048661 in _start ()
Each modules own __modinit() is written out as such:
struct _modref_t {
_modref_t * next;
ModuleInfo m;
}
extern (C) _modref_t * _Dmodule_ref;
private _modref_t our_mod_ref =
{ next: null, m: _ModuleInfo_xxx };
void ___modinit() { // a static constructor
our_mod_ref.next = _Dmodule_ref;
_Dmodule_ref = our_mod_ref;
}
These functions are created by the compiler, and inserted into the .ctor list.
Before I progress, posting to ask if anyone has any good implementation ideas
to get this fully functional?
Regards
Iain
Re: shared libraries in D
Iain Buclaw wrote: So I've been prototyping, and have built a fully working shared D2 druntime/phobos library on Linux (will come to caveats in a moment). Just for sake of visual proof. Awesome! Before I progress, posting to ask if anyone has any good implementation ideas to get this fully functional? Regards Iain I think dmd/druntime support shared libraries on MacOS. I'm not sure though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) -- Johannes Pfau signature.asc Description: PGP signature
Re: shared libraries in D
== Quote from Johannes Pfau (s...@example.com)'s article --Sig_/a6ST_/ke_QlFF5lFi4BGNvw Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Iain Buclaw wrote: So I've been prototyping, and have built a fully working shared D2 druntime/phobos library on Linux (will come to caveats in a moment). Just for sake of visual proof. Awesome! Before I progress, posting to ask if anyone has any good implementation ideas to get this fully functional? Regards Iain I think dmd/druntime support shared libraries on MacOS. I'm not sure though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau The support for OSX uses an entirely different, and questionable implementation. Regards
Re: shared libraries in D
== Quote from Iain Buclaw (ibuc...@ubuntu.com)'s article == Quote from Johannes Pfau (s...@example.com)'s article --Sig_/a6ST_/ke_QlFF5lFi4BGNvw Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Iain Buclaw wrote: So I've been prototyping, and have built a fully working shared D2 druntime/phobos library on Linux (will come to caveats in a moment). Just for sake of visual proof. Awesome! Before I progress, posting to ask if anyone has any good implementation ideas to get this fully functional? Regards Iain I think dmd/druntime support shared libraries on MacOS. I'm not sure though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau The support for OSX uses an entirely different, and questionable implementation. Regards Came across this obscure documentation in the tldp. Libraries should export initialization and cleanup routines using the gcc __attribute__((constructor)) and __attribute__((destructor)) function attributes. This is what gdc was doing anyway. Constructor routines are executed before dlopen returns (or before main() is started if the library is loaded at load time). Destructor routines are executed before dlclose returns (or after exit() or completion of main() if the library is loaded at load time). This is what should have been happening, but wasn't earlier because: Shared libraries must not be compiled with the gcc arguments `-nostartfiles' or `-nostdlib'. If those arguments are used, the constructor/destructor routines will not be executed (unless special measures are taken). Whoops! Fixed and it now works. :~) Will be making shared libraries default in GDC pretty soon now...
Re: shared libraries in D
On 2011-02-14 16:29, Iain Buclaw wrote: == Quote from Johannes Pfau (s...@example.com)'s article --Sig_/a6ST_/ke_QlFF5lFi4BGNvw Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Iain Buclaw wrote: So I've been prototyping, and have built a fully working shared D2 druntime/phobos library on Linux (will come to caveats in a moment). Just for sake of visual proof. Awesome! Before I progress, posting to ask if anyone has any good implementation ideas to get this fully functional? Regards Iain I think dmd/druntime support shared libraries on MacOS. I'm not sure though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau The support for OSX uses an entirely different, and questionable implementation. Regards If you know any other way of implementing it I very open for suggestions. -- /Jacob Carlborg
Re: shared libraries in D
Iain Buclaw wrote: Will be making shared libraries default in GDC pretty soon now... Did you adjust the GC to check the shared libraries' data sections for references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian
