On Thursday, 30 September 2021 at 18:09:46 UTC, Hipreme wrote:
I write this post as both a learning tool, a question and an
inquiry.
There are just a lot of drawbacks in trying to do function
exporting while using D.
The terms that people use are a bit sloppy. There are three kinds
of 'linking' here:
1. static linking, performed during compilation, once. If linking
fails, the compile files.
2. dynamic linking (option 1), performed when an executable
starts up, before your program gains control, by the system
linker. If linking fails, your program never gets control.
3. dynamic linking (option 2), performed arbitrarily at runtime,
by your program. If linking fails, you can do whatever you want
about that.
All of the loadSymbol and 'userdata module' hassle that you're
frustrated by is from option 2. Option 1 is really the normal way
to link large shared libraries and there's nothing to it. What
your code looks like that loads a shared library is just `import
biglib;`, and the rest of the work is in dub, pkg-config,
`LD_LIBRARY_PATH`, etc. Phobos is commonly linked in this way.
Pretty much anything that isn't a plugin in a plugin directory
can use option 1 instead of option 2.
extern(C) advantages:
- Code callable from any language as it is absolutely intuitive
- Well documented
You can call scalding water 'hot' even when you're fresh from
observing a lava flow. People still find the C ABI frustrating in
a lot of ways, and especially when they encounter it for the
first time.
But the C ABI rules the world right now, yes. The real advantages
are
- it 'never' changes
- 'everyone' already makes it easy to use
extern(C) disadvantages:
- You will need to declare your function pointer as extern(C)
or it will swap the arguments order.
- you're limited to using C's types
- you can't use overloading, lazy parameters, default values; you
can't rely on scope parameters, etc., etc.
- you can't casually hand over GC-allocated data and expect the
other side to handle it right, or structs with lifetime functions
that you expect to be called
- very little of importance is statically checked: to use a C ABI
right you need to very carefully read documentation that needs to
exist to even know who is expected to clean up a pointer and how,
how large buffers should be. (I wasn't feeling a lot of the C
ABI's "absolute intuitiveness" when I was passing libpcre an
ovector sized to the number of pairs I wanted back rather than
the correct number of `pairs*3/2`)
Option 2 dynamic linking of D libraries sounds pretty
frustrating. Even with a plugin architecture, maybe I'd prefer
just recompiling the application each time the plugins change to
retain option 1 dynamic linking. Using a C ABI instead is a good
idea if just to play nice with other languages.
And if you were wanting something like untrusted plugins, a way
to respond to a segfault in a plugin, like I think you mentioned
in Discord, then I'd still suggest not linking at all but having
separate applications and some form of interprocess communication
(pipes, unix sockets, TCP sockets) instead of function calls.
This is something that you could design, or with D's reflection,
generate code for against the function calls you already have.
But this is even more work that you'll have to do. If we add "a
separate process telling you what to do with some kind of
protocol" as a fourth kind of linking, then the respective effort
is
1. free! it compiles, it's probably good!
2. free! if the program starts, it's probably good!
3. wow, why don't you just write your own loadSymbol DSL?
4. wow, why don't you just reimplement Erlang/OTP and call it
std.distributed? maybe protobufs will be enough.
