Re: DIP45: fixing the dllimport/dllexport issue
On 09/23/2013 08:30 AM, Rainer Schuetze wrote: I just checked the OMF and COFF docs: it should be possible to wipe out the export records without having to rewrite the object files, so it's not too involved. Don't know about ELF or mach-o, though. I tried this and implemented it by rewriting the ELF sym flag for export, so it's was done by a rewrite of the object file. Maybe there are other solutions, but this seemed to be the simplest approach.
Re: DIP45: fixing the dllimport/dllexport issue
On 09/19/2013 01:55 PM, Benjamin Thaut wrote: B: You shouldn't mix static with dynamic linking, ODR issues are what you get in return. Well you should not. But it will happen. Espeically if you don't have any control over what others due. E.g. when using third party libraries. Let me put it differently, if you statically link against a library in your DLL you need to ensure not to leak (export) any symbols from this static library.
Re: DIP45: fixing the dllimport/dllexport issue
On 09/08/2013 09:33 AM, Benjamin Thaut wrote: There is yet another problem to be solved. What should happen with tempaltes? Does it make sense to export templates? Yes, we need to export/import symbols from templates. Recently the compiler was changed to skip instantiations when they have already been instantiated in imported modules. There is a problem similar to the lib case though. When you instantiate an exported template you don't want it's symbols exported from your shared library. What happens if a template class marked with export gets instanciated in different DLLs using the same template arguments? Should the TypeInfo objects match? Should templates be exportable in generall? Yes TypeInfos should match. For symbols which are allowed to have multiple definitions (like typeinfos from template classes) we can't solely rely on pointer comparison but also need to test typename equality.
Re: DIP45: fixing the dllimport/dllexport issue
On 19.09.2013 21:42, Benjamin Thaut wrote: Am 19.09.2013 20:24, schrieb Rainer Schuetze: On 19.09.2013 13:58, Benjamin Thaut wrote: Am 12.09.2013 07:54, schrieb Rainer Schuetze: This sounds interesting. Stripping an existing library isn't even needed because it is normally never created with exports anyway (-lib implies that no exports are created to start with). Stripping object files might be necessary, though, in case they have been built separately with -c. Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library? That could work, though it has it's own set of side effects, like splitting a module into pseudo modules per function which also ruins running unittests built into the library. Building modules with -c and combining these into a library afterwards avoids this. Ah ok, I didn't know that. How much work would it be to implement object file stripping? I'm not confident I can do this. I just checked the OMF and COFF docs: it should be possible to wipe out the export records without having to rewrite the object files, so it's not too involved. Don't know about ELF or mach-o, though.
Re: DIP45: fixing the dllimport/dllexport issue
On 23.09.2013 07:22, Benjamin Thaut wrote: Am 10.09.2013 05:50, schrieb Martin Nowak: Can we deal with vtable entries pointing to imported functions on Windows? The kind of thing that happens when you inherit across a DLL boundary. I can't answer that and with dmd's current implementation of export its not possible to try this. I would say we implement the DIP and see on the way if this becomes a issue. It is the same as initializing a global variable with a pointer into another DLL, i.e. it needs some init code to set the value.
Re: DIP45: fixing the dllimport/dllexport issue
Am 10.09.2013 05:50, schrieb Martin Nowak: Can we deal with vtable entries pointing to imported functions on Windows? The kind of thing that happens when you inherit across a DLL boundary. I can't answer that and with dmd's current implementation of export its not possible to try this. I would say we implement the DIP and see on the way if this becomes a issue. -- Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
Am 19.09.2013 20:24, schrieb Rainer Schuetze: On 19.09.2013 13:58, Benjamin Thaut wrote: Am 12.09.2013 07:54, schrieb Rainer Schuetze: This sounds interesting. Stripping an existing library isn't even needed because it is normally never created with exports anyway (-lib implies that no exports are created to start with). Stripping object files might be necessary, though, in case they have been built separately with -c. Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library? That could work, though it has it's own set of side effects, like splitting a module into pseudo modules per function which also ruins running unittests built into the library. Building modules with -c and combining these into a library afterwards avoids this. Ah ok, I didn't know that. How much work would it be to implement object file stripping? I'm not confident I can do this. Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On 19.09.2013 13:58, Benjamin Thaut wrote: Am 12.09.2013 07:54, schrieb Rainer Schuetze: This sounds interesting. Stripping an existing library isn't even needed because it is normally never created with exports anyway (-lib implies that no exports are created to start with). Stripping object files might be necessary, though, in case they have been built separately with -c. Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library? That could work, though it has it's own set of side effects, like splitting a module into pseudo modules per function which also ruins running unittests built into the library. Building modules with -c and combining these into a library afterwards avoids this.
Re: DIP45: fixing the dllimport/dllexport issue
Am 12.09.2013 07:54, schrieb Rainer Schuetze: This sounds interesting. Stripping an existing library isn't even needed because it is normally never created with exports anyway (-lib implies that no exports are created to start with). Stripping object files might be necessary, though, in case they have been built separately with -c. Can't we get around stripping object files by always using -c together with -lib in case the object file is supposed to go into a static library?
Re: DIP45: fixing the dllimport/dllexport issue
Am 10.09.2013 05:39, schrieb Martin Nowak: I also kind of feel that we need to export lib1 as well, as you may not know what lib2 does with what it gets from lib1 (passing objects around for instance). So certainly at least some part of lib1 need to be exported. For sure lib2 might forward to some lib1 symbols. Not sure what might accidentally require linkage. I don't agree if that statement. If you have three libs. lib1, lib2 and lib3. And lib2 and lib3 link statically against lib1 you are going to get linker errors because both lib2 and lib3 contain the symbols of lib1. A: Why do two DLLs which export the same symbols cause linker errors? Yes Because the corresponding import libraries contain the same symbols? B: You shouldn't mix static with dynamic linking, ODR issues are what you get in return. Well you should not. But it will happen. Espeically if you don't have any control over what others due. E.g. when using third party libraries. C: What would happen if you statically link against lib2 and lib3? Well you would get a linker error as well. But the reasoing behind this is as follows. You use a third party library that comes as a shared library. That third party library uses say libcurl internally. Now you use libcurl too. You link against libcurl. But because the third party library you link to also uses libcurl you will get linker errors. Also you might not want to use the same version of libcurl then the thirdparty library you use. This is going to become a problem especially in bigger projects, where you are not able to compile the third party libraries yourself because you don't have the source code. In such a case it can easily happen that you get a new version of that third party library, they added a new internal dependency to a library you already use and suddenly your executable won't linke anymore. Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On 12.09.2013 01:16, Martin Nowak wrote: On 09/11/2013 08:21 AM, Rainer Schuetze wrote: AFAIU the discussed scenario is that the phobos library exports a lot of symbols, and this library is used both for static and dynamic linking. An executable statically linked against this library will have all the exports. That still works but is ugly. Yes and it also has some runtime/memory overhead because the symbol table is much bigger. A DLL to be used as a plugin with a C interface might also want to link statically against phobos to avoid DLL hell. Consider multiple plugins built against different versions of phobos. These would also leak also the symbols. Right, that's the valid use-case. But here the additionally exported symbols are way less of a problem because your loading not linking. And as I wrote before if you link a program with two copies of the same library you're asking for trouble. Anyhow I realized there is even less a use-case for statically linking a lib into a DLL (shared library) while expecting the symbols to be exported. We use that to put druntime into libphobos2.so though. Also one doesn't need exported (visible) symbols for static linking. So how about going into the opposite direction? When the compiler creates a lib it strips all the exports (marks all symbols as hidden) by default. For the very special use-case where one wants to preserve the exports we can add a -libexports flag, only to be used with -lib. Another nice thing is, that dmd writes libs itself and already processes all object files so we could easily integrate the striping in the compiler. This sounds interesting. Stripping an existing library isn't even needed because it is normally never created with exports anyway (-lib implies that no exports are created to start with). Stripping object files might be necessary, though, in case they have been built separately with -c.
Re: DIP45: fixing the dllimport/dllexport issue
On 09/11/2013 08:21 AM, Rainer Schuetze wrote: AFAIU the discussed scenario is that the phobos library exports a lot of symbols, and this library is used both for static and dynamic linking. An executable statically linked against this library will have all the exports. That still works but is ugly. Yes and it also has some runtime/memory overhead because the symbol table is much bigger. A DLL to be used as a plugin with a C interface might also want to link statically against phobos to avoid DLL hell. Consider multiple plugins built against different versions of phobos. These would also leak also the symbols. Right, that's the valid use-case. But here the additionally exported symbols are way less of a problem because your loading not linking. And as I wrote before if you link a program with two copies of the same library you're asking for trouble. Anyhow I realized there is even less a use-case for statically linking a lib into a DLL (shared library) while expecting the symbols to be exported. We use that to put druntime into libphobos2.so though. Also one doesn't need exported (visible) symbols for static linking. So how about going into the opposite direction? When the compiler creates a lib it strips all the exports (marks all symbols as hidden) by default. For the very special use-case where one wants to preserve the exports we can add a -libexports flag, only to be used with -lib. Another nice thing is, that dmd writes libs itself and already processes all object files so we could easily integrate the striping in the compiler.
Re: DIP45: fixing the dllimport/dllexport issue
On Tuesday, 3 September 2013 at 07:02:33 UTC, Rainer Schuetze wrote: To initialize data with a pointer into another DLL, you cannot do that with a direct relocation, you have to run some code to copy the value from the import table into the data. Well, yes, static initialization is problematic, but you can have statically initialized pointer to an imported function, which is the same problem.
Re: DIP45: fixing the dllimport/dllexport issue
On 11.09.2013 00:39, Martin Nowak wrote: On 09/07/2013 02:47 PM, Benjamin Thaut wrote: If you compile lib1 into a static library and then copmpile lib2 into a DLL which statically links against lib1 both symbols lib1Func and lib2Func will be exported from the dll because lib1Func will get marked as dllexport when compiled into a static library and the static library transports that information into the linking process of lib2. This happens both on win32 and win64. But this is something you really don't want. Lets imagine you have a DLL that statically links against phobos. Suddenly you will have all phobos exported symbols in your dll. This can also lead to double defined symbols in case a user links against your dll and against the shared phobos dll. So we have to fix this somehow. I had another thought about this and tried some examples. If you do transitively link against lib1.lib symbols when using lib2.dll you might get ODR issues if there is another copy of lib1 in your process. This is to be expected because there are two copies of the same library. The safe choice is to make lib1 a DLL too. If you still want to statically link lib1 into lib2.dll you have to be very cautious. An actual problem only occurs when lib1 symbols get used from outside of lib2.dll. Of course a mean to prevent bugs from accidental linking lib1 symbols would be nice. One way of doing this would be to mark all lib1 symbols as PRIVATE in lib2.def. EXPORTS D4lib18lib1FuncFZi PRIVATE That's probably not feasible for a library like phobos. (At least optlink stumbles if the def file gets too large). Also, compressed symbol names cause troubles for OMF. Another way would be to write a small tool that turns all exported symbols of a library into non-exported symbols. I'm not sure we should do this in the compiler, because it simply hands over the libs to the linker. We'd have to make a copy of the library first and then do what a tool would do. Another note for the phobos case. You should really ask yourself why does your dll need a private copy of phobos. And if so why can phobos symbols leak out of your dll. AFAIU the discussed scenario is that the phobos library exports a lot of symbols, and this library is used both for static and dynamic linking. An executable statically linked against this library will have all the exports. That still works but is ugly. A DLL to be used as a plugin with a C interface might also want to link statically against phobos to avoid DLL hell. Consider multiple plugins built against different versions of phobos. These would also leak also the symbols.
Re: DIP45: fixing the dllimport/dllexport issue
On 09/07/2013 02:47 PM, Benjamin Thaut wrote: If you compile lib1 into a static library and then copmpile lib2 into a DLL which statically links against lib1 both symbols lib1Func and lib2Func will be exported from the dll because lib1Func will get marked as dllexport when compiled into a static library and the static library transports that information into the linking process of lib2. This happens both on win32 and win64. But this is something you really don't want. Lets imagine you have a DLL that statically links against phobos. Suddenly you will have all phobos exported symbols in your dll. This can also lead to double defined symbols in case a user links against your dll and against the shared phobos dll. So we have to fix this somehow. I had another thought about this and tried some examples. If you do transitively link against lib1.lib symbols when using lib2.dll you might get ODR issues if there is another copy of lib1 in your process. This is to be expected because there are two copies of the same library. The safe choice is to make lib1 a DLL too. If you still want to statically link lib1 into lib2.dll you have to be very cautious. An actual problem only occurs when lib1 symbols get used from outside of lib2.dll. Of course a mean to prevent bugs from accidental linking lib1 symbols would be nice. One way of doing this would be to mark all lib1 symbols as PRIVATE in lib2.def. EXPORTS D4lib18lib1FuncFZi PRIVATE Another way would be to write a small tool that turns all exported symbols of a library into non-exported symbols. I'm not sure we should do this in the compiler, because it simply hands over the libs to the linker. We'd have to make a copy of the library first and then do what a tool would do. Another note for the phobos case. You should really ask yourself why does your dll need a private copy of phobos. And if so why can phobos symbols leak out of your dll.
Re: DIP45: fixing the dllimport/dllexport issue
Can we deal with vtable entries pointing to imported functions on Windows? The kind of thing that happens when you inherit across a DLL boundary.
Re: DIP45: fixing the dllimport/dllexport issue
On 09/07/2013 02:47 PM, Benjamin Thaut wrote: If you compile lib1 into a static library and then copmpile lib2 into a DLL which statically links against lib1 both symbols lib1Func and lib2Func will be exported from the dll That might indeed cause some issues, but when you make lib1 part of your DLL by statically linking against it that's what you get. If you just want to use it, you should make lib1 a DLL as well, which also avoids ODR problems. Also could a .DEF file control which symbols are reexported?
Re: DIP45: fixing the dllimport/dllexport issue
I also kind of feel that we need to export lib1 as well, as you may not know what lib2 does with what it gets from lib1 (passing objects around for instance). So certainly at least some part of lib1 need to be exported. For sure lib2 might forward to some lib1 symbols. Not sure what might accidentally require linkage. I don't agree if that statement. If you have three libs. lib1, lib2 and lib3. And lib2 and lib3 link statically against lib1 you are going to get linker errors because both lib2 and lib3 contain the symbols of lib1. A: Why do two DLLs which export the same symbols cause linker errors? Because the corresponding import libraries contain the same symbols? B: You shouldn't mix static with dynamic linking, ODR issues are what you get in return. C: What would happen if you statically link against lib2 and lib3?
Re: DIP45: fixing the dllimport/dllexport issue
Am 08.09.2013 09:33, schrieb Benjamin Thaut: Am 27.08.2013 12:12, schrieb Benjamin Thaut: The current behaviour of export is not sufficient to create a shared runtime on windows. Its alos not working very well in a few other cases. For background information please read the links provided in the DIP. This DIP tries to solve the problem by imitating the proven preprocessor techniques used in C/C++ to create windows dlls. Destroy. Kind Regards Benjamin Thaut There is yet another problem to be solved. What should happen with tempaltes? Does it make sense to export templates? What happens if a template class marked with export gets instanciated in different DLLs using the same template arguments? Should the TypeInfo objects match? Should templates be exportable in generall? In C++ the only thing I needed so far was exporting static members of templated classes. So the different instances of the template can work on the same data. Kind Regards Benjamin Thaut Good question. Not sure how Ada, Modula-3, Eiffel, Delphi, MLton solve the issue, even though their generic capabilities are not as powerfull. -- Paulo
Re: DIP45: fixing the dllimport/dllexport issue
Am 27.08.2013 12:12, schrieb Benjamin Thaut: The current behaviour of export is not sufficient to create a shared runtime on windows. Its alos not working very well in a few other cases. For background information please read the links provided in the DIP. This DIP tries to solve the problem by imitating the proven preprocessor techniques used in C/C++ to create windows dlls. Destroy. Kind Regards Benjamin Thaut There is yet another problem to be solved. What should happen with tempaltes? Does it make sense to export templates? What happens if a template class marked with export gets instanciated in different DLLs using the same template arguments? Should the TypeInfo objects match? Should templates be exportable in generall? In C++ the only thing I needed so far was exporting static members of templated classes. So the different instances of the template can work on the same data. Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
Am 08.09.2013 06:19, schrieb deadalnix: On Saturday, 7 September 2013 at 21:41:01 UTC, Benjamin Thaut wrote: I don't agree if that statement. If you have three libs. lib1, lib2 and lib3. And lib2 and lib3 link statically against lib1 you are going to get linker errors because both lib2 and lib3 contain the symbols of lib1. And you don't have any options to avoid that (unless you got into the source of lib1 and remove all export attributes, which might not even be possible because you don't have the source) And you should have an error, especially is you pass object from one side to the other. typeid won't match (which will create a mess in the runtime), and worse, struct layout may not match (hello memory corruption). If the problem is that big, we can still have a -noexport flag or something, but that doesn't seems really safe to me. But only assuming that you are passing around objects of that type. I see a dll as a module of encapsulation. If the library is only unsed internally nothing of its dependencies should get exported. Because if they are your users might rely on the fact that they are, and from that point on you have to garantuee that all those additional exported functions from your dependencies stay in your library, even if you don't need the dependency anymore.
Re: DIP45: fixing the dllimport/dllexport issue
On 07.09.2013 14:47, Benjamin Thaut wrote: I propose that we add a command-line-paramter to the compiler (windows only) which actually enables dllexport. So all behavior described in the DIP will be enabled by default, but to actually make it mark symbols with dllexport you have to specifiy a additional command-line-parameter. We could call it "-dll" or something. I don't think the issue is Windows specific, symbol visibility in the GNU toolchain is also bound to the declaration. I remember seeing public (visibility=default) symbols exported from static libraries on OSX.
Re: DIP45: fixing the dllimport/dllexport issue
On Saturday, 7 September 2013 at 21:41:01 UTC, Benjamin Thaut wrote: I don't agree if that statement. If you have three libs. lib1, lib2 and lib3. And lib2 and lib3 link statically against lib1 you are going to get linker errors because both lib2 and lib3 contain the symbols of lib1. And you don't have any options to avoid that (unless you got into the source of lib1 and remove all export attributes, which might not even be possible because you don't have the source) And you should have an error, especially is you pass object from one side to the other. typeid won't match (which will create a mess in the runtime), and worse, struct layout may not match (hello memory corruption). If the problem is that big, we can still have a -noexport flag or something, but that doesn't seems really safe to me.
Re: DIP45: fixing the dllimport/dllexport issue
On Friday, 6 September 2013 at 02:00:22 UTC, Martin Nowak wrote: Yes, it has a bunch of special relocations and runtime linker support for this. It would be nice if someone could figure out and summarize all the TLS index and TIB business on Windows. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation http://en.wikipedia.org/wiki/Win32_Thread_Information_Block Best material I found on Windows TLS so far: http://www.nynaeve.net/?s=tls Haven't really looked into the interaction between multiple DLLs at all so far, though. David
Re: DIP45: fixing the dllimport/dllexport issue
Am 07.09.2013 20:42, schrieb deadalnix: On Saturday, 7 September 2013 at 12:47:19 UTC, Benjamin Thaut wrote: I propose that we add a command-line-paramter to the compiler (windows only) which actually enables dllexport. So all behavior described in the DIP will be enabled by default, but to actually make it mark symbols with dllexport you have to specifiy a additional command-line-parameter. We could call it "-dll" or something. I have to think about it. But I'd like to avoid any special behavior for windows language side. I also kind of feel that we need to export lib1 as well, as you may not know what lib2 does with what it gets from lib1 (passing objects around for instance). So certainly at least some part of lib1 need to be exported. I don't agree if that statement. If you have three libs. lib1, lib2 and lib3. And lib2 and lib3 link statically against lib1 you are going to get linker errors because both lib2 and lib3 contain the symbols of lib1. And you don't have any options to avoid that (unless you got into the source of lib1 and remove all export attributes, which might not even be possible because you don't have the source)
Re: DIP45: fixing the dllimport/dllexport issue
On Saturday, 7 September 2013 at 12:47:19 UTC, Benjamin Thaut wrote: I propose that we add a command-line-paramter to the compiler (windows only) which actually enables dllexport. So all behavior described in the DIP will be enabled by default, but to actually make it mark symbols with dllexport you have to specifiy a additional command-line-parameter. We could call it "-dll" or something. I have to think about it. But I'd like to avoid any special behavior for windows language side. I also kind of feel that we need to export lib1 as well, as you may not know what lib2 does with what it gets from lib1 (passing objects around for instance). So certainly at least some part of lib1 need to be exported.
Re: DIP45: fixing the dllimport/dllexport issue
Ok there is a new problem. Consider the following example:
module lib1;
export int lib1Func()
{
return 5;
}
module lib2;
import lib1;
export int lib2Func()
{
return lib1Func() + 1;
}
If you compile lib1 into a static library and then copmpile lib2 into a
DLL which statically links against lib1 both symbols lib1Func and
lib2Func will be exported from the dll because lib1Func will get marked
as dllexport when compiled into a static library and the static library
transports that information into the linking process of lib2. This
happens both on win32 and win64. But this is something you really don't
want. Lets imagine you have a DLL that statically links against phobos.
Suddenly you will have all phobos exported symbols in your dll. This can
also lead to double defined symbols in case a user links against your
dll and against the shared phobos dll. So we have to fix this somehow.
I propose that we add a command-line-paramter to the compiler (windows
only) which actually enables dllexport. So all behavior described in the
DIP will be enabled by default, but to actually make it mark symbols
with dllexport you have to specifiy a additional command-line-parameter.
We could call it "-dll" or something.
In case you want to try it out yourself, here is the repro case:
http://stuff.benjamin-thaut.de/D/dll2.zip
Kind Regards
Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
Am 06.09.2013 04:00, schrieb Martin Nowak: On 09/01/2013 09:33 PM, Benjamin Thaut wrote: I didn't know linux has a facility for accessing TLS data across shared library boundaries. I will put the implementation suggestion into the windows specific section then. Yes, it has a bunch of special relocations and runtime linker support for this. It would be nice if someone could figure out and summarize all the TLS index and TIB business on Windows. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation http://en.wikipedia.org/wiki/Win32_Thread_Information_Block What do we need this information for? -- Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
Am 06.09.2013 03:51, schrieb Martin Nowak: On 09/01/2013 05:26 PM, Rainer Schuetze wrote: Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Based on your saying that there are no absolute data relocations for this on windows we'd have to perform runtime initialization. At runtime it would simply copy _imp_var into pvar, right? Yes. As Rainer already mentioned we can either do this by running some initialiers before initializing druntime, or we could add additional information in a section of the dll and run some generic dll initialization code. Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment. No you can't take the address of a TLS var at compile time. Its not taking a address of a TLS variable. It is initializing a TLS variable with the address of a __gshared variable. -- Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On 09/01/2013 09:33 PM, Benjamin Thaut wrote: I didn't know linux has a facility for accessing TLS data across shared library boundaries. I will put the implementation suggestion into the windows specific section then. Yes, it has a bunch of special relocations and runtime linker support for this. It would be nice if someone could figure out and summarize all the TLS index and TIB business on Windows. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686997(v=vs.85).aspx http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation http://en.wikipedia.org/wiki/Win32_Thread_Information_Block
Re: DIP45: fixing the dllimport/dllexport issue
On 09/01/2013 05:26 PM, Rainer Schuetze wrote: Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Based on your saying that there are no absolute data relocations for this on windows we'd have to perform runtime initialization. At runtime it would simply copy _imp_var into pvar, right? Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment. No you can't take the address of a TLS var at compile time.
Re: DIP45: fixing the dllimport/dllexport issue
On 03.09.2013 08:17, Kagamin wrote: On Friday, 30 August 2013 at 10:50:40 UTC, Rainer Schuetze wrote: Just remembered that we don't have that in Windows Why? What's the difference between code and data symbols? I thought, they're just addresses. Inside an executable/DLL image, all relocations are relative to the image itself, only the import table has references to other DLLs. To initialize data with a pointer into another DLL, you cannot do that with a direct relocation, you have to run some code to copy the value from the import table into the data. Accessing data or functions in another DLL from code already has this extra code compiled in.
Re: DIP45: fixing the dllimport/dllexport issue
On Friday, 30 August 2013 at 10:50:40 UTC, Rainer Schuetze wrote: Just remembered that we don't have that in Windows Why? What's the difference between code and data symbols? I thought, they're just addresses.
Re: DIP45: fixing the dllimport/dllexport issue
Am 01.09.2013 21:19, schrieb Iain Buclaw: On 1 September 2013 19:16, David Nadlinger wrote: On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote: The DIP has a seperate section on TLS variables. Did you miss to read that? Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David Yes, no need to invent a new cog to go on top of what the base runtime already offers. Basically I thought whatever is inside the Description would go into the spec. Everything inside the implementation details is just a suggestion for implementing proper dllexport / dllimport on windows plattforms. We shouldn't hard wire these implementation suggestions into the spec.
Re: DIP45: fixing the dllimport/dllexport issue
Am 01.09.2013 19:58, schrieb Rainer Schuetze: On 01.09.2013 18:43, Benjamin Thaut wrote: Am 01.09.2013 17:26, schrieb Rainer Schuetze: On 01.09.2013 11:49, Benjamin Thaut wrote: I updated the DIP with all discussed content. Feedback is welcome. http://wiki.dlang.org/DIP45 Kind Regards Benjamin Thaut LGTM. Lets get this merged? ^^ First someone has to implement it ;-) What about the current use case of just exporting some functions for being loaded by a C application? It should still work using a def-file (which you have to use anyway most of the time due to http://d.puremagic.com/issues/show_bug.cgi?id=3956) Well you can still use a .def file of course. Also no one stops you just annotating simple extern(C) functions with the export attribute. I don't see any problem there. It would export the ModuleInfo and related symbols. Is this a issue? There just would be some unused exports in the dll, wouldn't it? If this really is a problem we could add a compiler switch to stop implict exporting of symbols. Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Yes, you are absolutely correct. I don't know enough about the internals of a compiler (dmd in perticular) though to answer this question. Can't we use the same mechanism C++ uses to run all its initializers? Yes, it would be an option to generate init functions to be run with the C initializers. But that might introduce a dependency on initialization order. We can write relocation info into another segment and do the relocations before runtime init. I would say whatever works best for the plattform at hand. No need to force one way or the other. Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment. The DIP has a seperate section on TLS variables. Did you miss to read that? No. I meant thread global variables that are preinitialized. The appropriate relocation doesn't exist, the __acccess_tls_var function has to be called in a thread local init function. Can't we initialize these right before running the shared module constructors? To make the implementation less work we could also make this a error for now and maybe implement it later. What would you prefer? Rainer
Re: DIP45: fixing the dllimport/dllexport issue
Am 01.09.2013 21:19, schrieb Iain Buclaw: On 1 September 2013 19:16, David Nadlinger wrote: On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote: The DIP has a seperate section on TLS variables. Did you miss to read that? Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David Yes, no need to invent a new cog to go on top of what the base runtime already offers. I didn't know linux has a facility for accessing TLS data across shared library boundaries. I will put the implementation suggestion into the windows specific section then.
Re: DIP45: fixing the dllimport/dllexport issue
On 1 September 2013 19:16, David Nadlinger wrote: > On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote: >> >> The DIP has a seperate section on TLS variables. Did you miss to read >> that? > > > Can't we just use the normal (g)libc facilities for handling TLS in shared > objects on Linux? The TLS data from the "DLL" is assigned to another tls > module, and everything is fine, or am I missing something? > > David Yes, no need to invent a new cog to go on top of what the base runtime already offers. -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Re: DIP45: fixing the dllimport/dllexport issue
On Sunday, 1 September 2013 at 16:43:14 UTC, Benjamin Thaut wrote: The DIP has a seperate section on TLS variables. Did you miss to read that? Can't we just use the normal (g)libc facilities for handling TLS in shared objects on Linux? The TLS data from the "DLL" is assigned to another tls module, and everything is fine, or am I missing something? David
Re: DIP45: fixing the dllimport/dllexport issue
On 01.09.2013 18:43, Benjamin Thaut wrote: Am 01.09.2013 17:26, schrieb Rainer Schuetze: On 01.09.2013 11:49, Benjamin Thaut wrote: I updated the DIP with all discussed content. Feedback is welcome. http://wiki.dlang.org/DIP45 Kind Regards Benjamin Thaut LGTM. Lets get this merged? ^^ First someone has to implement it ;-) What about the current use case of just exporting some functions for being loaded by a C application? It should still work using a def-file (which you have to use anyway most of the time due to http://d.puremagic.com/issues/show_bug.cgi?id=3956) Well you can still use a .def file of course. Also no one stops you just annotating simple extern(C) functions with the export attribute. I don't see any problem there. It would export the ModuleInfo and related symbols. Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Yes, you are absolutely correct. I don't know enough about the internals of a compiler (dmd in perticular) though to answer this question. Can't we use the same mechanism C++ uses to run all its initializers? Yes, it would be an option to generate init functions to be run with the C initializers. But that might introduce a dependency on initialization order. We can write relocation info into another segment and do the relocations before runtime init. Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment. The DIP has a seperate section on TLS variables. Did you miss to read that? No. I meant thread global variables that are preinitialized. The appropriate relocation doesn't exist, the __acccess_tls_var function has to be called in a thread local init function. Rainer
Re: DIP45: fixing the dllimport/dllexport issue
Am 01.09.2013 17:26, schrieb Rainer Schuetze: On 01.09.2013 11:49, Benjamin Thaut wrote: I updated the DIP with all discussed content. Feedback is welcome. http://wiki.dlang.org/DIP45 Kind Regards Benjamin Thaut LGTM. Lets get this merged? ^^ First someone has to implement it ;-) What about the current use case of just exporting some functions for being loaded by a C application? It should still work using a def-file (which you have to use anyway most of the time due to http://d.puremagic.com/issues/show_bug.cgi?id=3956) Well you can still use a .def file of course. Also no one stops you just annotating simple extern(C) functions with the export attribute. I don't see any problem there. Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Yes, you are absolutely correct. I don't know enough about the internals of a compiler (dmd in perticular) though to answer this question. Can't we use the same mechanism C++ uses to run all its initializers? Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment. The DIP has a seperate section on TLS variables. Did you miss to read that? Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On 01.09.2013 11:49, Benjamin Thaut wrote: I updated the DIP with all discussed content. Feedback is welcome. http://wiki.dlang.org/DIP45 Kind Regards Benjamin Thaut LGTM. What about the current use case of just exporting some functions for being loaded by a C application? It should still work using a def-file (which you have to use anyway most of the time due to http://d.puremagic.com/issues/show_bug.cgi?id=3956) Maybe already deep into implementation but how are we going to deal with the initialization of data? module sharedlib; __gshared int var; module exe; import sharedlib; __gshared int* pvar = &var; (This kind of relocation also has to be done for stuff like TypeInfo_Class). It needs initialization code to fill pvar with the correct pointer. Do we support the same for pointers in TLS? int* tlsvar = &var; This might need initialization per thread, or patching the original TLS segment.
Re: DIP45: fixing the dllimport/dllexport issue
I updated the DIP with all discussed content. Feedback is welcome. http://wiki.dlang.org/DIP45 Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
Am 30.08.2013 10:23, schrieb Benjamin Thaut: 1) Use alias symbols I'm convinced now that Martin Nowaks aliasing solution is the better solution as there will be less runtime overhead then I originally thought. Also the usability will be better. It also leaves room for further optimizations by doing some clever pointer patching, in case we find a solution that works on x64. If no one does a veto I'm going to update the DIP with this aliasing solution and all other discussed changes. (e.g. making export a attribute) Kind Regards Benjamin Thaut -- Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On 08/30/2013 08:20 AM, Rainer Schuetze wrote: It's a bit easier to see the code by adding debug symbols and using dumpbin /disasm main.exe. Unfortnately, this won't help us a lot, because the intermediate object files have some unknown format and in fact are just transferring some code representation to the linker that then invokes the compiler and optimizer on the full source code. This is very specific to the C/C++ toolchain and I don't think we can take advantage of it. This is sadly true for the dmd->COFF->VC link toolchain. But we'll likely see this from GDC or LDC in the future. Despite that I don't think this is a huge performance issue at all we could put all the import data pointers in one section so that they are all packed and allow efficient use of the CPU cache.
Re: DIP45: fixing the dllimport/dllexport issue
On 08/30/2013 12:50 PM, Rainer Schuetze wrote: References from the data segment use absolute relocations. Just remembered that we don't have that in Windows, so if we want to mimick what C++ does, we'll have to generate initialization-code to be run at startup to fill in the proper pointers from the import table. This is also done at runtime by the loader. The additional startup time is one reason why exporting data in ABIs is bad. The other is copy relocations for writeable data (http://docs.oracle.com/cd/E19082-01/819-0690/chapter4-84604/index.html).
Re: DIP45: fixing the dllimport/dllexport issue
On Friday, 30 August 2013 at 06:23:02 UTC, Rainer Schuetze wrote: On 29.08.2013 21:09, Martin Nowak wrote: References from the data segment use absolute relocations. References from PIC code use the GOT. So an indirection through the GOT is added to every access to a non-static global if you compile with -fPIC? My understanding was references using the GOT have the same number of indirections as in the PLT -- that is, a one-time cost for the dynamic linker to dereference and replace an entry with the absolute address -- and the major difference is PLT is done lazily, while GOT is fixed up in its entirety at runtime. Having said that, now I wonder if I'm misunderstanding the question? -Wyatt
Re: DIP45: fixing the dllimport/dllexport issue
On 29.08.2013 21:09, Martin Nowak wrote: On 08/29/2013 08:30 PM, Rainer Schuetze wrote: I meant the import part. How are accesses or data references to data inside another shared library implemented? References from the data segment use absolute relocations. Just remembered that we don't have that in Windows, so if we want to mimick what C++ does, we'll have to generate initialization-code to be run at startup to fill in the proper pointers from the import table.
Re: DIP45: fixing the dllimport/dllexport issue
Am 30.08.2013 11:52, schrieb deadalnix: On Friday, 30 August 2013 at 09:33:25 UTC, Benjamin Thaut wrote: Am 30.08.2013 11:17, schrieb deadalnix: My understanding is that it can be optimized away. Or can it ? It can in C/C++ because microsoft has really good LTO (link time optimization). As Rainer Schuetze explained the format of the object files used for LTO is propriatary and unkown to the public. So we can not easily use that, and its unlikely we can use it at all. Also for 32-bit windows it can not be optimized away because optlink does not do LTO. win32 is kind of a dead end anyway. It is still very alive today, but we want to aim for the future. And it WILL disappear. The object format is kind of anoying. In theory it could be optimized away, but in pratice I doubt we are going to see this kind of optimization in dmd any time soon. And as dmd is the only viable option on windows right now I would prefer the better performing second solution to this problem I understand. But sacrificing good design for immediate reward is bound to cost way more in the long run if D is successful. And if D fails, then it doesn't matter. I don't consider good it good design to add lots of indirections just to remove a command line parameter to the compiler. Its not like its going to make any difference in source code.
Re: DIP45: fixing the dllimport/dllexport issue
On Friday, 30 August 2013 at 09:33:25 UTC, Benjamin Thaut wrote: Am 30.08.2013 11:17, schrieb deadalnix: My understanding is that it can be optimized away. Or can it ? It can in C/C++ because microsoft has really good LTO (link time optimization). As Rainer Schuetze explained the format of the object files used for LTO is propriatary and unkown to the public. So we can not easily use that, and its unlikely we can use it at all. Also for 32-bit windows it can not be optimized away because optlink does not do LTO. win32 is kind of a dead end anyway. It is still very alive today, but we want to aim for the future. And it WILL disappear. The object format is kind of anoying. In theory it could be optimized away, but in pratice I doubt we are going to see this kind of optimization in dmd any time soon. And as dmd is the only viable option on windows right now I would prefer the better performing second solution to this problem I understand. But sacrificing good design for immediate reward is bound to cost way more in the long run if D is successful. And if D fails, then it doesn't matter.
Re: DIP45: fixing the dllimport/dllexport issue
Am 30.08.2013 11:17, schrieb deadalnix: My understanding is that it can be optimized away. Or can it ? It can in C/C++ because microsoft has really good LTO (link time optimization). As Rainer Schuetze explained the format of the object files used for LTO is propriatary and unkown to the public. So we can not easily use that, and its unlikely we can use it at all. Also for 32-bit windows it can not be optimized away because optlink does not do LTO. In theory it could be optimized away, but in pratice I doubt we are going to see this kind of optimization in dmd any time soon. And as dmd is the only viable option on windows right now I would prefer the better performing second solution to this problem
Re: DIP45: fixing the dllimport/dllexport issue
On Friday, 30 August 2013 at 08:23:20 UTC, Benjamin Thaut wrote: So let me summarize the currently discussed solutions: 1) Use alias symbols Pros: - Better usability (no additional command line parameters needed when compiling / linking against DLLs) Cons: - Less efficient code for cross DLL function calls (one additional call instruction) - Less efficient code for all accesses to global data, no matter if it will end up in a DLL or not. That means __gshared variables, global shared variables, hidden global data like module info, type info, vtables, etc (one additional level of indirection). It might be possible to avoid this in most cases using link time optimization. But it is unlikely that we can get LTO to work easily. - Additional level of indirection might confuse current debuggers My understanding is that it can be optimized away. Or can it ? 2) Use additional command line Pros: - More efficient code Cons: - Additional command line parameters needed when compiling / linking against a DLL. (can be hidden away inside sc.ini for phobos / druntime)
Re: DIP45: fixing the dllimport/dllexport issue
So let me summarize the currently discussed solutions: 1) Use alias symbols Pros: - Better usability (no additional command line parameters needed when compiling / linking against DLLs) Cons: - Less efficient code for cross DLL function calls (one additional call instruction) - Less efficient code for all accesses to global data, no matter if it will end up in a DLL or not. That means __gshared variables, global shared variables, hidden global data like module info, type info, vtables, etc (one additional level of indirection). It might be possible to avoid this in most cases using link time optimization. But it is unlikely that we can get LTO to work easily. - Additional level of indirection might confuse current debuggers 2) Use additional command line Pros: - More efficient code Cons: - Additional command line parameters needed when compiling / linking against a DLL. (can be hidden away inside sc.ini for phobos / druntime)
Re: DIP45: fixing the dllimport/dllexport issue
On 29.08.2013 21:09, Martin Nowak wrote: On 08/29/2013 08:30 PM, Rainer Schuetze wrote: I meant the import part. How are accesses or data references to data inside another shared library implemented? References from the data segment use absolute relocations. References from PIC code use the GOT. So an indirection through the GOT is added to every access to a non-static global if you compile with -fPIC?
Re: DIP45: fixing the dllimport/dllexport issue
On 29.08.2013 23:40, Martin Nowak wrote:
On 08/29/2013 10:17 PM, Benjamin Thaut wrote:
How about LTO, when statically linking it should be possible to optimize
away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this
optimizations. But I don't know aynthing further about this topic.
module libA;
export int var;
int* _imp_var = &var; // created by compiler
module foo;
import libA;
void bar()
{
auto val = var; // creates val = *_imp_var;
}
Yes that would work. Still there must be a reason why microsoft doesn't
do stuff like that in their C++ toolchain. Its certanly going to cost
performance.
I just tested it and it works.
lib.c
int var = 0xdeadbeaf;
int* _imp_var = &var;
main.c
#include
extern int* _imp_var;
void main()
{
printf("%d\n", *_imp_var);
}
cl /c /O2 /GL lib.c
cl /O2 /GL main.c lib.obj
get objconv from http://www.agner.org/optimize/
objconv -fasm main.exe
Search for deadbeaf in main.asm to get the symbol number (?_1176).
It directly loads the variable.
mov edx, dword ptr [?_1176]
It's a bit easier to see the code by adding debug symbols and using
dumpbin /disasm main.exe.
Unfortnately, this won't help us a lot, because the intermediate object
files have some unknown format and in fact are just transferring some
code representation to the linker that then invokes the compiler and
optimizer on the full source code. This is very specific to the C/C++
toolchain and I don't think we can take advantage of it.
Re: DIP45: fixing the dllimport/dllexport issue
On Thursday, 29 August 2013 at 21:40:08 UTC, Martin Nowak wrote:
On 08/29/2013 10:17 PM, Benjamin Thaut wrote:
How about LTO, when statically linking it should be possible
to optimize
away the indirection.
Rainer Schuetze stated that some linkers are capable of doing
this
optimizations. But I don't know aynthing further about this
topic.
module libA;
export int var;
int* _imp_var = &var; // created by compiler
module foo;
import libA;
void bar()
{
auto val = var; // creates val = *_imp_var;
}
Yes that would work. Still there must be a reason why
microsoft doesn't
do stuff like that in their C++ toolchain. Its certanly going
to cost
performance.
I just tested it and it works.
lib.c
int var = 0xdeadbeaf;
int* _imp_var = &var;
main.c
#include
extern int* _imp_var;
void main()
{
printf("%d\n", *_imp_var);
}
cl /c /O2 /GL lib.c
cl /O2 /GL main.c lib.obj
get objconv from http://www.agner.org/optimize/
objconv -fasm main.exe
Search for deadbeaf in main.asm to get the symbol number
(?_1176).
It directly loads the variable.
mov edx, dword ptr [?_1176]
I was doubting my idea, but you conviced me :P
Re: DIP45: fixing the dllimport/dllexport issue
On 08/29/2013 10:17 PM, Benjamin Thaut wrote:
How about LTO, when statically linking it should be possible to optimize
away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this
optimizations. But I don't know aynthing further about this topic.
module libA;
export int var;
int* _imp_var = &var; // created by compiler
module foo;
import libA;
void bar()
{
auto val = var; // creates val = *_imp_var;
}
Yes that would work. Still there must be a reason why microsoft doesn't
do stuff like that in their C++ toolchain. Its certanly going to cost
performance.
I just tested it and it works.
lib.c
int var = 0xdeadbeaf;
int* _imp_var = &var;
main.c
#include
extern int* _imp_var;
void main()
{
printf("%d\n", *_imp_var);
}
cl /c /O2 /GL lib.c
cl /O2 /GL main.c lib.obj
get objconv from http://www.agner.org/optimize/
objconv -fasm main.exe
Search for deadbeaf in main.asm to get the symbol number (?_1176).
It directly loads the variable.
mov edx, dword ptr [?_1176]
Re: DIP45: fixing the dllimport/dllexport issue
Am 29.08.2013 22:04, schrieb Martin Nowak:
So the alias trick would only work for functions.
yes
For data the _imp_* symbol would need to be a pointer to the data?
yes
How about LTO, when statically linking it should be possible to optimize
away the indirection.
Rainer Schuetze stated that some linkers are capable of doing this
optimizations. But I don't know aynthing further about this topic.
module libA;
export int var;
int* _imp_var = &var; // created by compiler
module foo;
import libA;
void bar()
{
auto val = var; // creates val = *_imp_var;
}
Yes that would work. Still there must be a reason why microsoft doesn't
do stuff like that in their C++ toolchain. Its certanly going to cost
performance.
Re: DIP45: fixing the dllimport/dllexport issue
On 08/29/2013 09:28 PM, Benjamin Thaut wrote:
Well no its not that simple. You can't just alias it. You still have to
create a import table. That would mean ALL data accesses to data symbols
would have another level of indirection. Always. The _imp_ symbols don't
refer to the data directly, they refer to the location the data is
stored at. As explained in the article I linked
(http://blog.omega-prime.co.uk/?p=115) this requires different assembly
to be generated. To make this work we would always have to generate
assembly with one level of indirection added for every data symbol
access, because we can not know if the symbol might be imported from a
DLL or not. As D tries to achive near C++ performance I would arther
want to avoid that.
So the alias trick would only work for functions.
For data the _imp_* symbol would need to be a pointer to the data?
How about LTO, when statically linking it should be possible to optimize
away the indirection. Also are there special relocations?
module libA;
export int var;
int* _imp_var = &var; // created by compiler
module foo;
import libA;
void bar()
{
auto val = var; // creates val = *_imp_var;
}
Re: DIP45: fixing the dllimport/dllexport issue
Am 29.08.2013 21:20, schrieb Martin Nowak: On 08/29/2013 09:12 PM, Martin Nowak wrote: On 08/29/2013 12:03 PM, Benjamin Thaut wrote: But what if you import a module that is linked statically? That would mean export would be treated as dllimport and it will fail to link because the _imp_ symbols are missing when linking statically? Could we create alias symbols? Indeed this seems to work. OMF has an ALIAS record (http://www.azillionmonkeys.com/qed/Omfg.pdf) and COFF has weak externals (http://blog.omega-prime.co.uk/?p=121). So we could add _imp_* aliases for every exported symbol. When someone links against the import library they are never used, when someone links against a static library they redirect to the actual definitions. Would that work? Well no its not that simple. You can't just alias it. You still have to create a import table. That would mean ALL data accesses to data symbols would have another level of indirection. Always. The _imp_ symbols don't refer to the data directly, they refer to the location the data is stored at. As explained in the article I linked (http://blog.omega-prime.co.uk/?p=115) this requires different assembly to be generated. To make this work we would always have to generate assembly with one level of indirection added for every data symbol access, because we can not know if the symbol might be imported from a DLL or not. As D tries to achive near C++ performance I would arther want to avoid that.
Re: DIP45: fixing the dllimport/dllexport issue
On 08/29/2013 09:12 PM, Martin Nowak wrote: On 08/29/2013 12:03 PM, Benjamin Thaut wrote: But what if you import a module that is linked statically? That would mean export would be treated as dllimport and it will fail to link because the _imp_ symbols are missing when linking statically? Could we create alias symbols? Indeed this seems to work. OMF has an ALIAS record (http://www.azillionmonkeys.com/qed/Omfg.pdf) and COFF has weak externals (http://blog.omega-prime.co.uk/?p=121). So we could add _imp_* aliases for every exported symbol. When someone links against the import library they are never used, when someone links against a static library they redirect to the actual definitions. Would that work?
Re: DIP45: fixing the dllimport/dllexport issue
Am 29.08.2013 21:12, schrieb Martin Nowak: On 08/29/2013 12:03 PM, Benjamin Thaut wrote: But what if you import a module that is linked statically? That would mean export would be treated as dllimport and it will fail to link because the _imp_ symbols are missing when linking statically? Could we create alias symbols? Yes, but that would add a indirection to all data accesses even if you link the library in statically. That would basically remove the benefit from linking statically. If we want control over what gets exported and what not we will need export anyway, and in my eyes a additional compiler switch for windows is not bad enough to justify adding a level of indirction to data accesses into static libraries. Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On 08/29/2013 12:03 PM, Benjamin Thaut wrote: But what if you import a module that is linked statically? That would mean export would be treated as dllimport and it will fail to link because the _imp_ symbols are missing when linking statically? Could we create alias symbols?
Re: DIP45: fixing the dllimport/dllexport issue
On 08/29/2013 08:30 PM, Rainer Schuetze wrote: I meant the import part. How are accesses or data references to data inside another shared library implemented? References from the data segment use absolute relocations. References from PIC code use the GOT.
Re: DIP45: fixing the dllimport/dllexport issue
I found a very good articel on the dllimport/dllexport issue on windows explaining both dynamic linking of function symbols and data symbols: http://blog.omega-prime.co.uk/?p=115
Re: DIP45: fixing the dllimport/dllexport issue
On 29.08.2013 08:54, Jacob Carlborg wrote: On 2013-08-29 08:43, Rainer Schuetze wrote: How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker? "export" is noop on Posix. Every symbol is accessible. I meant the import part. How are accesses or data references to data inside another shared library implemented?
Re: DIP45: fixing the dllimport/dllexport issue
On 29.08.2013 11:33, Kagamin wrote: On Thursday, 29 August 2013 at 06:43:39 UTC, Rainer Schuetze wrote: How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker? I guess, it adds missing functions which jump to the actual implementation in so. This is done on windows too in import libraries, but export qualifier makes code a little faster because you make one indirect call instead of a direct call + indirect jump (it can be direct, but for this loader should be able to write to the code section to relocate the jump). You also only need an import table for such call. That's how it works for functions, but what about accesses to data in another DLL? Suppose phobos being built as a shared library, how do you access typeid(Object) from another DLL, both from code or inside data, e.g. as the base class member of a class info struct?
Re: DIP45: fixing the dllimport/dllexport issue
Am 29.08.2013 11:42, schrieb deadalnix: import declaration marked as export as understood as importing a module compiled in a shared object. This time, declaration in the imported mopdule marked as export are understood as ddlimport. But what if you import a module that is linked statically? That would mean export would be treated as dllimport and it will fail to link because the _imp_ symbols are missing when linking statically?
Re: DIP45: fixing the dllimport/dllexport issue
On Thursday, 29 August 2013 at 09:13:02 UTC, Benjamin Thaut wrote: So what should be do? So far I like Andrej Mitrovic's solution best. But I think we should omit the placement holder so that instead of "-export std.*" we use "-export std" which basically means the same thing. Otherwise people might geht the idea stuff like this will work "-export *.somesubmodule". Personally I'm ok with any working solution. I wanted to use shared dlls since I started using D 3 years ago, and nothing has happend in that regard since then. Kind Regards Benjamin Thaut Here is my proposal. Everything is NOT exported on unixes system, just like on windows. It is consistent accross system and provide opportunities for optimizations and faster linkage. symobol marked as export are understood as ddlexport (or publicly visible on unixes) import declaration marked as export as understood as importing a module compiled in a shared object. This time, declaration in the imported mopdule marked as export are understood as ddlimport.
Re: DIP45: fixing the dllimport/dllexport issue
On Thursday, 29 August 2013 at 06:43:39 UTC, Rainer Schuetze wrote: How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker? I guess, it adds missing functions which jump to the actual implementation in so. This is done on windows too in import libraries, but export qualifier makes code a little faster because you make one indirect call instead of a direct call + indirect jump (it can be direct, but for this loader should be able to write to the code section to relocate the jump). You also only need an import table for such call.
Re: DIP45: fixing the dllimport/dllexport issue
Am 29.08.2013 10:46, schrieb David Nadlinger: On Thursday, 29 August 2013 at 08:36:03 UTC, Benjamin Thaut wrote: Am 29.08.2013 09:24, schrieb deadalnix: C++ is doing its best to move away from it. We should probably don't follow that convention. Moving away from what exactly? Everything being visible by default. So what should be do? So far I like Andrej Mitrovic's solution best. But I think we should omit the placement holder so that instead of "-export std.*" we use "-export std" which basically means the same thing. Otherwise people might geht the idea stuff like this will work "-export *.somesubmodule". Personally I'm ok with any working solution. I wanted to use shared dlls since I started using D 3 years ago, and nothing has happend in that regard since then. Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On Thursday, 29 August 2013 at 08:36:03 UTC, Benjamin Thaut wrote: Am 29.08.2013 09:24, schrieb deadalnix: C++ is doing its best to move away from it. We should probably don't follow that convention. Moving away from what exactly? Everything being visible by default.
Re: DIP45: fixing the dllimport/dllexport issue
Am 29.08.2013 09:24, schrieb deadalnix: On Thursday, 29 August 2013 at 06:54:46 UTC, Jacob Carlborg wrote: On 2013-08-29 08:43, Rainer Schuetze wrote: How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker? "export" is noop on Posix. Every symbol is accessible. C++ is doing its best to move away from it. We should probably don't follow that convention. Moving away from what exactly? -- Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On Thursday, 29 August 2013 at 06:54:46 UTC, Jacob Carlborg wrote: On 2013-08-29 08:43, Rainer Schuetze wrote: How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker? "export" is noop on Posix. Every symbol is accessible. C++ is doing its best to move away from it. We should probably don't follow that convention.
Re: DIP45: fixing the dllimport/dllexport issue
On 2013-08-29 08:43, Rainer Schuetze wrote: How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker? "export" is noop on Posix. Every symbol is accessible. -- /Jacob Carlborg
Re: DIP45: fixing the dllimport/dllexport issue
On 27.08.2013 15:40, Benjamin Thaut wrote: Am 27.08.2013 15:33, schrieb Andrej Mitrovic: On 8/27/13, Benjamin Thaut wrote: Well, a link would have been great: http://wiki.dlang.org/DIP45 Using export(identifier) is not going to be reliable, since chances of clashes are high. E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you won't be able to distinguish between the two. Instead I think 'identifier' should be a module name. However, I have a better idea. Why not only introduce compiler switches that are based on module names rather than having to annotate what export does in code? For example: dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib "-export libB.*" means all modules and subpackages of the libB package should be exported, whereas "-import libA.*" means the opposite for libA. This way you don't have to edit any existing code. I thought about something along those lines aswell. But wouldn't that create a very long command line if you do it for druntime? I think export (at module granularity) can be handled by building the modules with symbols to export using a command line switch "-exportall", and all other modules without. I'd prefer if the compilation against the imported library would be agnostic whether it is later linked against a static or dynamic library. How is this done on linux right now? It does not need "export"/"import" to build against a shared phobos library. Is "import" assumed for any data access and later removed by some magic in the linker?
Re: DIP45: fixing the dllimport/dllexport issue
Am 27.08.2013 15:33, schrieb Andrej Mitrovic: On 8/27/13, Benjamin Thaut wrote: Well, a link would have been great: http://wiki.dlang.org/DIP45 Using export(identifier) is not going to be reliable, since chances of clashes are high. E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you won't be able to distinguish between the two. Instead I think 'identifier' should be a module name. However, I have a better idea. Why not only introduce compiler switches that are based on module names rather than having to annotate what export does in code? For example: dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib "-export libB.*" means all modules and subpackages of the libB package should be exported, whereas "-import libA.*" means the opposite for libA. This way you don't have to edit any existing code. I thought about something along those lines aswell. But wouldn't that create a very long command line if you do it for druntime?
Re: DIP45: fixing the dllimport/dllexport issue
On 8/27/13, Benjamin Thaut wrote: > Well, a link would have been great: > > http://wiki.dlang.org/DIP45 Using export(identifier) is not going to be reliable, since chances of clashes are high. E.g. if libFoo uses export(Foo) and libBar also uses export(Foo), you won't be able to distinguish between the two. Instead I think 'identifier' should be a module name. However, I have a better idea. Why not only introduce compiler switches that are based on module names rather than having to annotate what export does in code? For example: dmd -m64 -export libB.* -import libA.* -of"libB.dll" dllmain.d libB.d -L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib "-export libB.*" means all modules and subpackages of the libB package should be exported, whereas "-import libA.*" means the opposite for libA. This way you don't have to edit any existing code.
Re: DIP45: fixing the dllimport/dllexport issue
Am 27.08.2013 13:28, schrieb deadalnix: On Tuesday, 27 August 2013 at 10:16:24 UTC, Benjamin Thaut wrote: Well, a link would have been great: http://wiki.dlang.org/DIP45 I don't think that is the way to go. We need a way to mark a module as being an "header" import for a shared lib. With that solution, you'll ends up duplicating all modules with import and export, and putting the export(foobar) all over the place in a module. Well ideally it should be enough to put one export(foobar): at the top of the file. Also I don't see why there is any duplication of modules involved? What exactly do you mean? And how should your header import for a shared lib work, if you link against exactly that lib statically? Kind Regards Benjamin Thaut
Re: DIP45: fixing the dllimport/dllexport issue
On Tuesday, 27 August 2013 at 10:16:24 UTC, Benjamin Thaut wrote: Well, a link would have been great: http://wiki.dlang.org/DIP45 I don't think that is the way to go. We need a way to mark a module as being an "header" import for a shared lib. With that solution, you'll ends up duplicating all modules with import and export, and putting the export(foobar) all over the place in a module.
Re: DIP45: fixing the dllimport/dllexport issue
Well, a link would have been great: http://wiki.dlang.org/DIP45
DIP45: fixing the dllimport/dllexport issue
The current behaviour of export is not sufficient to create a shared runtime on windows. Its alos not working very well in a few other cases. For background information please read the links provided in the DIP. This DIP tries to solve the problem by imitating the proven preprocessor techniques used in C/C++ to create windows dlls. Destroy. Kind Regards Benjamin Thaut
