Re: F*cked by memory corruption after assiging value to associative array
On Friday, 29 January 2021 at 15:09:23 UTC, ShadoLight wrote: Just to confirm... I assume you just neglected to show the line in fun template function that returns the object, right? Yes, that's pseudo code with a missed return :D
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 17:11:37 UTC, frame wrote:
On Monday, 25 January 2021 at 16:54:42 UTC, vitamin wrote:
On Monday, 25 January 2021 at 16:44:40 UTC, frame wrote:
On Monday, 25 January 2021 at 16:14:05 UTC, vitamin wrote:
Yes, I directly calling on every function that returns an
object:
T fun(T)(T object) {
GC.addRoot(cast(void*) object);
}
...
extern (C) export SomeObject bar() {
return fun(new SomeObject);
}
Just to confirm... I assume you just neglected to show the line
in fun template function that returns the object, right?
Like...
T fun(T)(T object) {
GC.addRoot(cast(void*) object);
return object;
}
Re: F*cked by memory corruption after assiging value to associative array
On Thursday, 28 January 2021 at 22:11:40 UTC, tsbockman wrote: Alternatively, you can design your APIs so that no pointer to GC memory is ever owned or mutated by any thread unknown to that GC. (This is the only option when working across language boundaries.) Yes, thank you for your input - I was already thinking about that as it shows the better design and also wouldn't require that the DLL itself build-in much redundant code. However, if I run GC.collect() after every DLL function was done then it clearly shows which data goes away and bite me. Basically most objects are allocated in the main EXE/DLL anyway - it only comes in trouble where a new separate object is returned by the sub DLL - GC.addRoot() really solves that problem.
Re: F*cked by memory corruption after assiging value to associative array
On Thursday, 28 January 2021 at 20:17:09 UTC, frame wrote: On Thursday, 28 January 2021 at 19:22:16 UTC, tsbockman wrote: It is possible to get things sort of working with on Windows, anyway. I'm ok with it as long as the memory is not re-used by the GC. It seems that it can be prevented with addRoot() successfully. GC.addRoot is not enough by itself. Each GC needs to know about every single thread that may own or mutate any pointer to memory managed by that particular GC. If a GC doesn't know, memory may be prematurely freed, and therefore wrongly re-used. This is because when it scans memory for pointers to find out which memory is still in use, an untracked thread may be hiding a pointer on its stack or in registers, or it might move a pointer value from a location late in the scanning order to a location early in the scanning order while the GC is scanning the memory in between, such that the pointer value is not in either location *at the time the GC checks it*. You won't be able to test for this problem easily, because it is non-deterministic and depends upon the precise timing with which each thread is scheduled and memory is synchronized. But, it will probably still bite you later. If you were just manually creating additional threads unknown to the GC, you could tell the GC about them with core.thread.osthread.thread_attachThis and thread_detachThis. But, I don't know if those work right when there are multiple disconnected copies of D runtime running at the same time like this. The official solution is to get the GC proxy connected properly from each DLL to the EXE. This is still very broken on Windows in other ways (again, explained at my link), but it should at least prevent the race condition I described above, as well as being more efficient than running multiple GCs in parallel. Alternatively, you can design your APIs so that no pointer to GC memory is ever owned or mutated by any thread unknown to that GC. (This is the only option when working across language boundaries.)
Re: F*cked by memory corruption after assiging value to associative array
On Thursday, 28 January 2021 at 19:22:16 UTC, tsbockman wrote: It is possible to get things sort of working with on Windows, anyway. I'm ok with it as long as the memory is not re-used by the GC. It seems that it can be prevented with addRoot() successfully. The other problem with shared slice data is somewhat logical as the DLL GC doesn't care on the origin of the data from another thread and the data's origin GC sees any reference to it gone after passing it to the DLL function. They are isolated and data which must be kept longer should be copied where it's necessary.
Re: F*cked by memory corruption after assiging value to associative array
On Thursday, 28 January 2021 at 07:50:43 UTC, frame wrote: Under Linux everything is shared. Under Windows each DLL seems to run in its own thread, has its own rt_options and do not see any __gshared variable value. Its completely isolated and so I assume that also GC is. This stuff works correctly under Linux, and is quite broken in Windows. This has been known for years, but hasn't been fixed yet. This link for my other reply gives more details: https://forum.dlang.org/post/veeksndchoppftluj...@forum.dlang.org Also https://wiki.dlang.org/Win32_DLLs_in_D says: Each EXE and DLL will have their own gc instance. They each have their own GC instance because no one has fully fixed the problems discussed at my link, above, not because it's actually a good idea for them each to have their own GC instance. It is possible to get things sort of working with on Windows, anyway. But, this requires either: A) Following all the same rules that you would need to follow if you wanted to share D GCed memory with another thread written in C. (Just adding GC roots is not enough.) B) Ensuring that the GC proxy connections are properly established before doing anything else. This doesn't actually work correctly or reliably, but it might work well enough for your use case. Maybe.
Re: F*cked by memory corruption after assiging value to associative array
On Wednesday, 27 January 2021 at 22:57:11 UTC, tsbockman wrote: There is supposed to only be one instance of the D GC running per process. If you have more than one running then either you aren't linking and loading the DLLs correctly, or you have run into a serious bug in the D tooling. What could I do wrong by just using SimpleDllMain and then put my exports? build line for DLL is: rdmd -shared --build-only -gf -m64 Under Linux everything is shared. Under Windows each DLL seems to run in its own thread, has its own rt_options and do not see any __gshared variable value. Its completely isolated and so I assume that also GC is. Also https://wiki.dlang.org/Win32_DLLs_in_D says: Each EXE and DLL will have their own gc instance. I also wonder why the static linked DLL should use a GC proxy while as SimpleDllMain does nothing with a proxy - should loadLibrary() take care off here automatically? It seems, it does not.
Re: F*cked by memory corruption after assiging value to associative array
On Wednesday, 27 January 2021 at 18:09:39 UTC, frame wrote: I have no idea if there are multiple runtimes. I just use the mixin SimpleDllMain. But there must be multiple instances of GCs running because Another thread is running right now which I think is touching upon these same issues. Adam D. Ruppe explains some of what's going on: https://forum.dlang.org/post/veeksndchoppftluj...@forum.dlang.org Sadly, it looks like shared D DLLs are just kind of broken on Windows, unless you want to go the betterC route...
Re: F*cked by memory corruption after assiging value to associative array
On Wednesday, 27 January 2021 at 18:09:39 UTC, frame wrote: there must be multiple instances of GCs running because Sharing data between multiple threads that each use a different instance of the D GC will definitely not work right, because each GC will only know to pause the threads and scan the roots that it has been directly informed of. There is supposed to only be one instance of the D GC running per process. If you have more than one running then either you aren't linking and loading the DLLs correctly, or you have run into a serious bug in the D tooling. Or my debugger lied to me. I have found the gdb debugger on Linux often breaks horribly on my D code, especially when it is multi-threaded, and the debugger is only semi-usable. Maybe the Windows debugger is better now? (I wouldn't know, since I haven't used it in a while.) I think skepticism is warranted here.
Re: F*cked by memory corruption after assiging value to associative array
On Wednesday, 27 January 2021 at 17:41:05 UTC, Ali Çehreli wrote: On 1/26/21 6:31 AM, frame wrote: > all GCs Multiple D runtimes? That might work I guess but I've never heard of anybody talking about having multiple runtimes. Does rt_init() initialize *a* D runtime or *the* D runtime? If it indeed works we definitely need much better documentation. I load my libraries with loadLibrary[1] so that "[if] the library contains a D runtime it will be integrated with the current runtime." Ali [1] https://dlang.org/library/core/runtime/runtime.load_library.html I have no idea if there are multiple runtimes. I just use the mixin SimpleDllMain. But there must be multiple instances of GCs running because 1) command line argument --DRT-gcopt=gc:manual was seen by the EXE but ignored by the DLL and still crashed 2) after "burning in" gc:manual in the DLL, observing GC.profileStats.numCollections shows in one DLL thread 0 and the other DLL thread > 0 and thus crashed. Or my debugger lied to me. I also use loadLibrary.
Re: F*cked by memory corruption after assiging value to associative array
On 1/26/21 6:31 AM, frame wrote: > all GCs Multiple D runtimes? That might work I guess but I've never heard of anybody talking about having multiple runtimes. Does rt_init() initialize *a* D runtime or *the* D runtime? If it indeed works we definitely need much better documentation. I load my libraries with loadLibrary[1] so that "[if] the library contains a D runtime it will be integrated with the current runtime." Ali [1] https://dlang.org/library/core/runtime/runtime.load_library.html
Re: F*cked by memory corruption after assiging value to associative array
On Tuesday, 26 January 2021 at 14:31:58 UTC, frame wrote: but why does not see DLL 1 then that sub objects of B are still alive? I may fool myself but could it be caused by an already gone slice data? It very looks like that only a specific string property is corrupted which got the same slice data as an input parameter. I thought that the slice data should stay referenced in the persistent object anyway but the GC seems not so smart to detect this. The error can be prevented with .dup so far.
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 17:11:37 UTC, frame wrote: Wrong way? Please, someone correct me if I'm getting this wrong: Structure: EXE/Main Thread: - GC: manual - requests DLL 1 object A - GC knows about object A DLL/Thread 1: - GC: conservative - allocates new object A -> addRoot(object A), return to EXE (out param) - requests DLL 2 object B - GC knows about object A and object B - requests sub objects of object B later DLL/Thread 2: - GC: manual - allocates new object B -> addRoot(object B), return to DLL 1 (out param) - GC knows about object B - allocates sub objects over object B when DLL 1 requests it, return to DLL 1 (out param) - sub objects are stored in object B - object B sub objects memory gets corrupted after DLL 1 becomes active thread again In this scenario only DLL 1 can cause the corruption as it does not occur if all GCs are set to manual. At this point I am confused about how memory allocation is ensured. Each thread should have assigned its own memory area. Each GC adopts the root by the returned object and knows about that area too. But if DLL 1 becomes active it writes into sub memory of DLL 2. It only can because it has adopted the root of object B - but why does not see DLL 1 then that sub objects of B are still alive?
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 16:54:42 UTC, vitamin wrote:
On Monday, 25 January 2021 at 16:44:40 UTC, frame wrote:
On Monday, 25 January 2021 at 16:14:05 UTC, vitamin wrote:
[...]
Yes with simple new operator.
Forgot to mention: the DLL itself calls a DLL.
[...]
If there are separated GCs for DLLs, then you must call
GC.addRoot() from DLL where you allocate that object.
Yes, I directly calling on every function that returns an object:
T fun(T)(T object) {
GC.addRoot(cast(void*) object);
}
...
extern (C) export SomeObject bar() {
return fun(new SomeObject);
}
Wrong way?
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 16:44:40 UTC, frame wrote: On Monday, 25 January 2021 at 16:14:05 UTC, vitamin wrote: [...] Yes with simple new operator. Forgot to mention: the DLL itself calls a DLL. [...] If there are separated GCs for DLLs, then you must call GC.addRoot() from DLL where you allocate that object.
Re: F*cked by memory corruption after assiging value to associative array
On 1/25/21 8:14 AM, vitamin wrote:
> If created on the default way mean allocated with new (=> GC)
I had the same thought. The following would be the "default way" for me
but passing that object's address to addRoot would be wrong:
import core.memory;
struct S {
}
void main() {
auto a = S();
GC.addRoot(); // Wrong: 'a' is on the stack
}
I'm pretty sure frame knows this but still...
Ali
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 16:14:05 UTC, vitamin wrote: If created on the default way mean allocated with new (=> GC) then I don't known where is problem, but if the object is allocated with other way, for example malloc, some allocator then you need tell GC about that object with GC.addRange. Yes with simple new operator. Forgot to mention: the DLL itself calls a DLL. I made following observation, don't know if it makes any sense: Assuming that executable and DLL have their own GC instance, it seems that the --DRT option is ignored by DLLs or they do not see the command line, idk. But using extern (C) __gshared string[] rt_options = ["gcopt=gc:manual"]; in main and DLLs, keeps the memory intact. Setting gc:conservative on a DLL corrupts again. Also GC.profileStats.numCollections says that there are cycles when the memory gets corrupted. So that means that GC.addRoot() isn't "global" and must be called after the DLL function has returned the object to adapt it or within the DLL itself? or both?
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 15:46:15 UTC, frame wrote: On Monday, 25 January 2021 at 14:34:23 UTC, vitamin wrote: Is the object returned from dll GC allocated? The object is created on the default way. No alternating allocation. Before the object is returned it's added to GC.addRoot() which should be enough but may I'm wrong. I also tried to add each object that holds the string member with GC.addRoot(). If created on the default way mean allocated with new (=> GC) then I don't known where is problem, but if the object is allocated with other way, for example malloc, some allocator then you need tell GC about that object with GC.addRange.
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 14:34:23 UTC, vitamin wrote: Is the object returned from dll GC allocated? The object is created on the default way. No alternating allocation. Before the object is returned it's added to GC.addRoot() which should be enough but may I'm wrong. I also tried to add each object that holds the string member with GC.addRoot().
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 13:44:52 UTC, frame wrote: On Monday, 25 January 2021 at 11:25:56 UTC, FeepingCreature wrote: I suspect the memory used by the original data got reused for the associative array somehow. But if the GC is off from program start, that should really not occur. Do you maybe turn the GC off before the AA assignment, but after it's already marked that memory freed? Try turning it off completely from the commandline with --DRT-gcopt=gc:manual With that option the bytes behind the strings look different, so it has an impact. But sadly it does not help. - I have comment out all GC.free(). - GC.disable() is called in main. - GC.profileStats.numCollections is 0. Is the object returned from dll GC allocated?
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 11:25:56 UTC, FeepingCreature wrote: I suspect the memory used by the original data got reused for the associative array somehow. But if the GC is off from program start, that should really not occur. Do you maybe turn the GC off before the AA assignment, but after it's already marked that memory freed? Try turning it off completely from the commandline with --DRT-gcopt=gc:manual With that option the bytes behind the strings look different, so it has an impact. But sadly it does not help. - I have comment out all GC.free(). - GC.disable() is called in main. - GC.profileStats.numCollections is 0.
Re: F*cked by memory corruption after assiging value to associative array
On Monday, 25 January 2021 at 11:15:28 UTC, frame wrote: After a while my program crashes. I'm inspecting in the debugger that some strings are overwritten after a struct is assigned to an associative array. - I have disabled the GC. - All happens in the same thread. - The strings belong to an object collection inside an object created from a d-DLL. - The object returned by the DLL function is added to the GC with GC.addRoot(). - This object also lives in a static array the whole time. - Not all objects are affected but many. - The struct itself looks okay also the key for the associative array has normal form. The data is not overwritten by another Thread (only one is running) but by the compiler. I'm watching it by memory location. It gets always visible first after that assignment. But how is this even possible? In theory, how could I ran into this issue? That should really not be possible. I suspect the memory used by the original data got reused for the associative array somehow. But if the GC is off from program start, that should really not occur. Do you maybe turn the GC off before the AA assignment, but after it's already marked that memory freed? Try turning it off completely from the commandline with --DRT-gcopt=gc:manual
F*cked by memory corruption after assiging value to associative array
After a while my program crashes. I'm inspecting in the debugger that some strings are overwritten after a struct is assigned to an associative array. - I have disabled the GC. - All happens in the same thread. - The strings belong to an object collection inside an object created from a d-DLL. - The object returned by the DLL function is added to the GC with GC.addRoot(). - This object also lives in a static array the whole time. - Not all objects are affected but many. - The struct itself looks okay also the key for the associative array has normal form. The data is not overwritten by another Thread (only one is running) but by the compiler. I'm watching it by memory location. It gets always visible first after that assignment. But how is this even possible? In theory, how could I ran into this issue?
