Re: threading issues with D - C - Python
On 12/07/2014 03:12 PM, Michael wrote: now to figure out how to use them in the general case. This is great.. Thank you. I'm looking forward to being able to try the finished result. My build servers are broken at the moment, but I think I have this fixed, on linux at least.
Re: threading issues with D - C - Python
On Monday, 8 December 2014 at 01:17:16 UTC, Ellery Newcomer wrote: On 12/07/2014 03:12 PM, Michael wrote: On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote: On 12/04/2014 10:55 PM, Ellery Newcomer wrote: I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case. This is great.. Thank you. I'm looking forward to being able to try the finished result. It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered. Actually, I suppose with a thread local flag the latter wouldn't be too bad. Mind if I incorporate your example into pyd's test suite? Not at all. Go for it.
Re: threading issues with D - C - Python
On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote: On 12/04/2014 10:55 PM, Ellery Newcomer wrote: I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case. This is great.. Thank you. I'm looking forward to being able to try the finished result.
Re: threading issues with D - C - Python
On 12/07/2014 03:12 PM, Michael wrote: On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote: On 12/04/2014 10:55 PM, Ellery Newcomer wrote: I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case. This is great.. Thank you. I'm looking forward to being able to try the finished result. It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered. Actually, I suppose with a thread local flag the latter wouldn't be too bad. Mind if I incorporate your example into pyd's test suite?
Re: threading issues with D - C - Python
On 12/04/2014 10:55 PM, Ellery Newcomer wrote: I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. by gum, thread_attachThis and thread_detachThis fix the issue! now to figure out how to use them in the general case.
Re: threading issues with D - C - Python
On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote: dustmite? Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors. I manually deleted a reasonable chunk of the code and I'm left with these files which still seem to cause segfaults: Main code: http://pastebin.com/zqgNTk9w PyD definitions: http://pastebin.com/6mRH3KZZ setup.py: http://pastebin.com/i9Ph78UC test code that causes segfaults: http://pastebin.com/1ukzShVh Cheers, Michael.
Re: threading issues with D - C - Python
On Thu, Dec 04, 2014 at 10:11:53PM +, Michael via Digitalmars-d-learn wrote: On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote: dustmite? Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors. [...] Nothing is wrong with dustmite; one of its standard techniques to reduce code is to delete source files and see if the problem still happens (i.e., it's independent of that file). What you need to do is to craft a test script such that only the specific error you're looking for will return a success status, and everything else won't. Then when dustmite deletes an essential file, it will know that that was a wrong step and backtrack. T -- Food and laptops don't mix.
Re: threading issues with D - C - Python
On 12/04/2014 02:11 PM, Michael wrote: On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote: dustmite? Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors. I manually deleted a reasonable chunk of the code and I'm left with these files which still seem to cause segfaults: Main code: http://pastebin.com/zqgNTk9w PyD definitions: http://pastebin.com/6mRH3KZZ setup.py: http://pastebin.com/i9Ph78UC test code that causes segfaults: http://pastebin.com/1ukzShVh Cheers, Michael. hmm.. looks like here it originates in python when it tries to acquire the GIL. specifically, pthread_cond_timedwait is segfaulting. in your code, execution inside a python thread makes it to receiveTimeout in get_image. it made it past receiveTimeout in acquire. then I guess there is a context switch. the main python thread throws an exception, but a number of things trigger the segfault. I think it's just the interpreter loop calling RestoreThread. backtrace looks like #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x003799d07bb3 in PyCOND_TIMEDWAIT (cond=0x379a063220 gil_cond, mut=0x379a0631e0 gil_mutex, us=5000) at /usr/src/debug/Python-3.3.2/Python/condvar.h:103 #2 take_gil (tstate=tstate@entry=0x604410) at /usr/src/debug/Python-3.3.2/Python/ceval_gil.h:224 #3 0x003799d081fb in PyEval_RestoreThread (tstate=tstate@entry=0x604410) ... It looks like this is the python main thread. I see two other threads. (i took out one of your python spawns) #2 looks to be your listener thread. std.concurrency.send seems to have gotten it into a gc_malloc, but it looks like it's just waiting. it's currently in sem_wait. this one would have been spawned in D code by #3 #3 is your other python thread. it is also in pthread_cond_timedwait. by its stack trace, receiveTimeout is just waiting. I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. not exactly my area of expertise, this.
Re: threading issues with D - C - Python
On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer wrote: are you looking at this pyd: https://bitbucket.org/ariovistus/pyd I'm looking at this one, which is what came up when googling python to D http://pyd.dsource.org/
Re: threading issues with D - C - Python
On Wednesday, 3 December 2014 at 06:30:07 UTC, Russel Winder via Digitalmars-d-learn wrote: As far as I can tell PyD is still active, but in a non-funded FOSS way, i.e. work happens as and when volunteers put time and effort in. I haven't tried PyD recently but it worked fine last time I did. If can set out what you tried and what didn't work, maybe there is a PyD solution, or a fix to PyD to give a solution? Yeah apparently I might have used the wrong PyD. This might end up being all I need to do to fix my problem. D's big problem is shared objects/dynamic link libraries. Without them you cannot interwork with Python at all. I have tried experiments on Linux creating shared libraries from D code with C linkage entry points to create classic Python extensions, and it appears to work fine. Except for having to start up the D heap and thread management, should they be needed. But that is what PyD is there for. If I took my experiments any further I would end up recreating PyD or something like it. This is what I'm doing. I'm using the rt_init() function to setup the heap/thread management. Am I missing anything else here? It seems the issue is definitely more complicated than that. I'll also point out that: http://dlang.org/interfaceToC had no information on calling rt_init first, although it seems like it should. It sounds like you are in a similar situation except that you appear to have an extra layer of C code. I am not sure a layer of C is needed between Python and D, it would be good to know more about why you seem to need it. Well I wanted to compile D code and directly call it in Python, and since PyD didn't work for me, I instead tried the python- C interface I already knew (https://docs.python.org/2/c-api/index.html) and worked my way through a C-D interface, which I understood to be relatively simple. My guess would be not properly initializing the D infrastructure from the incoming Python thread. I would suggest that you want to avoid threads crossing the boundaries and just pass data via a shared channel. Unix pipes seem to work well in this context since they provide a language independent data channel. Yeah I'm leaning in that direction myself, although I might try the other PyD library first. I wanted to be able to use the D message-passing libraries to do the thread-safety stuff at first, because it was much easier than the alternative, but I'm not sure that's true anymore.
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 20:41:46 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer wrote: are you looking at this pyd: https://bitbucket.org/ariovistus/pyd I'm looking at this one, which is what came up when googling python to D http://pyd.dsource.org/ ah, dsource strikes back! that vile site keep biting us again and again. let's hope that new admins will kill it for good. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via Digitalmars-d-learn wrote: ah, dsource strikes back! that vile site keep biting us again and again. let's hope that new admins will kill it for good. Yeah. I've got the new PyD and it compiles and does everything I want much nicer, but it appears to have the exact same problems. When calling a python thread to my code, it can cause segfaults and hanging issues. Cheers, Michael.
Re: threading issues with D - C - Python
On 12/03/2014 04:43 PM, Michael wrote: On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via Digitalmars-d-learn wrote: ah, dsource strikes back! that vile site keep biting us again and again. let's hope that new admins will kill it for good. Yeah. I've got the new PyD and it compiles and does everything I want much nicer, but it appears to have the exact same problems. When calling a python thread to my code, it can cause segfaults and hanging issues. Cheers, Michael. okay. that's not too surprising. If you can get me a minimal example, I'd be happy to have a look since pyd should probably support this case.
Re: threading issues with D - C - Python
On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer wrote: okay. that's not too surprising. If you can get me a minimal example, I'd be happy to have a look since pyd should probably support this case. Cool. Unfortunately most of the times I've attempted to reduce this down it always seems to work, but I think that's because I often did the example code in D. I'll play around with it and try to send you an example. Cheers, Michael.
Re: threading issues with D - C - Python
On 12/03/2014 06:56 PM, Michael wrote: On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer wrote: okay. that's not too surprising. If you can get me a minimal example, I'd be happy to have a look since pyd should probably support this case. Cool. Unfortunately most of the times I've attempted to reduce this down it always seems to work, but I think that's because I often did the example code in D. I'll play around with it and try to send you an example. Cheers, Michael. dustmite?
threading issues with D - C - Python
Hi. I'm new here and this is my first post. I'm not sure this is the right subforum for it, but wasn't sure where else to put it either. I've written a library to talk to some external hardware using a socket. It uses the std.concurrency threads to send messages between the main D-object for the hardware and the D-object for the sockets. I then wanted to be able to call these functions from Python. PyD appeared to be out of date, so I've been using a D - C interface, and a C - Python interface. The python code will often run from different python threads, so I then added yet another message-passing layer between the D-C interface and the D-hardware interface. My problem is that this code routinely causes segmentation faults. I've spent a long time going through trying to figure out exactly what the causes are. I think there have been some related to D-exceptions not being handled gracefully by the C/Python code. Some more by stdout writing from multiple threads (which surprised me). I'm fairly sure I have tackled both of these issues, but it still seems like Python threads and D threads don't mix well. When running the same functions from D, I am able to get no errors, but when run from Python/C it causes segfaults reliably. Sorry for the large exposition. I am currently at the point of suspecting bugs in Phobos, but I am unskilled enough to tell for sure, and would appreciate any help. The latest core dump gives a backtrace of almost entirely phobos commands: #0 0x7fe789ad3b97 in gc.gc.Gcx.fullcollect() () from /lib/libphobos2.so.0.66 #1 0x7fe789ad3294 in gc.gc.Gcx.bigAlloc() () from /lib/libphobos2.so.0.66 #2 0x7fe789ad0df1 in gc.gc.GC.mallocNoSync() () from /lib/libphobos2.so.0.66 #3 0x7fe789ad0c15 in gc.gc.GC.malloc() () from /lib/libphobos2.so.0.66 #4 0x7fe789ad6470 in gc_malloc () from /lib/libphobos2.so.0.66 #5 0x7fe789ae6d36 in _d_newitemT () from /lib/libphobos2.so.0.66 #6 0x7fe789e57112 in std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from /usr/lib/libv5camera.so #7 0x7fe789e570b5 in std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from /usr/lib/libv5camera.so #8 0x7fe789e562dc in std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from /usr/lib/libv5camera.so #9 0x7fe789e561ea in std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from /usr/lib/libv5camera.so #10 0x7fe789e5617d in std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() () from /usr/lib/libv5camera.so #11 0x7fe789e56132 in std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() () from /usr/lib/libv5camera.so #12 0x7fe789e61f09 in std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() () from /usr/lib/libv5camera.so #13 0x7fe789e5b4ac in std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() () from /usr/lib/libv5camera.so #14 0x7fe789e57e8d in std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.__ctor() () from /usr/lib/libv5camera.so #15 0x7fe789e581f1 in std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.handler() () from /usr/lib/libv5camera.so #16 0x7fe789e57d0f in std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.opEquals() () from /usr/lib/libv5camera.so #17 0x7fe789e57ba8 in std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from /usr/lib/libv5camera.so #18 0x7fe789e62087 in std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() () from /usr/lib/libv5camera.so #19 0x7fe789e621a3 in std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() () from /usr/lib/libv5camera.so #20 0x7fe789e5b7f6 in std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() () from /usr/lib/libv5camera.so #21 0x7fe789ac7d51 in core.thread.Thread.run() () from /lib/libphobos2.so.0.66 #22 0x7fe789ac6f95 in thread_entryPoint () from /lib/libphobos2.so.0.66 #23 0x7fe79cee5182 in start_thread (arg=0x7fe77aca5700) at pthread_create.c:312 #24 0x7fe79cc11fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Cheers, Michael.
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 01:07:42 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: I'm fairly sure I have tackled both of these issues, but it still seems like Python threads and D threads don't mix well. When running the same functions from D, I am able to get no errors, but when run from Python/C it causes segfaults reliably. you are right, D threads and other language/library threads aren't mix well. at least you have to use `thread_attachThis()` and `thread_detachThis()` from core.threads module to make sure that GC is aware of alien threads. and i assume that calling this functions from python will not be very easy. but it's better to not mix 'em at all if it is possible. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
Thanks for this. Its definitely a step in the right direction. Would you mind explaining a bit more about the problem here, if you can? I don't fully understand why the garbage collector needs to know about the threads, and if so for how long does it need to know? If I put in thread_attachThis();scope(exit)thread_detachThis(); it doesn't appear to fix my problems, so I'm definitely curious as to what is going on under the hood. Cheers, Michael. On Wednesday, 3 December 2014 at 01:17:43 UTC, ketmar via Digitalmars-d-learn wrote: On Wed, 03 Dec 2014 01:07:42 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: I'm fairly sure I have tackled both of these issues, but it still seems like Python threads and D threads don't mix well. When running the same functions from D, I am able to get no errors, but when run from Python/C it causes segfaults reliably. you are right, D threads and other language/library threads aren't mix well. at least you have to use `thread_attachThis()` and `thread_detachThis()` from core.threads module to make sure that GC is aware of alien threads. and i assume that calling this functions from python will not be very easy. but it's better to not mix 'em at all if it is possible.
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 02:21:45 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Thanks for this. Its definitely a step in the right direction. Would you mind explaining a bit more about the problem here, if you can? I don't fully understand why the garbage collector needs to know about the threads, and if so for how long does it need to know? If I put in thread_attachThis();scope(exit)thread_detachThis(); it doesn't appear to fix my problems, so I'm definitely curious as to what is going on under the hood. you have to call `thread_attachThis();` in alien thread, not in D thread. i.e. if you created thread from python code, you have to call `thread_attachThis();` in that python thread (i don't know how you'll do that, but you must ;-). and you must call `thread_detachThis();` from the same python thread before exiting from it. garbage collector must know about all running threads so it can scan their stacks, variables and so on. as there is no portable way to set application-wide hooks on thread creation and termination, you must inform GC about that events manually. the other thing you can do is to not use any D allocated data in alien threads. i.e. don't pass anything that was allocated by D code to python thread and vice versa. if you want to pass some data to alien thread, `malloc()` the necessary space, copy data to it and pass malloc'ed pointer. don't forget to free that data in alien thread. but i think that this is not what you really want, as it means alot of allocations and copying, and complicates the whole thing alot. alien is the thread that was created outside of D code. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
On Wednesday, 3 December 2014 at 02:41:11 UTC, ketmar via Digitalmars-d-learn wrote: On Wed, 03 Dec 2014 02:21:45 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Thanks for this. Its definitely a step in the right direction. Would you mind explaining a bit more about the problem here, if you can? I don't fully understand why the garbage collector needs to know about the threads, and if so for how long does it need to know? If I put in thread_attachThis();scope(exit)thread_detachThis(); it doesn't appear to fix my problems, so I'm definitely curious as to what is going on under the hood. you have to call `thread_attachThis();` in alien thread, not in D thread. i.e. if you created thread from python code, you have to call `thread_attachThis();` in that python thread (i don't know how you'll do that, but you must ;-). and you must call `thread_detachThis();` from the same python thread before exiting from it. garbage collector must know about all running threads so it can scan their stacks, variables and so on. as there is no portable way to set application-wide hooks on thread creation and termination, you must inform GC about that events manually. the other thing you can do is to not use any D allocated data in alien threads. i.e. don't pass anything that was allocated by D code to python thread and vice versa. if you want to pass some data to alien thread, `malloc()` the necessary space, copy data to it and pass malloc'ed pointer. don't forget to free that data in alien thread. but i think that this is not what you really want, as it means alot of allocations and copying, and complicates the whole thing alot. alien is the thread that was created outside of D code. Okay. Well I am already not passing any D-allocated data. I'm specifically creating variables/arrays on the C-stack, and then passing the pointer of that to D and overwriting the data of the C-stack pointer for any return values. I was worried about that specific problem and I thought this would be a solution. I am then able to tell python to use the C-stack variable without having to worry about D trying to run any garbage collection on it. Going the other way, I probably am passing some python strings etc.. into D, but I would assume they are valid for the lifetime of the function call, and that D would have no reason to try and perform any garbage collection on them.
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 02:52:27 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Okay. Well I am already not passing any D-allocated data. I'm specifically creating variables/arrays on the C-stack, and then passing the pointer of that to D and overwriting the data of the C-stack pointer for any return values. I was worried about that specific problem and I thought this would be a solution. I am then able to tell python to use the C-stack variable without having to worry about D trying to run any garbage collection on it. if D code has any pointer to that data stored anywhere, GC will walk it and hit another thread's stack. and now it doesn't know where it is, and it can't pause that uknown thread so it will not mutate the area GC is scanning now. this *may* work, but it will segfault sooner or later. Going the other way, I probably am passing some python strings etc.. into D, but I would assume they are valid for the lifetime of the function call, and that D would have no reason to try and perform any garbage collection on them. D has conservative GC, so it will try to walk with the unknown data just in case that data contains some pointers. and GC can hit false positives there (something that *looks* like a pointer to some area) and other things. so to make the long story short: you should either register and deregister *all* your threads in GC (for D threads it's automatic process; for other threads you must do it manually), or don't use GC at all. besides, if you are using your D library from C code, you must call `rt_init()` once before calling any D code. this function will initialize D runtime. and you have to call `rt_term()` before exiting your program to deinitialize D runtime. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 02:52:27 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: by using from C code i mean that your main program is not written in D, it has no D `main()` and so on. i.e. you wrote, for example, some .a library in D and now you want to use that library in C code. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 01:07:42 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: all in all, you'd better not mixing D code with alien mulththreaded code and not using .a/.so libraries written in D in another language until you are familiar with D runtime and GC. those mixes are very fragile. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
On Wed, 03 Dec 2014 01:07:42 + Michael via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: btw, Adam Ruppe's D Cookbook has a chapter which describes how to call D library from C code. don't remember if it describes threading, though. signature.asc Description: PGP signature
Re: threading issues with D - C - Python
On 12/02/2014 05:07 PM, Michael wrote: Hi. I'm new here and this is my first post. I'm not sure this is the right subforum for it, but wasn't sure where else to put it either. I've written a library to talk to some external hardware using a socket. It uses the std.concurrency threads to send messages between the main D-object for the hardware and the D-object for the sockets. I then wanted to be able to call these functions from Python. PyD appeared to be out of date, so I've been using a D - C interface, and a C - Python interface. The python code will often run from different python threads, so I then added yet another message-passing layer between the D-C interface and the D-hardware interface. are you looking at this pyd: https://bitbucket.org/ariovistus/pyd
Re: threading issues with D - C - Python
On Wed, 2014-12-03 at 01:07 +, Michael via Digitalmars-d-learn wrote: Hi. I'm new here and this is my first post. I'm not sure this is the right subforum for it, but wasn't sure where else to put it either. I've written a library to talk to some external hardware using a socket. It uses the std.concurrency threads to send messages between the main D-object for the hardware and the D-object for the sockets. I then wanted to be able to call these functions from Python. PyD appeared to be out of date, so I've been using a As far as I can tell PyD is still active, but in a non-funded FOSS way, i.e. work happens as and when volunteers put time and effort in. I haven't tried PyD recently but it worked fine last time I did. If can set out what you tried and what didn't work, maybe there is a PyD solution, or a fix to PyD to give a solution? D - C interface, and a C - Python interface. The python code will often run from different python threads, so I then added yet another message-passing layer between the D-C interface and the D-hardware interface. D's big problem is shared objects/dynamic link libraries. Without them you cannot interwork with Python at all. I have tried experiments on Linux creating shared libraries from D code with C linkage entry points to create classic Python extensions, and it appears to work fine. Except for having to start up the D heap and thread management, should they be needed. But that is what PyD is there for. If I took my experiments any further I would end up recreating PyD or something like it. It sounds like you are in a similar situation except that you appear to have an extra layer of C code. I am not sure a layer of C is needed between Python and D, it would be good to know more about why you seem to need it. Your use case is interesting as I have more or less given up on using D in a Python context. CPython naturally requires C linkage shared objects for non-Python code so C, C++ or D would be fine except that everyone does it in C or C++, very few people in the arena have even heard of D. PyPy brings it's own issues with non-Python code but the now have a C capability. With Cython, Pythran, and more recently Numba there is increasing less and less need for any user written non-Python code. Hardware control would be done in C and PyPy now has a C linkage mechanism. Of course for Python networking there is Twisted, Asyncio and Tornado so no Python folk are using non-Python code for handling networking. My problem is that this code routinely causes segmentation faults. I've spent a long time going through trying to figure out exactly what the causes are. I think there have been some related to D-exceptions not being handled gracefully by the C/Python code. Some more by stdout writing from multiple threads (which surprised me). I'm fairly sure I have tackled both of these issues, but it still seems like Python threads and D threads don't mix well. When running the same functions from D, I am able to get no errors, but when run from Python/C it causes segfaults reliably. Without seeing your code, it is difficult to say what may or may not be the problem, but I would guess it is about D infrastructure start up. I recollect being able to reliably get segfaults this way. Are you using ctypes or CFFI on the Python side? Sorry for the large exposition. I am currently at the point of suspecting bugs in Phobos, but I am unskilled enough to tell for sure, and would appreciate any help. The latest core dump gives a backtrace of almost entirely phobos commands: #0 0x7fe789ad3b97 in gc.gc.Gcx.fullcollect() () from /lib/libphobos2.so.0.66 #1 0x7fe789ad3294 in gc.gc.Gcx.bigAlloc() () from /lib/libphobos2.so.0.66 #2 0x7fe789ad0df1 in gc.gc.GC.mallocNoSync() () from /lib/libphobos2.so.0.66 #3 0x7fe789ad0c15 in gc.gc.GC.malloc() () from /lib/libphobos2.so.0.66 #4 0x7fe789ad6470 in gc_malloc () from /lib/libphobos2.so.0.66 #5 0x7fe789ae6d36 in _d_newitemT () from /lib/libphobos2.so.0.66 #6 0x7fe789e57112 in std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from /usr/lib/libv5camera.so #7 0x7fe789e570b5 in std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from /usr/lib/libv5camera.so #8 0x7fe789e562dc in std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from /usr/lib/libv5camera.so #9 0x7fe789e561ea in std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from /usr/lib/libv5camera.so #10 0x7fe789e5617d in std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() () from /usr/lib/libv5camera.so #11 0x7fe789e56132 in std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() () from /usr/lib/libv5camera.so #12 0x7fe789e61f09 in std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() () from /usr/lib/libv5camera.so #13
