Re: Embedding D Shared Library in WSGI Web Server
On Tuesday, 28 October 2014 at 09:34:33 UTC, Chris wrote: On Tuesday, 28 October 2014 at 00:16:03 UTC, John McFarlane wrote: Hi, I've written a modest shared library in D that I'd like to call directly from a Python web server (Linux/OS X, Apache, WSGI, Pyramid). I can call it reliably from within Python unit tests but on a running server, the library causes a SIGSEGV as soon as I try anything as complicated as a writeln call. Under Linux, I've tried using Pyd, CFFI and ctypes to bind the native .so to Python with the same result. I've tried calling Runtime.initialize();from an init function on server startup and from the D function being called as part of the web request. I've even tried writing a shim in C to make these calls and call through to the D. I've compiled with DMD and GDC with a variety of switches. Currently my build command looks like: dmd ./d/*.d -version=library -release -shared -O -inline -defaultlib=libphobos2.so -fPIC -oflibgusteau.so The extern(C) function takes a string and returns a string. I've tried holding onto a pointer to the returned string and passing in an adequately sized byte array from Python. Smaller strings seem to cause a crash with lower probability but typically the input and output strings are a few KB in size and consistently crash. I taken measures to ensure that they are correctly encoded and null terminated. I've also tried disabling the GC, calling terminate on function exit (after copying result into received buffer) and various other measures in an attempt to get something working. I'm guessing that the web server spawns/relies on a thread or process for each request and I've read that runtime initialization should be invoked from the main thread. Does this always mean the first thread that was created on process start up or does it just have to be consistent? If calling from a thread other than the one used to initialize, is that a problem? Are other threads created when GC is invoked which might last past the extern function being called and if so, can I prevent this by controlling collection explicitly? Thanks, John I had a similar problem with a DLL for Python. The reason my DLL would crash were some writeln statements. After removing them it worked fine. I suppose the writeln messes things up, be it because of threads or because of conflicts in the file system (remember writeln is a file system operation). I suggest you have your lib not print things to console, but use a different mechanism instead, e.g. pass a string back to Python and have Python do the printing (if needs be). You could also try to set your lib up as a small socket server that waits for input and answers. That usually works for me. Thanks for the suggestion. I looked into removing all of my writeln calls and found the I still had the same problem. Simply a call to to!string with the input char* is enough to cause a crash. I will go with the socket server as a last resort but it would be much easier (especially from a testing POV) if I could call the native D code directly from the web server.
Re: Embedding D Shared Library in WSGI Web Server
On Tuesday, 28 October 2014 at 00:16:03 UTC, John McFarlane wrote: Hi, I've written a modest shared library in D that I'd like to call directly from a Python web server (Linux/OS X, Apache, WSGI, Pyramid). I can call it reliably from within Python unit tests but on a running server, the library causes a SIGSEGV as soon as I try anything as complicated as a writeln call. Under Linux, I've tried using Pyd, CFFI and ctypes to bind the native .so to Python with the same result. I've tried calling Runtime.initialize();from an init function on server startup and from the D function being called as part of the web request. I've even tried writing a shim in C to make these calls and call through to the D. I've compiled with DMD and GDC with a variety of switches. Currently my build command looks like: dmd ./d/*.d -version=library -release -shared -O -inline -defaultlib=libphobos2.so -fPIC -oflibgusteau.so The extern(C) function takes a string and returns a string. I've tried holding onto a pointer to the returned string and passing in an adequately sized byte array from Python. Smaller strings seem to cause a crash with lower probability but typically the input and output strings are a few KB in size and consistently crash. I taken measures to ensure that they are correctly encoded and null terminated. I've also tried disabling the GC, calling terminate on function exit (after copying result into received buffer) and various other measures in an attempt to get something working. I'm guessing that the web server spawns/relies on a thread or process for each request and I've read that runtime initialization should be invoked from the main thread. Does this always mean the first thread that was created on process start up or does it just have to be consistent? If calling from a thread other than the one used to initialize, is that a problem? Are other threads created when GC is invoked which might last past the extern function being called and if so, can I prevent this by controlling collection explicitly? Thanks, John I had a similar problem with a DLL for Python. The reason my DLL would crash were some writeln statements. After removing them it worked fine. I suppose the writeln messes things up, be it because of threads or because of conflicts in the file system (remember writeln is a file system operation). I suggest you have your lib not print things to console, but use a different mechanism instead, e.g. pass a string back to Python and have Python do the printing (if needs be). You could also try to set your lib up as a small socket server that waits for input and answers. That usually works for me.
Embedding D Shared Library in WSGI Web Server
Hi, I've written a modest shared library in D that I'd like to call directly from a Python web server (Linux/OS X, Apache, WSGI, Pyramid). I can call it reliably from within Python unit tests but on a running server, the library causes a SIGSEGV as soon as I try anything as complicated as a writeln call. Under Linux, I've tried using Pyd, CFFI and ctypes to bind the native .so to Python with the same result. I've tried calling Runtime.initialize();from an init function on server startup and from the D function being called as part of the web request. I've even tried writing a shim in C to make these calls and call through to the D. I've compiled with DMD and GDC with a variety of switches. Currently my build command looks like: dmd ./d/*.d -version=library -release -shared -O -inline -defaultlib=libphobos2.so -fPIC -oflibgusteau.so The extern(C) function takes a string and returns a string. I've tried holding onto a pointer to the returned string and passing in an adequately sized byte array from Python. Smaller strings seem to cause a crash with lower probability but typically the input and output strings are a few KB in size and consistently crash. I taken measures to ensure that they are correctly encoded and null terminated. I've also tried disabling the GC, calling terminate on function exit (after copying result into received buffer) and various other measures in an attempt to get something working. I'm guessing that the web server spawns/relies on a thread or process for each request and I've read that runtime initialization should be invoked from the main thread. Does this always mean the first thread that was created on process start up or does it just have to be consistent? If calling from a thread other than the one used to initialize, is that a problem? Are other threads created when GC is invoked which might last past the extern function being called and if so, can I prevent this by controlling collection explicitly? Thanks, John
