Re: building shared library from D code to import into cython
Thanks for this. I am aware of pyd and will take a look at source/build process. Any thoughts on speed in 2014 of pyd vs using cython to talk to D directly via C/C++ interface? I saw this old coment here: prabhuramachandran.blogspot.co.uk/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig predict that the D version's relative slowness might have something to do with Pyd's somewhat awful handling of arrays (at least in part). More low-level and verbose, but possibly faster code could be written to compensate for this if this is indeed the problem. However, this is not a very attractive solution. (Optimally, Pyd should be capable of directly pointing D arrays at numpy arrays, but this is not actually implemented.) On Friday, 10 October 2014 at 02:19:17 UTC, Ellery Newcomer wrote: On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc wrote: Hi. Thanks for the quick response. The -defaultlib was left around from trying all kinds of combinations of dmd and gcc. I am not used to gcc, and it will take me some time to become properly acquainted with all the options. I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
Re: building shared library from D code to import into cython
On Sunday, 12 October 2014 at 16:07:19 UTC, Laeeth Isharc wrote: Any thoughts on speed in 2014 of pyd vs using cython to talk to D directly via C/C++ interface? I saw this old coment here: pyd is basically just a convenience layer on top of the C interface. The part that would most likely screw with performance would be conversion from D objects to python objects and vice versa since that normally does a value copy. At the time of that comment, pyd's strategy for arrays was just iterate and convert each element. Now pyd can take advantage of the buffer protocol if it is exposed (2.7+), so converting e.g. numpy arrays should be just a memcopy. If you want by ref semantics instead of by value, use PydObject instead of a D type in your function signature. You can also turn off all conveniences and just use the C api with raw_only (https://bitbucket.org/ariovistus/pyd/wiki/CeleriD)
Re: building shared library from D code to import into cython
On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc wrote: Hi. Thanks for the quick response. The -defaultlib was left around from trying all kinds of combinations of dmd and gcc. I am not used to gcc, and it will take me some time to become properly acquainted with all the options. I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
building shared library from D code to import into cython
Hi.
I am trying to create a shared library in D linked against phobos
so that I may use this in a cython extension module for Python.
Ultimately I would like to be able to use a D class or struct
(via the C++ interface) and call it from within cython, since
cython classes cannot be instantiated without the gil (and this
prevents easy parallelisation).
I feel a bit foolish asking the question as there is a nice
example here for working with plain C using dmd as the linker,
and using dmd and gcc to create a DMD shared library statically
linked to phobos. However, I have not succeeded in creating a D
library statically linked to phobos that works with cython and
python,
http://dlang.org/dll-linux.html#dso7
I tried it first with test C code to make sure I am able to get
the C library/cython/Python interaction working.
pytest.c:
#include stdio.h
long pytest(long a)
{
return a+1;
}
int main()
{
long a =pytest(100);
printf(%ld,a);
return 0;
}
pytestpy.pyx:
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
command line:
gcc -shared -o libpytest.so pytest.o
python setup.py build_ext -i
copied libpytest.so to /usr/local/lib
python
import pytestpy
pytestpy.pytestpy()
it works
now try pytest.d
import std.stdio;
extern (C) long pytest(long a)
{
return a*2;
}
void main()
{
auto a =pytest(100);
writefln(%d,a);
}
command line:
rm pytestd.o
rm libpytest.so
rm /usr/local/lib/libpytest.so
dmd -c pytest.d -fPIC
gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so
-L-rpath=/usr/local/lib
python
import pytestpy
Traceback (most recent call last):
File stdin, line 1, in module
ImportError: /usr/local/lib/libpytest.so: undefined symbol:
_D3std5stdio12__ModuleInfoZ
I guess it is not linking to the D runtime, but I am not sure
what I should be doing to fix.
Any thoughts appreciated.
(The next step I was going to try when this works was C++
interface vs importing as a Cython class, but I thought best to
start simple).
I am running this on 64 bit Fedora 20.
Thanks.
Laeeth.
Re: building shared library from D code to import into cython
On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:
Hi.
I am trying to create a shared library in D linked against
phobos so that I may use this in a cython extension module for
Python. Ultimately I would like to be able to use a D class or
struct (via the C++ interface) and call it from within cython,
since cython classes cannot be instantiated without the gil
(and this prevents easy parallelisation).
I feel a bit foolish asking the question as there is a nice
example here for working with plain C using dmd as the linker,
and using dmd and gcc to create a DMD shared library statically
linked to phobos. However, I have not succeeded in creating a
D library statically linked to phobos that works with cython
and python,
http://dlang.org/dll-linux.html#dso7
I tried it first with test C code to make sure I am able to get
the C library/cython/Python interaction working.
pytest.c:
#include stdio.h
long pytest(long a)
{
return a+1;
}
int main()
{
long a =pytest(100);
printf(%ld,a);
return 0;
}
pytestpy.pyx:
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
command line:
gcc -shared -o libpytest.so pytest.o
python setup.py build_ext -i
copied libpytest.so to /usr/local/lib
python
import pytestpy
pytestpy.pytestpy()
it works
now try pytest.d
import std.stdio;
extern (C) long pytest(long a)
{
return a*2;
}
void main()
{
auto a =pytest(100);
writefln(%d,a);
}
command line:
rm pytestd.o
rm libpytest.so
rm /usr/local/lib/libpytest.so
dmd -c pytest.d -fPIC
gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so
-L-rpath=/usr/local/lib
python
import pytestpy
Traceback (most recent call last):
File stdin, line 1, in module
ImportError: /usr/local/lib/libpytest.so: undefined symbol:
_D3std5stdio12__ModuleInfoZ
I guess it is not linking to the D runtime, but I am not sure
what I should be doing to fix.
Any thoughts appreciated.
(The next step I was going to try when this works was C++
interface vs importing as a Cython class, but I thought best to
start simple).
I am running this on 64 bit Fedora 20.
Thanks.
Laeeth.
Since when does gcc have a -defaultlib option?
---
$ man gcc | grep defaultlib
object-file-name -llibrary -nostartfiles -nodefaultlibs
This is useful when you use -nostdlib or
-nodefaultlibs but you do
-nodefaultlibs is used.
-nodefaultlibs
-nodefaultlibs is libgcc.a, a library of internal
subroutines which
-nodefaultlibs you should usually specify -lgcc as
well. This
---
and why do you have a main function when compiling a shared
library?
Anyway, I think the problem is that libpytest.so can't find
libphobos2.so. Try adding libphobos2.so's location
(/usr/lib/x86_64-linux-gnu/libphobos2.so on my linux mint) to
your -rpath .
Re: building shared library from D code to import into cython
Hi.
Thanks for the quick response.
The -defaultlib was left around from trying all kinds of
combinations of dmd and gcc. I am not used to gcc, and it will
take me some time to become properly acquainted with all the
options.
I simply could not get it to recognize libphobos no matter what
path settings I tried passing.
http://forum.dlang.org/thread/k3vfm9$1tq$1...@digitalmars.com?page=4
I don't know whether this was necessary, but I did manage to get
it to work after looking at a thread from a couple of years back.
I recompiled the D runtime and Phobos with position independent
code (PIC) and pointed everything to that version instead, and it
worked fine. (Just export PIC=True, rename the posix.mak to
Makefile, change the DMD path to /usr/sbin/dmd - in my case, and
it all worked).
Next step is to try calling D interface from Cython.
Laeeth
In case anyone else should struggle with this in future.
Makefile here:
PHOBOS_PATH=/opt/dmd2/src/phobos/generated/linux/release/64
pytestpy.so: pytest.d pytestpy.o
dmd -fPIC -c pytest.d #-shared -defaultlib=libphobos2.so
-L-rpath=.:${PHOBOS_PATH}
gcc -shared pytest.o pytestpy.o -o pytest.so -lphobos2 -lpthread
-lrt -L. -L${PHOBOS_PATH} -Wl,-rpath=.:${PHOBOS_PATH} -o
pytestpy.so -defaultlib=libphobos2.so -L-rpath=.:${PHOBOS_PATH}
pytest.o: %.d
dmd -c $
pytestpy.o: pytestpy.pyx
cython pytestpy.pyx
gcc -fPIC -c pytestpy.c -o pytestpy.o -I/usr/include/python2.7
clean:
rm -f pytestpy.c pytest.o pytestpy.o pytest.so
pytest.d
extern (C) long pytest(long a)
{
return a*2;
}
pytestpy.pyx
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
print pytest(102)
setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
On Tuesday, 7 October 2014 at 22:46:09 UTC, Freddy wrote:
On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:
Hi.
I am trying to create a shared library in D linked against
phobos so that I may use this in a cython extension module for
Python. Ultimately I would like to be able to use a D class
or struct (via the C++ interface) and call it from within
cython, since cython classes cannot be instantiated without
the gil (and this prevents easy parallelisation).
I feel a bit foolish asking the question as there is a nice
example here for working with plain C using dmd as the linker,
and using dmd and gcc to create a DMD shared library
statically linked to phobos. However, I have not succeeded in
creating a D library statically linked to phobos that works
with cython and python,
http://dlang.org/dll-linux.html#dso7
I tried it first with test C code to make sure I am able to
get the C library/cython/Python interaction working.
pytest.c:
#include stdio.h
long pytest(long a)
{
return a+1;
}
int main()
{
long a =pytest(100);
printf(%ld,a);
return 0;
}
pytestpy.pyx:
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
command line:
gcc -shared -o libpytest.so pytest.o
python setup.py build_ext -i
copied libpytest.so to /usr/local/lib
python
import pytestpy
pytestpy.pytestpy()
it works
now try pytest.d
import std.stdio;
extern (C) long pytest(long a)
{
return a*2;
}
void main()
{
auto a =pytest(100);
writefln(%d,a);
}
command line:
rm pytestd.o
rm libpytest.so
rm /usr/local/lib/libpytest.so
dmd -c pytest.d -fPIC
gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so
-L-rpath=/usr/local/lib
python
import pytestpy
Traceback (most recent call last):
File stdin, line 1, in module
ImportError: /usr/local/lib/libpytest.so: undefined symbol:
_D3std5stdio12__ModuleInfoZ
I guess it is not linking to the D runtime, but I am not sure
what I should be doing to fix.
Any thoughts appreciated.
(The next step I was going to try when this works was C++
interface vs importing as a Cython class, but I thought best
to start simple).
I am running this on 64 bit Fedora 20.
Thanks.
Laeeth.
Since when does gcc have a -defaultlib option?
---
$ man gcc | grep defaultlib
object-file-name -llibrary -nostartfiles -nodefaultlibs
This is useful when you use -nostdlib or
-nodefaultlibs but you do
-nodefaultlibs is used.
-nodefaultlibs
-nodefaultlibs is libgcc.a, a library of
