New issue 2434: Support pybind11 in conjunction with PyPy's cpyext: issue with heap types https://bitbucket.org/pypy/pypy/issues/2434/support-pybind11-in-conjunction-with-pypys
Wenzel Jakob: Hi, I'm the maintainer of [pybind11](https://github.com/pybind/pybind11), which is a modern library for creating bindings to C++ code (similar in spirit to Boost.Python). Hearing of the cpyext improvements in the latest release, I tried to see if pybind11 would work with the supported CPython API in PyPy. This would be huge, since it immediately make many projects available in PyPy. The good news: I just had to make a few minor changes to get a few simple examples to compile. This is all in the WIP branch in pybind11 here: https://github.com/pybind/pybind11/pull/521 The bad news: I'm running into fundamental issues involving creation of types. To reproduce: ``` $ git clone https://github.com/wjakob/pybind11 $ cd pybind11 $ cat > test.cpp <<EOF #include <pybind11/pybind11.h> namespace py = pybind11; PYBIND11_PLUGIN(test) { py::module m("test"); struct Test { }; py::class_<Test>(m, "Test"); return m.ptr(); } EOF $ g++ test.cpp -g3 -std=c++11 -I <path-to-pypy-include> -I include -undefined dynamic_lookup -shared -rdynamic -o test.so ``` Loading the module from PyPy leads to the following error: ```bash $ pypy Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 13 2016, 01:57:41) [PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>> import cpyext >>>> pyext.load_module('test.so', 'test') Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: global name 'pyext' is not defined >>>> cpyext.load_module('test.so', 'test') Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: TypeError: can't set attributes on type object 'Test' ``` The error message is thrown when pybind11 basically calls (via CPython API) ```python setattr(Test, '__module__', 'test') ``` Digging into the PyPy source code reveals the following source of this error message: ```python def setdictvalue(self, space, name, w_value): if not self.is_heaptype(): raise oefmt(space.w_TypeError, "can't set attributes on type object '%N'", self) ``` So PyPy believes that ``Test`` is not a heap type. Which is confusing, because ``cpyext`` believes it is. The CPython API calls boil down to: ```cpp PyHeapTypeObject *type = ((PyHeapTypeObject*) PyType_Type.tp_alloc(&PyType_Type, 0) type->ht_type.tp_flags |= Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HEAPTYPE; .. fill out remaining type fields like name, docs, ... PyType_Ready(&type->ht_type); PyObject_SetAttrString(type->ht_type, "__module__", ...); // BOOM, this fails ``` Interestingly, the type instance returned by ``tp_alloc`` *already* has the heap type flag set, and it is still set after the ``PyType_Ready`` call. I can only assume that PyPy has its own heap type flag that somehow isn't synchronized with the types created by cpyext. _______________________________________________ pypy-issue mailing list [email protected] https://mail.python.org/mailman/listinfo/pypy-issue
