Karl Nelson <nelso...@llnl.gov> added the comment:
Thanks, I will see if I can get additional diagnostics today. Otherwise I will have to start recompiling Python with diagnostic hooks. I have never had to rebuild python on windows so it may take a while to figure out how to make progress. The monkey patch is less scary than it looks. Since I only need these slots of specific basic types that I designate (not user selected), the extra types all extend without altering the underlaying type, and the class layout is controlled by a meta class so that only Java can create these type objects, it is reasonably robust. However, yes changes in Python can break it. It is basically an attempt to create a mixin for concrete classes without the benefit of a dictoffset type slot. It took about a month do develop something that safely added the slot without overrunning, leaking or creating other memory issues (and played nice when Python added its own dict slot). Unfortunately, as I do many 100s on JSlot tests during method resolution using a dict based approach was unacceptably slow. I would love if there was a formal method to create multiple inheritance mixins for Python basic types, but that is another issue. ``` /** * Allocate a new Python object with a slot for Java. * * We need extra space to store our values, but because there * is no way to do so without disturbing the object layout. * Fortunately, Python already handles this for dict and weakref. * Python aligns the ends of the structure and increases the * base type size to add additional slots to a standard object. * * We will use the same trick to add an additional slot for Java * after the end of the object outside of where Python is looking. * As the memory is aligned this is safe to do. We will use * the alloc and finalize slot to recognize which objects have this * extra slot appended. */ PyObject* PyJPValue_alloc(PyTypeObject* type, Py_ssize_t nitems ) { // Modification from Python to add size elements const size_t size = _PyObject_VAR_SIZE(type, nitems + 1) + sizeof (JPValue); PyObject *obj = (PyType_IS_GC(type)) ? _PyObject_GC_Malloc(size) : (PyObject *) PyObject_MALLOC(size); if (obj == NULL) return PyErr_NoMemory(); // GCOVR_EXCL_LINE memset(obj, 0, size); Py_ssize_t refcnt = ((PyObject*) type)->ob_refcnt; if (type->tp_itemsize == 0) PyObject_Init(obj, type); else PyObject_InitVar((PyVarObject *) obj, type, nitems); // This line is required to deal with Python bug (GH-11661) // Some versions of Python fail to increment the reference counter of // heap types properly. if (refcnt == ((PyObject*) type)->ob_refcnt) Py_INCREF(type); // GCOVR_EXCL_LINE if (PyType_IS_GC(type)) { PyObject_GC_Track(obj); } return obj; } bool PyJPValue_hasJavaSlot(PyTypeObject* type) { if (type == NULL || type->tp_alloc != (allocfunc) PyJPValue_alloc || type->tp_finalize != (destructor) PyJPValue_finalize) return false; // GCOVR_EXCL_LINE return true; } // Check for a Java slot (must work on all object types) Py_ssize_t PyJPValue_getJavaSlotOffset(PyObject* self) { PyTypeObject *type = Py_TYPE(self); if (type == NULL || type->tp_alloc != (allocfunc) PyJPValue_alloc || type->tp_finalize != (destructor) PyJPValue_finalize) return 0; Py_ssize_t offset; Py_ssize_t sz = Py_SIZE(self); // I have no clue what negative sizes mean if (sz < 0) sz = -sz; if (type->tp_itemsize == 0) offset = _PyObject_VAR_SIZE(type, 1); else offset = _PyObject_VAR_SIZE(type, sz + 1); return offset; } JPValue* PyJPValue_getJavaSlot(PyObject* self) { Py_ssize_t offset = PyJPValue_getJavaSlotOffset(self); if (offset == 0) return NULL; JPValue* value = (JPValue*) (((char*) self) + offset); if (value->getClass() == NULL) return NULL; return value; } void PyJPValue_free(void* obj) { // Normally finalize is not run on simple classes. PyTypeObject *type = Py_TYPE(obj); if (type->tp_finalize != NULL) type->tp_finalize((PyObject*) obj); if (type->tp_flags & Py_TPFLAGS_HAVE_GC) PyObject_GC_Del(obj); else PyObject_Free(obj); // GCOVR_EXCL_LINE } void PyJPValue_finalize(void* obj) { JPValue* value = PyJPValue_getJavaSlot((PyObject*) obj); ... destroy the objects hidden slot if it exists } ``` ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue42529> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com