On Fri, May 28, 2021 at 6:55 AM Tim Peters <tim.pet...@gmail.com> wrote: > I suppose I could ask why heap types were fiddled to point to their > module objects too - but that's really got nothing to do with getting > the release done, so I won't :-)
PyHeapTypeObject.ht_module was added by the PEP 573 "Module State Access from C Extension Methods": https://www.python.org/dev/peps/pep-0573/ It is set if you create a type using PyType_FromModuleAndSpec(). Why does it matter to store the module? Well, previously, it was not possible to unload a C extension which prevented to release memory at Python exit, and many Python objects survived to Py_Finalize(). This behavior is unpleasant or can cause issues when Python is embedded. This issue is getting worse if you consider the subinterpreter use case (also when embedding Python). See https://bugs.python.org/issue1635741: "Py_Finalize() doesn't clear all Python objects at exit". What is the relationship between unloading C extensions and PyHeapTypeObject.ht_module? Well, to be able to unload an extension, the extension must implement the multiphase initialization API (PEP 489) and it must be possible to create multiple instances of the same extension. For correctness, it becomes important that each extension has its own type instances and its own variables values. Global variables must be made per module instance. In practice, it means that global variables must be moved into a "module state": PyModule_GetState(). The new problem becomes: how can you access the module style from type methods which only get the instance in its parameter and not the module? Well, there is how PyHeapTypeObject.ht_module comes into the game: PyType_GetModule() gives you the module. Simple example: ------------ static int _abcmodule_exec(PyObject *module) { ... state->_abc_data_type = (PyTypeObject *)PyType_FromModuleAndSpec(module, &_abc_data_type_spec, NULL); if (state->_abc_data_type == NULL) { return -1; } ... } static PyObject * abc_data_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { ... state = PyType_GetModuleState(type); ... } ------------ The tp_new function of the _abc._abc_data gets the type. It retrieves the module state using PyType_FromModuleAndSpec(). This example is the simple case, since the type cannot be subclassed (it doesn't have the Py_TPFLAGS_BASETYPE flag). Otherwise, you would need the newly added _PyType_GetModuleByDef() function to find the right type in the MRO, since PyHeapTypeObject.ht_module can be different in a subclass (it can be NULL). You need to get the "defining class" to retrieve the module state related to your function. For example, for an ABC type, you would like to retrieve the ABC module state in your ABC functions. At the end, the great news is that extension modules behave as modules implemented in Python: each extension instance has its own types, variables values, internal state, etc. For example, it works as expected to "reload" an extension to get a "fresh module" for unit tests. Moreover, it is really possible to fully unload an extension. Finally, if your heap types fully implement the GC protocol (traverse, clear, GC type flag), unloading the extension is able to destroy all Python objects that it created ;-) (except of objects that you kept alive on purpose which can keep the extension partially alive ;-)) Hey, it's a funny issue! For more context about these issues and subinterpreters, see the talk that Dong-hee Na and me gave the Language Summit: * https://pyfound.blogspot.com/2021/05/the-2021-python-language-summit_16.html * https://github.com/vstinner/talks/raw/main/2021-PyconUS/subinterpreters.pdf Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-le...@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/CMASRUO54YNZFYELKNVP5MMO23CQ2CGL/ Code of Conduct: https://www.python.org/psf/codeofconduct/
