[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
On Mon, Jan 31, 2022 at 3:57 PM Petr Viktorin wrote: > Remove the point from the Abstract? > (...) > I think you should make it clear that the percentage was taken after > fixing many of the projects. Thanks for your review. I pushed a change to take your remarks in account: https://github.com/python/peps/commit/e9456773643c7dcca584ae0100db42639b93f04e I simplified the Abstract and added a new "Backward compatibility > Statistics" section to show all numbers at one place: https://python.github.io/peps/pep-0674/#statistics I also moved projects already fixed in the "Top 5000 PyPI" category or in a new "Other affected projects" category, depending if they are in the top 5000 PyPI projects or not. It should now be easier to understand statistics. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CJRZSM53LVDKJPVEEFJYUERW7B3LDF5L/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
On 31. 01. 22 15:30, Victor Stinner wrote: On Mon, Jan 31, 2022 at 2:31 PM Petr Viktorin wrote: PEP: 674 Title: Disallow using macros as l-value The current PEP is named "Disallow using Py_TYPE() and Py_SIZE() macros as l-values", which is misleading: it proposes to change 65 macros. Right, I made changes since I posted the "version 2" to python-dev 13 days ago. Since nobody replied to my thread (before you), I directly submitted the latest (updated) PEP to the Steering Council: https://github.com/python/steering-council/issues/100 I decided to put "Py_TYPE() and Py_SIZE()" in the title, since in practice, only these two macros are causing troubles. The PEP changes 65 macros in total. In the latest PEP version, exactly 2 macros are breaking projects: Py_TYPE and Py_SIZE. This is a reasonable assumption if you think that backwards-incompatible changes are OK to make if they don't affect "projects". (Presumably, projects that you know about.) If I don't agree wit that, I can't rely on the title/abstract to decide how much time I'll need to spend on the PEP :( * Allow evolving CPython internals; In the current PEP, this is now "Allow evolving CPython internals (change the PyObject structure);" But that's not really true: the PyObject won't become changeable if the PEP is accepted, since `ob_type` is part of the stable ABI, and direct access to this field is compiled into existing extensions that use the stable ABI. When I created https://bugs.python.org/issue39573, I chose the title "Make PyObject an opaque structure in the limited C API". Two years later, I changed the title to "[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type" since I understood that this project requires multiple incremental steps (and maybe changes should be done in multiple Python versions). You're right that the PEP 674 alone is not enough to fully make the structure opaque. The next interesting problem is that PyType_FromSpec() and PyType_Spec API (which are part of the stable ABI!) use indirectly sizeof(PyObject) in the PyType_Spec.basicsize member. One option to make the PyObject structure opaque would be to modify the PyObject structure to make it empty, and move its members into a new private _PyObject structure. This _PyObject structure would be allocated before the PyObject* pointer, same idea as the current PyGC_Head header which is also allocated before the PyObject* pointer. The main blocker issue is that it breaks the stable ABI. How should the PEP abstract be rephrased to be more accurate? Remove the point from the Abstract? On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 macro changes. Moreover, 24 projects just have to regenerate their Cython code to use ``Py_SET_TYPE()``. This data was gathered in 2022. Since this change was made in 2020 and then reverted because it broke too many projects (which were presumably notified of the breakage and had 2 years to update), the 0.3% is misleading. It is a measure of the current porting effort, not an estimate of the chance of a given project being affected. I tried to provide all data that I gathered. The following section list 14 projects that had to be fixed: https://www.python.org/dev/peps/pep-0674/#projects-released-with-a-fix I didn't check if they are part of the current top 5000 PyPI project or not. How should the abstract be rephrased to mention these projects? Does someone know how to check if these projects are part of the top 5000 PyPI projects? I think you should make it clear that the percentage was taken after fixing many of the projects. That change will probably make the numbers look rather meaningless. So either explain why they're meaningful, or remove the paragraph. (Somewhat unrelated: you could also put the motivation in a Motivation section, rather than the Abstract. If that leaves the abstract with just one sentence, it's good -- provided the sentence is still a good summary of the proposed changes.) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LDX6O76V7M3BWZKWXQHPJ26GODS6Z5UE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
On Mon, Jan 31, 2022 at 2:31 PM Petr Viktorin wrote: > > PEP: 674 > > Title: Disallow using macros as l-value > > The current PEP is named "Disallow using Py_TYPE() and Py_SIZE() macros > as l-values", which is misleading: it proposes to change 65 macros. Right, I made changes since I posted the "version 2" to python-dev 13 days ago. Since nobody replied to my thread (before you), I directly submitted the latest (updated) PEP to the Steering Council: https://github.com/python/steering-council/issues/100 I decided to put "Py_TYPE() and Py_SIZE()" in the title, since in practice, only these two macros are causing troubles. The PEP changes 65 macros in total. In the latest PEP version, exactly 2 macros are breaking projects: Py_TYPE and Py_SIZE. > > * Allow evolving CPython internals; > > In the current PEP, this is now "Allow evolving CPython internals > (change the PyObject structure);" > > But that's not really true: the PyObject won't become changeable if the > PEP is accepted, since `ob_type` is part of the stable ABI, and direct > access to this field is compiled into existing extensions that use the > stable ABI. When I created https://bugs.python.org/issue39573, I chose the title "Make PyObject an opaque structure in the limited C API". Two years later, I changed the title to "[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type" since I understood that this project requires multiple incremental steps (and maybe changes should be done in multiple Python versions). You're right that the PEP 674 alone is not enough to fully make the structure opaque. The next interesting problem is that PyType_FromSpec() and PyType_Spec API (which are part of the stable ABI!) use indirectly sizeof(PyObject) in the PyType_Spec.basicsize member. One option to make the PyObject structure opaque would be to modify the PyObject structure to make it empty, and move its members into a new private _PyObject structure. This _PyObject structure would be allocated before the PyObject* pointer, same idea as the current PyGC_Head header which is also allocated before the PyObject* pointer. The main blocker issue is that it breaks the stable ABI. How should the PEP abstract be rephrased to be more accurate? > > On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 > > macro changes. Moreover, 24 projects just have to regenerate their > > Cython code to use ``Py_SET_TYPE()``. > > This data was gathered in 2022. > Since this change was made in 2020 and then reverted because it broke > too many projects (which were presumably notified of the breakage and > had 2 years to update), the 0.3% is misleading. It is a measure of the > current porting effort, not an estimate of the chance of a given project > being affected. I tried to provide all data that I gathered. The following section list 14 projects that had to be fixed: https://www.python.org/dev/peps/pep-0674/#projects-released-with-a-fix I didn't check if they are part of the current top 5000 PyPI project or not. How should the abstract be rephrased to mention these projects? Does someone know how to check if these projects are part of the top 5000 PyPI projects? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BVBXIAMEH3UMDN5NWV4GLRIZ5L4RRI5U/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
On 18. 01. 22 9:30, Victor Stinner wrote: Hi, I made the following changes to the PEP since the version 1 (November 30): * Elaborate the HPy section and add a new "Relationship with the HPy project" section * Elaborate which projects are impacted and which changes are needed * Remove the PyPy section The PEP can be read online: https://python.github.io/peps/ep-0674/ And you can find the plain text below. Victor PEP: 674 Title: Disallow using macros as l-value The current PEP is named "Disallow using Py_TYPE() and Py_SIZE() macros as l-values", which is misleading: it proposes to change 65 macros. Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Nov-2021 Python-Version: 3.11 Abstract Incompatible C API change disallowing using macros as l-value to: * Allow evolving CPython internals; In the current PEP, this is now "Allow evolving CPython internals (change the PyObject structure);" But that's not really true: the PyObject won't become changeable if the PEP is accepted, since `ob_type` is part of the stable ABI, and direct access to this field is compiled into existing extensions that use the stable ABI. This PEP alone won't really help nogil. * Ease the C API implementation on other Python implementation; * Help migrating existing C extensions to the HPy API. On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 macro changes. Moreover, 24 projects just have to regenerate their Cython code to use ``Py_SET_TYPE()``. This data was gathered in 2022. Since this change was made in 2020 and then reverted because it broke too many projects (which were presumably notified of the breakage and had 2 years to update), the 0.3% is misleading. It is a measure of the current porting effort, not an estimate of the chance of a given project being affected. In practice, the majority of affected projects only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. I'll post my counterproposal, please tell me what's wrong with it (and ideally add it to Rejected ideas): - document that these macros should not be used as l-values - and that's it for CPython. When this is done, interested people can try compiling projects with HPy, and send fixes to various projects. The above documentation should be a strong reason to accept the PRs. And finding the issue is relatively easy -- compile with modified CPython headers. Meanwhile, projects that don't care about HPy (yet), or projects that want to keep working with minimum maintenance, can do the change on their own timeline. If and when the time comes and the proposed change is actually beneficial to all/most users in the short term (for example it is needed for some major performance improvement or nogil), or we can make it opt-in, or it becomes one of a few things that prevent CPython from switching to HPy, then we can do the change. Within one release, even. The PEP doesn't propose any deprecation period because it's impractical to produce compiler warnings; IMO we should handle this by having a long "docs-only deprecation" period, rather than not having a deprecation period at all. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TEKRMXAKLFTWSZBFOXIK35TAKWMWQISA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
On Thu, Jan 27, 2022 at 1:14 AM Larry Hastings wrote: > Just so I understand: is this effectively permanent? My first thought was, > we get SWIG to change its code generation, and in a couple of years we can > follow up and make this change. But perhaps there's so much SWIG code out > there already, and the change would propagate so slowly, that it's not even > worth considering. Is it something like that? Or is it just that it's such > a minor change that it's not worth fighting about? I'm interested to be able to change Python core structures like PyObject, PyTypeObject or PyListObject. It's just that PyDescrObject is out of my scope for now. As you explained, changing SWIG is not free, so it's better to have a good reason to do it :-) If someone wants to do it, I already did the work, I attached patches to: https://bugs.python.org/issue46538 -- About generated code, there is a similar concern with Cython. Hopefully, it seems like maintainers do regenerate the C code of their project using a recent Cython when they release a new version of their project (ex: numpy). So for Cython, fixing Cython is enough, then it's just a matter of time. In the case of the PEP 674, I already updated Cython in May 2020, and Cython is already released with my changes ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CTQU7LNZF2FXA34XCWIRXJHUYEKFXCAF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
On 1/26/22 3:02 PM, Victor Stinner wrote: Hi, My PEP 674 proposed to change PyDescr_TYPE() and PyDescr_NAME() macros. This change breaks M2Crypto and mecab-python3 projects in code generated by SWIG. I tried two solutions to prevent SWIG accessing PyDescrObject members directly: https://bugs.python.org/issue46538 At the end, IMO it's too much work, whereas there is no need in the short term to modify the PyDescrObject structure, or structure inheriting from PyDescrObject. So I excluded PyDescr_TYPE() and PyDescr_NAME() macros from PEP 674 to leave them unchanged: https://python.github.io/peps/pep-0674/#pydescr-name-and-pydescr-type-are-left-unchanged Just so I understand: is this effectively permanent? My first thought was, we get SWIG to change its code generation, and in a couple of years we can follow up and make this change. But perhaps there's so much SWIG code out there already, and the change would propagate so slowly, that it's not even worth considering. Is it something like that? Or is it just that it's such a minor change that it's not worth fighting about? //arry/ p.s. thanks for doing this! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KQD4MW2DIFQYDYP25Z6A5XHQHZW75NEV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)
Hi, My PEP 674 proposed to change PyDescr_TYPE() and PyDescr_NAME() macros. This change breaks M2Crypto and mecab-python3 projects in code generated by SWIG. I tried two solutions to prevent SWIG accessing PyDescrObject members directly: https://bugs.python.org/issue46538 At the end, IMO it's too much work, whereas there is no need in the short term to modify the PyDescrObject structure, or structure inheriting from PyDescrObject. So I excluded PyDescr_TYPE() and PyDescr_NAME() macros from PEP 674 to leave them unchanged: https://python.github.io/peps/pep-0674/#pydescr-name-and-pydescr-type-are-left-unchanged Today, the PEP 674 only affects 9 projects on the top 5000 PyPI projects (+ 22 projects which have to re-run Cython). Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SKZ5UTNJD3AAYIBTL6D4Y7XKGI2PZSIF/ Code of Conduct: http://python.org/psf/codeofconduct/
