[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Łukasz Langa]

Hi everybody,
I'd like to revive this thread as I feel like we have to do something here but 
some consensus is needed first.

To recap, the current state of things is as follows:
- in March 2000 (d724b23420f) Christian Tismer contributed the "trashcan" patch 
that added Py_TRASHCAN_SAFE_BEGIN and Py_TRASHCAN_SAFE_END macros which allow 
destroying nested objects non-recursively.
- in May 2019 (GH-11841 of BPO-35983) Antoine Pitrou merged a change by Jeroen 
Demeyer which made Py_TRASHCAN_SAFE_BEGIN/END (unintentionally?) backwards 
incompatible; this was released in Python 3.8.0.
- by the way, GH-11841 introduced a new pair of macros (because they have 
different signatures) called simply  Py_TRASHCAN_BEGIN and Py_TRASHCAN_END.
- by that time there was already a follow-up PR open (GH-12607) to improve 
backwards compatibility of the macros, as well as introduce tests for them; 
this was never merged.
- in Feb 2020 (0fa4f43db08) Victor Stinner removed the trashcan mechanism from 
the limited C API (note: not ABI, those are macros) since it accesses fields of 
structs not exposed in the limited C API; this was released in Python 3.9.0.
- in May 2020 Irit noticed that the backwards incompatibility (BPO-40608) 
causes segfaults for C API code that worked fine with Python 3.7. Using the new 
macros requires code changes but doesn't crash.

Now, there are a couple of things we can do here:
Option 1: Finish GH-12607 to fix the old macros, keeping in mind this will 
restore compatibility lost with Python 3.8 - 3.10 only for users of 3.11+
Option 2: Review and merge GH-20104 that reverts the macro changes that make 
old client code segfault -- unclear what else this needs and again, that would 
only fix it for users of 3.11+
Option 3: Abandon GH-12607 and GH-20104, instead declaring the old macros 
deprecated for 3.11 and remove them in 3.13

I personally agree with Irit, voting +1 for Option 3 since the old macros were 
soft-deprecated already by introducing new macros in 3.8, and more importantly 
made incompatible with pre-3.8 usage.

Let's talk on how to proceed.

- Ł



> On 26 Apr 2021, at 23:55, Irit Katriel via Python-Dev  
> wrote:
> 
> 
> Re https://bugs.python.org/issue40608 .
> 
> I think it will be an act of kindness to deprecate Py_TRASHCAN_SAFE_BEGIN/END 
> in 3.10 and tell people to use Py_TRASHCAN_BEGIN/END instead.
> 
> TL;DR:   There was a change in 3.8 that introduced the latter while leaving 
> the former for backwards compatibility, but also inadvertently breaking them. 
> This is not an easy bug to deal with in the wild, we found it because we have 
> a unit test in our codebase referencing https://bugs.python.org/issue16602 
> .  A deprecation note pointing to the new 
> macros would have made it easier.
> 
> Is there any reason not to deprecate the old macros?
> 
> Irit
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/LWP6MOLP5UW2TH3MROZQK4N64SX35N2B/
> Code of Conduct: http://python.org/psf/codeofconduct/



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TH4NILYOG3ZJO7SCCF6ECBLXJDDFJKS4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Antoine Pitrou]

On Tue, 17 Aug 2021 12:00:11 +0200
Łukasz Langa  wrote:
> 
> Now, there are a couple of things we can do here:
> Option 1: Finish GH-12607 to fix the old macros, keeping in mind this will 
> restore compatibility lost with Python 3.8 - 3.10 only for users of 3.11+
> Option 2: Review and merge GH-20104 that reverts the macro changes that make 
> old client code segfault -- unclear what else this needs and again, that 
> would only fix it for users of 3.11+
> Option 3: Abandon GH-12607 and GH-20104, instead declaring the old macros 
> deprecated for 3.11 and remove them in 3.13
> 
> I personally agree with Irit, voting +1 for Option 3 since the old macros 
> were soft-deprecated already by introducing new macros in 3.8, and more 
> importantly made incompatible with pre-3.8 usage.

Agreed as well.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5FKBYFBPCTS6F44B7ESOXCPJQYYU4V5I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Irit Katriel via Python-Dev]

On Tue, Aug 17, 2021 at 12:05 PM Duncan Grisby  wrote:

> I don't know if it is pertinent to this at all, but I raised
> https://bugs.python.org/issue9 in which the faulthandler module can
> lead to a segfault inside Py_TRASHCAN_SAFE_BEGIN. Would that be avoided if
> frameobject.c was changed to use Py_TRASHCAN_BEGIN / END?
>

We just changed frameobject.c to use the new macros. Can you check?
https://github.com/python/cpython/pull/27683
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4HBXLRBGNANOUMWTABWU7MPJZL5ML6V6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Petr Viktorin]

I'm late to the thread, and as I read it I see everything I wanted to 
say was covered already :)

So just a few clarifications.

The stable ABI is not defined by PEP 384 or PEP 652 or by the header 
something is defined in, but by the docs:

- https://docs.python.org/dev/c-api/stable.html
and changes to it are covered in the devguide:
- https://devguide.python.org/c-api/


We have 3 API layers:

1. Internal API, guarded by Py_BUILD_CORE, can break *even in point 
releases*. (Py_BUILD_CORE means just that: things like `_PyCode_New` can 
only be used safely if you build/embed CPython yourself.)
2. Regular C-API, covered by PEP 387 (breaking changes need deprecation 
for 2 releases, or an exception from the SC); `PyCode_New*` is here now

2. Stable ABI, which is hard to change, and thankfully isn't involved here.

I can see that having `.replace()` equivalent in the C API would be 
"worth the effort of [its users having to] keeping up with CPython 
internal changes" (to quote Patrick).
Looks like we could use something between layers 1 and 2 above for 
"high-maintenance" users (like Cython): API that will work for all of 
3.11.x, but can freely break for 3.12. I don't think this needs an 
explicit API layer, though: just a note in the docs that a new 
`PyCode_NewWithAllTheWithBellsAndWhistles` is expected to change in 
point releases. But...


Guido:

  [struct rather than N arguments] is the
  API style that _PyCode_New() uses (thanks to Eric who
  IIRC pushed for this and implemented it). You gave me an idea now:
  the C equivalent to .replace() could use the same input structure;
  one can leave fields NULL that should be copied from the original
  unmodified.


From a usability point of view, that's a much better idea than a 
function that's expected to change. It would probably also be easier to 
implement than an entirely separate public API.


Nick:

  P.S. Noting an idea that won't work, in case anyone else reading
  the thread was thinking the same thing: a "PyType_FromSpec"
  style API won't help here, as the issue is that the compiler is
  now doing more work up front and recording that extra info in
  the code object for the interpreter to use. There is no way to
  synthesise that info if it isn't passed to the constructor, as
  it isn't intrinsically recorded in the opcode sequence.


I guess it might be possible to add a flag that says a piece of bytecode 
object has exception handling and so it needs an exception table, and 
have the old API raise when the flag is on.

It's probably not worth the effort, though.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T32UC25S3R7MTAKTZRF22ZJ26K6WMFGO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Rejecting PEP 648

[Thomas Wouters]

Hi Mario,

Thank you for submitting PEP 648 (Extensible customizations of the
interpreter at startup). The Steering Council has reviewed the PEP and
after careful consideration, we have decided to reject the PEP. There are a
number of reasons why we have arrived at this decision so we would like to
explain our view of the PEP.

After careful analysis of the PEP, it is clear to us that the main focus of
the PEP (as stated in the document) is to provide a more clear mechanism
for system administrators and tools that repackage the interpreter. The PEP
also mentions “some libraries” but we are not actually convinced of the
positive impact on the PEP in this regard (see below).

Our main concern is that the improvements proposed in this PEP have a very
limited set of use cases and doesn’t really act as a replacement for the
library usage of pth files but further complicates the startup sequence
with even more things to consider, including the interaction between this
mechanism with virtualenvs. This not only incurs a maintenance cost but the
danger of further confusion when something is not working or “magic
behaviour” is noticed and the user wants to understand where it is coming
from.

There is a fundamental relationship between this PEP and pth files. There
is a general agreement that **the executable** behaviour of pth files is
undesirable and although this PEP doesn’t claim that is tackling this
problem directly, it mentions that it addresses some “limitations of pth
files”.  However, the PEP actually doesn’t provide a mechanism that is
intended to substitute or improve the situation regarding the executable
use cases of pth files. In particular there are even some incompatibilities
between the proposed mechanism and how pth files propagate to virtual
environments:

The customizations applied to an interpreter via the new __sitecustomize__
> solutions will continue to work when a user creates a virtual environment
> the same way that sitecustomize.py interacts with virtual environments.



This is a difference when compared to pth files, which are not propagated
> into virtual environments unless include-system-site-packages is enabled.



If library maintainers have features installed via __sitecustomize__ that
> they do not want to propagate into virtual environments, they should detect
> if they are running within a virtual environment by checking sys.prefix ==
> sys.base_prefix. This behavior is similar to packages that modify the
> global sitecustomize.py.


This is a fundamental difference that shifts the responsibility on how the
changes should behave from the user to the library developer, which is a
complicated decision as library developers won’t generally know what
behaviour is best for the user.  At the same time, if libraries add
customizations at startup it will “magically” affect created virtual
environments which is one of the “surprising” effects of existing pth files.

Apart from pth files, the PEP indeed proposes a more clear and organized
way to incorporate changes that otherwise will go into the
sitecustomize.py, but on the other hand this is a very specific use case
that mainly affects system administrators and repackaging tools.  Even in
these cases those stakeholders *already* control the sitecustomize.py and
they can indeed adopt this mechanism (or any other that suits their needs
better) by hardcoding it in their sitecustomize.py. This highlights what is
the impact of the benefit compared with the potential cost (including
maintenance and complication of the startup sequence). If this PEP were
accepted we will have 3 different ways to configure the startup sequence:
pth files, sitecustomize.py and __sitecustomize__, including all the
possible interactions. The PEP also mentions that:

Should the change happen, it will also improve the situation for these
> users, as rather than having a sitecustomize.py which performs all those
> actions, they can have custom isolated files named after the features they
> want to enhance.


But this is clearly not a feature directed for users because users are not
generally supposed to directly install files in the installation directory.
Users that wish to configure their startup configuration already have more
specific means to do that (such as the PYTHONSTARTUP environment variable
and related mechanisms).

Finally, we believe that the PEP lacks concrete use cases and endorsements
where this mechanism will make things easier for libraries or important
redistributors. Although the PEP mentions some libraries and some
situations it doesn’t clearly state what would be the improvements if the
PEP were affected nor includes some statements from library authors or
repackagers of the interpreter that indicate how this PEP would benefit
them.

For the whole SC,
Thomas.
-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
___
Python-Dev mailing list -- python-de

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Petr Viktorin]

On 17. 08. 21 12:00, Łukasz Langa wrote:

Hi everybody,
I'd like to revive this thread as I feel like we have to do something 
here but some consensus is needed first.


To recap, the current state of things is as follows:
- *in March 2000* (d724b23420f) Christian Tismer contributed the 
"trashcan" patch that added Py_TRASHCAN_SAFE_BEGIN 
and Py_TRASHCAN_SAFE_END macros which allow destroying nested objects 
non-recursively.
- *in May 2019* (GH-11841 of BPO-35983) Antoine Pitrou merged a change 
by Jeroen Demeyer which made Py_TRASHCAN_SAFE_BEGIN/END 
(unintentionally?) backwards incompatible; this was released in Python 
3.8.0.
- by the way, GH-11841 introduced a new pair of macros (because they 
have different signatures) called simply  Py_TRASHCAN_BEGIN and 
Py_TRASHCAN_END.
- by that time there was already a follow-up PR open (GH-12607) to 
improve backwards compatibility of the macros, as well as introduce 
tests for them; this was never merged.
- *in Feb 2020* (0fa4f43db08) Victor Stinner removed the trashcan 
mechanism from the limited C API (note: not ABI, those are macros) since 
it accesses fields of structs not exposed in the limited C API; this was 
released in Python 3.9.0.
- *in May 2020* Irit noticed that the backwards incompatibility 
(BPO-40608) causes segfaults for C API code that worked fine with Python 
3.7. Using the new macros requires code changes but doesn't crash.


Now, there are a couple of things we can do here:
*Option 1*: Finish GH-12607 to fix the old macros, keeping in mind this 
will restore compatibility lost with Python 3.8 - 3.10 only for users of 
3.11+
*Option 2*: Review and merge GH-20104 that reverts the macro changes 
that make old client code segfault -- unclear what else this needs and 
again, that would only fix it for users of 3.11+
*Option 3*: Abandon GH-12607 and GH-20104, instead declaring the old 
macros deprecated for 3.11 and remove them in 3.13


I personally agree with Irit, voting +1 for Option 3 since the old 
macros were soft-deprecated already by introducing new macros in 3.8, 
and more importantly made incompatible with pre-3.8 usage.


+1.
The deprecation should follow PEP 387, which means emitting a warning. I 
think a compiler warning is appropriate; that could be done by having 
the macro call a deprecated function. Depending on how broken the macros 
are, a compiler warning could be backported to older versions.




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KWFX6XX3HMZBQ2BYBVL7G74AIOPWO66Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Victor Stinner]

If the macro is deprecated, please provide an detailed explanation how
to port existing C code to the new C API, in What's New In Python 3.X
(version where the macro is deprecated, Python 3.11 if I understood
correctly).

Also, is there a way to write a single code base working on Python
3.6-Python 3.11? It seems like mypy uses macros to support Python <
3.8 and Python >= 3.8:

#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
#else
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)


A search in PyPI top 1000 project gives me 8 projects (I was too lazy
to check the full top 5000). It would be great to help these projects
(propose pull requests) to migrate to the new C API.

(*) pyrsistent-0.18.0.tar.gz

> ./pvectorcmodule.c:  Py_TRASHCAN_SAFE_BEGIN(self);
> ./pvectorcmodule.c:  Py_TRASHCAN_SAFE_BEGIN(self);

(*) PyGObject-3.40.1.tar.gz

> ./gi/pygi-resulttuple.c:Py_TRASHCAN_SAFE_BEGIN (self)

(*) pycurl-7.44.1.tar.gz

> ./src/easy.c:Py_TRASHCAN_SAFE_BEGIN(self);
> ./src/multi.c:Py_TRASHCAN_SAFE_BEGIN(self);
> ./src/share.c:Py_TRASHCAN_SAFE_BEGIN(self);

(*) mypy-0.910.tar.gz

> ./mypyc/lib-rt/CPy.h:
---
#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
#else
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)
#endif
---

(*) pypi-top-1000_2021-08-17/multidict-5.1.0.tar.gz

> ./multidict/_multidict.c:Py_TRASHCAN_SAFE_BEGIN(self);

(*) pypi-top-1000_2021-08-17/immutables-0.16.tar.gz

> ./immutables/_map.c:Py_TRASHCAN_SAFE_BEGIN(self)
> ./immutables/_map.c:Py_TRASHCAN_SAFE_BEGIN(self)
> ./immutables/_map.c:Py_TRASHCAN_SAFE_BEGIN(self)

(*) guppy3-3.1.1.tar.gz

> ./src/sets/nodeset.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/sets/immnodeset.c:Py_TRASHCAN_SAFE_BEGIN(it)
> ./src/sets/immnodeset.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/heapy/hv.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/heapy/classifier.c:Py_TRASHCAN_SAFE_BEGIN(op)
> ./src/heapy/nodegraph.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/heapy/hv_cli_rel.c:Py_TRASHCAN_SAFE_BEGIN(op)

(*) frozendict-2.0.6.tar.gz

> 

Victor

On Tue, Aug 17, 2021 at 12:02 PM Łukasz Langa  wrote:
>
> Hi everybody,
> I'd like to revive this thread as I feel like we have to do something here 
> but some consensus is needed first.
>
> To recap, the current state of things is as follows:
> - in March 2000 (d724b23420f) Christian Tismer contributed the "trashcan" 
> patch that added Py_TRASHCAN_SAFE_BEGIN and Py_TRASHCAN_SAFE_END macros which 
> allow destroying nested objects non-recursively.
> - in May 2019 (GH-11841 of BPO-35983) Antoine Pitrou merged a change by 
> Jeroen Demeyer which made Py_TRASHCAN_SAFE_BEGIN/END (unintentionally?) 
> backwards incompatible; this was released in Python 3.8.0.
> - by the way, GH-11841 introduced a new pair of macros (because they have 
> different signatures) called simply  Py_TRASHCAN_BEGIN and Py_TRASHCAN_END.
> - by that time there was already a follow-up PR open (GH-12607) to improve 
> backwards compatibility of the macros, as well as introduce tests for them; 
> this was never merged.
> - in Feb 2020 (0fa4f43db08) Victor Stinner removed the trashcan mechanism 
> from the limited C API (note: not ABI, those are macros) since it accesses 
> fields of structs not exposed in the limited C API; this was released in 
> Python 3.9.0.
> - in May 2020 Irit noticed that the backwards incompatibility (BPO-40608) 
> causes segfaults for C API code that worked fine with Python 3.7. Using the 
> new macros requires code changes but doesn't crash.
>
> Now, there are a couple of things we can do here:
> Option 1: Finish GH-12607 to fix the old macros, keeping in mind this will 
> restore compatibility lost with Python 3.8 - 3.10 only for users of 3.11+
> Option 2: Review and merge GH-20104 that reverts the macro changes that make 
> old client code segfault -- unclear what else this needs and again, that 
> would only fix it for users of 3.11+
> Option 3: Abandon GH-12607 and GH-20104, instead declaring the old macros 
> deprecated for 3.11 and remove them in 3.13
>
> I personally agree with Irit, voting +1 for Option 3 since the old macros 
> were soft-deprecated already by introducing new macros in 3.8, and more 
> importantly made incompatible with pre-3.8 usage.
>
> Let's talk on how to proceed.
>
> - Ł
>
>
>
> On 26 Apr 2021, at 23:55, Irit Katriel via Python-Dev  
> wrote:
>
>
> Re https://bugs.python.org/issue40608.
>
> I think it will be an act of kindness to deprecate Py_TRASHCAN_SAFE_BEGIN/END 
> in 3.10 and tell people to use Py_TRASHCAN_BEGIN/END instead.
>
> TL;DR: There was a change in 3.8 that introduced the latter while leaving the 

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Victor Stinner]

See also:

* https://bugs.python.org/issue44881 "Consider integration of
PyObject_GC_UnTrack() with the trashcan C API".
* https://bugs.python.org/issue44897 "Integrate trashcan mechanism
into _Py_Dealloc"

The issue44897 would make all these macros useless ;-)

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZTTQP5YYAKVBQWHZPOVQ4FREZJPPV3PQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

Since Cython is a common consumer of this C API, can somone please dig
into Cython to see exactly what it needs in terms of API? How does
Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
copy an existing function to only override some fields, something like
CodeType.replace(field=new_value)?

If possible, I would prefer that Cython only uses the *public* C API.
Otherwise, it will be very likely that Cython will break at every
single Python release. Cython has a small team to maintain the code
base, whereas CPython evolves much faster with a larger team.

Victor

On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
>
> Doing a search of a huge codebase (work), the predominant user of PyCode_New* 
> APIs appears to be checked in Cython generated code (in all sorts of 
> third_party OSS projects). It's in the boilerplate that Cython extensions 
> make use of via it's __Pyx_PyCode_New macro.  
> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
>
> I saw very few non-Cython uses.  There are some, but at a very quick first 
> glance they appear simple - easy enough to reach out to the projects with a 
> PR to update their code.
>
> The Cython use will require people to upgrade Cython and regenerate their 
> code before they can use the Python version that changes these. That is not 
> an uncommon thing for Cython. It's unfortunate that many projects on ship 
> generated sources rather than use Cython at build time, but that isn't _our_ 
> problem to solve. The more often we change internal APIs that things depend 
> on, the more people will move their projects towards doing the right thing 
> with regards to either not using said APIs or rerunning an up to date code 
> generator as part of their build instead of checking in generated unstable 
> API using sources.
>
> -gps
>
>
> On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum  wrote:
>>
>> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan  wrote:
>>>
>>> [...]
>>> A cloning-with-replacement API that accepted the base code object and the 
>>> "safe to modify" fields could be a good complement to the API deprecation 
>>> proposal.
>>
>>
>> Yes (I forgot to mention that).
>>
>>>
>>> Moving actual "from scratch" code object creation behind the Py_BUILD_CORE 
>>> guard with an underscore prefix on the name would also make sense, since it 
>>> defines a key piece of the compiler/interpreter boundary.
>>
>>
>> Yeah, we have _PyCode_New() for that.
>>
>>>
>>> Cheers,
>>> Nick.
>>>
>>> P.S. Noting an idea that won't work, in case anyone else reading the thread 
>>> was thinking the same thing: a "PyType_FromSpec" style API won't help here, 
>>> as the issue is that the compiler is now doing more work up front and 
>>> recording that extra info in the code object for the interpreter to use. 
>>> There is no way to synthesise that info if it isn't passed to the 
>>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>>
>>
>> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC pushed 
>> for this and implemented it). You gave me an idea now: the C equivalent to 
>> .replace() could use the same input structure; one can leave fields NULL 
>> that should be copied from the original unmodified.
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> Pronouns: he/him (why is my pronoun here?)
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/67DMIW7NQE6M6LEPLANXKZQEFOFVPBBL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FFE6ULML5CWOBMFCZKTRCSSHKTPBFRFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

[Duncan Grisby]

I don't know if it is pertinent to this at all, but I
raised https://bugs.python.org/issue9 in which the faulthandler
module can lead to a segfault inside Py_TRASHCAN_SAFE_BEGIN. Would that
be avoided if frameobject.c was changed to use Py_TRASHCAN_BEGIN / END?

Duncan.


On Tue, 2021-08-17 at 12:00 +0200, Łukasz Langa wrote:
> Hi everybody,
> I'd like to revive this thread as I feel like we have to do something
> here but some consensus is needed first.
> 
> To recap, the current state of things is as follows:
> - in March 2000 (d724b23420f) Christian Tismer contributed the
> "trashcan" patch that added Py_TRASHCAN_SAFE_BEGIN
> and Py_TRASHCAN_SAFE_END macros which allow destroying nested objects
> non-recursively.
> - in May 2019 (GH-11841 of BPO-35983) Antoine Pitrou merged a change
> by Jeroen Demeyer which made Py_TRASHCAN_SAFE_BEGIN/END
> (unintentionally?) backwards incompatible; this was released in
> Python 3.8.0.
> - by the way, GH-11841 introduced a new pair of macros (because they
> have different signatures) called simply  Py_TRASHCAN_BEGIN and
> Py_TRASHCAN_END.
> - by that time there was already a follow-up PR open (GH-12607) to
> improve backwards compatibility of the macros, as well as introduce
> tests for them; this was never merged.
> - in Feb 2020 (0fa4f43db08) Victor Stinner removed the trashcan
> mechanism from the limited C API (note: not ABI, those are macros)
> since it accesses fields of structs not exposed in the limited C API;
> this was released in Python 3.9.0.
> - in May 2020 Irit noticed that the backwards incompatibility (BPO-
> 40608) causes segfaults for C API code that worked fine with Python
> 3.7. Using the new macros requires code changes but doesn't crash.
> 
> Now, there are a couple of things we can do here:
> Option 1: Finish GH-12607 to fix the old macros, keeping in mind this
> will restore compatibility lost with Python 3.8 - 3.10 only for users
> of 3.11+
> Option 2: Review and merge GH-20104 that reverts the macro changes
> that make old client code segfault -- unclear what else this needs
> and again, that would only fix it for users of 3.11+
> Option 3: Abandon GH-12607 and GH-20104, instead declaring the old
> macros deprecated for 3.11 and remove them in 3.13
> 
> I personally agree with Irit, voting +1 for Option 3 since the old
> macros were soft-deprecated already by introducing new macros in 3.8,
> and more importantly made incompatible with pre-3.8 usage.
> 
> Let's talk on how to proceed.
> 
> - Ł

-- 
Duncan Grisby 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SB6ODKKHANAHDXST2W73RNDZSCZQFOHS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Problems with dict subclassing performance

[Marco Sulla]

On Sun, 15 Aug 2021 at 22:30, Marco Sulla  wrote:
> Oh, this is enough. The sense of the phrase was very clear and you all
> have understood it. Remarking grammatical errors is a gross violation
> of the Netiquette. I ask __immediately__ the intervent of a moderator,
> even if I quite sure, since I'm mister No One and you are Terry Reed
> and Steven D'Aprano, that this will be against me X-D

Ahahhaahh, call me Marco "Cassandra" Sulla.

Mr. Cannon, I was sure about your response. Please ban me, it will be
a pleasure to be banned by you for specious reasons. It's your
specialty.

I want to remember here that you banned me from the Py Forum because I
said "Even a child can understand my code", and you _demanded_ my
excuses. Since I found the accusation ridiculous, I've made my excuses
to all children that do not understand my code. You banned me and even
Steven defended me.

I've been the moderator for a forum for years, and let me say, you are
not a good moderator. You are hard-mannered. See Tim. He calmed me
down with two posts. You all have to learn from him, as coders,
moderators and human beings.

That said, please ban me permanently, because I'll _never_ give you my
excuses. What I said is the pure truth.

PS: as a side note, I started to get downvotes from when this useless
polemic started. Since, I repeat, I have more than 13k reputation on
SO, and since the question had about 20 votes without dirty tricks, as
Steven subtly insinuates, all of this makes me laugh. Do your worst,
you all little men. As a Mister No One, I will be **PROUD** to be
banned as Stephan Krah.


On Mon, 16 Aug 2021 at 18:52, Brett Cannon  wrote:
>
>
>
> On Sun, Aug 15, 2021 at 2:55 PM Marco Sulla  
> wrote:
>>
>> On Sun, 15 Aug 2021 at 23:33, Tim Peters 
>> wrote:ople have said now, including me, they had no idea what
>> > you meant.by "I pretend your immediate excuses". It's not a complaint
>> > that it's expressed inelegantly, but that they can't make _any_ sense
>> > of it. By my count, this is at least the second time you've declined
>> > to explain what you meant, but instead implied the person who said
>> > they couldn't understand it was being dishonest.
>>
>> I repeat, even the worst AI will understand from the context what I
>> meant. But let me do a very rude example:
>
>
> Rude examples are never necessary and are not acceptable (don't forget we 
> have kids who participate on this mailing list).
>
> I have referred the whole thread to the Conduct WG so they can settle who was 
> out of line. Otherwise I advise everyone to mute this thread.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BM4AQR347MPNXRDF3FHBYPOOXPHMYQU2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Problems with dict subclassing performance

[Marco Sulla]

My time is short, so thank you for focusing on the real subject.

On Mon, 16 Aug 2021 at 11:00, Jeff Allen  wrote:
> I have spent a lot of time reading typeobject.c over the years I've been 
> looking at an alternative implementation. It's quite difficult to follow, and 
> full of tweaks for special circumstances. So I'm impressed with the 
> understanding that "user2357112 supports Monica" brings to the subject.

I was impressed too. I suppose it worked on some C Extension and
discovered the behaviour herself.

> When a built-in type like dict is defined in C, pointers to its C 
> implementation functions are hard-coded into slots in the type object. In 
> order to make each appear as a method to Python, a descriptor is created when 
> building the type that delegates to the slot (so sq_contains generates a 
> descriptor __contains__ in the dictionary of the type.
>
> Conversely, if in a sub-class you define __contains__, then the type builder 
> will insert a function pointer in the slot of the new type that arranges a 
> call to __contains__. This will overwrite whatever was in the slot.
>
> In a C implementation, you can also define methods (by creating a PyMethodDef 
> the tp_methods table) that become descriptors in the dictionary of the type. 
> You would not normally define both a C function to place in the slot *and* 
> the corresponding method via a PyMethodDef. If you do, the version from the 
> dictionary of the type will win the slot, *unless* you mark the method 
> definition (in its PyMethodDef) as METH_COEXIST.
>
> This exception is used in the special case of dict (and hardly anywhere else 
> but set I think). I assume this is because some important code calls 
> __contains__ via the descriptor, rather than via the slot (which would be 
> quicker), and because an explicit definition is faster than a descriptor 
> created automatically to wrap the slot.
>
> Now, when you create a sub-class, the table of slots is copied first, then 
> the type is checked for definitions of special methods, and these are allowed 
> to overwrite the slot, unless they are slot wrappers on the same function 
> pointer the slot already contains. I think at this point the slot is 
> re-written to contain a wrapper on __contains__, which has been inherited 
> from dict.__contains__, because it isn't a *slot wrapper* on the same 
> function. For example:
>
> >>> dict.__contains__
> 
> >>> str.__contains__
> 
>
> >>> class S(str): pass
>
> >>> S.__contains__
> 
> >>> D.__contains__
> 

And is it not possible for subclasses to continue to use the optimized
version, if some contract will be maintained?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UGBL62CDO2XYXAVK6RAOS46ORGL545MV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Gregory P. Smith]

+cc: cython-devel

background reading for those new to the thread:
https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/

On Tue, Aug 17, 2021 at 9:47 AM Victor Stinner  wrote:

> Since Cython is a common consumer of this C API, can somone please dig
> into Cython to see exactly what it needs in terms of API? How does
> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
> copy an existing function to only override some fields, something like
> CodeType.replace(field=new_value)?
>
> If possible, I would prefer that Cython only uses the *public* C API.
> Otherwise, it will be very likely that Cython will break at every
> single Python release. Cython has a small team to maintain the code
> base, whereas CPython evolves much faster with a larger team.
>
> Victor
>

I don't claim knowledge of Cython internals, but the two places it appears
to call it's __Pyx_PyCode_New macro are:

https://github.com/cython/cython/blob/master/Cython/Utility/Exceptions.c#L769
in __Pyx_CreateCodeObjectForTraceback() - this one already has a `#if
CYTHON_COMPILING_IN_LIMITED_API` code path option in it.

 and

https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L9722
in CodeObjectNode.generate_result_code()  that creates PyCodeObject's for
CyFunction instances per its comment.  Slightly described in this comment
http://google3/third_party/py/cython/files/Cython/Compiler/ExprNodes.py?l=397.
I don't see anything obvious mentioning the limited API in that code
generator.

it'd be best to loop in Cython maintainers for more of an idea of Cython's
intents and needs with PyCode_New APIs.  I've cc'd cython-de...@python.org.

-Greg


> On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
> >
> > Doing a search of a huge codebase (work), the predominant user of
> PyCode_New* APIs appears to be checked in Cython generated code (in all
> sorts of third_party OSS projects). It's in the boilerplate that Cython
> extensions make use of via it's __Pyx_PyCode_New macro.
> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
> >
> > I saw very few non-Cython uses.  There are some, but at a very quick
> first glance they appear simple - easy enough to reach out to the projects
> with a PR to update their code.
> >
> > The Cython use will require people to upgrade Cython and regenerate
> their code before they can use the Python version that changes these. That
> is not an uncommon thing for Cython. It's unfortunate that many projects on
> ship generated sources rather than use Cython at build time, but that isn't
> _our_ problem to solve. The more often we change internal APIs that things
> depend on, the more people will move their projects towards doing the right
> thing with regards to either not using said APIs or rerunning an up to date
> code generator as part of their build instead of checking in generated
> unstable API using sources.
> >
> > -gps
> >
> >
> > On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum 
> wrote:
> >>
> >> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan 
> wrote:
> >>>
> >>> [...]
> >>> A cloning-with-replacement API that accepted the base code object and
> the "safe to modify" fields could be a good complement to the API
> deprecation proposal.
> >>
> >>
> >> Yes (I forgot to mention that).
> >>
> >>>
> >>> Moving actual "from scratch" code object creation behind the
> Py_BUILD_CORE guard with an underscore prefix on the name would also make
> sense, since it defines a key piece of the compiler/interpreter boundary.
> >>
> >>
> >> Yeah, we have _PyCode_New() for that.
> >>
> >>>
> >>> Cheers,
> >>> Nick.
> >>>
> >>> P.S. Noting an idea that won't work, in case anyone else reading the
> thread was thinking the same thing: a "PyType_FromSpec" style API won't
> help here, as the issue is that the compiler is now doing more work up
> front and recording that extra info in the code object for the interpreter
> to use. There is no way to synthesise that info if it isn't passed to the
> constructor, as it isn't intrinsically recorded in the opcode sequence.
> >>
> >>
> >> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
> pushed for this and implemented it). You gave me an idea now: the C
> equivalent to .replace() could use the same input structure; one can leave
> fields NULL that should be copied from the original unmodified.
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido)
> >> Pronouns: he/him (why is my pronoun here?)
> >> ___
> >> Python-Dev mailing list -- python-dev@python.org
> >> To unsubscribe send an email to python-dev-le...@python.org
> >> https://mail.python.org/mailman3/lists/python-dev.python.org/
> >> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
> >> Code of Conduct: http://python.org/psf/codeofconduct/
> >
> > 

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

Both locations Greg found seem to be creating a completely *empty* code
object except for the name, filename and lineno. So that would actually
work fine with my proposal.

On Tue, Aug 17, 2021 at 4:24 PM Gregory P. Smith  wrote:

> +cc: cython-devel
>
> background reading for those new to the thread:
> https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/
>
> On Tue, Aug 17, 2021 at 9:47 AM Victor Stinner 
> wrote:
>
>> Since Cython is a common consumer of this C API, can somone please dig
>> into Cython to see exactly what it needs in terms of API? How does
>> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
>> copy an existing function to only override some fields, something like
>> CodeType.replace(field=new_value)?
>>
>> If possible, I would prefer that Cython only uses the *public* C API.
>> Otherwise, it will be very likely that Cython will break at every
>> single Python release. Cython has a small team to maintain the code
>> base, whereas CPython evolves much faster with a larger team.
>>
>> Victor
>>
>
> I don't claim knowledge of Cython internals, but the two places it appears
> to call it's __Pyx_PyCode_New macro are:
>
>
> https://github.com/cython/cython/blob/master/Cython/Utility/Exceptions.c#L769
> in __Pyx_CreateCodeObjectForTraceback() - this one already has a `#if
> CYTHON_COMPILING_IN_LIMITED_API` code path option in it.
>
>  and
>
>
> https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L9722
> in CodeObjectNode.generate_result_code()  that creates PyCodeObject's for
> CyFunction instances per its comment.  Slightly described in this comment
> http://google3/third_party/py/cython/files/Cython/Compiler/ExprNodes.py?l=397.
> I don't see anything obvious mentioning the limited API in that code
> generator.
>
> it'd be best to loop in Cython maintainers for more of an idea of Cython's
> intents and needs with PyCode_New APIs.  I've cc'd cython-de...@python.org
> .
>
> -Greg
>
>
>> On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
>> >
>> > Doing a search of a huge codebase (work), the predominant user of
>> PyCode_New* APIs appears to be checked in Cython generated code (in all
>> sorts of third_party OSS projects). It's in the boilerplate that Cython
>> extensions make use of via it's __Pyx_PyCode_New macro.
>> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
>> >
>> > I saw very few non-Cython uses.  There are some, but at a very quick
>> first glance they appear simple - easy enough to reach out to the projects
>> with a PR to update their code.
>> >
>> > The Cython use will require people to upgrade Cython and regenerate
>> their code before they can use the Python version that changes these. That
>> is not an uncommon thing for Cython. It's unfortunate that many projects on
>> ship generated sources rather than use Cython at build time, but that isn't
>> _our_ problem to solve. The more often we change internal APIs that things
>> depend on, the more people will move their projects towards doing the right
>> thing with regards to either not using said APIs or rerunning an up to date
>> code generator as part of their build instead of checking in generated
>> unstable API using sources.
>> >
>> > -gps
>> >
>> >
>> > On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum 
>> wrote:
>> >>
>> >> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan 
>> wrote:
>> >>>
>> >>> [...]
>> >>> A cloning-with-replacement API that accepted the base code object and
>> the "safe to modify" fields could be a good complement to the API
>> deprecation proposal.
>> >>
>> >>
>> >> Yes (I forgot to mention that).
>> >>
>> >>>
>> >>> Moving actual "from scratch" code object creation behind the
>> Py_BUILD_CORE guard with an underscore prefix on the name would also make
>> sense, since it defines a key piece of the compiler/interpreter boundary.
>> >>
>> >>
>> >> Yeah, we have _PyCode_New() for that.
>> >>
>> >>>
>> >>> Cheers,
>> >>> Nick.
>> >>>
>> >>> P.S. Noting an idea that won't work, in case anyone else reading the
>> thread was thinking the same thing: a "PyType_FromSpec" style API won't
>> help here, as the issue is that the compiler is now doing more work up
>> front and recording that extra info in the code object for the interpreter
>> to use. There is no way to synthesise that info if it isn't passed to the
>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>> >>
>> >>
>> >> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
>> pushed for this and implemented it). You gave me an idea now: the C
>> equivalent to .replace() could use the same input structure; one can leave
>> fields NULL that should be copied from the original unmodified.
>> >>
>> >> --
>> >> --Guido van Rossum (python.org/~guido)
>> >> Pronouns: he/him (why is my pronoun here?)
>> >> ___
>> >> Python-Dev mailing list -- python-dev@

[Python-Dev] Re: Making code object APIs unstable

[Spencer Brown]

Victor Stinner wrote:
> Since Cython is a common consumer of this C API, can somone please dig
> into Cython to see exactly what it needs in terms of API? How does
> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
> copy an existing function to only override some fields, something like
> CodeType.replace(field=new_value)?
> If possible, I would prefer that Cython only uses the *public* C API.
> Otherwise, it will be very likely that Cython will break at every
> single Python release. Cython has a small team to maintain the code
> base, whereas CPython evolves much faster with a larger team.
> Victor

Cython only uses it here, to construct a fake code object with mostly blank 
values for each of the Cython function objects:
https://github.com/cython/cython/blob/8af0271186cc642436306274564986888d5e64c8/Cython/Compiler/ExprNodes.py#L9722
It's not actually intending to execute the code object at any point, it's 
created so that fake tracebacks get a filename/line number, the function 
objects have the argument count values for inspect to read and so Cython can 
emulate profiler/tracer events. So for Cython's case in particular it could a 
more stable API function, though said hypothetical API would have to have 
updated to handle pos-only args anyway also.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7RM4XJBH3KVSP3T43YCT6VCELTPCCBME/
Code of Conduct: http://python.org/psf/codeofconduct/