[Python-Dev] Re: Request to postpone some Python 3.9 incompatible changes to Python 3.10

[Petr Viktorin]

On 2020-02-03 16:44, Antoine Pitrou wrote:

On Mon, 3 Feb 2020 13:18:46 +0100
Petr Viktorin  wrote:


I agree with the sentiment that gradual deprecations are more easily
managed, this statement about Python 3.0 is not true. The unicode
transition was never thought to be small, and that's *why* 3.0 was such
a big change.


Alright, "very small" is an overstatement. But it did seem much smaller
than it turned out to be.
https://docs.python.org/3.0/whatsnew/3.0.html lists it as the last of
the big breaking changes, while most of the porting efforts were spent
on it.


I don't think the order in this page has any relationship to the
difficulty of each change.  I'm not sure there was any particular
reason to order them in this way, but it seems to me that the easier
to understand changes were put first.

AFAIR, we were all acutely aware that the text model overhaul was going
to be the hardest change for many users (especially those who have
never encountered non-ASCII input). Most other changes in Python 3 have
a predetermined context-free repair, while migrating to the new text
model requires *thinking* about the context in which each Python 2
bytestring is used.  It forces people to think about the problem
they're solving in ways many of them didn't think of it before.


I stand corrected. Thank you!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U327ZCIYDQ2QVSP2E7V3O4WGIA275GSS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to postpone some Python 3.9 incompatible changes to Python 3.10

[Petr Viktorin]

On 2020-02-03 12:55, Thomas Wouters wrote:



On Mon, Feb 3, 2020 at 10:53 AM Petr Viktorin <mailto:encu...@gmail.com>> wrote:


On 2020-01-31 19:47, Mike Miller wrote:
 >
 > On 2020-01-23 07:20, Victor Stinner wrote:
 >  > Python 3.9 introduces many small incompatible changes which
broke tons
 >
 >
 > There's a well-known and established way of signaling breaking
changes
 > in software platforms—it is to increment the major version number.
 >
 > Rather than debating the merits of breaking code on 3.9 or 3.10,
 > wouldn't it make more sense to do it in a Python 4.0 instead?  Well,
 > either of these strategies sound logical to me:
 >
 > - Python 4.0 with removal of all of the Python 3-era deprecations
 > - Continuing Python 3.1X with no breaks
 >
 > In other words, we should keep compatibility, or not.  In any
case, from
 > the looks of it these will be tiny breaks compared to the Unicode
 > transition.

The Unicode transition also looked very small back when 3.0 was planned.
It only takes one such not-so-little thing to make a big breaking
release like 3.0. And even if all the changes were little, I wouldn't
want to inflict 10 years of papercuts at once.


I agree with the sentiment that gradual deprecations are more easily 
managed, this statement about Python 3.0 is not true. The unicode 
transition was never thought to be small, and that's *why* 3.0 was such 
a big change.


Alright, "very small" is an overstatement. But it did seem much smaller 
than it turned out to be.
https://docs.python.org/3.0/whatsnew/3.0.html lists it as the last of 
the big breaking changes, while most of the porting efforts were spent 
on it.



We knew it was going to break everything, so we took the 
opportunity to break more things, like the behaviour of indexing 
bytestrings. (Bytestrings were even going to be *mutable* for a while.) 
I think we can all agree that it was a mistake, and that's certainly 
something we don't want to repeat: even if we defer actual removal of 
features for a x.0.0 release, it must not become carte blanche for 
breaking things. >


When the changes are rolled out gradually across minor releases, those
that cause unforeseen trouble in real-world code can be identified in
the alphas/betas, and rethought/reverted if necessary.


That's also the case if things are (loudly) deprecated *but not removed* 
until a x.0.0 release. The replacement would already be in use for years 
before the old way would go away.


I fear that this is only so if no unexpected issues come up.
If we do a loud deprecation, how can we be sure we did it right?





Ethan Furman wrote:
 > I've gotta say, I like that plan.  Instead of going to x.10, go
to x+1.0.  Every ten years we bump the major version and get rid of
all the deprecations.

I don't. I hope the 10-year (and counting) transition from Python 2 to
Python 3 will not become a tradition.
I'd rather iterate on making removals less drastic (e.g. by making the
DeprecationWarnings more visible). Iterate with a feedback loop, rather
than do a one-time change and hope that everything goes well.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X5RBXEEB5BFPXGGGT2MTG4KZL3OBXQFD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to postpone some Python 3.9 incompatible changes to Python 3.10

[Petr Viktorin]

On 2020-01-31 19:47, Mike Miller wrote:


On 2020-01-23 07:20, Victor Stinner wrote:
 > Python 3.9 introduces many small incompatible changes which broke tons


There's a well-known and established way of signaling breaking changes 
in software platforms—it is to increment the major version number.


Rather than debating the merits of breaking code on 3.9 or 3.10, 
wouldn't it make more sense to do it in a Python 4.0 instead?  Well, 
either of these strategies sound logical to me:


- Python 4.0 with removal of all of the Python 3-era deprecations
- Continuing Python 3.1X with no breaks

In other words, we should keep compatibility, or not.  In any case, from 
the looks of it these will be tiny breaks compared to the Unicode 
transition.


The Unicode transition also looked very small back when 3.0 was planned.
It only takes one such not-so-little thing to make a big breaking 
release like 3.0. And even if all the changes were little, I wouldn't 
want to inflict 10 years of papercuts at once.


When the changes are rolled out gradually across minor releases, those 
that cause unforeseen trouble in real-world code can be identified in 
the alphas/betas, and rethought/reverted if necessary.



Ethan Furman wrote:
I've gotta say, I like that plan.  Instead of going to x.10, go to x+1.0.  Every ten years we bump the major version and get rid of all the deprecations. 


I don't. I hope the 10-year (and counting) transition from Python 2 to 
Python 3 will not become a tradition.
I'd rather iterate on making removals less drastic (e.g. by making the 
DeprecationWarnings more visible). Iterate with a feedback loop, rather 
than do a one-time change and hope that everything goes well.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CHMOFONIBAACIW5A5SNHLTV6A5BEQXYT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to postpone some Python 3.9 incompatible changes to Python 3.10

[Petr Viktorin]

On 2020-01-30 07:12, Dima Tisnek wrote:

On Fri, 24 Jan 2020 at 00:27, Victor Stinner  wrote:


Miro and me consider that Python 3.9 is pushing too much pressure on
projects maintainers to either abandon Python 2.7 right now...


Let's not conflate py2 EOL with what-should-go-into-py39.


https://copr.fedorainfracloud.org/coprs/g/python/python3.9/monitor/
- Has package failures. Some packages fail because of broken dependencies.


Please provide a better summary.
I've looked at first 3 failed packages, and:

* `afflib` only has these error lines:
```
warning: extra tokens at the end of %endif directive in line 186:
%endif # with_python2
warning: extra tokens at the end of %endif directive in line 195:
%endif # with_python3
```
I don't know if that's why the build is considered failed, but that
issue seems to be on the build script side.


Those are just warnings; ignore them.
afflib didn't build because it's missing a dependency 
(libtermcap-devel). It wasn't counted in the ones with Python-related 
problems.



* `python-django-post_office` doesn't have any stderr output or other
indication why the build failed.


Same situation (this time missing python3-django-jsonfield).


* `ansible-lint` same, does this package even use Python?


And that one is missing python3-passlib, which fails several tests; one 
of them being ImportError: cannot import name 'MutableMapping' from 
'collections'.




https://bugzilla.redhat.com/showdependencytree.cgi?id=PYTHON39
- Has open Python 3.9 bug reports for Fedora packages. Some problems


I feel it's still early days for py39 and well-maintained packages are
expected to catch up.
> I, for one, just started testing our crud against py39a3 yesterday,
and yes some tools and libraries are broken, so I've opened issues and
might post a PR, depending.

I get the gist from your email that you are (or fedora folks are)
doing the same.


Definitely! That's exactly what we're doing: helping the well-maintained 
packages catch up, via bug reports and pull requests as capacity allows.


And now we're reporting back that the most common change these packages 
have to make is the collections.abc removal, and that since many 
projects are planning to drop py2 compat later in 2020, the pull request 
we need to send is ugly -- and it could be much nicer one-liner in 2021 
if the collections.abc removal was postponed.



So, what's the rationale to change py39 change set?


try:
 from collections.abc import Sequence
except ImprotError:
 # Python 2.7 doesn't have collections.abc
 from collections import Sequence




I thought that collections compatibility was kept up to 3.8
specifically because py2 was alive.
No that that requirement is gone, so should the shim, right?
Ofc., my assumption could be just wrong...


While if we remove collections.Sequence in 3.10, they will face this
decision early in 2021 and they will simply fix their code by adding
".abc" at the proper places, not needing any more compatibility
layers. Of course, there will be projects that will still have
declared Python 2 support in 2021, but it will arguably not be that
many.


I see some semi-abandoned projects, as well as actively maintained
projects not addressing DeprecationWarnings.
I don't know if some projects just need a harder push (will break, not
just deprecated).
Either way, my gut tells me, if py39 was changed and became very
compatible, we'd see the same discussion in 2021.


Not really. For projects that want to stay be py2/py3-compatible for the 
long term, the long "try/except ImportError" dance makes sense.



* Removed tostring/fromstring methods in array.array and base64 modules
* Removed collections aliases to ABC classes
* Removed fractions.gcd() function (which is similar to math.gcd())
* Remove "U" mode of open(): having to use io.open() just for Python 2
makes the code uglier
* Removed old plistlib API: 2.7 doesn't have the new API


my 2c: all of these are truly minor.

I have a feeling that the discussion is just wrong altogether.
It should not be "let's remove" or "let's keep".
It should not be "let's keep x but not y".

It should be something along the lines of:
* here's a great tool that fedora has that shows which libraries need
changes (your links + better output)
* here's a build server, where existing packages (maybe not fedora)
can be tested and built on py39, new ABI, etc.
* [optional] it would be awesome if anyone could submit their repo url
and the server would automatically build wheels for all arch,abi and
publish to pypi
* let's notify package maintainers (presumably done?) and if that
doesn't yield the desired result,
* let's organise the open-source community (via fame or bounties) and
fix (or fork) those that are important (common [transitive] deps to
many fedora packages)


Yes! Except a few details, that's just what I'd like to do in the long 
term. Do you want to help?

___
Python-Dev mailing list -- 

[Python-Dev] Re: Last call for comments on PEP 573 (Module State Access from C Extension Methods)

[Petr Viktorin]

On 2019-11-25 13:15, Stefan Behnel wrote:

Hi all,

I think that PEP 573 is ready to be accepted, to greatly improve the state
of extension modules in CPython 3.9.

https://www.python.org/dev/peps/pep-0573/

It has come a long way since the original proposal and went through several
iterations and discussions by various interested people, effectively
reducing its scope quite a bit. So this is the last call for comments on
the latest version of the PEP, before I will pronounce on it. Please keep
the discussion in this thread.


Hi everyone,
There was another round of changes, reducing the scope even more.

Are there any further comments on the PEP?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QMMLRQY4WVRGNMTUTMMS6QDOZVQ5LNTK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: extern "C" { ... } in Include/cpython/*.h

[Petr Viktorin]

On 2020-01-27 21:42, Skip Montanaro wrote:

(Apologies. Not sure where to ask this, and I'm not much of a C++
programmer. Maybe I should have just added a comment to the still-open
issue.)

I just noticed that Nick migrated the guts of Include/frameobject.h to
include/cpython/frameobject.h. It's not clear to me that the latter
should be #include'd directly from anywhere other than
Include/frameobject.h. If that's the case, does the extern "C" stuff
still need to be replicated in the lower level file? Won't the scope
of the extern "C" block in Include/frameobject.h be active at the
lower level?

Whatever the correct answer is, I suspect the same constraints should
apply to all Include/cpython/*.h files.


You're correct:
- The #ifndef/#error make a clear point that a include/cpython/*.h file 
is only to be included as part of its include/*.h
- The inner extern "C" block overrides the outer one. If they're the 
same, it's useless.


But I'd call it a minor issue – a style issue, even. Probably not worth 
worrying about.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PFQC4YIDJBKRFI77H42DPTTSRP6G44DK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Last call for comments on PEP 573 (Module State Access from C Extension Methods)

[Petr Viktorin]

On 2020-01-03 14:36, Petr Viktorin wrote:

On 2019-11-25 13:15, Stefan Behnel wrote:

Hi all,

I think that PEP 573 is ready to be accepted, to greatly improve the 
state

of extension modules in CPython 3.9.

https://www.python.org/dev/peps/pep-0573/

It has come a long way since the original proposal and went through 
several

iterations and discussions by various interested people, effectively
reducing its scope quite a bit. So this is the last call for comments on
the latest version of the PEP, before I will pronounce on it. Please keep
the discussion in this thread.


I have received substantial private feedback, and I'll need to make 
another update to:


## Better separate C-API additions:[...]  from CPython implementation details:
## Clarify that ht_module is not inherited
## Specify what "defining class" means more rigorously


The update is here: https://github.com/python/peps/pull/1275

It also includes a more drastic change: it removes the MRO walker from 
the proposal.
Reflecting on the feedback, it became clear to me that a MRO walker, as 
it was described, won't give correct results in all cases: specifically, 
is a slot is overridden by setting a special method from Python code, 
the walker won't be able to find module. Think something like:

c_add = Class.__add__  # where nb_add uses the MRO walker
Class.__add__ = lambda *args: "evil"
c_add(Class(), 0)  # Exception: Defining type has not been found.

This can be solved, but it will need a different design and more 
discussion. I'd rather defer it to the future.
Meanwhile, extension authors can use their own MRO walker if they're OK 
with some surprising edge cases.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQAD3GLDHE6Z7SZ2R6ML57GSN5N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't venv also install python3*-config?

[Petr Viktorin]

On 2020-01-08 17:53, Victor Stinner wrote:

Hi,

I dislike python-config for multiple reasons

You may get the wrong information if you pick the wrong python-config script :-(

IMHO we should add a new module (problem: how should it be called?
pyconfig?) which could be used using "python3 -m xxx ...". There is a
similar discussion between "pip ..." and "python3 -m pip ..."
commands: I understand that "python3 -m pip ..." is more reliable to
know which Python will be picked, especially in a virtual environment.


Indeed. It's a bad idea to have separate executables that use/affect 
"the Python" or are needed to work with "the Python", but don't 
precisely specify "the Python". And `python -m ...` is the most sane way 
out of the mess.



There are two implementations: one in Python used on macOS, one in
shell used on other platforms.

Moreover, if it's implemented in the stdlib, it can be implemented in
Python and it should be easier to maintain than a shell script. For
example, python-config has no unit test...


Could sysconfig be extended? Currently it doesn't respond to CLI flags; 
it should be possible to add config-like ones, e.g.:


python -m sysconfig --libs
python -m sysconfig --help

Would that help?
I must admit I don't really understand python-config. Does anyone know 
its purpose? History? Documentation? One thing I suspect is that it was 
meant as a helper for pkg-config, not to be used alone.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FTKE7GRT4IWHJOBX5JPZQXTR42ZT6ZH6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Last call for comments on PEP 573 (Module State Access from C Extension Methods)

[Petr Viktorin]

On 2019-11-25 13:15, Stefan Behnel wrote:

Hi all,

I think that PEP 573 is ready to be accepted, to greatly improve the state
of extension modules in CPython 3.9.

https://www.python.org/dev/peps/pep-0573/

It has come a long way since the original proposal and went through several
iterations and discussions by various interested people, effectively
reducing its scope quite a bit. So this is the last call for comments on
the latest version of the PEP, before I will pronounce on it. Please keep
the discussion in this thread.


I have received substantial private feedback, and I'll need to make 
another update to:


## Better separate C-API additions:

- PyType_FromModuleAndSpec
- PyType_GetModule
- PyType_DefiningTypeFromSlotFunc
- PyType_GetModuleState

- METH_METHOD flag
- PyCMethod (function signature)

from CPython implementation details:

- ht_module (member of the undocumented PyHeapTypeObject)
- PyCMethodObject (extends the undocumented  PyCFunctionObject; used 
internally to hold information to pass to PyCMethod)

- PyCFunction_GET_CLASS (helper for the internal class)
- defining_class clinic converter

(See 
https://mail.python.org/archives/list/capi-...@python.org/message/B2VDVLABM4RQ4ATEJXFZYWEGTBZPUBKW/ 
for a proposal to formalize this distinction as "rings", and for some 
reasons why it is good.)


## Clarify that ht_module is not inherited

## Specify what "defining class" means more rigorously
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/336BSKX6NTW6FBB5JCFZOHVBI2PUF2MW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python Documentation and AIX specifics - how to proceed?

[Petr Viktorin]

On 2019-12-26 14:35, Michael wrote:

First - best wishes all for a happy and healthy 2020!

As my nickname implies - my primary means to contribute to Python is
with regard to AIX. One of the things I recently came across is
Misc/README.AIX which was last updated sometime between 2010 and 2014. I
am thinking a facelift is in order. Assuming 2010-2011 as the original
date - much has changed and many of the comments are no longer accurate.

Before saying so, I would like to check here that - having all tests
pass on the 3.8 bot implies that there are no outstanding issues. As to
the historical issues in the current document - these could either be
deleted, or a short text describing when they were resolved (e.g.,
Python 3.7, or just resolved, but noone knows exactly how or when -
whether it was a Python change, or a platform (AIX) change.
Delete them. The README's history is in Git. If you know additional 
details (like when an issue was resolved), you can say that in the 
commit message.


What I see as being more relevant is the description re: how to build
Python for AIX - for the "do it youself"-ers.
So, besides the direct question re: what to say about the old "known
issues" and whether there are no known issues aka, no issues identified
via the standard tests - I would appreciate feedback on what is
considered appropriate - anno 2020 - for any Misc/README.platform text.

Additionally, I am willing to work on other areas of the standard
documentation where it is either needed or considered appropriate for
platform specific details and/or examples.

This is not something I would try to get done in a single PR, Instead I
am thinking a single -longer term- issue - and multiple PR's to work
through corrections and additions during 2020. Focus on Python 3.9 and
beyond yet where appropriate backlevel to Python 3.8 or even 3.7.

Sincerely,

Michael



Some devs have been setting GitHub repositories for larger projects, 
with issues (TODO lists), collaboration, wikis or even blog-like 
documentation. For example:


https://github.com/ericsnowcurrently/multi-core-python
https://faster-cpython.readthedocs.io/
https://github.com/python/typing
https://github.com/python/asyncio

It helps to have a space where you can focus on a task without having to 
prove that "this is good for CPython right now" *all* the time.


You could replace README.AIX text with a link there. Afterwards, I think 
the devguide or Python documentation could link to such a space for the 
AIX build instructions; then, when finalized, the instructions 
themselves could be in Python docs or in a new README.AIX. But I do 
recommend to separate writing the text from proposing it for inclusion 
in CPython.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y4I7CDFIRD7H64THTMHWWHIJ7GCXEWSH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Adding a scarier warning to object.__del__?

[Petr Viktorin]

Could there be a clearer separation between the language semantics and 
current CPython behavior?


On 2020-01-02 03:53, Yonatan Zunger wrote:

Oh, I'm absolutely thinking about clarity. Something like:



This method is called when the instance is about to be destroyed. 
Because it may be called by either the ordinary Python execution flow or 
by the garbage collector, it has very unusual semantics, and must be 
treated with great care.



The precise semantics of when __del__ is called on an object are 
implementation-dependent. For example:
* It might be invoked during the normal interpreter flow at a moment 
like function return, if the reference count for an object has dropped 
to zero;
* It might be invoked by the garbage collector, which happens during the 
normal interpreter flow but is called from an arbitrary thread;
* It might be invoked during interpreter shutdown, after various global 
variables (including modules!) have already been deleted, or other 
threads' interpreters have already stopped.
* It may not be invoked at all; it is /not guaranteed/ that __del__() 
methods are called for objects that still exist when the interpreter exits.


Note that similar caveats apply for alternatives to __del__: 
weakref.finalize and atexit.register.


Doing any of the following in an implementation of __del__() may easily 
lead to bugs, nondeterministic behavior, and inconsistent behavior 
across different implementations or versions of Python:


* Assuming that any variables outside self exist, including 
modules or global variables;
* Attempting to acquire locks or otherwise block, since they may 
(e.g.) be invoked inside a thread which already holds those resources;
* Relying on being able to signal other threads and wait for their 
responses (even thread joins), since those threads' interpreters may 
have exited;
* Causing failures (e.g. leaving external devices in an invalid 
state) if __del__ is /never/ invoked.


In particular, del x does not call x.__del__() directly: the call may be 
delayed or not happen at all.


In CPython, del x decrements the reference count for x by one, and 
x.__del__() is only called either when x’s reference count reaches zero, 
or by the cyclic garbage collector at an unspecified time (e.g. if a 
reference cycle prevents the reference count from going to zero).


If a base class has a __del__() method, the derived class’s __del__() 
method, if any, must explicitly call it to ensure proper deletion of the 
base class part of the instance.


It is possible (though not recommended!) for the __del__() method to 
postpone destruction of the instance by creating a new reference to it. 
This is called object resurrection. It is implementation-dependent 
whether __del__() is called a second time when a resurrected object is 
about to be destroyed.
The current CPython implementation usually calls __del__ only once for 
each object, but this is not guaranteed in all cases.




On Wed, Jan 1, 2020 at 12:57 PM Andrew Svetlov > wrote:


If the warning text tells about the delayed execution -- I'm +1.
-1 for something like "just don't use __del__, it is dangerous".

On Wed, Jan 1, 2020, 21:51 Gregory P. Smith mailto:g...@krypto.org>> wrote:



On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov
mailto:andrew.svet...@gmail.com>> wrote:

__del__ is very useful not for interpreter shutdown but as a
regular
destructor in object's lifecycle.


The reason we should warn people against ever implementing
__del__ is that people rarely actually understand object
lifecycle.  Its presence guarantees delayed garbage collection
and that its code will code execute in a context way outside of
normal control flow that all other methods are invoked from.

-gps


Action on the shutdown is another beast.
Personally, I prefer to do all finalization works by
explicit calls instead.

On Wed, Jan 1, 2020 at 2:39 AM Yonatan Zunger
mailto:zun...@humu.com>> wrote:
 >
 > Hey everyone,
 >
 > I just encountered yet another reason to beware of
__del__: when it's called during interpreter shutdown, for
reasons which are kind of obvious in retrospect, if it calls
notify() on a threading.Condition, the waiting thread may or
may not ever actually receive it, and so if it does that and
then tries to join() the thread the interpreter may hang in
a hard-to-debug way.
 >
 > This isn't something that can reasonably be fixed, and
(like in most cases involving __del__) there's a very simple
fix of using weakref.finalize instead. My question for the
dev list: How would people feel about changing the
documentation for the method to more 

[Python-Dev] Re: Should set objects maintain insertion order too?

[Petr Viktorin]

On 2019-12-16 06:00, Larry Hastings wrote:
[...]


These are all performance concerns.  As I mentioned previously in this 
thread, in my opinion we should figure out what semantics we want for 
the object, then implement those, and only after that should we worry 
about performance.  I think we should decide the question "should set 
objects maintain insertion order?" literally without any consideration 
about performance implications.


Then this thread is missing arguments saying *why* ordered dicts are 
actually better, semantics-wise.


Originally, making dicts ordered was all about performance (or rather 
memory efficiency, which falls in the same bucket.) It wasn't added 
because it's better semantics-wise.

Here's one (very simplified and maybe biased) view of the history of dicts:

* 2.x: Dicts are unordered, please don't rely on the order.
* 3.3: Dict iteration order is now randomized. We told you not to rely 
on it!
* 3.6: We now use an optimized implementation of dict that saves memory! 
As a side effect it makes dicts ordered, but that's an implementation 
detail, please don't rely on it.
* 3.7: Oops, people are now relying on dicts being ordered. Insisting on 
people not relying on it is battling windmills. Also, it's actually 
useful sometimes, and alternate implementations can support it pretty 
easily. Let's make it a language feature! (Later it turns out 
MicroPython can't support it easily. Tough luck.)



By itself, "we already made dicts do it" is not a great argument in the 
set *semantics* debate.
Of course, it may turn out ordering was a great idea semantics-wise as 
well, but if I'm reading correctly, so far this thread has one piece of 
anectodal evidence for that.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RFGF4IJ3RHW5QCQGY5P6IUWE336D4OU5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python 3.8 error?

[Petr Viktorin]

I'm quite interested in the rest of the story here. PySide is probably 
the biggest open-source user of the limited API, so IMO it is relevant 
to this list.


On 2019-12-11 23:48, Christian Tismer wrote:

Hi all,

Sorry for the noise, I was wrong, and I retract.
I was somehow mislead and hunted a phantom.


Does that mean that there was never a problem?
Or that a problem is still there, but we don't know if it's in CPython 
or PySide or their interaction?

Or was the problem solved in PySide?
Do we need to document something better, or just aware of some caveat?



Best - Chris


On 10.12.19 00:29, Christian Tismer wrote:

On 09.12.19 23:26, Nick Coghlan wrote:



On Tue., 10 Dec. 2019, 5:17 am MRAB, mailto:pyt...@mrabarnett.plus.com>> wrote:

 On 2019-12-09 18:22, Christian Tismer wrote:
 >
 >
 > Hi Nick,
 >
 > after staring long at the code, I fount something funny in
 > typeobject.c #286 ff:
 >
 >
 > static void
 > type_mro_modified(PyTypeObject *type, PyObject *bases) {
 >      /*
 >         Check that all base classes or elements of the MRO of type are
 >         able to be cached.  This function is called after the base
 >         classes or mro of the type are altered.
 >
 >         Unset HAVE_VERSION_TAG and VALID_VERSION_TAG if the type
 >         has a custom MRO that includes a type which is not officially
 >         super type, or if the type implements its own mro() method.
 >
 >         Called from mro_internal, which will subsequently be called on
 >         each subclass when their mro is recursively updated.
 >       */
 >      Py_ssize_t i, n;
 >      int custom = (Py_TYPE(type) != _Type);
 >      int unbound;
 >      PyObject *mro_meth = NULL;
 >      PyObject *type_mro_meth = NULL;
 >
 >      if (!PyType_HasFeature(type, Py_TPFLAGS_HAVE_VERSION_TAG))
 >          return;
 >
 >      if (custom) {
 >          _Py_IDENTIFIER(mro);
 >          mro_meth = lookup_maybe_method(
 >              (PyObject *)type, _mro, );
 >          if (mro_meth == NULL)
 >              goto clear;
 >          type_mro_meth = lookup_maybe_method(
 >              (PyObject *)_Type, _mro, );
 >          if (type_mro_meth == NULL)
 >              goto clear;
 >          if (mro_meth != type_mro_meth)
 >              goto clear;
 >          Py_XDECREF(mro_meth);
 >          Py_XDECREF(type_mro_meth);
 >      }
 >
 >
 > Look at the "if (custom)" clause.
 > "mro_meth = lookup_maybe_method(" uses lookup_maybe_method which
 > gives a borrowed reference. The same holds for "type_mro_meth".
 >
 > But then both are decreffed, which IMHO is not correct.
 >
 Look at what happens at the label "clear": it DECREFs them.

 If mro_meth != NULL or mro_meth != type_mro_meth, they'll get DECREFed
 at "clear".


I believe Christian's point is that this entire "if (custom) {" branch
looks suspicious, as it assumes "lookup_maybe_method" will increment the
refcount on the returned object. If that assumption is incorrect, we're
going to get DECREFs without a preceding INCREF.

The specific code path is also obscure enough that it's plausible the
test suite may not currently cover it (as it requires doing something
that calls "type_mro_modified" on a type with a custom metaclass).


Thanks Nick for this nice analysis. And it's exactly this codepath that
is taken in the case of PySide: custom types all the time :-)

what-a-relief - ly y'rs -- Chris


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AONFRH53HED5RN7VVT375JDMORGKEFQV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why limit coroutines?

[Petr Viktorin]

On 2019-12-11 22:45, Antoine Pitrou wrote:

On Mon, 9 Dec 2019 21:42:36 -0500
Kyle Stanley  wrote:


(b) Why limit coroutines? It's just another Python object and has no

operating resources associated with it. Perhaps your definition of
coroutine is different, and you are thinking of OS threads?

This was my primary concern with the proposed PEP. At the moment, it's
rather trivial to create one million coroutines, and the total memory taken
up by each individual coroutine object is very minimal compared to each OS
thread.

There's also a practical use case for having a large number of coroutine
objects, such as for asynchronously:

[...]

2) Sending a large number of concurrent database transactions to run on a
cluster of database servers.


1M concurrent database transactions?  Does that sound reasonable at
all?  Your database administrator probably won't like you.


Right. Instead, you use a pool of DB connections, making each coroutine 
await its turn to talk to the DB.

You can still have a million coroutines waiting.

(Or as Guido put it, a coroutine "just another Python object and has no 
operating resources associated with it". I still sense some confusion 
around what was meant there.)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3IJPT5OH6ISHFTSRHSJU5E6DA5WJCJ74/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Pass the Python thread state to internal C functions

[Petr Viktorin]

On 2019-11-12 23:03, Victor Stinner wrote:

Hi,

Are you ok to modify internal C functions to pass explicitly tstate?


In short, yes, but:
- don't make things slower :)
- don't break the public API or the stable ABI


I'm a fan of explicitly passing state everywhere, rather than keeping it 
in "global" variables.


Currently, surprisingly many internal functions do a PyThreadState_GET 
for themselves, then call another function that does the same. That's 
wasteful, but impossible to change in the public API.
Your changes (of which I only saw a very limited subset) seem to follow 
a simple rule: public API functions call PyThreadState_GET, and then 
call internal functions that pass it around.
That's sounds beautifully easy to explain! Later, we'll just need to 
find a way to make the tstate API public (and opt-in).



The "per-interpreter None", however, is a different issue. I don't see 
how that can be done without breaking the stable ABI. I still think 
immortal immutable objects could be shared across interpreters.






--

I started to modify internal C functions to pass explicitly "tstate"
when calling C functions: the Python thread state (PyThreadState).
Example of C code (after my changes):

 if (_Py_EnterRecursiveCall(tstate, " while calling a Python object")) {
 return NULL;
 }
 PyObject *result = (*call)(callable, args, kwargs);
 _Py_LeaveRecursiveCall(tstate);
 return _Py_CheckFunctionResult(tstate, callable, result, NULL);

In Python 3.8, the tstate is implicit:

 if (Py_EnterRecursiveCall(" while calling a Python object")) {
 return NULL;
 }
 PyObject *result = (*call)(callable, args, kwargs);
 Py_LeaveRecursiveCall();
 return _Py_CheckFunctionResult(callable, result, NULL);

There are different reasons to pass explicitly tstate, but my main
motivation is to rework Python code base to move away from implicit
global states to states passed explicitly, to implement the PEP 554
"Multiple Interpreters in the Stdlib". In short, the final goal is to
run multiple isolated Python interpreters in the same process: run
pure Python code on multiple CPUs in parallel with a single process
(whereas multiprocessing runs multiple processes).

Currently, subinterpreters are a hack: they still share a lot of
things, the code base is not ready to implement isolated interpreters
with one "GIL" (interpreter lock) per interpreter, and to run multiple
interpreters in parallel. Many _PyRuntimeState fields (the global
_PyRuntime variable) should be moved to PyInterpreterState (or maybe
PyThreadState): per interpreter.

Another simpler but more annoying example are Py_None and Py_True
singletons which are globals. We cannot share these singletons between
interpreters because updating their reference counter would be a
performance bottleneck. If we put a "superglobal-GIL" to ensure that
Py_None reference counter remains consistent, it would basically
"serialize" all threads, rather than running them in parallel.

The idea of passing tstate to internal C functions is to prepare code
to get the per-interpreter None from tstate.

tstate is basically the "root" to access all states which are per
interpreter. For example, PyInterpreterState can be read from
tstate->interp.

Right now, tstate is only passed to a few functions, but you should
expect to see it passed to way more functions later, once more
structures will be moved to PyInterpreterState.

--

On my latest merged PR 17052 ("Add _PyObject_VectorcallTstate()"),
Mark Shannon wrote: "I don't see how this could ever be faster, nor do
I see how it is more correct."
https://github.com/python/cpython/pull/17052#issuecomment-552538438

Currently, tstate is get using these internal APIs:

#define _PyRuntimeState_GetThreadState(runtime) \
 
((PyThreadState*)_Py_atomic_load_relaxed(&(runtime)->gilstate.tstate_current))
#define _PyThreadState_GET() _PyRuntimeState_GetThreadState(&_PyRuntime)

or using public APIs:

PyAPI_FUNC(PyThreadState *) PyThreadState_Get(void);
#define PyThreadState_GET() PyThreadState_Get()

I dislike _PyThreadState_GET() for 2 reasons:

* it relies on the _PyRuntime global variable: I would prefer to avoid
global variables
* it uses an atomic operation which can become a perofrmance issue
when more and more code will require tstate

--

An alternative would be to use PyGILState_GetThisThreadState() which
uses a thread local state (TLS) variable to get the Python thread
state ("tstate"), rather that _PyRuntime atomic variable. Except that
the PyGILState API doesn't support subinterpreters yet :-(

https://bugs.python.org/issue15751 "Support subinterpreters in the GIL
state API" is open since 2012.

Note: While the GIL is released, _PyThreadState_GET() is NULL, whereas
PyGILState_GetThisThreadState() is non-NULL.

--

Links:

* https://pythoncapi.readthedocs.io/runtime.html : my notes on moving
globals to per interpreter states
* 

[Python-Dev] Belatedly accepting PEP 579 (design discussion leading to vectorcall)

[Petr Viktorin]

I approve [PEP 579], with the understanding that acceptance of an 
Informational PEP doesn't really mean that much.


PEP 579 charts the design issues addressed in PEP 575, PEP 580,
PEP 590 (and hopefully more upcoming proposals).

As noted in PEP 1:

   Informational PEPs do not necessarily represent a Python community
   consensus or recommendation, so users and implementers are free to
   ignore Informational PEPs or follow their advice.

While there is no consensus on whether the issues or the solutions in
PEP 579 are valid, the list is still useful to guide further design.

I plan to note that in the PEP itself to inform drive-by readers: 
https://github.com/python/peps/pull/1233/files



I should have done this ages ago, but it fell off my TODO list. Thanks 
to Brett for reminding me!



[PEP 579]: https://www.python.org/dev/peps/pep-0579/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6DWOBYFL4VNPFZHVU2IUYYMS7NSEUNMR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Steering Council Update (July through October)

[Petr Viktorin]

[PEP 484](https://www.python.org/dev/peps/pep-0484/) policy for inclusion of 
stubs in typeshed was discussed and the 
[PR](https://github.com/python/peps/pull/1218/) around it. It was decided that 
a sentence would be changed to make type stubs an opt-out situation and not 
opt-in. 
- [PEP 608](https://www.python.org/dev/peps/pep-0608/) and 
[discussion](https://discuss.python.org/t/rfc-pep-608-coordinated-python-release/2539)

On November 9, 2019 5:25:53 PM GMT+01:00, Ewa Jodlowska  wrote:
>Hey folks -
>
>Here is an update from the Steering Council. It covers July through
>October.
>
>The update has been added to the Steering Council repo as well:
>https://github.com/python/steering-council/blob/master/updates/2019-11-09-steering-council-update.md
>.
>
># Steering Council Community Update
>July 2019 through October 2019
>
>
>## July recap:
>- PSF and the Steering Council worked on a proposal to hire a project
>manager to help with Python 2 EOL communication. The proposal was
>approved
>by the PSF Board and Changeset Consulting was contracted to help with
>this
>work. Progress is being tracked [here](
>https://github.com/orgs/python/projects/1#card-26091749)
>- Release cadence discussions happened.
>- [PEP 1](https://www.python.org/dev/peps/pep-0001/)’s
>copyright/license
>section was updated to include a dual license of public domain and
>CC0-1.0-Universal.
>
>
>## August recap:
>- [PEP 572](https://www.python.org/dev/peps/pep-0572/) was updated to
>replace `TargetScopeError` to `SyntaxError`:
>https://github.com/python/peps/commit/cd292d89972476fa485a8ac1b14c1ed85b8c4867
>- Vision document review and update happened.
>- Release cadence discussions happened.
>- Steering Council helped answer inquiries about proposed Python 2 EOL
>page.
>- Steering Council reviewed the newly approved PSF Code of Conduct.
>
>
>## September recap:
>- Slide deck “Vision Doc” was created based off of the original Vision
>Document and was discussed by the Steering Council. The Steering
>Council
>decided to continue in this direction.
>- The Steering Council discussed the Core Dev promotion topic and it
>was
>suggested by the Steering Council that the Core Devs create a Work
>Group to
>work on that:
>https://discuss.python.org/t/concerns-about-how-we-vote-on-new-core-developers/2351/19
>- A timeline was selected for the Steering Council election and
>informed
>the Core Devs:
>https://mail.python.org/archives/list/python-committ...@python.org/thread/PLDUMAJOLHM5W6FWOMSGUBN2Q7Z6D3JM/
>- Release cadence discussions happened
>- [Core Dev Sprint in London recap](
>http://pyfound.blogspot.com/2019/10/cpython-core-developer-sprint-2019.html)
>
>
>## October recap:
>- Steering Council discussed the new Code of Conduct and agreed that it
>should be sent out to the python-committers@ and python-dev@ mailing
>lists:
>https://mail.python.org/archives/list/python-committ...@python.org/thread/6QIMLZ65D2HVBRGXRMOF2KOLFRXH4IRG/
>- Steering Council plans on presenting the Vision Deck at PyCon pending
>any
>changes made by future Steering Council.
>- Steering Council discussed the proposal made by [Thomas Wouters](
>https://mail.python.org/archives/list/python-committ...@python.org/thread/UWOU47BVULIOZHEOFWGVVQG6LPEXTDNG/)
>for approval voting. The Steering Council guided Thomas in the
>direction of
>a [PR](https://github.com/python/peps/pull/1197) and a formal vote.
>That
>resulted in a change to [PEP
>13](https://www.python.org/dev/peps/pep-0013/)
>stating core team members can vote for zero or more of the candidates
>for
>the Steering Council.
>- Steering Council discussed communal ownership of Python repo. It was
>decided that the Steering Council will send an email to
>python-committers
>about the topic. Brett sent the email on Nov 6:
>https://mail.python.org/archives/list/python-committ...@python.org/message/DCKAPZVXEQHP7OPS3GLEVNDXMXX5QAE6/
>.
>- Steering Council met with GitHub team about the b.p.o. migration, PEP
>[581](https://www.python.org/dev/peps/pep-0581/)/[588](
>https://www.python.org/dev/peps/pep-0588/) plan and discussed next
>steps.
>Next steps for Python include finding a point person to lead the work
>(will
>be a project manager), the Steering Council to find out more
>information
>about search features and how mapping will work.
>- The SC reviewed and discussed PEP 602 (1 year release cycle proposal
>by
>Łukasz Langa) and PEP 605 (Nick Coghlan and Steve Dower's 2 year
>proposal)
>and selected [PEP 602](https://www.python.org/dev/peps/pep-0602/).
>- [PEP 484](https://www.python.org/dev/peps/pep-0484/) policy for
>inclusion
>of stubs in typeshed was discussed and the [PR](
>https://github.com/python/peps/pull/1218/) around it. It was decided
>that a
>sentence would be changed to make type stubs an opt-out situation and
>not
>opt-in.
>- [PEP 608](https://www.python.org/dev/peps/pep-0608/) and
>[discussion](
>https://discuss.python.org/t/rfc-pep-608-coordinated-python-release/2539)
>were reviewed by the Steering Council. It 

[Python-Dev] Re: Getting function arguments of PyCFunctionObject

[Petr Viktorin]

On 2019-10-19 00:31, Sümer Cip wrote:

Hi everyone,

I have been working on some feature of a deterministic 
profiler(github.com/sumerc/yappi ). The 
feature is about getting arguments for a given set of function names. 
For example: you can define something like foo 1,3,arg_name and when foo 
function is called, profiler will simply collect the first, third from 
*args and a named argument arg_name from *kwargs.


For Python functions I am using following approach: Please note that the 
code below is executing in the PyTrace_CALL event of the profiler in C side:
Look co_argcount and co_varnames to determine the names of arguments and 
then use these names to retrieve values from f_locals. It seems to be 
working fine for now. My first question is: Is there a better way to do 
this?


And for C functions, I am totally in dark. I have played with 
f_valuestack and retrieve some values from there but the indexes seem to 
change from Python version to version and also I think there is no way 
of getting named arguments...


I have been dealing with this for a while, so there might be unclear 
points in my explanation. I would really appreciate if anyone can help 
me on a correct direction on this.




Hi,
The best way we know of getting a callable's arguments is 
inspect.signature: 
https://docs.python.org/3.5/library/inspect.html#inspect.signature


Of course, some callables can lie about their arguments, some might have 
information missing, and sometimes there even are good reasons to do 
such things.
If you can't use inspect.signature directly, your best bet is probably 
to study its implementation. It's complex but hopefully well-enough 
commented. Just be aware it can change in the future, as we find better 
ways of documenting and finding signatures. And if you find a possible 
improvement, send it here!

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UR6S46WVGAVXSYKG5736VC24LJ2BHSID/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Petr Viktorin]

On 2019-09-30 00:33, Nick Coghlan wrote:
[...]
We'll never change the size of the config structs (or any other public 
struct) in a maintenance branch


But we *will* change it in alphas/betas, and then we will ask people to 
try their software out with these.
Of course, there are no actual API/ABI guarantees with alphas/betas: 
officially, you need to rebuild everything for each new pre-release. But 
with the goal being to find bugs rather than provide correct software, 
it's OK to take shortcuts (if you can then distinguish the bugs the 
shortcuts cause from actual bugs in the code).


Issuing proper version mismatch errors (instead of random cryptic 
segfaults) could make for a much nicer experience when testing several 
successive alphas/betas.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LEXOES2PE634F5JJQJGXFTFATX3CY3PA/

[Python-Dev] Re: What to do about invalid escape sequences

[Petr Viktorin]

On 8/15/19 10:40 AM, Greg Ewing wrote:

If we want a truly raw string format that allows all characters,
including any kind of quote, we could take a tip from Fortran:

     s = 31HThis is a "totally raw" string!


Or from Rust:

let s = r"Here's a raw string";
let s = r#"Here's a raw string with "quotes" in it"#;
let s = r##"Here's r#"the raw string syntax"# in raw string"##;
let s = r###"and here's a '##"' as well"###;
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AB565Z6Z6VVUQR74VWLRCA6R2J6NZGAP/

[Python-Dev] Re: typing: how to use names in result-tuples?

[Petr Viktorin]

On 7/29/19 4:36 PM, Christian Tismer wrote:

Hi friends,

I am meanwhile the PySide maintainer at The Qt Company,
and we are trying to make the mapping from Qt functions
to Python functions as comparable as possible.

One problem are primitive pointer variables:
In Qt, it is natural to use "sometype *varname" to make a
mutable variable. In Python, you have to turn such a thing
into a result-tuple. Example:

 void QPrinter::getPageMargins(qreal *left, qreal *top, \
 qreal *right, qreal *bottom, QPrinter::Unit unit) const

(meanwhile deprecated, but a good example)

is mapped to the Python signature

 def getPageMargins(self, \
 unit: PySide2.QtPrintSupport.QPrinter.Unit) \
 -> typing.Tuple[float, float, float, float]: ...

NOW my question:


I would like to retain the variable names in Python!
The first idea is to use typing.NamedTuple, but I get the impression
that this would be too much, because I would not only need to create
an extra named tuple definition for every set of names, but also
invent a name for that named tuple.

What I would like to have is something that looks like

 def getPageMargins(self, \
 unit: PySide2.QtPrintSupport.QPrinter.Unit) \
 -> typing.NamedTuple[left: float, top: float, \
  right:float, bottom:float]: ...

but that is obviously not a named tuple.
Possible would be to derive a name from the function, so maybe
a definition could be used like

class PageMargingResult(NamedTuple):
 left: float
 top: float
 right: float
 bottom: float

but then I would have some opaque PageMargingResult type. This would
work, but as said, this is a bit too much, since I only
wanted something like a tuple with names.

What do you suggest to do here? Something what I am missing?


Hello,
I'm afraid you're stuck with defining a new type for each of these.
One reason why that's better is that info about the names will only be 
stored once, in the type; not in every instance.


Since this is in PySide, I assume you're using the C-API. If that's the 
case, check out Struct Sequences, the "C equivalent of named tuples". 
For example, the result of "os.stat()" is a struct sequence.


https://docs.python.org/3/c-api/tuple.html#struct-sequence-objects

Note that like namedtuples, these are immutable, and they're proper 
subclasses of tuple.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DKGRAP7LHZGOHQON4UKXUT2TAMRSXQD/

[Python-Dev] Re: Comparing dict.values()

[Petr Viktorin]

On 7/25/19 2:25 AM, David Mertz wrote:
Exactly! that was my thought that the exception message could hint at 
likely approaches. The NumPy example seems to have a good pattern:


arr1 == arr2

|ValueError:Thetruth value of an array withmore than one element |isambiguous.

|Usea.any()ora.all().|


It's not the equality operator that errors: `==` means element-wise 
comparison in Numpy.

The error would come from a conversion of the array to bool:

>>> numpy.array([1, 2, 3]) == numpy.array([1, 3, 4])
array([ True, False, False])

>>> if numpy.array([ True, False, False]):
... print('Same!')
...
Traceback (most recent call last):
  File "", line 1, in 
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()



Numpy currently returns False when `==` “doesn't make sense”, but 
apparently has plans to change that:


>>> numpy.array([1, 2, 3]) == numpy.array([1, 2])
__main__:1: DeprecationWarning: elementwise comparison failed; this will 
raise an error in the future.

False

>>> numpy.array([1, 2, 3]) == numpy.array(['a', 'b'])
__main__:1: FutureWarning: elementwise comparison failed; returning 
scalar instead, but in the future will perform elementwise comparison

False

>>> numpy.__version__
'1.16.4'





On Wed, Jul 24, 2019, 8:06 PM Rob Cliffe via Python-Dev 
mailto:python-dev@python.org>> wrote:




On 25/07/2019 00:09:37, David Mertz wrote:
 > I agree with Greg.
 >
 > There are various possible behaviors that might make sense, but
having
 > `d.values() != d.values()` is about the only one I can see no
sense in.
+1
 >
 > This really feels like a good cade for reading a descriptive
 > exception. If someone wants too compare `set(d.values())` that's
 > great. If they want `list(d.values())`, also a sensible question.
But
 > the programmer should spell it explicitly.
 >
 >
So, a helpful error message including something like "Cannot compare
dict.values directly, consider converting to sets / lists / sorted
lists
before comparing" ?
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/CSTSLCDEJYKDLADQV5PJRCSSVTMB5RIG/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2IMLY36A6K45LHCXO2OVQJZBQKL47U6T/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DRG757EQKEDNKPPM2KXMZ5NVVSBSGMCL/

[Python-Dev] Re: EuroPython 2019 sprints

[Petr Viktorin]

Hello,
I'll be at the EuroPython CPython sprints too, happy to help out.


On Thu, Jun 13, 2019 at 6:36 PM Steve Dower  wrote:
> Just letting everyone know that I signed up any core devs who are going
> to be at EuroPython to run sprints :) (including myself, since I'll be
> there)

FWIW, I read this as a done deal, not inviting a confirmation for me.
A [recent discourse post] suggests otherwise :)

[recent discourse post]:
https://discuss.python.org/t/europython-2019-talks-sprints/1900/5?u=encukou
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6BYA75HSN6RX6F5LWAYEKV6PBO3IHKXZ/

[Python-Dev] Re: Expected stability of PyCode_New() and types.CodeType() signatures

[Petr Viktorin]

On 6/12/19 7:40 AM, Stefan Behnel wrote:

Victor Stinner schrieb am 12.06.19 um 00:09:

So yeah, the PyCode_New() change is very annoying in practical, since
every single project using Cython requires a new release in practice.


I think Cython's deal with regard to this is:

"""
If you use Cython, we will try hard to cover up the adaptations for
upcoming (and existing) CPython versions for you. But you'll likely have to
rerun Cython on your project before a new CPython major release comes out.
"""

That's obviously not ideal for projects that end up being unmaintained. But
then again, you can't freeze time forever, and /any/ change to a dependency
can end up being fatal to them.

I personally think that rerunning Cython when a new CPython release comes
out is an acceptable price to pay for a project. In the worst case, this
can even be done by others, as you suggested as a common Fedora practice
elsewhere, Victor.


I hope this is something that improvements in Python's packaging story 
(specifically, PEP 518) should help with.
I see the current practice of including Cython's output in releases as a 
workaround for the fact that you can't (reasonably) specify Cython as a 
build dependency. Cython is a much lighter dependency than a C compiler 
-- though a less common one. When there's a reliable way to specify 
build-time dependencies, running Cython on each build will hopefully 
become the obvious way to do it.


Or is there something else blocking that future?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGWDJ44NSCAWP2IC7DNFBJRHQL2JHQTL/

[Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures

[Petr Viktorin]

Hello,
PEP 570 (Positional-Only Parameters) changed the signatures of 
PyCode_New() and types.CodeType(), adding a new argument for "posargcount".
Our policy for such changes seems to be fragmented tribal knowledge. I'm 
writing to check if my understanding is reasonable, so I can apply it 
and document it explicitly.


There is a surprisingly large ecosystem of tools that create code objects.
The expectation seems to be that these tools will need to be adapted for 
each minor version of Python.


But that's not the same as saying that if you use types.CodeType(), 
you're on your own. To me [PR 13271], seems to go too far when it adds:

> These types are not supposed to be instantiated outside of
> CPython internals and constructor signatures will vary
> between Python versions.

This kind of "washing our hands" doesn't do justice to tools like 
Cython, whose use of PyCode_New is (IMO) perfectly justified. As is 
adapting to Python minor versions.


So, we should document the changes as any backwards-incompatible change.
Specifically, changes should be (or should have been, in hindsight) 
mentioned in:

* The "API changes" and/or "Porting" sections of the What's New document
* Version history of the documentation (e.g. versionchanged blocks)
* Any relevant PEPs

Also, the expected level of API stability should be documented in the docs.

Does that sound right?


PEP 570: https://www.python.org/dev/peps/pep-0570/
PyCode_New: https://docs.python.org/3.8/c-api/code.html#c.PyCode_New
[PR 13271]: https://github.com/python/cpython/pull/13271/files
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590: vectorcall without tp_call

[Petr Viktorin]

On 5/29/19 3:36 PM, Jeroen Demeyer wrote:

On 2019-05-29 15:29, Petr Viktorin wrote:

That sounds like a good idea for PyType_FromSpec.


I don't think we're planning to support vectorcall in PyType_FromSpec 
for now. That's maybe for 3.9 when vectorcall is no longer provisional.



For static types I either wouldn't bother at all, or only check in debug
builds and fail with Py_FatalError.


So basically an assert(...)?


Yes. That's the usual way to let C extension authors know they did 
something wrong.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590: vectorcall without tp_call

[Petr Viktorin]

On 5/29/19 2:25 PM, Jeroen Demeyer wrote:

Hello,

I have one implementation question about vectorcall which is not 
specified in PEP 590: what should happen if a type implements vectorcall 
(i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set tp_call (i.e. 
tp_call == NULL)? I see the following possibilities:


1. Ignore this problem/assume that it won't happen. This would be bad, 
since callable(obj) would be False even though obj() would succeed.


2. Raise SystemError.

3. Automatically set tp_call to PyVectorcall_Call.

I would vote for 3 since it's the most user-friendly option. There is 
also no way how it could be wrong: it ensures that tp_call and 
vectorcall are consistent.


That sounds like a good idea for PyType_FromSpec.

For static types I either wouldn't bother at all, or only check in debug 
 builds and fail with Py_FatalError.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Accepting PEP 590 (Vectorcall)

[Petr Viktorin]

As BDFL-Delegate, I am accepting PEP 590 -- Vectorcall: a fast calling 
protocol for CPython.

https://www.python.org/dev/peps/pep-0590/

There was a major last-minute change to the text (though one that was 
discussed since PyCon US): The API is not finalized yet, and it's 
provided using private names (with leading underscores). We intend to 
use the Python 3.8 cycle to get feedback from consumers, and finalize it 
in 3.9.




Thanks to Jeroen Demeyer, who spent a lot of time pushing this forward, 
and probably has some new gray hair from interactions with me and other 
core devs. Without his PEP 580 (and its implementation), the simpler PEP 
590 would not be possible.
It might not be apparent if some of his interactions are taken out of 
context, but throughout the process Jeroen always worked to make Python 
better, and he acted professionally even as we made decisions he didn't 
agree with.


Thanks to Mark Shannon for providing major insights, patiently 
explaining that they actually are good ideas, and for the initial 
implementation.


Thanks to Stefan Behnel, Nick Coghlan, Victor Stinner and everyone else 
involved in PEPs 576, 579, 580 and 590.


I learned a lot in this process.

This is my first time as BDFL-delegate. If you have any criticism of the 
way I handled things, please share! (Assume I have tough skin, but keep 
to the code of conduct.) I'd be happy to know how I can improve.



And now, I'm off to get the implementation reviewed!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 (Vectorcall) discussion

[Petr Viktorin]

Discussion on PEP 590 (Vectorcall) has been split over several PRs, 
issues and e-mails, so let me post an update.


I am planning to approve PEP 590 with the following changes, if Mark 
doesn't object to them:


* https://github.com/python/peps/pull/1064 (Mark the main API as private 
to allow changes in Python 3.9)
* https://github.com/python/peps/pull/1066 (Use size_t for "number of 
arguments + flag")



The resulting text, for reference:

PEP: 590
Title: Vectorcall: a fast calling protocol for CPython
Author: Mark Shannon , Jeroen Demeyer 
BDFL-Delegate: Petr Viktorin 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Mar-2019
Python-Version: 3.8
Post-History:

Abstract


This PEP introduces a new C API to optimize calls of objects.
It introduces a new "vectorcall" protocol and calling convention.
This is based on the "fastcall" convention, which is already used 
internally by CPython.

The new features can be used by any user-defined extension class.

Most of the new API is private in CPython 3.8.
The plan is to finalize semantics and make it public in Python 3.9.

**NOTE**: This PEP deals only with the Python/C API,
it does not affect the Python language or standard library.


Motivation
==

The choice of a calling convention impacts the performance and 
flexibility of code on either side of the call.

Often there is tension between performance and flexibility.

The current ``tp_call`` [2]_ calling convention is sufficiently flexible 
to cover all cases, but its performance is poor.
The poor performance is largely a result of having to create 
intermediate tuples, and possibly intermediate dicts, during the call.
This is mitigated in CPython by including special-case code to speed up 
calls to Python and builtin functions.
Unfortunately, this means that other callables such as classes and third 
party extension objects are called using the

slower, more general ``tp_call`` calling convention.

This PEP proposes that the calling convention used internally for Python 
and builtin functions is generalized and published

so that all calls can benefit from better performance.
The new proposed calling convention is not fully general, but covers the 
large majority of calls.
It is designed to remove the overhead of temporary object creation and 
multiple indirections.


Another source of inefficiency in the ``tp_call`` convention is that it 
has one function pointer per class,

rather than per object.
This is inefficient for calls to classes as several intermediate objects 
need to be created.
For a class ``cls``, at least one intermediate object is created for 
each call in the sequence

``type.__call__``, ``cls.__new__``, ``cls.__init__``.

This PEP proposes an interface for use by extension modules.
Such interfaces cannot effectively be tested, or designed, without 
having the

consumers in the loop.
For that reason, we provide private (underscore-prefixed) names.
The API may change (based on consumer feedback) in Python 3.9, where we 
expect

it to be finalized, and the underscores removed.


Specification
=

The function pointer type
-

Calls are made through a function pointer taking the following parameters:

* ``PyObject *callable``: The called object
* ``PyObject *const *args``: A vector of arguments
* ``size_t nargs``: The number of arguments plus the optional flag 
``PY_VECTORCALL_ARGUMENTS_OFFSET`` (see below)
* ``PyObject *kwnames``: Either ``NULL`` or a tuple with the names of 
the keyword arguments


This is implemented by the function pointer type:
``typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject 
*const *args, size_t nargs, PyObject *kwnames);``


Changes to the ``PyTypeObject`` struct
--

The unused slot ``printfunc tp_print`` is replaced with 
``tp_vectorcall_offset``. It has the type ``Py_ssize_t``.

A new ``tp_flags`` flag is added, ``_Py_TPFLAGS_HAVE_VECTORCALL``,
which must be set for any class that uses the vectorcall protocol.

If ``_Py_TPFLAGS_HAVE_VECTORCALL`` is set, then ``tp_vectorcall_offset`` 
must be a positive integer.
It is the offset into the object of the vectorcall function pointer of 
type ``vectorcallfunc``.
This pointer may be ``NULL``, in which case the behavior is the same as 
if ``_Py_TPFLAGS_HAVE_VECTORCALL`` was not set.


The ``tp_print`` slot is reused as the ``tp_vectorcall_offset`` slot to 
make it easier for for external projects to backport the vectorcall 
protocol to earlier Python versions.
In particular, the Cython project has shown interest in doing that (see 
https://mail.python.org/pipermail/python-dev/2018-June/153927.html).


Descriptor behavior
---

One additional type flag is specified: ``Py_TPFLAGS_METHOD_DESCRIPTOR``.

``Py_TPFLAGS_METHOD_DESCRIPTOR`` should be set if the callable uses the 
descriptor protocol to create a bound method-like object.
This is used by the

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Petr Viktorin]

On 5/21/19 12:06 PM, Christian Heimes wrote:

On 21/05/2019 11.49, Nathaniel Smith wrote:

On Tue, May 21, 2019 at 2:40 AM Walter Dörwald  wrote:


On 20 May 2019, at 22:15, Christian Heimes wrote:


Hi,

here is the first version of my PEP 594 to deprecate and eventually
remove modules from the standard library. The PEP started last year
with talk during the Python Language Summit 2018,
https://lwn.net/Articles/755229/.

[...]

colorsys


The `colorsys `_
module
defines color conversion functions between RGB, YIQ, HSL, and HSV
coordinate
systems. The Pillow library provides much faster conversation between
color systems.

Module type
   pure Python
Deprecated in
   3.8
To be removed in
   3.10
Substitute
   `Pillow `_,
   `colorspacious `_


I'm using colorsys constantly as the basis for a tool that converts CSS
colors between different coordinate systems. I don't see how that could
be done via Pillow (which AFAICT only converts complete images).
RGB<->HSV<->HLS conversion seems to be not available (or not obvious) in
colorspacious.


Correct, colorspacious doesn't support HSV or HLS. I suppose it would
be trivial enough to add...

The 'colour' package has them (along with everything else you can
dream of): https://colour.readthedocs.io/en/latest/colour.models.html


Nice catch, I just added https://python-colormath.readthedocs.io/en/latest/ to 
my update PR. I'll add colour to the list, too.

(It didn't pop up on my radar because I wasn't checking for British spelling)

Christian



Please, don't remove colorsys. HSL and HSV aren't outdated, and AFAIK, 
the module is not a maintenance burden.


The packages on PyPI offer more advanced models. Perhaps they could be 
linked in the documentation, if we want to endorse them. But just as 
we're not removing `array` in favor of `numpy`, we shouldn't remove 
colorsys either. Colorsys is not dead, it's just small.


I assume one of the reasons colorspacious doesn't support HSV or HLS is 
that HSV/HLS tends to be used for quite different purposes than the more 
fancy color spaces. You'd use HSL/HSV for a simple color picker or a 
saturation shift on a web page, GUI or game (where you don't have 
*exact* colors anyway); things like sRGB or CIELab are for color 
management, photos, printing, etc.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-36829: Add sys.unraisablehook()

[Petr Viktorin]

On 5/16/19 3:23 AM, Victor Stinner wrote:
[...]

I modified my API to create an object to pack arguments. The new API
becomes sys.unraisablehook(unraisable) where unraisable has 4 fields:
exc_type, exc_value, exc_tb, obj.

[...]
I always thought the classic (exc_type, exc_value, exc_tb) triple is a 
holdover from older Python versions, and all the information is now in 
the exception instance.

Is the triple ever different from (type(exc), exc, exc.__traceback__)?
(possibly with a getattr for __traceback__)
Should new APIs use it?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 5/9/19 5:33 PM, Jeroen Demeyer wrote:

On 2019-05-09 20:30, Petr Viktorin wrote:

The underlying C function should not need to know how to extract
"self" from the function object, or how to handle the argument
offsetting.
Those should be implementation details.


Maybe you misunderstood my proposal. I want to allow both for extra 
flexibility:


- METH_FASTCALL (possibly combined with METH_KEYWORDS) continues to work 
as before. If you don't want to care about the implementation details of 
vectorcall, this is the right thing to use.


- METH_VECTORCALL (using exactly the vectorcallfunc signature) is a new 
calling convention for applications that want the lowest possible 
overhead at the cost of being slightly harder to use.


Then we can, in the spirit of minimalism, not add METH_VECTORCALL at all.


Personally, I consider the discussion about who is supposed to check 
that a function returns NULL if and if an error occurred a tiny detail 
which shouldn't dictate the design. There are two solutions for this: 
either we move that check one level up and do it for all vectorcall 
functions. Or, we keep the existing checks in place but we don't do that 
check for METH_VECTORCALL (this is already more specialized anyway, so 
dropping that check doesn't hurt much). We could also decide to enable 
this check only for debug builds, especially if debug builds are going 
to be easier to use thank to Victor Stinner's work.



I see the value in having METH_VECTORCALL equivalent to the existing
METH_FASTCALL|METH_KEYWORDS.


But why invent a new name for that? METH_FASTCALL|METH_KEYWORDS already 
works. The alias METH_VECTORCALL could only make things more confusing 
(having two ways to specify exactly the same thing). Or am I missing 
something?


METH_FASTCALL is currently not documented, and it should be renamed 
before it's documented. Names with "fast" or "new" generally don't age well.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

PEP 590 is on its way to be accepted, with some details still to be
discussed. I've rejected PEP 580 so we can focus on one place.

Here are things we discussed on GitHub but now seem to agree on:

* The vectorcall function's kwname argument can be NULL.
* Let's use `vectorcallfunc`, not `vectorcall`, and stop the bikeshedding.
* `tp_vectorcall_offset` can be `Py_ssize_t` (The discussions around
signedness and C standards and consistency are interesting, but
ultimately irrelevant here.)
* `PyCall_MakeTpCall` can be removed.
* `PyVectorcall_Function` (for getting the `vectorcallfunc` of an
object) can be an internal helper. External code should go through
`PyCall_Vectorcall` (whatever we name it).
* `PY_VECTORCALL_ARGUMENTS_OFFSET` is OK, bikeshedding over variants
like `PY_VECTORCALL_PREPEND` won't bring much benefit.

Anyone against, make your point now :)

The following have discussion PRs open:

* `PyCall_MakeVectorCall` name: https://github.com/python/peps/pull/1037
* Passing a dict to `PyObject_Vectorcall`:
https://github.com/python/peps/pull/1038
* Type of the kwnames argument (PyObject/PyTupleObject):
https://github.com/python/peps/pull/1039


The remaining points are:


### Making things private

For Python 3.8, the public API should be private, so the API can get
some contact with the real world. I'd especially like to be able to
learn from
Cython's experience using it.
That would mean:

* _PyObject_Vectorcall
* _PyCall_MakeVectorCall
* _PyVectorcall_NARGS
* _METH_VECTORCALL
* _Py_TPFLAGS_HAVE_VECTORCALL
* _Py_TPFLAGS_METHOD_DESCRIPTOR


### Can the kwnames tuple be empty?

Disallowing empty tuples means it's easier for the *callee* to detect
the case of no keyword arguments. Instead of:

if (kwnames != NULL && PyTuple_GET_SIZE(kwnames))

you have:

if (kwnames != NULL)

On the other hand, the *caller* would now be responsible for handling
the no-kwarg case specially.

Jeroen points out:
> The side of the caller (which should ensure not to send an empty tuple)
> is CPython and there the issue of people implementing the protocol wrongly
> doesn't arise.
> External C code is not expected to manually use tp_vectorcall_offset to make
> vectorcalls: it is expected to use an API like PyCall_Vectorcall() and that
> API will ensure to replace an empty tuple with NULL.
>
> I see it as an application of 
> https://en.wikipedia.org/wiki/Robustness_principle
> (Be conservative in what you send, be liberal in what you accept):
> PyCall_Vectorcall should accept an empty tuple but it should not send an
> empty tuple to the vectorcall function.

But, if you apply the robustness principle to vectorcallfunc, it
should accept empty tuples.


### `METH_VECTORCALL` function type

Jeroen suggested changing this from:

`PyObject *(*call) (PyObject *self, PyObject *const *args,
Py_ssize_t nargs, PyObject *kwname)`

to `vectorcallfunc`, i.e.:

`PyObject *(*call) (PyObject *callable, Py_ssize_t n, PyObject
*const *args, PyObject *kwnames)`

Mark argues that this is a major change and prevents the interpreter
from sanity checking the return value of PyMethodDef defined
functions.
(Since the functions are defined by extension code, they need to be
sanity-checked, and this will be done by PyCFunction's vectorcall
adapter. Tools like Cython can bypass the check if needed.)

The underlying C function should not need to know how to extract
"self" from the function object, or how to handle the argument
offsetting.
Those should be implementation details.

I see the value in having METH_VECTORCALL equivalent to the existing
METH_FASTCALL|METH_KEYWORDS.
(Even though PEP 573 will need to add to the calling convention.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hello

[Petr Viktorin]

On 5/7/19 9:08 PM, Ben Kane wrote:
Sorry Victor, I must have misinterpreted this snippet from the mailbot. 
Maybe it would be a good idea to re-word it? I'll be happy to open an 
issue for that in an appropriate mailing list if you think that would help.


 > Welcome to the Python-Dev@python.org  
mailing list! If you are a new subscriber, please take the time to 
introduce yourself briefly in your first post. It is appreciated if you 
lurk around for a while before posting! :-)


Not sure what the original intent was, but I read that as: "Whenever you 
get to sending your first post, include an introduction in it. But, 
before you start actively participating, take your time to hang around 
and absorb the culture."


Perhaps we want to reverse the two sentences to put them in 
chronological order?


I'm not sure what the proper forum for this discussion would be.

I talked to Barry Warsaw about https://bugs.python.org/issue36837 and he 
said I should post to python-dev to get feedback. I'll work on that 
email and post back here once I'm happy with it.


Thanks,
Ben


On Tue, May 7, 2019 at 4:47 PM Victor Stinner > wrote:


Welcome Ben!

Le mar. 7 mai 2019 à 16:28, Ben Kane mailto:bbk1...@gmail.com>> a écrit :
 > My name is Ben Kane. I'm joining this mailing list and
introducing myself as asked in the "welcome to python-dev" email.
I've been using Python for maybe 7 years now. I started to learn it
in college to help with accounting homework and I continue to use it
now for personal projects ranging from small scripts to larger
applications and in my day job as a LinkedIn Site Reliability Engineer.
 >
 > I'd like to contribute back to the language that's helped me so
much; and talking with core devs at PyCon 2019 has really encouraged
me. So I'll be lurking and trying to pick things up as I see them.

python-dev is not the proper mail to introduce yourself to contribute.
I suggest you to start reading http://devguide.python.org/ and please
send the same email to core-mentorship mailing list:
https://mail.python.org/mailman3/lists/core-mentorship.python.org/?x=22

It would help if you can elaborate on which parts of Python you would
be interested to contribute ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.




--

Ben Kane

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 5/6/19 3:43 AM, Jeroen Demeyer wrote:

On 2019-05-06 00:04, Petr Viktorin wrote:

- Single bound method class for all kinds of function classes: This
would be a cleaner design, yes, but I don't see a pressing need. As
PEP 579 says, "this is a compounding issue", not a goal. As I recall,
that is the only major reason for CCALL_DEFARG.


Just a minor correction here: I guess that you mean CCALL_SELFARG. The 
flag CCALL_DEFARG is for passing the PyCCallDef* in PEP 580, which is 
mostly equivalent to passing the callable object in PEP 590.


The signature of PEP 580 is

func(const PyCCallDef *def, PyObject *self, PyObject *const *args, 
Py_ssize_t nargs, PyObject *kwnames)


And with PEP 590 it is

func(PyObject *callable, PyObject *const *args, Py_ssize_t nargs, 
PyObject *kwnames)


with the additional special role for the PY_VECTORCALL_ARGUMENTS_OFFSET 
bit (which is meant to solve the problem of "self" in a different way).


I worded that badly, sorry.

From PEP 590's `callable`, the called function can get any of these if 
it needs to (and if they're stored somewhere). But you can't write 
generic code would get them from any callable.


If we're not going for the "single bound method class" idea, that is OK; 
`def` & `self` can be implementation details of the callables that need 
them.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 5/6/19 4:24 AM, Jeroen Demeyer wrote:

Hello Petr,

Thanks for your time. I suggest you (or somebody else) to officially 
reject PEP 580.


I'll do that shortly.
I hope that you are not taking this personally. PEP 580 is a good 
design. PEP 590 even says that it's built on your ideas.


I start working on reformulating PEP 590, adding some elements from PEP 
580. At the same time, I work on the implementation of PEP 590. I want 
to implement Mark's idea of having a separate wrapper for each old-style 
calling convention.


In the mean time, we can continue the discussion about the details, for 
example whether to store the flags inside the instance (I don't have an 
answer for that right now, I'll need to think about it).


I'm abandoning per-instance flag proposal. It's an unnecessary 
complication; per-type flags are fine.


Petr, did you discuss with the Steering Council? It would be good to 
have some kind of pre-approval that PEP 590 and its implementation will 
be accepted. I want to work on PEP 590, but I'm not the right person to 
"defend" it (I know that it's worse in some ways than PEP 580).


As BDFL-delegate, I'm "pre-approving" PEP 590.
I mentioned some details of PEP 590 that still need attention. If there 
are any more, now's the time to bring them up.


And yes, I know that in some ways it's worse than PEP 580. That's what 
makes it a hard decision.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

Hello!
Sorry for the delay; PyCon is keeping me busy. On the other hand, I
did get to talk to a lot of smart people here!

I'm leaning toward accepting PEP 590 (with some changes still). Let's
start focusing on it.
As for the changes, I have these 4 points:

I feel that the API needs some contact with real users before it's set
in stone. That was the motivation behind my proposal for PEP 590 with
additional flags.
At PyCon, Nick Coghlan suggested another option make the API
"provisional":  make it formally private.
Py_TPFLAGS_HAVE_VECTORCALL would be underscore-prefixed, and the docs
would say that it can change. in Python 3.9, the semantics will be
finalized and the underscore removed.
This would allow high-maintenance projects (like Cython) to start
using it and give their feedback, and we'd have a chance to respond to
the feedback.

tp_vectorcall_offset should be what's replacing tp_print in the
struct. The current implementation has tp_vectorcall there. This way,
Cython can create vectorcall callables for older Pythons. (See PEP
580: https://www.python.org/dev/peps/pep-0580/#replacing-tp-print).

Subclassing should not be forbidden. Jeroen, do you want write a
section for how subclassing should work?

Given Jeroen's research and ideas that went into the PEP (and
hopefully, we'll incorporate some PEP 580 text as well), it seems fair
to list him as co-author of the accepted PEP, instead of just listing
PEP 580 in the acknowledgement section.


On some other points:

- Single bound method class for all kinds of function classes: This
would be a cleaner design, yes, but I don't see a pressing need. As
PEP 579 says, "this is a compounding issue", not a goal. As I recall,
that is the only major reason for CCALL_DEFARG.
PEP 590 says that x64 Windows passes 4 arguments in registers.
Admittedly, I haven't checked this, nor the performance implications
(so this would be a good point to argue!), but it seems like a good
reason to keep the argument count down. So, no CCALL_DEFARG.

- In reply to this Mark's note:
> PEP 590 is fully universal, it supports callables that can do anything with 
> anything. There is no need for it to be extended because it already supports 
> any possible behaviour.

I don't buy this point. The current tp_call also supports any possible
behavior. Here we want to support any behavior *efficiently*.
As a specific example: for calling PEP 590 callable with a kwarg dict,
there'll need to be an extra allocation. That's inefficient relative
to PEP 580 (or PEP 590 plus allowing a dict in "kwnames"). But I'm
willing to believe the inefficiency is acceptable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 4/25/19 5:12 AM, Jeroen Demeyer wrote:

On 2019-04-25 00:24, Petr Viktorin wrote:

PEP 590 defines a new simple/fast protocol for its users, and instead of
making existing complexity faster and easier to use, it's left to be
deprecated/phased out (or kept in existing classes for backwards
compatibility). It makes it possible for future code to be 
faster/simpler.


Can you elaborate on what you mean with this deprecating/phasing out?


Kept for backwards compatibility, but not actively recommended or 
optimized. Perhaps made slower if that would help performance elsewhere.



What's your view on dealing with method classes (not necessarily right 
now, but in the future)? Do you think that having separate method 
classes like method-wrapper (for example [].__add__) is good or bad?


I fully agree with PEP 579's point on complexity:

There are a huge number of classes involved to implement all variations of 
methods. This is not a problem by itself, but a compounding issue.


The main problem is that, currently, you sometimes need to care about 
this (due to CPython special casing its own classes, without fallback to 
some public API). Ideally, what matters is the protocols the class 
implements rather than the class itself. If that is solved, having so 
many different classes becomes curious but unimportant -- merging them 
shouldn't be a priority.


I'd concentrate on two efforts instead:

- Calling should have a fast public API. (That's this PEP.)
- Introspection should have well-defined, consistently used public API 
(but not necessarily fast).


For introspection, I think the way is implementing the necessary API 
(e.g. dunder attributes) and changing things like inspect, traceback 
generation, etc. to use them. CPython's callable classes should stay as 
internal implementation details. (Specifically: I'm against making them 
subclassable: allowing subclasses basically makes everything about the 
superclass an API.)


Since the way how PEP 580 and PEP 590 deal with bound method classes is 
very different, I would like to know the roadmap for this.


My thoughts are not the roadmap, of course :)


Speaking about roadmaps, I often use PEP 579 to check what I'm 
forgetting. Here are my thoughts on it:



## Naming (The word "built-in" is overused in Python)

This is a social/docs problem, and out of scope of the technical 
efforts. PEPs should always define the terms they use (even in the case 
where there is an official definition, but it doesn't match popular usage).



## Not extendable

As I mentioned above, I'm against opening the callables for subclassing. 
We should define and use protocols instead.



## cfunctions do not become methods

If we were designing Python from scratch, this should have been done 
differently.
Now this is a problem for Cython to solve. CPython should provide the 
tools to do so.



## Semantics of inspect.isfunction

I don't like inspect.isfunction, because "Is it a function?" is almost 
never what you actually want to ask. I'd like to deprecate it in favor 
of explicit functions like "Does it have source code?", "Is it 
callable?", or even "Is it exactly types.FunctionType?".
But I'm against changing its behavior -- people are expecting the 
current answer.



## C functions should have access to the function object

That's where my stake in all this is; I want to move on with PEP 573 
after 580/590 is sorted out.



## METH_FASTCALL is private and undocumented

This is the intersection of PEP 580 and 590.


## Allowing native C arguments

This would be a very experimental feature. Argument Clinic itself is not 
intended for public use, locking its "impl" functions as part of public 
API is off the table at this point.
Cython's cpdef allows this nicely, and CPython's API is full of C 
functions. That should be good enough good for now.



## Complexity

We should simpify, but I think the number of callable classes is not the 
best metric to focus on.



## PyMethodDef is too limited

This is a valid point. But the PyMethodDef array is little more than a 
shortcut to creating methods directly in a loop. The immediate 
workaround could be to create a new constructor for methods. Then we can 
look into expressing the data declaratively again.



## Slot wrappers have no custom documentation

I think this can now be done with a new custom slot wrapper class. 
Perhaps that can be added to CPython when it matures.



## Static methods and class methods should be callable

This is a valid, though minor, point. I don't event think it would be a 
PEP-level change.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 4/25/19 10:42 AM, Jeroen Demeyer wrote:

On 2019-04-25 00:24, Petr Viktorin wrote:

I believe we can achieve
that by having PEP 590's (o+offset) point not just to function pointer,
but to a {function pointer; flags} struct with flags defined for two
optimizations:


What's the rationale for putting the flags in the instance? Do you 
expect flags to be different between one instance and another instance 
of the same class?


I'm not tied to that idea. If there's a more reasonable place to put the 
flags, let's go for it, but it's not a big enough issue so it shouldn't 
complicate the protocol much. Quoting Mark from the other subthread:

Callables are either large or transient. If large, then the extra few bytes 
makes little difference. If transient then, it matters even less.




Both type flags and
nargs bits are very limited resources.


Type flags are only a limited resource if you think that all flags ever 
added to a type must be put into tp_flags. There is nothing wrong with 
adding new fields tp_extraflags or tp_vectorcall_flags to a type.


Indeed. Extra flags are just what I think PEP 590 is missing.


What I don't like about it is that it has
the extensions built-in; mandatory for all callers/callees.


I don't agree with the above sentence about PEP 580:
- callers should use APIs like PyCCall_FastCall() and shouldn't need to 
worry about the implementation details at all.
- callees can opt out of all the extensions by not setting any special 
flags and setting cr_self to a non-NULL value. When using the flags 
CCALL_FASTCALL | CCALL_KEYWORDS, then implementing the callee is exactly 
the same as PEP 590.


Imagine an extension author sitting down to read the docs and implement 
a callable:


- PEP 580 introduces 6 CCALL_* combinations: you need to select the best 
one for your use case. Also, add two structs to the instance & link them 
via pointers, make sure you support descriptor behavior and the __name__ 
attribute. (Plus there are features for special purposes: CCALL_DEFARG, 
CCALL_OBJCLASS, self-slicing, but you can skip that initially.)
- My proposal: to the instance, add a function pointer with known 
signature and flags which you set to zero. Add an offset to the type, 
and set a type flag. (There are additional possible optimizations, but 
you can skip them initially.)


PEP 580 makes a lot of sense if you read it all, but I fear there'll be 
very few people who read and understand it.
And is not important just for extension authors (admittedly, 
implementing a callable directly using the C API is often a bad idea). 
The more people understand the mechanism, the more people can help with 
further improvements.



I don't see the benefit of supporting METH_VARARGS, METH_NOARGS, and 
METH_O calling conventions (beyond backwards compatibility and 
comptibility with Python's *args syntax).
For keywords, I see a benefit in supporting *only one* of kwarg dict or 
kwarg tuple: if the caller and callee don't agree on which one to use, 
you need an expensive conversion. If we say tuple is the way, some of 
them will need to adapt, but within the set of those that do it any 
caller/callee combination will be fast. (And if tuple only turns out to 
be wrong choice, adding dict support in the future shouldn't be hard.)


That leaves fastcall (with tuple only) as the focus of this PEP, and the 
other calling conventions essentially as implementation details of 
builtin functions/methods.




As in PEP 590, any class that uses this mechanism shall not be usable as
a base class.


Can we please lift this restriction? There is really no reason for it. 
I'm not aware of any similar restriction anywhere in CPython. Note that 
allowing subclassing is not the same as inheriting the protocol.


Sure, let's use PEP 580 treatment of inheritance.
Even if we don't, I don't think dropping this restriction would be a 
PEP-level change. It can be dropped as soon as an implementation and 
tests are ready, and inheritance issues ironed out. But it doesn't need 
to be in the initial implementation.



As a compromise, we could simply never inherit the protocol.


That also sounds reasonable for the initial implementation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 4/10/19 7:05 PM, Jeroen Demeyer wrote:

On 2019-04-10 18:25, Petr Viktorin wrote:

Hello!
I've had time for a more thorough reading of PEP 590 and the reference
implementation. Thank you for the work!


And thank you for the review!


I'd now describe the fundamental
difference between PEP 580 and PEP 590 as:
- PEP 580 tries to optimize all existing calling conventions
- PEP 590 tries to optimize (and expose) the most general calling
convention (i.e. fastcall)


And PEP 580 has better performance overall, even for METH_FASTCALL. See 
this thread:

https://mail.python.org/pipermail/python-dev/2019-April/156954.html

Since these PEPs are all about performance, I consider this a very 
relevant argument in favor of PEP 580.


All about performance as well as simplicity, correctness, testability, 
teachability... And PEP 580 touches some introspection :)



PEP 580 also does a number of other things, as listed in PEP 579. But I
think PEP 590 does not block future PEPs for the other items.
On the other hand, PEP 580 has a much more mature implementation -- and
that's where it picked up real-world complexity.

About complexity, please read what I wrote in
https://mail.python.org/pipermail/python-dev/2019-March/156853.html

I claim that the complexity in the protocol of PEP 580 is a good thing, 
as it removes complexity from other places, in particular from the users 
of the protocol (better have a complex protocol that's simple to use, 
rather than a simple protocol that's complex to use).


I think we're talking past each other. I see now it as:

PEP 580 takes existing complexity and makes it available to all users, 
in a simpler way. It makes existing code faster.


PEP 590 defines a new simple/fast protocol for its users, and instead of 
making existing complexity faster and easier to use, it's left to be 
deprecated/phased out (or kept in existing classes for backwards 
compatibility). It makes it possible for future code to be faster/simpler.


I think things should be simple by default, but if people want some 
extra performance, they can opt in to some extra complexity.



As a more concrete example of the simplicity that PEP 580 could bring, 
CPython currently has 2 classes for bound methods implemented in C:

- "builtin_function_or_method" for normal C methods
- "method-descriptor" for slot wrappers like __eq__ or __add__

With PEP 590, these classes would need to stay separate to get maximal 
performance. With PEP 580, just one class for bound methods would be 
sufficient and there wouldn't be any performance loss. And this extends 
to custom third-party function/method classes, for example as 
implemented by Cython.


Yet, for backwards compatibility reasons, we can't merge the classes.
Also, I think CPython and Cython are exactly the users that can trade 
some extra complexity for better performance.



Jeroen's analysis from
https://mail.python.org/pipermail/python-dev/2018-July/154238.html seems
to miss a step at the top:

a. CALL_FUNCTION* / CALL_METHOD opcode
   calls
b. _PyObject_FastCallKeywords()
   which calls
c. _PyCFunction_FastCallKeywords()
   which calls
d. _PyMethodDef_RawFastCallKeywords()
   which calls
e. the actual C function (*ml_meth)()

I think it's more useful to say that both PEPs bridge a->e (via
_Py_VectorCall or PyCCall_Call).


Not quite. For a builtin_function_or_method, we have with PEP 580:

a. call_function()
     calls
d. PyCCall_FastCall
     which calls
e. the actual C function

and with PEP 590 it's more like:

a. call_function()
     calls
c. _PyCFunction_FastCallKeywords
     which calls
d. _PyMethodDef_RawFastCallKeywords
     which calls
e. the actual C function

Level c. above is the vectorcall wrapper, which is a level that PEP 580 
doesn't have.


PEP 580 optimizes all the code paths, where PEP 590 optimizes the fast 
path, and makes sure most/all use cases can use (or switch to) the fast 
path.
Both fast paths are fast: bridging a->e using zero-copy arg passing with 
some C calls and flag checks.


The PEP 580 approach is faster; PEP 590's is simpler.



Jeroen, is there something in PEPs 579/580 that PEP 590 blocks, or
should address?


Well, PEP 580 is an extensible protocol while PEP 590 is not. But, 
PyTypeObject is extensible, so even with PEP 590 one can always extend 
that (for example, PEP 590 uses a type flag Py_TPFLAGS_METHOD_DESCRIPTOR 
where PEP 580 instead uses the structs for the C call protocol). But I 
guess that extending PyTypeObject will be harder to justify (say, in a 
future PEP) than extending the C call protocol.


That's a good point.


Also, it's explicitly allowed for users of the PEP 580 protocol to 
extend the PyCCallDef structure with custom fields. But I don't have a 
concrete idea of whether that will be useful.


Unless I'm missing something, that would be effectively the same as 
extending their own instance struct. To bring any benefits, the extended 
PyCCallDef would need t

[Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

So, I spent another day pondering the PEPs.

I love PEP 590's simplicity and PEP 580's extensibility. As I hinted 
before, I hope they can they be combined, and I believe we can achieve 
that by having PEP 590's (o+offset) point not just to function pointer, 
but to a {function pointer; flags} struct with flags defined for two 
optimizations:

- "Method-like", i.e. compatible with LOAD_METHOD/CALL_METHOD.
- "Argument offsetting request", allowing PEP 590's 
PY_VECTORCALL_ARGUMENTS_OFFSET optimization.


This would mean one basic call signature (today's METH_FASTCALL | 
METH_KEYWORD), with individual optimizations available if both the 
caller and callee support them.




In case you want to know my thoughts or details, let me indulge in some 
detailed comparisons and commentary that led to this.

I also give a more detailed proposal below.
Keep in mind I wrote this before I distilled it to the paragraph above, 
and though the distillation is written as a diff to PEP 590, I still 
think of this as merging both PEPs.



PEP 580 tries hard to work with existing call conventions (like METH_O, 
METH_VARARGS), making them fast.
PEP 590 just defines a new convention. Basically, any callable that 
wants performance improvements must switch to METH_VECTORCALL (fastcall).
I believe PEP 590's approach is OK. To stay as performant as possible, C 
extension authors will need to adapt their code regularly. If they 
don't, no harm -- the code will still work as before, and will still be 
about as fast as it was before.
In exchange for this, Python (and Cython, etc.) can focus on optimizing 
one calling convention, rather than a variety, each with its own 
advantages and drawbacks.


Extending PEP 580 to support a new calling convention will involve 
defining a new CCALL_* constant, and adding to existing dispatch code.
Extending PEP 590 to support a new calling convention will most likely 
require a new type flag, and either changing the vectorcall semantics or 
adding a new pointer.
To be a bit more concrete, I think of possible extensions to PEP 590 as 
things like:
- Accepting a kwarg dict directly, without copying the items to 
tuple/array (as in PEP 580's CCALL_VARARGS|CCALL_KEYWORDS)
- Prepending more than one positional argument, or appending positional 
arguments
- When an optimization like LOAD_METHOD/CALL_METHOD turns out to no 
longer be relevant, removing it to simplify/speed up code.
I expect we'll later find out that something along these lines might 
improve performance. PEP 590 would make it hard to experiment.


I mentally split PEP 590 into two pieces: formalizing fastcall, plus one 
major "extension" -- making bound methods fast.
When seen this way, this "extension" is quite heavy: it adds an 
additional type flag, Py_TPFLAGS_METHOD_DESCRIPTOR, and uses a bit in 
the "Py_ssize_t nargs" argument as additional flag. Both type flags and 
nargs bits are very limited resources. If I was sure vectorcall is the 
final best implementation we'll have, I'd go and approve it – but I 
think we still need room for experimentation, in the form of more such 
extensions.
PEP 580, with its collection of per-instance data and flags, is 
definitely more extensible. What I don't like about it is that it has 
the extensions built-in; mandatory for all callers/callees.


PEP 580 adds a common data struct to callable instances. Currently these 
are all data bound methods want to use (cc_flags, cc_func, cc_parent, 
cr_self). Various flags are consulted in order to deliver the needed 
info to the underlying function.
PEP 590 lets the callable object store data it needs independently. It 
provides a clever mechanism for pre-allocating space for bound methods' 
prepended "self" argument, so data can be provided cheaply, though it's 
still done by the callable itself.
Callables that would need to e.g. prepend more than one argument won't 
be able to use this mechanism, but y'all convinced me that is not worth 
optimizing for.


PEP 580's goal seems to be that making a callable behave like a Python 
function/method is just a matter of the right set of flags. Jeroen 
called this "complexity in the protocol".
PEP 590, on the other hand, leaves much to individual callable types. 
This is "complexity in the users of the protocol".
I now don't see a problem with PEP 590's approach. Not all users will 
need the complexity. We need to give CPython and Cython the tools to 
make implementing "def"-like functions possible (and fast), but if other 
extensions need to match the behavior of Python functions, they should 
just use Cython. Emulating Python functions is a special-enough use case 
that it doesn't justify complicating the protocol, and the same goes for 
implementing Python's built-in functions (with all their historical 
baggage).




My more full proposal for a compromise between PEP 580 and 590 would go 
something like below.


The type flag (Py_TPFLAGS_HAVE_VECTORCALL/Py_TPFLAGS_HAVE_CCALL) and 
offset 

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

Hi Mark! See my more general reply; here I'll just tie loose ends with a 
few +1s.


On 4/14/19 7:30 AM, Mark Shannon wrote:

On 10/04/2019 5:25 pm, Petr Viktorin wrote:

[...]
PEP 590 is built on a simple idea, formalizing fastcall. But it is 
complicated by PY_VECTORCALL_ARGUMENTS_OFFSET and 
Py_TPFLAGS_METHOD_DESCRIPTOR.
As far as I understand, both are there to avoid intermediate 
bound-method object for LOAD_METHOD/CALL_METHOD. (They do try to be 
general, but I don't see any other use case.)

Is that right?


Not quite.
Py_TPFLAGS_METHOD_DESCRIPTOR is for LOAD_METHOD/CALL_METHOD, it allows 
any callable descriptor to benefit from the LOAD_METHOD/CALL_METHOD 
optimisation.


PY_VECTORCALL_ARGUMENTS_OFFSET exists so that callables that make onward 
calls with an additional argument can do so efficiently. The obvious 
example is bound-methods, but classes are at least as important.

cls(*args) -> cls.new(cls, *args) -> cls.__init__(self, *args)


I see. Thanks!

(I'm running out of time today, but I'll write more on why I'm asking, 
and on the case I called "impossible" (while avoiding creation of a 
"bound method" object), later.)


Let me drop this thread; I stand corrected.

Another point I'd like some discussion on is that vectorcall function 
pointer is per-instance. It looks this is only useful for type 
objects, but it will add a pointer to every new-style callable object 
(including functions). That seems wasteful.
Why not have a per-type pointer, and for types that need it (like 
PyTypeObject), make it dispatch to an instance-specific function?


Firstly, each callable has different behaviour, so it makes sense to be 
able to do the dispatch from caller to callee in one step. Having a 
per-object function pointer allows that.
Secondly, callables are either large or transient. If large, then the 
extra few bytes makes little difference. If transient then, it matters 
even less.
The total increase in memory is likely to be only a few tens of 
kilobytes, even for a large program.


That makes sense.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 394: Distributions can choose what does python command mean

[Petr Viktorin]

On 4/12/19 4:53 PM, Miro Hrončok wrote:

Hello.

Based on discussions in [1], Petr Viktorin and me have drafted a new 
update [2] to the PEP 394 (The "python" Command on Unix-Like Systems).


The update gives distributors the opportunity to decide where does the 
"python" command lead to, whether it is present etc.


Please, see the PR [2] for the suggested changes.

[1]: https://mail.python.org/pipermail/python-dev/2019-February/156272.html
[2]: https://github.com/python/peps/pull/989


The text is available at 
https://github.com/hroncok/peps/blob/pep394-2019/pep-0394.txt


As a summary, I'll paste the rationale sections here:

History of this PEP
===

In 2011, the majority of distributions
aliased the ``python`` command to Python 2, but some started switching it to
Python 3 ([5]_). As some of the former distributions did not provide a
``python2`` command by default, there was previously no way for Python 2 
code

(or any code that invokes the Python 2 interpreter directly rather than via
``sys.executable``) to reliably run on all Unix-like systems without
modification, as the ``python`` command would invoke the wrong interpreter
version on some systems, and the ``python2`` command would fail completely
on others. This PEP originally provided a very simple mechanism
to restore cross-platform support, with minimal additional work required
on the part of distribution maintainers. Simplified, the recommendation was:

1. The ``python`` command was preferred for code compatible with both
   Python 2 and 3 (since it was available on all systems, even those that
   already aliased it to Python 3).
2. The ``python`` command should always invoke Python 2 (to prevent
   hard-to-diagnose errors when Python 2 code is run on Python 3).
3. The ``python2`` and ``python3`` commands should be available to specify
   the version explicitly.

However, these recommendations implicitly assumed that Python 2 would 
always be
available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 
404),

distributions are making Python 2 optional or removing entirely.
This means either removing the ``python`` command or switching it to invoke
Python 3, invalidating respectively the first or second recommendation.
Also, some distributors decided that their users are better served by
ignoring the PEP's recommendations, making the PEP's supposedly
cross-platform recommendations on ``python`` and ``python2`` in shebangs
increasingly unreliable.


Current Rationale
=

As of 2019, nearly all new systems include Python 3 and the ``python3``
command. This makes the ``python3`` command the best general choice for
code that can run on either Python 3.x or 2.x, even though it is not
available everywhere.

The recommendation is skewed toward current and future systems, leaving
behind “*old systems*” (like RHEL 6 or default Python installed on macOS).
On these systems, Python software is rarely updated and any recommendations
this PEP makes would likely be ignored.

Also, since distributors often ignored recommendations the PEP gave
regarding the ``python`` command (for what they saw as legitimate special
needs), this PEP now gives them broad control over the command.
Correspondingly, users are advised to not use the ``python`` command
in cross-platform code.
Instead, this PEP specifies the expected behavior of the ``python3`` and
``python2`` commands, which is not controversial.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 4/11/19 1:05 AM, Jeroen Demeyer wrote:

On 2019-04-10 18:25, Petr Viktorin wrote:

Hello!
I've had time for a more thorough reading of PEP 590 and the reference
implementation. Thank you for the work!


And thank you for the review!


One general note: I am not (yet) choosing between PEP 580 and PEP 590.
I am not looking for arguments for/against whole PEPs, but individual 
ideas which, I believe, can still be mixed & matched.


I see the situation this way:
- I get about one day per week when I can properly concentrate on 
CPython. It's frustrating to be the bottleneck.
- Jeroen has time, but it would frustrating to work on something that 
will later be discarded, and it's frustrating to not be able to move the 
project forward.
- Mark has good ideas, but seems to lack the time to polish them, or 
even test out if they are good. It is probably frustrating to see 
unpolished ideas rejected.


I'm looking for ways to reduce the frustration, given where we are.


Jeroen, thank you for the comments. Apologies for not having the time to 
reply to all of them properly right now.


Mark, if you could find the time to answer (even just a few of the 
points), it would be great. I ask you to share/clarify your thoughts, 
not defend your PEP.




I'd now describe the fundamental
difference between PEP 580 and PEP 590 as:
- PEP 580 tries to optimize all existing calling conventions
- PEP 590 tries to optimize (and expose) the most general calling
convention (i.e. fastcall)


And PEP 580 has better performance overall, even for METH_FASTCALL. See 
this thread:

https://mail.python.org/pipermail/python-dev/2019-April/156954.html

Since these PEPs are all about performance, I consider this a very 
relevant argument in favor of PEP 580.



PEP 580 also does a number of other things, as listed in PEP 579. But I
think PEP 590 does not block future PEPs for the other items.
On the other hand, PEP 580 has a much more mature implementation -- and
that's where it picked up real-world complexity.

About complexity, please read what I wrote in
https://mail.python.org/pipermail/python-dev/2019-March/156853.html

I claim that the complexity in the protocol of PEP 580 is a good thing, 
as it removes complexity from other places, in particular from the users 
of the protocol (better have a complex protocol that's simple to use, 
rather than a simple protocol that's complex to use).


Sadly, I need more time on this than I have today; I'll get back to it 
next week.


As a more concrete example of the simplicity that PEP 580 could bring, 
CPython currently has 2 classes for bound methods implemented in C:

- "builtin_function_or_method" for normal C methods
- "method-descriptor" for slot wrappers like __eq__ or __add__

With PEP 590, these classes would need to stay separate to get maximal 
performance. With PEP 580, just one class for bound methods would be 
sufficient and there wouldn't be any performance loss. And this extends 
to custom third-party function/method classes, for example as 
implemented by Cython.



PEP 590's METH_VECTORCALL is designed to handle all existing use cases,
rather than mirroring the existing METH_* varieties.
But both PEPs require the callable's code to be modified, so requiring
it to switch calling conventions shouldn't be a problem.


Agreed.


Jeroen's analysis from
https://mail.python.org/pipermail/python-dev/2018-July/154238.html seems
to miss a step at the top:

a. CALL_FUNCTION* / CALL_METHOD opcode
   calls
b. _PyObject_FastCallKeywords()
   which calls
c. _PyCFunction_FastCallKeywords()
   which calls
d. _PyMethodDef_RawFastCallKeywords()
   which calls
e. the actual C function (*ml_meth)()

I think it's more useful to say that both PEPs bridge a->e (via
_Py_VectorCall or PyCCall_Call).


Not quite. For a builtin_function_or_method, we have with PEP 580:

a. call_function()
     calls
d. PyCCall_FastCall
     which calls
e. the actual C function

and with PEP 590 it's more like:

a. call_function()
     calls
c. _PyCFunction_FastCallKeywords
     which calls
d. _PyMethodDef_RawFastCallKeywords
     which calls
e. the actual C function

Level c. above is the vectorcall wrapper, which is a level that PEP 580 
doesn't have.


Again, I'll get back to this next week.


The way `const` is handled in the function signatures strikes me as too
fragile for public API.


That's a detail which shouldn't influence the acceptance of either PEP.


True.
I guess what I want from the answer is to know how much thought went 
into const handling: is what's in the PEP an initial draft, or does it 
solve some hidden issue?



Why not have a per-type pointer, and for types that need it (like
PyTypeObject), make it dispatch to an instance-specific function?


That would be exactly https://bugs.python.org/issue29259

I'll let Mark comment on this.


Minor things:
- "Continued prohibition of callable classes as base classes" -- this
section read

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

Hello!
I've had time for a more thorough reading of PEP 590 and the reference 
implementation. Thank you for the work!
Overall, I like PEP 590's direction. I'd now describe the fundamental 
difference between PEP 580 and PEP 590 as:

- PEP 580 tries to optimize all existing calling conventions
- PEP 590 tries to optimize (and expose) the most general calling 
convention (i.e. fastcall)


PEP 580 also does a number of other things, as listed in PEP 579. But I 
think PEP 590 does not block future PEPs for the other items.
On the other hand, PEP 580 has a much more mature implementation -- and 
that's where it picked up real-world complexity.


PEP 590's METH_VECTORCALL is designed to handle all existing use cases, 
rather than mirroring the existing METH_* varieties.
But both PEPs require the callable's code to be modified, so requiring 
it to switch calling conventions shouldn't be a problem.


Jeroen's analysis from 
https://mail.python.org/pipermail/python-dev/2018-July/154238.html seems 
to miss a step at the top:


a. CALL_FUNCTION* / CALL_METHOD opcode
  calls
b. _PyObject_FastCallKeywords()
  which calls
c. _PyCFunction_FastCallKeywords()
  which calls
d. _PyMethodDef_RawFastCallKeywords()
  which calls
e. the actual C function (*ml_meth)()

I think it's more useful to say that both PEPs bridge a->e (via 
_Py_VectorCall or PyCCall_Call).



PEP 590 is built on a simple idea, formalizing fastcall. But it is 
complicated by PY_VECTORCALL_ARGUMENTS_OFFSET and 
Py_TPFLAGS_METHOD_DESCRIPTOR.
As far as I understand, both are there to avoid intermediate 
bound-method object for LOAD_METHOD/CALL_METHOD. (They do try to be 
general, but I don't see any other use case.)

Is that right?
(I'm running out of time today, but I'll write more on why I'm asking, 
and on the case I called "impossible" (while avoiding creation of a 
"bound method" object), later.)



The way `const` is handled in the function signatures strikes me as too 
fragile for public API.
I'd like if, as much as possible, PY_VECTORCALL_ARGUMENTS_OFFSET was 
treated as a special optimization that extension authors can either opt 
in to, or blissfully ignore.

That might mean:
- vectorcall, PyObject_VectorCallWithCallable, PyObject_VectorCall, 
PyCall_MakeTpCall all formally take "PyObject *const *args"
- a naïve callee must do "nargs &= ~PY_VECTORCALL_ARGUMENTS_OFFSET" 
(maybe spelled as "nargs &= PY_VECTORCALL_NARGS_MASK"), but otherwise 
writes compiler-enforced const-correct code.
- if PY_VECTORCALL_ARGUMENTS_OFFSET is set, the callee may modify 
"args[-1]" (and only that, and after the author has read the docs).



Another point I'd like some discussion on is that vectorcall function 
pointer is per-instance. It looks this is only useful for type objects, 
but it will add a pointer to every new-style callable object (including 
functions). That seems wasteful.
Why not have a per-type pointer, and for types that need it (like 
PyTypeObject), make it dispatch to an instance-specific function?



Minor things:
- "Continued prohibition of callable classes as base classes" -- this 
section reads as a final. Would you be OK wording this as something 
other PEPs can tackle?
- "PyObject_VectorCall" -- this looks extraneous, and the reference 
imlementation doesn't need it so far. Can it be removed, or justified?
- METH_VECTORCALL is *not* strictly "equivalent to the currently 
undocumented METH_FASTCALL | METH_KEYWORD flags" (it has the 
ARGUMENTS_OFFSET complication).
- I'd like to officially call this PEP "Vectorcall", see 
https://github.com/python/peps/pull/984




Mark, what are your plans for next steps with PEP 590? If a volunteer 
wanted to help you push this forward, what would be the best thing to 
work on?


Jeroen, is there something in PEPs 579/580 that PEP 590 blocks, or 
should address?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 3/30/19 11:36 PM, Jeroen Demeyer wrote:

On 2019-03-30 17:30, Mark Shannon wrote:

2. The claim that PEP 580 allows "certain optimizations because other
code can make assumptions" is flawed. In general, the caller cannot make
assumptions about the callee or vice-versa. Python is a dynamic language.


PEP 580 is meant for extension classes, not Python classes. Extension 
classes are not dynamic. When you implement tp_call in a given way, the 
user cannot change it. So if a class implements the C call protocol or 
the vectorcall protocol, callers can make assumptions about what that 
means.



PEP 579 is mainly a list of supposed flaws with the
'builtin_function_or_method' class.
The general thrust of PEP 579 seems to be that builtin-functions and
builtin-methods should be more flexible and extensible than they are. I
don't agree. If you want different behaviour, then use a different
object. Don't try an cram all this extra behaviour into a pre-existing
object.


I think that there is a misunderstanding here. I fully agree with the 
"use a different object" solution. This isn't a new solution: it's 
already possible to implement those different objects (Cython does it). 
It's just that this solution comes at a performance cost and that's what 
we want to avoid.


It does seem like there is some misunderstanding.

PEP 580 defines a CCall structure, which includes the function pointer, 
flags, "self" and "parent". Like the current implementation, it has 
various METH_ flags for various C signatures. When called, the info from 
CCall is matched up (in relatively complex ways) to what the C function 
expects.


PEP 590 only adds the "vectorcall". It does away with flags and only has 
one C signatures, which is designed to fit all the existing ones, and is 
well optimized. Storing the "self"/"parent", and making sure they're 
passed to the C function is the responsibility of the callable object.
There's an optimization for "self" (offsetting using 
PY_VECTORCALL_ARGUMENTS_OFFSET), and any supporting info can be provided 
as part of "self".



I'll reiterate that PEP 590 is more general than PEP 580 and that once
the callable's code has access to the callable object (as both PEPs
allow) then anything is possible. You can't can get more extensible than
that.


Anything is possible, but if one of the possibilities becomes common and 
useful, PEP 590 would make it hard to optimize for it.
Python has grown many "METH_*" signatures over the years as we found 
more things that need to be passed to callables. Why would 
"METH_VECTORCALL" be the last? If it won't (if you think about it as one 
more way to call functions), then dedicating a tp_* slot to it sounds 
quite expensive.



In one of the ways to call C functions in PEP 580, the function gets 
access to:

- the arguments,
- "self", the object
- the class that the method was found in (which is not necessarily 
type(self))
I still have to read the details, but when combined with 
LOAD_METHOD/CALL_METHOD optimization (avoiding creation of a "bound 
method" object), it seems impossible to do this efficiently with just 
the callable's code and callable's object.



I would argue the opposite: PEP 590 defines a fixed protocol that is not 
easy to extend. PEP 580 on the other hand uses a new data structure 
PyCCallDef which could easily be extended in the future (this will 
intentionally never be part of the stable ABI, so we can do that).


I have also argued before that the generality of PEP 590 is a bad thing 
rather than a good thing: by defining a more rigid protocol as in PEP 
580, more optimizations are possible.



PEP 580 has the same limitation for the same reasons. The limitation is
necessary for correctness if an object supports calls via `__call__` and
through another calling convention.


I don't think that this limitation is needed in either PEP. As I 
explained at the top of this email, it can easily be solved by not using 
the protocol for Python classes. What is wrong with my proposal in PEP 
580: https://www.python.org/dev/peps/pep-0580/#inheritance



I'll add Jeroen's notes from the review of the proposed PEP 590
(https://github.com/python/peps/pull/960):

The statement "PEP 580 is specifically targetted at function-like 
objects, and doesn't support other callables like classes, partial 
functions, or proxies" is factually false. The motivation for PEP 580 is 
certainly function/method-like objects but it's a general protocol that 
every class can implement. For certain classes, it may not be easy or 
desirable to do that but it's always possible.


Given that `PY_METHOD_DESCRIPTOR` is a flag for tp_flags, shouldn't it 
be called `Py_TPFLAGS_METHOD_DESCRIPTOR` or something?


Py_TPFLAGS_HAVE_VECTOR_CALL should be Py_TPFLAGS_HAVE_VECTORCALL, to be 
consistent with tp_vectorcall_offset and other uses of "vectorcall" (not 
"vector call")



And mine, so far:

I'm not clear on the constness of the "args" array.
If it is mutable (PyObject **), you 

Re: [Python-Dev] BDFL-Delegate appointments for several PEPs

[Petr Viktorin]

On Sun, Mar 24, 2019 at 4:22 PM Mark Shannon  wrote:
>
> Hi Petr,
>
> Regarding PEPs 576 and 580.
> Over the new year, I did a thorough analysis of possible approaches to
> possible calling conventions for use in the CPython ecosystems and came
> up with a new PEP.
> The draft can be found here:
> https://github.com/markshannon/peps/blob/new-calling-convention/pep-.rst
>
> I was hoping to profile a branch with the various experimental changes
> cherry-picked together, but don't seemed to have found the time :(
>
> I'd like to have a testable branch, before formally submitting the PEP,
> but I'd thought you should be aware of the PEP.
>
> Cheers,
> Mark.

Hello Mark,
Thank you for letting me know! I wish I knew of this back in January,
when you committed the first draft. This is unfair to the competing
PEP, which is ready and was waiting for the new govenance. We have
lost three months that could be spent pondering the ideas in the
pre-PEP.
Do you think you will find the time to piece things together? Is there
anything that you already know should be changed?

Do you have any comments on [Jeroen's comparison]?

The pre-PEP is simpler then PEP 580, because it solves simpler issues.
I'll need to confirm that it won't paint us into a corner -- that
there's a way to address all the issues in PEP 579 in the future.

The pre-PEP claims speedups of 2% in initial experiments, with
expected overall performance gain of 4% for the standard benchmark
suite. That's pretty big.
As far as I can see, PEP 580 claims not much improvement in CPython,
but rather large improvements for extensions (Mistune with Cython).

The pre-PEP has a complication around offsetting arguments by 1 to
allow bound methods forward calls cheaply. I fear that this optimizes
for current usage with its limitations.
PEP 580's cc_parent allows bound methods to have access to the class,
and through that, the module object where they are defined and the
corresponding module state. To support this, vector calls would need a
two-argument offset.
(That seems to illustrate the main difference between the motivations
of the two PEPs: one focuses on extensibility; the other on optimizing
existing use cases.)

The pre-PEP's "any third-party class implementing the new call
interface will not be usable as a base class" looks quite limiting.



[Jeroen's comparison]:
https://mail.python.org/pipermail/python-dev/2018-July/154238.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Petr Viktorin]

On 2/28/19 6:56 PM, Gregory P. Smith wrote:


On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi > wrote:



On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
mailto:nas-pyt...@arctrix.com>> wrote:

On 2019-02-26, Gregory P. Smith wrote:
 > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
mailto:ba...@python.org>> wrote:
 > For an OS distro provided interpreter, being able to restrict
its use to
 > only OS distro provided software would be ideal (so ideal
that people who
 > haven't learned the hard distro maintenance lessons may hate
me for it).

This idea has some definite problems.  I think enforcing it via
convention is about as much as would be good to do.  Anything more
and you make it hard for people who really need to use the vendor
provided interpreter from being able to do so.

Why might someone need to use the distro provided interpreter?

* Vendor provides some python modules in their system packages which
are not installable from pip (possibly even a proprietary extension
module, so not even buildable from source or copyable from the
system location) which the end user needs to use to do something to
their system.
* End user writes a python module which is a plugin to a system tool
which has to be installed into the system python to from which that
system tool runs.  The user then wants to write a script which uses
the system tool with the plugin in order to do something to their
system outside of the system tool (perhaps the system tool is
GUI-driven and the user wants to automate a part of it via the
python module).  They need their script to use the system python so
that they are using the same code as the system tool itself would use.

There's probably other scenarios where the benefits of locking the
user out of the system python outweigh the benefits but these are
the ones that I've run across lately.


Agreed.  The convention approach as someone said RHEL 8 has apparently 
done with an os distro reserved interpreter (yay) is likely good enough 
for most situations.


I'd go a /little/ further than that and suggest such an os distro 
reserved interpreter attempt to prevent installation of packages (ie: 
remove pip/ensurepip/distutils) via any other means than the OS package 
manager (rpms, debs).  Obviously that can't actually prevent someone 
from figuring out how to run getpip or manually installing trees of 
packages within its sys.path, but it acts as a deterrent suggesting that 
this interpreter is not intended for arbitrary software installation.


Currently, in RHEL/Fedora:
- "sudo pip" installs to /usr/local/, RPM packages install to /usr/
- with "-I", the interpreter does not look into /usr/local/.
AFAIK, Debian/Ubuntu have something very similar, and were first to do it.

What remains to tie this together is making "platform-python" always run 
with -I. This is PEP 432's "system-python" use case / design goal.
(The RHEL/Fedora platform-python was initially called system-python. We 
renamed to it so it doesn't clash with the PEP. It's the same use case, 
but we don't want to assign semantics to the name prematurely. 
Cutrrently, system-python is waiting for the larger-scale restructuring, 
and hopefully wider/longer discussion.)


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/26/19 6:54 PM, Barry Warsaw wrote:

There haven't been many new ideas since this summary – mostly it was explaining 
and re-hashing what's been mentioned before.


Thanks for the summary Petr.

Here’s another way to think about the problem.  I know Nick and I have talked 
about this before, but I don’t think any distros have actually done this, 
though I’ve been out of that business a while now so correct me if I’m wrong.

I see this question as having several parts, and the conflation of them is part 
of the reason why the unversioned `python` command is so problematic.  Python 
is used for:

* OS functionality
* to run applications that aren’t critical to the OS but are delivered on the OS
* as the entry point to the interactive interpreter
* to run applications written and deployed on the OS but completely outside of 
it

Which `python` are we trying to change?  All of them?

For OS functionality, there should probably be a separate command not conflated 
with /usr/bin/python.  The OS can make any adjustments it needs, calling it 
`spython` (as I think Nick once suggested), or whatever.  Nobody but OS 
maintainers cares what this is called or what version of Python it exposes.


Yup. RHEL 8 actually has exactly that. (It's called 
/usr/libexec/platform-python; please don't use it!)
Fedora (and most other distros) makes this the same as the interpreter 
for other packaged software. For Fedora the main reason is that don't 
want to maintain two full separate Python stacks.



I strongly believe that (eventually) the interactive interpreter command should 
be /usr/bin/python and that this should point to Python 3, since this provides 
the best experience for beginners, dabblers, etc.


+1


So what about the other two use cases?  Well, for applications packages within 
the OS but aren’t critical to it, I think they should always use the versioned 
shebang, never the unversioned shebang.  Distros can control this, so that 
transition should be easier.


+1


The tricky part then seems to me what to do for 3rd parties which are using the 
distro Python in their shebangs?  Nobody sees their code but them, and changing 
the shebang out from under them could cause their code to break.  But don’t 
they already take lots of precautions and planning for any OS upgrade?  
Changing the shebang for Python 2 would be just one of the things they’d have 
to worry about in an OS upgrade.


Also, things will break for them anyway, it's just a matter of time. 
Python 2 *is* going away, eventually. (Right?)
I don't think we're doing that many people a favor by keeping 
/usr/bin/python → python2 around. Instead, we're *hiding* the problem 
from them. Too many people think python2 is still the "default".


Making /usr/bin/python be missing for some time, rather than pointing it 
to python3 now, is the more explicit way to do the transition.



I don’t know whether this analysis is complete or correct, but perhaps it helps 
inform a way forward on PEP 394.


I have two very different questions in mind for moving this forward.

Who gets to decide on PEP 394 changes?
Since so many people on python-dev are in agreement, where do I go for 
opposing voices?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/14/19 9:56 AM, Petr Viktorin wrote:

On 2/13/19 4:24 PM, Petr Viktorin wrote:

I think it's time for another review.

[...]
Please see this PR for details and a suggested change: 
https://github.com/python/peps/pull/893


Summary of the thread so far.

Antoine Pitrou noted that the PEP should acknowledge that there are now 
years of
established usage of `python` as Python 3 for many conda users, often as 
the "main" Python.


Victor Stinner expressed support for "python" being the latest Python 
version, citing PHP, Ruby, Perl; containers; mentions of "python" in our 
docs.


Steve Dower later proposed concrete points how to make "python" the 
default command:

   * our docs should say "python" consistently
   * we should recommend that distributors use the same workaround
   * our docs should describe the recommended workaround in any places 
people are likely to first encounter it (tutorial, sys.executable, etc.)


Chris Barker added that "python3" should still be available, even if 
"python" is default.


Barry Warsaw gave a +1 to making "python" default, noting that there 
were plans to change this when Python 2 is officially deprecated. But 
distros need to make decisions about 2020 now.


Chris Barker noted that users won't see any discuntinuity in 2020. 
That's just a date support from CPython devs ends.


Victor pointed to discussions on 4.0 vs. 3.10. (I'll ignore discussions 
on 4.0 in this summary.)

Victor also posted some interesting info and links on Fedora and RHEL.
There was a discussion on the PSF survey about how many people use 
Python 3. (I'll ignore this sub-thread, it's not really about the 
"python" command.)


Steve noted that the Windows Store package of Python 3 provides 
"python", but he is still thinking how to make this reasonable/reliable 
in the full installer.


Several people think "py" on Unix would be a good thing. Neil 
Schemenauer supposes we would encourage people to use it over 
"python"/"python2"/"python3", so "python" would be less of an issue.


Neil Schemenauer is not opposed to making "python" configurable or 
eventually pointing it to Python 3.


Jason Swails shared experience from running software with a 
"#!/usr/bin/env python" shebang on a system that didn't have Python 2 
(and followed the PEP, so no "python" either). The workaround was ugly.


There haven't been many new ideas since this summary – mostly it was 
explaining and re-hashing what's been mentioned before.
Matthias Klose pointed out some Debian/Ubuntu points, to which I'll add 
the situation in other distros I know of.


*Debian* is concerned that python → python3 will break software
after an upgrade. Debian appears to not want to ship the unversioned 
command after py2 removal.


For *Ubuntu*, Matthias is not sure if he wants a python executable at 
all. He notes that pypi.org recommends pip, and pip still breaks 
system-installed packages when asked to.


For both Ubuntu 20.04 LTS and Debian bullseye, the goal is that distro 
packages don't use the unversioned shebang.


*Fedora* packages don't use the unversioned shebang. If it was up to me, 
the unversioned command would be removed in F31 (released in the first 
half of 2019) and then pointed to python3 in F32 (second half). But we'd 
rather happy to follow upstream consensus. (And the PEP, if it reflects 
the consensus.)


In *RHEL*, the unversioned command is missing by default. Sysadmins can 
change it, but are advised to use python2/python3 instead. RHEL decision 
makers don't give the PEP much weight.


*Arch* did the switch to python3 a long time ago (and the resulting fire 
wasn't all that bright).


With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if 
installed) otherwise the macOS system Python. That's exactly according 
to the PEP. They tried to switch python to python3 before, and got 
rather nasty backlash citing PEP 394. I assume they will follow the PEP 
quite strictly from now on.


The *macOS* system Python is out of our hands; Apple ignores upstream 
recommendations.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/13/19 5:45 PM, Victor Stinner wrote:

Some more context about Petr's change, Fedora, RHEL and Red Hat.

[...]

Fedora could switch "python" to Python 3 now*, if the PEP changes to 
allow it.


* "now" has a release date of around October 2019. The next release 
after that should then be around May 2020.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/13/19 4:24 PM, Petr Viktorin wrote:

I think it's time for another review.

[...]
Please see this PR for details and a suggested change: 
https://github.com/python/peps/pull/893


Summary of the thread so far.

Antoine Pitrou noted that the PEP should acknowledge that there are now 
years of
established usage of `python` as Python 3 for many conda users, often as 
the "main" Python.


Victor Stinner expressed support for "python" being the latest Python 
version, citing PHP, Ruby, Perl; containers; mentions of "python" in our 
docs.


Steve Dower later proposed concrete points how to make "python" the 
default command:

  * our docs should say "python" consistently
  * we should recommend that distributors use the same workaround
  * our docs should describe the recommended workaround in any places 
people are likely to first encounter it (tutorial, sys.executable, etc.)


Chris Barker added that "python3" should still be available, even if 
"python" is default.


Barry Warsaw gave a +1 to making "python" default, noting that there 
were plans to change this when Python 2 is officially deprecated. But 
distros need to make decisions about 2020 now.


Chris Barker noted that users won't see any discuntinuity in 2020. 
That's just a date support from CPython devs ends.


Victor pointed to discussions on 4.0 vs. 3.10. (I'll ignore discussions 
on 4.0 in this summary.)

Victor also posted some interesting info and links on Fedora and RHEL.
There was a discussion on the PSF survey about how many people use 
Python 3. (I'll ignore this sub-thread, it's not really about the 
"python" command.)


Steve noted that the Windows Store package of Python 3 provides 
"python", but he is still thinking how to make this reasonable/reliable 
in the full installer.


Several people think "py" on Unix would be a good thing. Neil 
Schemenauer supposes we would encourage people to use it over 
"python"/"python2"/"python3", so "python" would be less of an issue.


Neil Schemenauer is not opposed to making "python" configurable or 
eventually pointing it to Python 3.


Jason Swails shared experience from running software with a 
"#!/usr/bin/env python" shebang on a system that didn't have Python 2 
(and followed the PEP, so no "python" either). The workaround was ugly.


-

Since this list is public, I'd like to remind all readers that it is 
full of people who work extensively with Python 3, and tend to drive it 
forward at any opportunity. (Myself included, but on this thread I'll 
leave the arguments to someone else – they're covered adequately.)


Thoughts of Python developers are important, but we're not hearing any 
other voices. Perhaps everyone with a different opinion has already 
self-selected out.


I don't know of a good place for this discussion, and I'm not a good 
person to give arguments to support the original "python" should be 
Python 2 direction. (But if I did, I imagine posting them here would 
feel a bit scary.)
But I would not be surprised, or annoyed, if the Council had a private 
discussion and pronounced "No, sorry, not yet".


Anyway, how should this be decided? Where should it be discussed?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/13/19 4:46 PM, Antoine Pitrou wrote:

On Wed, 13 Feb 2019 16:24:48 +0100
Petr Viktorin  wrote:

PEP 394 says:

  > This recommendation will be periodically reviewed over the next few
  > years, and updated when the core development team judges it
  > appropriate. As a point of reference, regular maintenance releases
  > for the Python 2.7 series will continue until at least 2020.

I think it's time for another review.
I'm especially worried about the implication of these:

- If the `python` command is installed, it should invoke the same
version of Python as the `python2` command
- scripts that are deliberately written to be source compatible
with both Python 2.x and 3.x [...] may continue to use `python` on
their shebang line.

So, to support scripts that adhere to the recommendation, Python 2
needs to be installed :(


I think PEP 394 should acknowledge that there are now years of
established usage of `python` as Python 3 for many conda users.


The intention is that Conda environments are treated the same as venv 
environments, i.e.:


When a virtual environment (created by the PEP 405 venv package or a 
similar tool) is active, the python command should refer to the virtual 
environment's interpreter. In other words, activating a virtual 
environment counts as deliberate user action to change the default 
python interpreter.



Do you think conda should be listed explicitly along with venv?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

PEP 394 says:

> This recommendation will be periodically reviewed over the next few
> years, and updated when the core development team judges it
> appropriate. As a point of reference, regular maintenance releases
> for the Python 2.7 series will continue until at least 2020.

I think it's time for another review.
I'm especially worried about the implication of these:

- If the `python` command is installed, it should invoke the same
  version of Python as the `python2` command
- scripts that are deliberately written to be source compatible
  with both Python 2.x and 3.x [...] may continue to use `python` on
  their shebang line.

So, to support scripts that adhere to the recommendation, Python 2
needs to be installed :(


Please see this PR for details and a suggested change: 
https://github.com/python/peps/pull/893

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/7/19 5:16 PM, Stephane Wirtel wrote:

On 02/06, Petr Viktorin wrote:

On 2/6/19 8:43 AM, Stephane Wirtel wrote:

On 02/05, Barry Warsaw wrote:

On Feb 5, 2019, at 02:24, Stephane Wirtel  wrote:
You’re welcome!  I just pushed an update to add 3.8.0a1 to the set 
of Python’s (including git head).  Do you think there’s a better way 
to publicize these images?


I know that Julien Palard wanted a docker image with all the versions of
Python, see: https://github.com/docker-library/python/issues/373

For my part, I wanted to propose a docker image with the last version of
Python and try to use it for the detection of bugs in the main python
projects (django, numpy, flask, pandas, etc...) with a CI (example:
Gitlab-CI)

First issue: pytest uses the ast module of python and since 3.8.0a1, the
tests do not pass -> new issue for pytest


FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
alphas/betas, so everything's tested when 3.8.0 is released: 
https://fedoraproject.org/wiki/Changes/Python3.8

Hi Petr,

Will you execute the tests of these packages?


It's best practice to include the test suite in Fedora packages.
Sometimes it's not – e.g. if the tests need network access, or all extra 
testing dependencies aren't available, or most frequently, the 
maintainer is just lazy.
If you have a specific package in mind, I can check. Currently django & 
numpy get tested; flask & pandas don't.


For 3.7, we did the rebuild much later in the cycle. The builds 
themselves caught async/await SyntaxErrors, and tests caught a lot of 
StopIteration leaking. At the time it felt like no one really knew what 
porting to 3.7.0 would look like – similar to how people didn't think 
"unicode" would be a big problem in py3k. That's what we're trying to 
avoid for 3.8.0.



I have a small discussion with Julien Palard and I wanted to create a
small CI where I will execute the tests of the updated packages from
the RSS feed of PyPI.

The first one was pytest 


That sounds exciting! Something like that is on my "interesting possible 
projects" list, but alas, not at the top :(


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/6/19 2:26 PM, Matthias Klose wrote:

On 06.02.19 13:23, Petr Viktorin wrote:

FWIW, we're preparing to rebuild all Fedora packages with the 3.8 alphas/betas,
so everything's tested when 3.8.0 is released:
https://fedoraproject.org/wiki/Changes/Python3.8

That should cover the main Python projects, too.


well, the real challenge is that all test suites of third party packages still
pass on all architectures.  From past transitions, I know that this costs the
most time and resources. 


Same experience here.

In Fedora, tests are generally run as part of the build.
(Sorry, that was definitely not obvious from my message!)


But yes, targeting 3.8 for Ubuntu 20.04 LTS as well.


\o/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/6/19 8:43 AM, Stephane Wirtel wrote:

On 02/05, Barry Warsaw wrote:

On Feb 5, 2019, at 02:24, Stephane Wirtel  wrote:
You’re welcome!  I just pushed an update to add 3.8.0a1 to the set of 
Python’s (including git head).  Do you think there’s a better way to 
publicize these images?


I know that Julien Palard wanted a docker image with all the versions of
Python, see: https://github.com/docker-library/python/issues/373

For my part, I wanted to propose a docker image with the last version of
Python and try to use it for the detection of bugs in the main python
projects (django, numpy, flask, pandas, etc...) with a CI (example:
Gitlab-CI)

First issue: pytest uses the ast module of python and since 3.8.0a1, the
tests do not pass -> new issue for pytest


FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
alphas/betas, so everything's tested when 3.8.0 is released: 
https://fedoraproject.org/wiki/Changes/Python3.8


That should cover the main Python projects, too.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

[Petr Viktorin]

On 1/23/19 3:33 AM, Stephan Reiter wrote:

Thanks for the answers so far. I appreciate them!

Nathaniel, I'd like to allow Python plugins in my application. A
plugin should be allowed to bring its own modules along (i.e.
plugin-specific subdir is in sys.path when the plugin is active) and
hence some isolation of them will be needed, so that they can use
different versions of a given module. That's my main motivation for
using subinterpreters.
I thought about running plugins out-of-processes - a separate process
for every plugin - and allow them to communicate with my application
via RPC. But that makes it more complex to implement the API my
application will offer and will slow down things due to the need to
copy data.
Maybe you have another idea for me? :)


Try to make the plugins work together. Look into using pip/PyPI for your 
plugins. Try to make it so each package ("plugin") would have only one 
module/package, and dependencies would be other packages that can be 
installed individually and shared. And keep in mind you can set up your 
own package index, or distribute/install individual package files.


If that's not possible, and you want things to work now, go with subprocess.

If you want to help make subinterpreters work better, there are several 
people scratching at the problem from different angles. Most/all would 
welcome help, but don't expect any short-term benefits.
(FWIW, my own effort is currently blocked on PEP 580, and I hope to move 
forward after a Council is elected.)




Henry, Antoine, thanks for your input; I'll check out the tests and
see what I can learn from issue 10915.

Stephan

Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith :


There are currently numerous incompatibilities between numpy and 
subinterpreters, and no concrete plan for fixing them. The numpy team does not 
consider subinterpreters to be a supported configuration, and can't help you 
with any issues you run into. I know the concept of subinterpreters is really 
appealing, but unfortunately the CPython implementation is not really mature or 
widely supported... are you absolutely certain you need to use subinterpreters 
for your application?

On Tue, Jan 22, 2019, 08:27 Stephan Reiter 

Hi all!

I am new to the list and arriving with a concrete problem that I'd
like to fix myself.

I am embedding Python (3.6) into my C++ application and I would like
to run Python scripts isolated from each other using sub-interpreters.
I am not using threads; everything is supposed to run in the
application's main thread.

I noticed that if I create an interpreter, switch to it and execute
code that imports numpy (1.13), my application will hang.

   ntdll.dll!NtWaitForSingleObject() Unknown
   KernelBase.dll!WaitForSingleObjectEx() Unknown

python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x748a67a0, 
_RTL_CRITICAL_SECTION * cs=0x748a6778, unsigned long ms=5) Line 245 C

   [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
   python36.dll!take_gil(_ts * tstate=0x023251cbc260) Line 224 C
   python36.dll!PyEval_RestoreThread(_ts * tstate=0x023251cbc260) Line 370 C
   python36.dll!PyGILState_Ensure() Line 855 C
   umath.cp36-win_amd64.pyd!7ff8c6306ab2() Unknown
   umath.cp36-win_amd64.pyd!7ff8c630723c() Unknown
   umath.cp36-win_amd64.pyd!7ff8c6303a1d() Unknown
   umath.cp36-win_amd64.pyd!7ff8c63077c0() Unknown
   umath.cp36-win_amd64.pyd!7ff8c62ff926() Unknown
   [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
   [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
   python36.dll!call_function(_object * * *
pp_stack=0x0048be5f5e40, __int64 oparg, _object * kwnames) Line
4822 C

Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
PyEval_RestoreThread on the (auto) threadstate of the main
interpreter. And that's wrong.
We are already holding the GIL with the threadstate of our current
sub-interpreter, so there's no need to switch.

I know that the GIL API is not fully compatible with sub-interpreters,
as issues #10915 and #15751 illustrate.

But since I need to support calls to PyGILState_Ensure - numpy is the
best example -, I am trying to improve the situation here:
https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597

That change may be naive, but it does the trick for my use case. If
totally wrong, I don't mind pursuing another alley.

Essentially, I'd like to ask for some guidance in how to tackle this
problem while keeping the current GIL API unchanged (to avoid breaking
modules).

I am also wondering how I can test any changes I am proposing. Is
there a test suite for interpreters, for example?

Thank you very much,
Stephan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 

Re: [Python-Dev] Missing functions [Was: Re: Experiment an opt-in new C API for Python? (leave current API unchanged)]

[Petr Viktorin]

On 11/21/18 4:11 PM, Matěj Cepl wrote:

On 2018-11-21, 14:54 GMT, Benjamin Peterson wrote:

In Python 3, there is no underlying FILE* because the io
module is implemented using fds directly rather than C stdio.


OK, so the proper solution is to kill all functions which expect
FILE


Indeed.
This has another side to it: there are file-like objects that aren't 
backed by FILE*. In most case, being a "real" file is an unnecessary 
distinction, like that between the old `int` vs. `long`. "Fits in the 
machine register" is a detail from a level below Python, and so is "the 
kernel treats this as a file".


Of course, this is not how C libraries work -- so, sadly, it makes 
wrappers harder to write. And a perfect solution might require adding 
more generic I/O to the C library.




and if you are anal retentive about stability of API, then
you have to fake it by creating FILE structure around the
underlying fd handler as I did in M2Crypto, right?


Yes, AFAIK that is the least bad solution.
I did something very similar here: 
https://github.com/encukou/py3c/blob/master/include/py3c/fileshim.h


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Petr Viktorin]

On 11/19/18 12:14 PM, Victor Stinner wrote:

To design a new C API, I see 3 options:

(1) add more functions to the existing Py_LIMITED_API
(2) "fork" the current public C API: remove functions and hide as much
implementation details as possible
(3) write a new C API from scratch, based on the current C API.
Something like #define newcapi_Object_GetItem PyObject_GetItem"?
Sorry, but "#undef " doesn't work. Only very few
functions are defined using "#define ...".

I dislike (1) because it's too far from what is currently used in
practice. Moreover, I failed to find anyone who can explain me how the
C API is used in the wild, which functions are important or not, what
is the C API, etc.


One big, complex project that now uses the limited API is PySide. They 
do some workarounds, but the limited API works. Here's a writeup of the 
troubles they have with it: 
https://github.com/pyside/pyside2-setup/blob/5.11/sources/shiboken2/libshiboken/pep384impl_doc.rst



I propose (2). We control how much changes we do at each milestone,
and we start from the maximum compatibility with current C API. Each
change can be discussed and experimented to define what is the C API,
what we want, etc. I'm working on this approach for 1 year, that's why
many discussions popped up around specific changes :-)


I hope the new C API will be improvements (and clarifications) of the 
stable ABI, rather than a completely new thing.
My ideal would be that Python 4.0 would keep the same API (with 
questionable things emulated & deprecated), but break *ABI*. The "new C 
API" would become that new stable ABI -- and this time it'd be something 
we'd really want to support, without reservations.


One thing that did not work with the stable ABI was that it's "opt-out"; 
I think we can agree that a new one must be "opt-in" from the start.
I'd also like the "new API" to be a *strict subset* of the stable ABI: 
if a new function needs to be added, it should be added to both.



Some people recently proposed (3) on python-dev. I dislike this option
because it starts by breaking the backward compatibility. It looks
like (1), but worse. The goal and the implementation are unclear to
me.

--

Replacing PyDict_GetItem() (specialized call) with PyObject_Dict()
(generic API) is not part of my short term plan. I wrote it in the
roadmap, but as I wrote before, each change should be discusssed,
experimented, benchmarked, etc.

Victor
Le lun. 19 nov. 2018 à 12:02, M.-A. Lemburg  a écrit :


On 19.11.2018 11:53, Antoine Pitrou wrote:

On Mon, 19 Nov 2018 11:28:46 +0100
Victor Stinner  wrote:

Python internals rely on internals to implement further optimizations,
than modifying an "immutable" tuple, bytes or str object, because you
can do that at the C level. But I'm not sure that I would like 3rd
party extensions to rely on such things.


I'm not even talking about *modifying* tuples or str objects, I'm
talking about *accessing* their value without going through an abstract
API that does slot lookups, indirect function calls and object unboxing.

For example, people may need a fast way to access the UTF-8
representation of a unicode object.  Without making indirect function
calls, and ideally without making a copy of the data either.  How do
you do that using the generic C API?


Something else you need to consider is creating instances of
types, e.g. a tuple. In C you will have to be able to put
values into the data structure before it is passed outside
the function in order to build the tuple.

If you remove this possibility to have to copy data all the
time, losing the advantages of having a rich C API.
  --
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2018)

Python Projects, Coaching and Consulting ...  http://www.egenix.com/
Python Database Interfaces ...   http://products.egenix.com/
Plone/Zope Database Interfaces ...   http://zope.egenix.com/



::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
   http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org

Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command

[Petr Viktorin]

On 10/18/18 7:44 PM, VanL wrote:
Primarily for non-windows platforms, but I also think for Windows users 
without any compilers or similar tools installed. There is also some 
discussion of removing some of the older toolchain-specific versions 
(leaving only -14), but that is a subject for another day.


Also I am not sure that bug report applies to 3.5/3.6/3.7.


It does.



On Thu, Oct 18, 2018 at 11:26 AM Petr Viktorin <mailto:encu...@gmail.com>> wrote:


On 10/18/18 4:40 PM, Zachary Ware wrote:
 > On Thu, Oct 18, 2018 at 9:09 AM VanL mailto:van.lindb...@gmail.com>> wrote:
 >> Hi all,
 >>
 >> I am looking into an issue associated with the wininst-*.exe
files in the distutils/command subdirectory. It looks like these are
the executable stubs used to create self-extracting zips for
installation - but I am not 100% sure. It also looks like they
include the calls to standard Windows functions to display the
installer window.
 >>
 >> I have a couple questions I need help with:
 >> 1) Am I correct about the function, and if not, what are they?
 >
 > You are correct.  IIUC, they are checked in to allow creating those
 > installers from non-Windows platforms.

Is that the only reason for them?
At least on Linux, bdist_wininst does not work since at least Python
3.2, as it tries to use a Windows-only encoding internally.
https://bugs.python.org/issue10945

If they're only there for non-Windows platforms, they're useless.

 >> 2) Where did these come from, and where is their source code?
 >
 > Source can be found here:
 > https://github.com/python/cpython/tree/master/PC/bdist_wininst
 >
 > The individual checked-in .exe files were each originally built by
 > whoever updated the Windows toolchain to a new version of MSVC
 > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
 > ones were added by Thomas Heller, presumably using whatever the
 > current toolchain(s) was (were) at the time).  A few of them have
been
 > rebuilt after bug fixes in the source since they were added,
mostly by
 > the same people, though I also see Mark Hammond and Raymond Hettinger
 > in the history (and Georg Brandl via svnmerge).  I notice that there
 > are a few very minor code cleanups (the three latest commits here
 >
https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
 > that have not made it into a rebuilt exe yet.
 >
 > FTR, we really ought to remove all but the 14.0 version from the
 > master branch.  We don't support building Python with any toolchain
 > older than 14.0 anymore, and the older toolchains are nigh impossible
 > to find anymore anyway.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command

[Petr Viktorin]

On 10/18/18 4:40 PM, Zachary Ware wrote:

On Thu, Oct 18, 2018 at 9:09 AM VanL  wrote:

Hi all,

I am looking into an issue associated with the wininst-*.exe files in the 
distutils/command subdirectory. It looks like these are the executable stubs 
used to create self-extracting zips for installation - but I am not 100% sure. 
It also looks like they include the calls to standard Windows functions to 
display the installer window.

I have a couple questions I need help with:
1) Am I correct about the function, and if not, what are they?


You are correct.  IIUC, they are checked in to allow creating those
installers from non-Windows platforms.


Is that the only reason for them?
At least on Linux, bdist_wininst does not work since at least Python 
3.2, as it tries to use a Windows-only encoding internally.

https://bugs.python.org/issue10945

If they're only there for non-Windows platforms, they're useless.


2) Where did these come from, and where is their source code?


Source can be found here:
https://github.com/python/cpython/tree/master/PC/bdist_wininst

The individual checked-in .exe files were each originally built by
whoever updated the Windows toolchain to a new version of MSVC
(Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
ones were added by Thomas Heller, presumably using whatever the
current toolchain(s) was (were) at the time).  A few of them have been
rebuilt after bug fixes in the source since they were added, mostly by
the same people, though I also see Mark Hammond and Raymond Hettinger
in the history (and Georg Brandl via svnmerge).  I notice that there
are a few very minor code cleanups (the three latest commits here
https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
that have not made it into a rebuilt exe yet.

FTR, we really ought to remove all but the 14.0 version from the
master branch.  We don't support building Python with any toolchain
older than 14.0 anymore, and the older toolchains are nigh impossible
to find anymore anyway.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580

[Petr Viktorin]

On 10/4/18 1:38 PM, Łukasz Langa wrote:


On 3 Oct 2018, at 23:10, Terry Reedy <mailto:tjre...@udel.edu>> wrote:


On 10/3/2018 8:12 AM, Jeroen Demeyer wrote:

Hello,
I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
titled "The C call protocol". He has co-authored several PEPs (PEP 
394, PEP 489, PEP 534, PEP 547, PEP 573), several of which involve 
extension modules.
Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
BDFL-Delegate.


To me, three experienced core devs approving of a 4th person as 
PEP-examiner is sufficient to proceed on a CPython implementation 
proposal.  I don't think we need to be paralyzed on this.


What you're saying is sensible, the team is small enough and tightly 
knit that we trust each other. However, trust is not the point. It's 
about clear expectations and avoiding anarchy. As Nick points out 
elsewhere, circumventing the lack of governance by "asking a few 
friends" on the core team creates a need for the new leadership to 
ratify those changes. Speaking frankly, it would be a major shit show if 
any of those changes were to be reverted. As the release manager of this 
version of Python, can I ask you please not to risk this?


Ironically, the governance model I am championing is one that would 
closely resemble what you're describing. A community of experts, no 
kings: https://discuss.python.org/t/pep-8012-the-community-model/156/


So it's really not that I disagree with you, I do. It's not that I don't 
trust Petr, I do. It's that I believe the core team needs to formalize 
how they want the project to proceed *before* they go run approving PEPs.


Łukasz, as the release manager for 3.8 you're the closest we have to an 
authority, so I defer to your judgment. No PEPs can currently be accepted.



Anyway, even if I was a *-delegate here, I would need to hear Mark 
Shannon's opinion on the PEP. Convincing him will probably be harder 
than convincing me.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dear core-devs

[Petr Viktorin]

On 10/4/18 9:34 AM, Victor Stinner wrote:

Hi,

If IBM wants a better Python support, it would help a lot if IBM pays 
for this development. With money, you can easily find core dev 
contractors. Antoine Pitrou has been paid in the past to enhance Python 
support in Solaris and it worked well.


Michael explicitly said this is a personal effort. IBM or other big 
money is not involved.


Is paying the best way to get features into Python? Does becoming a core 
dev mean you can now get paid for approving changes? Some of the 
implications are quite disturbing :(



Le mercredi 3 octobre 2018, Michael Felt > a écrit :

 >
 >
 > On 10/2/2018 11:34 PM, Terry Reedy wrote:
 >> On 10/2/2018 12:41 PM, Simon Cross wrote:
 >>> Are there any core devs that Michael or Erik could collaborate with?
 >>> Rather than rely on adhoc patch review from random core developers.
 >>
 >> You two might collaborate with each other to the extent of reviewing
 >> some of each other's PRs.
 > Might be difficult. We both, or at least I, claim ignorance of the
 > others platform. I still have a lot of PEP to learn, and my idea of a
 > bug-fix (for Python2) was seen by core-dev as a feature change. I would
 > not feel comfortable trying to mentor someone in things PEP, etc..
 >> That still leaves the issue of merging.
 > How much confidence is there in all the "CI" tests? Does that not offer
 > sufficient confidence for a core-dev to press merge.
 > How about "master" continuing to be what it is, but insert a new
 > "pre-master" branch that the buildbots actually test on (e.g., what is
 > now the 3.X) and have a 3.8 buildbot - for what is now the "master".
 >
 > PR would still be done based on master, but an "initial" merge would be
 > via the pre-master aka 3.X buildbot tests.
 >
 > How "friendly" git is - that it not become such a workload to keep it
 > clean - I cannot say. Still learning to use git. Better, but still do
 > not want to assume it would be easy.
 >
 > My hope is that it would make it easier to consider a "merge" step that
 > gets all the buildbots involved for even broader CI tests.
 >
 >>
 >>> Michael and Eric: Question -- are you interested in becoming core
 >>> developers at least for the purposes of maintaining these platforms in
 >>> future?
 >>
 >> Since adhoc is not working to get merges, I had this same suggestion.
 >> Michael and Erik, I presume you have gotten some guidelines on what
 >> modifications to C code might be accepted, and what concerns people 
have.

 > imho: guidelines - paraphrased - as little as possible :)
 >
 > I have many assumptions, and one of those is that my assumptions are
 > probably incorrect.
 > Goal: have AIX recognized as a Stable platform, even if not in the
 > highest supported category.
 > And that implies, support as far as I am able, to keep it "Stable".
 >>
 >> I think for tests, a separate test_aix.py might be a good idea for
 >> aix-only tests
 > Unclear to me how this would work. Too young in Python I guess (or just
 > a very old dog), but what test would be needed for AIX, or any other
 > platform, that would not need to be tested in some fashion for the
 > 'other' platforms. At a hunch, where there are many platform.system()
 > dependencies expected (e.g., test_posix, maybe doing something in the
 > class definition (is there a "Root" Object/Class that all inherit from.
 > Maybe a (read-only) "root" attribute (or is property better?) could be
 > the value of platform.system(), and iirc, might be used by as @property
 > in unittest. (so, if not in "root" class, then in something like
 > unittest/__init__.py.
 >
 > I hope to be "close" in "Python thinking" - enough that someone who
 > actually knows how the pieces fit together could come with a better, and
 > more appropriate guideline/implementation.
 >
 >> , while modification of other tests might be limited to adding skips.
 >> The idea would be to make it easy to remove aix stuff in the future if
 >> it again became unsupported.
 > IMHO: IBM and AIX do not mention it, but for openstack cloudmanagement
 > (very specifically cloud-init) AIX needs a recognized stable Python
 > implementation. I am "surprised" in the level of communication of IBM
 > with Python community.
 >
 > Personally, I do not see AIX as a specialized platform. Feels more like
 > the "last-standing" fully supported (commercial OEM) 'POSIX-UNIX'. Of
 > course my focus is narrow - so maybe there is a lot of support for
 > commercial platforms such as HPUX, Solaris, and other mainstream UNIXes.
 > Feel free to correct me!!
 >> Ditto for other specialized platforms.
 >>
 >>
 >>
 >>
 >
 > ___
 > Python-Dev mailing list
 > Python-Dev@python.org 
 > https://mail.python.org/mailman/listinfo/python-dev
 > Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

 >

___
Python-Dev 

Re: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580

[Petr Viktorin]

On 10/3/18 2:12 PM, Jeroen Demeyer wrote:

Hello,

I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
titled "The C call protocol". He has co-authored several PEPs (PEP 394, 
PEP 489, PEP 534, PEP 547, PEP 573), several of which involve extension 
modules.


Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
BDFL-Delegate.


I am well aware of the current governance issues, but several people 
have mentioned that the BDFL-Delegate process can still continue for 
now. I created a PR for the peps repository at 
https://github.com/python/peps/pull/797




Hello,
I don't think it's formally possible to do that now, but the following 
from elsewhere in the thread does make sense:


Antoine Pitrou:

Consensus would obviously work (if no-one opposes the proposed person,
then surely we don't need an elaborate governance model to decree that
said person can become the PEP delegate, no?).


Yes, it would feel very silly to have consensus and not be able to act 
on it. But without a governance (and approval process) it's hard to 
*ensure* we have consensus where all relevant voices have been heard.


On the other hand, I don't agree with Nick Coghlan here:

In this case, I'd consider it unlikely for either the PEP delegate appointment 
or any decisions about the PEP itself to be overturned - while it's a complex 
topic that definitely needs to go through the PEP process in order to work out 
the technical details, it isn't especially controversial in its own right (the 
most controversial aspect is whether it needs a new C level slot or not, and 
the PEP should clearly lay out the pros and cons of that)


PEP 580 *is* controversial -- there's the competing PEP 576 by Mark 
Shannon, who hasn't commented on this recently. Either would be an 
improvement, but choosing between them is a hard trade-off.
I'll leave technical stuff to another thread and concentrate on the 
process here.


When I'm happy with the PEP *and* if Mark says he's OK with it, I'll 
post a summary to Python-dev & Discourse with a special focus on the 
cons (e.g. the size increase of classes, which will affect everyone). If 
there are no "-1"s on the PEP itself or on the way it's discussed, let's 
treat PEP 580 as *provisionally* accepted, to be reverted if the new 
governance doesn't ratify it.


If there is no consensus, we'll need to wait for the new governance to 
decide.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Heap-allocated StructSequences

[Petr Viktorin]

On 09/13/18 23:34, Neil Schemenauer wrote:

On 2018-09-04, Eddie Elizondo wrote:

Solution:

   *   Fix the implementation of PyStructSequence_NewType:

The best solution would be to fix the implementation of this
function. This can easily be done by dynamically creating a
PyType_Spec and calling PyType_FromSpec


Hello Eddie,

Thank you for spending time to look into this.  Without studying the
details of your patch, your approach sounds correct to me.  I think
we should be allocating types from the heap and use PyType_FromSpec.
Having static type definitions living in the data segment cause too
many issues.

We have to assess how 3rd party extension modules would be affected
by this change.  Unless it is too hard to do, they should still
compile (perhaps with warnings) after your fix.  Do you know if
that's the case?  Looking at your changes to structseq.c, I can't
tell easily.

In any case, this should go into Victor's pythoncapi fork.  That
fork includes all the C-API cleanup we are hoping to make to CPython
(assuming we can figure out the backwards and forwards compatibility
issues).


Nope, Victor's fork doesn't include all C-API cleanup. There's an older 
long-term effort (PEP-384, PEP-489, the current contenders 576/579/580, 
and PEP-573 for the future). Converting things to use PyType_FromSpec 
falls in there. As long as the old API still works, these changes should 
go in (but they might need a PEP).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34595: How to format a type name?

[Petr Viktorin]

On 09/13/18 14:08, Victor Stinner wrote:

Le jeu. 13 sept. 2018 à 16:01, Eric V. Smith  a écrit :

* Add !t conversion to format string


I'm strongly opposed to this. This !t conversion would not be widely
applicable enough to be generally useful, and would need to be exposed
in the f-string and str.format() documentation, even though 99% of
programmers would never need or see it.


(I'm thinking aloud.)

In the Python code base, I found 115 lines using type(obj).__name__
and 228 lines using obj.__class__.__name__.

[...]

"!t" is not a big improvement over ":T" and "type(obj)".


I'm not sure if type(obj) or obj.__class__ should be used, but I can
say that they are different: obj.__class__ can be overriden:

[...]



Moreover, it's also possible to override the "type" symbol in the
global or local scope:

[...]

I don't think either of those are problematic. If you override 
`__class__` or `type`, things will behave weirdly, and that's OK.



One advantage of having a builtin formatter would be to always use
internally the builtin type() function to get the type of an object,
or not use "type()" in the current scope. The second advantage is to
prevent the need of having to decide between type(obj) and
obj.__class__ :-)



raise TypeError(f"must be str, not {obj!t}")

Should be written as:
raise TypeError(f"must be str, not {type(obj)}")

[...]


Do you want to modify str(type) to return a value different than repr(type)?

Or maybe it's just a typo and you wanted to write f"{type(obj):T}"?


Yes, AFAIK that was a typo.


I think "T" is a good idea, but I think you're adding in obj vs
type(obj) just because of the borrowed reference issue in Py_TYPE().
That issue is so much larger than string formatting the type of an
object that it shouldn't be addressed here.


Right, that's a side effect of the discussion on the C API. It seems
like Py_TYPE() has to go in the new C API. Sorry, the rationale is not
written down yet, but Dino convinced me that Py_TYPE() has to go :-)


I'll be happy when we get rid of Py_TYPE and get to use moving garbage 
collectors... but now is not the time.
The API for "%T" should be "give me the type". The best way to do that 
might change in the future.



But at this point, we're bikeshedding. I think all the relevant voices 
have been heard.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

[Petr Viktorin]

On 09/13/18 02:22, Jeroen Demeyer wrote:

On 2018-09-13 02:26, Petr Viktorin wrote:

The reference to PEP 573 is premature.


It seems to me that PEP 580 helps with the use case of PEP 573. In fact, 
it implements part of what PEP 573 proposes. So I don't see the problem 
with the reference to PEP 573. Even if the implementation of PEP 573 
changes, the problem statement will remain and that's what I'm referring 
to. 


If you agree I can
summarize rationale for "parent", as much as it concerns 580.


Sure. I still think that we should refer to PEP 573, but maybe we can 
summarize it also in PEP 580.


I want to make it clear that PEP 580 doesn't depend on 579. Reviewers 
don't need to agree with PEP 579 to accept 580.


Here's my proposed rewording: 
https://github.com/python/peps/pull/775/files?short_path=b34f00e#diff-b34f00eeb75773c32f9b22fd7fee9771



# Using tp_print

The tp_print gimmick is my biggest worry.
AFAIK there's no guarantee that a function pointer and Py_ssize_t are
the same size.


I'm not actually claiming anywhere that it is the same size.


Indeed, I was thinking ahead here. Backporting this to earlier versions 
of CPython will not be completely trivial, but let's leave it to Cython.




# Descriptor behavior

I'd say "SHOULD" rather than "MUST" here. The section describes how to
implement expected/reasonable behavior, but I see no need to limit that.


There *is* actually an important reason to limit it: it allows code to 
make assumptions on what __get__ does. This enables optimizations which 
wouldn't be possible otherwise. If you cannot be sure what __get__ does, 
then you cannot optimize


obj.method(x)

to

type(obj).method(obj, x)


I see now. Yes, that's reasonable.



"if func supports the C call protocol, then func.__set__ must not be
implemented." -- also, __delete__ should not be implemented, right?.


Indeed. I write Python but I think C API, so for me these are both 
really tp_descr_set.



PyCCall_FASTCALL is not a macro, shouldn't it be named PyCCall_FastCall?


What's the convention for that anyway? I assumed that capital letters 
meant a "really know what you are doing" function which could segfault 
if used badly.


Well, I don't think that's a useful distinction either. This is C; 
pretty much anything can segfault when used badly.


Macros tend to be "fast": Py_TYPE just gets a member of a struct; 
Py_INCREF just increments a number. METH_NOARGS is just a number. None 
of them are very dangerous.
IMO, PyTuple_GET_ITEM is not uppercase because it's dangerous, but 
because it just reaches into memory.


For me, whether something is a function or macro is just an 
implementation detail (which can change between Python versions) which 
should not affect the naming.


True. I'm not saying the convention is very strict or useful.



# C API functions

The function PyCFunction_GetFlags is, for better or worse, part of the
stable ABI. We shouldn't just give up on it. I'm fine with documenting
that it shouldn't be used, but for functions defined using
PyCFunction_New etc. it should continue behaving as before.
One solution could be to preserve the "definition time" METH_* flags in
the 0xFFF bits of cc_flags and use the other bits for CCALL_*.


I'm fine with that if you insist. However, it would be a silly solution 
to formally satisfy the "stable ABI" requirement without actually helping.


Yes, it's definitely very silly. But that's not a reason to break our 
promise to the users. After all it's called "stable ABI", not "useful 
ABI" :)


I agree with your other points that I didn't reply to and will make some 
edits to PEP 580.


Thank you!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

[Petr Viktorin]

On 06/20/18 01:53, Jeroen Demeyer wrote:

Hello,

Let me present PEP 579 and PEP 580.

PEP 579 is an informational meta-PEP, listing some of the issues with 
functions/methods implemented in C. The idea is to create several PEPs 
each fix some part of the issues mentioned in PEP 579.


PEP 580 is a standards track PEP to introduce a new "C call" protocol, 
which is an important part of PEP 579. In the reference implementation 
(which is work in progress), this protocol will be used by built-in 
functions and methods. However, it should be used by more classes in the 
future.


You find the texts at
https://www.python.org/dev/peps/pep-0579
https://www.python.org/dev/peps/pep-0580


Hi!
I finally had time to read the PEPs carefully.

Overall, great work! PEP 580 does look complicated, but it's well 
thought out and addresses real problems.


I think the main advantage over the competing PEP 576 is that it's a 
better foundation for solving Cython (and other C-API users) and my PEP 
573 (module state access from methods).



With that, I do have some comments.

The reference to PEP 573 is premature. If PEP 580 is implemented then 
PEP 573 will build on top, and I don't plan to update PEP 573 before 
that. So, I think 580 should be independent. If you agree I can 
summarize rationale for "parent", as much as it concerns 580.


# Using tp_print

The tp_print gimmick is my biggest worry.
AFAIK there's no guarantee that a function pointer and Py_ssize_t are 
the same size. That makes the backwards-compatibility typedef in the 
implementation is quite worrying:

typedef Py_ssize_t printfunc
I can see the benefit for backporting to earlier Python versions, and 
maybe that outweighs worries about exotic architectures, but the PEP 
should at least have more words on why this is not a problem.



# The C Call protocol

I really like the fact that, in the reference implementation, the flags 
are arranged in a way that allows a switch statement to select what to 
call. That should be noted, if only to explain why there's no guarantee 
of compatibility between Python versions.



# Descriptor behavior

I'd say "SHOULD" rather than "MUST" here. The section describes how to 
implement expected/reasonable behavior, but I see no need to limit that.


"if func supports the C call protocol, then func.__set__ must not be 
implemented." -- also, __delete__ should not be implemented, right?.



# Generic API functions

I'm a bit worried about PyCCall_FASTCALL's "kwds" argument accepting a 
dict, which is mutable. I wouldn't mind dropping that capability, but if 
it stays, we need to require that the callable promises to not modify it.


PyCCall_FASTCALL is not a macro, shouldn't it be named PyCCall_FastCall?


# C API functions

The function PyCFunction_GetFlags is, for better or worse, part of the 
stable ABI. We shouldn't just give up on it. I'm fine with documenting 
that it shouldn't be used, but for functions defined using 
PyCFunction_New etc. it should continue behaving as before.
One solution could be to preserve the "definition time" METH_* flags in 
the 0xFFF bits of cc_flags and use the other bits for CCALL_*.



# Stable ABI

The section should repeat that PyCFunction_ClsNew is added to the stable 
ABI (but nothing else).



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34595: How to format a type name?

[Petr Viktorin]

On 09/11/18 15:23, Victor Stinner wrote:

Hi,

Last week, I opened an issue to propose to add a new %T formatter to
PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat()
and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add
something to get the "fully qualified name" (FQN) of a type, ex
"datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name).
I proposed a second pull request to add %t (short) in addition to %T
(FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get
a wider discussion. So here I am.


After a discussion with Victor. I'll summarize where we are now.

There are actually two inconsistencies to fix:
- Python modules use `type(obj).__name__` and C extensions use 
`Py_TYPE(obj)->tp_name`, which inconsistent.
- Usage __name__ or __qualname__, and prepending __module__ or not, is 
inconsistent across types/modules.


It turns out that today, when you want to print out a type name, you 
nearly always want the fully qualified name (including the module unless 
it's "builtins"). So we can just have "%T" and not "%t". (Or we can add 
"%t" if a use case arises).


It should be possible to do this also in Python, preferably using a name 
similar to "%T".


Most of the usage is in error messages and __repr__, where we don't need 
to worry about compatibility too much.


It should be possible to get the name if you have the type object, but 
not an instance of it. So, the proposed `PyUnicode_FromFormat("%T", 
obj)` is incomplete -- if we go that way, we'll also need a function 
like PyType_GetFullName. Making "%T" work on the type, e.g. 
`PyUnicode_FromFormat("%T", Py_TYPE(obj))`, would be more general.


---

So, I propose adding a "%T" formatter to PyUnicode_FromFormat, to be 
used like this:

PyUnicode_FromFormat("%T", Py_TYPE(obj))
and a "T" format code for type.__format__, to be used like this:
f"{type(obj):T}"
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34595: How to format a type name?

[Petr Viktorin]

On 09/11/18 15:23, Victor Stinner wrote:

Hi,

Last week, I opened an issue to propose to add a new %T formatter to
PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat()
and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add
something to get the "fully qualified name" (FQN) of a type, ex
"datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name).
I proposed a second pull request to add %t (short) in addition to %T
(FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get
a wider discussion. So here I am.


The rationale for this change is to fix multiple issues:

* C extensions use Py_TYPE(obj)->tp_name which returns a fully
qualified name for C types, but the name (without the module) for
Python name. Python modules use type(obj).__name__ which always return
the short name.


That might be a genuine problem, but I wonder if "%T" is fixing the 
symptom rather than the cause here.

Or is this only an issue for PyUnicode_FromFormat()?


* currently, many C extensions truncate the type name: use "%.80s"
instead of "%s" to format a type name


That's an orthogonal issue -- you can change "%.80s" to "%s", and 
presumably you could use "%.80t" as well.



* "%s" with Py_TYPE(obj)->tp_name is used more than 200 times in the C
code, and I dislike this complex pattern. IMHO "%t" with obj would be
simpler to read, write and maintain.


I consider `Py_TYPE(obj)->tp_name` much more understandable than "%t".
It's longer to spell out, but it's quite self-documenting.


* I want C extensions and Python modules to have the same behavior:
respect the PEP 399. Petr considers that error messages are not part
of the PEP 399, but the issue is wider than only error messages.


The other major use is for __repr__, which AFAIK we also don't guarantee 
to be stable, so I don't think PEP 399 applies to it.
Having the same behavior between C and Python versions of a module is 
nice, but PEP 399 doesn't prescribe it. There are other differences as 
well -- for example, `_datetime.datetime` is immutable, and that's OK.


If error messages and __repr__s should be consistent between Python and 
the C accelerator, are you planning to write tests for all the affected 
modules when switching them to %T/%t?




The main issue is that at the C level, Py_TYPE(obj)->tp_name is
"usually" the fully qualified name for types defined in C, but it's
only the "short" name for types defined in Python.

For example, if you get the C accelerator "_datetime",
PyTYPE(obj)->tp_name of a datetime.timedelta object gives you
"datetime.timedelta", but if you don't have the accelerator, tp_name
is just "timedelta".

Another example, this script displays "mytimedelta(0)" if you have the
C accelerator, but "__main__.mytimedelta(0)" if you use the Python
implementation:
---
import sys
#sys.modules['_datetime'] = None
import datetime

class mytimedelta(datetime.timedelta):
 pass

print(repr(mytimedelta()))
---

So I would like to fix this kind of issue.


Type names are mainly used for two purposes:

* format an error message
* obj.__repr__()

It's unclear to me if we should use the "short" or the "fully
qualified" name. It should maybe be decided on a case by case basis.

There is also a 3rd usage: to implement __reduce__, here backward
compatibility matters.


Note: The discussion evolved since my first implementation of %T which
just used the not well defined Py_TYPE(obj)->tp_name.

--

Petr asked me why not exposing functions to get these names. For
example, with my second PR (not merged), there are 3 (private)
functions:

/* type.__name__ */
const char* _PyType_Name(PyTypeObject *type);
/* type.__qualname__ */
PyObject* _PyType_QualName(PyTypeObject *type);
* type.__module__ "." type.__qualname__ (but type.__qualname__ for
builtin types) */
PyObject * _PyType_FullName(PyTypeObject *type);

My concern here is that each caller has to handler error:

   PyErr_Format(PyExc_TypeError, "must be str, not %.100s",
Py_TYPE(obj)->tp_name);

would become:

   PyObject *type_name = _PyType_FullName(Py_TYPE(obj));
   if (name == NULL) { /* do something with this error ... */
   PyErr_Format(PyExc_TypeError, "must be str, not %U", type_name);
   Py_DECREF(name);

When I report an error, I dislike having to handle *new* errors... I
prefer that the error handling is done inside PyErr_Format() for me,
to reduce the risk of additional bugs.

--

Serhiy also asked if we could expose the same feature at the *Python*
level: provide something to get the fully qualified name of a type.
It's not just f"{type(obj).__module}.{type(obj).__name__}", but you
have to skip the module for b

Re: [Python-Dev] Can we split PEP 489 (extension module init) ?

[Petr Viktorin]

On 08/10/18 12:21, Stefan Behnel wrote:

Petr Viktorin schrieb am 10.08.2018 um 11:51:

On 08/10/18 11:21, Stefan Behnel wrote:

coming back to PEP 489 [1], the multi-phase extension module
initialization. We originally designed it as an "all or nothing" feature,
but as it turns out, the "all" part is so difficult to achieve that most
potential users end up with "nothing". So, my question is: could we split
it up so that projects can get at least the main advantages: module spec
and unicode module naming.

PEP 489 is a great protocol in the sense that it allows extension modules
to set themselves up in the same way that Python modules do: load, create
module, execute module code. Without it, creating the module and executing
its code are a single step that is outside of the control of CPython, which
prevents the module from knowing its metadata and CPython from knowing
up-front what the module will actually be.

Now, the problem with PEP 489 is that it requires support for reloading and
subinterpreters at the same time [2]. For this, extension modules must
essentially be free of static global state, which comprises both the module
code itself and any external native libraries that it uses. That is
somewhere between difficult and impossible to achieve. PEP 573 [3] explains
some of the reasons, and lists solutions for some of the issues, but cannot
solve the general problem that some extension modules simply cannot get rid
of their global state, and are therefore inherently incompatible with
reloading and subinterpreters.


Are there any issues that aren't explained in PEP 573?
I don't think Python modules should be *inherently* incompatible with
subinterpreters. Static global state is perhaps unavoidable in some cases,
but IMO it should be managed when it's exposed to Python.
If there are issues not in the PEPs, I'd like to collect the concrete cases
in some document.


There's always the case where an external native library simply isn't
re-entrant and/or requires configuration to be global. I know, there's
static linking and there are even ways to load an external shared library
multiple times, but that's just adding to the difficulties. Let's just
accept that some things are not easy enough to make for a good requirement.


For that case, I think the right thing to do is for the module to raise 
an extension when it's being initialized for the second time, or when 
the underlying library would be initialized for the second time.


"Avoid static global state" is a good rule of thumb for supporting 
subinterpreters nicely, but other strategies are possible.
If an underlying library just expects to be initialized once, and then 
work from several modules, the Python wrapper should ensure that (using 
global state, most likely). Other ways of handling things should be 
possible, depending on the underlying library.



I would like the requirement in [2] to be lifted in PEP 489, to make the
main features of the PEP generally available to all extension modules.

The question is then how to opt out of the subinterpreter support. The PEP
explicitly does not allow backporting new init slot functions/feeatures:

"Unknown slot IDs will cause the import to fail with SystemError."

But at least changing this in Py3.8 should be doable and would be really
nice.


I don't think we can just silently skip unknown slots -- that would mean
modules wouldn't be getting features they asked for.
Do you have some more sophisticated model for slots in mind, or is this
something to be designed?


Sorry for not being clear here. I was asking for changing the assumptions
that PEP 489 makes about modules that claim to support the multi-step
initialisation part of the PEP. Adding a new (flag?) slot was just one idea
for opting out of multi-initialisation support.


Would this be better than a flag + raising an error on init?
One big disadvantage of a big opt-out-of-everything button is that it 
doesn't encourage people to think about what the actual non-reentrant 
piece of code is.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can we split PEP 489 (extension module init) ?

[Petr Viktorin]

On 08/10/18 11:21, Stefan Behnel wrote:

Hi,

coming back to PEP 489 [1], the multi-phase extension module
initialization. We originally designed it as an "all or nothing" feature,
but as it turns out, the "all" part is so difficult to achieve that most
potential users end up with "nothing". So, my question is: could we split
it up so that projects can get at least the main advantages: module spec
and unicode module naming.

PEP 489 is a great protocol in the sense that it allows extension modules
to set themselves up in the same way that Python modules do: load, create
module, execute module code. Without it, creating the module and executing
its code are a single step that is outside of the control of CPython, which
prevents the module from knowing its metadata and CPython from knowing
up-front what the module will actually be.

Now, the problem with PEP 489 is that it requires support for reloading and
subinterpreters at the same time [2]. For this, extension modules must
essentially be free of static global state, which comprises both the module
code itself and any external native libraries that it uses. That is
somewhere between difficult and impossible to achieve. PEP 573 [3] explains
some of the reasons, and lists solutions for some of the issues, but cannot
solve the general problem that some extension modules simply cannot get rid
of their global state, and are therefore inherently incompatible with
reloading and subinterpreters.


Are there any issues that aren't explained in PEP 573?
I don't think Python modules should be *inherently* incompatible with 
subinterpreters. Static global state is perhaps unavoidable in some 
cases, but IMO it should be managed when it's exposed to Python.
If there are issues not in the PEPs, I'd like to collect the concrete 
cases in some document.



I would like the requirement in [2] to be lifted in PEP 489, to make the
main features of the PEP generally available to all extension modules.

The question is then how to opt out of the subinterpreter support. The PEP
explicitly does not allow backporting new init slot functions/feeatures:

"Unknown slot IDs will cause the import to fail with SystemError."

But at least changing this in Py3.8 should be doable and would be really nice.


I don't think we can just silently skip unknown slots -- that would mean 
modules wouldn't be getting features they asked for.
Do you have some more sophisticated model for slots in mind, or is this 
something to be designed?




What do you think?

Stefan



[1] https://www.python.org/dev/peps/pep-0489/
[2]
https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading
[3] https://www.python.org/dev/peps/pep-0573/


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Comparing PEP 576 and PEP 580

[Petr Viktorin]

On 07/05/18 13:59, Jeroen Demeyer wrote:

On 2018-07-05 13:32, INADA Naoki wrote:

Core devs interested in this area is limited resource.


I know and unfortunately there is nothing that I can do about that. It 
would be a pity that PEP 580 (or a variant like PEP 576) is not accepted 
simply because no core developer cares enough.


Hi,
I do care about this, and I'm really sorry I've been mostly silent here.
Unfortunately, this is the kind of work that can't be done with a few 
hours in the evenings, and currently an urgent project is sucking up all 
the big blocks of time I have :(

That project should be done in a month or two, however.





As far as I understand, there are some important topics to discuss.

a. Low level calling convention, including argument parsing API.
b. New API for calling objects without argument tuple and dict.
c. How more types can support FASTCALL, LOAD_METHOD and CALL_METHOD.
d. How to reorganize existing builtin types, without breaking stable ABI.


Right, that's why I wanted PEP 580 to be only about (c) and nothing 
else. I made the mistake in PEP 575 of also involving (d).


I still don't understand why we must finish (a) before we can even start 
discussing (c).



Reference implementation helps discussion.


METH_FASTCALL and argument parsing for METH_FASTCALL is already 
implemented in CPython. Not in documented public functions, but the 
implementation exists.


And PEP 580 also has a reference implementation:
https://github.com/jdemeyer/cpython/tree/pep580


Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failing tests (on a Linux distro)

[Petr Viktorin]

On 07/02/18 00:59, Miro Hrončok wrote:

On 1.7.2018 23:48, Matěj Cepl wrote:

On 2018-06-28, 00:58 GMT, Ned Deily wrote:

On behalf of the Python development community and the Python 3.7 release
team, we are pleased to announce the availability of Python 3.7.0.


I am working on updating openSUSE packages to python 3.7, but
I have hit quite large number of failing tests (the testsuite
obviously passed with 3.6), see
https://build.opensuse.org/package/show/home:mcepl:work/python3
(click on the red "failed" label to get logs). I fell into
a bout of depression, only to discover that we are not alone in
this problem ... Debian doesn't seem to do much better
https://is.gd/HKBU4j. Surprisingly, Fedora seems to pass the
testsuite https://is.gd/E0KA53; interesting, I will have to
investigate which of their many patches did the trick.


Note that we (=Fedora) unfortunately skip some tests.

https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec#_1051

https://src.fedoraproject.org/rpms/python3/blob/master/f/00160-disable-test_fs_holes-in-rpm-build.patch 



https://src.fedoraproject.org/rpms/python3/blob/master/f/00163-disable-parts-of-test_socket-in-rpm-build.patch 


[with my Fedora hat on]

Fedora* has been building python37 since the alphas, so the final update 
to rc/stable was smoother. But it also means we aren't solving the same 
issues as SUSE now, so we won't be able to help all that much :(

Do consider trying out alphas/betas in SUSE next time!
Anyway, the SUSE tests seem  to fail on .pyc files. The main change in 
that area was [PEP 552], try starting there. AFAIK, SUSE is ahead of 
Fedora in the reproducible builds area; perhaps that's where the 
difference is.


And while I'm responding here, a bit of reflection and a heads-up:
What Fedora as a distro should do better next time is re-build the 
entire ecosystem with a new Python version. For 3.7 we started doing 
that too late, and there are way too many projects that weren't prepared 
for `async` as keyword and PEP 479 (StopIteration handling).
If you run into similar problems in SUSE, you might want to take a look 
at issues tracked under [Fedora bug 1565020].



[PEP 552]: https://www.python.org/dev/peps/pep-0552/

[Fedora bug 1565020]: 
https://bugzilla.redhat.com/showdependencytree.cgi?id=1565020


* Thanks to Miro Hrončok for most of the work in Fedora
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Policy on refactoring/clean up

[Petr Viktorin]

On 06/26/18 14:13, Jeroen Demeyer wrote:

On 2018-06-26 13:54, Ivan Pozdeev via Python-Dev wrote:

This is exactly what that the YAGNI principle is about, and Inada was
right to point to it. Until you have an immediate practical need for
something, you don't really know the shape and form for it that you will
be the most comfortable with. Thus any "would be nice to have"
tinkerings are essentially a waste of time and possibly a degradation,
too: you'll very likely have to change them again when the real need
arises -- while having to live with any drawbacks in the meantime.


It is important to clarify that this is exactly what I did. I *have* an 
implementation of PEP 580 and it's based on that PR 7909.


I just think that this PR makes sense independently of whether PEP 580 
will be accepted.



So, if you suggest those changes together with the PEP 580 PR


That sounds like a bad idea because that would be mixing two issues in 
one PR. If I want to increase my chances of getting PEP 580 and its 
implementation accepted, I shouldn't bring in unrelated changes.


To put it in a different perspective: if somebody else would make a PR 
to one of my projects doing a refactoring and adding new features, I 
would ask them to split it up.


Actually, that's exactly what we *did* ask Jeroen with his earlier 
proposal for PEP 575, where the implementation ended up being quite big. 
Split the changes to make it more manageable.


Unfortunately I haven't had time to study this PR yet (work is taking 
all my time lately), but I trust that Jeroen will propose actual 
improvements on top of the clean-up.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] the new(-ish) dict ordering vs hash randomization

[Petr Viktorin]

On 06/18/18 15:13, Ethan Furman wrote:
I'm sure we've already had this conversation, but my google-fu is 
failing me.


Can someone provide a link to a discussion explaining why the new 
ordering of dictionaries does not defeat the hash-randomization 
non-ordering we added a few versions ago?


Hi,
Modern dicts have an array of contents (which is used for iterating the 
dict, and thus iteration doesn't touch hashes at all), and a separate 
hash table of indexes (which still enjoys the benefits of hash 
randomization).


See Raymond Hettinger's initial post from 2012: 
https://mail.python.org/pipermail/python-dev/2012-December/123028.html


A technical overview of the idea is on the PyPy blog:
https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/15/18 17:55, Jeroen Demeyer wrote:

On 2018-05-15 18:36, Petr Viktorin wrote:

Naturally, large-scale
changes have less of a chance there.


Does it really matter that much how large the change is? I think you are 
focusing too much on the change instead of the end result.


As I said in my previous post, I could certainly make less disruptive 
changes. But would that really be better? (If you think that the answer 
is "yes" here, I honestly want to know).


Yes, I believe it is better.
The larger a change is, the harder it is to understand, meaning that 
less people can meaningfully join the conversation, think about how it 
interacts with their own use cases, and notice (and think through) any 
unpleasant details.

Less disruptive changes tend to have a better backwards compatibility story.
A less intertwined change makes it easier to revert just a single part, 
in case that becomes necessary.


I could make the code less different than today but at the cost of added 
complexity. Building on top of the existing code is like building on a 
bad foundation: the higher you build, the messier it gets. Instead, I 
propose a solid new foundation. Of course, that requires more work to 
build but once it is built, the finished building looks a lot better.


To continue the analogy: the tenants have been customizing their 
apartments inside that building, possibly depending on structural 
details that we might think should be hidden from them. And they expect 
to continue living there while the foundation is being swapped under them :)



With such a "finished product" PEP, it's hard to see if some of the
various problems could be solved in a better way -- faster, more
maintainable, or less disruptive.


With "faster", you mean runtime speed? I'm pretty confident that we 
won't lose anything there.


As I argued above, my PEP might very well make things "more 
maintainable", but this is of course very subjective. And "less 
disruptive" was never a goal for this PEP.



It's also harder from a psychological point of view: you obviously
already put in a lot of good work, and it's harder to waste that work if
an even better solution is found.


I hope that this won't be my psychology. As a developer, I prefer to 
focus on problems rather than on solutions: I don't want to push a 
particular solution, I want to fix a particular problem. If an even 
better solution is accepted, I will be a very happy man.


What I would hate is that this PEP gets rejected because some people 
claim that the problem can be solved in a better way, but without 
actually suggesting such a better way.


Mark Shannon has an upcoming PEP with an alternative to some of the 
issues. (Not all of them – but less intertwined is better, all else 
being equal.)



Is a branching class hierarchy, with quite a few new of flags for
feature selection, the kind of simplicity we want?


Maybe yes because it *concentrates* all complexity in one small place. 
Currently, we have several independent classes 
(builtin_function_or_method, method_descriptor, function, method) which 
all require various forms of special casing in the interpreter with some 
code duplication. With my PEP, this all goes away and instead we need to 
understand just one class, namely base_function.



Would it be possible to first decouple things, reducing the complexity,
and then tackle the individual problems?


What do you mean with "decouple things"? Can you be more concrete?


Currently, the "outside" of a function (how it looks when introspected) 
is tied to the "inside" (what happens internally when it's called). 
That's what I'd like to see decoupled.
Can we better enable pydoc/IPython developers to tackle introspection 
problems without wading deep in the internals and call optimizations?



The class hierarchy still makes it hard to decouple the introspection
side (how functions look on the outside) from the calling mechanism (how
the calling works internally).


Any class who wants to profit from fast function calls can inherit from 
base_function. It can add whatever attributes it wants and it can choose 
to implement documentation and/or introspection in whatever way it 
wants. It can choose to not care about that at all. That looks very 
decoupled to me.


But, it still has to inherit from base_function to "look like a 
function". Can we remove that limitation in favor of duck typing?



Starting from an idea and ironing out the details it lets you (and, if
since you published results, everyone else) figure out the tricky
details. But ultimately it's exploring one path of doing things – it
doesn't necessarily lead to the best way of doing something.


So far I haven't seen any other proposals...


That's a good question. Maybe inspect.isfunction() serves too many use
cases to be useful. Cython functons should behave like "def" functions
in some cases, and like buil

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/15/18 05:15, Jeroen Demeyer wrote:

On 2018-05-14 19:56, Petr Viktorin wrote:

It does quite a lot of things, and the changes are all intertwined,
which will make it hard to get reviewed and accepted.


The problem is that many things *are* already intertwined currently. You 
cannot deal with functions without involving methods for example.


An important note is that it was never my goal to create a minimal PEP. 
I did not aim for changing as little as possible. I was thinking: we are 
changing functions, what would be the best way to implement them?


That might be a problem. For the change to be accepted, a core developer 
will need to commit to maintaining the code, understand it, and accept 
responsibility for anything that's broken. Naturally, large-scale 
changes have less of a chance there.


With such a "finished product" PEP, it's hard to see if some of the 
various problems could be solved in a better way -- faster, more 
maintainable, or less disruptive.
It's also harder from a psychological point of view: you obviously 
already put in a lot of good work, and it's harder to waste that work if 
an even better solution is found. (I always tell Marcel to view 
large-scale changes as a hands-on learning experiment -- more likely to 
be thrown away than accepted -- rather than as creating a finished project.)


The main goal was fixing introspection but a secondary goal was fixing 
many of the existing warts with functions. Probably this secondary goal 
will in the end be more important for the general Python community.


I would argue that my PEP may look complicated, but I'm sure that the 
end result will be a simpler implementation than we have today. Instead 
of having four related classes implementing similar functionality 
(builtin_function_or_method, method, method_descriptor and function), we 
have just one (base_function). The existing classes like method still 
exist with my PEP but a lot of the core functionality is implemented in 
the common base_function.


Is a branching class hierarchy, with quite a few new of flags for 
feature selection, the kind of simplicity we want?
Would it be possible to first decouple things, reducing the complexity, 
and then tackle the individual problems?


This is really one of the key points: while my PEP *could* be 
implemented without the base_function class, the resulting code would be 
far more complicated.



Are there parts that can be left to a subsequent PEP, to simplify the
document (and implementation)?


It depends. The current PEP is more or less a finished product. You can 
of course pick parts of the PEP and implement those, but then those 
parts will be somewhat meaningless individually.


But if PEP 575 is accepted "in principle" (you accept the new class 
hierarchy for functions), then the details could be spread over several 
PEPs. But those individual PEPs would only make sense in the light of 
PEP 575.


Well, that's the thing I'm not sure about.
The class hierarchy still makes it hard to decouple the introspection 
side (how functions look on the outside) from the calling mechanism (how 
the calling works internally). It fear that it is replacing complexity 
with a different kind of complexity.
So my main question now is, can this all be *simplified* rather than 
*reorganized*? It's a genuine question – I don't know, but I feel it 
should be explored more.


A few small details could be left out, such as METH_BINDING. But that 
wouldn't yield a significant simplification.



It seems to me that the current complexity is (partly) due to the fact
that how functions are *called* is tied to how they are *introspected*.


The *existing* situation is that introspection is totally tied to how 
functions are called. So I would argue that my PEP improves on that by 
removing some of those ties by moving __call__ to a common base class.



Maybe we can change `inspect` to use duck-typing instead of isinstance?


That was rejected on https://bugs.python.org/issue30071


Then, if built-in functions were subclassable, Cython functions could
need to provide appropriate __code__/__defaults__/__kwdefaults__
attributes that inspect would pick up.


Of course, that's possible. I don't think that it would be a *better* 
solution than my PEP though.


Essentially, my PEP started from that idea. But then you realize that 
you'll need to handle not only built-in functions but also method 
descriptors (unbound methods of extension types). And you'll want to 
allow __get__ for the new subclasses. For efficiency, you really want to 
implement __get__ in the base classes (both builtin_function_or_method 
and method_descriptor) because of optimizations combining __get__ and 
__call__ (the LOAD_METHOD and CALL_METHOD opcodes). And then you realize 
that it makes no sense to duplicate all that functionality in both 
classes. So you add a new base class. You already end up with a major 
part of my PEP this way.


Starting from an 

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.


The PEP says:


User flags: METH_CUSTOM and METH_USRx
These flags are meant for applications that want to use tp_methods for an 
extension type or m_methods for a module but that do not want the default 
built-in functions to be created. Those applications would set METH_CUSTOM. The 
application is also free to use METH_USR0, ..., METH_USR7 for its own purposes, 
for example to customize the creation of special function instances.

There is no immediate concrete use case, but we expect that tools which 
auto-generate functions or extension types may want to define custom flags. 
Given that it costs essentially nothing to have these flags, it seems like a 
good idea to allow it.


Why are these flags added?
They aren't free – the space of available flags is not infinite. If 
something (Cython?) needs eight of them, it would be nice to mention the 
use case, at least as an example.


What should Python do with a m_methods entry that has METH_CUSTOM set? 
Again it would be nice to have an example or use case.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.



Hi,
I'm reading the PEP thoroughly, trying to "swap it into my brain" for 
the next few days.
It does quite a lot of things, and the changes are all intertwined, 
which will make it hard to get reviewed and accepted.
Are there parts that can be left to a subsequent PEP, to simplify the 
document (and implementation)?
It seems to me that the current complexity is (partly) due to the fact 
that how functions are *called* is tied to how they are *introspected*. 
Perhaps starting to separate that is a better way to untangle things 
than arranging a class hierarchy?



Can the problem of allowing introspection ("It is currently not possible 
to implement a function efficiently in C (only built-in functions can do 
that) while still allowing introspection like inspect.signature or 
inspect.getsourcefile (only Python functions can do that)") be solved in 
a better way?


Maybe we can change `inspect` to use duck-typing instead of isinstance? 
Then, if built-in functions were subclassable, Cython functions could 
need to provide appropriate __code__/__defaults__/__kwdefaults__ 
attributes that inspect would pick up.
Maybe we could eve add more attributes (__isgenerator__?) to separate 
how a function is called from how it should be introspected -- e.g. make 
inspect not consult co_flags.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 394: Allow the `python` command to not be installed (and other minor edits)

[Petr Viktorin]

Hello,
After discussion on the [Pull Request], my update to PEP 394 changed 
scope somewhat. The new major changes are:


- The `python` command may not exist at all in some cases (see the PEP 
for details)
- The paragraph about the anticipated future where python points to 
Python 3 is removed. (We'd rather see a future where `python` doesn't 
exist and one always has to specify `python2` or `python3`.)
- The PEP now explicitly says that in an active venv, python means that 
venv's interpreter. (Some view this as a design mistake, but one not 
worth reverting now.)


There are also other edits and clarifications.

Thanks for everyone involved, especially Guido for pitching in with the 
intended direction -- which was not clear from (or is genuinely 
different from) the 7-year-old PEP!


I'll keep the PR open for a day or so, in case someone still wants to 
comment.


[Pull Request]: https://github.com/python/peps/pull/630


On 04/26/18 19:21, Ben Finney wrote:

Petr Viktorin <encu...@gmail.com> writes:


In Fedora, I found that PEP 394's strict recommendation that `python`
points to `python2` is holding us back.


I have read the message, but I don't see how you draw the link that PEP
394 is holding you back.


The problems are:
- For developers that are not following the language's development,
the fact that `python` invokes `python2` sends a strong signal that 2
is somehow the preferred version, and it's OK to start new projects in
it.


I agree with the statement you make later in the message:


[…] we feel that the only way to *enforce* that guidelines is to
provide environments where the `python` command does not work (unless
explicitly installed).


Yes. The ‘python’ command is confusing, for the reasons you say. There
should be ‘python2’ and ‘python3’ commands for Python 2 and Python 3
respectively, and no ‘python’ command should be installed by the
operating system.

The fact that ‘/usr/bin/python’ exists is an historical accident, and I
agree with the proposal you state: the best way to correct the confusion
is to bar the confusing command from being installed by packages.


- Users and sysadmins that *do* want to “live in the future” are
switching the symlink to `python3` themselves. We would like to give
them a supported, documented way to do so -- and make surer they're
aware of the caveats.


The supported, documented way to add a command pointing to a different
command already exists, and there is no need to make a Python-specific
special case.

Users who want to make a ‘python’ alias can do so in their shell; this
is supported and documented.

Users who want to add a new command file can add a suitable directory
(e.g. ‘$HOME/bin’) to their ‘PATH’ variable, and put a symlink in there
named ‘python’. This is supported and documented.

Sysadmins who want to create a system-wide command ‘python’ can put a
symlink at ‘/usr/local/bin/python’. This is supported and documented.

I disagree with making some special-case extra way; that would be both
cunfusing and superfluous.


- The `python` command is still not available out-of-the box on macOS,
so it didn't completely live up to the expectation of being the
cross-platform way to launch 2/3 source compatile scripts.


That is one of the minor ways which macOS fails to conform to
community-agreed conventions. We should not let that intransigence
distort our discussion of best practices.


To help solve these, I would like to relax recommendations on the Unix
``python -> python2`` symlink in these cases:


For the above reasons, I disagree that PEP 394 is limiting what you want
to do on free-software operating systems.

For non-free operating systems, I don't think the already-discussed PEP
394 should be weakened if the operating system vendor fails to conform.


- Users and administrators can, by a deliberate action, change
``python`` to invoke Python 3.


Yes. That is well-known and long-standardised on Unix operating systems,
and is much more broadly understood than any Python-specific special
case would be. So I don't see how anyone is being held back.

I trust that PEP 394 will not be weakened in its effect, and I wish you
well with using the already-supported, already-documented, PEP-394
compatible means to add local customisations for a ‘python’ command.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 394 update proposal: Allow changing the `python` command in some cases

[Petr Viktorin]

On 04/27/18 02:03, Ben Finney wrote:

Ben Finney <ben+pyt...@benfinney.id.au> writes:


Petr Viktorin <encu...@gmail.com> writes:


[…] we feel that the only way to *enforce* that guidelines is to
provide environments where the `python` command does not work
(unless explicitly installed).


Yes. The ‘python’ command is confusing, for the reasons you say. There
should be ‘python2’ and ‘python3’ commands for Python 2 and Python 3
respectively, and no ‘python’ command should be installed by the
operating system.

The fact that ‘/usr/bin/python’ exists is an historical accident, and I
agree with the proposal you state: the best way to correct the confusion
is to bar the confusing command from being installed by packages.


Because the above is ambiguous, I'll clarify: I am not calling for, and
PEP 394 does not call for, the banishment of the ‘python’ command.


Well, Guido *is* calling for it :)
It would break too many things, but after discussions on the PR, it's 
clear that we want a future where the "python" doesn't exist.

But while it's available, it should point to Python 2.


What I'm saying is that muddying the rules further on what ‘python’ may
or may not mean is *worse than* banishing the ‘python’ command entirely.


That's also consistent with the PR discussion. (But not that much with 
the original PEP, which said `python` is expected to eventually mean 
`python3`.)



So, short of banishing ‘python’ entirely, I think PEP 394 is already a
good clear way to address the issue. Existing, documented and supported
means to locally modify a ‘python’ command already exist and should be
sufficient.


I trust that PEP 394 will not be weakened in its effect, and I wish you
well with using the already-supported, already-documented, PEP-394
compatible means to add local customisations for a ‘python’ command.


Right. But some already-supported, already-documented mechanisms like 
Debian Alternatives or alternate package repos, are not compatible with 
PEP 394.
And as a PEP 394 compliant distro, we also won't be promoting the 
/usr/local or $HOME/bin ways to change `python` (which makes me a bit 
sad, because that documentation might have included a link to the 
caveats listed in the PEP).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/26/18 09:32, Thomas Wouters wrote:


Thanks for working on this, Marcel (and Petr). This looks like an 
ambitious intern project :) Couple of questions and comments in-line.


Indeed, but also a multi-year one. Sometimes it seems like Python is 
moving too fast under us!


Specifically, PEP 573 (Unifying function/method classes) andto a smaller 
extent PEP 567 (Context Variables) seem to be solving similar problems 
as this PEP. I'm inclined to put this PEP temporarily on hold, help 575 
get accepted, and then rebase on top of that.



On Mon, Apr 23, 2018 at 12:36 PM, Marcel Plch > wrote:

PEP: 573

[...]


Abstract



[...]


Additionaly, support for easier creation of immutable exception
classes is added.


I'm not a fan of using 'immutable' here, or in the API function name. I 
understand the types are to some extent immutable (apart from their 
refcount, I assume), but I think it's going to be too easy to confuse it 
with types whose *instances* are immutable. (We do occasionally say 
things like "tuples are an immutable type".) Since the point is that 
they behave like statically defined ones, perhaps 'Static' would be a 
reasonable replacement.


That was the first naming choice, but then on the other hand, "static 
heap type" sounds like an oxymoron -- in C, static things don't live on 
the heap.

Naming is hard :(


[...]

Slot methods


The above changes don't cover slot methods, such as ``tp_iter`` or
``nb_add``.

The problem with slot methods is that their C API is fixed, so we can't
simply add a new argument to pass in the defining class.
Two possible solutions have been proposed to this problem:

     * Look up the class through walking the MRO.
       This is potentially expensive, but will be useful if
performance is not
       a problem (such as when raising a module-level exception).
     * Storing a pointer to the defining class of each slot in a
separate table,
       ``__typeslots__`` [#typeslots-mail]_.  This is technically
feasible and fast,
       but quite invasive.

Due to the invasiveness of the latter approach, this PEP proposes
adding an MRO walking
helper for use in slot method implementations, deferring the more
complex alternative
as a potential future optimisation. Modules affected by this concern
also have the
option of using thread-local state or PEP 567 context variables, or
else defining their
own reload-friendly lookup caching scheme.


I do not believe walking the MRO is going to work without reworking the 
implementation of types, specifically how typeobject.c deals with slots 
of subclasses: in some cases copies the slots from the base class (see 
inherit_slots() and from where it's called). I believe this would cause 
problems if, for example, you define type X in module A, subclass it 
from type Y in module B without overriding the slot, and try to find the 
module object for A from the slot implementation. I don't think copying 
slots is a requirement for the desired semantics, but it's going to be 
fairly involved to rewrite it to do something else. There's also 
backward-compatibility to consider: third-party libraries can be 
inheriting from builtin types (e.g. numpy does this extensively) using 
the same copying-slot mechanism, which means those builtin types can't 
use the MRO walking to find their module without breaking compatibility 
with those third-party libraries.


The MRO walking code needs to deal with copied slots. It's not 
straigtforward, but I do think it's possible.



[...]

Static exceptions
-

A new function will be added::

     int PyErr_PrepareImmutableException(PyTypeObject **exc,
                                      const char *name,
                                      const char *doc,
                                      PyObject *base)

Creates an immutable exception type which can be shared
across multiple module objects.


How is this going to deal with type.__subclasses__()? Is re-using the 
static type object between reloads and sub-interpreters important enough 
to warrant the different behaviour? What if sub-interpreters end up 
wanting to disallow sharing objects between them?


Argh. Yes, subclasses seem to be the obvious shortcoming that a fresh 
pair of eyes is bound to find. Thanks!

This sends the proposal back to the drawing board.



[...]

Modules Converted in the Initial Implementation
---

To validate the approach, several modules will be modified during
the initial implementation:

The ``zipimport``, ``_io``, ``_elementtree``, and ``_csv`` modules
will be ported to PEP 489 multiphase initialization.


zipimport currently caches things in C globals. Changing it to use PEP 
489 multi-phase 

[Python-Dev] PEP 394 update proposal: Allow changing the `python` command in some cases

[Petr Viktorin]

Hello,
In Fedora, I found that PEP 394's strict recommendation that `python` 
points to `python2` is holding us back. From discussions on Zulip and 
elsewhere it's clear that this recommendation is not changing any time 
soon, but I would like to officially relax it in several cases.


The problems are:
- For developers that are not following the language's development, the 
fact that `python` invokes `python2` sends a strong signal that 2 is 
somehow the preferred version, and it's OK to start new projects in it.
- Users and sysadmins that *do* want to “live in the future” are 
switching the symlink to `python3` themselves. We would like to give 
them a supported, documented way to do so -- and make surer they're 
aware of the caveats.
- The `python` command is still not available out-of-the box on macOS, 
so it didn't completely live up to the expectation of being the 
cross-platform way to launch 2/3 source compatile scripts.
- `python` in the shebang line can mean *either* that a script is 
carefully written to be 2/3 compatible, *or* that the author/maintainer 
is lazy or unaware of the recommendations. While Fedora guidelines have 
long banned the unversioned command, we feel that the only way to 
*enforce* that guidelines is to provide environments where the `python` 
command does not work (unless explicitly installed).


To help solve these, I would like to relax recommendations on the Unix 
``python -> python2`` symlink in these cases:


- Users and administrators can, by a deliberate action, change 
``python`` to invoke Python 3. (Activating a venv counts as such an 
action, but so would e.g. using alternates, installing a non-default 
overriding package, or replacing /usr/bin/python.)
- In controlled environments where being explicit is valued more than 
user experience (test environments, build systems, etc.), distributions 
can omit the `python` command even when `python2` is installed.


I have filed these changes as a pull request here:

  https://github.com/python/peps/pull/630

The PR also spells out several other things, which I felt were hidden 
between the lines -- but correct me if you disagree with my reading.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/25/18 14:46, Jeroen Demeyer wrote:

On 2018-04-25 20:33, Petr Viktorin wrote:

Perhaps "m_objclass" could point to the module in this case


That was exactly my idea also today. Instead of treating m_objclass as 
the defining class, we should generalize it to be the "parent" of the 
function: either the class or the module.


Great to hear we think alike.

However, I think that while reusing the pointer is nice to save space, 
the two concepts should still be separate, because "defining module" is 
a reasonable concept even for methods. In particular:

- There should be *separate* accessor functions for:
  - getting the defining class
  - getting the defining module
- The latter would later (in PEP 573) be extended to return the defining 
module even for class methods (when available)

- In Python code, __objclass__ should be the defining class, not the module.
- The C field should have a different name (m_parent?), so it isn't that 
strongly associated with __objclass__.


Does that sound reasonable?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/24/18 13:12, Jeroen Demeyer wrote:

On 2018-04-24 16:34, Jeroen Demeyer wrote:

On the other hand, if you are passing the function object, then you can
get __self__ from it (unless it's an unbound method: in that case
__self__ is NULL and self is really args[0]). So there wouldn't be a
need for passing "self". I'm not saying that this is better than passing
"self" explicitly... I haven't yet decided what is best.


One thing I realized from PEP 573: the fact that __self__ for built-in 
functions is set to the module is considered a feature. I never 
understood the reason for it (and I don't know if the original reason 
was the same as the reason in PEP 573).


If we want to continue supporting that and we also want to support 
__get__ for built-in functions (to make them act as methods), then there 
are really two "selfs": there is the "self" from the method (the object 
that it's bound to) and the "self" from the built-in function (the 
module). To support that, passing *both* the function and "self" seems 
like the best way.


You're talking about functions with METH_BINDING here, right?
There the other "self" would be the defining module.
It might make sense to pass that also in the struct, rather than as an 
additional argument.
Perhaps "m_objclass" could point to the module in this case, or a new 
pointer could be added.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/23/18 14:04, Jeroen Demeyer wrote:

Hello,

I just saw this PEP. There is a bit of overlap between PEP 573 and PEP 
575 since these both change the calling convention for built-in methods. 
In particular, PEP 575 also proposes to add a "defining class" member 
(for different reasons). In PEP 575, this is added to the PyCFunction 
struct itself instead of a separate struct PyCMethod.


The reason you just saw the pep is because it wasn't posted to 
python-dev yet, and it wasn't posted yet because we first want to look 
for potential conflicts with 573.


(There's no one forking on PEP 573 full-time, so unfortunately it's 
progressing slower than I'd like.)


It would be nice to justify whether you really need a new class 
(PyCMethod_Type) to support METH_METHOD. It looks strange to me that the 
class of some object depends on an implementation detail like whether 
METH_METHOD is specified.


The current PEP 573 implies that backwards compatibility concerns would 
arise every time that METH_METHOD is added to an existing method. People 
have asked questions on PEP 575 about that: it would break code 
depending on "types.BuiltinFunctionType" for example. You could instead 
just change PyCFunctionObject to add that field (that's what I did in 
PEP 575).


For practical reasons, it would be nice to implement PEP 573 and PEP 575 
together as they affect the same code (assuming that both PEPs are 
accepted of course).


I currently even think PEP 575 can go forward *before* PEP 573. Having 
__objclass__ on methods of extension classes does sounds like a more 
elegant solution! And for PEP 573 it would mean one less annoying 
problem to solve.



Reading PEP 575, I miss an explanation of what __objclass__ *is* -- it 
only says the technical restrictions on it. For PEP 575 I found it 
pretty important that the reader gets a good mental picture of what it 
is, but it's a bit difficult to explain succinctly.

Maybe something like this would help make it clearer in PEP 573?

If set, `__objclass__` is the class that defines the method (which might 
be a superclass of `type(self)`). Functionally it's equivalent to the 
`__class__` cell [0] in Python code.


[0] 
https://docs.python.org/3/reference/datamodel.html#creating-the-class-object


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state

[Petr Viktorin]

On 12/19/2017 04:19 PM, Antoine Pitrou wrote:

On Tue, 19 Dec 2017 16:10:06 +0100
Petr Viktorin <encu...@gmail.com> wrote:


Speaking of which, the doc is not very clear: is PEP 489 required for
multi-interpreter support or is PyModule_GetState() sufficient?


Yes, it is possible to have proper subinterpreter support without
multi-phase init.


Thanks.  I guess the C API docs need a user-friendly section laying out
the various methods for initializing a module, and their various
advantages :-)


That, or eventually remove multi-phase init's disadvantages, and have 
just one way to do it :)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state

[Petr Viktorin]

On Thu, Dec 14, 2017 at 12:00 PM, Antoine Pitrou  wrote:
> On Thu, 14 Dec 2017 17:00:10 +1000
> Nick Coghlan  wrote:
>> On 14 Dec. 2017 9:19 am, "Antoine Pitrou"  wrote:
>>
>>
>> Hello,
>>
>> After debugging a crash on AppVeyor for a submitter's PR
>> (see https://github.com/python/cpython/pull/4611 ), I came to the
>> following diagnosis: converting the "atexit" module (which is a
>> built-in C extension) to PEP 489 multiphase initialization can lead to
>> its m_traverse function (and presumably also m_clear and m_free) to be
>> called while not module state is yet registered: that is,
>> `PyModule_GetState(self)` when called from m_traverse returns NULL!
>>
>> Is that an expected or known subtlety?
>>
>>
>> Not that I'm aware of, so I'd be inclined to classify it as a bug in the
>> way we're handling multi-phase initialisation unless/until we determine
>> there's no way to preserve the existing invariant from the single phase
>> case.
>
> Speaking of which, the doc is not very clear: is PEP 489 required for
> multi-interpreter support or is PyModule_GetState() sufficient?

Yes, it is possible to have proper subinterpreter support without
multi-phase init.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state

[Petr Viktorin]

On 12/14/2017 12:00 PM, Antoine Pitrou wrote:

On Thu, 14 Dec 2017 17:00:10 +1000
Nick Coghlan  wrote:

On 14 Dec. 2017 9:19 am, "Antoine Pitrou"  wrote:


Hello,

After debugging a crash on AppVeyor for a submitter's PR
(see https://github.com/python/cpython/pull/4611 ), I came to the
following diagnosis: converting the "atexit" module (which is a
built-in C extension) to PEP 489 multiphase initialization can lead to
its m_traverse function (and presumably also m_clear and m_free) to be
called while not module state is yet registered: that is,
`PyModule_GetState(self)` when called from m_traverse returns NULL!

Is that an expected or known subtlety?


Thank you for looking into this, Antoine!


Not that I'm aware of, so I'd be inclined to classify it as a bug in the
way we're handling multi-phase initialisation unless/until we determine
there's no way to preserve the existing invariant from the single phase
case.


Yes, it's a bug – at least in documentation.
From initial investigation, the problem is that between the two phases 
of multi-phase init, module state is NULL, and Python code can run.
This is expected, so I'm thinking m_traverse for all modules using 
multi-phase init should have a check for NULL. And this should be 
documented.

Let's have Marcel run with this a bit further.



Speaking of which, the doc is not very clear: is PEP 489 required for
multi-interpreter support or is PyModule_GetState() sufficient?


I'm not exactly sure what you're asking; which doc are you referring to?

PEP 489 gives you good defaults, if you use it and avoid global state 
(roughly: C-level mutable static variables), then you should get 
multi-interpreter support for free in simple cases.
It's also possible to use PyModule_GetState() and other APIs directly. 
However, I'd like to avoid solving subinterpreter support separately 
(and slightly differently) in each module.



For a slightly bigger picture: as a part-time internship, Marcel is 
identifying where PEP 489 is inadequate, and solving the problems for 
the complex cases. This is part of better support for subinterpreter 
support in general. Going with a PEP 489-based solution with atexit 
would help us in that effort.
I'm assuming fixing the atexit bug from 2009 [0] can be delayed a bit as 
issues with PEP 489 are investigated & solved.

Does that sound fair?


[0] https://bugs.python.org/issue6531
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v4 (new interpreters module)

[Petr Viktorin]

On 12/06/2017 03:51 AM, Eric Snow wrote:

Hi all,

I've finally updated PEP 554.  Feedback would be most welcome.  The
PEP is in a pretty good place now and I hope to we're close to a
decision to accept it. :)


[...]

C-extension opt-in/opt-out
--

By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily
add a mechanism by which C-extension modules could opt out of support
for subinterpreters.  Then the import machinery, when operating in
a subinterpreter, would need to check the module for support.  It would
raise an ImportError if unsupported. >
Alternately we could support opting in to subinterpreter support.
However, that would probably exclude many more modules (unnecessarily)
than the opt-out approach.


Currently it's already opt-in, as modules that use PyModuleDef are 
expected to support subinterpreters:

https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading

[...]

.. [global-atexit]
https://bugs.python.org/issue6531


Oh dear; there's now also https://bugs.python.org/issue31901
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing files from the repository

[Petr Viktorin]

On 11/29/2017 07:26 PM, Serhiy Storchaka wrote:

[...]  Perhaps it is
worth to track all removals in a special file, so if later you will find 
that the removed file can be useful you could restore it instead of 
recreating its functionality from zero in the case if you even don't 
know that similar file existed.


All removals are tracked by Git, necessarily. It's the command to show 
them that's not obvious (unless you're Finnish):


git log --oneline --diff-filter=D --summary -- :^/Misc/NEWS.d/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Petr Viktorin]

On 11/07/2017 09:00 AM, Wolfgang wrote:
[...]

Also it is fine to teach people that dict (Mapping) is not ordered
but CPython has an implementation detail and it is ordered.
But if you want the guarantee use OrderedDict.


I don't think that is fine.
When I explained this in 3.5, dicts rearranging themselves seemed quite 
weird to the newcomers.
This year, I'm not looking forward to saying that dicts behave 
"intuitively", but you shouldn't rely on that, because they're 
theoretically allowed to rearrange themselves.
The concept of "implementation detail" and language spec vs. multiple 
interpreter implementations isn't easy to explain to someone in a "basic 
coding literacy" course.


Today I can still show an example on Python 3.5. But most people I teach 
today won't run their code on 3.5, or on MicroPython or Brython, and 
quite soon they'll forget that there's no dict ordering guarantee.


Also: I happen to read python-dev and the language docs. I suspect not 
all teachers do, and when they see that dict order randomization was 
"fixed", they might just remove the explanation from the lesson and 
teach something practical instead.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Namedtuple on startup time

[Petr Viktorin]

On 07/17/2017 10:31 PM, Giampaolo Rodola' wrote:
I completely agree. I love namedtuples but I've never been too happy 
about the additional overhead vs. plain tuples (both for creation and 
attribute access times), to the point that I explicitly avoid to use 
them in certain circumstances (e.g. a busy loop) and only for public 
end-user APIs returning multiple values.


To be entirely honest, I'm not even sure why they need to be forcefully 
declared upfront in the first place, instead of just having a 
first-class function (builtin?) written in C:


 >>> ntuple(x=1, y=0)
(x=1, y=0)

...or even a literal as in:

 >>> (x=1, y=0)
(x=1, y=0)

Most of the times this is what I really want: quickly returning an 
anonymous tuple with named attributes and nothing else, similarly to 
os.times() & others. [...]


It seems that you want `types.SimpleNamespace(x=1, y=0)`.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Snap Python for simple distribution across multiple Linux distros

[Petr Viktorin]

On 05/23/2017 01:34 AM, Brett Cannon wrote:


On Tue, 16 May 2017 at 08:08 Martin Wimpress 
> 
wrote:


Hi all,

I work at Canonical as part of the engineering team developing Ubuntu
and Snapcraft [1] and I'm a long time Python fan :-)

We've created snaps, a platform that enables projects to directly
control delivery of software updates to users. This video of a
lightning talk by dlang developers at DConf2017 [2] shows how they've
made good use of snaps to distribute their compiler. They found the
release channels particularly useful so their users can track a
specific release.

Is there someone here who'd be interested in doing the same for Python?


So the problem with adding Snap is it's yet one more thing for us to 
create at release time. And if we do this for Snap are there competitors 
from e.g. Fedora that people would want supported?


Not Fedora per se, Flatpak [0] (formerly `xdg-app`) is distro-agnostic [1].


[0] http://flatpak.org/
[1] http://flatpak.org/faq.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Petr Viktorin]

On 03/30/2017 06:31 AM, Nick Coghlan wrote:

On 29 March 2017 at 02:18, Paul Moore  wrote:

On 28 March 2017 at 12:24, Miro Hrončok  wrote:

I'd like some clarification on what ABI compatibility we can expect.
 * Should the ABI be stable across patch releases (so calling
PySlice_AdjustIndices from an existing macro would be a bug)?
 * Should the ABI be forward-compatible within a minor release (so modules
built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
 * Or should we expect the ABI to change even across patch releases?


Given that binary wheels are built against a specific minor version
(3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
release. That would fit with my expectations of the compatibility
guarantees on patch releases.

So I from what you describe, I'd consider this as a bug. Certainly, if
someone built a C extension as a wheel using Python 3.6.1, it would be
tagged as compatible with cp36, and pip would happily use it when
installing to a Python 3.6.0 system, where it would fail.


Right, this is the main problem - while "build against the X.Y.0
headers" is useful advice, it's not something we've ever explicitly
stated, and it's not something we can reasonably expect all providers
of pre-built binary modules to do.


Also, while building against 3.6.0 headers will ensure compatibility, it 
will also restore the original bug that PySlice_AdjustIndices fixes.
Expecting extension authors to build against x.y.0 would lock them out 
of such bug fixes in later releases.



Instead, it makes sense to explicitly strengthen the ABI guarantees
within CPython maintenance releases, and add some automated testing to
avoid accidental changes and oversights (similar to the pending test
to ensure magic number stability for cached bytecode files)

Cheers,
Nick.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py 3.6 on Ubuntu Zesty

[Petr Viktorin]

On 02/07/2017 11:38 PM, Barry Warsaw wrote:

On Feb 07, 2017, at 02:15 PM, Mike Miller wrote:


Does anyone know why Python 3.6 is not the default Python 3 under the
upcoming Ubuntu Zesty, or what may be holding it back?


I guess that would be me. :)



Thank for the in-depth explanation!



The Fedora/RH ecosystem probably has their own list, which I'd expect to
mostly overlap with ours, but I don't have those links handy.


Hello,

Python 3.6 is the python3 in Fedora Rawhide, on track to become Fedora 
26 in June.
These packages fail to build (which, for properly packaged software, 
includes passing upstream tests where feasible):


insight  (failure unrelated to Python)
python-django-admin-honeypot  (not ported for django 1.10)
python-os-brick  (unrelated packaging problems)
python-oslo-messaging  (blocked by python-oslo-middleware)
python-oslo-middleware  (not ported for webob 1.7+)
python-oslo-versionedobjects  (blocked by python3-oslo-messaging)
python-recommonmark  (not ported to latest commonmark)
python-repoze-who-plugins-sa  (test failures)
rb_libtorrent  (probably unrelated to Python)
shogun  (probably unrelated to Python)

That's 10 of 3349 python3-compatible packages.
The list used to be much longer and contain much more important 
packages, but got to the current state thanks to individual maintainers 
and people like Adam Williamson, Zbigniew Jędrzejewski-Szmek, and Igor 
Gnatenko (and the python-maint team, but we're paid for it).
The list was also longer than usual because there was no mass rebuild of 
packages for Fedora 25, so some build failures introduced in the last 
year are only popping up now.



Released versions of Fedora will be getting getting a "python36" package 
soon; for now it's installable using:


   dnf install --enablerepo=updates-testing python36

This is one of Fedora's "other Pythons" (interpreters other than the 
"current" 2.x and 3.x, i.e.  python26, python34, python33). It's enough 
to make virtualenv/venv and tox work, but no system packages are built 
for these.




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drastically improving list.sort() for lists of strings/ints

[Petr Viktorin]

On 09/11/2016 10:48 PM, Terry Reedy wrote:
[...]

Second, with respect to timsort in particular: timsort is designed to
exploit structure and run faster than O(n*logn) in special cases.  If a
list is already sorted, timsort will do one O(n) scan and stop.  Any
radix sort will take several times longer.  If a list is reverse sorted,
timsort will do one O(n) scan and do an O(n) reverse.  If a list is the
concatenation of two sorted lists, timsort will find the two sorted
sublists and merge them.  If a sorted list has unsorted items appended
to the end, timsort will sort the appended items and then do a merge.  I
expect any radix sort to be slower for all these cases.  Tim Peters
somewhere documented his experiments and results with various special
but plausible cases of non-randomness.


That write-up is included in Python source:
https://github.com/python/cpython/blob/master/Objects/listsort.txt

A good read if you want to know what sort of thinking, benchmarking, and 
justification should go into a new sorting algorithm.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Petr Viktorin]

On 07/06/2016 05:11 AM, Steven D'Aprano wrote:
> On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:
> 
>> In the tkinter case, compiling for source is easy on a developer's
>> computer, but doing that on a headless server brings in devel files for
>> the entire graphical environment.
>> Are you saying Python on servers should have a way to do turtle
>> graphics, otherwise it's not Python?
> 
> That's a really good question.
> 
> I don't think we have an exact answer to "What counts as Python?". It's 
> not like EMCAScript (Javascript) or C where there's a standard that 
> defines the language and standard modules. We just have some de facto 
> guidelines:
> 
> - CPython is definitely Python;
> - Jython is surely Python, even if it lacks the byte-code of CPython and 
>   some things behave slightly differently;
> - MicroPython is probably Python, because nobody expects to be able to 
>   run Tkinter GUI apps on an embedded device with 256K or RAM;
> 
> but it's hard to make that judgement except on a case-by-case basis.
> 
> I think though that even if there's no documented line, most people 
> recognise that there are "core" and "non-core" standard modules. dis and 
> tkinter are non-core: if µPython leaves out tkinter, nobody will be 
> surprised; if Jython leaves out dis, nobody will hold it against them; 
> but if they leave out math or htmllib that's another story.

For MicroPython, I would definitely expect htmllib to be an optional
add-on – it's not useful for reading data off a thermometer saving it to
an SD card. But I guess that's getting too deep into specifics.

> So a headless server can probably leave out tkinter; but a desktop 
> shouldn't.

Up till recently this wasn't possible to express in terms of RPM
dependencies. Now, it's on the ever-growing TODO list...

Another problem here is that you don't explicitly "install Python" on
Fedora: when you install the system, you get a minimal set of packages
to make everything work, and most of Python is part of that – but
tkinter is not. This is in contrast to python.org releases, where you
explicitly ask for (all of) Python.
Technically it would now be possible to have to install Python to use
it, but we run into another "batteries included" problem: Python (or,
"most-of-Python") is a pretty good battery for an OS.

Maybe a good short-term solution would be to make "import tkinter" raise
ImportError("Run `dnf install tkinter` to install the tkinter module")
if not found. This would prevent confusion while keeping the status quo.
I'll look into that.


> [...]
>>> The other extreme is Javascript/Node.js, where the "just use pip" (or 
>>> npm in this case) philosophy has been taken to such extremes that one 
>>> developer practically brought down the entire Node.js ecosystem by 
>>> withdrawing an eleven line module, left-pad, in a fit of pique.
>>>
>>> Being open source, the damage was routed around quite quickly, but 
>>> still, I think it's a good cautionary example of how a technological 
>>> advance can transform a programming culture to the worse.
>>
>> I don't understand the analogy. Should the eleven-line module have been
>> in Node's stdlib? Outside of stdlib, people are doing this.
> 
> The point is that Javascript/Node.js is so lacking in batteries that the 
> community culture has gravitated to an extreme version of "just use 
> pip". I'm not suggesting that you, or anyone else, has proposed that 
> Python do the same, only that there's a balance to be found between the 
> extremes of "everything in the Python ecosystem should be part of the 
> standard installation" and "next to nothing should be part of the 
> standard installation".
> 
> The hard part is deciding where that balance should be :-)

I think the balance is where it needs to be for CPython, and it's also
where it needs to be for Fedora. The real hard part is acknowledging
that it needs to be in different places for different use cases, and
making sure work to support the different use cases is coordinated.

So, I guess I'm starting to form a concrete proposal:

1) Document what should happen when a stdlib module is not available.
This should be an ImportError informative error message, something along
the lines of 'This build of Python does not include SQLite support.' or
'MicroPython does not support turtle' or 'Use `sudo
your-package-manager` install tkinter` to install this module'.

2) Document leaf modules (or "clusters") that can be removed from the
stdlib, and their dependencies. Make no guarantees about cross-version
compatibility of this metadata.

3) Standardize a way to query which stdlib modules a