Re: [Python-Dev] Please reconsider PEP 479.

[Petr Viktorin]

On Wed, Nov 26, 2014 at 12:24 PM, Nick Coghlan  wrote:
> On 26 November 2014 at 18:30, Greg Ewing  wrote:
>> Guido van Rossum wrote:
>>>
>>> Hm, that sounds like you're either being contrarian or Chris and I have
>>> explained it even worse than I thought.
>>
>> I'm not trying to be contrary, I just think the PEP could
>> explain more clearly what you're trying to achieve. The
>> rationale is too vague and waffly at the moment.
>>
>>> Currently, there are cases where list(x for x in xs if P(x)) works while
>>> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the
>>> PEP, both cases will raise some exception
>>
>> That's a better explanation, I think.
>
> The other key aspect is that it changes the answer to the question
> "How do I gracefully terminate a generator function?". The existing
> behaviour has an "or" in the answer: "return from the generator frame,
> OR raise StopIteration from the generator frame". That then leads to
> the follow on question: "When should I use one over the other?".
>
> The "from __future__ import generator_stop" answer drops the "or", so
> it's just: "return from the generator frame".
>
> Raising *any* exception inside the generator, including StopIteration,
> then counts as non-graceful termination, bringing generators into line
> with the PEP 343 philosophy that "hiding flow control in macros makes
> your code inscrutable", where here, the hidden flow control is relying
> on the fact that a called function raising StopIteration will
> currently always gracefully terminate generator execution.
>
> The key downside is that it means relatively idiomatic code like:
>
> def my_generator():
> ...
> yield next(it)
> ...
>
> Now needs to be written out explicitly as:
>
> def my_generator():
> ...
>try:
> yield next(it)
> except StopIteration
> return
> ...

It could also be written as:

 def my_generator():
try:
...
yield next(it)
...
except StopIteration
 return

i.e. put the try-block around the whole body, not just the individual
yield. This emulates what's happenning in current Python, and it would
be faster than individual try blocks.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] advice needed: best approach to enabling "metamodules"?

[Petr Viktorin]

On Sat, Nov 29, 2014 at 8:37 PM, Nathaniel Smith  wrote:
[...]
> The python-ideas thread did also consider several methods of
> implementing strategy (c), but they're messy enough that I left them
> out here. The problem is that somehow we have to execute code to
> create the new subtype *before* we have an entry in sys.modules for
> the package that contains the code for the subtype. So one option
> would be to add a new rule, that if a file pkgname/__new__.py exists,
> then this is executed first and is required to set up
> sys.modules["pkgname"] before we exec pkgname/__init__.py. So
> pkgname/__new__.py might look like:
>
> import sys
> from pkgname._metamodule import MyModuleSubtype
> sys.modules[__name__] = MyModuleSubtype(__name__, docstring)

> This runs into a lot of problems though. To start with, the 'from
> pkgname._metamodule ...' line is an infinite loop, b/c this is the
> code used to create sys.modules["pkgname"].

As Greg Ewing said – you don't want to import from the package whose
metamodule you're defining. You'd want to do as little work as
possible in __new__.py.

I'd use something like this:

import types

class __metamodule__(types.ModuleType):
def __call__(self):
return self.main()

where Python would get the attribute __metamodule__ from __new__.py,
and use `__metamodule__(name, doc)` as the thing to execute __main__
in.

> It's not clear where the
> globals dict for executing __new__.py comes from (who defines
> __name__? Currently that's done by ModuleType.__init__).

Well, it could still be in __metamodule__.__init__().

> It only works for packages, not modules.

I don't see a need for this treatment for modules in a package – if
you want `from mypkg import callme`, you can make "callme" a function
rather than a callable module. If you *also* want `from mypkg.callme
import something_else`, I say you should split "callme" into two
differently named things; names are cheap inside a package.
If really needed, modules in a package can use an import hook defined
in the package, or be converted to subpackages.
Single-module projects would be left out, yes – but those can be
simply converted to a package.

> The need to provide the docstring here,
> before __init__.py is even read, is weird.

Does it have to be before __init__.py is read? Can't __init__.py be
compiled beforehand, to get __doc__, and only *run* in the new
namespace?
(Or should __new__.py define import hooks that say how __init__.py
should be loaded/compiled? I don't see a case for that.)

> It adds extra stat() calls to every package lookup.

Fair.

> And, the biggest showstopper IMHO: AFAICT
> it's impossible to write a polyfill to support this code on old python
> versions, so it's useless to any package which needs to keep
> compatibility with 2.7 (or even 3.4). Sure, you can backport the whole
> import system like importlib2, but telling everyone that they need to
> replace every 'import numpy' with 'import importlib2; import numpy' is
> a total non-starter.

I'm probably missing something obvious, but where would this not work?
- As the first thing it does, __init__.py imports the polyfill and
calls polyfill(__name__)
- The polyfill, if running non-recursively* under old Python:
-- compiles __init__.py
-- imports __new__.py to get __metamodule__
-- instantiates metamodule with name, and docstring from compiled code
-- * remembers the instance, to check for recursion later
-- puts it in sys.modules
-- execs __init__ in it
- afterwards the original __init__.py execution continues, filling up
a now-unused module's namespace
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] advice needed: best approach to enabling "metamodules"?

[Petr Viktorin]

On Sun, Nov 30, 2014 at 12:05 AM, Steven D'Aprano  wrote:
> On Sun, Nov 30, 2014 at 11:07:57AM +1300, Greg Ewing wrote:
>> Nathaniel Smith wrote:
>> >So pkgname/__new__.py might look like:
>> >
>> >import sys
>> >from pkgname._metamodule import MyModuleSubtype
>> >sys.modules[__name__] = MyModuleSubtype(__name__, docstring)
>> >
>> >To start with, the 'from
>> >pkgname._metamodule ...' line is an infinite loop,
>>
>> Why does MyModuleSubtype have to be imported from pkgname?
>> It would make more sense for it to be defined directly in
>> __new__.py, wouldn't it? Isn't the purpose of separating
>> stuff out into __new__.py precisely to avoid circularities
>> like that?
>
> Perhaps I'm missing something, but won't that imply that every module
> which wants to use a "special" module type has to re-invent the wheel?
>
> If this feature is going to be used, I would expect to be able to re-use
> pre-written module types. E.g. having written "module with properties"
> (so to speak) once, I can just import it and use it in my next project.

I expect you'd package the special metamodule class in a stand-alone
package, not directly in the ones that use it.
You could import other packages freely, just the one that you're
currently defining would be unavailable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[Petr Viktorin]

Also keep in mind that not all Python libraries are on PyPI.
For non-Python projects with Python bindings (think video players,
OpenCV, systemd, Samba), distribution via PyPI doesn't make much
sense. And since the Python bindings are usually second-class
citizens, the porting doesn't have a high priority.

If anyone is wondering why their favorite Linux distribution is stuck
with Python 2 – well, I can only speak for Fedora, but nowadays most
of what's left are CPython bindings.
No pylint --py3k or 2to3 will help there...


On Fri, Dec 12, 2014 at 7:24 PM, Mark Roberts  wrote:
> So, I'm more than aware of how to write Python 2/3 compatible code. I've
> ported 10-20 libraries to Python 3 and write Python 2/3 compatible code at
> work. I'm also aware of how much writing 2/3 compatible code makes me hate
> Python as a language. It'll be a happy day when one of the two languages
> dies so that I never have to write code like that again. However, my point
> was that just because the core libraries by usage are *starting* to roll out
> Python 3 support doesn't mean that things are "easy" or "convenient" yet.
> There are too many libraries in the long tail which fulfill semi-common
> purposes and haven't been moved over yet. Yeah, sure, they haven't been
> updated in years... but neither has the language they're built on.
>
> I suppose what I'm saying is that the long tail of libraries is far more
> valuable than it seems the Python3 zealots are giving it credit for. Please
> don't claim it's "easy" to move over just because merely most of the top 20
> libraries have been moved over. :-/
>
> -Mark
>
> On Thu, Dec 11, 2014 at 12:14 PM, Dan Stromberg  wrote:
>>
>> On Thu, Dec 11, 2014 at 11:35 AM, Mark Roberts  wrote:
>> > I disagree. I know there's a huge focus on The Big Libraries (and
>> > wholesale
>> > migration is all but impossible without them), but the long tail of
>> > libraries is still incredibly important. It's like saying that migrating
>> > the
>> > top 10 Perl libraries to Perl 6 would allow people to completely ignore
>> > all
>> > of CPAN. It just doesn't make sense.
>>
>> Things in the Python 2.x vs 3.x world aren't that bad.
>>
>> See:
>> https://python3wos.appspot.com/ and
>> https://wiki.python.org/moin/PortingPythonToPy3k
>> http://stromberg.dnsalias.org/~strombrg/Intro-to-Python/ (writing code
>> to run on 2.x and 3.x)
>>
>> I believe just about everything I've written over the last few years
>> either ran on 2.x and 3.x unmodified, or ran on 3.x alone.  If you go
>> the former route, you don't need to wait for your libraries to be
>> updated.
>>
>> I usually run pylint twice for my projects (after each change, prior
>> to checkin), once with a 2.x interpreter, and once with a 3.x
>> interpreter (using
>> http://stromberg.dnsalias.org/svn/this-pylint/trunk/this-pylint) , but
>> I gather pylint has the option of running on a 2.x interpreter and
>> warning about anything that wouldn't work on 3.x.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[Petr Viktorin]

On Sun, Dec 14, 2014 at 1:14 AM, Nick Coghlan  wrote:
[...]
> Barry, Petr, any of the other folks working on distro level C extension
> ports, perhaps one of you would be willing to consider an update to the C
> extension porting guide to be more in line with Brett's latest version of
> the Python level porting guide?

I can make it a 20%-time project starting in January, if no-one beats me to it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Disassembly of generated comprehensions

[Petr Viktorin]

On Sun, Jan 25, 2015 at 12:55 PM, Neil Girdhar  wrote:
> How do I disassemble a generated comprehension?
>
> For example, I am trying to debug the following:
>
 dis.dis('{**{} for x in [{1:2}]}')
>   1   0 LOAD_CONST   0 ( at
> 0x10160b7c0, file "", line 1>)
>   3 LOAD_CONST   1 ('')
>   6 MAKE_FUNCTION0
>   9 LOAD_CONST   2 (2)
>  12 LOAD_CONST   3 (1)
>  15 BUILD_MAP1
>  18 BUILD_LIST   1
>  21 GET_ITER
>  22 CALL_FUNCTION1 (1 positional, 0 keyword pair)
>  25 RETURN_VALUE
>
> (This requires the new patch in issue 2292.)
>
> The code here looks fine to me, so I need to look into the code object
> .  How do I do that?

Put it in a function, then get it from the function's code's constants.
I don't have the patch applied but it should work like this even for
the new syntax:

>>> import dis
>>> def f(): return {{} for x in [{1:2}]}
...
>>> dis.dis(f)
  1   0 LOAD_CONST   1 ( at
0x7ff2c0647420, file "", line 1>)
  3 LOAD_CONST   2 ('f..')
  6 MAKE_FUNCTION0
  9 BUILD_MAP1
 12 LOAD_CONST   3 (2)
 15 LOAD_CONST   4 (1)
 18 STORE_MAP
 19 BUILD_LIST   1
 22 GET_ITER
 23 CALL_FUNCTION1 (1 positional, 0 keyword pair)
 26 RETURN_VALUE
>>> f.__code__.co_consts[1]  # from "LOAD_CONST 1"
 at 0x7ff2c0647420, file "", line 1>
>>> dis.dis(f.__code__.co_consts[1])
  1   0 BUILD_SET0
  3 LOAD_FAST0 (.0)
>>6 FOR_ITER12 (to 21)
  9 STORE_FAST   1 (x)
 12 BUILD_MAP0
 15 SET_ADD  2
 18 JUMP_ABSOLUTE6
>>   21 RETURN_VALUE
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[Petr Viktorin]

On Mon, Jan 26, 2015 at 8:29 PM, Ethan Furman  wrote:
> On 01/26/2015 11:24 AM, Barry Warsaw wrote:
>> On Jan 26, 2015, at 10:55 AM, Ethan Furman wrote:
>>
>>> In the your example
>>>
>>>  from_env = {'a': 12}
>>>  from_config = {'a': 13}
>>>
>>>  f(**from_env, **from_config)
>>>
>>> I would think 'a' should be 13, as from_config is processed /after/ 
>>> from_env.
>>>
>>> So which is it?
>>
>> In the face of ambiguity, refuse the temptation to guess.
>
> Lots of things are ambiguous until one learns the rules.  ;)

I don't see why `f(**{'a': 12}, **{'a': 13})` should not be equivalent
to `f(a=12, **{'a':13})` – iow, raise TypeError.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Petr Viktorin]

On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore  wrote:
> So the usage would be something like
>
> python -m zipapp [options] dir_to_zip
>
> Options:
> -p The interpreter to use on the shebang line
> (defaulting to /usr/bin/env python)

On many systems this default would mean Python 2. Even if the official
recommendation changes for 3.5, the status quo might linger for a few
years.
On the other hand, the number of distros that don't ship Python 3 is
small, and the reason they're slow-moving tends to be stability and/or
compliance, so they're not the target audience for zipapp. And the
python3 symlink is not going away any time soon.
So I'd suggest `/usr/bin/env python3` for the default.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Petr Viktorin]

On Sun, Feb 15, 2015 at 5:43 PM, Paul Moore  wrote:
> On 15 February 2015 at 16:15, Petr Viktorin  wrote:
>> On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore  wrote:
>>> So the usage would be something like
>>>
>>> python -m zipapp [options] dir_to_zip
>>>
>>> Options:
>>> -p The interpreter to use on the shebang line
>>> (defaulting to /usr/bin/env python)
>>
>> On many systems this default would mean Python 2. Even if the official
>> recommendation changes for 3.5, the status quo might linger for a few
>> years.
>> On the other hand, the number of distros that don't ship Python 3 is
>> small, and the reason they're slow-moving tends to be stability and/or
>> compliance, so they're not the target audience for zipapp. And the
>> python3 symlink is not going away any time soon.
>> So I'd suggest `/usr/bin/env python3` for the default.
>
> But that will fail for users who only have Python 2 installed. And it
> won't pick up an activated virtualenv (assuming virtualenvs on Unix
> don't include a python3 command, which I believe is true - it
> certainly won't on Windows).

Yes, it would fail with only Python 2 installed. I don't think that's a problem.

On POSIXy systems the "python3" symlink is created in all venvs. I
thought (perhaps naïvely) that Windows doesn't do shebangs natively,
so there's some Python-specific mechanism around them, which should
handle "python3".

If the app was single-source compatible with both 2 and 3, then
`/usr/bin/env python` would be a good choice. But I don't think apps
(as opposed to libraries) have much incentive to keep the backwards
compatibility.

> As a Windows user, I believe that the command "python" should run the
> version of Python that you choose to be your default. I know it's not
> quite that simple on Unix, and the system "python" command is
> typically Python 2, with "python3" for the system Python 3 version,
> but I also know that tools like pyenv work like that. Equally, I've
> seen lots of scripts with "#!/usr/bin/env python" as the shebang line,
> and none with "#!/usr/bin/env python3". That may just be my limited
> experience with Unix though.

On Linux, what version "python" is depends on the distro. Currently
the recommendation from Python upstream is for it to be Python 2,
though, and I know of just one distro that does not do this (Arch,
because it switched before the recommendation was put in place).

If you're in a venv, then of "python" is that venv's Python, of
course. But the default mode of operation for Python-unaware users is
to not be in a venv.

> I don't really want "#!/usr/bin/env python3" as the default shebang on
> Windows (at a minimum, it would do the wrong thing for my current
> environment). I'm open to advice from the Unix users as to how to set
> things up better on Unix, but there's a whole new set of problems that
> will arise as soon as we have different defaults on Unix and Windows,
> so IMO there would need to be pretty significant benefits to justify
> doing that.

I'm not familiar with how the shebang works on Windows, but if
"/usr/bin/env python3" breaks things, I find it highly unfortunate.

> And of course it *is* only a default - users can set whatever they
> want. (But getting defaults right is important, so that's not to
> dismiss the point).

I'm afraid the point is whether it's even possible to have a shebang
that does the right thing on both platforms.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Petr Viktorin]

On Mon, Feb 16, 2015 at 12:00 AM, Chris Angelico  wrote:
> On Sat, Feb 14, 2015 at 11:07 AM, Nick Coghlan  wrote:
>> OTOH, it may be time to reconsider our recommendation to distros as well,
>> suggesting that for Python 3.5+, we will consider it appropriate to have the
>> "python" symlink refer to "python3".
>
> If *all* distros provide a "python2" symlink, then the recommendation
> can become "use python if it's 2/3 compatible, or python2/python3 to
> choose", and then it won't hurt to switch python to be python3.
> Are there known distros in which current versions (or, those that will be
> current when 3.5 is released) aren't providing "python2"?

That wouldn't be a good recommendation for the long term.
Fedora probably won't drop python2 from the default installation
before 3.5 is released, but hopes are high that it'll happen when 3.5
is still current. Not sure about others.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Petr Viktorin]

On Wed, Feb 18, 2015 at 10:16 PM, Paul Moore  wrote:
> On 18 February 2015 at 20:48, Jim J. Jewett  wrote:
>> Barry Warsaw wrote:
 I don't know exactly what the procedure would be to claim .pyz for *nix,
 e.g. updating /etc/mime.types, but I think the PEP should at least mention
 this.  I think we want to get as official support for .pyz files on *nix as
 possible.
>>
>> Paul Moore wrote:
>>> I'll add a note to the PEP, but I have no idea how we would even go
>>> about that, so that's all I can do, unfortunately.
>>
>> Are you just looking for
>>
>> http://www.iana.org/assignments/media-types/media-types.xhtml
>>
>> and its references, including the registration procedures
>>
>> http://tools.ietf.org/html/rfc6838#section-4.2.5
>>
>> and the application form at
>>
>> http://www.iana.org/form/media-types
>>
>> ?
>
> That covers mime types, but not file extensions, so it's not really
> what *I* thought Barry was talking about.
>
> Actually, I guess the first question is whether ".py" is reserved for
> Python scripts. If it is, then certainly /pyz should be reserved in a
> similar way for Python Zip Applications. If there's no formal
> registration of .py, then it seems a bit unreasonable to bother for
> .pyz...
>
> Also, I don't think reserving anything is something I, as an
> individual (and specifically a non-Unix user) should do. It probably
> should be handled by the PSF, as the process seems to need a contact
> email address...

There is no extension registry for Unixy systems, because extensions
traditionally don't matter. File types are identified by their
contents; there is a "magic"* database that maps file contents to
types (MIME types, nowadays). The `file` command is the CLI interface
for that. Some programs will fall back on the extension if that
doesn't give a result, but that would depend on the program.

For a pyz file to run when you "open" it (for most values of "open"),
it needs to have the executable bit set, and have a shebang line.

* /usr/share/misc/magic – it originally contained just magic numbers,
such as those you might find at the beginning of a .pyc file.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 489: Redesigning extension module loading

[Petr Viktorin]

Hello,
On import-sig, I've agreed to continue Nick Coghlan's work on making 
extension modules act more like Python ones, work well with PEP 451 
(ModuleSpec), and encourage proper subinterpreter and reloading support. 
Here is the resulting PEP.


I don't have a patch yet, but I'm working on it.

There's a remaining open issue: providing a tool that can be run in test 
suites to check if a module behaves well with subinterpreters/reloading. 
I believe it's out of scope for this PEP but speak out if you disagree.


Please discuss on import-sig.

===

PEP: 489
Title: Redesigning extension module loading
Version: $Revision$
Last-Modified: $Date$
Author: Petr Viktorin ,
Stefan Behnel ,
Nick Coghlan 
Discussions-To: import-...@python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Aug-2013
Python-Version: 3.5
Post-History: 23-Aug-2013, 20-Feb-2015
Resolution:


Abstract


This PEP proposes a redesign of the way in which extension modules interact
with the import machinery. This was last revised for Python 3.0 in PEP
3121, but did not solve all problems at the time. The goal is to solve them
by bringing extension modules closer to the way Python modules behave;
specifically to hook into the ModuleSpec-based loading mechanism
introduced in PEP 451.

Extensions that do not require custom memory layout for their module objects
may be executed in arbitrary pre-defined namespaces, paving the way for
extension modules being runnable with Python's ``-m`` switch.
Other extensions can use custom types for their module implementation.
Module types are no longer restricted to types.ModuleType.

This proposal makes it easy to support properties at the module
level and to safely store arbitrary global state in the module that is
covered by normal garbage collection and supports reloading and
sub-interpreters.
Extension authors are encouraged to take these issues into account
when using the new API.



Motivation
==

Python modules and extension modules are not being set up in the same way.
For Python modules, the module is created and set up first, then the module
code is being executed (PEP 302).
A ModuleSpec object (PEP 451) is used to hold information about the module,
and passed to the relevant hooks.
For extensions, i.e. shared libraries, the module
init function is executed straight away and does both the creation and
initialisation. The initialisation function is not passed ModuleSpec
information about the loaded module, such as the __file__ or fully-qualified
name. This hinders relative imports and resource loading.

This is specifically a problem for Cython generated modules, for which it's
not uncommon that the module init code has the same level of complexity as
that of any 'regular' Python module. Also, the lack of __file__ and __name__
information hinders the compilation of __init__.py modules, i.e. packages,
especially when relative imports are being used at module init time.

The other disadvantage of the discrepancy is that existing Python 
programmers

learning C cannot effectively map concepts between the two domains.
As long as extension modules are fundamentally different from pure 
Python ones
in the way they're initialised, they are harder for people to pick up 
without
relying on something like cffi, SWIG or Cython to handle the actual 
extension

module creation.

Currently, extension modules are also not added to sys.modules until 
they are

fully initialized, which means that a (potentially transitive)
re-import of the module will really try to reimport it and thus run into an
infinite loop when it executes the module init function again.
Without the fully qualified module name, it is not trivial to correctly add
the module to sys.modules either.

Furthermore, the majority of currently existing extension modules has
problems with sub-interpreter support and/or reloading, and, while it is
possible with the current infrastructure to support these
features, it is neither easy nor efficient.
Addressing these issues was the goal of PEP 3121, but many extensions,
including some in the standard library, took the least-effort approach
to porting to Python 3, leaving these issues unresolved.
This PEP keeps the backwards-compatible behavior, which should reduce 
pressure
and give extension authors adequate time to consider these issues when 
porting.



The current process
===

Currently, extension modules export an initialisation function named
"PyInit_modulename", named after the file name of the shared library. This
function is executed by the import machinery and must return either NULL in
the case of an exception, or a fully initialised module object. The
function receives no arguments, so it has no way of knowing about its
import context.

During its execution, the module init function creates a module object
based on a PyModuleDef struc

Re: [Python-Dev] PEP 489: Redesigning extension module loading

[Petr Viktorin]

On Mon, Mar 16, 2015 at 4:42 PM, Jim J. Jewett  wrote:
>
> On 16 March 2015 Petr Viktorin wrote:
>
>> If PyModuleCreate is not defined, PyModuleExec is expected to operate
>> on any Python object for which attributes can be added by PyObject_GetAttr*
>> and retrieved by PyObject_SetAttr*.
>
> I assume it is the other way around (add with Set and retrieve with Get),
> rather than a description of the required form of magic.

Right you are, I mixed that up.

>> PyObject *PyModule_AddCapsule(
>> PyObject *module,
>> const char *module_name,
>> const char *attribute_name,
>> void *pointer,
>> PyCapsule_Destructor destructor)
>
> What happens if module_name doesn't match the module's __name__?
> Does it become a hidden attribute?  A dotted attribute?  Is the
> result undefined?

The module_name is used to name the capsule, following the convention
from PyCapsule_Import. The "module.__name__" is not used or checked.
The function would do this:
capsule_name = module_name + '.' + attribute_name
capsule = PyCapsule_New(pointer, capsule_name, destructor)
PyModule_AddObject(module, attribute_name, capsule)
just with error handling, and suitable C code for the "+".
I will add the pseudocode to the PEP.

> Later, there is
>
>> void *PyModule_GetCapsulePointer(
>> PyObject *module,
>> const char *module_name,
>> const char *attribute_name)
>
> with the same apparently redundant arguments,

Here the behavior would be:
capsule_name = module_name + '.' + attribute_name
capsule = PyObject_GetAttr(module, attribute_name)
return PyCapsule_GetPointer(capsule, capsule_name)

> but not a
> PyModule_SetCapsulePointer.  Are capsule pointers read-only, or can
> they be replaced with another call to PyModule_AddCapsule, or by a
> simple PyObject_SetAttr?

You can replace the capsule using any of those two, or set the pointer
using PyCapsule_SetPointer, or (most likely) change the data the
pointer points to.
The added functions are just simple helpers for common operations,
meant to encourage keeping per-module state.

>> Subinterpreters and Interpreter Reloading
> ...
>> No user-defined functions, methods, or instances may leak to different
>> interpreters.
>
> By "user-defined" do you mean "defined in python, as opposed to in
> the extension itself"?

Yes.

> If so, what is the recommendation for modules that do want to support,
> say, callbacks?  A dual-layer mapping that uses the interpreter as the
> first key?  Naming it _module and only using it indirectly through
> module.py, which is not shared across interpreters?  Not using this
> API at all?

There is a separate module object, with its own dict, for each
subinterpreter (as when creating the module with "PyModuleDef.m_size
== 0" today).
Callbacks should be stored on the appropriate module instance.
Does that answer your question? I'm not sure how you meant "callbacks".

>> To achieve this, all module-level state should be kept in either the module
>> dict, or in the module object.
>
> I don't see how that is related to leakage.
>
>> A simple rule of thumb is: Do not define any static data, except
>> built-in types
>> with no mutable or user-settable class attributes.
>
> What about singleton instances?  Should they be per-interpreter?

Yes, definitely.

> What about constants, such as PI?

In PyModuleExec, create the constant using PyFloat_FromDouble, and add
it using PyModule_FromObject. That will do the right thing.
(Float constants can be shared, since they cannot refer to
user-defined code. But this PEP shields you from needing to know this
for every type.)

> Where should configuration variables (e.g., MAX_SEARCH_DEPTH) be
> kept?

On the module object.

> What happens if this no-leakage rule is violated?  Does the module
> not load, or does it just maybe lead to a crash down the road?

It may, as today, lead to unexpected behavior down the road. This is
explained here:
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
Unfortunately, there's no good way to detect such leakage. This PEP
adds the tools, documentation, and guidelines to make it easy to do
the right thing, but won't prevent you from shooting yourself in the
foot in C code.


Thank you for sharing your concerns! I will keep them in mind when
writing the docs for this.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Changing PyModuleDef.m_reload to m_slots

[Petr Viktorin]

Hello,

PEP 489, Redesigning extension module loading [0], is currently 
discussed on import-sig, but one question calls for a wider audience.


As a background, the PyModuleDef structure [1] is currently:

struct PyModuleDef{
  PyModuleDef_Base m_base;
  const char* m_name;
  const char* m_doc;
  Py_ssize_t m_size;
  PyMethodDef *m_methods;
  inquiry m_reload;
  traverseproc m_traverse;
  inquiry m_clear;
  freefunc m_free;
};

... where the m_reload pointer is unused, and must be NULL.
My proposal is to repurpose this pointer to hold an array of slots, in 
the style of PEP 384's PyType_Spec [2], which would allow adding 
extensions -- both those needed for PEP 489 and future ones.


The result would be:

typedef struct {
int slot;
void *value;
} PyModuleDesc_Slot;

typedef struct PyModuleDef {
PyModuleDef_Base m_base;
const char* m_name;
const char* m_doc;
Py_ssize_t m_size;
PyMethodDef *m_methods;
PyModuleDesc_Slot* m_slots;  /* <-- change here */
traverseproc m_traverse;
inquiry m_clear;
freefunc m_free;
} PyModuleDef;

The alternative is defining another struct for module definitions, with 
a parallel versions of functions to operate on it, and duplication on 
the lower level (e.g. a variant of PyModule_GetDef, variants for 
PyState_FindModule &c with the supporting registry, extra storage in 
module objects, two places to look in for GC/deallocation hooks).


The downside is that, strictly speaking, this would break API/ABI, 
though in pedantic details.


The field name could conceivably be used to get the NULL or in C99-style 
designated initializers, but since NULL is the only valid value, I don't 
see a reason for that. It could also be used for zeroing the structure 
after allocating a PyModuleDef dynamically.


ABI-wise, this changes a function pointer to a data pointer. To my 
knowledge, Python doesn't support a platform where this would matter, 
but I admit my knowledge is not complete.


Please share any concerns you might have about this change.


[0] https://www.python.org/dev/peps/pep-0489/
[1] https://docs.python.org/3/c-api/module.html#c.PyModuleDef
[2] https://www.python.org/dev/peps/pep-0384/#type-objects
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] A macro for easier rich comparisons

[Petr Viktorin]

It seems the discussion on python-ideas, and also the patch review,
died down. So I'm posting to python-dev.

A macro like this would reduce boilerplate in stdlib and third-party C
extensions. It would ease porting C extensions to Python 3, where rich
comparison is mandatory.

#define Py_RETURN_RICHCOMPARE(val1, val2, op)   \
do {\
switch (op) {   \
case Py_EQ: if ((val1) == (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
case Py_NE: if ((val1) != (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
case Py_LT: if ((val1) < (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;   \
case Py_GT: if ((val1) > (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;   \
case Py_LE: if ((val1) <= (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
case Py_GE: if ((val1) >= (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
}   \
Py_RETURN_NOTIMPLEMENTED;   \
} while (0)

Is any of the core devs interested in this macro? Anything I can do to
help get it in?

http://bugs.python.org/issue23699
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A macro for easier rich comparisons

[Petr Viktorin]

On Tue, Apr 28, 2015 at 11:13 AM, Victor Stinner
 wrote:
> Hi,
>
> 2015-04-27 16:02 GMT+02:00 Petr Viktorin :
>> A macro like this would reduce boilerplate in stdlib and third-party C
>> extensions. It would ease porting C extensions to Python 3, where rich
>> comparison is mandatory.
>
> It would be nice to have a six module for C extensions. I'm quite sure
> that many projects are already full of #ifdef PYTHON3 ... #else ...
> #endif macros.

The idea actually came from my work on such a library:
http://py3c.readthedocs.org/en/latest/

>> #define Py_RETURN_RICHCOMPARE(val1, val2, op)   \
>> do {\
>> switch (op) {   \
>> case Py_EQ: if ((val1) == (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
>> case Py_NE: if ((val1) != (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
>> case Py_LT: if ((val1) < (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;   \
>> case Py_GT: if ((val1) > (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;   \
>> case Py_LE: if ((val1) <= (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
>> case Py_GE: if ((val1) >= (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
>> }   \
>> Py_RETURN_NOTIMPLEMENTED;   \
>> } while (0)
>
> I would prefer a function for that:
>
> PyObject *Py_RichCompare(long val1, long2, int op);

The original version of the macro used ternary statements. This was
shot down because a chain of comparisons would be slower than a case
statement. (See discussion on the issue.) Wouldn't a function call
also be slower?
Also, a function with long arguments won't work on unsigned long or long long.

> You should also handle invalid operator. PyUnicode_RichCompare() calls
> PyErr_BadArgument() in this case.

There are many different precedents, from ignoring this case to doing
an assert. Is PyErr_BadArgument() better than returning
NotImplemented?

> Anyway, please open an issue for this idea.

http://bugs.python.org/issue23699
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A macro for easier rich comparisons

[Petr Viktorin]

On Tue, Apr 28, 2015 at 4:59 PM, Barry Warsaw  wrote:
> On Apr 28, 2015, at 11:13 AM, Victor Stinner wrote:
>
>>It would be nice to have a six module for C extensions. I'm quite sure
>>that many projects are already full of #ifdef PYTHON3 ... #else ...
>>#endif macros.
>
> Maybe encapsulating some of the recommendations here:
>
> https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef#Python_extension_modules

py3c (or its documentation) now has all that except REPRV (with an
alias for the native string type, e.g. "PyStr", you can use that in
all reprs, so REPRV strikes me as somewhat redundant).

> (We really need to collect all this information in on place.)
>
>>> #define Py_RETURN_RICHCOMPARE(val1, val2, op)
>
> I think this macro would make a nice addition to the C API.  It might read
> better as `Py_RETURN_RICHCOMPARE(val1, op, val2)`.

(val1, val2, op) mirrors richcmp and PyObject_RichCompareBool; I think
a different order of arguments would just be confusing.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Petr Viktorin]

On 11/07/2017 09:00 AM, Wolfgang wrote:
[...]

Also it is fine to teach people that dict (Mapping) is not ordered
but CPython has an implementation detail and it is ordered.
But if you want the guarantee use OrderedDict.


I don't think that is fine.
When I explained this in 3.5, dicts rearranging themselves seemed quite 
weird to the newcomers.
This year, I'm not looking forward to saying that dicts behave 
"intuitively", but you shouldn't rely on that, because they're 
theoretically allowed to rearrange themselves.
The concept of "implementation detail" and language spec vs. multiple 
interpreter implementations isn't easy to explain to someone in a "basic 
coding literacy" course.


Today I can still show an example on Python 3.5. But most people I teach 
today won't run their code on 3.5, or on MicroPython or Brython, and 
quite soon they'll forget that there's no dict ordering guarantee.


Also: I happen to read python-dev and the language docs. I suspect not 
all teachers do, and when they see that dict order randomization was 
"fixed", they might just remove the explanation from the lesson and 
teach something practical instead.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing files from the repository

[Petr Viktorin]

On 11/29/2017 07:26 PM, Serhiy Storchaka wrote:

[...]  Perhaps it is
worth to track all removals in a special file, so if later you will find 
that the removed file can be useful you could restore it instead of 
recreating its functionality from zero in the case if you even don't 
know that similar file existed.


All removals are tracked by Git, necessarily. It's the command to show 
them that's not obvious (unless you're Finnish):


git log --oneline --diff-filter=D --summary -- :^/Misc/NEWS.d/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v4 (new interpreters module)

[Petr Viktorin]

On 12/06/2017 03:51 AM, Eric Snow wrote:

Hi all,

I've finally updated PEP 554.  Feedback would be most welcome.  The
PEP is in a pretty good place now and I hope to we're close to a
decision to accept it. :)


[...]

C-extension opt-in/opt-out
--

By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily
add a mechanism by which C-extension modules could opt out of support
for subinterpreters.  Then the import machinery, when operating in
a subinterpreter, would need to check the module for support.  It would
raise an ImportError if unsupported. >
Alternately we could support opting in to subinterpreter support.
However, that would probably exclude many more modules (unnecessarily)
than the opt-out approach.


Currently it's already opt-in, as modules that use PyModuleDef are 
expected to support subinterpreters:

https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading

[...]

.. [global-atexit]
https://bugs.python.org/issue6531


Oh dear; there's now also https://bugs.python.org/issue31901
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state

[Petr Viktorin]

On 12/14/2017 12:00 PM, Antoine Pitrou wrote:

On Thu, 14 Dec 2017 17:00:10 +1000
Nick Coghlan  wrote:

On 14 Dec. 2017 9:19 am, "Antoine Pitrou"  wrote:


Hello,

After debugging a crash on AppVeyor for a submitter's PR
(see https://github.com/python/cpython/pull/4611 ), I came to the
following diagnosis: converting the "atexit" module (which is a
built-in C extension) to PEP 489 multiphase initialization can lead to
its m_traverse function (and presumably also m_clear and m_free) to be
called while not module state is yet registered: that is,
`PyModule_GetState(self)` when called from m_traverse returns NULL!

Is that an expected or known subtlety?


Thank you for looking into this, Antoine!


Not that I'm aware of, so I'd be inclined to classify it as a bug in the
way we're handling multi-phase initialisation unless/until we determine
there's no way to preserve the existing invariant from the single phase
case.


Yes, it's a bug – at least in documentation.
From initial investigation, the problem is that between the two phases 
of multi-phase init, module state is NULL, and Python code can run.
This is expected, so I'm thinking m_traverse for all modules using 
multi-phase init should have a check for NULL. And this should be 
documented.

Let's have Marcel run with this a bit further.



Speaking of which, the doc is not very clear: is PEP 489 required for
multi-interpreter support or is PyModule_GetState() sufficient?


I'm not exactly sure what you're asking; which doc are you referring to?

PEP 489 gives you good defaults, if you use it and avoid global state 
(roughly: C-level mutable static variables), then you should get 
multi-interpreter support for free in simple cases.
It's also possible to use PyModule_GetState() and other APIs directly. 
However, I'd like to avoid solving subinterpreter support separately 
(and slightly differently) in each module.



For a slightly bigger picture: as a part-time internship, Marcel is 
identifying where PEP 489 is inadequate, and solving the problems for 
the complex cases. This is part of better support for subinterpreter 
support in general. Going with a PEP 489-based solution with atexit 
would help us in that effort.
I'm assuming fixing the atexit bug from 2009 [0] can be delayed a bit as 
issues with PEP 489 are investigated & solved.

Does that sound fair?


[0] https://bugs.python.org/issue6531
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state

[Petr Viktorin]

On Thu, Dec 14, 2017 at 12:00 PM, Antoine Pitrou  wrote:
> On Thu, 14 Dec 2017 17:00:10 +1000
> Nick Coghlan  wrote:
>> On 14 Dec. 2017 9:19 am, "Antoine Pitrou"  wrote:
>>
>>
>> Hello,
>>
>> After debugging a crash on AppVeyor for a submitter's PR
>> (see https://github.com/python/cpython/pull/4611 ), I came to the
>> following diagnosis: converting the "atexit" module (which is a
>> built-in C extension) to PEP 489 multiphase initialization can lead to
>> its m_traverse function (and presumably also m_clear and m_free) to be
>> called while not module state is yet registered: that is,
>> `PyModule_GetState(self)` when called from m_traverse returns NULL!
>>
>> Is that an expected or known subtlety?
>>
>>
>> Not that I'm aware of, so I'd be inclined to classify it as a bug in the
>> way we're handling multi-phase initialisation unless/until we determine
>> there's no way to preserve the existing invariant from the single phase
>> case.
>
> Speaking of which, the doc is not very clear: is PEP 489 required for
> multi-interpreter support or is PyModule_GetState() sufficient?

Yes, it is possible to have proper subinterpreter support without
multi-phase init.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state

[Petr Viktorin]

On 12/19/2017 04:19 PM, Antoine Pitrou wrote:

On Tue, 19 Dec 2017 16:10:06 +0100
Petr Viktorin  wrote:


Speaking of which, the doc is not very clear: is PEP 489 required for
multi-interpreter support or is PyModule_GetState() sufficient?


Yes, it is possible to have proper subinterpreter support without
multi-phase init.


Thanks.  I guess the C API docs need a user-friendly section laying out
the various methods for initializing a module, and their various
advantages :-)


That, or eventually remove multi-phase init's disadvantages, and have 
just one way to do it :)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/23/18 14:04, Jeroen Demeyer wrote:

Hello,

I just saw this PEP. There is a bit of overlap between PEP 573 and PEP 
575 since these both change the calling convention for built-in methods. 
In particular, PEP 575 also proposes to add a "defining class" member 
(for different reasons). In PEP 575, this is added to the PyCFunction 
struct itself instead of a separate struct PyCMethod.


The reason you just saw the pep is because it wasn't posted to 
python-dev yet, and it wasn't posted yet because we first want to look 
for potential conflicts with 573.


(There's no one forking on PEP 573 full-time, so unfortunately it's 
progressing slower than I'd like.)


It would be nice to justify whether you really need a new class 
(PyCMethod_Type) to support METH_METHOD. It looks strange to me that the 
class of some object depends on an implementation detail like whether 
METH_METHOD is specified.


The current PEP 573 implies that backwards compatibility concerns would 
arise every time that METH_METHOD is added to an existing method. People 
have asked questions on PEP 575 about that: it would break code 
depending on "types.BuiltinFunctionType" for example. You could instead 
just change PyCFunctionObject to add that field (that's what I did in 
PEP 575).


For practical reasons, it would be nice to implement PEP 573 and PEP 575 
together as they affect the same code (assuming that both PEPs are 
accepted of course).


I currently even think PEP 575 can go forward *before* PEP 573. Having 
__objclass__ on methods of extension classes does sounds like a more 
elegant solution! And for PEP 573 it would mean one less annoying 
problem to solve.



Reading PEP 575, I miss an explanation of what __objclass__ *is* -- it 
only says the technical restrictions on it. For PEP 575 I found it 
pretty important that the reader gets a good mental picture of what it 
is, but it's a bit difficult to explain succinctly.

Maybe something like this would help make it clearer in PEP 573?

If set, `__objclass__` is the class that defines the method (which might 
be a superclass of `type(self)`). Functionally it's equivalent to the 
`__class__` cell [0] in Python code.


[0] 
https://docs.python.org/3/reference/datamodel.html#creating-the-class-object


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/24/18 13:12, Jeroen Demeyer wrote:

On 2018-04-24 16:34, Jeroen Demeyer wrote:

On the other hand, if you are passing the function object, then you can
get __self__ from it (unless it's an unbound method: in that case
__self__ is NULL and self is really args[0]). So there wouldn't be a
need for passing "self". I'm not saying that this is better than passing
"self" explicitly... I haven't yet decided what is best.


One thing I realized from PEP 573: the fact that __self__ for built-in 
functions is set to the module is considered a feature. I never 
understood the reason for it (and I don't know if the original reason 
was the same as the reason in PEP 573).


If we want to continue supporting that and we also want to support 
__get__ for built-in functions (to make them act as methods), then there 
are really two "selfs": there is the "self" from the method (the object 
that it's bound to) and the "self" from the built-in function (the 
module). To support that, passing *both* the function and "self" seems 
like the best way.


You're talking about functions with METH_BINDING here, right?
There the other "self" would be the defining module.
It might make sense to pass that also in the struct, rather than as an 
additional argument.
Perhaps "m_objclass" could point to the module in this case, or a new 
pointer could be added.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/25/18 14:46, Jeroen Demeyer wrote:

On 2018-04-25 20:33, Petr Viktorin wrote:

Perhaps "m_objclass" could point to the module in this case


That was exactly my idea also today. Instead of treating m_objclass as 
the defining class, we should generalize it to be the "parent" of the 
function: either the class or the module.


Great to hear we think alike.

However, I think that while reusing the pointer is nice to save space, 
the two concepts should still be separate, because "defining module" is 
a reasonable concept even for methods. In particular:

- There should be *separate* accessor functions for:
  - getting the defining class
  - getting the defining module
- The latter would later (in PEP 573) be extended to return the defining 
module even for class methods (when available)

- In Python code, __objclass__ should be the defining class, not the module.
- The C field should have a different name (m_parent?), so it isn't that 
strongly associated with __objclass__.


Does that sound reasonable?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 394 update proposal: Allow changing the `python` command in some cases

[Petr Viktorin]

Hello,
In Fedora, I found that PEP 394's strict recommendation that `python` 
points to `python2` is holding us back. From discussions on Zulip and 
elsewhere it's clear that this recommendation is not changing any time 
soon, but I would like to officially relax it in several cases.


The problems are:
- For developers that are not following the language's development, the 
fact that `python` invokes `python2` sends a strong signal that 2 is 
somehow the preferred version, and it's OK to start new projects in it.
- Users and sysadmins that *do* want to “live in the future” are 
switching the symlink to `python3` themselves. We would like to give 
them a supported, documented way to do so -- and make surer they're 
aware of the caveats.
- The `python` command is still not available out-of-the box on macOS, 
so it didn't completely live up to the expectation of being the 
cross-platform way to launch 2/3 source compatile scripts.
- `python` in the shebang line can mean *either* that a script is 
carefully written to be 2/3 compatible, *or* that the author/maintainer 
is lazy or unaware of the recommendations. While Fedora guidelines have 
long banned the unversioned command, we feel that the only way to 
*enforce* that guidelines is to provide environments where the `python` 
command does not work (unless explicitly installed).


To help solve these, I would like to relax recommendations on the Unix 
``python -> python2`` symlink in these cases:


- Users and administrators can, by a deliberate action, change 
``python`` to invoke Python 3. (Activating a venv counts as such an 
action, but so would e.g. using alternates, installing a non-default 
overriding package, or replacing /usr/bin/python.)
- In controlled environments where being explicit is valued more than 
user experience (test environments, build systems, etc.), distributions 
can omit the `python` command even when `python2` is installed.


I have filed these changes as a pull request here:

  https://github.com/python/peps/pull/630

The PR also spells out several other things, which I felt were hidden 
between the lines -- but correct me if you disagree with my reading.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/26/18 09:32, Thomas Wouters wrote:


Thanks for working on this, Marcel (and Petr). This looks like an 
ambitious intern project :) Couple of questions and comments in-line.


Indeed, but also a multi-year one. Sometimes it seems like Python is 
moving too fast under us!


Specifically, PEP 573 (Unifying function/method classes) andto a smaller 
extent PEP 567 (Context Variables) seem to be solving similar problems 
as this PEP. I'm inclined to put this PEP temporarily on hold, help 575 
get accepted, and then rebase on top of that.



On Mon, Apr 23, 2018 at 12:36 PM, Marcel Plch > wrote:

PEP: 573

[...]


Abstract



[...]


Additionaly, support for easier creation of immutable exception
classes is added.


I'm not a fan of using 'immutable' here, or in the API function name. I 
understand the types are to some extent immutable (apart from their 
refcount, I assume), but I think it's going to be too easy to confuse it 
with types whose *instances* are immutable. (We do occasionally say 
things like "tuples are an immutable type".) Since the point is that 
they behave like statically defined ones, perhaps 'Static' would be a 
reasonable replacement.


That was the first naming choice, but then on the other hand, "static 
heap type" sounds like an oxymoron -- in C, static things don't live on 
the heap.

Naming is hard :(


[...]

Slot methods


The above changes don't cover slot methods, such as ``tp_iter`` or
``nb_add``.

The problem with slot methods is that their C API is fixed, so we can't
simply add a new argument to pass in the defining class.
Two possible solutions have been proposed to this problem:

     * Look up the class through walking the MRO.
       This is potentially expensive, but will be useful if
performance is not
       a problem (such as when raising a module-level exception).
     * Storing a pointer to the defining class of each slot in a
separate table,
       ``__typeslots__`` [#typeslots-mail]_.  This is technically
feasible and fast,
       but quite invasive.

Due to the invasiveness of the latter approach, this PEP proposes
adding an MRO walking
helper for use in slot method implementations, deferring the more
complex alternative
as a potential future optimisation. Modules affected by this concern
also have the
option of using thread-local state or PEP 567 context variables, or
else defining their
own reload-friendly lookup caching scheme.


I do not believe walking the MRO is going to work without reworking the 
implementation of types, specifically how typeobject.c deals with slots 
of subclasses: in some cases copies the slots from the base class (see 
inherit_slots() and from where it's called). I believe this would cause 
problems if, for example, you define type X in module A, subclass it 
from type Y in module B without overriding the slot, and try to find the 
module object for A from the slot implementation. I don't think copying 
slots is a requirement for the desired semantics, but it's going to be 
fairly involved to rewrite it to do something else. There's also 
backward-compatibility to consider: third-party libraries can be 
inheriting from builtin types (e.g. numpy does this extensively) using 
the same copying-slot mechanism, which means those builtin types can't 
use the MRO walking to find their module without breaking compatibility 
with those third-party libraries.


The MRO walking code needs to deal with copied slots. It's not 
straigtforward, but I do think it's possible.



[...]

Static exceptions
-

A new function will be added::

     int PyErr_PrepareImmutableException(PyTypeObject **exc,
                                      const char *name,
                                      const char *doc,
                                      PyObject *base)

Creates an immutable exception type which can be shared
across multiple module objects.


How is this going to deal with type.__subclasses__()? Is re-using the 
static type object between reloads and sub-interpreters important enough 
to warrant the different behaviour? What if sub-interpreters end up 
wanting to disallow sharing objects between them?


Argh. Yes, subclasses seem to be the obvious shortcoming that a fresh 
pair of eyes is bound to find. Thanks!

This sends the proposal back to the drawing board.



[...]

Modules Converted in the Initial Implementation
---

To validate the approach, several modules will be modified during
the initial implementation:

The ``zipimport``, ``_io``, ``_elementtree``, and ``_csv`` modules
will be ported to PEP 489 multiphase initialization.


zipimport currently caches things in C globals. Changing it to use PEP 
489 multi-phase initialisation is very likely goi

Re: [Python-Dev] PEP 394 update proposal: Allow changing the `python` command in some cases

[Petr Viktorin]

On 04/27/18 02:03, Ben Finney wrote:

Ben Finney  writes:


Petr Viktorin  writes:


[…] we feel that the only way to *enforce* that guidelines is to
provide environments where the `python` command does not work
(unless explicitly installed).


Yes. The ‘python’ command is confusing, for the reasons you say. There
should be ‘python2’ and ‘python3’ commands for Python 2 and Python 3
respectively, and no ‘python’ command should be installed by the
operating system.

The fact that ‘/usr/bin/python’ exists is an historical accident, and I
agree with the proposal you state: the best way to correct the confusion
is to bar the confusing command from being installed by packages.


Because the above is ambiguous, I'll clarify: I am not calling for, and
PEP 394 does not call for, the banishment of the ‘python’ command.


Well, Guido *is* calling for it :)
It would break too many things, but after discussions on the PR, it's 
clear that we want a future where the "python" doesn't exist.

But while it's available, it should point to Python 2.


What I'm saying is that muddying the rules further on what ‘python’ may
or may not mean is *worse than* banishing the ‘python’ command entirely.


That's also consistent with the PR discussion. (But not that much with 
the original PEP, which said `python` is expected to eventually mean 
`python3`.)



So, short of banishing ‘python’ entirely, I think PEP 394 is already a
good clear way to address the issue. Existing, documented and supported
means to locally modify a ‘python’ command already exist and should be
sufficient.


I trust that PEP 394 will not be weakened in its effect, and I wish you
well with using the already-supported, already-documented, PEP-394
compatible means to add local customisations for a ‘python’ command.


Right. But some already-supported, already-documented mechanisms like 
Debian Alternatives or alternate package repos, are not compatible with 
PEP 394.
And as a PEP 394 compliant distro, we also won't be promoting the 
/usr/local or $HOME/bin ways to change `python` (which makes me a bit 
sad, because that documentation might have included a link to the 
caveats listed in the PEP).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 394: Allow the `python` command to not be installed (and other minor edits)

[Petr Viktorin]

Hello,
After discussion on the [Pull Request], my update to PEP 394 changed 
scope somewhat. The new major changes are:


- The `python` command may not exist at all in some cases (see the PEP 
for details)
- The paragraph about the anticipated future where python points to 
Python 3 is removed. (We'd rather see a future where `python` doesn't 
exist and one always has to specify `python2` or `python3`.)
- The PEP now explicitly says that in an active venv, python means that 
venv's interpreter. (Some view this as a design mistake, but one not 
worth reverting now.)


There are also other edits and clarifications.

Thanks for everyone involved, especially Guido for pitching in with the 
intended direction -- which was not clear from (or is genuinely 
different from) the 7-year-old PEP!


I'll keep the PR open for a day or so, in case someone still wants to 
comment.


[Pull Request]: https://github.com/python/peps/pull/630


On 04/26/18 19:21, Ben Finney wrote:

Petr Viktorin  writes:


In Fedora, I found that PEP 394's strict recommendation that `python`
points to `python2` is holding us back.


I have read the message, but I don't see how you draw the link that PEP
394 is holding you back.


The problems are:
- For developers that are not following the language's development,
the fact that `python` invokes `python2` sends a strong signal that 2
is somehow the preferred version, and it's OK to start new projects in
it.


I agree with the statement you make later in the message:


[…] we feel that the only way to *enforce* that guidelines is to
provide environments where the `python` command does not work (unless
explicitly installed).


Yes. The ‘python’ command is confusing, for the reasons you say. There
should be ‘python2’ and ‘python3’ commands for Python 2 and Python 3
respectively, and no ‘python’ command should be installed by the
operating system.

The fact that ‘/usr/bin/python’ exists is an historical accident, and I
agree with the proposal you state: the best way to correct the confusion
is to bar the confusing command from being installed by packages.


- Users and sysadmins that *do* want to “live in the future” are
switching the symlink to `python3` themselves. We would like to give
them a supported, documented way to do so -- and make surer they're
aware of the caveats.


The supported, documented way to add a command pointing to a different
command already exists, and there is no need to make a Python-specific
special case.

Users who want to make a ‘python’ alias can do so in their shell; this
is supported and documented.

Users who want to add a new command file can add a suitable directory
(e.g. ‘$HOME/bin’) to their ‘PATH’ variable, and put a symlink in there
named ‘python’. This is supported and documented.

Sysadmins who want to create a system-wide command ‘python’ can put a
symlink at ‘/usr/local/bin/python’. This is supported and documented.

I disagree with making some special-case extra way; that would be both
cunfusing and superfluous.


- The `python` command is still not available out-of-the box on macOS,
so it didn't completely live up to the expectation of being the
cross-platform way to launch 2/3 source compatile scripts.


That is one of the minor ways which macOS fails to conform to
community-agreed conventions. We should not let that intransigence
distort our discussion of best practices.


To help solve these, I would like to relax recommendations on the Unix
``python -> python2`` symlink in these cases:


For the above reasons, I disagree that PEP 394 is limiting what you want
to do on free-software operating systems.

For non-free operating systems, I don't think the already-discussed PEP
394 should be weakened if the operating system vendor fails to conform.


- Users and administrators can, by a deliberate action, change
``python`` to invoke Python 3.


Yes. That is well-known and long-standardised on Unix operating systems,
and is much more broadly understood than any Python-specific special
case would be. So I don't see how anyone is being held back.

I trust that PEP 394 will not be weakened in its effect, and I wish you
well with using the already-supported, already-documented, PEP-394
compatible means to add local customisations for a ‘python’ command.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.



Hi,
I'm reading the PEP thoroughly, trying to "swap it into my brain" for 
the next few days.
It does quite a lot of things, and the changes are all intertwined, 
which will make it hard to get reviewed and accepted.
Are there parts that can be left to a subsequent PEP, to simplify the 
document (and implementation)?
It seems to me that the current complexity is (partly) due to the fact 
that how functions are *called* is tied to how they are *introspected*. 
Perhaps starting to separate that is a better way to untangle things 
than arranging a class hierarchy?



Can the problem of allowing introspection ("It is currently not possible 
to implement a function efficiently in C (only built-in functions can do 
that) while still allowing introspection like inspect.signature or 
inspect.getsourcefile (only Python functions can do that)") be solved in 
a better way?


Maybe we can change `inspect` to use duck-typing instead of isinstance? 
Then, if built-in functions were subclassable, Cython functions could 
need to provide appropriate __code__/__defaults__/__kwdefaults__ 
attributes that inspect would pick up.
Maybe we could eve add more attributes (__isgenerator__?) to separate 
how a function is called from how it should be introspected -- e.g. make 
inspect not consult co_flags.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.


The PEP says:


User flags: METH_CUSTOM and METH_USRx
These flags are meant for applications that want to use tp_methods for an 
extension type or m_methods for a module but that do not want the default 
built-in functions to be created. Those applications would set METH_CUSTOM. The 
application is also free to use METH_USR0, ..., METH_USR7 for its own purposes, 
for example to customize the creation of special function instances.

There is no immediate concrete use case, but we expect that tools which 
auto-generate functions or extension types may want to define custom flags. 
Given that it costs essentially nothing to have these flags, it seems like a 
good idea to allow it.


Why are these flags added?
They aren't free – the space of available flags is not infinite. If 
something (Cython?) needs eight of them, it would be nice to mention the 
use case, at least as an example.


What should Python do with a m_methods entry that has METH_CUSTOM set? 
Again it would be nice to have an example or use case.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/15/18 05:15, Jeroen Demeyer wrote:

On 2018-05-14 19:56, Petr Viktorin wrote:

It does quite a lot of things, and the changes are all intertwined,
which will make it hard to get reviewed and accepted.


The problem is that many things *are* already intertwined currently. You 
cannot deal with functions without involving methods for example.


An important note is that it was never my goal to create a minimal PEP. 
I did not aim for changing as little as possible. I was thinking: we are 
changing functions, what would be the best way to implement them?


That might be a problem. For the change to be accepted, a core developer 
will need to commit to maintaining the code, understand it, and accept 
responsibility for anything that's broken. Naturally, large-scale 
changes have less of a chance there.


With such a "finished product" PEP, it's hard to see if some of the 
various problems could be solved in a better way -- faster, more 
maintainable, or less disruptive.
It's also harder from a psychological point of view: you obviously 
already put in a lot of good work, and it's harder to waste that work if 
an even better solution is found. (I always tell Marcel to view 
large-scale changes as a hands-on learning experiment -- more likely to 
be thrown away than accepted -- rather than as creating a finished project.)


The main goal was fixing introspection but a secondary goal was fixing 
many of the existing warts with functions. Probably this secondary goal 
will in the end be more important for the general Python community.


I would argue that my PEP may look complicated, but I'm sure that the 
end result will be a simpler implementation than we have today. Instead 
of having four related classes implementing similar functionality 
(builtin_function_or_method, method, method_descriptor and function), we 
have just one (base_function). The existing classes like method still 
exist with my PEP but a lot of the core functionality is implemented in 
the common base_function.


Is a branching class hierarchy, with quite a few new of flags for 
feature selection, the kind of simplicity we want?
Would it be possible to first decouple things, reducing the complexity, 
and then tackle the individual problems?


This is really one of the key points: while my PEP *could* be 
implemented without the base_function class, the resulting code would be 
far more complicated.



Are there parts that can be left to a subsequent PEP, to simplify the
document (and implementation)?


It depends. The current PEP is more or less a finished product. You can 
of course pick parts of the PEP and implement those, but then those 
parts will be somewhat meaningless individually.


But if PEP 575 is accepted "in principle" (you accept the new class 
hierarchy for functions), then the details could be spread over several 
PEPs. But those individual PEPs would only make sense in the light of 
PEP 575.


Well, that's the thing I'm not sure about.
The class hierarchy still makes it hard to decouple the introspection 
side (how functions look on the outside) from the calling mechanism (how 
the calling works internally). It fear that it is replacing complexity 
with a different kind of complexity.
So my main question now is, can this all be *simplified* rather than 
*reorganized*? It's a genuine question – I don't know, but I feel it 
should be explored more.


A few small details could be left out, such as METH_BINDING. But that 
wouldn't yield a significant simplification.



It seems to me that the current complexity is (partly) due to the fact
that how functions are *called* is tied to how they are *introspected*.


The *existing* situation is that introspection is totally tied to how 
functions are called. So I would argue that my PEP improves on that by 
removing some of those ties by moving __call__ to a common base class.



Maybe we can change `inspect` to use duck-typing instead of isinstance?


That was rejected on https://bugs.python.org/issue30071


Then, if built-in functions were subclassable, Cython functions could
need to provide appropriate __code__/__defaults__/__kwdefaults__
attributes that inspect would pick up.


Of course, that's possible. I don't think that it would be a *better* 
solution than my PEP though.


Essentially, my PEP started from that idea. But then you realize that 
you'll need to handle not only built-in functions but also method 
descriptors (unbound methods of extension types). And you'll want to 
allow __get__ for the new subclasses. For efficiency, you really want to 
implement __get__ in the base classes (both builtin_function_or_method 
and method_descriptor) because of optimizations combining __get__ and 
__call__ (the LOAD_METHOD and CALL_METHOD opcodes). And then you realize 
that it makes no sense to duplicate all that functionality in both 
classes. So you add a new base class. You already

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/15/18 17:55, Jeroen Demeyer wrote:

On 2018-05-15 18:36, Petr Viktorin wrote:

Naturally, large-scale
changes have less of a chance there.


Does it really matter that much how large the change is? I think you are 
focusing too much on the change instead of the end result.


As I said in my previous post, I could certainly make less disruptive 
changes. But would that really be better? (If you think that the answer 
is "yes" here, I honestly want to know).


Yes, I believe it is better.
The larger a change is, the harder it is to understand, meaning that 
less people can meaningfully join the conversation, think about how it 
interacts with their own use cases, and notice (and think through) any 
unpleasant details.

Less disruptive changes tend to have a better backwards compatibility story.
A less intertwined change makes it easier to revert just a single part, 
in case that becomes necessary.


I could make the code less different than today but at the cost of added 
complexity. Building on top of the existing code is like building on a 
bad foundation: the higher you build, the messier it gets. Instead, I 
propose a solid new foundation. Of course, that requires more work to 
build but once it is built, the finished building looks a lot better.


To continue the analogy: the tenants have been customizing their 
apartments inside that building, possibly depending on structural 
details that we might think should be hidden from them. And they expect 
to continue living there while the foundation is being swapped under them :)



With such a "finished product" PEP, it's hard to see if some of the
various problems could be solved in a better way -- faster, more
maintainable, or less disruptive.


With "faster", you mean runtime speed? I'm pretty confident that we 
won't lose anything there.


As I argued above, my PEP might very well make things "more 
maintainable", but this is of course very subjective. And "less 
disruptive" was never a goal for this PEP.



It's also harder from a psychological point of view: you obviously
already put in a lot of good work, and it's harder to waste that work if
an even better solution is found.


I hope that this won't be my psychology. As a developer, I prefer to 
focus on problems rather than on solutions: I don't want to push a 
particular solution, I want to fix a particular problem. If an even 
better solution is accepted, I will be a very happy man.


What I would hate is that this PEP gets rejected because some people 
claim that the problem can be solved in a better way, but without 
actually suggesting such a better way.


Mark Shannon has an upcoming PEP with an alternative to some of the 
issues. (Not all of them – but less intertwined is better, all else 
being equal.)



Is a branching class hierarchy, with quite a few new of flags for
feature selection, the kind of simplicity we want?


Maybe yes because it *concentrates* all complexity in one small place. 
Currently, we have several independent classes 
(builtin_function_or_method, method_descriptor, function, method) which 
all require various forms of special casing in the interpreter with some 
code duplication. With my PEP, this all goes away and instead we need to 
understand just one class, namely base_function.



Would it be possible to first decouple things, reducing the complexity,
and then tackle the individual problems?


What do you mean with "decouple things"? Can you be more concrete?


Currently, the "outside" of a function (how it looks when introspected) 
is tied to the "inside" (what happens internally when it's called). 
That's what I'd like to see decoupled.
Can we better enable pydoc/IPython developers to tackle introspection 
problems without wading deep in the internals and call optimizations?



The class hierarchy still makes it hard to decouple the introspection
side (how functions look on the outside) from the calling mechanism (how
the calling works internally).


Any class who wants to profit from fast function calls can inherit from 
base_function. It can add whatever attributes it wants and it can choose 
to implement documentation and/or introspection in whatever way it 
wants. It can choose to not care about that at all. That looks very 
decoupled to me.


But, it still has to inherit from base_function to "look like a 
function". Can we remove that limitation in favor of duck typing?



Starting from an idea and ironing out the details it lets you (and, if
since you published results, everyone else) figure out the tricky
details. But ultimately it's exploring one path of doing things – it
doesn't necessarily lead to the best way of doing something.


So far I haven't seen any other proposals...


That's a good question. Maybe inspect.isfunction() serves too many use
cases to be useful. Cython functons sho

Re: [Python-Dev] the new(-ish) dict ordering vs hash randomization

[Petr Viktorin]

On 06/18/18 15:13, Ethan Furman wrote:
I'm sure we've already had this conversation, but my google-fu is 
failing me.


Can someone provide a link to a discussion explaining why the new 
ordering of dictionaries does not defeat the hash-randomization 
non-ordering we added a few versions ago?


Hi,
Modern dicts have an array of contents (which is used for iterating the 
dict, and thus iteration doesn't touch hashes at all), and a separate 
hash table of indexes (which still enjoys the benefits of hash 
randomization).


See Raymond Hettinger's initial post from 2012: 
https://mail.python.org/pipermail/python-dev/2012-December/123028.html


A technical overview of the idea is on the PyPy blog:
https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Policy on refactoring/clean up

[Petr Viktorin]

On 06/26/18 14:13, Jeroen Demeyer wrote:

On 2018-06-26 13:54, Ivan Pozdeev via Python-Dev wrote:

This is exactly what that the YAGNI principle is about, and Inada was
right to point to it. Until you have an immediate practical need for
something, you don't really know the shape and form for it that you will
be the most comfortable with. Thus any "would be nice to have"
tinkerings are essentially a waste of time and possibly a degradation,
too: you'll very likely have to change them again when the real need
arises -- while having to live with any drawbacks in the meantime.


It is important to clarify that this is exactly what I did. I *have* an 
implementation of PEP 580 and it's based on that PR 7909.


I just think that this PR makes sense independently of whether PEP 580 
will be accepted.



So, if you suggest those changes together with the PEP 580 PR


That sounds like a bad idea because that would be mixing two issues in 
one PR. If I want to increase my chances of getting PEP 580 and its 
implementation accepted, I shouldn't bring in unrelated changes.


To put it in a different perspective: if somebody else would make a PR 
to one of my projects doing a refactoring and adding new features, I 
would ask them to split it up.


Actually, that's exactly what we *did* ask Jeroen with his earlier 
proposal for PEP 575, where the implementation ended up being quite big. 
Split the changes to make it more manageable.


Unfortunately I haven't had time to study this PR yet (work is taking 
all my time lately), but I trust that Jeroen will propose actual 
improvements on top of the clean-up.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 489 (Multi-phase extension module initialization)

[Petr Viktorin]

On Fri, May 22, 2015 at 11:44 PM, Eric Snow  wrote:
> Hi all,
>
> After extended discussion over the last several months on import-sig,
> the resulting proposal for multi-phase (PEP 451) extension module
> initialization has finalized.  The resulting PEP provides a clean,
> straight-forward, and backward-compatible way to import extension
> modules using ModuleSpecs.
>
> With that in mind and given the improvement it provides, PEP 489 is
> now accepted.  I want to thank Petr, Nick, and Stefan for the time,
> thought, and effort they put into the proposal (and implementation).
> It was a disappointment to me when, at the time, we couldn't find a
> good way to apply PEP 451 to builtins and extension modules.  So
> thanks for easing my anxiety!

Thank you for the thorough review, Eric! Also thanks to everyone
involved for insightful discussion, and especially Nick for guiding me
through writing my first PEP.
Let me know if there's anything more I can (or should -- still my
first time) do.
I'm off now but I'll be online tomorrow all day (UTC-ish hours).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Avoiding reference leaks in heap types with custom tp_dealloc

[Petr Viktorin]

Hello,
The new test_importlib.extension.test_loader is currently leaking
references, (issue24268). There is a simple hack to stop this, but I'm
inclined to not apply quick hacks and rather dig into the root cause.
(It's a test module, the refleaks are relatively harmless.)

The tests are based directly on the "xxlimited" example,
xxlimited.Xxo, which exhibits the same bug -- it's just not tested.
It's is caused by a combination of a few factors, but I'm not sure
what's a bug and what's just undocumented behavior, so I'm asking for
input to put me on the right track.

As reported in issue16690, heap types with a naïve custom tp_dealloc
leak references to the type when instantiated. According to [0], it
seems that tp_dealloc should check if it has been overridden, and if
so, decref the type. This needs to be documented (regardless of the
solution to the other problems), and I intend to document it.
We can change xxlimited to do the check and decref, but isn't it too
ugly for a module that showcases the extension module API?
(xxlimited.Xxo can technically skip the check, since it doesn't allow
subclasses, but that would be setting a nasty trap for anyone learning
from that example.)

The nice way out would be taking advantage of PEP 442: xxlimited.Xxo
can ditch tp_dealloc in favor of tp_traverse and tp_finalize (the
former of which it needs anyway to behave correctly). Unfortunately,
tp_finalize is not available in the stable ABI (issue24345). I think
it should be added; is it too late for 3.5?

Another problem is that xxlimited is untested. It's only built for
non-debug builds, because Py_LIMITED_API and Py_DEBUG are
incompatible. Would it make sense to build and test it without
Py_LIMITED_API in debug mode, instead of not building it at all?


[0] http://bugs.python.org/issue15653#msg168449
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Avoiding reference leaks in heap types with custom tp_dealloc

[Petr Viktorin]

On Mon, Jun 1, 2015 at 5:33 PM, Antoine Pitrou  wrote:
> On Mon, 1 Jun 2015 16:38:35 +0200
> Petr Viktorin  wrote:
[...]
>> The nice way out would be taking advantage of PEP 442: xxlimited.Xxo
>> can ditch tp_dealloc in favor of tp_traverse and tp_finalize (the
>> former of which it needs anyway to behave correctly). Unfortunately,
>> tp_finalize is not available in the stable ABI (issue24345). I think
>> it should be added; is it too late for 3.5?
>
> Well, but the stable ABI is supposed to be a subset of the API
> that's safe to program against, regardless of the Python version (at
> least from the point where the stable ABI was introduced).

That part's not true. From the PEP:
During evolution of Python, new ABI functions will be added.
Applications using them will then have a requirement on a minimum
version of Python; this PEP provides no mechanism for such
applications to fall back when the Python library is too old.

> What happens
> if you define a Py_tp_finalize and run your C extension type on a
> pre-3.5 version? Do you get an error at definition time? A resource
> leak? A crash?

Looking at PyType_FromSpecWithBases code, you should get
RuntimeError("invalid slot offset") in this case.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Avoiding reference leaks in heap types with custom tp_dealloc

[Petr Viktorin]

On Mon, Jun 1, 2015 at 6:00 PM, Antoine Pitrou  wrote:
[...]
> I think we have been laxist with additions to the stable ABI:
> apparently, they should be conditioned on the API version requested by
> the user.  For example, in pystate.h:
>
> #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x0303
> /* New in 3.3 */
> PyAPI_FUNC(int) PyState_AddModule(PyObject*, struct PyModuleDef*);
> PyAPI_FUNC(int) PyState_RemoveModule(struct PyModuleDef*);
> #endif
>
> (those were added by Martin, so I assume he knew what he was doing :-))
>
> This way, failing to restrict yourself to a given API version fails at
> compile time, not at runtime. However, it's also more work to do so
> when adding stuff, which is why we tend to skimp on it.

I see! I completely missed that memo.
I filed a patch that wraps my 3.5 additions as issue 24365.

I think this should be in the PEP, so people like me can find it. Does
the attached wording look good?
diff --git a/pep-0384.txt b/pep-0384.txt
index f042478..311bae4 100644
--- a/pep-0384.txt
+++ b/pep-0384.txt
@@ -360,6 +360,19 @@ whether their modules conform to the ABI. To avoid users having to
 rewrite their type definitions, a script to convert C source code
 containing type definitions will be provided [3]_.
 
+Considerations for CPython developers
+=
+
+When making additions to the stable ABI, the new definitions should be wrapped
+in a preprocessor conditional block that limits their visibility to
+a minimum stable ABI version. For example::
+
+#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x0303
+/* New in 3.3 */
+PyAPI_FUNC(int) PyState_AddModule(PyObject*, struct PyModuleDef*);
+PyAPI_FUNC(int) PyState_RemoveModule(struct PyModuleDef*);
+#endif
+
 References
 ==
 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

[Petr Viktorin]

On 01/14/2016 10:45 AM, Nick Coghlan wrote:
> On 14 January 2016 at 15:42, Benjamin Peterson  wrote:
>> This is a "well-known" issue. Parts of the interpreter (and especially,
>> extension modules) cheerfully stash objects in global variables with no
>> way to clean them up. Fixing this is a large project, which probably
>> involves implementing PEP 489.
> 
> The actual multi-phase extension module import system from 489 was
> implemented for 3.5, but indeed, the modules with stashed global state
> haven't been converted yet.

The hairy details on why the global variables haven't yet gone away are
on import-sig [0]. Nick suggested a workable solution there that I
really need to go back to and implement.

[0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reference cycle on the module dict (globals())

[Petr Viktorin]

On 01/19/2016 10:42 AM, Victor Stinner wrote:
> Hi,
> 
> While working on my FAT Python optimizer project, I found an annoying
> bug in my code. When at least one guard is created with a reference to
> the global namespace (globals(), the module dictionary), objects of
> the module are no more removed at exit.
> 
> Example:
> ---
> import sys
> 
> class MessageAtExit:
> def __del__(self):
> print('__del__ called')
> 
> # display a message at exit, when message_at_exit is removed
> message_at_exit = MessageAtExit()
> 
> # create a reference cycle:
> # module -> module dict -> Guard -> module dict
> guard = sys.Guard(globals())
> ---
> (the code is adapted from a test of test_gc)
> 
> Apply attached patch to Python 3.6 to get the sys.Guard object. It's a
> minimalist object to keep a strong reference to an object.
> 
> I expected the garbage collector to break such (simple?) reference cycle.
> 
> The Guard object implements a traverse module, but it is never called.
> 
> Did I miss something obvious, or is it a known issue of the garbage
> collector on modules?

The default type flags are for objects that don't store references.
Since you're creating a mutable container, you need to set
Py_TPFLAGS_HAVE_GC. See https://docs.python.org/3/c-api/gcsupport.html
for all the details.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Petr Viktorin]

On 02/11/2016 11:07 AM, Nick Coghlan wrote:
> On 11 February 2016 at 19:59, Victor Stinner  wrote:
>> 2016-02-11 9:11 GMT+01:00 Georg Brandl :
>>> On 02/11/2016 12:04 AM, Victor Stinner wrote:
 It looks like the implementation https://bugs.python.org/issue26331
 only changes the Python parser.

 What about other functions converting strings to numbers at runtime
 like int(str) and float(str)? Paul also asked for Decimal(str).
>>>
>>> I added these as "Open Questions" to the PEP.
>>
>> Ok nice. Now another question :-)
>>
>> Would it be useful to add an option to repr(int) and repr(float), or a
>> formatter to int.__format__() and float.__float__() to add an
>> underscore for thousands.
> 
> Given that str.format supports a thousands separator:
> 
 "{:,d}".format(1)
> '100,000,000'
> 
> it might be reasonable to permit "_" in place of "," in the format specifier.
> 
> However, I'm not sure when you'd use it aside from code generation,
> and you can already insert the thousands separator and then replace
> "," with "_".

It would make "SI style" [0] numbers a little bit more straightforward
to generate, since the order of operations wouldn't matter.
Currently it's:

"{:,}".format(1234.5678).replace(',', ' ').replace('.', ',')

Also it would make numbers with decimal comma and dot as separator a bit
easier to generate. Currently, that's (from PEP 378):

format(n, "6,f").replace(",", "X").replace(".", ",").replace("X", ".")

[0] https://en.wikipedia.org/wiki/Decimal_mark#Examples_of_use


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When should pathlib stop being provisional?

[Petr Viktorin]

On 04/06/2016 08:53 AM, Nathaniel Smith wrote:
> On Tue, Apr 5, 2016 at 11:29 PM, Nick Coghlan  wrote:
>> On 6 April 2016 at 15:57, Serhiy Storchaka  wrote:
>>> On 06.04.16 05:44, Nick Coghlan wrote:

 The most promising option for that is probably "getattr(path, 'path',
 path)", since the "path" attribute is being added to pathlib, and the
 given idiom can be readily adopted in Python 2/3 compatible code
 (since normal strings and any other object without a "path" attribute
 are passed through unchanged). Alternatively, since it's a protocol,
 double-underscores on the property name may be appropriate (i.e.
 "getattr(path, '__path__', path)")
>>>
>>> This was already discussed. Current conclusion is using the "path"
>>> attribute. See http://bugs.python.org/issue22570 .
>>
>> I'd missed the existing precedent in DirEntry.path, so simply taking
>> that and running with it sounds good to me.
> 
> This makes me twitch slightly, because NumPy has had a whole set of
> problems due to the ancient and minimally-considered decision to
> assume a bunch of ad hoc non-namespaced method names fulfilled some
> protocol -- like all .sum methods will have a signature that's
> compatible with numpy's, and if an object has a .log method then
> surely that computes the logarithm (what else in computing could "log"
> possibly refer to?), etc. This experience may or may not be relevant,
> I'm not sure -- sometimes these kinds of twitches are good guides to
> intuition, and sometimes they are just knee-jerk responses to an old
> and irrelevant problem :-). But you might want to at least think about
> how common it might be to have existing objects with unrelated
> attributes that happen to be called "path", and the bizarro problems
> that might be caused if someone accidentally passes one of them to a
> function that expects all .path attributes to be instances of this new
> protocol.
> 
> -n
> 

Python was in a similar situation with the .next method on iterators,
which changed to __next__ in Python 3. PEP 3114 (which explains this
change) says:

> Code that nowhere contains an explicit call to a next method can
> nonetheless be silently affected by the presence of such
> a method. Therefore, this PEP proposes that iterators should have
> a __next__ method instead of a next method (with no change in
> semantics).

How well does that apply to path/__path__?

That PEP also introduced the next() builtin. This suggests that a
protocol with __path__/__fspath__ would need a corresponding
path()/fspath() builtin.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failing tests (on a Linux distro)

[Petr Viktorin]

On 07/02/18 00:59, Miro Hrončok wrote:

On 1.7.2018 23:48, Matěj Cepl wrote:

On 2018-06-28, 00:58 GMT, Ned Deily wrote:

On behalf of the Python development community and the Python 3.7 release
team, we are pleased to announce the availability of Python 3.7.0.


I am working on updating openSUSE packages to python 3.7, but
I have hit quite large number of failing tests (the testsuite
obviously passed with 3.6), see
https://build.opensuse.org/package/show/home:mcepl:work/python3
(click on the red "failed" label to get logs). I fell into
a bout of depression, only to discover that we are not alone in
this problem ... Debian doesn't seem to do much better
https://is.gd/HKBU4j. Surprisingly, Fedora seems to pass the
testsuite https://is.gd/E0KA53; interesting, I will have to
investigate which of their many patches did the trick.


Note that we (=Fedora) unfortunately skip some tests.

https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec#_1051

https://src.fedoraproject.org/rpms/python3/blob/master/f/00160-disable-test_fs_holes-in-rpm-build.patch 



https://src.fedoraproject.org/rpms/python3/blob/master/f/00163-disable-parts-of-test_socket-in-rpm-build.patch 


[with my Fedora hat on]

Fedora* has been building python37 since the alphas, so the final update 
to rc/stable was smoother. But it also means we aren't solving the same 
issues as SUSE now, so we won't be able to help all that much :(

Do consider trying out alphas/betas in SUSE next time!
Anyway, the SUSE tests seem  to fail on .pyc files. The main change in 
that area was [PEP 552], try starting there. AFAIK, SUSE is ahead of 
Fedora in the reproducible builds area; perhaps that's where the 
difference is.


And while I'm responding here, a bit of reflection and a heads-up:
What Fedora as a distro should do better next time is re-build the 
entire ecosystem with a new Python version. For 3.7 we started doing 
that too late, and there are way too many projects that weren't prepared 
for `async` as keyword and PEP 479 (StopIteration handling).
If you run into similar problems in SUSE, you might want to take a look 
at issues tracked under [Fedora bug 1565020].



[PEP 552]: https://www.python.org/dev/peps/pep-0552/

[Fedora bug 1565020]: 
https://bugzilla.redhat.com/showdependencytree.cgi?id=1565020


* Thanks to Miro Hrončok for most of the work in Fedora
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Comparing PEP 576 and PEP 580

[Petr Viktorin]

On 07/05/18 13:59, Jeroen Demeyer wrote:

On 2018-07-05 13:32, INADA Naoki wrote:

Core devs interested in this area is limited resource.


I know and unfortunately there is nothing that I can do about that. It 
would be a pity that PEP 580 (or a variant like PEP 576) is not accepted 
simply because no core developer cares enough.


Hi,
I do care about this, and I'm really sorry I've been mostly silent here.
Unfortunately, this is the kind of work that can't be done with a few 
hours in the evenings, and currently an urgent project is sucking up all 
the big blocks of time I have :(

That project should be done in a month or two, however.





As far as I understand, there are some important topics to discuss.

a. Low level calling convention, including argument parsing API.
b. New API for calling objects without argument tuple and dict.
c. How more types can support FASTCALL, LOAD_METHOD and CALL_METHOD.
d. How to reorganize existing builtin types, without breaking stable ABI.


Right, that's why I wanted PEP 580 to be only about (c) and nothing 
else. I made the mistake in PEP 575 of also involving (d).


I still don't understand why we must finish (a) before we can even start 
discussing (c).



Reference implementation helps discussion.


METH_FASTCALL and argument parsing for METH_FASTCALL is already 
implemented in CPython. Not in documented public functions, but the 
implementation exists.


And PEP 580 also has a reference implementation:
https://github.com/jdemeyer/cpython/tree/pep580


Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can we split PEP 489 (extension module init) ?

[Petr Viktorin]

On 08/10/18 11:21, Stefan Behnel wrote:

Hi,

coming back to PEP 489 [1], the multi-phase extension module
initialization. We originally designed it as an "all or nothing" feature,
but as it turns out, the "all" part is so difficult to achieve that most
potential users end up with "nothing". So, my question is: could we split
it up so that projects can get at least the main advantages: module spec
and unicode module naming.

PEP 489 is a great protocol in the sense that it allows extension modules
to set themselves up in the same way that Python modules do: load, create
module, execute module code. Without it, creating the module and executing
its code are a single step that is outside of the control of CPython, which
prevents the module from knowing its metadata and CPython from knowing
up-front what the module will actually be.

Now, the problem with PEP 489 is that it requires support for reloading and
subinterpreters at the same time [2]. For this, extension modules must
essentially be free of static global state, which comprises both the module
code itself and any external native libraries that it uses. That is
somewhere between difficult and impossible to achieve. PEP 573 [3] explains
some of the reasons, and lists solutions for some of the issues, but cannot
solve the general problem that some extension modules simply cannot get rid
of their global state, and are therefore inherently incompatible with
reloading and subinterpreters.


Are there any issues that aren't explained in PEP 573?
I don't think Python modules should be *inherently* incompatible with 
subinterpreters. Static global state is perhaps unavoidable in some 
cases, but IMO it should be managed when it's exposed to Python.
If there are issues not in the PEPs, I'd like to collect the concrete 
cases in some document.



I would like the requirement in [2] to be lifted in PEP 489, to make the
main features of the PEP generally available to all extension modules.

The question is then how to opt out of the subinterpreter support. The PEP
explicitly does not allow backporting new init slot functions/feeatures:

"Unknown slot IDs will cause the import to fail with SystemError."

But at least changing this in Py3.8 should be doable and would be really nice.


I don't think we can just silently skip unknown slots -- that would mean 
modules wouldn't be getting features they asked for.
Do you have some more sophisticated model for slots in mind, or is this 
something to be designed?




What do you think?

Stefan



[1] https://www.python.org/dev/peps/pep-0489/
[2]
https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading
[3] https://www.python.org/dev/peps/pep-0573/


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can we split PEP 489 (extension module init) ?

[Petr Viktorin]

On 08/10/18 12:21, Stefan Behnel wrote:

Petr Viktorin schrieb am 10.08.2018 um 11:51:

On 08/10/18 11:21, Stefan Behnel wrote:

coming back to PEP 489 [1], the multi-phase extension module
initialization. We originally designed it as an "all or nothing" feature,
but as it turns out, the "all" part is so difficult to achieve that most
potential users end up with "nothing". So, my question is: could we split
it up so that projects can get at least the main advantages: module spec
and unicode module naming.

PEP 489 is a great protocol in the sense that it allows extension modules
to set themselves up in the same way that Python modules do: load, create
module, execute module code. Without it, creating the module and executing
its code are a single step that is outside of the control of CPython, which
prevents the module from knowing its metadata and CPython from knowing
up-front what the module will actually be.

Now, the problem with PEP 489 is that it requires support for reloading and
subinterpreters at the same time [2]. For this, extension modules must
essentially be free of static global state, which comprises both the module
code itself and any external native libraries that it uses. That is
somewhere between difficult and impossible to achieve. PEP 573 [3] explains
some of the reasons, and lists solutions for some of the issues, but cannot
solve the general problem that some extension modules simply cannot get rid
of their global state, and are therefore inherently incompatible with
reloading and subinterpreters.


Are there any issues that aren't explained in PEP 573?
I don't think Python modules should be *inherently* incompatible with
subinterpreters. Static global state is perhaps unavoidable in some cases,
but IMO it should be managed when it's exposed to Python.
If there are issues not in the PEPs, I'd like to collect the concrete cases
in some document.


There's always the case where an external native library simply isn't
re-entrant and/or requires configuration to be global. I know, there's
static linking and there are even ways to load an external shared library
multiple times, but that's just adding to the difficulties. Let's just
accept that some things are not easy enough to make for a good requirement.


For that case, I think the right thing to do is for the module to raise 
an extension when it's being initialized for the second time, or when 
the underlying library would be initialized for the second time.


"Avoid static global state" is a good rule of thumb for supporting 
subinterpreters nicely, but other strategies are possible.
If an underlying library just expects to be initialized once, and then 
work from several modules, the Python wrapper should ensure that (using 
global state, most likely). Other ways of handling things should be 
possible, depending on the underlying library.



I would like the requirement in [2] to be lifted in PEP 489, to make the
main features of the PEP generally available to all extension modules.

The question is then how to opt out of the subinterpreter support. The PEP
explicitly does not allow backporting new init slot functions/feeatures:

"Unknown slot IDs will cause the import to fail with SystemError."

But at least changing this in Py3.8 should be doable and would be really
nice.


I don't think we can just silently skip unknown slots -- that would mean
modules wouldn't be getting features they asked for.
Do you have some more sophisticated model for slots in mind, or is this
something to be designed?


Sorry for not being clear here. I was asking for changing the assumptions
that PEP 489 makes about modules that claim to support the multi-step
initialisation part of the PEP. Adding a new (flag?) slot was just one idea
for opting out of multi-initialisation support.


Would this be better than a flag + raising an error on init?
One big disadvantage of a big opt-out-of-everything button is that it 
doesn't encourage people to think about what the actual non-reentrant 
piece of code is.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34595: How to format a type name?

[Petr Viktorin]

On 09/11/18 15:23, Victor Stinner wrote:

Hi,

Last week, I opened an issue to propose to add a new %T formatter to
PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat()
and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add
something to get the "fully qualified name" (FQN) of a type, ex
"datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name).
I proposed a second pull request to add %t (short) in addition to %T
(FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get
a wider discussion. So here I am.


The rationale for this change is to fix multiple issues:

* C extensions use Py_TYPE(obj)->tp_name which returns a fully
qualified name for C types, but the name (without the module) for
Python name. Python modules use type(obj).__name__ which always return
the short name.


That might be a genuine problem, but I wonder if "%T" is fixing the 
symptom rather than the cause here.

Or is this only an issue for PyUnicode_FromFormat()?


* currently, many C extensions truncate the type name: use "%.80s"
instead of "%s" to format a type name


That's an orthogonal issue -- you can change "%.80s" to "%s", and 
presumably you could use "%.80t" as well.



* "%s" with Py_TYPE(obj)->tp_name is used more than 200 times in the C
code, and I dislike this complex pattern. IMHO "%t" with obj would be
simpler to read, write and maintain.


I consider `Py_TYPE(obj)->tp_name` much more understandable than "%t".
It's longer to spell out, but it's quite self-documenting.


* I want C extensions and Python modules to have the same behavior:
respect the PEP 399. Petr considers that error messages are not part
of the PEP 399, but the issue is wider than only error messages.


The other major use is for __repr__, which AFAIK we also don't guarantee 
to be stable, so I don't think PEP 399 applies to it.
Having the same behavior between C and Python versions of a module is 
nice, but PEP 399 doesn't prescribe it. There are other differences as 
well -- for example, `_datetime.datetime` is immutable, and that's OK.


If error messages and __repr__s should be consistent between Python and 
the C accelerator, are you planning to write tests for all the affected 
modules when switching them to %T/%t?




The main issue is that at the C level, Py_TYPE(obj)->tp_name is
"usually" the fully qualified name for types defined in C, but it's
only the "short" name for types defined in Python.

For example, if you get the C accelerator "_datetime",
PyTYPE(obj)->tp_name of a datetime.timedelta object gives you
"datetime.timedelta", but if you don't have the accelerator, tp_name
is just "timedelta".

Another example, this script displays "mytimedelta(0)" if you have the
C accelerator, but "__main__.mytimedelta(0)" if you use the Python
implementation:
---
import sys
#sys.modules['_datetime'] = None
import datetime

class mytimedelta(datetime.timedelta):
 pass

print(repr(mytimedelta()))
---

So I would like to fix this kind of issue.


Type names are mainly used for two purposes:

* format an error message
* obj.__repr__()

It's unclear to me if we should use the "short" or the "fully
qualified" name. It should maybe be decided on a case by case basis.

There is also a 3rd usage: to implement __reduce__, here backward
compatibility matters.


Note: The discussion evolved since my first implementation of %T which
just used the not well defined Py_TYPE(obj)->tp_name.

--

Petr asked me why not exposing functions to get these names. For
example, with my second PR (not merged), there are 3 (private)
functions:

/* type.__name__ */
const char* _PyType_Name(PyTypeObject *type);
/* type.__qualname__ */
PyObject* _PyType_QualName(PyTypeObject *type);
* type.__module__ "." type.__qualname__ (but type.__qualname__ for
builtin types) */
PyObject * _PyType_FullName(PyTypeObject *type);

My concern here is that each caller has to handler error:

   PyErr_Format(PyExc_TypeError, "must be str, not %.100s",
Py_TYPE(obj)->tp_name);

would become:

   PyObject *type_name = _PyType_FullName(Py_TYPE(obj));
   if (name == NULL) { /* do something with this error ... */
   PyErr_Format(PyExc_TypeError, "must be str, not %U", type_name);
   Py_DECREF(name);

When I report an error, I dislike having to handle *new* errors... I
prefer that the error handling is done inside PyErr_Format() for me,
to reduce the risk of additional bugs.

--

Serhiy also asked if we could expose the same feature at the *Python*
level: provide something to get the fully qualified name of a type.
It's not just f"{type(obj).__module}.{ty

Re: [Python-Dev] bpo-34595: How to format a type name?

[Petr Viktorin]

On 09/11/18 15:23, Victor Stinner wrote:

Hi,

Last week, I opened an issue to propose to add a new %T formatter to
PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat()
and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add
something to get the "fully qualified name" (FQN) of a type, ex
"datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name).
I proposed a second pull request to add %t (short) in addition to %T
(FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get
a wider discussion. So here I am.


After a discussion with Victor. I'll summarize where we are now.

There are actually two inconsistencies to fix:
- Python modules use `type(obj).__name__` and C extensions use 
`Py_TYPE(obj)->tp_name`, which inconsistent.
- Usage __name__ or __qualname__, and prepending __module__ or not, is 
inconsistent across types/modules.


It turns out that today, when you want to print out a type name, you 
nearly always want the fully qualified name (including the module unless 
it's "builtins"). So we can just have "%T" and not "%t". (Or we can add 
"%t" if a use case arises).


It should be possible to do this also in Python, preferably using a name 
similar to "%T".


Most of the usage is in error messages and __repr__, where we don't need 
to worry about compatibility too much.


It should be possible to get the name if you have the type object, but 
not an instance of it. So, the proposed `PyUnicode_FromFormat("%T", 
obj)` is incomplete -- if we go that way, we'll also need a function 
like PyType_GetFullName. Making "%T" work on the type, e.g. 
`PyUnicode_FromFormat("%T", Py_TYPE(obj))`, would be more general.


---

So, I propose adding a "%T" formatter to PyUnicode_FromFormat, to be 
used like this:

PyUnicode_FromFormat("%T", Py_TYPE(obj))
and a "T" format code for type.__format__, to be used like this:
f"{type(obj):T}"
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

[Petr Viktorin]

On 06/20/18 01:53, Jeroen Demeyer wrote:

Hello,

Let me present PEP 579 and PEP 580.

PEP 579 is an informational meta-PEP, listing some of the issues with 
functions/methods implemented in C. The idea is to create several PEPs 
each fix some part of the issues mentioned in PEP 579.


PEP 580 is a standards track PEP to introduce a new "C call" protocol, 
which is an important part of PEP 579. In the reference implementation 
(which is work in progress), this protocol will be used by built-in 
functions and methods. However, it should be used by more classes in the 
future.


You find the texts at
https://www.python.org/dev/peps/pep-0579
https://www.python.org/dev/peps/pep-0580


Hi!
I finally had time to read the PEPs carefully.

Overall, great work! PEP 580 does look complicated, but it's well 
thought out and addresses real problems.


I think the main advantage over the competing PEP 576 is that it's a 
better foundation for solving Cython (and other C-API users) and my PEP 
573 (module state access from methods).



With that, I do have some comments.

The reference to PEP 573 is premature. If PEP 580 is implemented then 
PEP 573 will build on top, and I don't plan to update PEP 573 before 
that. So, I think 580 should be independent. If you agree I can 
summarize rationale for "parent", as much as it concerns 580.


# Using tp_print

The tp_print gimmick is my biggest worry.
AFAIK there's no guarantee that a function pointer and Py_ssize_t are 
the same size. That makes the backwards-compatibility typedef in the 
implementation is quite worrying:

typedef Py_ssize_t printfunc
I can see the benefit for backporting to earlier Python versions, and 
maybe that outweighs worries about exotic architectures, but the PEP 
should at least have more words on why this is not a problem.



# The C Call protocol

I really like the fact that, in the reference implementation, the flags 
are arranged in a way that allows a switch statement to select what to 
call. That should be noted, if only to explain why there's no guarantee 
of compatibility between Python versions.



# Descriptor behavior

I'd say "SHOULD" rather than "MUST" here. The section describes how to 
implement expected/reasonable behavior, but I see no need to limit that.


"if func supports the C call protocol, then func.__set__ must not be 
implemented." -- also, __delete__ should not be implemented, right?.



# Generic API functions

I'm a bit worried about PyCCall_FASTCALL's "kwds" argument accepting a 
dict, which is mutable. I wouldn't mind dropping that capability, but if 
it stays, we need to require that the callable promises to not modify it.


PyCCall_FASTCALL is not a macro, shouldn't it be named PyCCall_FastCall?


# C API functions

The function PyCFunction_GetFlags is, for better or worse, part of the 
stable ABI. We shouldn't just give up on it. I'm fine with documenting 
that it shouldn't be used, but for functions defined using 
PyCFunction_New etc. it should continue behaving as before.
One solution could be to preserve the "definition time" METH_* flags in 
the 0xFFF bits of cc_flags and use the other bits for CCALL_*.



# Stable ABI

The section should repeat that PyCFunction_ClsNew is added to the stable 
ABI (but nothing else).



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

[Petr Viktorin]

On 09/13/18 02:22, Jeroen Demeyer wrote:

On 2018-09-13 02:26, Petr Viktorin wrote:

The reference to PEP 573 is premature.


It seems to me that PEP 580 helps with the use case of PEP 573. In fact, 
it implements part of what PEP 573 proposes. So I don't see the problem 
with the reference to PEP 573. Even if the implementation of PEP 573 
changes, the problem statement will remain and that's what I'm referring 
to. 


If you agree I can
summarize rationale for "parent", as much as it concerns 580.


Sure. I still think that we should refer to PEP 573, but maybe we can 
summarize it also in PEP 580.


I want to make it clear that PEP 580 doesn't depend on 579. Reviewers 
don't need to agree with PEP 579 to accept 580.


Here's my proposed rewording: 
https://github.com/python/peps/pull/775/files?short_path=b34f00e#diff-b34f00eeb75773c32f9b22fd7fee9771



# Using tp_print

The tp_print gimmick is my biggest worry.
AFAIK there's no guarantee that a function pointer and Py_ssize_t are
the same size.


I'm not actually claiming anywhere that it is the same size.


Indeed, I was thinking ahead here. Backporting this to earlier versions 
of CPython will not be completely trivial, but let's leave it to Cython.




# Descriptor behavior

I'd say "SHOULD" rather than "MUST" here. The section describes how to
implement expected/reasonable behavior, but I see no need to limit that.


There *is* actually an important reason to limit it: it allows code to 
make assumptions on what __get__ does. This enables optimizations which 
wouldn't be possible otherwise. If you cannot be sure what __get__ does, 
then you cannot optimize


obj.method(x)

to

type(obj).method(obj, x)


I see now. Yes, that's reasonable.



"if func supports the C call protocol, then func.__set__ must not be
implemented." -- also, __delete__ should not be implemented, right?.


Indeed. I write Python but I think C API, so for me these are both 
really tp_descr_set.



PyCCall_FASTCALL is not a macro, shouldn't it be named PyCCall_FastCall?


What's the convention for that anyway? I assumed that capital letters 
meant a "really know what you are doing" function which could segfault 
if used badly.


Well, I don't think that's a useful distinction either. This is C; 
pretty much anything can segfault when used badly.


Macros tend to be "fast": Py_TYPE just gets a member of a struct; 
Py_INCREF just increments a number. METH_NOARGS is just a number. None 
of them are very dangerous.
IMO, PyTuple_GET_ITEM is not uppercase because it's dangerous, but 
because it just reaches into memory.


For me, whether something is a function or macro is just an 
implementation detail (which can change between Python versions) which 
should not affect the naming.


True. I'm not saying the convention is very strict or useful.



# C API functions

The function PyCFunction_GetFlags is, for better or worse, part of the
stable ABI. We shouldn't just give up on it. I'm fine with documenting
that it shouldn't be used, but for functions defined using
PyCFunction_New etc. it should continue behaving as before.
One solution could be to preserve the "definition time" METH_* flags in
the 0xFFF bits of cc_flags and use the other bits for CCALL_*.


I'm fine with that if you insist. However, it would be a silly solution 
to formally satisfy the "stable ABI" requirement without actually helping.


Yes, it's definitely very silly. But that's not a reason to break our 
promise to the users. After all it's called "stable ABI", not "useful 
ABI" :)


I agree with your other points that I didn't reply to and will make some 
edits to PEP 580.


Thank you!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34595: How to format a type name?

[Petr Viktorin]

On 09/13/18 14:08, Victor Stinner wrote:

Le jeu. 13 sept. 2018 à 16:01, Eric V. Smith  a écrit :

* Add !t conversion to format string


I'm strongly opposed to this. This !t conversion would not be widely
applicable enough to be generally useful, and would need to be exposed
in the f-string and str.format() documentation, even though 99% of
programmers would never need or see it.


(I'm thinking aloud.)

In the Python code base, I found 115 lines using type(obj).__name__
and 228 lines using obj.__class__.__name__.

[...]

"!t" is not a big improvement over ":T" and "type(obj)".


I'm not sure if type(obj) or obj.__class__ should be used, but I can
say that they are different: obj.__class__ can be overriden:

[...]



Moreover, it's also possible to override the "type" symbol in the
global or local scope:

[...]

I don't think either of those are problematic. If you override 
`__class__` or `type`, things will behave weirdly, and that's OK.



One advantage of having a builtin formatter would be to always use
internally the builtin type() function to get the type of an object,
or not use "type()" in the current scope. The second advantage is to
prevent the need of having to decide between type(obj) and
obj.__class__ :-)



raise TypeError(f"must be str, not {obj!t}")

Should be written as:
raise TypeError(f"must be str, not {type(obj)}")

[...]


Do you want to modify str(type) to return a value different than repr(type)?

Or maybe it's just a typo and you wanted to write f"{type(obj):T}"?


Yes, AFAIK that was a typo.


I think "T" is a good idea, but I think you're adding in obj vs
type(obj) just because of the borrowed reference issue in Py_TYPE().
That issue is so much larger than string formatting the type of an
object that it shouldn't be addressed here.


Right, that's a side effect of the discussion on the C API. It seems
like Py_TYPE() has to go in the new C API. Sorry, the rationale is not
written down yet, but Dino convinced me that Py_TYPE() has to go :-)


I'll be happy when we get rid of Py_TYPE and get to use moving garbage 
collectors... but now is not the time.
The API for "%T" should be "give me the type". The best way to do that 
might change in the future.



But at this point, we're bikeshedding. I think all the relevant voices 
have been heard.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Heap-allocated StructSequences

[Petr Viktorin]

On 09/13/18 23:34, Neil Schemenauer wrote:

On 2018-09-04, Eddie Elizondo wrote:

Solution:

   *   Fix the implementation of PyStructSequence_NewType:

The best solution would be to fix the implementation of this
function. This can easily be done by dynamically creating a
PyType_Spec and calling PyType_FromSpec


Hello Eddie,

Thank you for spending time to look into this.  Without studying the
details of your patch, your approach sounds correct to me.  I think
we should be allocating types from the heap and use PyType_FromSpec.
Having static type definitions living in the data segment cause too
many issues.

We have to assess how 3rd party extension modules would be affected
by this change.  Unless it is too hard to do, they should still
compile (perhaps with warnings) after your fix.  Do you know if
that's the case?  Looking at your changes to structseq.c, I can't
tell easily.

In any case, this should go into Victor's pythoncapi fork.  That
fork includes all the C-API cleanup we are hoping to make to CPython
(assuming we can figure out the backwards and forwards compatibility
issues).


Nope, Victor's fork doesn't include all C-API cleanup. There's an older 
long-term effort (PEP-384, PEP-489, the current contenders 576/579/580, 
and PEP-573 for the future). Converting things to use PyType_FromSpec 
falls in there. As long as the old API still works, these changes should 
go in (but they might need a PEP).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580

[Petr Viktorin]

On 10/3/18 2:12 PM, Jeroen Demeyer wrote:

Hello,

I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
titled "The C call protocol". He has co-authored several PEPs (PEP 394, 
PEP 489, PEP 534, PEP 547, PEP 573), several of which involve extension 
modules.


Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
BDFL-Delegate.


I am well aware of the current governance issues, but several people 
have mentioned that the BDFL-Delegate process can still continue for 
now. I created a PR for the peps repository at 
https://github.com/python/peps/pull/797




Hello,
I don't think it's formally possible to do that now, but the following 
from elsewhere in the thread does make sense:


Antoine Pitrou:

Consensus would obviously work (if no-one opposes the proposed person,
then surely we don't need an elaborate governance model to decree that
said person can become the PEP delegate, no?).


Yes, it would feel very silly to have consensus and not be able to act 
on it. But without a governance (and approval process) it's hard to 
*ensure* we have consensus where all relevant voices have been heard.


On the other hand, I don't agree with Nick Coghlan here:

In this case, I'd consider it unlikely for either the PEP delegate appointment 
or any decisions about the PEP itself to be overturned - while it's a complex 
topic that definitely needs to go through the PEP process in order to work out 
the technical details, it isn't especially controversial in its own right (the 
most controversial aspect is whether it needs a new C level slot or not, and 
the PEP should clearly lay out the pros and cons of that)


PEP 580 *is* controversial -- there's the competing PEP 576 by Mark 
Shannon, who hasn't commented on this recently. Either would be an 
improvement, but choosing between them is a hard trade-off.
I'll leave technical stuff to another thread and concentrate on the 
process here.


When I'm happy with the PEP *and* if Mark says he's OK with it, I'll 
post a summary to Python-dev & Discourse with a special focus on the 
cons (e.g. the size increase of classes, which will affect everyone). If 
there are no "-1"s on the PEP itself or on the way it's discussed, let's 
treat PEP 580 as *provisionally* accepted, to be reverted if the new 
governance doesn't ratify it.


If there is no consensus, we'll need to wait for the new governance to 
decide.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dear core-devs

[Petr Viktorin]

On 10/4/18 9:34 AM, Victor Stinner wrote:

Hi,

If IBM wants a better Python support, it would help a lot if IBM pays 
for this development. With money, you can easily find core dev 
contractors. Antoine Pitrou has been paid in the past to enhance Python 
support in Solaris and it worked well.


Michael explicitly said this is a personal effort. IBM or other big 
money is not involved.


Is paying the best way to get features into Python? Does becoming a core 
dev mean you can now get paid for approving changes? Some of the 
implications are quite disturbing :(



Le mercredi 3 octobre 2018, Michael Felt > a écrit :

 >
 >
 > On 10/2/2018 11:34 PM, Terry Reedy wrote:
 >> On 10/2/2018 12:41 PM, Simon Cross wrote:
 >>> Are there any core devs that Michael or Erik could collaborate with?
 >>> Rather than rely on adhoc patch review from random core developers.
 >>
 >> You two might collaborate with each other to the extent of reviewing
 >> some of each other's PRs.
 > Might be difficult. We both, or at least I, claim ignorance of the
 > others platform. I still have a lot of PEP to learn, and my idea of a
 > bug-fix (for Python2) was seen by core-dev as a feature change. I would
 > not feel comfortable trying to mentor someone in things PEP, etc..
 >> That still leaves the issue of merging.
 > How much confidence is there in all the "CI" tests? Does that not offer
 > sufficient confidence for a core-dev to press merge.
 > How about "master" continuing to be what it is, but insert a new
 > "pre-master" branch that the buildbots actually test on (e.g., what is
 > now the 3.X) and have a 3.8 buildbot - for what is now the "master".
 >
 > PR would still be done based on master, but an "initial" merge would be
 > via the pre-master aka 3.X buildbot tests.
 >
 > How "friendly" git is - that it not become such a workload to keep it
 > clean - I cannot say. Still learning to use git. Better, but still do
 > not want to assume it would be easy.
 >
 > My hope is that it would make it easier to consider a "merge" step that
 > gets all the buildbots involved for even broader CI tests.
 >
 >>
 >>> Michael and Eric: Question -- are you interested in becoming core
 >>> developers at least for the purposes of maintaining these platforms in
 >>> future?
 >>
 >> Since adhoc is not working to get merges, I had this same suggestion.
 >> Michael and Erik, I presume you have gotten some guidelines on what
 >> modifications to C code might be accepted, and what concerns people 
have.

 > imho: guidelines - paraphrased - as little as possible :)
 >
 > I have many assumptions, and one of those is that my assumptions are
 > probably incorrect.
 > Goal: have AIX recognized as a Stable platform, even if not in the
 > highest supported category.
 > And that implies, support as far as I am able, to keep it "Stable".
 >>
 >> I think for tests, a separate test_aix.py might be a good idea for
 >> aix-only tests
 > Unclear to me how this would work. Too young in Python I guess (or just
 > a very old dog), but what test would be needed for AIX, or any other
 > platform, that would not need to be tested in some fashion for the
 > 'other' platforms. At a hunch, where there are many platform.system()
 > dependencies expected (e.g., test_posix, maybe doing something in the
 > class definition (is there a "Root" Object/Class that all inherit from.
 > Maybe a (read-only) "root" attribute (or is property better?) could be
 > the value of platform.system(), and iirc, might be used by as @property
 > in unittest. (so, if not in "root" class, then in something like
 > unittest/__init__.py.
 >
 > I hope to be "close" in "Python thinking" - enough that someone who
 > actually knows how the pieces fit together could come with a better, and
 > more appropriate guideline/implementation.
 >
 >> , while modification of other tests might be limited to adding skips.
 >> The idea would be to make it easy to remove aix stuff in the future if
 >> it again became unsupported.
 > IMHO: IBM and AIX do not mention it, but for openstack cloudmanagement
 > (very specifically cloud-init) AIX needs a recognized stable Python
 > implementation. I am "surprised" in the level of communication of IBM
 > with Python community.
 >
 > Personally, I do not see AIX as a specialized platform. Feels more like
 > the "last-standing" fully supported (commercial OEM) 'POSIX-UNIX'. Of
 > course my focus is narrow - so maybe there is a lot of support for
 > commercial platforms such as HPUX, Solaris, and other mainstream UNIXes.
 > Feel free to correct me!!
 >> Ditto for other specialized platforms.
 >>
 >>
 >>
 >>
 >
 > ___
 > Python-Dev mailing list
 > Python-Dev@python.org 
 > https://mail.python.org/mailman/listinfo/python-dev
 > Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

 >

___
Python-Dev mai

Re: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580

[Petr Viktorin]

On 10/4/18 1:38 PM, Łukasz Langa wrote:


On 3 Oct 2018, at 23:10, Terry Reedy <mailto:tjre...@udel.edu>> wrote:


On 10/3/2018 8:12 AM, Jeroen Demeyer wrote:

Hello,
I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
titled "The C call protocol". He has co-authored several PEPs (PEP 
394, PEP 489, PEP 534, PEP 547, PEP 573), several of which involve 
extension modules.
Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
BDFL-Delegate.


To me, three experienced core devs approving of a 4th person as 
PEP-examiner is sufficient to proceed on a CPython implementation 
proposal.  I don't think we need to be paralyzed on this.


What you're saying is sensible, the team is small enough and tightly 
knit that we trust each other. However, trust is not the point. It's 
about clear expectations and avoiding anarchy. As Nick points out 
elsewhere, circumventing the lack of governance by "asking a few 
friends" on the core team creates a need for the new leadership to 
ratify those changes. Speaking frankly, it would be a major shit show if 
any of those changes were to be reverted. As the release manager of this 
version of Python, can I ask you please not to risk this?


Ironically, the governance model I am championing is one that would 
closely resemble what you're describing. A community of experts, no 
kings: https://discuss.python.org/t/pep-8012-the-community-model/156/


So it's really not that I disagree with you, I do. It's not that I don't 
trust Petr, I do. It's that I believe the core team needs to formalize 
how they want the project to proceed *before* they go run approving PEPs.


Łukasz, as the release manager for 3.8 you're the closest we have to an 
authority, so I defer to your judgment. No PEPs can currently be accepted.



Anyway, even if I was a *-delegate here, I would need to hear Mark 
Shannon's opinion on the PEP. Convincing him will probably be harder 
than convincing me.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command

[Petr Viktorin]

On 10/18/18 4:40 PM, Zachary Ware wrote:

On Thu, Oct 18, 2018 at 9:09 AM VanL  wrote:

Hi all,

I am looking into an issue associated with the wininst-*.exe files in the 
distutils/command subdirectory. It looks like these are the executable stubs 
used to create self-extracting zips for installation - but I am not 100% sure. 
It also looks like they include the calls to standard Windows functions to 
display the installer window.

I have a couple questions I need help with:
1) Am I correct about the function, and if not, what are they?


You are correct.  IIUC, they are checked in to allow creating those
installers from non-Windows platforms.


Is that the only reason for them?
At least on Linux, bdist_wininst does not work since at least Python 
3.2, as it tries to use a Windows-only encoding internally.

https://bugs.python.org/issue10945

If they're only there for non-Windows platforms, they're useless.


2) Where did these come from, and where is their source code?


Source can be found here:
https://github.com/python/cpython/tree/master/PC/bdist_wininst

The individual checked-in .exe files were each originally built by
whoever updated the Windows toolchain to a new version of MSVC
(Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
ones were added by Thomas Heller, presumably using whatever the
current toolchain(s) was (were) at the time).  A few of them have been
rebuilt after bug fixes in the source since they were added, mostly by
the same people, though I also see Mark Hammond and Raymond Hettinger
in the history (and Georg Brandl via svnmerge).  I notice that there
are a few very minor code cleanups (the three latest commits here
https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
that have not made it into a rebuilt exe yet.

FTR, we really ought to remove all but the 14.0 version from the
master branch.  We don't support building Python with any toolchain
older than 14.0 anymore, and the older toolchains are nigh impossible
to find anymore anyway.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command

[Petr Viktorin]

On 10/18/18 7:44 PM, VanL wrote:
Primarily for non-windows platforms, but I also think for Windows users 
without any compilers or similar tools installed. There is also some 
discussion of removing some of the older toolchain-specific versions 
(leaving only -14), but that is a subject for another day.


Also I am not sure that bug report applies to 3.5/3.6/3.7.


It does.



On Thu, Oct 18, 2018 at 11:26 AM Petr Viktorin <mailto:encu...@gmail.com>> wrote:


On 10/18/18 4:40 PM, Zachary Ware wrote:
 > On Thu, Oct 18, 2018 at 9:09 AM VanL mailto:van.lindb...@gmail.com>> wrote:
 >> Hi all,
 >>
 >> I am looking into an issue associated with the wininst-*.exe
files in the distutils/command subdirectory. It looks like these are
the executable stubs used to create self-extracting zips for
installation - but I am not 100% sure. It also looks like they
include the calls to standard Windows functions to display the
installer window.
 >>
 >> I have a couple questions I need help with:
 >> 1) Am I correct about the function, and if not, what are they?
 >
 > You are correct.  IIUC, they are checked in to allow creating those
 > installers from non-Windows platforms.

Is that the only reason for them?
At least on Linux, bdist_wininst does not work since at least Python
3.2, as it tries to use a Windows-only encoding internally.
https://bugs.python.org/issue10945

If they're only there for non-Windows platforms, they're useless.

 >> 2) Where did these come from, and where is their source code?
 >
 > Source can be found here:
 > https://github.com/python/cpython/tree/master/PC/bdist_wininst
 >
 > The individual checked-in .exe files were each originally built by
 > whoever updated the Windows toolchain to a new version of MSVC
 > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
 > ones were added by Thomas Heller, presumably using whatever the
 > current toolchain(s) was (were) at the time).  A few of them have
been
 > rebuilt after bug fixes in the source since they were added,
mostly by
 > the same people, though I also see Mark Hammond and Raymond Hettinger
 > in the history (and Georg Brandl via svnmerge).  I notice that there
 > are a few very minor code cleanups (the three latest commits here
 >
https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
 > that have not made it into a rebuilt exe yet.
 >
 > FTR, we really ought to remove all but the 14.0 version from the
 > master branch.  We don't support building Python with any toolchain
 > older than 14.0 anymore, and the older toolchains are nigh impossible
 > to find anymore anyway.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Petr Viktorin]

On 11/19/18 12:14 PM, Victor Stinner wrote:

To design a new C API, I see 3 options:

(1) add more functions to the existing Py_LIMITED_API
(2) "fork" the current public C API: remove functions and hide as much
implementation details as possible
(3) write a new C API from scratch, based on the current C API.
Something like #define newcapi_Object_GetItem PyObject_GetItem"?
Sorry, but "#undef " doesn't work. Only very few
functions are defined using "#define ...".

I dislike (1) because it's too far from what is currently used in
practice. Moreover, I failed to find anyone who can explain me how the
C API is used in the wild, which functions are important or not, what
is the C API, etc.


One big, complex project that now uses the limited API is PySide. They 
do some workarounds, but the limited API works. Here's a writeup of the 
troubles they have with it: 
https://github.com/pyside/pyside2-setup/blob/5.11/sources/shiboken2/libshiboken/pep384impl_doc.rst



I propose (2). We control how much changes we do at each milestone,
and we start from the maximum compatibility with current C API. Each
change can be discussed and experimented to define what is the C API,
what we want, etc. I'm working on this approach for 1 year, that's why
many discussions popped up around specific changes :-)


I hope the new C API will be improvements (and clarifications) of the 
stable ABI, rather than a completely new thing.
My ideal would be that Python 4.0 would keep the same API (with 
questionable things emulated & deprecated), but break *ABI*. The "new C 
API" would become that new stable ABI -- and this time it'd be something 
we'd really want to support, without reservations.


One thing that did not work with the stable ABI was that it's "opt-out"; 
I think we can agree that a new one must be "opt-in" from the start.
I'd also like the "new API" to be a *strict subset* of the stable ABI: 
if a new function needs to be added, it should be added to both.



Some people recently proposed (3) on python-dev. I dislike this option
because it starts by breaking the backward compatibility. It looks
like (1), but worse. The goal and the implementation are unclear to
me.

--

Replacing PyDict_GetItem() (specialized call) with PyObject_Dict()
(generic API) is not part of my short term plan. I wrote it in the
roadmap, but as I wrote before, each change should be discusssed,
experimented, benchmarked, etc.

Victor
Le lun. 19 nov. 2018 à 12:02, M.-A. Lemburg  a écrit :


On 19.11.2018 11:53, Antoine Pitrou wrote:

On Mon, 19 Nov 2018 11:28:46 +0100
Victor Stinner  wrote:

Python internals rely on internals to implement further optimizations,
than modifying an "immutable" tuple, bytes or str object, because you
can do that at the C level. But I'm not sure that I would like 3rd
party extensions to rely on such things.


I'm not even talking about *modifying* tuples or str objects, I'm
talking about *accessing* their value without going through an abstract
API that does slot lookups, indirect function calls and object unboxing.

For example, people may need a fast way to access the UTF-8
representation of a unicode object.  Without making indirect function
calls, and ideally without making a copy of the data either.  How do
you do that using the generic C API?


Something else you need to consider is creating instances of
types, e.g. a tuple. In C you will have to be able to put
values into the data structure before it is passed outside
the function in order to build the tuple.

If you remove this possibility to have to copy data all the
time, losing the advantages of having a rich C API.
  --
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2018)

Python Projects, Coaching and Consulting ...  http://www.egenix.com/
Python Database Interfaces ...   http://products.egenix.com/
Plone/Zope Database Interfaces ...   http://zope.egenix.com/



::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
   http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/py

Re: [Python-Dev] Missing functions [Was: Re: Experiment an opt-in new C API for Python? (leave current API unchanged)]

[Petr Viktorin]

On 11/21/18 4:11 PM, Matěj Cepl wrote:

On 2018-11-21, 14:54 GMT, Benjamin Peterson wrote:

In Python 3, there is no underlying FILE* because the io
module is implemented using fds directly rather than C stdio.


OK, so the proper solution is to kill all functions which expect
FILE


Indeed.
This has another side to it: there are file-like objects that aren't 
backed by FILE*. In most case, being a "real" file is an unnecessary 
distinction, like that between the old `int` vs. `long`. "Fits in the 
machine register" is a detail from a level below Python, and so is "the 
kernel treats this as a file".


Of course, this is not how C libraries work -- so, sadly, it makes 
wrappers harder to write. And a perfect solution might require adding 
more generic I/O to the C library.




and if you are anal retentive about stability of API, then
you have to fake it by creating FILE structure around the
underlying fd handler as I did in M2Crypto, right?


Yes, AFAIK that is the least bad solution.
I did something very similar here: 
https://github.com/encukou/py3c/blob/master/include/py3c/fileshim.h


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

[Petr Viktorin]

On 1/23/19 3:33 AM, Stephan Reiter wrote:

Thanks for the answers so far. I appreciate them!

Nathaniel, I'd like to allow Python plugins in my application. A
plugin should be allowed to bring its own modules along (i.e.
plugin-specific subdir is in sys.path when the plugin is active) and
hence some isolation of them will be needed, so that they can use
different versions of a given module. That's my main motivation for
using subinterpreters.
I thought about running plugins out-of-processes - a separate process
for every plugin - and allow them to communicate with my application
via RPC. But that makes it more complex to implement the API my
application will offer and will slow down things due to the need to
copy data.
Maybe you have another idea for me? :)


Try to make the plugins work together. Look into using pip/PyPI for your 
plugins. Try to make it so each package ("plugin") would have only one 
module/package, and dependencies would be other packages that can be 
installed individually and shared. And keep in mind you can set up your 
own package index, or distribute/install individual package files.


If that's not possible, and you want things to work now, go with subprocess.

If you want to help make subinterpreters work better, there are several 
people scratching at the problem from different angles. Most/all would 
welcome help, but don't expect any short-term benefits.
(FWIW, my own effort is currently blocked on PEP 580, and I hope to move 
forward after a Council is elected.)




Henry, Antoine, thanks for your input; I'll check out the tests and
see what I can learn from issue 10915.

Stephan

Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith :


There are currently numerous incompatibilities between numpy and 
subinterpreters, and no concrete plan for fixing them. The numpy team does not 
consider subinterpreters to be a supported configuration, and can't help you 
with any issues you run into. I know the concept of subinterpreters is really 
appealing, but unfortunately the CPython implementation is not really mature or 
widely supported... are you absolutely certain you need to use subinterpreters 
for your application?

On Tue, Jan 22, 2019, 08:27 Stephan Reiter 

Hi all!

I am new to the list and arriving with a concrete problem that I'd
like to fix myself.

I am embedding Python (3.6) into my C++ application and I would like
to run Python scripts isolated from each other using sub-interpreters.
I am not using threads; everything is supposed to run in the
application's main thread.

I noticed that if I create an interpreter, switch to it and execute
code that imports numpy (1.13), my application will hang.

   ntdll.dll!NtWaitForSingleObject() Unknown
   KernelBase.dll!WaitForSingleObjectEx() Unknown

python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x748a67a0, 
_RTL_CRITICAL_SECTION * cs=0x748a6778, unsigned long ms=5) Line 245 C

   [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
   python36.dll!take_gil(_ts * tstate=0x023251cbc260) Line 224 C
   python36.dll!PyEval_RestoreThread(_ts * tstate=0x023251cbc260) Line 370 C
   python36.dll!PyGILState_Ensure() Line 855 C
   umath.cp36-win_amd64.pyd!7ff8c6306ab2() Unknown
   umath.cp36-win_amd64.pyd!7ff8c630723c() Unknown
   umath.cp36-win_amd64.pyd!7ff8c6303a1d() Unknown
   umath.cp36-win_amd64.pyd!7ff8c63077c0() Unknown
   umath.cp36-win_amd64.pyd!7ff8c62ff926() Unknown
   [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
   [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
   python36.dll!call_function(_object * * *
pp_stack=0x0048be5f5e40, __int64 oparg, _object * kwnames) Line
4822 C

Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
PyEval_RestoreThread on the (auto) threadstate of the main
interpreter. And that's wrong.
We are already holding the GIL with the threadstate of our current
sub-interpreter, so there's no need to switch.

I know that the GIL API is not fully compatible with sub-interpreters,
as issues #10915 and #15751 illustrate.

But since I need to support calls to PyGILState_Ensure - numpy is the
best example -, I am trying to improve the situation here:
https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597

That change may be naive, but it does the trick for my use case. If
totally wrong, I don't mind pursuing another alley.

Essentially, I'd like to ask for some guidance in how to tackle this
problem while keeping the current GIL API unchanged (to avoid breaking
modules).

I am also wondering how I can test any changes I am proposing. Is
there a test suite for interpreters, for example?

Thank you very much,
Stephan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.co

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/6/19 8:43 AM, Stephane Wirtel wrote:

On 02/05, Barry Warsaw wrote:

On Feb 5, 2019, at 02:24, Stephane Wirtel  wrote:
You’re welcome!  I just pushed an update to add 3.8.0a1 to the set of 
Python’s (including git head).  Do you think there’s a better way to 
publicize these images?


I know that Julien Palard wanted a docker image with all the versions of
Python, see: https://github.com/docker-library/python/issues/373

For my part, I wanted to propose a docker image with the last version of
Python and try to use it for the detection of bugs in the main python
projects (django, numpy, flask, pandas, etc...) with a CI (example:
Gitlab-CI)

First issue: pytest uses the ast module of python and since 3.8.0a1, the
tests do not pass -> new issue for pytest


FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
alphas/betas, so everything's tested when 3.8.0 is released: 
https://fedoraproject.org/wiki/Changes/Python3.8


That should cover the main Python projects, too.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/6/19 2:26 PM, Matthias Klose wrote:

On 06.02.19 13:23, Petr Viktorin wrote:

FWIW, we're preparing to rebuild all Fedora packages with the 3.8 alphas/betas,
so everything's tested when 3.8.0 is released:
https://fedoraproject.org/wiki/Changes/Python3.8

That should cover the main Python projects, too.


well, the real challenge is that all test suites of third party packages still
pass on all architectures.  From past transitions, I know that this costs the
most time and resources. 


Same experience here.

In Fedora, tests are generally run as part of the build.
(Sorry, that was definitely not obvious from my message!)


But yes, targeting 3.8 for Ubuntu 20.04 LTS as well.


\o/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/7/19 5:16 PM, Stephane Wirtel wrote:

On 02/06, Petr Viktorin wrote:

On 2/6/19 8:43 AM, Stephane Wirtel wrote:

On 02/05, Barry Warsaw wrote:

On Feb 5, 2019, at 02:24, Stephane Wirtel  wrote:
You’re welcome!  I just pushed an update to add 3.8.0a1 to the set 
of Python’s (including git head).  Do you think there’s a better way 
to publicize these images?


I know that Julien Palard wanted a docker image with all the versions of
Python, see: https://github.com/docker-library/python/issues/373

For my part, I wanted to propose a docker image with the last version of
Python and try to use it for the detection of bugs in the main python
projects (django, numpy, flask, pandas, etc...) with a CI (example:
Gitlab-CI)

First issue: pytest uses the ast module of python and since 3.8.0a1, the
tests do not pass -> new issue for pytest


FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
alphas/betas, so everything's tested when 3.8.0 is released: 
https://fedoraproject.org/wiki/Changes/Python3.8

Hi Petr,

Will you execute the tests of these packages?


It's best practice to include the test suite in Fedora packages.
Sometimes it's not – e.g. if the tests need network access, or all extra 
testing dependencies aren't available, or most frequently, the 
maintainer is just lazy.
If you have a specific package in mind, I can check. Currently django & 
numpy get tested; flask & pandas don't.


For 3.7, we did the rebuild much later in the cycle. The builds 
themselves caught async/await SyntaxErrors, and tests caught a lot of 
StopIteration leaking. At the time it felt like no one really knew what 
porting to 3.7.0 would look like – similar to how people didn't think 
"unicode" would be a big problem in py3k. That's what we're trying to 
avoid for 3.8.0.



I have a small discussion with Julien Palard and I wanted to create a
small CI where I will execute the tests of the updated packages from
the RSS feed of PyPI.

The first one was pytest 


That sounds exciting! Something like that is on my "interesting possible 
projects" list, but alas, not at the top :(


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

PEP 394 says:

> This recommendation will be periodically reviewed over the next few
> years, and updated when the core development team judges it
> appropriate. As a point of reference, regular maintenance releases
> for the Python 2.7 series will continue until at least 2020.

I think it's time for another review.
I'm especially worried about the implication of these:

- If the `python` command is installed, it should invoke the same
  version of Python as the `python2` command
- scripts that are deliberately written to be source compatible
  with both Python 2.x and 3.x [...] may continue to use `python` on
  their shebang line.

So, to support scripts that adhere to the recommendation, Python 2
needs to be installed :(


Please see this PR for details and a suggested change: 
https://github.com/python/peps/pull/893

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/13/19 4:46 PM, Antoine Pitrou wrote:

On Wed, 13 Feb 2019 16:24:48 +0100
Petr Viktorin  wrote:

PEP 394 says:

  > This recommendation will be periodically reviewed over the next few
  > years, and updated when the core development team judges it
  > appropriate. As a point of reference, regular maintenance releases
  > for the Python 2.7 series will continue until at least 2020.

I think it's time for another review.
I'm especially worried about the implication of these:

- If the `python` command is installed, it should invoke the same
version of Python as the `python2` command
- scripts that are deliberately written to be source compatible
with both Python 2.x and 3.x [...] may continue to use `python` on
their shebang line.

So, to support scripts that adhere to the recommendation, Python 2
needs to be installed :(


I think PEP 394 should acknowledge that there are now years of
established usage of `python` as Python 3 for many conda users.


The intention is that Conda environments are treated the same as venv 
environments, i.e.:


When a virtual environment (created by the PEP 405 venv package or a 
similar tool) is active, the python command should refer to the virtual 
environment's interpreter. In other words, activating a virtual 
environment counts as deliberate user action to change the default 
python interpreter.



Do you think conda should be listed explicitly along with venv?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/13/19 4:24 PM, Petr Viktorin wrote:

I think it's time for another review.

[...]
Please see this PR for details and a suggested change: 
https://github.com/python/peps/pull/893


Summary of the thread so far.

Antoine Pitrou noted that the PEP should acknowledge that there are now 
years of
established usage of `python` as Python 3 for many conda users, often as 
the "main" Python.


Victor Stinner expressed support for "python" being the latest Python 
version, citing PHP, Ruby, Perl; containers; mentions of "python" in our 
docs.


Steve Dower later proposed concrete points how to make "python" the 
default command:

  * our docs should say "python" consistently
  * we should recommend that distributors use the same workaround
  * our docs should describe the recommended workaround in any places 
people are likely to first encounter it (tutorial, sys.executable, etc.)


Chris Barker added that "python3" should still be available, even if 
"python" is default.


Barry Warsaw gave a +1 to making "python" default, noting that there 
were plans to change this when Python 2 is officially deprecated. But 
distros need to make decisions about 2020 now.


Chris Barker noted that users won't see any discuntinuity in 2020. 
That's just a date support from CPython devs ends.


Victor pointed to discussions on 4.0 vs. 3.10. (I'll ignore discussions 
on 4.0 in this summary.)

Victor also posted some interesting info and links on Fedora and RHEL.
There was a discussion on the PSF survey about how many people use 
Python 3. (I'll ignore this sub-thread, it's not really about the 
"python" command.)


Steve noted that the Windows Store package of Python 3 provides 
"python", but he is still thinking how to make this reasonable/reliable 
in the full installer.


Several people think "py" on Unix would be a good thing. Neil 
Schemenauer supposes we would encourage people to use it over 
"python"/"python2"/"python3", so "python" would be less of an issue.


Neil Schemenauer is not opposed to making "python" configurable or 
eventually pointing it to Python 3.


Jason Swails shared experience from running software with a 
"#!/usr/bin/env python" shebang on a system that didn't have Python 2 
(and followed the PEP, so no "python" either). The workaround was ugly.


-

Since this list is public, I'd like to remind all readers that it is 
full of people who work extensively with Python 3, and tend to drive it 
forward at any opportunity. (Myself included, but on this thread I'll 
leave the arguments to someone else – they're covered adequately.)


Thoughts of Python developers are important, but we're not hearing any 
other voices. Perhaps everyone with a different opinion has already 
self-selected out.


I don't know of a good place for this discussion, and I'm not a good 
person to give arguments to support the original "python" should be 
Python 2 direction. (But if I did, I imagine posting them here would 
feel a bit scary.)
But I would not be surprised, or annoyed, if the Council had a private 
discussion and pronounced "No, sorry, not yet".


Anyway, how should this be decided? Where should it be discussed?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/13/19 5:45 PM, Victor Stinner wrote:

Some more context about Petr's change, Fedora, RHEL and Red Hat.

[...]

Fedora could switch "python" to Python 3 now*, if the PEP changes to 
allow it.


* "now" has a release date of around October 2019. The next release 
after that should then be around May 2020.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/14/19 9:56 AM, Petr Viktorin wrote:

On 2/13/19 4:24 PM, Petr Viktorin wrote:

I think it's time for another review.

[...]
Please see this PR for details and a suggested change: 
https://github.com/python/peps/pull/893


Summary of the thread so far.

Antoine Pitrou noted that the PEP should acknowledge that there are now 
years of
established usage of `python` as Python 3 for many conda users, often as 
the "main" Python.


Victor Stinner expressed support for "python" being the latest Python 
version, citing PHP, Ruby, Perl; containers; mentions of "python" in our 
docs.


Steve Dower later proposed concrete points how to make "python" the 
default command:

   * our docs should say "python" consistently
   * we should recommend that distributors use the same workaround
   * our docs should describe the recommended workaround in any places 
people are likely to first encounter it (tutorial, sys.executable, etc.)


Chris Barker added that "python3" should still be available, even if 
"python" is default.


Barry Warsaw gave a +1 to making "python" default, noting that there 
were plans to change this when Python 2 is officially deprecated. But 
distros need to make decisions about 2020 now.


Chris Barker noted that users won't see any discuntinuity in 2020. 
That's just a date support from CPython devs ends.


Victor pointed to discussions on 4.0 vs. 3.10. (I'll ignore discussions 
on 4.0 in this summary.)

Victor also posted some interesting info and links on Fedora and RHEL.
There was a discussion on the PSF survey about how many people use 
Python 3. (I'll ignore this sub-thread, it's not really about the 
"python" command.)


Steve noted that the Windows Store package of Python 3 provides 
"python", but he is still thinking how to make this reasonable/reliable 
in the full installer.


Several people think "py" on Unix would be a good thing. Neil 
Schemenauer supposes we would encourage people to use it over 
"python"/"python2"/"python3", so "python" would be less of an issue.


Neil Schemenauer is not opposed to making "python" configurable or 
eventually pointing it to Python 3.


Jason Swails shared experience from running software with a 
"#!/usr/bin/env python" shebang on a system that didn't have Python 2 
(and followed the PEP, so no "python" either). The workaround was ugly.


There haven't been many new ideas since this summary – mostly it was 
explaining and re-hashing what's been mentioned before.
Matthias Klose pointed out some Debian/Ubuntu points, to which I'll add 
the situation in other distros I know of.


*Debian* is concerned that python → python3 will break software
after an upgrade. Debian appears to not want to ship the unversioned 
command after py2 removal.


For *Ubuntu*, Matthias is not sure if he wants a python executable at 
all. He notes that pypi.org recommends pip, and pip still breaks 
system-installed packages when asked to.


For both Ubuntu 20.04 LTS and Debian bullseye, the goal is that distro 
packages don't use the unversioned shebang.


*Fedora* packages don't use the unversioned shebang. If it was up to me, 
the unversioned command would be removed in F31 (released in the first 
half of 2019) and then pointed to python3 in F32 (second half). But we'd 
rather happy to follow upstream consensus. (And the PEP, if it reflects 
the consensus.)


In *RHEL*, the unversioned command is missing by default. Sysadmins can 
change it, but are advised to use python2/python3 instead. RHEL decision 
makers don't give the PEP much weight.


*Arch* did the switch to python3 a long time ago (and the resulting fire 
wasn't all that bright).


With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if 
installed) otherwise the macOS system Python. That's exactly according 
to the PEP. They tried to switch python to python3 before, and got 
rather nasty backlash citing PEP 394. I assume they will follow the PEP 
quite strictly from now on.


The *macOS* system Python is out of our hands; Apple ignores upstream 
recommendations.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Petr Viktorin]

On 2/26/19 6:54 PM, Barry Warsaw wrote:

There haven't been many new ideas since this summary – mostly it was explaining 
and re-hashing what's been mentioned before.


Thanks for the summary Petr.

Here’s another way to think about the problem.  I know Nick and I have talked 
about this before, but I don’t think any distros have actually done this, 
though I’ve been out of that business a while now so correct me if I’m wrong.

I see this question as having several parts, and the conflation of them is part 
of the reason why the unversioned `python` command is so problematic.  Python 
is used for:

* OS functionality
* to run applications that aren’t critical to the OS but are delivered on the OS
* as the entry point to the interactive interpreter
* to run applications written and deployed on the OS but completely outside of 
it

Which `python` are we trying to change?  All of them?

For OS functionality, there should probably be a separate command not conflated 
with /usr/bin/python.  The OS can make any adjustments it needs, calling it 
`spython` (as I think Nick once suggested), or whatever.  Nobody but OS 
maintainers cares what this is called or what version of Python it exposes.


Yup. RHEL 8 actually has exactly that. (It's called 
/usr/libexec/platform-python; please don't use it!)
Fedora (and most other distros) makes this the same as the interpreter 
for other packaged software. For Fedora the main reason is that don't 
want to maintain two full separate Python stacks.



I strongly believe that (eventually) the interactive interpreter command should 
be /usr/bin/python and that this should point to Python 3, since this provides 
the best experience for beginners, dabblers, etc.


+1


So what about the other two use cases?  Well, for applications packages within 
the OS but aren’t critical to it, I think they should always use the versioned 
shebang, never the unversioned shebang.  Distros can control this, so that 
transition should be easier.


+1


The tricky part then seems to me what to do for 3rd parties which are using the 
distro Python in their shebangs?  Nobody sees their code but them, and changing 
the shebang out from under them could cause their code to break.  But don’t 
they already take lots of precautions and planning for any OS upgrade?  
Changing the shebang for Python 2 would be just one of the things they’d have 
to worry about in an OS upgrade.


Also, things will break for them anyway, it's just a matter of time. 
Python 2 *is* going away, eventually. (Right?)
I don't think we're doing that many people a favor by keeping 
/usr/bin/python → python2 around. Instead, we're *hiding* the problem 
from them. Too many people think python2 is still the "default".


Making /usr/bin/python be missing for some time, rather than pointing it 
to python3 now, is the more explicit way to do the transition.



I don’t know whether this analysis is complete or correct, but perhaps it helps 
inform a way forward on PEP 394.


I have two very different questions in mind for moving this forward.

Who gets to decide on PEP 394 changes?
Since so many people on python-dev are in agreement, where do I go for 
opposing voices?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Petr Viktorin]

On 2/28/19 6:56 PM, Gregory P. Smith wrote:


On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi > wrote:



On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
mailto:nas-pyt...@arctrix.com>> wrote:

On 2019-02-26, Gregory P. Smith wrote:
 > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
mailto:ba...@python.org>> wrote:
 > For an OS distro provided interpreter, being able to restrict
its use to
 > only OS distro provided software would be ideal (so ideal
that people who
 > haven't learned the hard distro maintenance lessons may hate
me for it).

This idea has some definite problems.  I think enforcing it via
convention is about as much as would be good to do.  Anything more
and you make it hard for people who really need to use the vendor
provided interpreter from being able to do so.

Why might someone need to use the distro provided interpreter?

* Vendor provides some python modules in their system packages which
are not installable from pip (possibly even a proprietary extension
module, so not even buildable from source or copyable from the
system location) which the end user needs to use to do something to
their system.
* End user writes a python module which is a plugin to a system tool
which has to be installed into the system python to from which that
system tool runs.  The user then wants to write a script which uses
the system tool with the plugin in order to do something to their
system outside of the system tool (perhaps the system tool is
GUI-driven and the user wants to automate a part of it via the
python module).  They need their script to use the system python so
that they are using the same code as the system tool itself would use.

There's probably other scenarios where the benefits of locking the
user out of the system python outweigh the benefits but these are
the ones that I've run across lately.


Agreed.  The convention approach as someone said RHEL 8 has apparently 
done with an os distro reserved interpreter (yay) is likely good enough 
for most situations.


I'd go a /little/ further than that and suggest such an os distro 
reserved interpreter attempt to prevent installation of packages (ie: 
remove pip/ensurepip/distutils) via any other means than the OS package 
manager (rpms, debs).  Obviously that can't actually prevent someone 
from figuring out how to run getpip or manually installing trees of 
packages within its sys.path, but it acts as a deterrent suggesting that 
this interpreter is not intended for arbitrary software installation.


Currently, in RHEL/Fedora:
- "sudo pip" installs to /usr/local/, RPM packages install to /usr/
- with "-I", the interpreter does not look into /usr/local/.
AFAIK, Debian/Ubuntu have something very similar, and were first to do it.

What remains to tie this together is making "platform-python" always run 
with -I. This is PEP 432's "system-python" use case / design goal.
(The RHEL/Fedora platform-python was initially called system-python. We 
renamed to it so it doesn't clash with the PEP. It's the same use case, 
but we don't want to assign semantics to the name prematurely. 
Cutrrently, system-python is waiting for the larger-scale restructuring, 
and hopefully wider/longer discussion.)


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL-Delegate appointments for several PEPs

[Petr Viktorin]

On Sun, Mar 24, 2019 at 4:22 PM Mark Shannon  wrote:
>
> Hi Petr,
>
> Regarding PEPs 576 and 580.
> Over the new year, I did a thorough analysis of possible approaches to
> possible calling conventions for use in the CPython ecosystems and came
> up with a new PEP.
> The draft can be found here:
> https://github.com/markshannon/peps/blob/new-calling-convention/pep-.rst
>
> I was hoping to profile a branch with the various experimental changes
> cherry-picked together, but don't seemed to have found the time :(
>
> I'd like to have a testable branch, before formally submitting the PEP,
> but I'd thought you should be aware of the PEP.
>
> Cheers,
> Mark.

Hello Mark,
Thank you for letting me know! I wish I knew of this back in January,
when you committed the first draft. This is unfair to the competing
PEP, which is ready and was waiting for the new govenance. We have
lost three months that could be spent pondering the ideas in the
pre-PEP.
Do you think you will find the time to piece things together? Is there
anything that you already know should be changed?

Do you have any comments on [Jeroen's comparison]?

The pre-PEP is simpler then PEP 580, because it solves simpler issues.
I'll need to confirm that it won't paint us into a corner -- that
there's a way to address all the issues in PEP 579 in the future.

The pre-PEP claims speedups of 2% in initial experiments, with
expected overall performance gain of 4% for the standard benchmark
suite. That's pretty big.
As far as I can see, PEP 580 claims not much improvement in CPython,
but rather large improvements for extensions (Mistune with Cython).

The pre-PEP has a complication around offsetting arguments by 1 to
allow bound methods forward calls cheaply. I fear that this optimizes
for current usage with its limitations.
PEP 580's cc_parent allows bound methods to have access to the class,
and through that, the module object where they are defined and the
corresponding module state. To support this, vector calls would need a
two-argument offset.
(That seems to illustrate the main difference between the motivations
of the two PEPs: one focuses on extensibility; the other on optimizing
existing use cases.)

The pre-PEP's "any third-party class implementing the new call
interface will not be usable as a base class" looks quite limiting.



[Jeroen's comparison]:
https://mail.python.org/pipermail/python-dev/2018-July/154238.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 3/30/19 11:36 PM, Jeroen Demeyer wrote:

On 2019-03-30 17:30, Mark Shannon wrote:

2. The claim that PEP 580 allows "certain optimizations because other
code can make assumptions" is flawed. In general, the caller cannot make
assumptions about the callee or vice-versa. Python is a dynamic language.


PEP 580 is meant for extension classes, not Python classes. Extension 
classes are not dynamic. When you implement tp_call in a given way, the 
user cannot change it. So if a class implements the C call protocol or 
the vectorcall protocol, callers can make assumptions about what that 
means.



PEP 579 is mainly a list of supposed flaws with the
'builtin_function_or_method' class.
The general thrust of PEP 579 seems to be that builtin-functions and
builtin-methods should be more flexible and extensible than they are. I
don't agree. If you want different behaviour, then use a different
object. Don't try an cram all this extra behaviour into a pre-existing
object.


I think that there is a misunderstanding here. I fully agree with the 
"use a different object" solution. This isn't a new solution: it's 
already possible to implement those different objects (Cython does it). 
It's just that this solution comes at a performance cost and that's what 
we want to avoid.


It does seem like there is some misunderstanding.

PEP 580 defines a CCall structure, which includes the function pointer, 
flags, "self" and "parent". Like the current implementation, it has 
various METH_ flags for various C signatures. When called, the info from 
CCall is matched up (in relatively complex ways) to what the C function 
expects.


PEP 590 only adds the "vectorcall". It does away with flags and only has 
one C signatures, which is designed to fit all the existing ones, and is 
well optimized. Storing the "self"/"parent", and making sure they're 
passed to the C function is the responsibility of the callable object.
There's an optimization for "self" (offsetting using 
PY_VECTORCALL_ARGUMENTS_OFFSET), and any supporting info can be provided 
as part of "self".



I'll reiterate that PEP 590 is more general than PEP 580 and that once
the callable's code has access to the callable object (as both PEPs
allow) then anything is possible. You can't can get more extensible than
that.


Anything is possible, but if one of the possibilities becomes common and 
useful, PEP 590 would make it hard to optimize for it.
Python has grown many "METH_*" signatures over the years as we found 
more things that need to be passed to callables. Why would 
"METH_VECTORCALL" be the last? If it won't (if you think about it as one 
more way to call functions), then dedicating a tp_* slot to it sounds 
quite expensive.



In one of the ways to call C functions in PEP 580, the function gets 
access to:

- the arguments,
- "self", the object
- the class that the method was found in (which is not necessarily 
type(self))
I still have to read the details, but when combined with 
LOAD_METHOD/CALL_METHOD optimization (avoiding creation of a "bound 
method" object), it seems impossible to do this efficiently with just 
the callable's code and callable's object.



I would argue the opposite: PEP 590 defines a fixed protocol that is not 
easy to extend. PEP 580 on the other hand uses a new data structure 
PyCCallDef which could easily be extended in the future (this will 
intentionally never be part of the stable ABI, so we can do that).


I have also argued before that the generality of PEP 590 is a bad thing 
rather than a good thing: by defining a more rigid protocol as in PEP 
580, more optimizations are possible.



PEP 580 has the same limitation for the same reasons. The limitation is
necessary for correctness if an object supports calls via `__call__` and
through another calling convention.


I don't think that this limitation is needed in either PEP. As I 
explained at the top of this email, it can easily be solved by not using 
the protocol for Python classes. What is wrong with my proposal in PEP 
580: https://www.python.org/dev/peps/pep-0580/#inheritance



I'll add Jeroen's notes from the review of the proposed PEP 590
(https://github.com/python/peps/pull/960):

The statement "PEP 580 is specifically targetted at function-like 
objects, and doesn't support other callables like classes, partial 
functions, or proxies" is factually false. The motivation for PEP 580 is 
certainly function/method-like objects but it's a general protocol that 
every class can implement. For certain classes, it may not be easy or 
desirable to do that but it's always possible.


Given that `PY_METHOD_DESCRIPTOR` is a flag for tp_flags, shouldn't it 
be called `Py_TPFLAGS_METHOD_DESCRIPTOR` or something?


Py_TPFLAGS_HAVE_VECTOR_CALL should be Py_TPFLAGS_HAVE_VECTORCALL, to be 
consistent with tp_vectorcall_offset and other uses of "vectorcall" (not 
"vector call")



And mine, so far:

I'm not clear on the constness of the "args" array.
If it is mutable (PyObject **), you ca

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

Hello!
I've had time for a more thorough reading of PEP 590 and the reference 
implementation. Thank you for the work!
Overall, I like PEP 590's direction. I'd now describe the fundamental 
difference between PEP 580 and PEP 590 as:

- PEP 580 tries to optimize all existing calling conventions
- PEP 590 tries to optimize (and expose) the most general calling 
convention (i.e. fastcall)


PEP 580 also does a number of other things, as listed in PEP 579. But I 
think PEP 590 does not block future PEPs for the other items.
On the other hand, PEP 580 has a much more mature implementation -- and 
that's where it picked up real-world complexity.


PEP 590's METH_VECTORCALL is designed to handle all existing use cases, 
rather than mirroring the existing METH_* varieties.
But both PEPs require the callable's code to be modified, so requiring 
it to switch calling conventions shouldn't be a problem.


Jeroen's analysis from 
https://mail.python.org/pipermail/python-dev/2018-July/154238.html seems 
to miss a step at the top:


a. CALL_FUNCTION* / CALL_METHOD opcode
  calls
b. _PyObject_FastCallKeywords()
  which calls
c. _PyCFunction_FastCallKeywords()
  which calls
d. _PyMethodDef_RawFastCallKeywords()
  which calls
e. the actual C function (*ml_meth)()

I think it's more useful to say that both PEPs bridge a->e (via 
_Py_VectorCall or PyCCall_Call).



PEP 590 is built on a simple idea, formalizing fastcall. But it is 
complicated by PY_VECTORCALL_ARGUMENTS_OFFSET and 
Py_TPFLAGS_METHOD_DESCRIPTOR.
As far as I understand, both are there to avoid intermediate 
bound-method object for LOAD_METHOD/CALL_METHOD. (They do try to be 
general, but I don't see any other use case.)

Is that right?
(I'm running out of time today, but I'll write more on why I'm asking, 
and on the case I called "impossible" (while avoiding creation of a 
"bound method" object), later.)



The way `const` is handled in the function signatures strikes me as too 
fragile for public API.
I'd like if, as much as possible, PY_VECTORCALL_ARGUMENTS_OFFSET was 
treated as a special optimization that extension authors can either opt 
in to, or blissfully ignore.

That might mean:
- vectorcall, PyObject_VectorCallWithCallable, PyObject_VectorCall, 
PyCall_MakeTpCall all formally take "PyObject *const *args"
- a naïve callee must do "nargs &= ~PY_VECTORCALL_ARGUMENTS_OFFSET" 
(maybe spelled as "nargs &= PY_VECTORCALL_NARGS_MASK"), but otherwise 
writes compiler-enforced const-correct code.
- if PY_VECTORCALL_ARGUMENTS_OFFSET is set, the callee may modify 
"args[-1]" (and only that, and after the author has read the docs).



Another point I'd like some discussion on is that vectorcall function 
pointer is per-instance. It looks this is only useful for type objects, 
but it will add a pointer to every new-style callable object (including 
functions). That seems wasteful.
Why not have a per-type pointer, and for types that need it (like 
PyTypeObject), make it dispatch to an instance-specific function?



Minor things:
- "Continued prohibition of callable classes as base classes" -- this 
section reads as a final. Would you be OK wording this as something 
other PEPs can tackle?
- "PyObject_VectorCall" -- this looks extraneous, and the reference 
imlementation doesn't need it so far. Can it be removed, or justified?
- METH_VECTORCALL is *not* strictly "equivalent to the currently 
undocumented METH_FASTCALL | METH_KEYWORD flags" (it has the 
ARGUMENTS_OFFSET complication).
- I'd like to officially call this PEP "Vectorcall", see 
https://github.com/python/peps/pull/984




Mark, what are your plans for next steps with PEP 590? If a volunteer 
wanted to help you push this forward, what would be the best thing to 
work on?


Jeroen, is there something in PEPs 579/580 that PEP 590 blocks, or 
should address?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 4/11/19 1:05 AM, Jeroen Demeyer wrote:

On 2019-04-10 18:25, Petr Viktorin wrote:

Hello!
I've had time for a more thorough reading of PEP 590 and the reference
implementation. Thank you for the work!


And thank you for the review!


One general note: I am not (yet) choosing between PEP 580 and PEP 590.
I am not looking for arguments for/against whole PEPs, but individual 
ideas which, I believe, can still be mixed & matched.


I see the situation this way:
- I get about one day per week when I can properly concentrate on 
CPython. It's frustrating to be the bottleneck.
- Jeroen has time, but it would frustrating to work on something that 
will later be discarded, and it's frustrating to not be able to move the 
project forward.
- Mark has good ideas, but seems to lack the time to polish them, or 
even test out if they are good. It is probably frustrating to see 
unpolished ideas rejected.


I'm looking for ways to reduce the frustration, given where we are.


Jeroen, thank you for the comments. Apologies for not having the time to 
reply to all of them properly right now.


Mark, if you could find the time to answer (even just a few of the 
points), it would be great. I ask you to share/clarify your thoughts, 
not defend your PEP.




I'd now describe the fundamental
difference between PEP 580 and PEP 590 as:
- PEP 580 tries to optimize all existing calling conventions
- PEP 590 tries to optimize (and expose) the most general calling
convention (i.e. fastcall)


And PEP 580 has better performance overall, even for METH_FASTCALL. See 
this thread:

https://mail.python.org/pipermail/python-dev/2019-April/156954.html

Since these PEPs are all about performance, I consider this a very 
relevant argument in favor of PEP 580.



PEP 580 also does a number of other things, as listed in PEP 579. But I
think PEP 590 does not block future PEPs for the other items.
On the other hand, PEP 580 has a much more mature implementation -- and
that's where it picked up real-world complexity.

About complexity, please read what I wrote in
https://mail.python.org/pipermail/python-dev/2019-March/156853.html

I claim that the complexity in the protocol of PEP 580 is a good thing, 
as it removes complexity from other places, in particular from the users 
of the protocol (better have a complex protocol that's simple to use, 
rather than a simple protocol that's complex to use).


Sadly, I need more time on this than I have today; I'll get back to it 
next week.


As a more concrete example of the simplicity that PEP 580 could bring, 
CPython currently has 2 classes for bound methods implemented in C:

- "builtin_function_or_method" for normal C methods
- "method-descriptor" for slot wrappers like __eq__ or __add__

With PEP 590, these classes would need to stay separate to get maximal 
performance. With PEP 580, just one class for bound methods would be 
sufficient and there wouldn't be any performance loss. And this extends 
to custom third-party function/method classes, for example as 
implemented by Cython.



PEP 590's METH_VECTORCALL is designed to handle all existing use cases,
rather than mirroring the existing METH_* varieties.
But both PEPs require the callable's code to be modified, so requiring
it to switch calling conventions shouldn't be a problem.


Agreed.


Jeroen's analysis from
https://mail.python.org/pipermail/python-dev/2018-July/154238.html seems
to miss a step at the top:

a. CALL_FUNCTION* / CALL_METHOD opcode
   calls
b. _PyObject_FastCallKeywords()
   which calls
c. _PyCFunction_FastCallKeywords()
   which calls
d. _PyMethodDef_RawFastCallKeywords()
   which calls
e. the actual C function (*ml_meth)()

I think it's more useful to say that both PEPs bridge a->e (via
_Py_VectorCall or PyCCall_Call).


Not quite. For a builtin_function_or_method, we have with PEP 580:

a. call_function()
     calls
d. PyCCall_FastCall
     which calls
e. the actual C function

and with PEP 590 it's more like:

a. call_function()
     calls
c. _PyCFunction_FastCallKeywords
     which calls
d. _PyMethodDef_RawFastCallKeywords
     which calls
e. the actual C function

Level c. above is the vectorcall wrapper, which is a level that PEP 580 
doesn't have.


Again, I'll get back to this next week.


The way `const` is handled in the function signatures strikes me as too
fragile for public API.


That's a detail which shouldn't influence the acceptance of either PEP.


True.
I guess what I want from the answer is to know how much thought went 
into const handling: is what's in the PEP an initial draft, or does it 
solve some hidden issue?



Why not have a per-type pointer, and for types that need it (like
PyTypeObject), make it dispatch to an instance-specific function?


That would be exactly https://bugs.python.org/issue29259

I'll let Mark comment on this.


Mino

Re: [Python-Dev] Update PEP 394: Distributions can choose what does python command mean

[Petr Viktorin]

On 4/12/19 4:53 PM, Miro Hrončok wrote:

Hello.

Based on discussions in [1], Petr Viktorin and me have drafted a new 
update [2] to the PEP 394 (The "python" Command on Unix-Like Systems).


The update gives distributors the opportunity to decide where does the 
"python" command lead to, whether it is present etc.


Please, see the PR [2] for the suggested changes.

[1]: https://mail.python.org/pipermail/python-dev/2019-February/156272.html
[2]: https://github.com/python/peps/pull/989


The text is available at 
https://github.com/hroncok/peps/blob/pep394-2019/pep-0394.txt


As a summary, I'll paste the rationale sections here:

History of this PEP
===

In 2011, the majority of distributions
aliased the ``python`` command to Python 2, but some started switching it to
Python 3 ([5]_). As some of the former distributions did not provide a
``python2`` command by default, there was previously no way for Python 2 
code

(or any code that invokes the Python 2 interpreter directly rather than via
``sys.executable``) to reliably run on all Unix-like systems without
modification, as the ``python`` command would invoke the wrong interpreter
version on some systems, and the ``python2`` command would fail completely
on others. This PEP originally provided a very simple mechanism
to restore cross-platform support, with minimal additional work required
on the part of distribution maintainers. Simplified, the recommendation was:

1. The ``python`` command was preferred for code compatible with both
   Python 2 and 3 (since it was available on all systems, even those that
   already aliased it to Python 3).
2. The ``python`` command should always invoke Python 2 (to prevent
   hard-to-diagnose errors when Python 2 code is run on Python 3).
3. The ``python2`` and ``python3`` commands should be available to specify
   the version explicitly.

However, these recommendations implicitly assumed that Python 2 would 
always be
available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 
404),

distributions are making Python 2 optional or removing entirely.
This means either removing the ``python`` command or switching it to invoke
Python 3, invalidating respectively the first or second recommendation.
Also, some distributors decided that their users are better served by
ignoring the PEP's recommendations, making the PEP's supposedly
cross-platform recommendations on ``python`` and ``python2`` in shebangs
increasingly unreliable.


Current Rationale
=

As of 2019, nearly all new systems include Python 3 and the ``python3``
command. This makes the ``python3`` command the best general choice for
code that can run on either Python 3.x or 2.x, even though it is not
available everywhere.

The recommendation is skewed toward current and future systems, leaving
behind “*old systems*” (like RHEL 6 or default Python installed on macOS).
On these systems, Python software is rarely updated and any recommendations
this PEP makes would likely be ignored.

Also, since distributors often ignored recommendations the PEP gave
regarding the ``python`` command (for what they saw as legitimate special
needs), this PEP now gives them broad control over the command.
Correspondingly, users are advised to not use the ``python`` command
in cross-platform code.
Instead, this PEP specifies the expected behavior of the ``python3`` and
``python2`` commands, which is not controversial.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

Hi Mark! See my more general reply; here I'll just tie loose ends with a 
few +1s.


On 4/14/19 7:30 AM, Mark Shannon wrote:

On 10/04/2019 5:25 pm, Petr Viktorin wrote:

[...]
PEP 590 is built on a simple idea, formalizing fastcall. But it is 
complicated by PY_VECTORCALL_ARGUMENTS_OFFSET and 
Py_TPFLAGS_METHOD_DESCRIPTOR.
As far as I understand, both are there to avoid intermediate 
bound-method object for LOAD_METHOD/CALL_METHOD. (They do try to be 
general, but I don't see any other use case.)

Is that right?


Not quite.
Py_TPFLAGS_METHOD_DESCRIPTOR is for LOAD_METHOD/CALL_METHOD, it allows 
any callable descriptor to benefit from the LOAD_METHOD/CALL_METHOD 
optimisation.


PY_VECTORCALL_ARGUMENTS_OFFSET exists so that callables that make onward 
calls with an additional argument can do so efficiently. The obvious 
example is bound-methods, but classes are at least as important.

cls(*args) -> cls.new(cls, *args) -> cls.__init__(self, *args)


I see. Thanks!

(I'm running out of time today, but I'll write more on why I'm asking, 
and on the case I called "impossible" (while avoiding creation of a 
"bound method" object), later.)


Let me drop this thread; I stand corrected.

Another point I'd like some discussion on is that vectorcall function 
pointer is per-instance. It looks this is only useful for type 
objects, but it will add a pointer to every new-style callable object 
(including functions). That seems wasteful.
Why not have a per-type pointer, and for types that need it (like 
PyTypeObject), make it dispatch to an instance-specific function?


Firstly, each callable has different behaviour, so it makes sense to be 
able to do the dispatch from caller to callee in one step. Having a 
per-object function pointer allows that.
Secondly, callables are either large or transient. If large, then the 
extra few bytes makes little difference. If transient then, it matters 
even less.
The total increase in memory is likely to be only a few tens of 
kilobytes, even for a large program.


That makes sense.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

So, I spent another day pondering the PEPs.

I love PEP 590's simplicity and PEP 580's extensibility. As I hinted 
before, I hope they can they be combined, and I believe we can achieve 
that by having PEP 590's (o+offset) point not just to function pointer, 
but to a {function pointer; flags} struct with flags defined for two 
optimizations:

- "Method-like", i.e. compatible with LOAD_METHOD/CALL_METHOD.
- "Argument offsetting request", allowing PEP 590's 
PY_VECTORCALL_ARGUMENTS_OFFSET optimization.


This would mean one basic call signature (today's METH_FASTCALL | 
METH_KEYWORD), with individual optimizations available if both the 
caller and callee support them.




In case you want to know my thoughts or details, let me indulge in some 
detailed comparisons and commentary that led to this.

I also give a more detailed proposal below.
Keep in mind I wrote this before I distilled it to the paragraph above, 
and though the distillation is written as a diff to PEP 590, I still 
think of this as merging both PEPs.



PEP 580 tries hard to work with existing call conventions (like METH_O, 
METH_VARARGS), making them fast.
PEP 590 just defines a new convention. Basically, any callable that 
wants performance improvements must switch to METH_VECTORCALL (fastcall).
I believe PEP 590's approach is OK. To stay as performant as possible, C 
extension authors will need to adapt their code regularly. If they 
don't, no harm -- the code will still work as before, and will still be 
about as fast as it was before.
In exchange for this, Python (and Cython, etc.) can focus on optimizing 
one calling convention, rather than a variety, each with its own 
advantages and drawbacks.


Extending PEP 580 to support a new calling convention will involve 
defining a new CCALL_* constant, and adding to existing dispatch code.
Extending PEP 590 to support a new calling convention will most likely 
require a new type flag, and either changing the vectorcall semantics or 
adding a new pointer.
To be a bit more concrete, I think of possible extensions to PEP 590 as 
things like:
- Accepting a kwarg dict directly, without copying the items to 
tuple/array (as in PEP 580's CCALL_VARARGS|CCALL_KEYWORDS)
- Prepending more than one positional argument, or appending positional 
arguments
- When an optimization like LOAD_METHOD/CALL_METHOD turns out to no 
longer be relevant, removing it to simplify/speed up code.
I expect we'll later find out that something along these lines might 
improve performance. PEP 590 would make it hard to experiment.


I mentally split PEP 590 into two pieces: formalizing fastcall, plus one 
major "extension" -- making bound methods fast.
When seen this way, this "extension" is quite heavy: it adds an 
additional type flag, Py_TPFLAGS_METHOD_DESCRIPTOR, and uses a bit in 
the "Py_ssize_t nargs" argument as additional flag. Both type flags and 
nargs bits are very limited resources. If I was sure vectorcall is the 
final best implementation we'll have, I'd go and approve it – but I 
think we still need room for experimentation, in the form of more such 
extensions.
PEP 580, with its collection of per-instance data and flags, is 
definitely more extensible. What I don't like about it is that it has 
the extensions built-in; mandatory for all callers/callees.


PEP 580 adds a common data struct to callable instances. Currently these 
are all data bound methods want to use (cc_flags, cc_func, cc_parent, 
cr_self). Various flags are consulted in order to deliver the needed 
info to the underlying function.
PEP 590 lets the callable object store data it needs independently. It 
provides a clever mechanism for pre-allocating space for bound methods' 
prepended "self" argument, so data can be provided cheaply, though it's 
still done by the callable itself.
Callables that would need to e.g. prepend more than one argument won't 
be able to use this mechanism, but y'all convinced me that is not worth 
optimizing for.


PEP 580's goal seems to be that making a callable behave like a Python 
function/method is just a matter of the right set of flags. Jeroen 
called this "complexity in the protocol".
PEP 590, on the other hand, leaves much to individual callable types. 
This is "complexity in the users of the protocol".
I now don't see a problem with PEP 590's approach. Not all users will 
need the complexity. We need to give CPython and Cython the tools to 
make implementing "def"-like functions possible (and fast), but if other 
extensions need to match the behavior of Python functions, they should 
just use Cython. Emulating Python functions is a special-enough use case 
that it doesn't justify complicating the protocol, and the same goes for 
implementing Python's built-in functions (with all their historical 
baggage).




My more full proposal for a compromise between PEP 580 and 590 would go 
something like below.


The type flag (Py_TPFLAGS_HAVE_VECTORCALL/Py_TPFLAGS_HAVE_CCALL) and 
offset (tp_vectorcall_offset/tp

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 4/10/19 7:05 PM, Jeroen Demeyer wrote:

On 2019-04-10 18:25, Petr Viktorin wrote:

Hello!
I've had time for a more thorough reading of PEP 590 and the reference
implementation. Thank you for the work!


And thank you for the review!


I'd now describe the fundamental
difference between PEP 580 and PEP 590 as:
- PEP 580 tries to optimize all existing calling conventions
- PEP 590 tries to optimize (and expose) the most general calling
convention (i.e. fastcall)


And PEP 580 has better performance overall, even for METH_FASTCALL. See 
this thread:

https://mail.python.org/pipermail/python-dev/2019-April/156954.html

Since these PEPs are all about performance, I consider this a very 
relevant argument in favor of PEP 580.


All about performance as well as simplicity, correctness, testability, 
teachability... And PEP 580 touches some introspection :)



PEP 580 also does a number of other things, as listed in PEP 579. But I
think PEP 590 does not block future PEPs for the other items.
On the other hand, PEP 580 has a much more mature implementation -- and
that's where it picked up real-world complexity.

About complexity, please read what I wrote in
https://mail.python.org/pipermail/python-dev/2019-March/156853.html

I claim that the complexity in the protocol of PEP 580 is a good thing, 
as it removes complexity from other places, in particular from the users 
of the protocol (better have a complex protocol that's simple to use, 
rather than a simple protocol that's complex to use).


I think we're talking past each other. I see now it as:

PEP 580 takes existing complexity and makes it available to all users, 
in a simpler way. It makes existing code faster.


PEP 590 defines a new simple/fast protocol for its users, and instead of 
making existing complexity faster and easier to use, it's left to be 
deprecated/phased out (or kept in existing classes for backwards 
compatibility). It makes it possible for future code to be faster/simpler.


I think things should be simple by default, but if people want some 
extra performance, they can opt in to some extra complexity.



As a more concrete example of the simplicity that PEP 580 could bring, 
CPython currently has 2 classes for bound methods implemented in C:

- "builtin_function_or_method" for normal C methods
- "method-descriptor" for slot wrappers like __eq__ or __add__

With PEP 590, these classes would need to stay separate to get maximal 
performance. With PEP 580, just one class for bound methods would be 
sufficient and there wouldn't be any performance loss. And this extends 
to custom third-party function/method classes, for example as 
implemented by Cython.


Yet, for backwards compatibility reasons, we can't merge the classes.
Also, I think CPython and Cython are exactly the users that can trade 
some extra complexity for better performance.



Jeroen's analysis from
https://mail.python.org/pipermail/python-dev/2018-July/154238.html seems
to miss a step at the top:

a. CALL_FUNCTION* / CALL_METHOD opcode
   calls
b. _PyObject_FastCallKeywords()
   which calls
c. _PyCFunction_FastCallKeywords()
   which calls
d. _PyMethodDef_RawFastCallKeywords()
   which calls
e. the actual C function (*ml_meth)()

I think it's more useful to say that both PEPs bridge a->e (via
_Py_VectorCall or PyCCall_Call).


Not quite. For a builtin_function_or_method, we have with PEP 580:

a. call_function()
     calls
d. PyCCall_FastCall
     which calls
e. the actual C function

and with PEP 590 it's more like:

a. call_function()
     calls
c. _PyCFunction_FastCallKeywords
     which calls
d. _PyMethodDef_RawFastCallKeywords
     which calls
e. the actual C function

Level c. above is the vectorcall wrapper, which is a level that PEP 580 
doesn't have.


PEP 580 optimizes all the code paths, where PEP 590 optimizes the fast 
path, and makes sure most/all use cases can use (or switch to) the fast 
path.
Both fast paths are fast: bridging a->e using zero-copy arg passing with 
some C calls and flag checks.


The PEP 580 approach is faster; PEP 590's is simpler.



Jeroen, is there something in PEPs 579/580 that PEP 590 blocks, or
should address?


Well, PEP 580 is an extensible protocol while PEP 590 is not. But, 
PyTypeObject is extensible, so even with PEP 590 one can always extend 
that (for example, PEP 590 uses a type flag Py_TPFLAGS_METHOD_DESCRIPTOR 
where PEP 580 instead uses the structs for the C call protocol). But I 
guess that extending PyTypeObject will be harder to justify (say, in a 
future PEP) than extending the C call protocol.


That's a good point.


Also, it's explicitly allowed for users of the PEP 580 protocol to 
extend the PyCCallDef structure with custom fields. But I don't have a 
concrete idea of whether that will be useful.


Unless I'm missing something, that would be effectively the same as 
extendi

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 4/25/19 10:42 AM, Jeroen Demeyer wrote:

On 2019-04-25 00:24, Petr Viktorin wrote:

I believe we can achieve
that by having PEP 590's (o+offset) point not just to function pointer,
but to a {function pointer; flags} struct with flags defined for two
optimizations:


What's the rationale for putting the flags in the instance? Do you 
expect flags to be different between one instance and another instance 
of the same class?


I'm not tied to that idea. If there's a more reasonable place to put the 
flags, let's go for it, but it's not a big enough issue so it shouldn't 
complicate the protocol much. Quoting Mark from the other subthread:

Callables are either large or transient. If large, then the extra few bytes 
makes little difference. If transient then, it matters even less.




Both type flags and
nargs bits are very limited resources.


Type flags are only a limited resource if you think that all flags ever 
added to a type must be put into tp_flags. There is nothing wrong with 
adding new fields tp_extraflags or tp_vectorcall_flags to a type.


Indeed. Extra flags are just what I think PEP 590 is missing.


What I don't like about it is that it has
the extensions built-in; mandatory for all callers/callees.


I don't agree with the above sentence about PEP 580:
- callers should use APIs like PyCCall_FastCall() and shouldn't need to 
worry about the implementation details at all.
- callees can opt out of all the extensions by not setting any special 
flags and setting cr_self to a non-NULL value. When using the flags 
CCALL_FASTCALL | CCALL_KEYWORDS, then implementing the callee is exactly 
the same as PEP 590.


Imagine an extension author sitting down to read the docs and implement 
a callable:


- PEP 580 introduces 6 CCALL_* combinations: you need to select the best 
one for your use case. Also, add two structs to the instance & link them 
via pointers, make sure you support descriptor behavior and the __name__ 
attribute. (Plus there are features for special purposes: CCALL_DEFARG, 
CCALL_OBJCLASS, self-slicing, but you can skip that initially.)
- My proposal: to the instance, add a function pointer with known 
signature and flags which you set to zero. Add an offset to the type, 
and set a type flag. (There are additional possible optimizations, but 
you can skip them initially.)


PEP 580 makes a lot of sense if you read it all, but I fear there'll be 
very few people who read and understand it.
And is not important just for extension authors (admittedly, 
implementing a callable directly using the C API is often a bad idea). 
The more people understand the mechanism, the more people can help with 
further improvements.



I don't see the benefit of supporting METH_VARARGS, METH_NOARGS, and 
METH_O calling conventions (beyond backwards compatibility and 
comptibility with Python's *args syntax).
For keywords, I see a benefit in supporting *only one* of kwarg dict or 
kwarg tuple: if the caller and callee don't agree on which one to use, 
you need an expensive conversion. If we say tuple is the way, some of 
them will need to adapt, but within the set of those that do it any 
caller/callee combination will be fast. (And if tuple only turns out to 
be wrong choice, adding dict support in the future shouldn't be hard.)


That leaves fastcall (with tuple only) as the focus of this PEP, and the 
other calling conventions essentially as implementation details of 
builtin functions/methods.




As in PEP 590, any class that uses this mechanism shall not be usable as
a base class.


Can we please lift this restriction? There is really no reason for it. 
I'm not aware of any similar restriction anywhere in CPython. Note that 
allowing subclassing is not the same as inheriting the protocol.


Sure, let's use PEP 580 treatment of inheritance.
Even if we don't, I don't think dropping this restriction would be a 
PEP-level change. It can be dropped as soon as an implementation and 
tests are ready, and inheritance issues ironed out. But it doesn't need 
to be in the initial implementation.



As a compromise, we could simply never inherit the protocol.


That also sounds reasonable for the initial implementation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 discussion

[Petr Viktorin]

On 4/25/19 5:12 AM, Jeroen Demeyer wrote:

On 2019-04-25 00:24, Petr Viktorin wrote:

PEP 590 defines a new simple/fast protocol for its users, and instead of
making existing complexity faster and easier to use, it's left to be
deprecated/phased out (or kept in existing classes for backwards
compatibility). It makes it possible for future code to be 
faster/simpler.


Can you elaborate on what you mean with this deprecating/phasing out?


Kept for backwards compatibility, but not actively recommended or 
optimized. Perhaps made slower if that would help performance elsewhere.



What's your view on dealing with method classes (not necessarily right 
now, but in the future)? Do you think that having separate method 
classes like method-wrapper (for example [].__add__) is good or bad?


I fully agree with PEP 579's point on complexity:

There are a huge number of classes involved to implement all variations of 
methods. This is not a problem by itself, but a compounding issue.


The main problem is that, currently, you sometimes need to care about 
this (due to CPython special casing its own classes, without fallback to 
some public API). Ideally, what matters is the protocols the class 
implements rather than the class itself. If that is solved, having so 
many different classes becomes curious but unimportant -- merging them 
shouldn't be a priority.


I'd concentrate on two efforts instead:

- Calling should have a fast public API. (That's this PEP.)
- Introspection should have well-defined, consistently used public API 
(but not necessarily fast).


For introspection, I think the way is implementing the necessary API 
(e.g. dunder attributes) and changing things like inspect, traceback 
generation, etc. to use them. CPython's callable classes should stay as 
internal implementation details. (Specifically: I'm against making them 
subclassable: allowing subclasses basically makes everything about the 
superclass an API.)


Since the way how PEP 580 and PEP 590 deal with bound method classes is 
very different, I would like to know the roadmap for this.


My thoughts are not the roadmap, of course :)


Speaking about roadmaps, I often use PEP 579 to check what I'm 
forgetting. Here are my thoughts on it:



## Naming (The word "built-in" is overused in Python)

This is a social/docs problem, and out of scope of the technical 
efforts. PEPs should always define the terms they use (even in the case 
where there is an official definition, but it doesn't match popular usage).



## Not extendable

As I mentioned above, I'm against opening the callables for subclassing. 
We should define and use protocols instead.



## cfunctions do not become methods

If we were designing Python from scratch, this should have been done 
differently.
Now this is a problem for Cython to solve. CPython should provide the 
tools to do so.



## Semantics of inspect.isfunction

I don't like inspect.isfunction, because "Is it a function?" is almost 
never what you actually want to ask. I'd like to deprecate it in favor 
of explicit functions like "Does it have source code?", "Is it 
callable?", or even "Is it exactly types.FunctionType?".
But I'm against changing its behavior -- people are expecting the 
current answer.



## C functions should have access to the function object

That's where my stake in all this is; I want to move on with PEP 573 
after 580/590 is sorted out.



## METH_FASTCALL is private and undocumented

This is the intersection of PEP 580 and 590.


## Allowing native C arguments

This would be a very experimental feature. Argument Clinic itself is not 
intended for public use, locking its "impl" functions as part of public 
API is off the table at this point.
Cython's cpdef allows this nicely, and CPython's API is full of C 
functions. That should be good enough good for now.



## Complexity

We should simpify, but I think the number of callable classes is not the 
best metric to focus on.



## PyMethodDef is too limited

This is a valid point. But the PyMethodDef array is little more than a 
shortcut to creating methods directly in a loop. The immediate 
workaround could be to create a new constructor for methods. Then we can 
look into expressing the data declaratively again.



## Slot wrappers have no custom documentation

I think this can now be done with a new custom slot wrapper class. 
Perhaps that can be added to CPython when it matures.



## Static methods and class methods should be callable

This is a valid, though minor, point. I don't event think it would be a 
PEP-level change.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

Hello!
Sorry for the delay; PyCon is keeping me busy. On the other hand, I
did get to talk to a lot of smart people here!

I'm leaning toward accepting PEP 590 (with some changes still). Let's
start focusing on it.
As for the changes, I have these 4 points:

I feel that the API needs some contact with real users before it's set
in stone. That was the motivation behind my proposal for PEP 590 with
additional flags.
At PyCon, Nick Coghlan suggested another option make the API
"provisional":  make it formally private.
Py_TPFLAGS_HAVE_VECTORCALL would be underscore-prefixed, and the docs
would say that it can change. in Python 3.9, the semantics will be
finalized and the underscore removed.
This would allow high-maintenance projects (like Cython) to start
using it and give their feedback, and we'd have a chance to respond to
the feedback.

tp_vectorcall_offset should be what's replacing tp_print in the
struct. The current implementation has tp_vectorcall there. This way,
Cython can create vectorcall callables for older Pythons. (See PEP
580: https://www.python.org/dev/peps/pep-0580/#replacing-tp-print).

Subclassing should not be forbidden. Jeroen, do you want write a
section for how subclassing should work?

Given Jeroen's research and ideas that went into the PEP (and
hopefully, we'll incorporate some PEP 580 text as well), it seems fair
to list him as co-author of the accepted PEP, instead of just listing
PEP 580 in the acknowledgement section.


On some other points:

- Single bound method class for all kinds of function classes: This
would be a cleaner design, yes, but I don't see a pressing need. As
PEP 579 says, "this is a compounding issue", not a goal. As I recall,
that is the only major reason for CCALL_DEFARG.
PEP 590 says that x64 Windows passes 4 arguments in registers.
Admittedly, I haven't checked this, nor the performance implications
(so this would be a good point to argue!), but it seems like a good
reason to keep the argument count down. So, no CCALL_DEFARG.

- In reply to this Mark's note:
> PEP 590 is fully universal, it supports callables that can do anything with 
> anything. There is no need for it to be extended because it already supports 
> any possible behaviour.

I don't buy this point. The current tp_call also supports any possible
behavior. Here we want to support any behavior *efficiently*.
As a specific example: for calling PEP 590 callable with a kwarg dict,
there'll need to be an extra allocation. That's inefficient relative
to PEP 580 (or PEP 590 plus allowing a dict in "kwnames"). But I'm
willing to believe the inefficiency is acceptable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 5/6/19 4:24 AM, Jeroen Demeyer wrote:

Hello Petr,

Thanks for your time. I suggest you (or somebody else) to officially 
reject PEP 580.


I'll do that shortly.
I hope that you are not taking this personally. PEP 580 is a good 
design. PEP 590 even says that it's built on your ideas.


I start working on reformulating PEP 590, adding some elements from PEP 
580. At the same time, I work on the implementation of PEP 590. I want 
to implement Mark's idea of having a separate wrapper for each old-style 
calling convention.


In the mean time, we can continue the discussion about the details, for 
example whether to store the flags inside the instance (I don't have an 
answer for that right now, I'll need to think about it).


I'm abandoning per-instance flag proposal. It's an unnecessary 
complication; per-type flags are fine.


Petr, did you discuss with the Steering Council? It would be good to 
have some kind of pre-approval that PEP 590 and its implementation will 
be accepted. I want to work on PEP 590, but I'm not the right person to 
"defend" it (I know that it's worse in some ways than PEP 580).


As BDFL-delegate, I'm "pre-approving" PEP 590.
I mentioned some details of PEP 590 that still need attention. If there 
are any more, now's the time to bring them up.


And yes, I know that in some ways it's worse than PEP 580. That's what 
makes it a hard decision.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 5/6/19 3:43 AM, Jeroen Demeyer wrote:

On 2019-05-06 00:04, Petr Viktorin wrote:

- Single bound method class for all kinds of function classes: This
would be a cleaner design, yes, but I don't see a pressing need. As
PEP 579 says, "this is a compounding issue", not a goal. As I recall,
that is the only major reason for CCALL_DEFARG.


Just a minor correction here: I guess that you mean CCALL_SELFARG. The 
flag CCALL_DEFARG is for passing the PyCCallDef* in PEP 580, which is 
mostly equivalent to passing the callable object in PEP 590.


The signature of PEP 580 is

func(const PyCCallDef *def, PyObject *self, PyObject *const *args, 
Py_ssize_t nargs, PyObject *kwnames)


And with PEP 590 it is

func(PyObject *callable, PyObject *const *args, Py_ssize_t nargs, 
PyObject *kwnames)


with the additional special role for the PY_VECTORCALL_ARGUMENTS_OFFSET 
bit (which is meant to solve the problem of "self" in a different way).


I worded that badly, sorry.

From PEP 590's `callable`, the called function can get any of these if 
it needs to (and if they're stored somewhere). But you can't write 
generic code would get them from any callable.


If we're not going for the "single bound method class" idea, that is OK; 
`def` & `self` can be implementation details of the callables that need 
them.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hello

[Petr Viktorin]

On 5/7/19 9:08 PM, Ben Kane wrote:
Sorry Victor, I must have misinterpreted this snippet from the mailbot. 
Maybe it would be a good idea to re-word it? I'll be happy to open an 
issue for that in an appropriate mailing list if you think that would help.


 > Welcome to the Python-Dev@python.org  
mailing list! If you are a new subscriber, please take the time to 
introduce yourself briefly in your first post. It is appreciated if you 
lurk around for a while before posting! :-)


Not sure what the original intent was, but I read that as: "Whenever you 
get to sending your first post, include an introduction in it. But, 
before you start actively participating, take your time to hang around 
and absorb the culture."


Perhaps we want to reverse the two sentences to put them in 
chronological order?


I'm not sure what the proper forum for this discussion would be.

I talked to Barry Warsaw about https://bugs.python.org/issue36837 and he 
said I should post to python-dev to get feedback. I'll work on that 
email and post back here once I'm happy with it.


Thanks,
Ben


On Tue, May 7, 2019 at 4:47 PM Victor Stinner > wrote:


Welcome Ben!

Le mar. 7 mai 2019 à 16:28, Ben Kane mailto:bbk1...@gmail.com>> a écrit :
 > My name is Ben Kane. I'm joining this mailing list and
introducing myself as asked in the "welcome to python-dev" email.
I've been using Python for maybe 7 years now. I started to learn it
in college to help with accounting homework and I continue to use it
now for personal projects ranging from small scripts to larger
applications and in my day job as a LinkedIn Site Reliability Engineer.
 >
 > I'd like to contribute back to the language that's helped me so
much; and talking with core devs at PyCon 2019 has really encouraged
me. So I'll be lurking and trying to pick things up as I see them.

python-dev is not the proper mail to introduce yourself to contribute.
I suggest you to start reading http://devguide.python.org/ and please
send the same email to core-mentorship mailing list:
https://mail.python.org/mailman3/lists/core-mentorship.python.org/?x=22

It would help if you can elaborate on which parts of Python you would
be interested to contribute ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.




--

Ben Kane

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

PEP 590 is on its way to be accepted, with some details still to be
discussed. I've rejected PEP 580 so we can focus on one place.

Here are things we discussed on GitHub but now seem to agree on:

* The vectorcall function's kwname argument can be NULL.
* Let's use `vectorcallfunc`, not `vectorcall`, and stop the bikeshedding.
* `tp_vectorcall_offset` can be `Py_ssize_t` (The discussions around
signedness and C standards and consistency are interesting, but
ultimately irrelevant here.)
* `PyCall_MakeTpCall` can be removed.
* `PyVectorcall_Function` (for getting the `vectorcallfunc` of an
object) can be an internal helper. External code should go through
`PyCall_Vectorcall` (whatever we name it).
* `PY_VECTORCALL_ARGUMENTS_OFFSET` is OK, bikeshedding over variants
like `PY_VECTORCALL_PREPEND` won't bring much benefit.

Anyone against, make your point now :)

The following have discussion PRs open:

* `PyCall_MakeVectorCall` name: https://github.com/python/peps/pull/1037
* Passing a dict to `PyObject_Vectorcall`:
https://github.com/python/peps/pull/1038
* Type of the kwnames argument (PyObject/PyTupleObject):
https://github.com/python/peps/pull/1039


The remaining points are:


### Making things private

For Python 3.8, the public API should be private, so the API can get
some contact with the real world. I'd especially like to be able to
learn from
Cython's experience using it.
That would mean:

* _PyObject_Vectorcall
* _PyCall_MakeVectorCall
* _PyVectorcall_NARGS
* _METH_VECTORCALL
* _Py_TPFLAGS_HAVE_VECTORCALL
* _Py_TPFLAGS_METHOD_DESCRIPTOR


### Can the kwnames tuple be empty?

Disallowing empty tuples means it's easier for the *callee* to detect
the case of no keyword arguments. Instead of:

if (kwnames != NULL && PyTuple_GET_SIZE(kwnames))

you have:

if (kwnames != NULL)

On the other hand, the *caller* would now be responsible for handling
the no-kwarg case specially.

Jeroen points out:
> The side of the caller (which should ensure not to send an empty tuple)
> is CPython and there the issue of people implementing the protocol wrongly
> doesn't arise.
> External C code is not expected to manually use tp_vectorcall_offset to make
> vectorcalls: it is expected to use an API like PyCall_Vectorcall() and that
> API will ensure to replace an empty tuple with NULL.
>
> I see it as an application of 
> https://en.wikipedia.org/wiki/Robustness_principle
> (Be conservative in what you send, be liberal in what you accept):
> PyCall_Vectorcall should accept an empty tuple but it should not send an
> empty tuple to the vectorcall function.

But, if you apply the robustness principle to vectorcallfunc, it
should accept empty tuples.


### `METH_VECTORCALL` function type

Jeroen suggested changing this from:

`PyObject *(*call) (PyObject *self, PyObject *const *args,
Py_ssize_t nargs, PyObject *kwname)`

to `vectorcallfunc`, i.e.:

`PyObject *(*call) (PyObject *callable, Py_ssize_t n, PyObject
*const *args, PyObject *kwnames)`

Mark argues that this is a major change and prevents the interpreter
from sanity checking the return value of PyMethodDef defined
functions.
(Since the functions are defined by extension code, they need to be
sanity-checked, and this will be done by PyCFunction's vectorcall
adapter. Tools like Cython can bypass the check if needed.)

The underlying C function should not need to know how to extract
"self" from the function object, or how to handle the argument
offsetting.
Those should be implementation details.

I see the value in having METH_VECTORCALL equivalent to the existing
METH_FASTCALL|METH_KEYWORDS.
(Even though PEP 573 will need to add to the calling convention.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 580/590 discussion

[Petr Viktorin]

On 5/9/19 5:33 PM, Jeroen Demeyer wrote:

On 2019-05-09 20:30, Petr Viktorin wrote:

The underlying C function should not need to know how to extract
"self" from the function object, or how to handle the argument
offsetting.
Those should be implementation details.


Maybe you misunderstood my proposal. I want to allow both for extra 
flexibility:


- METH_FASTCALL (possibly combined with METH_KEYWORDS) continues to work 
as before. If you don't want to care about the implementation details of 
vectorcall, this is the right thing to use.


- METH_VECTORCALL (using exactly the vectorcallfunc signature) is a new 
calling convention for applications that want the lowest possible 
overhead at the cost of being slightly harder to use.


Then we can, in the spirit of minimalism, not add METH_VECTORCALL at all.


Personally, I consider the discussion about who is supposed to check 
that a function returns NULL if and if an error occurred a tiny detail 
which shouldn't dictate the design. There are two solutions for this: 
either we move that check one level up and do it for all vectorcall 
functions. Or, we keep the existing checks in place but we don't do that 
check for METH_VECTORCALL (this is already more specialized anyway, so 
dropping that check doesn't hurt much). We could also decide to enable 
this check only for debug builds, especially if debug builds are going 
to be easier to use thank to Victor Stinner's work.



I see the value in having METH_VECTORCALL equivalent to the existing
METH_FASTCALL|METH_KEYWORDS.


But why invent a new name for that? METH_FASTCALL|METH_KEYWORDS already 
works. The alias METH_VECTORCALL could only make things more confusing 
(having two ways to specify exactly the same thing). Or am I missing 
something?


METH_FASTCALL is currently not documented, and it should be renamed 
before it's documented. Names with "fast" or "new" generally don't age well.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-36829: Add sys.unraisablehook()

[Petr Viktorin]

On 5/16/19 3:23 AM, Victor Stinner wrote:
[...]

I modified my API to create an object to pack arguments. The new API
becomes sys.unraisablehook(unraisable) where unraisable has 4 fields:
exc_type, exc_value, exc_tb, obj.

[...]
I always thought the classic (exc_type, exc_value, exc_tb) triple is a 
holdover from older Python versions, and all the information is now in 
the exception instance.

Is the triple ever different from (type(exc), exc, exc.__traceback__)?
(possibly with a getattr for __traceback__)
Should new APIs use it?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Petr Viktorin]

On 5/21/19 12:06 PM, Christian Heimes wrote:

On 21/05/2019 11.49, Nathaniel Smith wrote:

On Tue, May 21, 2019 at 2:40 AM Walter Dörwald  wrote:


On 20 May 2019, at 22:15, Christian Heimes wrote:


Hi,

here is the first version of my PEP 594 to deprecate and eventually
remove modules from the standard library. The PEP started last year
with talk during the Python Language Summit 2018,
https://lwn.net/Articles/755229/.

[...]

colorsys


The `colorsys `_
module
defines color conversion functions between RGB, YIQ, HSL, and HSV
coordinate
systems. The Pillow library provides much faster conversation between
color systems.

Module type
   pure Python
Deprecated in
   3.8
To be removed in
   3.10
Substitute
   `Pillow `_,
   `colorspacious `_


I'm using colorsys constantly as the basis for a tool that converts CSS
colors between different coordinate systems. I don't see how that could
be done via Pillow (which AFAICT only converts complete images).
RGB<->HSV<->HLS conversion seems to be not available (or not obvious) in
colorspacious.


Correct, colorspacious doesn't support HSV or HLS. I suppose it would
be trivial enough to add...

The 'colour' package has them (along with everything else you can
dream of): https://colour.readthedocs.io/en/latest/colour.models.html


Nice catch, I just added https://python-colormath.readthedocs.io/en/latest/ to 
my update PR. I'll add colour to the list, too.

(It didn't pop up on my radar because I wasn't checking for British spelling)

Christian



Please, don't remove colorsys. HSL and HSV aren't outdated, and AFAIK, 
the module is not a maintenance burden.


The packages on PyPI offer more advanced models. Perhaps they could be 
linked in the documentation, if we want to endorse them. But just as 
we're not removing `array` in favor of `numpy`, we shouldn't remove 
colorsys either. Colorsys is not dead, it's just small.


I assume one of the reasons colorspacious doesn't support HSV or HLS is 
that HSV/HLS tends to be used for quite different purposes than the more 
fancy color spaces. You'd use HSL/HSV for a simple color picker or a 
saturation shift on a web page, GUI or game (where you don't have 
*exact* colors anyway); things like sRGB or CIELab are for color 
management, photos, printing, etc.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590 (Vectorcall) discussion

[Petr Viktorin]

Discussion on PEP 590 (Vectorcall) has been split over several PRs, 
issues and e-mails, so let me post an update.


I am planning to approve PEP 590 with the following changes, if Mark 
doesn't object to them:


* https://github.com/python/peps/pull/1064 (Mark the main API as private 
to allow changes in Python 3.9)
* https://github.com/python/peps/pull/1066 (Use size_t for "number of 
arguments + flag")



The resulting text, for reference:

PEP: 590
Title: Vectorcall: a fast calling protocol for CPython
Author: Mark Shannon , Jeroen Demeyer 
BDFL-Delegate: Petr Viktorin 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Mar-2019
Python-Version: 3.8
Post-History:

Abstract


This PEP introduces a new C API to optimize calls of objects.
It introduces a new "vectorcall" protocol and calling convention.
This is based on the "fastcall" convention, which is already used 
internally by CPython.

The new features can be used by any user-defined extension class.

Most of the new API is private in CPython 3.8.
The plan is to finalize semantics and make it public in Python 3.9.

**NOTE**: This PEP deals only with the Python/C API,
it does not affect the Python language or standard library.


Motivation
==

The choice of a calling convention impacts the performance and 
flexibility of code on either side of the call.

Often there is tension between performance and flexibility.

The current ``tp_call`` [2]_ calling convention is sufficiently flexible 
to cover all cases, but its performance is poor.
The poor performance is largely a result of having to create 
intermediate tuples, and possibly intermediate dicts, during the call.
This is mitigated in CPython by including special-case code to speed up 
calls to Python and builtin functions.
Unfortunately, this means that other callables such as classes and third 
party extension objects are called using the

slower, more general ``tp_call`` calling convention.

This PEP proposes that the calling convention used internally for Python 
and builtin functions is generalized and published

so that all calls can benefit from better performance.
The new proposed calling convention is not fully general, but covers the 
large majority of calls.
It is designed to remove the overhead of temporary object creation and 
multiple indirections.


Another source of inefficiency in the ``tp_call`` convention is that it 
has one function pointer per class,

rather than per object.
This is inefficient for calls to classes as several intermediate objects 
need to be created.
For a class ``cls``, at least one intermediate object is created for 
each call in the sequence

``type.__call__``, ``cls.__new__``, ``cls.__init__``.

This PEP proposes an interface for use by extension modules.
Such interfaces cannot effectively be tested, or designed, without 
having the

consumers in the loop.
For that reason, we provide private (underscore-prefixed) names.
The API may change (based on consumer feedback) in Python 3.9, where we 
expect

it to be finalized, and the underscores removed.


Specification
=

The function pointer type
-

Calls are made through a function pointer taking the following parameters:

* ``PyObject *callable``: The called object
* ``PyObject *const *args``: A vector of arguments
* ``size_t nargs``: The number of arguments plus the optional flag 
``PY_VECTORCALL_ARGUMENTS_OFFSET`` (see below)
* ``PyObject *kwnames``: Either ``NULL`` or a tuple with the names of 
the keyword arguments


This is implemented by the function pointer type:
``typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject 
*const *args, size_t nargs, PyObject *kwnames);``


Changes to the ``PyTypeObject`` struct
--

The unused slot ``printfunc tp_print`` is replaced with 
``tp_vectorcall_offset``. It has the type ``Py_ssize_t``.

A new ``tp_flags`` flag is added, ``_Py_TPFLAGS_HAVE_VECTORCALL``,
which must be set for any class that uses the vectorcall protocol.

If ``_Py_TPFLAGS_HAVE_VECTORCALL`` is set, then ``tp_vectorcall_offset`` 
must be a positive integer.
It is the offset into the object of the vectorcall function pointer of 
type ``vectorcallfunc``.
This pointer may be ``NULL``, in which case the behavior is the same as 
if ``_Py_TPFLAGS_HAVE_VECTORCALL`` was not set.


The ``tp_print`` slot is reused as the ``tp_vectorcall_offset`` slot to 
make it easier for for external projects to backport the vectorcall 
protocol to earlier Python versions.
In particular, the Cython project has shown interest in doing that (see 
https://mail.python.org/pipermail/python-dev/2018-June/153927.html).


Descriptor behavior
---

One additional type flag is specified: ``Py_TPFLAGS_METHOD_DESCRIPTOR``.

``Py_TPFLAGS_METHOD_DESCRIPTOR`` should be set if the callable uses the 
descriptor protocol to create a bound method-like object.
This is used b

[Python-Dev] Accepting PEP 590 (Vectorcall)

[Petr Viktorin]

As BDFL-Delegate, I am accepting PEP 590 -- Vectorcall: a fast calling 
protocol for CPython.

https://www.python.org/dev/peps/pep-0590/

There was a major last-minute change to the text (though one that was 
discussed since PyCon US): The API is not finalized yet, and it's 
provided using private names (with leading underscores). We intend to 
use the Python 3.8 cycle to get feedback from consumers, and finalize it 
in 3.9.




Thanks to Jeroen Demeyer, who spent a lot of time pushing this forward, 
and probably has some new gray hair from interactions with me and other 
core devs. Without his PEP 580 (and its implementation), the simpler PEP 
590 would not be possible.
It might not be apparent if some of his interactions are taken out of 
context, but throughout the process Jeroen always worked to make Python 
better, and he acted professionally even as we made decisions he didn't 
agree with.


Thanks to Mark Shannon for providing major insights, patiently 
explaining that they actually are good ideas, and for the initial 
implementation.


Thanks to Stefan Behnel, Nick Coghlan, Victor Stinner and everyone else 
involved in PEPs 576, 579, 580 and 590.


I learned a lot in this process.

This is my first time as BDFL-delegate. If you have any criticism of the 
way I handled things, please share! (Assume I have tough skin, but keep 
to the code of conduct.) I'd be happy to know how I can improve.



And now, I'm off to get the implementation reviewed!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590: vectorcall without tp_call

[Petr Viktorin]

On 5/29/19 2:25 PM, Jeroen Demeyer wrote:

Hello,

I have one implementation question about vectorcall which is not 
specified in PEP 590: what should happen if a type implements vectorcall 
(i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set tp_call (i.e. 
tp_call == NULL)? I see the following possibilities:


1. Ignore this problem/assume that it won't happen. This would be bad, 
since callable(obj) would be False even though obj() would succeed.


2. Raise SystemError.

3. Automatically set tp_call to PyVectorcall_Call.

I would vote for 3 since it's the most user-friendly option. There is 
also no way how it could be wrong: it ensures that tp_call and 
vectorcall are consistent.


That sounds like a good idea for PyType_FromSpec.

For static types I either wouldn't bother at all, or only check in debug 
 builds and fail with Py_FatalError.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 590: vectorcall without tp_call

[Petr Viktorin]

On 5/29/19 3:36 PM, Jeroen Demeyer wrote:

On 2019-05-29 15:29, Petr Viktorin wrote:

That sounds like a good idea for PyType_FromSpec.


I don't think we're planning to support vectorcall in PyType_FromSpec 
for now. That's maybe for 3.9 when vectorcall is no longer provisional.



For static types I either wouldn't bother at all, or only check in debug
builds and fail with Py_FatalError.


So basically an assert(...)?


Yes. That's the usual way to let C extension authors know they did 
something wrong.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures

[Petr Viktorin]

Hello,
PEP 570 (Positional-Only Parameters) changed the signatures of 
PyCode_New() and types.CodeType(), adding a new argument for "posargcount".
Our policy for such changes seems to be fragmented tribal knowledge. I'm 
writing to check if my understanding is reasonable, so I can apply it 
and document it explicitly.


There is a surprisingly large ecosystem of tools that create code objects.
The expectation seems to be that these tools will need to be adapted for 
each minor version of Python.


But that's not the same as saying that if you use types.CodeType(), 
you're on your own. To me [PR 13271], seems to go too far when it adds:

> These types are not supposed to be instantiated outside of
> CPython internals and constructor signatures will vary
> between Python versions.

This kind of "washing our hands" doesn't do justice to tools like 
Cython, whose use of PyCode_New is (IMO) perfectly justified. As is 
adapting to Python minor versions.


So, we should document the changes as any backwards-incompatible change.
Specifically, changes should be (or should have been, in hindsight) 
mentioned in:

* The "API changes" and/or "Porting" sections of the What's New document
* Version history of the documentation (e.g. versionchanged blocks)
* Any relevant PEPs

Also, the expected level of API stability should be documented in the docs.

Does that sound right?


PEP 570: https://www.python.org/dev/peps/pep-0570/
PyCode_New: https://docs.python.org/3.8/c-api/code.html#c.PyCode_New
[PR 13271]: https://github.com/python/cpython/pull/13271/files
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: Expected stability of PyCode_New() and types.CodeType() signatures

[Petr Viktorin]

On 6/12/19 7:40 AM, Stefan Behnel wrote:

Victor Stinner schrieb am 12.06.19 um 00:09:

So yeah, the PyCode_New() change is very annoying in practical, since
every single project using Cython requires a new release in practice.


I think Cython's deal with regard to this is:

"""
If you use Cython, we will try hard to cover up the adaptations for
upcoming (and existing) CPython versions for you. But you'll likely have to
rerun Cython on your project before a new CPython major release comes out.
"""

That's obviously not ideal for projects that end up being unmaintained. But
then again, you can't freeze time forever, and /any/ change to a dependency
can end up being fatal to them.

I personally think that rerunning Cython when a new CPython release comes
out is an acceptable price to pay for a project. In the worst case, this
can even be done by others, as you suggested as a common Fedora practice
elsewhere, Victor.


I hope this is something that improvements in Python's packaging story 
(specifically, PEP 518) should help with.
I see the current practice of including Cython's output in releases as a 
workaround for the fact that you can't (reasonably) specify Cython as a 
build dependency. Cython is a much lighter dependency than a C compiler 
-- though a less common one. When there's a reliable way to specify 
build-time dependencies, running Cython on each build will hopefully 
become the obvious way to do it.


Or is there something else blocking that future?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGWDJ44NSCAWP2IC7DNFBJRHQL2JHQTL/

[Python-Dev] Re: EuroPython 2019 sprints

[Petr Viktorin]

Hello,
I'll be at the EuroPython CPython sprints too, happy to help out.


On Thu, Jun 13, 2019 at 6:36 PM Steve Dower  wrote:
> Just letting everyone know that I signed up any core devs who are going
> to be at EuroPython to run sprints :) (including myself, since I'll be
> there)

FWIW, I read this as a done deal, not inviting a confirmation for me.
A [recent discourse post] suggests otherwise :)

[recent discourse post]:
https://discuss.python.org/t/europython-2019-talks-sprints/1900/5?u=encukou
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6BYA75HSN6RX6F5LWAYEKV6PBO3IHKXZ/

[Python-Dev] Re: Comparing dict.values()

[Petr Viktorin]

On 7/25/19 2:25 AM, David Mertz wrote:
Exactly! that was my thought that the exception message could hint at 
likely approaches. The NumPy example seems to have a good pattern:


arr1 == arr2

|ValueError:Thetruth value of an array withmore than one element |isambiguous.

|Usea.any()ora.all().|


It's not the equality operator that errors: `==` means element-wise 
comparison in Numpy.

The error would come from a conversion of the array to bool:

>>> numpy.array([1, 2, 3]) == numpy.array([1, 3, 4])
array([ True, False, False])

>>> if numpy.array([ True, False, False]):
... print('Same!')
...
Traceback (most recent call last):
  File "", line 1, in 
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()



Numpy currently returns False when `==` “doesn't make sense”, but 
apparently has plans to change that:


>>> numpy.array([1, 2, 3]) == numpy.array([1, 2])
__main__:1: DeprecationWarning: elementwise comparison failed; this will 
raise an error in the future.

False

>>> numpy.array([1, 2, 3]) == numpy.array(['a', 'b'])
__main__:1: FutureWarning: elementwise comparison failed; returning 
scalar instead, but in the future will perform elementwise comparison

False

>>> numpy.__version__
'1.16.4'





On Wed, Jul 24, 2019, 8:06 PM Rob Cliffe via Python-Dev 
mailto:python-dev@python.org>> wrote:




On 25/07/2019 00:09:37, David Mertz wrote:
 > I agree with Greg.
 >
 > There are various possible behaviors that might make sense, but
having
 > `d.values() != d.values()` is about the only one I can see no
sense in.
+1
 >
 > This really feels like a good cade for reading a descriptive
 > exception. If someone wants too compare `set(d.values())` that's
 > great. If they want `list(d.values())`, also a sensible question.
But
 > the programmer should spell it explicitly.
 >
 >
So, a helpful error message including something like "Cannot compare
dict.values directly, consider converting to sets / lists / sorted
lists
before comparing" ?
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/CSTSLCDEJYKDLADQV5PJRCSSVTMB5RIG/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2IMLY36A6K45LHCXO2OVQJZBQKL47U6T/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DRG757EQKEDNKPPM2KXMZ5NVVSBSGMCL/

[Python-Dev] Re: typing: how to use names in result-tuples?

[Petr Viktorin]

On 7/29/19 4:36 PM, Christian Tismer wrote:

Hi friends,

I am meanwhile the PySide maintainer at The Qt Company,
and we are trying to make the mapping from Qt functions
to Python functions as comparable as possible.

One problem are primitive pointer variables:
In Qt, it is natural to use "sometype *varname" to make a
mutable variable. In Python, you have to turn such a thing
into a result-tuple. Example:

 void QPrinter::getPageMargins(qreal *left, qreal *top, \
 qreal *right, qreal *bottom, QPrinter::Unit unit) const

(meanwhile deprecated, but a good example)

is mapped to the Python signature

 def getPageMargins(self, \
 unit: PySide2.QtPrintSupport.QPrinter.Unit) \
 -> typing.Tuple[float, float, float, float]: ...

NOW my question:


I would like to retain the variable names in Python!
The first idea is to use typing.NamedTuple, but I get the impression
that this would be too much, because I would not only need to create
an extra named tuple definition for every set of names, but also
invent a name for that named tuple.

What I would like to have is something that looks like

 def getPageMargins(self, \
 unit: PySide2.QtPrintSupport.QPrinter.Unit) \
 -> typing.NamedTuple[left: float, top: float, \
  right:float, bottom:float]: ...

but that is obviously not a named tuple.
Possible would be to derive a name from the function, so maybe
a definition could be used like

class PageMargingResult(NamedTuple):
 left: float
 top: float
 right: float
 bottom: float

but then I would have some opaque PageMargingResult type. This would
work, but as said, this is a bit too much, since I only
wanted something like a tuple with names.

What do you suggest to do here? Something what I am missing?


Hello,
I'm afraid you're stuck with defining a new type for each of these.
One reason why that's better is that info about the names will only be 
stored once, in the type; not in every instance.


Since this is in PySide, I assume you're using the C-API. If that's the 
case, check out Struct Sequences, the "C equivalent of named tuples". 
For example, the result of "os.stat()" is a struct sequence.


https://docs.python.org/3/c-api/tuple.html#struct-sequence-objects

Note that like namedtuples, these are immutable, and they're proper 
subclasses of tuple.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DKGRAP7LHZGOHQON4UKXUT2TAMRSXQD/

[Python-Dev] Re: Speeding up CPython

[Petr Viktorin]

Let me explain an impression I'm getting. It is *just one aspect* of my 
opinion, one that doesn't make sense to me. Please tell me where it is 
wrong.



In the C API, there's a somewhat controversial refactoring going on, 
which involves passing around tstate arguments. I'm not saying [the 
first discussion] was perfect, and there are still issues, but, however 
flawed the "do-ocracy" process is, it is the best way we found to move 
forward. No one who can/wants to do the work has a better solution.


Later, Mark says there is an even better way – or at least, a less 
intrusive one! In [the second discussion], he hints at it vaguely (from 
that limited info I have, it involves switching to C11 and/or using 
compiler-specific extensions -- not an easy change to do). But 
frustratingly, Mark doesn't reveal any actual details, and a lot of the 
complaints are about churn and merge conflicts.
And now, there's news -- the better solution won't be revealed unless 
the PSF pays for it!


That's a very bad situation to be in for having discussions: basically, 
either we disregard Mark and go with the not-ideal solution, or 
virtually all work on changing the C API and internal structures is blocked.


I sense a similar thing happening here: 
https://github.com/ericsnowcurrently/multi-core-python/issues/69 -- 
there's a vague proposal to do things very differently, but I find it 
hard to find anything actionable. I would like to change my plans to 
align with Mark's fork, or to better explain some of the non-performance 
reasons for recent/planned changes. But I can't, because details are 
behind a paywall.



[the first discussion]: 
https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/#Q4IPXMQIM5YRLZLHADUGSUT4ZLXQ6MYY


[the second discussion]: 
https://mail.python.org/archives/list/python-dev@python.org/thread/KGBXVVJQZJEEZD7KDS5G3GLBGZ6XNJJX/#WOKAUQYDJDVRA7SJRJDEAHXTRXSVPNMO



On 10/20/20 2:53 PM, Mark Shannon wrote:

Hi everyone,

CPython is slow. We all know that, yet little is done to fix it.

I'd like to change that.
I have a plan to speed up CPython by a factor of five over the next few 
years. But it needs funding.


I am aware that there have been several promised speed ups in the past 
that have failed. You might wonder why this is different.


Here are three reasons:
1. I already have working code for the first stage.
2. I'm not promising a silver bullet. I recognize that this is a 
substantial amount of work and needs funding.
3. I have extensive experience in VM implementation, not to mention a 
PhD in the subject.


My ideas for possible funding, as well as the actual plan of 
development, can be found here:


https://github.com/markshannon/faster-cpython

I'd love to hear your thoughts on this.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RDXLCH22T2EZDRCBM6ZYYIUTBWQVVVWH/ 


Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7DKURFZ3JEZTKCUAUDCPR527FUBYMY7N/
Code of Conduct: http://python.org/psf/codeofconduct/