Re: [Python-ideas] Logging Levels
On 29 November 2017 at 06:46, Mike Miller <python-id...@mgmiller.net> wrote: > Hi, the reason I use note is that I want it to be output by default. So it > must be above warning, or perhaps the default level changed. If the message to be displayed is part of the actual UX of a command line tool, our advice is "You don't want the logging module, you want the print() builtin": https://docs.python.org/3/howto/logging.html#when-to-use-logging As a relevant technical detail, it's also worth noting that the default handler emits messages on stderr, while CLI UX messages should generally be displayed on stdout. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Repurpose `assert' into a general-purpose check
On 29 November 2017 at 02:03, Ivan Pozdeev via Python-ideas <python-ideas@python.org> wrote: > On 28.11.2017 16:36, Nick Coghlan wrote: >> I'll make the same observation I usually do each time one of these >> threads comes up: >> >> * I'm opposed to making assert substantially different from the way it >> works now >> * I'm in favour of adding a new "ensure()" builtin that encapsulates >> the check-and-raise logic >> >> The reasons I prefer this approach: >> >> - assert is a statement *solely* so that the compiler can optimise it >> out. If it's not optional, >>it doesn't need to be a statement any more > > Another benefit of a statement vs function is only evaluating the > error-related arguments when there's an error If you're worried about that level of micro-optimisation, it's straightforward enough to write your own wrapper function that accepts the components needed to build a suitably formatted message. There's no need to make the builtin more complicated to cope with it (that smooth transition from the builtin behaviour to customised behaviour is one of the other advantages of using a plain function). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Repurpose `assert' into a general-purpose check
On 28 November 2017 at 15:41, Steven D'Aprano <st...@pearwood.info> wrote:
> On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas wrote:
>> Unlike C, Python does the aforementioned checks all the time, i.e. it's
>> effectively always in "debug mode".
>
> Apart from -O which disables assertions. But in any case, the best use
> of assertions is not checking things which the interpreter is going to
> do anyway, but checking things which the interpreter can not and does
> not check automatically: your program logic. There is no way that the
> Python interpreter is going to do this check automatically, unless I
> write the assertion:
>
> assert 0 <= r < abs(y)
>
> That is copied straight out of one of my functions.
I'll make the same observation I usually do each time one of these
threads comes up:
* I'm opposed to making assert substantially different from the way it works now
* I'm in favour of adding a new "ensure()" builtin that encapsulates
the check-and-raise logic
The reasons I prefer this approach:
- assert is a statement *solely* so that the compiler can optimise it
out. If it's not optional,
it doesn't need to be a statement any more
- if the existing assert statements are left alone, there are no
performance or compatibility
concerns for folks that rely on the current behaviour
- if it's a function, it doesn't need to be called "assert", it can use the more
imperative term "ensure" (meaning "ensure this condition is true
before continuing")
- if it's a function, it can easily be emulated on old versions via
compatibility libraries
- "ensure() is required, assert is optional" is a better answer to
complaints about
assertions being optional than suggesting "if cond: raise AssertionError(msg)"
as a reasonable alternative to "assert cond, msg"
- if it's a function, we get access to all the regular function
machinery, so we're
not restricted to positional-only arguments the way the assert statement is
My initial proposed behaviour for the function:
def ensure(cond, msg=None, exc_type=RuntimeError):
"""Raise an exception if the given condition is not true"""
if not cond:
if msg is None:
frame = sys._getframe(1)
line = frame.f_lineno
modname = frame.f_globals.get("__name__", "")
msg = f"Condition not met on line {line:d} in {modname!r}"
raise exc_type(msg)
Cheers,
Nick.
P.S. No, I'm not offering to write that PEP myself, I'd just be in
favour of the idea if someone else were to write it :)
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq)
On 28 November 2017 at 13:22, bunslow <buns...@gmail.com> wrote: > My first submission to this list was predicated on what I'd read in PEPs -- > and many of those, since they recommend major-enough changes to require a > PEP, have sections (often lengthy) dedicated to "what's wrong with the > status quo". My attempt to imitate that obviously crossed some boundaries in > retrospect, and of course now that it's brought up here I see that spinning > it as "what can be done to make it better" is psychologically much more > effective than "why the current way sucks" (because semantically these are > either approximately or exactly the same). But that's where it came from, at > least with some of my earlier threads, and I suspect the author of the topic > message of the OP will have a similar sentiment. Yeah, by the time someone reaches the point of writing a PEP, there's usually some level of existing awareness along the lines of "The status quo might not be OK any more, so let's explicitly document the risks and benefits associated with a possible change". That means part of the role of the PEP is to summarise the relevant problems with the status quo, such that future readers can understand why any change is being proposed at all. In cases where the proposed change is relatively simple at a technical level, like PEP 479 (converting an unhandled StopIteration to RuntimeError), PEP 538 (coercing the C locale to a UTF-8 based locale), or PEP 565 (tweaking the way we handle DeprecationWarning), the motivation & rationale may end up being the majority of the PEP, since the actual change to be made is relatively minor, but the potential consequences aren't necessarily obvious. By contrast, python-ideas threads usually start at a point earlier in the decision making process: asking ourselves the question "Is the status quo still OK?". In most cases the answer is "Yeah, it's still fine", but we keep asking, because sometimes the answer is "Actually, we could probably improve it by doing...". The easiest trap to fall into on that front is to think to ourselves "The status quo doesn't solve my problems, therefore it doesn't solve anyone's problems", which usually isn't a productive mindset. A more productive framing is typically "The problems that the status quo solves are not the problems that I currently have". It may seem like a small change, but in the second version, we're thinking: - the status quo solves problems for someone, just not for me - whatever I propose should try to avoid making the status quo worse at what it already does - I need to explain the problem I have, not just the potential solution I see and that ends up coming through in the way we write. I'll also note that nobody expects perfection on this front - most of us are thoroughly familiar with the lure of "But someone is *wrong* on the internet" [1], and we know that sometimes it's too late by the time we finally think "Oh, I really should have toned that down a bit before hitting Send...". We just strive to ensure our typical approach is to be respectful of each other, and of past contributors. Cheers, Nick. [1] https://xkcd.com/386/ P.S. For a longer version of the "What problem does it solve?" question in relation to the respective APIs of the requests and urllib modules, folks may be interested in https://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.html -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?)
On 28 November 2017 at 16:11, Stephen J. Turnbull <turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Steven D'Aprano writes: > > > The subset of iterators which are created as generators are *also* > > called generators, > > As long as we're being precise, I don't think that is precisely correct: > > >>> (x for x in range(1)) > at 0x10dee5e08> > >>> iter(range(1)) > > >>> iter((1,)) > > > The two iterators have the same duck-type, the generator is different. > A generator (object) is, of course, an interable. While it's not obvious with the genexp (since they're anonymous), the main reason for the difference in the repr layouts here is just because generator iterators can have names: >>> def g(): yield ... >>> g() So the statement that "generator iterators are iterators" is correct. The functions that create them are called generator functions because they really are functions: >>> g What's more unfortunate here is that the usage of "generator" in the generator-iterator representation doesn't actually align with the preferred terminology in the documentation: https://docs.python.org/3/glossary.html#term-generator So I can understand the confusion here. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq
On 23 November 2017 at 17:13, Grant Jenks <grant.je...@gmail.com> wrote: > And it will work! The heap algorithm is exposed through a high-level > functional interface so that you can take advantage of duck-typing. This is > an important Python feature. > Nobody is proposing taking the functional interface away (just as nobody is proposing to take the bisect module away). Instead, the argument is: - the heapq functions need to be combined with concrete storage to be useful - a simple wrapper class that combines them with a given list (or other mutable sequence) is short and easy to maintain - such a wrapper class is also useful in its own right for basic heap use cases - so let's provide one in the collections module The potential arguments against that are: - this means yet-another-data-type in the collections module, so it will make the collections module harder to learn - people will be confused as to whether they should use collections.Heap or the heapq functions The risks of both of those outcomes seem low, since the main current source of confusion appears to be folks looking for a concrete "Heap" container type, and being surprised when they get told they either need to be build a DIY one from a list + the heapq functions, or else install a 3rd party dependency to get a preassembled one. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow additional separator character in variables
On 23 November 2017 at 16:34, Stephen J. Turnbull < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Nick Coghlan writes: > > > We're not going to start second-guessing the Unicode Consortium on this > > point - human languages are complicated, and we don't have any special > > insight on this point that they don't. > > Agreed. Python, however, is NOT a (natural) human language, and the > Unicode Consortium definition of conformance does NOT prohibit > subsetting appropriate to the purpose. We DO know more than the > Unicode Consortium about Python. For example, I suspect that your > catholic appetite for XID in identifiers does not apply to syntactic > keywords or names of builtins. > We already have a stricter ASCII-only naming policy for the standard library: https://www.python.org/dev/peps/pep-3131/#policy-specification That's different from placing additional constraints on end-user code, though, as that's where the line between "programming language" and "natural language" gets blurry (variable, function, attribute, and method names are often nouns and verbs in the author's language, and this is also the case for data-derived APIs like pandas column names) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq
On 23 November 2017 at 15:22, bunslow <buns...@gmail.com> wrote: > Something *should* be object oriented with the functions in question all > operate on the same data type, and in particular, those functions/verbs > *are only well defined for that type*. heapq.heappush(list-not-heap, item) > is perfectly valid code in the current interface, but doesn't make any > sense at all. And list.sort is *not* function based, it's an object > oriented method. sorted() provides the same functionality for other types, > given that it's a well defined function for a wide variety of sequences > (unlike heappush). (list.sort is a method because unlike sorted(), it > operates inplace, and is thus only meaningful for mutable, "complete" (all > in memory, not "lazy") sequences -- i.e., a list.) > > I've never used bisect, so I'll refrain from commenting on it. > > At the end of the day, the patch proposed is merely a wrapper around the > functional approach; you are welcome to continue using it as you like, it's > not going anywhere. I would propose that the docs put the OOP version first > though. > Ensuring the docs are clearly separated is part of why I think "collections.Heap" would make more sense as a proposal than "heapq.Heap" - collections already imports heapq for use in the Counter implementation, and adding another primitive container type to collections is a smaller conceptual change than updating heapq to being more than just a heap algorithm library. I'd also note that a collections.Heap class that accepts the underlying list to use as its sole parameter would compose nicely with the list builtin to allow creation of a heap from an arbitrary iterable: heap = collections.Heap(list(iterable)) rather than the current: heap = list(iterable) heapq.heapify(heap) That's akin to the difference between: data = sorted(iterable) and: data = list(iterable) data.sort() I don't think it would be a disaster if we continued to leave this out, but I don't think it would be a major maintenance burden or future barrier to learning to add it, either. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow additional separator character in variables
On 21 November 2017 at 21:55, Stephen J. Turnbull < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Personally, I think that Python probably should ban non-ASCII > non-letter characters in identifiers and whitespace, and maybe add > them later in response to requests from native speakers of the > relevant languages. I don't know how easy that would be to do, > though, since I think the rule is already that identifiers must be > composed only of letters, numbers, and ASCII "_". Since Serhiy's > examples are valid, we'd have to rule them out explicitly, rather than > by reference to the Unicode database. Yuck. > We're not going to start second-guessing the Unicode Consortium on this point - human languages are complicated, and we don't have any special insight on this point that they don't. https://www.python.org/dev/peps/pep-3131/#specification-of-language-changes delegated this aspect of the language to them by way of the XID_Start and the XID_Continue categories, and we're not going to change that. Any hybrid Python 2/3 application or library is necessarily restricted to ASCII-only identifiers, since that's all that Python 2 supports. We've also explicitly retained the ASCII-only restriction for PyPI distribution names (see https://www.python.org/dev/peps/pep-0508/#names), but that doesn't restrict the names used for import packages, only the names used to publish and install those components. If we ever decide to lift that restriction, it will likely be by way of https://en.wikipedia.org/wiki/Punycode, similar to the way internationalized domain names work, as well as the way multi-phase extension module initialization locates init functions for extension modules with non-ASCII names. Beyond that, I'll note that these questions were all raised in the original PEP: https://www.python.org/dev/peps/pep-3131/#open-issues The reference interpreter really isn't the place to experiment with answering them - rather, they're more a question for opt-in code analysis, since that makes it possible for folks to choose settings that are right *for them* (e.g. by defining a set of "permitted scripts" [1], specifying the Unicode characters that should be allowed in identifiers beyond the core set of "Latin" code points allowed by ASCII) Cheers, Nick. [1] https://en.wikipedia.org/wiki/Script_(Unicode) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism
On 21 November 2017 at 12:34, Nick Coghlan <ncogh...@gmail.com> wrote: > Right, but once you do that, then the existing resolver is already > able to handle things: > > >>> class C: pass > ... > >>> class S(C): pass > ... > >>> class E: pass > ... > >>> class B(S, E, C): pass > ... > >>> class R(E, C): pass > ... > >>> class Z(B, R): pass > ... > >>> > > If you wanted to pick up cases where two classes generate inconsistent > MROs that will prevent mutual subclasses (like "class B(S, E)" vs > "class R(E, C)"), that feels like a job for a type checker, since it > isn't obvious whether it's the definition of B or the definition of R > that should be changed to reorder their MRO. I do wonder if we might be able to make the error message here less cryptic by mentioning which *listed* base classes brought in the conflicting MRO entries. Consider the current: >>> class Z(B, R): pass ... Traceback (most recent call last): File "", line 1, in TypeError: Cannot create a consistent method resolution order (MRO) for bases C, E vs something like: TypeError: Cannot create a consistent method resolution order (MRO) for bases C, E (inherited through B, R) (Note: I haven't actually checked how practical it would be to implement something like that) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism
On 21 November 2017 at 11:09, Neil Girdhar <mistersh...@gmail.com> wrote: > > On Sat, Nov 18, 2017 at 9:29 PM Nick Coghlan <ncogh...@gmail.com> wrote: >> >> >>> class C: pass >> ... >> >>> class S(C): pass >> ... >> >>> class E: pass >> ... >> >>> class B(S, E): pass >> ... >> >>> class R(E, C): pass >> ... >> >>> class Z(B, S, R, E, C): pass >> ... >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: Cannot create a consistent method resolution order >> (MRO) for bases C, E > Sorry, I wrote back too quickly. I meant also to change B's requested MRO > to be: > > (B, S, E, C) > > It works with that change. Right, but once you do that, then the existing resolver is already able to handle things: >>> class C: pass ... >>> class S(C): pass ... >>> class E: pass ... >>> class B(S, E, C): pass ... >>> class R(E, C): pass ... >>> class Z(B, R): pass ... >>> If you wanted to pick up cases where two classes generate inconsistent MROs that will prevent mutual subclasses (like "class B(S, E)" vs "class R(E, C)"), that feels like a job for a type checker, since it isn't obvious whether it's the definition of B or the definition of R that should be changed to reorder their MRO. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow additional separator character in variables
On 19 November 2017 at 13:22, Mikhail V <mikhail...@gmail.com> wrote: > For me, one "cheap" solution against underscores is to use > syntax highlighting which grays them out, but if those become like > spaces, then it becomes a bit confusing, e.g. in function with many arguments. > Also, unfortunately, not many editors allow easy (if any) highlighting > customisation on that level. Changing the way editors display underscore-using variable names still seems like a more productive direction to explore than changing the text encoding read by the compiler. The current source code structure is well-defined and unambiguous, so there's no clear benefit to change things at that level, and significant downsides in terms of complexity, forwards and backwards compatibility concerns, and high barriers to pervasive adoption. By contrast, if the argument for using a different Unicode character is "Editors will reliably display Unicode hyphen characters differently from the way they display minus signs (or vice-versa)", then we can just as easily say "If users are finding the way that text editors display snake_cased_names to be consistently hard to read, then text editors should change the way that they display snake_cased_names (or at least make it easy for users to opt-in to displaying them differently)". For example, they could decide to replace underscores in variable names for display purposes with hyphens plus the underscore combining diacritic, or the combining macron below: - https://en.wikipedia.org/wiki/Underline#Unicode - https://en.wikipedia.org/wiki/Macron_below Then when the cursor was placed inside the variable name, they could revert to displaying those characters as regular underscores. This kind of editor level modification would also extend itself well to underscores in numeric literals, as there the appropriate pseudo-separator shown when the literal wasn't being edited would be locale dependent. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow additional separator character in variables
On 19 November 2017 at 12:32, Nick Coghlan <ncogh...@gmail.com> wrote: > On 19 November 2017 at 12:01, Mikhail V <mikhail...@gmail.com> wrote: >> Python allows underscore character as a separator in variables. >> This is better than nothing, still it does not make the look much better. >> >> **Proposal**: allow additional separator, namely hyphen character. > > Regardless of any potential readability merits, backwards > compatibility requirements combined with the use of the hyphen > character as a binary operator prohibit such a change: > > >>> my = variable = 1 > >>> my-variable > 0 Ah, sorry - I now see you addressed the basic version of that. The alternative of "Use a character that computers can distinguish, but humans can't" isn't an improvement, since it means introducing the exact kind of ambiguity that Python seeks to avoid by using indentation for block delimeters (rather than having the computer read braces, and humans read indentation). The difficulty of reliably distinguishing backticks from regular single quotes is also the main reason they're generally discounted from reintroduction for any other use case after their usage as an alternative to the repr builtin was dropped in Python 3.0, and it's why Python 3 prohibits mixing tabs and spaces for indentation by default. For anyone tempted to suggest "What about multiple underscores indicating continuation of the variable name?", that's still a compatibility problem due to the unary minus operator: >>> my--variable 2 >>> my---variable 0 Would hyphens in variable names improve readability sometimes? Potentially, but not enough to live with make binary subtraction expressions ambiguous (hence the consistency amongst almost all current text based programming languages on this point). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow additional separator character in variables
On 19 November 2017 at 12:01, Mikhail V <mikhail...@gmail.com> wrote:
> Python allows underscore character as a separator in variables.
> This is better than nothing, still it does not make the look much better.
>
> **Proposal**: allow additional separator, namely hyphen character.
Regardless of any potential readability merits, backwards
compatibility requirements combined with the use of the hyphen
character as a binary operator prohibit such a change:
>>> my = variable = 1
>>> my-variable
0
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism
On 19 November 2017 at 06:56, Neil Girdhar <mistersh...@gmail.com> wrote: > Would you mind explaining why it's necessary for C3 to complain? > > In: > > S < C > B < S, E > R < E, C > Z < B, R > > If Z is told to have MRO: > > (Z, B, S, R, E, C) > > then there are no conflicts with any base classes. I don't actually know what C3 allows in principle, I only know that CPython's resolver still complains in practice: >>> class C: pass ... >>> class S(C): pass ... >>> class E: pass ... >>> class B(S, E): pass ... >>> class R(E, C): pass ... >>> class Z(B, S, R, E, C): pass ... Traceback (most recent call last): File "", line 1, in TypeError: Cannot create a consistent method resolution order (MRO) for bases C, E I think the problem is that the resolver isn't looking at the declared bases of "B", and "R", it's looking at their full MRO: >>> B.__mro__ (, , , , ) >>> R.__mro__ (, , , ) Changing the heuristics used to generate B's MRO such that "C" and "E" appeared in the opposite order wouldn't really help, since that would just flip the problematic case to be the "R(C, E)" declaration. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism
On 18 November 2017 at 09:03, Neil Girdhar <mistersh...@gmail.com> wrote:
> On Fri, Nov 17, 2017 at 3:15 AM Nick Coghlan <ncogh...@gmail.com> wrote:
>> I'll note that an interesting side effect of
>> https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow
>> folks to write:
>>
>> class MyCustomMRO:
>> def __init__(self, *bases):
>> self._resolved_bases = my_mro_resolver(bases)
>> def __mro_entries(self, orig_bases):
>> if len(orig_bases) > 1 or orig_bases[0] is not self:
>> raise TypeError("CustomMRO instance must be sole base
>> class")
>> return self._resolved_bases
>>
>>
>> class Z(MyCustomMRO(B, R)):
>> ...
>>
>> The custom MRO algorithm may then allow for use case specific hints to
>> handle situations that the default C3 resolver will reject as
>> inconsistent or ambiguous. (I'll also note that such a custom resolver
>> would be able to manufacture and inject synthetic metaclasses if
>> that's what someone decided they really wanted to do, by also
>> synthesising a custom base class to stick at the head of the list of
>> bases).
>
> This is really cool!
Unfortunately, as Koos noted, it doesn't actually work as simply as I
presented it: even if you spell out a full MRO in the most-derived
class, the C3 resolution algorithm will complain that it's
inconsistent with the order in one of the two conflicting base
classes.
So to truly get a custom method resolution order, you're likely going
to end up needing a custom metaclass involved.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism
On 17 November 2017 at 15:52, Steven D'Aprano <st...@pearwood.info> wrote:
> There's a lot of benefit to having a relatively simple, understandable
> algorithm for determining the MRO, as opposed to some sort of adaptive
> rule that will try to reorder classes according to potentially clashing
> constraints. If that means that some classes cannot go together in
> multiple inheritence because their MRO would be inconsistent, I think
> that's a price worth paying for not having to debug inheritence bugs
> caused by weirdly reordered MROs.
I'll note that an interesting side effect of
https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow
folks to write:
class MyCustomMRO:
def __init__(self, *bases):
self._resolved_bases = my_mro_resolver(bases)
def __mro_entries(self, orig_bases):
if len(orig_bases) > 1 or orig_bases[0] is not self:
raise TypeError("CustomMRO instance must be sole base class")
return self._resolved_bases
class Z(MyCustomMRO(B, R)):
...
The custom MRO algorithm may then allow for use case specific hints to
handle situations that the default C3 resolver will reject as
inconsistent or ambiguous. (I'll also note that such a custom resolver
would be able to manufacture and inject synthetic metaclasses if
that's what someone decided they really wanted to do, by also
synthesising a custom base class to stick at the head of the list of
bases).
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 17 November 2017 at 05:15, Chris Barker <chris.bar...@noaa.gov> wrote: > On Wed, Nov 15, 2017 at 11:07 AM, Steve Dower <steve.do...@python.org> > wrote: > >> If you write such a PEP, please also research and write up the issues >> with modifying PATH on Windows (they're largely scattered throughout >> bugs.p.o and earlier discussions on python-dev). >> > > Is anyone proposing doing anything new with that? (other than changing the > default) > > My preferred solution for this is to rename "py.exe" to "python.exe" > > > I was going to propose that in this thread, but then thought: "there has > GOT to be a reason why that reall obvious solution wan't done in the first > place", and didn't have time to go back and research it. > As far as I recall, the arguments against it are: - wanting the regular executable and the launcher to be able to coexist in the same build target directory - not wanting the regular python executable to prevent access to the launcher at a shell prompt - not wanting the launcher at a shell prompt to prevent access to the regular executable at a shell prompt However, https://www.python.org/dev/peps/pep-0397/ doesn't spell those out, it just states what the launcher's name will be, and emphasises that the main purpose is to provide a sensible target for file associations after the "always use the most recently installed version" assumption broke down: https://www.python.org/dev/peps/pep-0397/#rationale Addressing them now: * as long as the extra hard links are only generated at install time, there also won't be any problems with build directory name conflicts. * the launcher will always be available a `py`, regardless of the current PATH * PATH in a venv will still put the regular python executable ahead of the launcher Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 16 November 2017 at 05:29, Zachary Ware <zachary.ware+py...@gmail.com> wrote: > On Wed, Nov 15, 2017 at 1:07 PM, Steve Dower <steve.do...@python.org> > wrote: > > My preferred solution for this is to rename "py.exe" to "python.exe" (or > > rather, make a copy of it with the new name), and extend (or more likely, > > rewrite) the launcher such that: > > > > * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and > > launch Python based on first command line argument > > * if argv[0] == "python.exe", find the matching > > PythonCore/ install (where tag may be a partial match - e.g. > > "python3.exe" finds the latest PythonCore/3.x) > > * else, if argv[0] == ".exe, find the matching > > PythonCore/ install and launch "-m " > > > > With the launcher behaving like this, we can make as many hard links as > we > > want in its install directory (it only gets installed once, so only needs > > one PATH entry, and this is C:\Windows for admin installs): > > * python.exe > > * python2.exe > > * python3.exe > > * python3.6.exe > > * pip.exe > > * pip2.exe > > * pip3.exe > > I haven't been following this thread closely, but this sounds lovely. > I'm not terribly keen on cluttering up C:\Windows with this, but > that's a minor issue. > I'd missed Steve's post before writing my last one. This sounds like a really nice technical solution to me, too, especially as it will handle Python 2 as well (even for Python 2 only systems, the launcher is available as an independently installable executable). Regardless of the underlying implementation details though, a PEP would be a helpful way of writing it up so we can make sure packaging.python.org and other resources properly account for it. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 15 November 2017 at 22:46, Michel Desmoulin <desmoulinmic...@gmail.com> wrote: > Le 13/11/2017 à 19:57, Chris Barker a écrit : > > 3) Make --user be be automatic for pip install. Not actually the > > default, but pip could do a user install if you don't have the > > permissions for a non-user install. > > Breaking compat ? Not sure people will accept. > This isn't actually a Python level change - it's a pip UX level one, and already on their todo list: https://github.com/pypa/pip/issues/1668 I believe the next concrete step that can be taken there is to actually add an explicit `--global` flag, so I've belatedly broken that out as its own issue: https://github.com/pypa/pip/issues/4865 = > > Should I do a PEP with a summary of all the stuff we discussed ? > I think a Windows-specific PEP covering adding PATH updates back to the default installer behaviour, and adding pythonX and pythonX.Y commands would be useful (and Guido would presumably delegate resolving that to Steve Dower as the Windows installer maintainer). The one thing I'd ask is that any such PEP *not* advocate for promoting ther variants as the preferred way of invoking Python on Windows - rather, they should be positioned as a way of making online instructions written for Linux more likely to "just work" for folks on Windows (similar to the utf-8 encoding changes in https://www.python.org/dev/peps/pep-0529/) Instead, the focus should be on ensuring the "python -m pip install" and "pip install" both work after clicking through the installer without changing any settings, and devising a troubleshooting guide to help folks that are familiar with computers and Python, but perhaps not with Windows, guide folks to a properly working environment. The update to the Windows installer would then start offering benefits as soon as Python 3.7 becomes the default download, while the troubleshooting guide would be beneficial as soon as folks start learning about it's existence. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 15 November 2017 at 19:51, Paul Moore <p.f.mo...@gmail.com> wrote: > On 15 November 2017 at 08:22, Nick Coghlan <ncogh...@gmail.com> wrote: > > On 15 November 2017 at 16:13, Steve Barnes <gadgetst...@live.co.uk> > wrote: > >> > >> - "pip -X[.Y][-32|-64] operation ..." tries to find a python matching > >> -X[.Y][-32|-64] and if it succeeds executes "python -m pip operation > >> ..." with that python, (if it doesn't find a matching python is should > >> fail with a sensible error message and possibly list the valid python > >> specifiers). > > > > > > This is a genuinely interesting option, especially as `pipenv` has > already > > implemented a feature somewhat akin to this: > > https://docs.pipenv.org/basics.html#specifying-versions-of-python > > > > `pipenv` also allows `pipenv --two` and `pipenv --three` when setting up > > your initial virtual environment. > > This is an interesting idea for *any* tool that's about "working with > Python environments" as opposed to "writing Python code". So pip, > virtualenv, tox, pipenv, ... Many of these tools are variously > reinventing "tell me which Python environment to work on" options. > Having a common way to do this would be really useful. I'm not sure > how likely it would be for pip to be able to use it (pip introspects > sys.executable to get site-packages, etc), but it's certainly a > possibility. > > Having a standardised library/wrapper that handles the "select a > Python environment" process would be a huge plus. There's > https://github.com/zooba/pep514tools which is a start on handling the > Windows registry scanning logic, and building on that to include Unix > and anything else we needed would be great. > While I suspect pep514tools itself may not end up being the right place, I filed https://github.com/zooba/pep514tools/issues/2 to start discussing that idea further. For other platforms, just naively scanning `PATH` is often good enough, so it's mainly Windows where discovery needs to be a bit more aware of platform specific details (i.e. PEP 514's registry entries). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas
On 14 November 2017 at 10:27, Brett Cannon <br...@python.org> wrote: > > On Mon, Nov 13, 2017, 15:55 Steven D'Aprano, <st...@pearwood.info> wrote: > >> On Mon, Nov 13, 2017 at 06:37:05PM -0500, Barry Warsaw wrote: >> > Brett Cannon wrote: >> > >> > > And possibly the easiest way to reach them is on the pyqa-dev mailing >> list. >> > >> > What's that? I can't find it on python.org, Gmane, or the Googles. >> >> Brett may have meant >> >> https://mail.python.org/mailman/listinfo/code-quality > > > Steve's right. http://meta.pycqa.org/en/latest/ > I think that will be the right way to go about it, as this seems similar to a lot of the problems that arise in the packaging interoperability space: what really matters is what tool developers support in practice, so documenting a proposed convention without a clear implementation plan across common tools wouldn't really achieve a great deal. This means that any new convention ideas need to be discussed and supported by the tool developers, with python-dev & python-ideas really only getting involved if some standard library adjustments are needed (e.g. updating doctest to abide by any proposed conventions), or if the convention is going to be captured as an informational PEP, rather than solely as a recommended comment parsing library. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 15 November 2017 at 16:13, Steve Barnes <gadgetst...@live.co.uk> wrote: > - "pip -X[.Y][-32|-64] operation ..." tries to find a python matching > -X[.Y][-32|-64] and if it succeeds executes "python -m pip operation > ..." with that python, (if it doesn't find a matching python is should > fail with a sensible error message and possibly list the valid python > specifiers). > This is a genuinely interesting option, especially as `pipenv` has already implemented a feature somewhat akin to this: https://docs.pipenv.org/basics.html#specifying-versions-of-python `pipenv` also allows `pipenv --two` and `pipenv --three` when setting up your initial virtual environment. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 15 November 2017 at 01:23, Stephan Houben <stephan...@gmail.com> wrote: > Hi Nick, > > 2017-11-14 11:07 GMT+01:00 Nick Coghlan <ncogh...@gmail.com>: > >> On 14 November 2017 at 16:47, Michel Desmoulin <desmoulinmic...@gmail.com >> > wrote: >> >>> Proposal A: >>> --- >>> >>> Suffix Python executable on Windows like on Unix, so that people will >>> type pythonX.X if they want a specify version. >>> >>> Pros: easy and discoverable. >>> >>> Cons: you need a lot of stuff in the system path. >>> >> >> Con: we hope to have the problem resolved on the Linux distro side such >> that "python" typically means "python" by the time community support for >> Python 2 ends in 2020. Since Windows has gone the better part of two >> decades without version Python commands, adding them because we're >> impatient with the pace of change at the Linux distro level doesn't really >> make sense (especially when Linux holds such a small fraction of the >> non-phone client device market). >> > > Perhaps I could sell you on the idea of a Windows "python3" executable, > not as the New Official Way to do things, > but rather as a pragmatic measure to make code from those Linux weirdos > ;-) > Aye, I'm not opposed to adding pythonX and pythonX.Y binaries or symlinks to the Windows installers as a way to make Linux-assuming tutorials slightly more likely to work for Windows users. I'm only opposed to promoting that as the new preferred way of launching Python from the Windows command line, if for no other reason than if the release after Python 3.9 actually does end up being Python 4.0, then we only have around 5 more years of "the latest Python" and "the latest Python 3.x" being the same thing :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Modules as global namespaces rather than dicts
On 15 November 2017 at 06:34, Neil Schemenauer <nas-python-id...@arctrix.com > wrote: > So, what is the purpose of all this trouble? > > - I believe quite a lot of Python internals can be simpler. For > example, importlib is complicated by the fact that a dict is > passed around when most of the logic would prefer to have the > module. Grubbing in sys.modules to lookup the module object is > ugly. The expression "exec(code, module)" is elegant to me. > I like the idea in principle, but highlighting a particular backwards compatibility pain point: one of the complications in importlib is that we promise to keep the "sys.modules[__name__] = some_other_object" idiom working. That means the need to do that check exists regardless of whether importlib is passing the module itself around, or the module's dict. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
pip and venv part of the standard and request debian that they > provide it. > > Pros: straight forward. > > Cons: holy war in the making. > I don't think it's *that* bad. Debian does have support for weak dependencies, so the resolution here may be as simple as asking Matthias to add a Recommends from python3 to python3-venv so that ensurepip is present by default on typical workstation configurations, but can still be easily omitted from server configurations. > Favorite personnal combination > > > 1 - So far, I like "providing py on unix" better, but I think it's too > hard to do. So provide suffix on windows seems more achievable. > > So +1 for having pythonX.X on windows. > I think this is actually similar to the situation with 'py' on *nix systems: with Windows having never had these before, adding and promoting them would be at best ineffective, and at worst outright confusing (especially if "python" starts meaning Python 3 by default on Linux as well within the next few years). > 2 - Decide on one form, and promote it everywhere is again the simplest > IMO. I love pipenv but the strings to pull to get something like this > means it probably would not happen. > We've been evaluating a potential pipenv tutorial for the Python Packaging User Guide, and while we're likely going to add it soon, it will be as a new "application dependency management" tutorial alongside the existing ones, rather than as a direct replacement for the pip-based "package installation" tutorial: https://github.com/pypa/python-packaging-user-guide/issues/394#issuecomment-343343760 > If we stick to suffix for issue 1, this mean we can promote: > > pythonX.X -m pip > pythonX.X -m venv > > Everywhere. In the docs. In the tutorials. In teaching classes. > > This will help a lot to remove all confusions and make the Python start > up experience way easier. > Except it won't, because now you can't seamlessly update your class instructions to new versions of Python - you'll have to go through and change all the "X.Y" references to refer to the right version. Writing cross-platform scripts will also get even harder, as you wouldn't be able to say "I just need a version of Python, I don't really care which one" any more. Python feature releases are already disruptive enough due to filesystem layout changes and file format changes - we don't need to make them worse by encouraging people to embed the exact target version in their documentation and helper scripts. > 3 - getting angry debian devs is probably not wise. The "mock" pip/venv > command also plays very well with the 2 other solutions. > While I was a bit snarky about "python -m venv" being broken by default earlier in the thread, *please* don't take that as meaning that we can't collaborate constructively with the Debian folks. We can, and do, but the friction between language level package management and system level package management is one with a *long* history that we're all working towards resolving together (See [1] & [2] for write-ups of a couple of linux.conf.au talks I've given about the problem), as is the question of how best to handle the `/usr/bin/python` symlink. > It means we can promote everywhere: > > pythonX.X -m cmd > > It will work most of the time. The combination of: pip install package python -m venv already works in most cases, *except* apparently the critical one of "New Python user on Windows downloads the python.org installer and clicks through all the buttons without changing any settings". So I think the main near term step forward would be to convince Steve Dower (as the Windows installer maintainer) to change that default behaviour yet again for 3.7, and then work towards coming up with a solid environment troubleshooting guide to include on packaging.python.org. Cheers, Nick. [1] https://lwn.net/Articles/580399/ [2] https://lwn.net/Articles/711906/ -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 14 November 2017 at 13:08, Nathaniel Smith <n...@pobox.com> wrote:
> On Nov 13, 2017 6:47 PM, "Nick Coghlan" <ncogh...@gmail.com> wrote:
>
> On 14 November 2017 at 11:51, Nathaniel Smith <n...@pobox.com> wrote:
> > What if instead of installing a standard entry point, the pip
> > executable was installed as
> >
> > #!/bin/sh
> > exec python -m pip "$@"
> >
> > on Unix-likes
>
> It would technically be enough to make the shebang line
> `#!/usr/bin/env python` so the interpreter used was picked up from the
> environment, rather than being preconfigured at install time. However,
> the problem is that you don't know for certain that that python will
> actually have `pip` installed, so it might just fail with a cryptic
> error instead.
>
>
> This would still be a massive improvement over the status quo, which in
> this situation would present a perfect simulacrum of downloading and
> installing the package you asked for, except then when you start python the
> import still fails.
>
> I did think of another issue: when installing into a virtualenv, we
> probably want to keep the current system, so explicit/path/bin/pip
> continues to work as expected.
>
My essential concern is that I don't think we should be opening ourselves
up to *even more* platform dependent behaviours in this area.
Instead, if we decide that we would like to request that pip to start doing
something different when:
* `sys.executable` and `shutil.which('python')` are inconsistent with each
other
* `sys.argv[0]` doesn't end with `site-packages/pip/__main__.py` and isn't
`shutil.which('pip')`
then we'd be better off putting that logic in pip's Python code, rather
than in the platform dependent script wrappers where we can't control the
error messages presented when our expectations aren't met.
I do like the idea of making that distinction in principle though, as it's
a better one than "inside a virtual environment or not?".
While Python level venvs are *a* way to obtain the desired pip/python
consistency, they're not the only one:
- Arch Linux does it by default
- Docker's Python 3 images do it by default (Alpine Linux may even do it by
default in general)
- Windows installs do it by default (if PATH is configured correctly)
- conda env does it
- Software Collections do it (including in the Python 3 Docker images for
RHEL & CentOS)
- *nix environment modules do it
- system admins may set up per-user install profiles to do it
- etc...
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 560 (second post)
On 14 November 2017 at 09:41, Guido van Rossum <gu...@python.org> wrote: > On Fri, Nov 10, 2017 at 8:54 AM, Ivan Levkivskyi <levkivs...@gmail.com> > wrote: >> >> On 10 November 2017 at 17:43, Guido van Rossum <gu...@python.org> wrote: >>> >>> There seem to be some action items from this thread that I haven't seen >>> reflected in the PEP source code yet. >>> [...snip...] >>> Then the next step I propose is a PR with a full implementation. After >>> that I'll likely approve the PEP (or we'll have minor feedback based on >>> trying the implementation). >> >> >> Yes, sorry, I wanted to make updates to the PEP and reference >> implementation, >> but last two weeks were very busy. >> Hopefully, I will work on it this weekend. > > Thanks, I am happy now with the PEP, except for one detail: maybe > `__mro_entry__` should always return a tuple and then maybe renamed to > `__mro_entries__`. (See debate at > https://github.com/python/peps/pull/460#issuecomment-343969528 .) I like that - very nice refinement. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 14 November 2017 at 11:51, Nathaniel Smith <n...@pobox.com> wrote:
> What if instead of installing a standard entry point, the pip
> executable was installed as
>
> #!/bin/sh
> exec python -m pip "$@"
>
> on Unix-likes
It would technically be enough to make the shebang line
`#!/usr/bin/env python` so the interpreter used was picked up from the
environment, rather than being preconfigured at install time. However,
the problem is that you don't know for certain that that python will
actually have `pip` installed, so it might just fail with a cryptic
error instead.
However, `pip` could potentially be updated with a `checkenv`
subcommand that complains if `sys.executable` and
`shutil.which('python')` don't match (and could presumably include
other checks as well).
> and a pip.bat with the equivalent contents on Windows?
> (Bonus: maybe this would fix the problem with upgrading pip on
> Windows?)
Depending on how the batch file was written, I think the answer to
that is "maybe":
https://stackoverflow.com/questions/2888976/how-to-make-bat-file-delete-it-self-after-completion/20333152#20333152
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu
On 13 November 2017 at 20:43, Antoine Pitrou <solip...@pitrou.net> wrote: > On Mon, 13 Nov 2017 11:17:54 +0100 > Petr Viktorin <encu...@gmail.com> wrote: >> > >> > Wow. I had forgotten Debian could be such a user-hostile >> > distribution. I'm not sure what the reason would be to use it as a >> > basis for a training course in Python programming, then (other than the >> > teacher having their own ideological preferences). >> >> For us, it's the *student's* preference. I believe it's better to let >> students use the machine and environment they're used to, even if it >> means extra trouble for the instructors. >> So, we get a healthy mix of Windows, Mac, Debian, Fedora, and sometimes >> some surprises. > > In that case the student must be ready to deal with the perils of their > own preferences. It's also currently missing from https://packaging.python.org/guides/installing-using-linux-tools/, but we can amend that now that we know what the required fix is. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu
On 13 November 2017 at 17:46, Stephan Houben <stephan...@gmail.com> wrote: > 2017-11-13 3:32 GMT+01:00 Nick Coghlan <ncogh...@gmail.com>: >> So technically it's ensurepip that's broken by default, but that >> translates to venv also being broken by default. >> >> I haven't worked out what the actual steps needed to fix it are > > On Debian, ensurepip is in the python3.5-venv package. > > https://packages.debian.org/stretch/amd64/python3.5-venv/filelist Thanks! I've tweaked the ncoghlan/debian-python Dockerfile to install that in addition to the base Python package (as well as finally enabling automatic rebuilds of that image whenever the base Debian image updates). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu
On 13 November 2017 at 04:38, Antoine Pitrou <solip...@pitrou.net> wrote: > On Sun, 12 Nov 2017 12:20:45 + > Paul Moore <p.f.mo...@gmail.com> wrote: >> >> > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or >> > pythonx -m venv ? Wait, it's not installed by default on debian. >> >> Seriously? Debian don't provide venv in the standard Python install? >> That's just broken. > > Frankly, I don't know where the current discussion comes from, but on > two recent Debian and Ubuntu setups, I get: > > $ dpkg -S /usr/lib/python3.5/venv/__init__.py > libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py The discussion comes from the fact that even though the code is present it doesn't actually work: $ docker run --rm -it ncoghlan/debian-python bash root@e9c0aa482aeb:/# python3 -m venv test_venv Error: Command '['/test_venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1 (I just refreshed my Docker image to 9.2, so that's the current default behaviour after "apt-get install -y python3") So technically it's ensurepip that's broken by default, but that translates to venv also being broken by default. I haven't worked out what the actual steps needed to fix it are (but I do know that installing "python3-pip" isn't enough the way it is on Fedora - we reconcile ensurepip with our security management policies by having a Recommends dep from python3 to python3-pip, and then patching ensurepip to use rewheel to inject a copy of the system pip into freshly created virtual environments. At least in theory, Debian should be able to do something similar with dirtbike, but whether or not they actually will would be a question for Matthias Klose as the Debian Python maintainer). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu
On 13 November 2017 at 09:29, Nick Coghlan <ncogh...@gmail.com> wrote: > On 13 November 2017 at 07:11, Chris Angelico <ros...@gmail.com> wrote: >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben <stephan...@gmail.com> wrote: >>> Hi Antoine, >>> >>> The venv module is included, >>> however the pyvenv script is in a separate package >>> python3.5-venv . >>> >>> By the way, I was totally confused by the following text form the doc. >>> >>> https://docs.python.org/3/library/venv.html >>> >>> >>> Deprecated since version 3.6: pyvenv was the recommended tool for creating >>> virtual environments for Python 3.3 and 3.4, and is deprecated in Python >>> 3.6. >>> >>> Changed in version 3.5: The use of venv is now recommended for creating >>> virtual environments. >>> >>> >> >> Not sure where you're reading that. I'm seeing: >> >> """ >> Note >> The pyvenv script has been deprecated as of Python 3.6 in favor of >> using python3 -m venv to help prevent any potential confusion as to >> which Python interpreter a virtual environment will be based on. >> """ >> >> I think that's pretty clear. "python3 -m venv env" is the standard and >> recommended way to spin up a virtual environment. > > It's further down in the page, under > https://docs.python.org/3/library/venv.html#creating-virtual-environments > > I think the deprecation notice for pyvenv should just be deleted, > since it renders like the *module* is deprecated. That is, the confusing one starting with "Deprecated since version 3.6: ...". The note Chris quoted is fine, and should be kept. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu
On 13 November 2017 at 07:11, Chris Angelico <ros...@gmail.com> wrote: > On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben <stephan...@gmail.com> wrote: >> Hi Antoine, >> >> The venv module is included, >> however the pyvenv script is in a separate package >> python3.5-venv . >> >> By the way, I was totally confused by the following text form the doc. >> >> https://docs.python.org/3/library/venv.html >> >> >> Deprecated since version 3.6: pyvenv was the recommended tool for creating >> virtual environments for Python 3.3 and 3.4, and is deprecated in Python >> 3.6. >> >> Changed in version 3.5: The use of venv is now recommended for creating >> virtual environments. >> >> > > Not sure where you're reading that. I'm seeing: > > """ > Note > The pyvenv script has been deprecated as of Python 3.6 in favor of > using python3 -m venv to help prevent any potential confusion as to > which Python interpreter a virtual environment will be based on. > """ > > I think that's pretty clear. "python3 -m venv env" is the standard and > recommended way to spin up a virtual environment. It's further down in the page, under https://docs.python.org/3/library/venv.html#creating-virtual-environments I think the deprecation notice for pyvenv should just be deleted, since it renders like the *module* is deprecated. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 13 November 2017 at 02:59, Brendan Barnwell <brenb...@brenbarn.net> wrote: > On 2017-11-12 05:18, Nick Coghlan wrote: >> >> * the `pip install` option really is nicer looking than `python -m pip >> install`, and it only has actual problems in the presence of multiple >> Python versions and when upgrading pip itself on Windows (plus: lots >> of third party guides recommend it, as do pypi.org project pages) > > > Is there any *advantage* to using `pip install` instead of `python > -m install`? If not, could we at least change everything under Python/pip > control (e.g., pip documentation) to never recommend `pip` and always > recommend `python -m pip` instead, and encourage all third-party > documentation to always use `python -m pip` and never use `pip`? We've already changed most of them (pypi.org itself is the main one that we haven't changed it yet). However, there are still per-project READMEs out there that suggest "easy_install project" and direct invocation of "python setup.py install", so it really isn't appealing to layer yet another mandatory change in the recommended spelling of the installation command and create yet another point of confusion - it will be much nicer overall if we can retroactively make the existing "pip install" instructions correct for most users, and leave "python -m pip install" to the "Multiple versions of Python" and "Self-upgrading pip on Windows". Cheers, Nick. P.S. As a user, it's also genuinely irritating to have to always type the "python -m " prefix when inside an active virtual environment, as in that case, there isn't any ambiguity about which environment pip should be manipulating or which version of Python it should be using. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
rties", like those Django Girls runs (i.e. running a pre-tutorial event, specifically focused on getting a working environment set up) * highly prescriptive learning environments, whether online ones (like Grok Learning, trinket.io, PythonAnywhere, etc), or locally installed ones (like PyCharm Educational Edition, the Anaconda distribution, etc) Without the kinds of constraints suggested in the last option, there are too many potential starting points, and it isn't even possible to ask potential learners to self-assess what their starting point actually is, since it's a tech support problem where the first task is "assess the current state of the user's system" (hence the first two options). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 12 November 2017 at 16:20, Michel Desmoulin <desmoulinmic...@gmail.com> wrote: > Le 10/11/2017 à 09:01, Nick Coghlan a écrit : >> On 10 November 2017 at 17:05, Michel Desmoulin >> <desmoulinmic...@gmail.com> wrote: >> When we choose not to use venv, then it becomes necessary to ensure >> each of those things individually for each potential system starting >> state >> > The way we do things is not the only way. Take JS: they don't have this > problem, you npm install the same way everywhere. You don't have > virtualenv but local node_modules. And you have transpilers to help with > the langage version differences. And they have browser manufacturers pouring millions of dollars a year into their tooling ecosystem into order to influence whose ad networks get clicked on most often. Python doesn't have that level of investment, but we do have fine folks volunteering to work on projects like `pip`, `virtualenv`, PyPI, and the packaging.python.org documentation project, as well as backports of standard library modules to earlier Python versions. When someone attempts to explain to you the practical challenges that limit both python-dev's and PyPA's ability to reliably control the starting experience of new Python users, the appropriate response is to *start listening*, not harangue them for failing to immediately follow your peremptory orders to make things simpler for you. "Do it because I said so" is bad management style even in an actual company - it's even worse in a peer production environment like an open source community. > Now I'm not advocating we do it the JS way. I'm just saying that you are > very keen to defend a statu quo instead of offering ideas to solve the > problem. I already opened https://github.com/pypa/python-packaging-user-guide/issues/396 to track possible areas of concrete near term improvement (I would have tagged you on the issue, but I couldn't find a GitHub account under your name). > Besides, using venv have the same issues. It's not installed on linux by > defaut. That depends greatly on which Linux you install - the choice on whether to deliberately cripple "python3 -m venv" or not is made by the maintainers for that distribution, and Python-friendly distros make sure Python's native tooling works properly by default. > And on windows you'll have to do py -x.x -m but on mac pythonx -m. CPython updates take years to reliably roll out to end user systems, so if you're looking to have an impact in a shorter time frame than that, the differences in cross-platform invocation are a constraint you're going to have to learn to live with. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 10 November 2017 at 19:50, Paul Moore <p.f.mo...@gmail.com> wrote: > On 10 November 2017 at 08:01, Nick Coghlan <ncogh...@gmail.com> wrote: >> That tooling is venv: >> >> * it ensures you have "pip" on your PATH >> * it ensures you have "python" on your PATH >> * it ensures that you have the required permissions to install new packages >> * it ensures that any commands you install from PyPI will be also on your >> PATH >> >> When we choose not to use venv, then it becomes necessary to ensure >> each of those things individually for each potential system starting >> state > > Currently, the reality is that people use virtualenv, not venv. All > higher-level tools I'm aware of wrap virtualenv (to allow Python 2.7 > support). Enhancing the capabilities of venv is fine, but promoting > venv over virtualenv involves technical challenges across the whole > toolset, not just documentation/education. We already assume there will be a step in understanding from "working with the latest Python 3.x locally" to "dealing with multiple Python versions". Switching from venv to virtualenv just becomes part of that process (and will often be hidden behind a higher level tool like pipenv, pew, or vex anyway). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 10 November 2017 at 17:05, Michel Desmoulin <desmoulinmic...@gmail.com> wrote: > >> Which is why we advise getting into a virtual environment ASAP, such >> that the only platform specific thing folks necessarily need to learn >> to get started is how to get to that first working virtual >> environment. >> > > You can't start by teaching virtualenv. I tried. It doesn't work. And > it's a terrible prerequisit if you write docs, tutorial, teaching > materials, etc. You can't have it both ways - the only way we can systematically mask the environmental differences between Windows, Linux and Mac OS X is by providing tooling that actually masks those differences, which means introducing that tooling becomes a prerequisite for teaching. It doesn't completely solve the problem (as getting into and out of the environment is still platform specific), but it does mean that the ubiquitous online instructions to run "pip install package-name" and "python -m command" will actually work once people are inside their working environment. That tooling is venv: * it ensures you have "pip" on your PATH * it ensures you have "python" on your PATH * it ensures that you have the required permissions to install new packages * it ensures that any commands you install from PyPI will be also on your PATH When we choose not to use venv, then it becomes necessary to ensure each of those things individually for each potential system starting state That said, we'd *like* the default to be is per-user package installations into the user home directory, but that creates additional problems: * "python" may be missing, and you'll have to use "python3" or "py" instead * "pip" may be missing (or mean "install for Python 2.7") * you have to specify "--user" on *every* call to pip, and most online guides won't say that * there's a major backwards compatibility problem with switching pip over to per-user package installs as the default (we still want to do it eventually, though) * on Windows, system-wide Python installs can't adjust per-user PATH settings, and per-user installs are subject to being broken by system-wide installs * on Windows, the distinction between a per-user install of Python, and per-user installs of a package is hard to teach * on Debian, I believe ~/.local/bin still isn't on PATH by default That said, I think there is one improvement we could feasibly make, which would be to introduce the notion of a "default user environment" into `venv`, such that there was a single "python -m venv shell" command that: * created a default user environment if it didn't already exist * launched a subshell with that environment already activated This wouldn't be a full environment manager like vex or pew - it would just be a way to bootstrap a single usable package management environment in a cross-platform way. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 10 November 2017 at 16:55, Michel Desmoulin <desmoulinmic...@gmail.com> wrote: > ensurepip may depend of youself having an internet connection when you > install it. And without a proxy. And it's not used on debian flavours. No, ensurepip doesn't depend on internet access (by design - it's the main reason why CPython bundles the wheel files). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Proposal to change Python version release cycle
On 6 November 2017 at 22:19, Stephan Houben <stephan...@gmail.com> wrote: > 2017-11-06 12:53 GMT+01:00 Brice Parent <cont...@brice.xyz>: >> I think the only problem we can reach here, not only in our lifetimes, but >> in the next years, is not Python3.10 vs Python31.0 (Python3.x will be long >> dead when we reach this point!), but the ordering of versions, like >> (python310 < python40). But it probably is a false problem, as after a >> two-digit minor version, we can fix the length of minor versions to two >> digits when needed (python310 < python400). > > No probs with either of my proposals: > >>>> "python39.dll" < "python3A.dll" < "python40.dll" > True Ah, you're right, I forgot about that option (I think Ezio Melotti suggested it previously). Yes, going with "3.10", but encoding it as "3A" in lexically ambiguous contexts is another option that would let us get as far as 3.35 (aka "3Z") before encountering ambiguity problems. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 7 November 2017 at 03:52, Michel Desmoulin <desmoulinmic...@gmail.com> wrote: > > > Le 06/11/2017 à 09:47, Nick Coghlan a écrit : >> On 6 November 2017 at 16:47, Michel Desmoulin <desmoulinmic...@gmail.com> >> wrote: >>> I really want some people from this list to discuss here so we can find >>> a way to either unify a bit the way we install and use pip, or find a >>> way to express a tutorial that always works for people on the most >>> popular platforms and spread the word so that any doc uses it. >> >> https://docs.python.org/3/installing/#basic-usage is as close as we've >> been able to get to that for the time being. > > I know and you still: > > - have to use py -m on windows, python3 linux, python in virtualenv... Which is why we advise getting into a virtual environment ASAP, such that the only platform specific thing folks necessarily need to learn to get started is how to get to that first working virtual environment. > - install pip manually on linux s/Linux/Ubuntu/ Other distros (like Fedora) provide pip by default. > - make sure the system path is correctly set Recent python.org Windows installers do this automatically, but there are unfortunately still lots of ways for things to go wrong. > Stuff that they will forget on the next install, or miss when changing > plateform. Yes, because computers are awful, and incredibly user hostile. We don't have a magic wand to wave to fix that. > And assume that stuff in any tutorial you make they know this stuff. > > This is a strong barrier or entry IMO. Sure, but it's not one we can readily fix - the user hostility of command line environments and the compromises we have to make to abide by platform conventions are in the hands of operating system vendors, and there's only so much we can do to paper over those distinctions when user lock-in and putting barriers in the way of cross-device portability is a core part of commercial OS vendors' business models. This is a big part of why mobile client devices with cloud backends are so popular, even for development purposes: they allow for a much simpler developer experience that avoids decades of accumulated cruft in the desktop operating system command line experience. Even there though, you're faced with the fact that once you choose a provider, whatever you do there will probably be locked into that provider and not transferable elsewhere. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 6 November 2017 at 18:50, Stephan Houben <stephan...@gmail.com> wrote:
> Hi Michel,
>
> That's exactly why I proposed a `pip` function available from the Python
> prompt.
> I suppose you could still tell your students to copy/paste the following
> into their
> Python interpreter.
>
> def pip(args):
> import sys
> import subprocess
> subprocess.check_call([sys.executable, "-m", "pip"] + args.split())
> print("Please restart Python now to use installed or upgraded
> packages.")
Depending on where and how Python is installed, this may still not
work (while I'll grant that Linux users are more likely to already be
familiar with the command line than Windows and Mac OS X users, it's
still a non-trivial step from there to realising why "sudo pip
install" is almost always a bad idea)
> I suppose an alternative is to set up jupyterhub
>
> https://jupyterhub.readthedocs.io/en/latest/
>
> and let all your students just access that from a webbrowser.
Yep, and lots of teachers use services like PythonAnywhere,
trinket.io, and similar, precisely because the only thing the learners
need locally is a web browser. The main downside is that learning that
way isn't quite as transferable a skill, since it completely hides the
code packaging and distribution step. However, it's a good option when
the primary aim is to teach computational skills, and Python is just
the vehicle used for that purpose.
Software Carpentry starts out with the Anaconda distribution, as it
not only improves the cross-platform UX consistent situation, it also
deals with the external binary dependency problem (at least for the
core set of packages provided either natively or via conda-forge).
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Looking for input to help with the pip situation
On 6 November 2017 at 16:47, Michel Desmoulin <desmoulinmic...@gmail.com> wrote: > I really want some people from this list to discuss here so we can find > a way to either unify a bit the way we install and use pip, or find a > way to express a tutorial that always works for people on the most > popular platforms and spread the word so that any doc uses it. https://docs.python.org/3/installing/#basic-usage is as close as we've been able to get to that for the time being. For Linux, you'll still need to do the initial "python3 -m venv" to get your students out of the system Python. I expect you'll also need an initial "py -m venv" for Windows users, to get their PATH configured appropriately with both a "python" command and any scripts they install from PyPI. But the key point: do *NOT* try to teach without creating a virtual environment as the first step, because it doesn't actually make anything simpler, and in fact makes a lot of things harder and more platform dependent. The tutorial in the Python Packaging User Guide similarly starts with venv creation: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On 6 November 2017 at 16:42, Lukasz Langa <luk...@langa.pl> wrote: > >> On 5 Nov, 2017, at 10:30 PM, Michel Desmoulin <desmoulinmic...@gmail.com> >> wrote: >> >> Le 06/11/2017 à 07:07, Nick Coghlan a écrit : >> >>> It's the default on Unix as well - you have to do "make install >>> ENSUREPIP=no" to avoid getting it. (And some distros also modify their >>> Python installations so that pip is missing by default) >> >> On debian and derivatives (so Ubuntu) you need to install python-pip to >> be able to use pip. >> >> Now it's annoying already. Because you have to write every tutorial to >> include a special case for them. But at least it's not a required step >> to run your program. >> >> However, if you do code using type hints and typing is not installed, >> you can't even run the program without installing something. So we >> finally have Python 3 by default on most Linux system, but still would >> not be able to assume we can run a modern script on it. > > This is a valid concern. Although this particular problem is self-inflicted > by Debian, I can understand their rationale behind explicit packaging. They > need to have control over the entire package graph. I wonder if there's a way > in .deb to specify a required installed package. I'm not calling it a > "dependency" since obviously it would rather be "python3-typing" that depends > on "python3". Fedora just lives with the circular dependency between python3 and python3-pip, which ensures both are installed by default (this arrangement allows "python3 -m venv" to still install pip, without actually needing a second copy of pip inside the python3 package) A bundled typing module that "python -m venv" installed by default would probably require similar treatment. > But even if Debian installs python3-typing by default, will "pip install -U > typing" be possible in this scenario? I guess it wouldn't be terrible if that > only worked in virtualenvs, although ideally it would work also for the raw > host installation. "sudo pip install " remains a terrible idea on any distro, because it leads to pip and the system package manager getting into a fight over which tool is responsible for managing the affected files. "pip install --user --upgrade typing" should work OK, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On 6 November 2017 at 09:08, Chris Angelico <ros...@gmail.com> wrote: > On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott <ba...@barrys-emacs.org> wrote: >> If this is a mechanism that python kitting has then you would be able to >> bundle other packages like requests or six as well as typing, but because >> you can use pip to override the one shipped a user can optionally keep >> up with the latest versions. > > If this were to happen, I would be inclined to put these "bootstrap" > modules into their own directory in sys.path, after the rest of the > stdlib. Then someone who's paranoid about stdlib shadowing could put > pip-installed modules after the bulk of the stdlib (thus preventing > any third-party package from overriding "import random", for instance) > but still update modules that are specifically intended for updating; > plus it'd mean you can get a directory listing of that, and go grab > all the "blessed by python.org as an extension of the stdlib" > packages. When we say bundled, we mean bundled: the exact same bits you would get from PyPI, installed the same way, and if you upgrade it system wide, there's no trace of the one we bundled. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On 5 November 2017 at 23:30, Paul Moore <p.f.mo...@gmail.com> wrote: > On 5 November 2017 at 10:48, Antoine Pitrou <solip...@pitrou.net> wrote: >> On Sun, 5 Nov 2017 13:46:59 +1000 >> Nick Coghlan <ncogh...@gmail.com> wrote: >>> * ensurepip gains the ability to also install bundled wheel files >> >> Why? Why wouldn't you put the wheel directly in site-packages on >> install? > > I'm not quite sure what you mean? It needs to be "installed", in the > sense of being unpacked into site-packages, and the ensurepip > mechanism is already able to do that for pip and setuptools, so adding > an extra wheel to install wouldn't be too hard. If we don't install > like that, people won't be able to easily upgrade typing by just using > "pip install --upgrade typing". > > Wheels don't support simply being added to sys.path the way that eggs > did, if that's what you mean. What Paul said here. While wheels *can* be designed to support use-without-extraction (and pip's own wheel file is constructed that way so you can use a pip wheel to install itself), the general design principle is that we expect them to be installed prior to use, such that they get proper PEP 376 metadata entries, and so that their subcomponents end up in the right sysconfig directories for the deployment environment. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On 5 November 2017 at 06:22, Guido van Rossum <gu...@python.org> wrote: > On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan <ncogh...@gmail.com> wrote: >> >> Perhaps typing could switch to being a bundled module, such that it >> had its own version, independent of the Python standard library >> version, but was still present by default in new installations? > > > This is beginning to sound like the most attractive solution. We could > possibly do away with typing_extensions. Are there precedents of how to > bundle a module in this way? Or is it going to be another special case like > pip? You'd be blazing new trails, but you'd be blazing them with a single-file pure Python module without any binary dependencies outside CPython itself, which is the simplest possible precedent setting scenario we could hope for. I can think of a couple of possible approaches you could take, depending on how you want to manage development on the typing module itself. = External dev = * Typing moves entirely out of the CPython source tree, with a separate issue tracker, repository, branch structure, etc (which you already have) * We add a "_bundled" directory to the CPython source tree in 3.7+ and put a typing sdist archive there * the release process PEP gains a "Confirm the bundled modules are up to date" step * the CPython build process is updated to use a venv to generate wheel files from bundled sdists * regrtest gains a "bundled" resource (and/or dedicated command line option) * when asked to test bundled modules, regrtest spins up a test venv, installs pip and the bundled modules, then runs the tests for those modules * ensurepip gains the ability to also install bundled wheel files * the Windows and Mac OS X installers start including the typing wheel file = In tree dev = * Similar to external dev for testing and distribution * Source code would stay in the main CPython tree and track CPython branch structure * The sdist file would be built by CPython's build process, rather than being checked in The external dev model is probably a better fit for your use case, though, since you'd like something that you can readily keep consistent across all supported 3.x versions (or at least 3.7+), and that's harder to do when you have multiple copies of the source code to keep in sync (vs having a single development branch where you run your CI testing across all supported Python versions). The external dev option also establishes a more useful precedent, since we could use it to officially migrate ongoing distutils maintenance into the setuptools project, and then bring it back via bundling of setuptools. Similarly, PEP 434 (which already exempts IDLE from the usual "No new features in maintenance releases" rules) could potentially be switched over to handling IDLE as an independently updated bundled application. The above idea is somewhat similar to what I suggested for the recommended modules list in https://mail.python.org/pipermail/python-ideas/2017-October/047508.html, but the difference is that to simplify the testing and installer building process, we'd bundle a suitable source archive in the git repo, rather than requiring PyPI access. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Proposal to change Python version release cycle
On 5 November 2017 at 01:29, Wolfgang <tds...@mailbox.org> wrote: > On 04.11.2017 16:01, Nick Coghlan wrote: >> We're currently more likely to go the other direction, and stick with >> the 3.x numbering for an extended period (potentially reaching 3.10, >> 3.11, 3.12, etc), so that the ongoing disruption caused by the 2.x -> >> 3.x transition has had a chance to settle down before we ask anyone to >> start calling anything "Python 4". > > A good point but two digits minor version numbers have the possibility > to break a lot code. There is a lot of stuff out where a single digit > major version is assumed. Even the official Python build for windows > with python27.dll, python36.dll can be problematic because the dot > is omitted between numbers. > Other do the same for compilation they concatenate only majorminor for a > name. > Then version 3.10 is the same as 31.0. > Ok I know this will take a while. > But for the first 2.7.10 version even there some other library code > was broken because of such assumptions. Aye, we're aware :) We're not in a situation where we'll have any *good* options available, so the question we're considering is "What do we think will be the least bad approach?". At our current release cadence, 3.7 will be in mid 2018, 3.8 in late 2019 or early 2020, and then 3.9 in mid 2021. The open question will be what we call the release after that (in late 2022), with the main leading contenders being "4.0" (on the grounds that so many changes will have accumulated since 2008's 3.0 release by then that it makes sense to call them different major versions, similar to the rationale the Linux kernel now uses for major version updates), and "3.10" (on the grounds that the release won't be any more different from 3.9 than 3.9 will be from 3.8). Other variants (like making the major release number "40" or "2022", and using the minor version field for some new purpose other than indicating compatibility breaks in the compiler AST, interpreter opcode format, code caching tags, and C ABI), tend to introduce the same pragmatic questions that will already need to be considered in choosing between the 3.10 and 4.0 alternatives. It's not just the version numbering aesthetics that need to be considered either - in addition to the point you raise around how the Python version gets embedded in file path names (such that the 3-digit form is technically ambiguous without a separator, and may break parsers that are expecting exactly two digits), there are other backwards compatibility concerns around how folks do version compatibility checks based on sys.version and sys.version_info. Python and CPython will be more than 3 decades old by 2022, and an ecosystem can build up a *lot* of implicit assumptions regarding how a platform's versioning scheme works in that kind of time frame :) Personally, I doubt we'll make a firm decision on how we're going to handle this until some time after 2.7 goes EOL in 2020, as by then we'll necessarily have a better idea of how the various Linux distros (including the LTS commercial ones) ended up handling the Python 2 -> Python 3 switch, and the fact that the next release at that point will be 3.9 making it eminently clear that we've run out of time to continue postponing the question. We'll hopefully also have more experience with folks relying on the stable runtime ABI, and hence whether or not it might be beneficial to define new "py4" and "abi4" compatibility tags for the binary wheel distribution format. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Proposal to change Python version release cycle
On 5 November 2017 at 00:40, Wolfgang <tds...@mailbox.org> wrote: > > > On 04.11.2017 14:29, Antoine Pitrou wrote: >> >> >> Hello Wolfgang, >> >> On Sat, 4 Nov 2017 12:25:57 +0100 (CET) >> tds...@mailbox.org wrote: >>> >>> Hello, >>> >>> one of my long standing ideas to improve Python is to adjust the >>> release cycle and version number handling. In short, to simplify it. >> >> >> There has been ample discussion in the past about changing our release >> cycle one way or another. In short, the meta-problem is there are many >> contradicting interests which would each favour a different solution to >> the problem. See for example this PEP (ultimately rejected): >> https://www.python.org/dev/peps/pep-0407/ >> >> and the discussion that ensued: >> >> https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 >> >> I haven't read your proposal in detail, but I suspect that it may be >> vulnerable to some of the same objections. The big objection being >> that a significant part of our ecosystem (that is, to put it roughly, >> the more corporate-minded segment, though I believe it is a >> simplification and may include other segments, such as Debian stable >> users and maintainers) doesn't want to deal more frequent feature >> releases. > > > I read this PEP and some of the discussion. > > The difference to my idea is not to propose a LTS version and feature > preview releases. > > The simplest form of my change is to switch from today major.minor to simply > major for the feature release cycle. We're currently more likely to go the other direction, and stick with the 3.x numbering for an extended period (potentially reaching 3.10, 3.11, 3.12, etc), so that the ongoing disruption caused by the 2.x -> 3.x transition has had a chance to settle down before we ask anyone to start calling anything "Python 4". While the problems Linux distros are facing with whether "python" should refer to "python2" or "python3" are at least arguably self-inflicted (especially for those that already dealt with the Python 1.5.2 -> Python 2.0 migration back in the early 2000s), nobody is really looking forward to having to figure out how to adjust to a potential future Python 4.0 that invalidates all the current "python3" based naming schemes that have been introduced to work around the fact that Python 3.x needed to be parallel installable with Python 2.x without breaking any existing Python 2.x applications. We're not even sure yet when we're going to update PEP 394 to say that we think it's reasonable for distros to start using "python" to mean "python3" (see https://www.python.org/dev/peps/pep-0394/ and https://github.com/python/redistributor-guide/issues/1 for more discussion on the latter point). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Proposal to change Python version release cycle
On 4 November 2017 at 23:29, Antoine Pitrou <solip...@pitrou.net> wrote: > > Hello Wolfgang, > > On Sat, 4 Nov 2017 12:25:57 +0100 (CET) > tds...@mailbox.org wrote: >> Hello, >> >> one of my long standing ideas to improve Python is to adjust the >> release cycle and version number handling. In short, to simplify it. > > There has been ample discussion in the past about changing our release > cycle one way or another. In short, the meta-problem is there are many > contradicting interests which would each favour a different solution to > the problem. See for example this PEP (ultimately rejected): > https://www.python.org/dev/peps/pep-0407/ > > and the discussion that ensued: > https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 PEP 413 was another competing proposal from around the same time: https://www.python.org/dev/peps/pep-0413/ Technically neither proposal has been formally Rejected, but they're also Deferred/Withdrawn because we knew that if we *had* asked for an explicit determination, the answer would have been "No, too high a cost, for not enough of a demonstrated benefit" (and that hasn't really changed in the past 5 years). > I haven't read your proposal in detail, but I suspect that it may be > vulnerable to some of the same objections. The big objection being > that a significant part of our ecosystem (that is, to put it roughly, > the more corporate-minded segment, though I believe it is a > simplification and may include other segments, such as Debian stable > users and maintainers) doesn't want to deal more frequent feature > releases. It's not just redistributors that get affected - it's the compatibility testing matrices for community projects as well. With our current release cycles (and excluding the 2.7 LTS release from consideration) we tend to have the following situation for the 3.x branches: - 1 actively maintained version - 2 or 3 security-fix only branches This stems from the default commitment of security fixes ending around 5 years after an X.Y.0 release, and releases happening 18-24 months apart: * Month 0: X.Y.0 released * Month 18: X.(Y+1).0 released * Month 36: X.(Y+2).0 released * Month 54: X.(Y+3).0 released * Month 60: X.Y.Z goes end-of-life * Month 72: X.(Y+4).0 released That same pattern also held for the 2.x series until the 2.7 release in 2010 (and it only changed then because that was the last 2.x feature release). The current year or so where the number of supported branches drops down to only 3 means we have some scope to speed up the rate of feature releases a bit (i.e. a release cadence of every 15 months instead of every 18 still means the community compatibility testing matrix consistently stays around 4 versions, or 5 if you're also testing against the dev release), but going to 6 monthly version updates means folks will either keep their test matrices the same as they are now (increasing the odds of the standard-library-only updates breaking things and that not being detected for some time), or else feeling obliged to expand their test matrices to cover even more concurrently supported versions. There are potential ways around both of those problems, but they require making the defined support periods for minor releases significantly shorter, unless they're the last minor release before the next major release (which means that instead of being simpler, these kinds of arrangements tend to end up being *more* complicated than the "every feature release receives security updates for 5 years" status quo). So rather than making fundamental changes to the way Python is versioned just to make standard library updates more readily available on older base versions of Python, we've instead gone with a more needs driven approach: making backport modules available for older versions where we find it is beneficial to do so (e.g. contextlib2, unittest2, importlib2). While Python 2.7 is a major driver for that change in practice, the benefits apply to older 3.x releases as well: - the updated modules are available for all Python versions and implementations where the packaging tools work (which is most of them) - users can decide on a module-by-module basis whether they want the baseline version or fast track updates - those decisions are explicitly tracked in the project's dependency metadata, rather than being implicit in the choice of deployment platform Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] install pip packages from Python prompt
On 3 November 2017 at 02:22, Ivan Levkivskyi <levkivs...@gmail.com> wrote: > Just another random idea: What about simply having two menu items in IDLE: > > * Install/update package manager > * Open package manager > > The first one will install the pipgui from PyPI (and pip if not already > installed). > The second one will run the GUI. > > This way it looks like pipgui can be simply published on PyPI without > special-casing at all, or am I missing something? This would also deal with the case where "ensurepip" hadn't been run at install time for some reason. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] install pip packages from Python prompt
On 2 November 2017 at 18:46, Nathaniel Smith <n...@pobox.com> wrote: > But unfortuately this hasn't been standardized, and there's currently > no way to do the lookup from the stdlib, so maybe this is not so > helpful for IDLE... > The entry point file format was recently promoted to a PyPA interoperability spec (without a PEP, as we documented it as-is, rather than changing anything): https://packaging.python.org/specifications/entry-points/ While the point about the standard library lacking the ability to read the metadata for installed packages still stands, it's also not too hard to implement a rudimentary version that just iterates over sys.path looking for `entry_points.txt` files in `*.dist-info` subdirectories. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 2 November 2017 at 04:29, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> SPOILER ALERT! At the moment, Nick's statement is in fact **always** true
> in **all** cases (at least when ignoring untypical cases and some
> inaccuracies in phrasing). Another question is whether the statement
> **should** be true at all.
>
>
> PyErr_CheckSignals(), the function that checks for pending signals, now
> **implicitly** uses the strictest possible memory-order requirement
> (SEQ_CST) for checking the `is_tripped` flag, a value which can be used to
> peek whether there are any pending signals. This means that two threads
> that call PyErr_CheckSignals can't even **check** the value of that flag
> at the same time, and they'll have to wait for each other and whatever the
> CPU hardware implementation does for cache synchronzation.
>
Nice, that's deeper than I went - I just assumed there was an
interlocked_increment in there somewhere, without checking whether or not
there were any optimised code paths that managed to avoid that call :)
> From a correctness point of view, that is absolutely great: if
> PyErr_CheckSignals() is called, it is guaranteed to notice a new signal
> regardles of how small the number of picoseconds after the `is_tripped`
> flag has been set. But is that really important? Do we really need things
> to be slowed down by this? And do we want people to "optimize" code by
> avoiding signal checking?
>
It isn't signal handling latency that's the concern here, it's potentially
missing signals. Consider the case where we have 3 threads: A, B, C. A is
the main thread that will actually handle signals, B and C both happened to
receive them. We'll also assume the two threads receive *different*
signals (otherwise they'll potentially get collapsed into one notification
regardless).
The scenario we want to avoid is having one or both of the signals set, but
is_tripped cleared. With an interlocked check (where both 0->1 and 1->0 are
memory barriers), that clearly can't happen. I suspect you're right that
this could also be achieved with a less strict memory sync requirement on
the 0->1 check, but that's much harder to reason about than simply limiting
the number of iterations we make through an iterator consumption loop
before checking for signals.
> The signals won't be caught until checked anyway, and as we've seen, one
> solution is to use a counter to determine if enough time has passed that
> another check for potential signals should happen. That does, however,
> raise the question of who should own the counter, and if it's OK for
> multiple threads to use the same counter. If yes, then would we be able to
> avoid slow atomic decrements (or increments)?
>
> But another solution might be to make a less strict but almost equivalent
> functionality with much less overhead. Something like this in a header file:
>
> static inline int PyErr_PROBE_SIGNALS() {
> static int volatile *flag = (int volatile *) _tripped;
> if (*flag) {
> return PyErr_CheckSignals();
> }
> else {
> return 0;
> }
> }
>
> Here, the flag is casted to volatile to force a check to happen each time
> from memory/cache. However, to make it more standard and to make sure it
> works with all compilers/platforms, it might be better to, instead of
> *flag, use an atomic load with "MEMORY_ORDER_RELAXED". Another thing is
> that `is_tripped`, which is currently declared as static inside
> signalmodule.c [4], should somehow be exposed in the API to make it
> available for the inline function in headers.
>
> This solution might be fast enough for a lot of cases, although still
> slightly slower than the counter approach, at least if the counter approach
> would completely avoid per-iteration memory access by requiring each
> function to own the counter that is used for this.
>
One thing I like about a nested inner loop is that it's easy to understand
the rationale for it *without* knowing any of the details of how memory
synchronisation works, as it's just:
- we want to check for signals regularly so the latency in interrupt
handling isn't too high
- PyErr_CheckSignals() is too expensive to call on every iteration
- so we only check every N iterations
Memory synchronisation then only comes up if someone asks why
"PyErr_CheckSignals" is expensive to call. And while it wouldn't surprise
at all if you're right and there are ways to make that call cheaper,
they're still never going to be cheaper than explicitly reducing the
frequency with which it is called.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] install pip packages from Python prompt
On 1 November 2017 at 08:50, Terry Reedy <tjre...@udel.edu> wrote: > In April 2016, after posting the idea to pydev list and getting 'go > ahead's from Nick Coughlin and someone else, with no negatives, I approved > Upendra Kumar's GSOC proposal to write a pip gui. This was > https://bugs.python.org/issue27051. On June 20, Ned Deily and Nick > Coughlin vetoed adding a pip gui anywhere in the stdlib since it depended > on something not in the stdlib, and perhaps for other reasons I don't fully > understand. > Clarifying the objection here (since the linked issue is a fairly long one): what I'm against is tightly coupling the pip-gui development & release process to the CPython development & release process when we don't have any compelling reason to do so. While PEP 434 provides a workaround that helps to keep IDLE consistent across different support branches, it still isn't as robust a development model as folks being able to install an application and try it out on existing Python versions *before* we bundle it with a CPython release. https://packaging.python.org/tutorials/distributing-packages/ provides a general guide on how to publish new packages, and the combination of tox and Travis CI makes it reasonably straightforward to run pre-merge CI testing across multiple Python versions. Given an independently released pip-gui on PyPI (with its own version numbering and release cadence), then I'd be +1 on bundling that as an optional IDLE addon, ensurepip style. Such a project could also lay the foundation for switching IDLE itself to a similar bundling model, which would allow the need for PEP 434 to be eliminated as well (since bundled applications are already permitted to add new features in maintenance releases). Cheers, Nick. P.S. I'll also note that a useful feature Travis CI offers is the ability to automate PyPI releases: https://docs.travis-ci.com/user/deployment/pypi/ -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"
On 1 November 2017 at 05:56, Guido van Rossum <gu...@python.org> wrote: > On Tue, Oct 31, 2017 at 12:24 PM, MRAB <pyt...@mrabarnett.plus.com> wrote: > >> At least Python 3.6 is only 1 year/release behind, which is fine! >> > > OK, so presumably that argument doesn't preclude inclusion in the 3.7 (or > later) stdlib. I'm beginning to warm up to the idea again... Maybe we > should just bite the bullet. Nick, what do you think? Is it worth a small > PEP? > I'm personally still in favour of swapping out the current _sre based implementation for a _regex based implementation (such that 3.7+ still only contained a single regex engine, and the stdlib re module and a PyPI regex backport module could still happily coexist), and a PEP + draft patch would be the way to do that. The framing of a PEP for that approach would be "Replace the regex engine backing the re module" rather than "Add regex to the standard library". The existing engine could then potentially be spun out as a new "sre" project on PyPI, such that folks that *were* depending on _sre internals had access to an upgrade path that didn't necessarily require them porting their code to a new regex engine (the PEP author wouldn't have to commit to doing that work - we'd just ask the PyPI admins to reserve the name in case there proved to be demand for such a project). However, I'm also not one of the folks that would be on the hook for handling any compatibility regressions that were subsequently reported against the 3.7 re module, so I'd also take my +1 with a rather large grain of salt - it's easy to be positive about a plan when the potential downsides don't affect me personally :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"
On 1 November 2017 at 00:53, Guido van Rossum <gu...@python.org> wrote: > On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan <ncogh...@gmail.com> wrote: > >> On 31 October 2017 at 02:29, Guido van Rossum <gu...@python.org> wrote: >> >>> What's your proposed process to arrive at the list of recommended >>> packages? >>> >> >> I'm thinking it makes the most sense to treat inclusion in the >> recommended packages list as a possible outcome of proposals for standard >> library inclusion, rather than being something we'd provide a way to >> propose specifically. >> > > I don't think that gets you off the hook for a process proposal. We need > some criteria to explain why a module should be on the recommended list -- > not just a ruling as to why it shouldn't be in the stdlib. > The developer guide already has couple of sections on this aspect: * https://devguide.python.org/stdlibchanges/#acceptable-types-of-modules * https://devguide.python.org/stdlibchanges/#requirements I don't think either of those sections is actually quite right (since we've approved new modules that wouldn't meet them), but they're not terrible as a starting point in general, and they're accurate for the recommended packages use case. > We'd only use it in cases where a proposal would otherwise meet the >> criteria for stdlib inclusion, but the logistics of actually doing so don't >> work for some reason. >> > > But that would exclude most of the modules you mention below, since one of > the criteria is that their development speed be matched with Python's > release cycle. I think there must be some form of "popularity" combined > with "best of breed". In particular I'd like to have a rule that explains > why flask and Django would never make the list. (I don't know what that > rule is, or I would tell you -- my gut tells me it's something to do with > having their own community *and* competing for the same spot.) > The developer guide words this as "The module needs to be considered best-of-breed.". In some categories (like WSGI frameworks), there are inherent trade-offs that mean there will *never* be a single best-of-breed solution, since projects like Django, Flask, and Pyramid occupy deliberately different points in the space of available design decisions. Running the initial 5 proposals through that filter: >> >> * six: a cross-version compatibility layer clearly needs to be outside >> the standard library >> > > Hm... Does six still change regularly? If not I think it *would* be a > candidate for actual stdlib inclusion. Just like we added u"..." literals > to Python 3.4. > It still changes as folks porting new projects discover additional discrepancies between the 2.x and 3.x standard library layouts and behaviour (e.g. we found recently that the 3.x subprocess module's emulation of the old commands module APIs actually bit shifts the status codes relative to the 2.7 versions). The rate of change has definitely slowed down a lot since the early days, but it isn't zero. In addition, the only folks that need it are those that already care about older versions of Python - if you can start with whatever the latest version of Python is, and don't have any reason to support users still running older version, you can happily pretend those older versions don't exist, and hence don't need a compatibility library like six. As a separate library, it can just gracefully fade away as folks stop depending on it as they drop Python 2.7 support. By contrast, if we were to bring it into the 3.x standard library, then we'd eventually have to figure out when we could deprecate and remove it again. > * setuptools: we want to update this in line with the PyPA interop specs, >> not the Python language version >> > > But does that exclude stdlib inclusion? Why would those specs change, and > why couldn't they wait for a new Python release? > The specs mainly change when we want to offer publishers new capabilities while still maintaining compatibility with older installation clients (and vice-versa: we want folks still running Python 2.7 to be able to publish wheel files and use recently added metadata fields like Description-Content-Type). The reason we can't wait for new Python releases is because when we add such things, we need them to work on *all* supported Python releases (including 2.7 and security-release-only 3.x versions). There are also other drivers for setuptools updates, which include: - operating system build toolchain changes (e.g. finding new versions of Visual Studio or XCode) - changes to PyPI's operations (e.g. the legacy upload API getting turned off due to persistent service stability problems, switching to HTTPS only access) With setuptools as
Re: [Python-ideas] install pip packages from Python prompt
On 31 October 2017 at 05:57, Alex Walters <tritium-l...@sdamon.com> wrote: > > While I completely agree with this in principle, I think you > > overestimate the average beginner. > > Nope. I totally get that they don’t know what a shell or command prompt > is. THEY. NEED. TO. LEARN. Hiding it is not a good idea for anyone. If > this is an insurmountable problem for the newbie, maybe they really > shouldn’t be attempting to program. This field is not for everyone. > We're not in the business of making judgements about who should and shouldn't become Python programmers - we're in the business of making sure that Python is accessible to as many people as possible by removing irrelevant barriers to adoption, whether that's translating documentation so folks can start learning with instructions in their native language, or making it possible for them to defer learning the idiosyncrasies of the Windows, Linux, and Mac OS X command line environments. On the latter front, the details of the development interfaces offered by traditional desktop operating systems may *never* be relevant to the new generation of folks coming through that are learning to program by manipulating remote coding environments on tablets and other app-centric devices, just as most developers nowadays are able to get by without even learning C, let alone any flavour of assembly language. Our role in this process isn't to create future generations that think and work in exactly the same ways we do, it's to enable them to discover new ways of working that build on top of whatever we create. Jupyter notebooks are a decent example of this, where the difference between a Python statement and a "command line statement" is just an exclamation mark at the beginning of the line - exactly where the backing environment lives is mostly a hidden implementation detail from the user's perspective. Eclipse Che and other online coding environments are another case - there, the "command line" is a different window inside the editor app (this is also going to be a familiar option for heavy IDE users on traditional desktop operating systems). And putting it in those terms makes me think that we should explicitly exclude the default REPL from consideration here, as we've long taken the position that that *isn't* a good teaching environment, and you certainly can't access it remotely without some kind of other service in front of it to manage the network connection (even if that service is just ssh). That means I now see a few potential RFEs from this thread: 1. An import system feature that allows a running Python program to report a timestamp (with the same granularity as pyc file timestamp headers) for *when* the currently loaded modules were last modified. This could be as simple as a new `__mtime__` attribute in each module to store that number. 2. A new importlib.util API to check for potentially out of date modules in sys.modules (those where a freshly calculated module mtime doesn't match the stored __mtime__ attribute) 3. Support in IDLE for Jupyter-style "!" commands 4. Having IDLE call that importlib API and warn about any stale modules after each command line operation The first two features would be about enabling learning environments to more easily detect when the currently loaded modules may not match what's actually on disk (hot reloaders already do this by watching for filesystem changes, but we're currently missing a simpler polling based alternative that will also pick up package updates). The second two would be about enhancing IDLE's capabilities in this area, as we *do* suggest that as a reasonable initial learning environment, even though there are also plenty of alternatives out there now. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"
On 31 October 2017 at 02:29, Guido van Rossum <gu...@python.org> wrote: > What's your proposed process to arrive at the list of recommended packages? > I'm thinking it makes the most sense to treat inclusion in the recommended packages list as a possible outcome of proposals for standard library inclusion, rather than being something we'd provide a way to propose specifically. We'd only use it in cases where a proposal would otherwise meet the criteria for stdlib inclusion, but the logistics of actually doing so don't work for some reason. Running the initial 5 proposals through that filter: * six: a cross-version compatibility layer clearly needs to be outside the standard library * setuptools: we want to update this in line with the PyPA interop specs, not the Python language version * cffi: updates may be needed for PyPA interop specs, Python implementation updates or C language definition updates * requests: updates are more likely to be driven by changes in network protocols and client platform APIs than Python language changes * regex: we don't want two regex engines in the stdlib, transparently replacing _sre would be difficult, and _sre is still good enough for most purposes Of the 5, I'd suggest that regex is the only one that could potentially still make its way into the standard library some day - it would just require someone with both the time and inclination to create a CPython variant that used _regex instead of _sre as the default regex engine, and then gathered evidence to show that it was "compatible enough" with _sre to serve as the default engine for CPython. For the first four, there are compelling arguments that their drivers for new feature additions are such that their release cycles shouldn't ever be tied to the rate at which we update the Python language definition. > And is it really just going to be a list of names, or is there going to be > some documentation (about the vetting, not about the contents of the > packages) for each name? > I'm thinking a new subsection in https://docs.python.org/devguide/stdlibchanges.html for "Recommended Third Party Packages" would make sense, covering what I wrote above. It also occurred to me that since the recommendations are independent of the Python version, they don't really belong in the version specific documentation. While the Developer's Guide isn't really the right place for the list either (except as an easier way to answer "Why isn't in the standard library?" questions), it could be a good interim option until I get around to actually writing a first draft of https://github.com/python/redistributor-guide/ (which I was talking to Barry about at the dev sprint, but didn't end up actually creating any content for since I went down a signal handling rabbit hole instead). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] install pip packages from Python prompt
On 31 October 2017 at 02:06, Paul Moore <p.f.mo...@gmail.com> wrote: > On 30 October 2017 at 15:53, Antoine Pitrou <solip...@pitrou.net> wrote: > > On Tue, 31 Oct 2017 01:44:10 +1000 > > Nick Coghlan <ncogh...@gmail.com> wrote: > >> (We'd want a real process restart, rather than emulating it by calling > >> Py_Initialize & Py_Finalize multiple times, as not every module properly > >> supports multiple initialise/finalise cycles within a single process, > and > >> module-specific quirks are exactly what we'd be trying to avoid by > forcing > >> an interpreter restart) > > > > The main difference, though, is that a notebook will reload and > > replay all your session, while restarting the regular REPL will simply > > lose all current work. I think that makes the idea much less > > appealing. > Right, but if you want an installation to work reliably, you're going to lose that state anyway. Erik's original comment included the suggestion to "give them an opportunity to cancel the action in case they have any work they need to save", and I think some kind of warning's going to be necessary no matter how we handle the restart. > Also, on Windows, I believe that any emulation of execve either leaves > the original process in memory, or has problems getting console > inheritance right. It's been a long time since I worked at that level, > and things may be better now, but getting a robust "restart this > process" interface in Windows would need some care (that's one of the > reasons the py launcher runs Python as a subprocess rather than doing > any sort of exec equivalent). > As long as the standard streams are passed along correctly, whatever the py launcher does would presumably be adequate for a REPL restart as well, assuming we decided to go down that path. It would also be reasonable to say that the regular REPL just issues a warning that a restart might be needed, and it's only REPLs with a separate client process that offer a way to restart the subprocess where code actually executes. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"
On 31 October 2017 at 00:28, Guido van Rossum <gu...@python.org> wrote:
> I just feel that when you're talking about an org like PayPal they can
> take care of themselves and don't need our help. They will likely have
> packages they want installed everywhere that would never make in on your
> list. So it feels this whole discussion is a distraction and a waste of
> time (yours, too).
>
Just because companies are big doesn't mean they necessarily have anyone
internally that's already up to speed on the specifics of recommended
practices in a sprawling open source community like Python's. The genesis
of this idea is that I think we can offer a more consistent initial
experience for those folks than "Here's PyPI and Google, y'all have fun
now" (and in so doing, help folks writing books and online tutorials to
feel more comfortable with the idea of assuming that libraries like
requests will be available in even the most restrictive institutional
environments that still allow the use of Python).
One specific situation this idea is designed to help with is the one where:
- there's a centrally managed Standard Operating Environment that dictates
what gets installed
- they've approved the python.org installers
- they *haven't* approved anything else yet
Now, a lot of large orgs simply won't get into that situation in the first
place, since their own supplier management rules will push them towards a
commercial redistributor, in which case they'll get their chosen
redistributor's preferred package set, which will then typically cover at
least a few hundred of the most popular PyPI packages.
But some of them will start from the narrower "standard library only"
baseline, and I spent enough time back at Boeing arguing for libraries to
be added to our approved component list to appreciate the benefits of
transitive declarations of trust ("we trust supplier X, they unambigously
state their trust in supplier Y, so that's an additional point in favour of
our also trusting supplier Y") when it comes time to make your case to your
supplier management organisation. Such declarations still aren'y always
sufficient, but they definitely don't hurt, and they sometimes help.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] install pip packages from Python prompt
On 30 October 2017 at 20:35, Erik Bray <erik.m.b...@gmail.com> wrote:
> I should add--another case that is becoming extremely common is
> beginners learning Python for the first time inside the
> Jupyter/IPython Notebook. And in my experience it can be very
> difficult for beginners to understand the connection between what's
> happening in the notebook ("it's in the web-browser--what does that
> have to do with anything on my computer??") and the underlying Python
> interpreter, file system, etc. Being able to pip install from within
> the Notebook would be a big win. This is already possible since
> IPython allows running system commands and it is possible to run the
> pip executable from the notebook, then manually restart the Jupyter
> kernel.
>
> It's not 100% clear to me how my proposal below would work within a
> Jupyter Notebook, so that would also be an angle worth looking into.
>
A few specific notes here:
1. As you say, this sort of already works in notebooks, since instructors
can say to run "!pip install requests" and then restart the language kernel.
2. We could probably replicate that style in IDLE, since that runs user
code in a subprocess, similar to the way Jupyter language kernels are
separate from the frontend client
3. We can't replicate it as readily in the regular REPL, since that runs
Python code directly in the current process, but even there I believe we
could potentially trigger a full process restart via execve (or the C++
style _execve on Windows)
(We'd want a real process restart, rather than emulating it by calling
Py_Initialize & Py_Finalize multiple times, as not every module properly
supports multiple initialise/finalise cycles within a single process, and
module-specific quirks are exactly what we'd be trying to avoid by forcing
an interpreter restart)
So the main missing piece if we went down that path would be to offer a way
to say from within the interpreter itself "Restart the current interactive
session". One possible approach to that would be to define a
RestartInterpreter subclass of SystemExit, which the interpreter would
intercept at around the same point where it checks for the PYTHONINSPECT
flag, and then initiate a graceful process shutdown and restart, rather
than a normal exit. We'd probably want that capability to be off by default
and enable it explicitly from the CPython CLI though, as otherwise it could
have some really annoying side effects in runtime embedding use cases.
I'm sure there'd be some thorny edge cases that would arise in trying to
make this work in practice, but at first glance, the general idea sounds
potentially feasible to me.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"
On 30 October 2017 at 04:56, Guido van Rossum <gu...@python.org> wrote: > The two use cases you describe (scripters and teachers) leave me luke-warm > -- scripters live in the wild west and can just pip install whatever > (that's what it means to be scripting) > For personal scripting, we can install whatever, but "institutional scripting" isn't the same thing - there we're scripting predefined "Standard Operating Environments", like those Mahmoud Hashemi describes for PayPal at the start of https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/ "Just use PyInstaller" isn't an adequate answer in large-scale environments because of the "zlib problem": you don't want to have to rebuild and redeploy the world to handle a security update in a low level frequently used component, you want to just update that component and have everything else pick it up dynamically. While "Just use conda" is excellent advice nowadays for any organisation contemplating defining their own bespoke Python SOE (hence Mahmoud's post), if that isn't being driven by the folks that already maintain the SOE (as happened in PayPal's case) convincing an org to add a handful of python-dev endorsed libraries to an established SOE is going to be easier than rebasing their entire Python SOE on conda. > and teachers tend to want a customized bundle anyway -- let the edu world > get together and create their own recommended bundle. > > As long as it's not going to be bundled, i.e. there's just going to be > some list of packages that we recommend to 3rd party repackagers, then I'm > fine with it. But they must remain clearly marked as 3rd party packages in > whatever docs we provide, and live in site-packages. > Yep, that was my intent, although it may not have been clear in my initial proposal. I've filed two separate RFEs in relation to that: * Documentation only: https://bugs.python.org/issue31898 * Regression testing resource: https://bugs.python.org/issue31899 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Composition over Inheritance
On 29 October 2017 at 21:44, Soni L. <fakedme...@gmail.com> wrote: > ORMs use this kind of descriptor based composition management extensively > in order to reliably model database foreign key relationships in a way > that's mostly transparent to users of the ORM classes. > > > And this is how you miss the whole point of being able to dynamically > add/remove arbitrary components on objects you didn't create, at runtime. > You can already do that by adding new properties to classes post-definition, or by changing __class__ to refer to a different type, or by wrapping objects in transparent proxy types the way wrapt does. We *allow* that kind of thing, because it's sometimes beneficial in order to get two libraries to play nicely together at runtime without having to patch one or the other. However, it's a last resort option that you use when you've exhausted the other more maintainable alternatives, not something we actually want to encourage. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"
On 29 October 2017 at 15:16, Guido van Rossum <gu...@python.org> wrote: > Why? What's wrong with pip install? > At a technical level, this would just be a really thin wrapper around 'pip install' (even thinner than ensurepip in general, since these libraries *wouldn't* be bundled for offline installation, only listed by name). > Why complicate things? Your motivation is really weak here. "beneficial"? > "difficult cases"? > The main recurring problems with "pip install" are a lack of discoverability and a potential lack of availability (depending on the environment). This then causes a couple of key undesirable outcomes: - folks using Python as a teaching language have to choose between teaching with just the standard library APIs, requiring that learners restrict themselves to a particular preconfigured learning environment, or make a detour into package management tools in order to ensure learners have access to the APIs they actually want to use (this isn't hypothetical - I was a technical reviewer for a book that justified teaching XML-RPC over HTTPS+JSON on the basis that xmlrpc was in the standard library, and requests wasn't) - folks using Python purely as a scripting language (i.e without app level dependency management) may end up having to restrict themselves to the standard library API, even when there's a well-established frequently preferred alternative for what they're doing (e.g. requests for API management, regex for enhanced regular expressions) The underlying problem is that our reasons for omitting these particular libraries from the standard library relate mainly to publisher side concerns like the logistics of ongoing bug fixing and support, *not* end user concerns like software reliability or API usability. This means that if educators aren't teaching them, or redistributors aren't providing them, then they're actively doing their users a disservice (as opposed to other cases like web frameworks and similar, where there are multiple competing options, you're only going to want one of them in any given application, and the relevant trade-offs between the available options depend greatly on exactly what you're doing) Now, the Python-for-data-science community have taken a particular direction around handling this, and there's an additional library set beyond the standard library that's pretty much taken for granted in a data science context. While conda has been the focal point for those efforts more recently, it started a long time ago with initiatives like Python(x, y) and the Enthought Python Distribution. Similarly, initiatives like Raspberry Pi are able to assume a particular learning environment (Raspbian in the Pi's case), rather than coping with arbitrary starting points. Curated lists like the "awesome-python" one that Stephan linked don't really help that much with the discoverability problem, since they become just another thing for people to learn: How do they find out such lists exist in the first place? Given such a list, how do they determine if the recommendations it offers are actually relevant to their needs? Since assessing a published package API against your needs as a user is a skill that has to be learned like any other, it can be a lot easier to get started in a more prescriptive environment that says "This is what you have to work with for now, we'll explain more about your options for branching out later". The proposal in this thread thus stems from asking the question "Who is going to be best positioned to offer authoritative advice on which third party modules may be preferable to their standard library counterparts for end users of Python?" and answering it with "The standard library module maintainers that are already responsible for deciding whether or not to place appropriate See Also links in the module documentation". All the proposal does is to suggest taking those existing recommendations from the documentation and converting them into a more readibly executable form. I'm not particularly wedded to any particular approach to making the recommendations available in a more machine-friendly form, though - it's just the "offer something more machine friendly than scraping the docs for recommendation links" aspect that I'm interested in. For example, we could skip touching ensurepip or venv at all, and instead limit this to a documentation proposal to collect these recommendations from the documentation, and publish them within the `venv` module docs as a "recommended-libraries.txt" file (using pip's requirements.txt format). That would be sufficient to allow straightforward 3rd party automation, without necessarily committing to providing such automation ourselves. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Pytho
Re: [Python-ideas] Composition over Inheritance
On 29 October 2017 at 12:25, Brendan Barnwell <brenb...@brenbarn.net> wrote:
> On 2017-10-28 19:13, Soni L. wrote:
>
>> And to have all cars have engines, you'd do:
>>
>> class Car:
>> def __init__(self, ???):
>> self[Engine] = GasEngine()
>>
>> car = Car()
>> car[Engine].kickstart() # kickstart gets the car as second argument.
>>
>> And if you can't do that, then you can't yet do what I'm proposing, and
>> thus the proposal makes sense, even if it still needs some refining...
>>
>
> As near as I can tell you can indeed do that, although it's still
> not clear to me why you'd want to. You can give Car a __getitem__ that
> on-the-fly generates an Engine object that knows which Car it is attached
> to, and then you can make Engine.kickstart a descriptor that knows which
> Engine it is attached to, and from that can figure out which Car it is
> attached to.
>
Right, I think a few different things are getting confused here related to
how different folks use composition.
For most data modeling use cases, the composition model you want is either
a tree or an acyclic graph, where the subcomponents don't know anything
about the whole that they're a part of. This gives you good component
isolation, and avoids circular dependencies.
However, for other cases, you *do* want the child object to be aware of the
parent - XML etrees are a classic example of this, where we want to allow
navigation back up the tree, so each node gains a reference to its parent
node. This often takes the form of a combination of delegation
(parent->child references) and dependency inversion (child->parent
reference).
For the car/engine example, this relates to explicitly modeling the
relationship whereby a car can have one or more engines (but the engine may
not currently be installed), while an engine can be installed in at most
one car at any given point in time.
You don't even need the descriptor protocol for that though, you just need
the subcomponent to accept the parent reference as a constructor parameter:
class Car:
def __init__(self, engine_type):
self.engine = engine_type(self)
However, this form of explicit dependency inversion wouldn't work as well
if you want to be able to explicitly create an "uninstalled engine"
instance, and then pass the engine in as a parameter to the class
constructor:
class Car:
def __init__(self, engine):
self.engine = engine # How would we ensure the engine is marked as
installed here?
As it turns out, Python doesn't need new syntax for this either, as it's
all already baked into the regular attribute access syntax, whereby
descriptor methods get passed a reference not only to the descriptor, but
*also* to the object being accessed:
https://docs.python.org/3/howto/descriptor.html#descriptor-protocol
And then the property builtin lets you ignore the existence of the
descriptor object entirely, and only care about the original object,
allowing the above example to be written as:
class Car:
def __init__(self, engine):
self.engine = engine # This implicitly marks the engine as installed
@property
def engine(self):
return self._engine
@engine.setter
def engine(self, engine):
if engine is not None:
if self._engine is not None:
raise RuntimeError("Car already has an engine installed")
if engine._car is not None:
raise RuntimeError("Engine is already installed in another
car")
engine._car = self
self._engine = engine
car = Car(GasEngine())
ORMs use this kind of descriptor based composition management extensively
in order to reliably model database foreign key relationships in a way
that's mostly transparent to users of the ORM classes.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Defining an easily installable "Recommended baseline package set"
Over on python-dev, the question of recommending MRAB's "regex" module over the standard library's "re" module for more advanced regular expressions recently came up again. Because of various logistical issues and backwards compatibility risks, it's highly unlikely that we'll ever be able to swap out the current _sre based re module implementation in the standard library for an implementation based on the regex module. At the same time, it would be beneficial to have a way to offer an even stronger recommendation to redistributors that we think full-featured general purpose Python scripting environments should offer the regex module as an opt-in alternative to the baseline re feature set, since that would also help with other currently difficult cases like the requests module. What I'm thinking is that we could make some relatively simple additions to the `ensurepip` and `venv` modules to help with this: 1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library module names containing dependency specifiers for recommended third party packages for particular tasks (e.g. "regex" as an enhanced alternative to "re", "requests" as an enhanced HTTPS-centric alternative to "urllib.request") 2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap 3. Add a corresponding `--install-recommended flag to the `python -m ensurepip` CLI 4. Add a corresponding `--install-recommended flag to the `python -m venv` CLI (when combined with `--without-pip`, this would run pip directly from the bundled wheel file to do the installations) We'd also need either a new informational PEP or else a section in the developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES` are up to the individual module maintainers (hence keying the mapping by standard library module name, rather than having a single flat list for the entire standard library). For redistributors with weak dependency support, these reference interpreter level recommendations could become redistributor level recommendations. Redistributors without weak dependency support could still make a distinction between "default" installations (which would include them) and "minimal" installations (which would exclude them). Folks writing scripts and example code for independent distribution (i.e. no explicitly declared dependencies) could then choose between relying on just the standard library (as now), or on the standard library plus independently versioned recommended packages. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Dollar operator suggestion
On 27 October 2017 at 02:23, Chris Barker <chris.bar...@noaa.gov> wrote: > > > On Thu, Oct 26, 2017 at 6:32 AM, Paul Moore <p.f.mo...@gmail.com> wrote: > >> >> Procedural languages, and Python in particular, simply don't work like >> that. Functions have arbitrary numbers of arguments, > > > And can return an arbitrary number of values. OK, technically a single > tuple of values, but that does complicate the whole simple chaining thing. > > In short -- Python is not a functional language, even though is supports a > number of functional idioms. > https://bugs.python.org/issue1506122 has a brief discussion of the non-syntactic variant of this proposal: functools.compose(len, set, str)(foo) => -> len(set(str(foo))) The main concerns that resulted in the suggestion being rejected are: * it isn't clear to folks that aren't already familiar with FP why the call order for the composed functions should be right to left * it isn't clear why every function other than the rightmost one must accept a single positional arg * it isn't clear why every function other than the leftmost one must return a single result And it doesn't make sense to provide something more general, because if you're writing genuinely functional code, you do tend to abide by those restrictions. So given that our position is "We don't even want to add this to the standard library, because the learning curve for using it successfully is too steep", it's even less likely we'd be willing to add syntax for the operation. By contrast, "FP-for-Python" libraries like toolz [1] can make familiarity with those kinds of concepts and a willingness to abide by the related conventions a pre-requisite for using them. It's just opt-in, the same way that learning to define your own classes (rather than importing existing ones defined elsewhere) is opt-in. Cheers, Nick. [1] https://toolz.readthedocs.io/en/latest/ -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 19 October 2017 at 08:34, Greg Ewing <greg.ew...@canterbury.ac.nz> wrote: > Nick Coghlan wrote: > >> since breaking up the current single level loops as nested loops would be >> a pre-requisite for allowing these APIs to check for signals while they're >> running while keeping the per-iteration overhead low >> > > Is there really much overhead? Isn't it just checking a flag? > It's checking an atomically updated flag, so it forces CPU cache synchronisation, which means you don't want to be doing it on every iteration of a low level loop. However, reviewing Serhiy's PR reminded me that PyErr_CheckSignals() already encapsulates the "Should this thread even be checking for signals in the first place?" logic, which means the code change to make the itertools iterators inherently interruptible with Ctrl-C is much smaller than I thought it would be. That approach is also clearly safe from an exception handling point of view, since all consumer loops already need to cope with the fact that itr.__next__() may raise arbitrary exceptions (including KeyboardInterrupt). So that change alone already offers a notable improvement, and combining it with a __length_hint__() implementation that keeps container constructors from even starting to iterate would go even further towards making the infinite iterators more user friendly. Similar signal checking changes to the consumer loops would also be possible, but I don't think that's an either/or decision: changing the iterators means they'll be interruptible for any consumer, while changing the consumers would make them interruptible for any iterator, and having checks in both the producer & the consumer merely means that you'll be checking for signals twice every 65k iterations, rather than once. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 18 October 2017 at 22:53, Serhiy Storchaka <storch...@gmail.com> wrote: > 18.10.17 13:22, Nick Coghlan пише: > >> 2.. These particular cases can be addressed locally using existing >> protocols, so the chances of negative side effects are low >> > > Only the particular case `count() in count()` can be addressed without > breaking the following examples: > You're right, the potential impact on objects with weird __eq__ implementations would mean that even the `__contains__` approach would require deprecation warnings for the APIs that allow short-circuiting. So we can discard it in favour of exploring the "Can we make a beneficial improvement via __length_hint__?" question. > 3. The total amount of code involved is likely to be small (a dozen or so >> lines of C, a similar number of lines of Python in the tests) in >> well-isolated protocol functions, so the chances of introducing future >> maintainability problems are low >> > > It depends on what you want to achieve. Just prevent an infinity loop in > `count() in count()`, or optimize `int in count()`, or optimize more > special cases. > My interest lies specifically in reducing the number of innocent looking ways we offer to provoke uninterruptible infinite loops or unbounded memory consumption. > 4. We have a potential contributor who is presumably offering to do the >> work (if that's not the case, then the question is moot anyway until a >> sufficiently interested volunteer turns up) >> > > Maintaining is more than writing an initial code. > Aye, that's why the preceding point was to ask how large a change we'd be offering to maintain indefinitely, and how well isolated that change would be. > If we were to do that, then we *could* make the solution to the reported >> problem more general by having all builtin and standard library operations >> that expect to be working with finite iterators (the containment testing >> fallback, min, max, sum, any, all, functools.reduce, etc) check for a >> length hint, even if they aren't actually pre-allocating any memory. >> > > This will add a significant overhead for relatively short (hundreds of > items) sequences. I already did benchmarking for similar cases in the past. I did wonder about that, so I guess the baseline zero-risk enhancement idea would be to only prevent the infinite loop in cases that already request a length hint as a memory pre-allocation check. That would reduce the likelihood of the most painful case (grinding the machine to a halt), without worrying about the less painful cases (which will break the current process, but the rest of the machine will be fine). Given that, adding TypeError raising __length_hint__ implementations to itertools.count(), itertools.cycle(), and itertools.repeat() would make sense as an independent RFE, without worrying about any APIs that don't already check for a length hint. A more intrusive option would then be to look at breaking the other tight iteration loops into two phases, such that checking for potentially infinite iterators could be delayed until after the first thousand iterations or so. That option is potentially worth exploring anyway, since breaking up the current single level loops as nested loops would be a pre-requisite for allowing these APIs to check for signals while they're running while keeping the per-iteration overhead low (only one pre-requisite of many though, and probably one of the easier ones). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]
On 18 October 2017 at 22:51, Stefan Krah <ste...@bytereef.org> wrote: > On Wed, Oct 18, 2017 at 10:43:57PM +1000, Nick Coghlan wrote: > > Per-process memory quotas *can* help avoid this, but enforcing them > > requires that every process run in a resource controlled sandbox. Hence, > > it's not a coincidence that mobile operating systems and container-based > > server environments already work that way, and the improved ability to > cope > > with misbehaving applications is part of why desktop operating systems > > would like to follow the lead of their mobile and server counterparts :) > > Does this also fall under the sandbox definition? > > $ softlimit -m 10 python3 > Yeah, Linux offers good opt-in tools for this kind of thing, and the combination of Android and containerised server environments means they're only getting better. But we're still some time away from it being routine for your desktop to be well protected from memory management misbehaviour in arbitrary console or GUI applications. The resource module (which Steven mentioned in passing) already provides opt-in access to some of those features from within the program itself: https://docs.python.org/3/library/resource.html For example: >>> import sys, resource >>> data = bytes(2**32) >>> resource.setrlimit(resource.RLIMIT_DATA, (2**31, sys.maxsize)) >>> data = bytes(2**32) Traceback (most recent call last): File "", line 1, in MemoryError >>> resource.setrlimit(resource.RLIMIT_DATA, (sys.maxsize, sys.maxsize)) >>> data = bytes(2**32) (Bulk memory allocations start failing on my machine somewhere between 2**33 and 2**34, which is about what I'd expect, since it has 8 GiB of physical RAM installed) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]
On 18 October 2017 at 21:38, Steven D'Aprano <st...@pearwood.info> wrote: > > But should it be fixed in list or in count? > > Neither. There are too many other places this can break for it to be > effective to try to fix each one in place. > > e.g. set(xrange(2**64)), or tuple(itertools.repeat([1])) > A great many of these call operator.length_hint() these days in order to make a better guess as to how much memory to pre-allocate, so while that still wouldn't intercept everything, it would catch a lot of them. > Rather, I think we should set a memory limit that applies to the whole > process. Once you try to allocate more memory, you get an MemoryError > exception rather than have the OS thrash forever trying to allocate a > terabyte on a 4GB machine. > > (I don't actually understand why the OS can't fix this.) > Trying to allocate enormous amounts of memory all at once isn't the problem, as that just fails outright with "Not enough memory": >>> data = bytes(2**62) Traceback (most recent call last): File "", line 1, in MemoryError The machine-killing case is repeated allocation requests that the operating system *can* satisfy, but require paging almost everything else out of RAM. And that's exactly what "list(infinite_iterator)" entails, since the interpreter will make an initial guess as to the correct size, and then keep resizing the allocation to 125% of its previous size each time it fills up (or so - I didn't check the current overallocation factor) . Per-process memory quotas *can* help avoid this, but enforcing them requires that every process run in a resource controlled sandbox. Hence, it's not a coincidence that mobile operating systems and container-based server environments already work that way, and the improved ability to cope with misbehaving applications is part of why desktop operating systems would like to follow the lead of their mobile and server counterparts :) So here is my suggestion: > > 1. Let's add a function in sys to set the "maximum memory" available, > for some definition of memory that makes the most sense on your > platform. Ordinary Python programmers shouldn't have to try to decipher > the ulimit interface. > Historically, one key reason we didn't do that was because the `PyMem_*` APIs bypassed CPython's memory allocator, so such a limit wouldn't have been particularly effective. As of 3.6 though, even bulk memory allocations pass through pymalloc, making a Python level memory allocation limit potentially more viable (since it would pick up almost all of the interpeter's own allocations, even if it missed those in extension modules): https://docs.python.org/dev/whatsnew/3.6.html#optimizations Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 18 October 2017 at 20:39, Koos Zevenhoven <k7ho...@gmail.com> wrote: > On Oct 18, 2017 13:29, "Nick Coghlan" <ncogh...@gmail.com> wrote: > > On 18 October 2017 at 19:56, Koos Zevenhoven <k7ho...@gmail.com> wrote: > >> I'm unable to reproduce the "uninterruptible with Ctrl-C" problem with >> infinite iterators. At least itertools doesn't seem to have it: >> >> >>> import itertools >> >>> for i in itertools.count(): >> ... pass >> ... >> > > That's interrupting the for loop, not the iterator. This is the test case > you want for the problem Jason raised: > > >>> "a" in itertools.count() > > Be prepared to suspend and terminate the affected process, because Ctrl-C > isn't going to help :) > > > I'm writing from my phone now, cause I was dumb enough to try list(count()) > Yeah, that's pretty much the worst case example, since the machine starts thrashing memory long before it actually gives up and starts denying the allocation requests :( > But should it be fixed in list or in count? > That one can only be fixed in count() - list already checks operator.length_hint(), so implementing itertools.count.__length_hint__() to always raise an exception would be enough to handle the container constructor case. The open question would then be the cases that don't pre-allocate memory, but still always attempt to consume the entire iterator: min(itr) max(itr) sum(itr) functools.reduce(op, itr) "".join(itr) And those which *may* attempt to consume the entire iterator, but won't necessarily do so: x in itr any(itr) all(itr) The items in the first category could likely be updated to check length_hint and propagate any errors immediately, since they don't provide any short circuiting behaviour - feeding them an infinite iterator is a guaranteed uninterruptible infinite loop, so checking for a length hint won't break any currently working code (operator.length_hint defaults to returning zero if a type doesn't implement __length_hint__). I'm tempted to say the same for the APIs in the latter category as well, but their short-circuiting semantics mean those can technically have well-defined behaviour, even when given an infinite iterator: >>> any(itertools.count()) True >>> all(itertools.count()) False >>> 1 in itertools.count() True It's only the "never short-circuits" branch that is ill-defined for non-terminating input. So for these, the safer path would be to emit DeprecationWarning if length_hint fails in 3.7, and then pass the exception through in 3.8+. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 18 October 2017 at 19:56, Koos Zevenhoven <k7ho...@gmail.com> wrote: > I'm unable to reproduce the "uninterruptible with Ctrl-C" problem with > infinite iterators. At least itertools doesn't seem to have it: > > >>> import itertools > >>> for i in itertools.count(): > ... pass > ... > That's interrupting the for loop, not the iterator. This is the test case you want for the problem Jason raised: >>> "a" in itertools.count() Be prepared to suspend and terminate the affected process, because Ctrl-C isn't going to help :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 18 October 2017 at 03:39, Koos Zevenhoven <k7ho...@gmail.com> wrote: > On Tue, Oct 17, 2017 at 5:26 PM, Serhiy Storchaka <storch...@gmail.com> > wrote: > >> 17.10.17 17:06, Nick Coghlan пише: >> >>> Keep in mind we're not talking about a regular loop you can break out of >>> with Ctrl-C here - we're talking about a tight loop inside the interpreter >>> internals that leads to having to kill the whole host process just to get >>> out of it. >>> >> >> And this is the root of the issue. Just let more tight loops be >> interruptible with Ctrl-C, and this will fix the more general issue. >> >> > Not being able to interrupt something with Ctrl-C in the repl or with the > interrupt command in Jupyter notebooks is definitely a thing I sometimes > encounter. A pity I don't remember when it happens, because I usually > forget it very soon after I've restarted the kernel and continued working. > But my guess is it's usually not because of an infinite iterator. > Fixing the general case is hard, because the assumption that signals are only checked between interpreter opcodes is a pervasive one throughout the interpreter internals. We certainly *could* redefine affected C APIs as potentially raising KeyboardInterrupt (adjusting the signal management infrastructure accordingly), and if someone actually follows through and implements that some day, then the argument could then be made that given such change, it might be reasonable to drop any a priori guards that we have put in place for particular *detectable* uninterruptible infinite loops. However, that's not the design question being discussed in this thread. The design question here is "We have 3 known uninterruptible infinite loops that are easy to detect and prevent. Should we detect and prevent them?". "We shouldn't allow anyone to do this easy thing, because it would be preferable for someone to instead do this hard and complicated thing that nobody is offering to do" isn't a valid design argument in that situation. And I have a four step check for that which prompts me to say "Yes, we should detect and prevent them": 1. Uninterruptible loops are bad, so having fewer of them is better 2. These particular cases can be addressed locally using existing protocols, so the chances of negative side effects are low 3. The total amount of code involved is likely to be small (a dozen or so lines of C, a similar number of lines of Python in the tests) in well-isolated protocol functions, so the chances of introducing future maintainability problems are low 4. We have a potential contributor who is presumably offering to do the work (if that's not the case, then the question is moot anyway until a sufficiently interested volunteer turns up) As an alternative implementation approach, the case could also be made that these iterators should be raising TypeError in __length_hint__, as that protocol method is explicitly designed to be used for finite container pre-allocation. That way things like "list(itertools.count())" would fail immediately (similar to the way "list(range(10**100))" already does) rather than attempting to consume all available memory before (hopefully) finally failing with MemoryError. If we were to do that, then we *could* make the solution to the reported problem more general by having all builtin and standard library operations that expect to be working with finite iterators (the containment testing fallback, min, max, sum, any, all, functools.reduce, etc) check for a length hint, even if they aren't actually pre-allocating any memory. Then the general purpose marker for "infinite iterator" would be "Explicitly defines __length_hint__ to raise TypeError", and it would prevent a priori all operations that attempted to fully consume the iterator. That more general approach would cause some currently "working" code (like "any(itertools.count())" and "all(itertools.count())", both of which consume at most 2 items from the iterator) to raise an exception instead, and hence would require the introduction of a DeprecationWarning in 3.7 (where the affected APIs would start calling length hint, but suppress any exceptions from it), before allowing the exception to propagate in 3.8+. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 17 October 2017 at 23:17, Koos Zevenhoven <k7ho...@gmail.com> wrote: > On Tue, Oct 17, 2017 at 2:46 PM, Serhiy Storchaka <storch...@gmail.com> > wrote: > >> 17.10.17 14:10, Nick Coghlan пише: >> >>> 1. It's pretty easy to write "for x in y in y" when you really meant to >>> write "for x in y", and if "y" is an infinite iterator, the "y in y" part >>> will become an unbreakable infinite loop when executed instead of the >>> breakable one you intended (especially annoying if it means you have to >>> discard and restart a REPL session due to it, and that's exactly where that >>> kind of typo is going to be easiest to make) >>> >> >> I think it is better to left this on linters. > > > Just to note that there is currently nothing that would prevent making > `for x in y in z` a syntax error. There is nothing meaningful that it > could do, really, because y in z can only return True or False (or raise an > Exception or loop infinitely). > That was just an example of one of the ways we can accidentally end up writing "x in y" at the REPL, where "y" is an infinite iterator, since it's the kind that's specific to "x in y", whereas other forms (like accidentally using the wrong variable name) also apply to other iterator consuming APIs (like the ones Serhiy mentioned). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 17 October 2017 at 21:46, Serhiy Storchaka <storch...@gmail.com> wrote: > 17.10.17 14:10, Nick Coghlan пише: > >> 1. It's pretty easy to write "for x in y in y" when you really meant to >> write "for x in y", and if "y" is an infinite iterator, the "y in y" part >> will become an unbreakable infinite loop when executed instead of the >> breakable one you intended (especially annoying if it means you have to >> discard and restart a REPL session due to it, and that's exactly where that >> kind of typo is going to be easiest to make) >> > > I think it is better to left this on linters. I never encountered this > mistake and doubt it is common. In any case the first execution of this > code will expose the mistake. > People don't run linters at the REPL, and it's at the REPL where accidentally getting an unbreakable infinite loop is most annoying. Keep in mind we're not talking about a regular loop you can break out of with Ctrl-C here - we're talking about a tight loop inside the interpreter internals that leads to having to kill the whole host process just to get out of it. > 2. Containment testing already has a dedicated protocol so containers can >> implement optimised containment tests, which means it's also trivial for an >> infinite iterator to intercept and explicitly disallow containment checks >> if it chooses to do so >> > > But this has non-zero maintaining cost. As the one who made many changes > in itertools.c I don't like the idea of increasing its complexity for > optimizing a pretty rare case. > It's not an optimisation, it's a UX improvement for the interactive prompt. The maintenance burden should be low, as it's highly unlikely we'd ever need to change this behaviour again in the future (I do think deprecating the success case would be more trouble than it would be worth though). > And note that the comparison can have side effect. You can implement the > optimization of `x in count()` only for the limited set of builtin types. > For example `x in range()` is optimized only for exact int and bool. You > can't guarantee the finite time for cycle() and repeat() either since they > can emit values of arbitrary types, with arbitrary __eq__. We're not trying to guarantee finite execution time in general, we're just making it more likely that either Ctrl-C works, or else you don't get stuck in an infinite loop in the first place. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 17 October 2017 at 19:19, Serhiy Storchaka <storch...@gmail.com> wrote:
> 17.10.17 09:42, Nick Coghlan пише:
>
>> On 17 October 2017 at 16:32, Nick Coghlan <ncogh...@gmail.com > ncogh...@gmail.com>> wrote:
>>
>> So this sounds like a reasonable API UX improvement to me, but you'd
>> need to ensure that you don't inadvertently change the external
>> behaviour of *successful* containment tests.
>>
>>
>> I should also note that there's another option here beyond just returning
>> "False": it would also be reasonable to raise an exception like
>> "RuntimeError('Attempted negative containment check on infinite iterator')".
>>
>
> What about other operations with infinite iterators? min(count()),
> max(count()), all(count(1))? Do you want to implement special cases for all
> of them?
No, as folks expect those to iterate without the opportunity to break out,
and are hence more careful with them when infinite iterators are part of
their application. We also don't have any existing protocols we could use
to intercept them, even if we decided we *did* want to do so.
The distinction I see with "x in y" is:
1. It's pretty easy to write "for x in y in y" when you really meant to
write "for x in y", and if "y" is an infinite iterator, the "y in y" part
will become an unbreakable infinite loop when executed instead of the
breakable one you intended (especially annoying if it means you have to
discard and restart a REPL session due to it, and that's exactly where that
kind of typo is going to be easiest to make)
2. Containment testing already has a dedicated protocol so containers can
implement optimised containment tests, which means it's also trivial for an
infinite iterator to intercept and explicitly disallow containment checks
if it chooses to do so
So the problem is more likely to be encountered due to "x in y" appearing
in both the containment test syntax and as part of the iteration syntax,
*and* it's straightforward to do something about it because the
__contains__ hook already exists. Those two points together are enough for
me to say "Sure, it makes sense to replace the current behaviour with
something more user friendly".
If either of them was false, then I'd say "No, that's not worth the hassle
of changing anything".
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 17 October 2017 at 17:44, Steven D'Aprano <st...@pearwood.info> wrote:
> On Tue, Oct 17, 2017 at 04:42:35PM +1000, Nick Coghlan wrote:
>
> > I should also note that there's another option here beyond just returning
> > "False": it would also be reasonable to raise an exception like
> > "RuntimeError('Attempted negative containment check on infinite
> iterator')".
>
> I don't think that works, even if we limit discussion to just
> itertools.count() rather than arbitrary iterators. Obviously we
> cannot wait until the entire infinite iterator is checked (that
> might take longer than is acceptible...) but if you only check a
> *finite* number before giving up, you lead to false-negatives:
>
> # say we only check 100 values before raising
> 0 in itertools.count(1) # correctly raises
> 101 in itertools.count(1) # wrongly raises
>
Nobody suggested that, as it's obviously wrong. This discussion is solely
about infinite iterators that have closed form containment tests, either
because they're computed (itertools.count()), or because they're based on
an underlying finite sequence of values (cycle(), repeat()).
> If we do a computed membership test, then why raise at all? We quickly
> know whether or not the value is in the sequence, so there's no error to
> report.
>
Because we should probably always be raising for these particular
containment checks, and it's safe to start doing so in the negative case,
since that's currently a guaranteed infinite loop.
And unlike a "while True" loop (which has many real world applications),
none of these implicit infinite loops allow for any kind of useful work on
each iteration, they just end up in a tight loop deep inside the
interpreter internals, doing absolutely nothing. They won't even check for
signals or release the GIL, so you'll need to ask the operating system to
clobber the entire process to break out of it - Ctrl-C will be ignored.
I'd also have no major objection to deprecating containment tests on these
iterators entirely, but that doesn't offer the same kind of UX benefit that
replacing an infinite loop with an immediate exception does, so I think the
two questions should be considered separately.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 17 October 2017 at 16:32, Nick Coghlan <ncogh...@gmail.com> wrote:
> So this sounds like a reasonable API UX improvement to me, but you'd need
> to ensure that you don't inadvertently change the external behaviour of
> *successful* containment tests.
>
I should also note that there's another option here beyond just returning
"False": it would also be reasonable to raise an exception like
"RuntimeError('Attempted negative containment check on infinite iterator')".
That way currently working code would be semantically unchanged, but code
that otherwise results in an infinite loop would turn into an immediate
exception instead.
The rationale for this approach would be "What you are trying to do doesn't
really make sense, so we're going to complain about it, rather than just
giving you an answer".
The rationale against the error is that "If this item is present, advance
past it, otherwise don't do anything" would be an at least arguably
reasonable operation to request.
Whether "x in itr" is a reasonable *spelling* of that operation would then
be a different question that still favoured the "raise an exception"
approach.
Cheers,
NIck.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Membership of infinite iterators
On 16 October 2017 at 11:12, Jason Campbell <j_campbe...@hotmail.com> wrote: > I recently came across a bug where checking negative membership > (__contains__ returns False) of an infinite iterator will freeze the > program. > > It may seem a bit obvious, but you can check membership in range for > example without iterating over the entire range. > As Terry noted, this is due to a semantic difference between range (compact representation of a tuple of integers) and arbitrary iterators. However, if we can avoid seemingly innocent code triggering an infinite loop, that's probably a good thing. > `int(1e100) in range(int(1e101))` on my machine takes about 1.5us > `int(1e7) in itertools.count()` on my machine takes about 250ms (and gets > worse quite quickly). > > Any membership test on the infinite iterators that is negative will freeze > the program as stated earlier, which is odd to me. > The other relevant behavioural difference is that even a successful membership test will consume the iterator up to that point. > itertools.count can use the same math as range > +1, with the caveat that a successful containment test should still advance the iterator to the point immediately after the requested element, just as it does today (if people don't want that behaviour, they should be defining a suitable finite range instead). > itertools.cycle could use membership from the underlying iterable > Sort of - you'll still need to iterate through the underlying iterator at least once in order to see all of the candidate elements. However, the containment test could still be defined as running through the full cycle at most once, such that you end up either at the point immediately after the item or else back where you started (if the item wasn't found). > itertools.repeat is even easier, just compare to the repeatable element > +1 So this sounds like a reasonable API UX improvement to me, but you'd need to ensure that you don't inadvertently change the external behaviour of *successful* containment tests. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 16 October 2017 at 21:08, Juancarlo Añez <apal...@gmail.com> wrote: > > Interestingly, thinking about the problem in terms of exception handling >> flow reminded me of the fact that having a generator-iterator yield while >> inside a with statement or try/except block is already considered an >> anti-pattern in many situations, precisely because it means that any >> exceptions that get thrown in (including GeneratorExit) will be intercepted >> when that may not be what the author really intended. >> >> > It all works fine now: > > https://github.com/neogeny/TatSu/blob/master/tatsu/contexts.py > > > So, I have a strong requirement: whatever is decided on this PEP... > > Please don't break it? (or make it illegal) > The "anti-pattern in many situations" qualifier was there because there are cases where it's explicitly expected to work, and isn't an anti-pattern at all (e.g. when the generator is decorated with contextlib.contextmanager, or when you're using a context manager to hold open an external resource like a file until the generator is closed). So this wasn't intended as an argument for changing anything - rather, it's about my changing my perspective on how beneficial it would be to have generators default to maintaining their own distinct logical context (which then becomes an argument for *not* changing anything). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 15 October 2017 at 20:45, Paul Moore <p.f.mo...@gmail.com> wrote: > On 15 October 2017 at 06:43, Nick Coghlan <ncogh...@gmail.com> wrote: > > > # Generator form > > def _results_gen(data): > > for item in data: > > with adjusted_context(): > > yield calculate_result(item) > > > > results = _results_gen(data) > > > > Today, while these two forms look like they *should* be comparable, > they're > > not especially close to being semantically equivalent, as there's no > > mechanism that allows for implicit context reversion at the yield point > in > > the generator form. > > I'll have to take your word for this, as I can't think of an actual > example that follows the pattern of your abstract description, for > which I can immediately see the difference. > Interestingly, thinking about the problem in terms of exception handling flow reminded me of the fact that having a generator-iterator yield while inside a with statement or try/except block is already considered an anti-pattern in many situations, precisely because it means that any exceptions that get thrown in (including GeneratorExit) will be intercepted when that may not be what the author really intended. Accordingly, the canonical guaranteed-to-be-consistent-with-the-previous-behaviour iterator -> generator transformation already involves the use of a temporary variable to move the yield outside any exception handling constructs and ensure that the exception handling only covers the code that it was originally written to cover: def _results_gen(data): for item in data: with adjusted_context(): result_for_item = calculate_result(item) yield result_for_item results = _results_gen(data) The exception handling expectations with coroutines are different, since an "await cr" expression explicitly indicates that any exceptions "cr" fails to handle *will* propagate back through to where the await appears, just as "f()" indicates that unhandled exceptions in "f" will be seen by the current frame. And even if as a new API context variables were to be defined in a yield-tolerant way, a lot of existing context managers still wouldn't be "yield safe", since they may be manipulating thread local or process global state, rather than context variables or a particular object instance. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why not picoseconds?
On 16 October 2017 at 04:28, Victor Stinner <victor.stin...@gmail.com> wrote: > I proposed to use nanoseconds because UNIX has 1 ns resolution in > timespec, the most recent API, and Windows has 100 ns. > > Using picoseconds would confuse users who may expect sub-nanosecond > resolution, whereas no OS support them currently. > > Moreover, nanoseconds as int already landed in os.stat and os.utime. > And this precedent also makes sense to me as the rationale for using an "_ns" API suffix within the existing module rather than introducing a new module. > Last but not least, I already strugle in pytime.c to prevent integer > overflow with 1 ns resolution. It can quickly become much more complex if > there is no native C int type supporting a range large enough to more 1 > picosecond resolution usable. I really like using int64_t for _PyTime_t, > it's well supported, very easy to use (ex: "t = t2 - t1"). 64-bit int > supports year after 2200 for delta since 1970. > Hopefully by the time we decide it's worth worrying about picoseconds in "regular" code, compiler support for decimal128 will be sufficiently ubiquitous that we'll be able to rely on that as our 3rd generation time representation (where the first gen is seconds as a 64 bit binary float and the second gen is nanoseconds as a 64 bit integer). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 15 October 2017 at 15:49, Nathaniel Smith <n...@pobox.com> wrote:
> It's not like this is a new and weird concept in Python either -- e.g.
> when you raise an exception, the relevant 'except' block is determined
> based on where the 'raise' happens (the runtime stack), not where the
> 'raise' was written:
>
> try:
> def foo():
> raise RuntimeError
> except RuntimeError:
> print("this is not going to execute, because Python doesn't work that
> way")
> foo()
>
Exactly - this is a better formulation of what I was trying to get at when
I said that we want the semantics of context variables in synchronous code
to reliably align with the semantics of the synchronous call stack as it
appears in an exception traceback.
Attempting a pithy summary of PEP 550's related semantics for use in
explanations to folks that don't care about all the fine details:
The currently active execution context aligns with the expected flow of
exception handling for any exceptions raised in the code being executed.
And with a bit more detail:
* If the code in question will see the exceptions your code raises, then
your code will also be able to see the context variables that it defined or
set
* By default, this relationship is symmetrical, such that if your code will
see the exceptions that other code raises as a regular Python exception,
then you will also see the context changes that that code makes.
* However, APIs and language features that enable concurrent code execution
within a single operating system level thread (like event loops, coroutines
and generators) may break that symmetry to avoid context variable
management conflicts between concurrently executing code. This is the key
behavioural difference between context variables (which enable this by
design) and thread local variables (which don't).
* Pretty much everything else in the PEP 550 API design is a lower level
performance optimisation detail to make management of this dynamic state
sharing efficient in event-driven code
Even PEP 550's proposal for how yield would work aligns with that "the
currently active execution context is the inverse of how exceptions will
flow" notion: the idea there is that if a context manager's __exit__ method
wouldn't see an exception raised by a piece of code, then that piece of
code also shouldn't be able to see any context variable changes made by
that context manager's __enter__ method (since the changes may not get
reverted correctly on failure in that case).
Exceptions raised in a for loop body *don't* typically get thrown back into
the body of the generator-iterator, so generator-iterators' context
variable changes should be reverted at their yield points.
By contrast, exceptions raised in a with statement body *do* get thrown
back into the body of a generator decorated with contextlib.contextmanager,
so those context variable changes should *not* be reverted at yield points,
and instead left for __exit__ to handle.
Similarly, coroutines are in the exception handling path for the other
coroutines they call (just like regular functions), so those coroutines
should share an execution context rather than each having their own.
All of that leads to it being specifically APIs that already need to do
special things to account for exception handling flows within a single
thread (e.g. asyncio.gather, asyncio.ensure_future,
contextlib.contextmanager) that are likely to have to put some thought into
how they will impact the active execution context.
Code for which the existing language level exception handling semantics
already work just fine should then also be able to rely on the default
execution context management semantics.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 15 October 2017 at 15:05, Guido van Rossum <gu...@python.org> wrote: > I would like to reboot this discussion (again). It feels to me we're > getting farther and farther from solving any of the problems we might solve. > > I think we need to give up on doing anything about generators; the use > cases point in too many conflicting directions. So we should keep the > semantics there, and if you don't want your numeric or decimal context to > leak out of a generator, don't put `yield` inside `with`. (Yury and Stefan > have both remarked that this is not a problem in practice, given that there > are no bug reports or StackOverflow questions about this topic.) > Let me have another go at building up the PEP 550 generator argument from first principles. The behaviour that PEP 550 says *shouldn't* change is the semantic equivalence of the following code: # Iterator form class ResultsIterator: def __init__(self, data): self._itr = iter(data) def __next__(self): return calculate_result(next(self._itr)) results = _ResultsIterator(data) # Generator form def _results_gen(data): for item in data: yield calculate_result(item) results = _results_gen(data) This *had* been non-controversial until recently, and I still don't understand why folks suddenly decided we should bring it into question by proposing that generators should start implicitly capturing state at creation time just because it's technically possible for them to do so (yes we can implicitly change the way all generators work, but no, we can't implicitly change the way all *iterators* work). The behaviour that PEP 550 thinks *should* change is for the following code to become roughly semantically equivalent, given the constraint that the context manager involved either doesn't manipulate any shared state at all (already supported), or else only manipulates context variables (the new part that PEP 550 adds): # Iterator form class ResultsIterator: def __init__(self, data): self._itr = iter(data) def __next__(self): with adjusted_context(): return calculate_result(next(self._itr)) results = _ResultsIterator(data) # Generator form def _results_gen(data): for item in data: with adjusted_context(): yield calculate_result(item) results = _results_gen(data) Today, while these two forms look like they *should* be comparable, they're not especially close to being semantically equivalent, as there's no mechanism that allows for implicit context reversion at the yield point in the generator form. While I think PEP 550 would still be usable without fixing this discrepancy, I'd be thoroughly disappointed if the only reason we decided not to do it was because we couldn't clearly articulate the difference in reasoning between: * "Generators currently have no way to reasonably express the equivalent of having a context-dependent return statement inside a with statement in a __next__ method implementation, so let's define one" (aka "context variable changes shouldn't leak out of generators, as that will make them *more* like explicit iterator __next__ methods"); and * "Generator functions should otherwise continue to be unsurprising syntactic sugar for objects that implement the regular iterator protocol" (aka "generators shouldn't implicitly capture their creation context, as that would make them *less* like explicit iterator __init__ methods"). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 15 October 2017 at 14:53, M.-A. Lemburg <m...@egenix.com> wrote: > On 15.10.2017 06:39, Nick Coghlan wrote: > > On 15 October 2017 at 05:47, Paul Moore <p.f.mo...@gmail.com > > <mailto:p.f.mo...@gmail.com>> wrote: > > > > On 14 October 2017 at 17:50, Nick Coghlan <ncogh...@gmail.com > > <mailto:ncogh...@gmail.com>> wrote: > > > If you capture the context eagerly, then there are fewer > opportunities to > > > get materially different values from "data = list(iterable)" and > "data = > > > iter(context_capturing_iterable)". > > > > > > While that's a valid intent for folks to want to be able to > express, I > > > personally think it would be more clearly requested via an > expression like > > > "data = iter_in_context(iterable)" rather than having it be > implicit in the > > > way generators work (especially since having eager context capture > be > > > generator-only behaviour would create an odd discrepancy between > generators > > > and other iterators like those in itertools). > > > > OK. I understand the point here - but I'm not sure I see the > practical > > use case for iter_in_context. When would something like that be used? > > > > > > Suppose you have some existing code that looks like this: > > > > results = [calculate_result(a, b) for a, b in data] > > > > If calculate_result is context dependent in some way (e.g. a & b might > > be decimal values), then eager evaluation of "calculate_result(a, b)" > > will use the context that's in effect on this line for every result. > > > > Now, suppose you want to change the code to use lazy evaluation, so that > > you don't need to bother calculating any results you don't actually use: > > > > results = (calculate_result(a, b) for a, b in data) > > > > In a PEP 550 world, this refactoring now has a side-effect that goes > > beyond simply delaying the calculation: since "calculate_result(a, b)" > > is no longer executed immediately, it will default to using whatever > > execution context is in effect when it actually does get executed, *not* > > the one that's in effect on this line. > > > > A context capturing helper for iterators would let you decide whether or > > not that's what you actually wanted by instead writing: > > > > results = iter_in_context(calculate_result(a, b) for a, b in data) > > > > Here, "iter_in_context" would indicate explicitly to the reader that > > whenever another item is taken from this iterator, the execution context > > is going to be temporarily reset back to the way it was on this line. > > And since it would be a protocol based iterator-in-iterator-out > > function, you could wrap it around *any* iterator, not just > > generator-iterator objects. > > I have a hard time seeing the advantage of having a default > where the context at the time of execution is dependent on > where it happens rather than where it's defined. > The underlying rationale is that the generator form should continue to be as close as we can reasonably make it to being pure syntactic sugar for the iterator form: class ResultsIterator: def __init__(self, data): self._itr = iter(data) def __next__(self): return calculate_result(next(self._itr)) results = _ResultsIterator(data) The logical context adjustments in PEP 550 then serve to make using a with statement around a yield expression in a generator closer in meaning to using one around a return statement in a __next__ method implementation. > IMO, the default should be to use the context where the line > was defined in the code, since that matches the intuitive > way of writing and defining code. > This would introduce a major behavioural discrepancy between generators and iterators. > The behavior of also deferring the context to time of > execution should be the non-standard form to not break > this intuition, otherwise debugging will be a pain and > writing fully working code would be really hard in the > face of changing contexts (e.g. say decimal rounding > changes in different parts of the code). > No, it really wouldn't, since "the execution context is the context that's active when the code is executed" is relatively easy to understand based entirely on the way functions, methods, and other forms of delayed execution work (including iterators). "The execution context is the context that's active when the code is executed, *unless* the code is in a generator, in which case, it's the context that was active when the generator-iterator was instantiated" is harder to follow. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 15 October 2017 at 05:47, Paul Moore <p.f.mo...@gmail.com> wrote: > On 14 October 2017 at 17:50, Nick Coghlan <ncogh...@gmail.com> wrote: > > If you capture the context eagerly, then there are fewer opportunities to > > get materially different values from "data = list(iterable)" and "data = > > iter(context_capturing_iterable)". > > > > While that's a valid intent for folks to want to be able to express, I > > personally think it would be more clearly requested via an expression > like > > "data = iter_in_context(iterable)" rather than having it be implicit in > the > > way generators work (especially since having eager context capture be > > generator-only behaviour would create an odd discrepancy between > generators > > and other iterators like those in itertools). > > OK. I understand the point here - but I'm not sure I see the practical > use case for iter_in_context. When would something like that be used? > Suppose you have some existing code that looks like this: results = [calculate_result(a, b) for a, b in data] If calculate_result is context dependent in some way (e.g. a & b might be decimal values), then eager evaluation of "calculate_result(a, b)" will use the context that's in effect on this line for every result. Now, suppose you want to change the code to use lazy evaluation, so that you don't need to bother calculating any results you don't actually use: results = (calculate_result(a, b) for a, b in data) In a PEP 550 world, this refactoring now has a side-effect that goes beyond simply delaying the calculation: since "calculate_result(a, b)" is no longer executed immediately, it will default to using whatever execution context is in effect when it actually does get executed, *not* the one that's in effect on this line. A context capturing helper for iterators would let you decide whether or not that's what you actually wanted by instead writing: results = iter_in_context(calculate_result(a, b) for a, b in data) Here, "iter_in_context" would indicate explicitly to the reader that whenever another item is taken from this iterator, the execution context is going to be temporarily reset back to the way it was on this line. And since it would be a protocol based iterator-in-iterator-out function, you could wrap it around *any* iterator, not just generator-iterator objects. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
f > "micro tasks" are something non-experts might need to write, but not > realise they are straying into deep waters. But I've no way of knowing > how likely that is. > A micro-task is just a fancier name for the "iter_in_context" idea above (save the current context when the iterator is created, switch back to that context every time you're asked for a new value). > One final point, this is all pretty deeply intertwined with the > comprehensibility of async as a whole. At the moment, as I said > before, async is a specialised area that's largely only used in > projects that centre around it. In the same way that Twisted is its > own realm - people write network applications without Twisted, or they > write them using Twisted. Nobody uses Twisted in the middle of some > normal non-async application like pip to handle grabbing a webpage. > I'm uncertain whether the intent is for the core async features to > follow this model, or whether we'd expect in the longer term for > "utility adoption" of async to happen (tactical use of async for > something like web crawling or collecting subprocess output in a > largely non-async app). If that *does* happen, then async needs to be > much more widely understandable - maintenance programmers who have > never used async will start encountering it in corners of their > non-async applications, or find it used under the hood in libraries > that they use. This discussion is a good example of the implications > of that - async quirks leaking out into the "normal" world (decimal > contexts) and as a result the async experts needing to be able to > communicate their concerns and issues to non-experts. > Aye, this is why I'd like the semantics of context variables to be almost indistinguishable from those of thread local variables for synchronous code (aside from avoiding context changes leaking out of generator-iterators when they yield from inside a with statement). PEP 550 currently does a good job of ensuring that, but we'd break that near equivalence if generators were to implicitly capture their creation context. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution
On 14 October 2017 at 18:21, Antoine Pitrou <solip...@pitrou.net> wrote: > On Sat, 14 Oct 2017 10:49:11 +0300 > Serhiy Storchaka <storch...@gmail.com> > wrote: > > I don't like the idea of adding a parallel set of functions. > > > > In the list of alternatives in PEP 410 there is no an idea about fixed > > precision float type with nanoseconds precision. It can be implemented > > internally as a 64-bit integer, but provide all methods required for > > float-compatible number. It would be simpler and faster than general > > Decimal. > > I agree a parallel set of functions is not ideal, but I think a parallel > set of functions is still more appropriate than a new number type > specific to the time module. > > Also, if you change existing functions to return a new type, you risk > breaking compatibility even if you are very careful about designing the > new type. > Might it make more sense to have a parallel *module* that works with a different base data type rather than parallel functions within the existing API? That is, if folks wanted to switch to 64-bit nanosecond time, they would use: * time_ns.time() * time_ns.monotonic() * time_ns.perf_counter() * time_ns.clock_gettime() * time_ns.clock_settime() The idea here would be akin to the fact we have both math and cmath as modules, where the common APIs conceptually implement the same algorithms, they just work with a different numeric type (floats vs complex numbers). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add a module itertools.recipes
On 14 October 2017 at 16:06, Antoine Rozo <antoine.r...@gmail.com> wrote:
> I am not searching for an external library (as I pointed, there are some
> on PyPI like iterutils or more-itertools).
> My point was that recipes are documented in itertools module, but not
> implemented in standard library, and it would be useful to have them
> available.
>
Not providing the recipes as an importable API is a deliberate design
decision, as what folks often need is code that is
similar-to-but-not-exactly-the-same-as the code in the recipe. If they've
copied the code into their own utility library, then that's not a problem -
they can just edit their version to have the exact semantics they need.
Individual recipes may occasionally get promoted to be part of the module
API, but that only happens on a case by case basis, and requires a
compelling justification for the change ("It's sometimes useful" isn't
compelling enough - we know it's sometimes useful, that's why it's listed
as an example recipe).
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 14 October 2017 at 08:44, Steve Dower <steve.do...@python.org> wrote: > > It's not possible to special case __aenter__ and __aexit__ reliably >> (supporting wrappers, decorators, and possible side effects). >> > > Why not? Can you not add a decorator that sets a flag on the code object > that means "do not create a new context when called", and then it doesn't > matter where the call comes from - these functions will always read and > write to the caller's context. That seems generally useful anyway, and then > you just say that __aenter__ and __aexit__ are special and always have that > flag set. > One example where giving function names implicit semantic significance becomes problematic: async def start_transaction(self): ... async def end_transaction(self, *exc_details): ... __aenter__ = start_transaction __aexit__ = end_transaction There *are* ways around that (e.g. type.__new__ implicitly wraps __init_subclass__ with classmethod since it makes no sense as a regular instance method), but then you still run into problems like this: async def __aenter__(self): return await self.start_transaction() async def __aexit__(self, *exc_details): return await self.end_transaction(*exc_details) If coroutines were isolated from their parents by default, then the above method implementations would be broken, even though the exact same invocation pattern works fine for synchronous function calls. To try and bring this back to synchronous examples that folks may find more intuitive, I figure it's worth framing the question this way: do we want people to reason about context variables like the active context is implicitly linked to the synchronous call stack, or do we want to encourage them to learn to reason about them more like they're a new kind of closure? The reason I ask that is because there are three "interesting" times in the life of a coroutine or generator: - definition time (when the def statement runs - this determines the lexical closure) - instance creation time (when the generator-iterator or coroutine is instantiated) - execution time (when the frame actually starts running - this determines the runtime call stack) For synchronous functions, instance creation time and execution time are intrinsically linked, since the execution frame is allocated and executed directly as part of calling the function. For asynchronous operations, there's more of a question, since actual execution is deferred until you call await or next() - the original synchronous call to the factory function instantiates an object, it doesn't actually *do* anything. The current position of PEP 550 (which I agree with) is that context variables should default to being closely associated with the active call stack (regardless of whether those calls are regular synchronous ones, or asynchronous ones with await), as this keeps the synchronous and asynchronous semantics of context variables as close to each other as we can feasibly make them. When implicit isolation takes place, it's either to keep concurrently active logical call stacks isolated from each other (the event loop case), and else to keep context changes from implicitly leaking *up* a stack (the generator case), not to keep context changes from propagating *down* a call stack. When we do want to prevent downward propagation for some reason, then that's what "run_in_execution_context" is for: deliberate creation of a new concurrently active call stack (similar to running something in another thread to isolate the synchronous call stack). Don't get me wrong, I'm not opposed to the idea of making it trivial to define "micro tasks" (iterables that perform a context switch to a specified execution context every time they retrieve a new value) that can provide easy execution context isolation without an event loop to manage it, I just think that would be more appropriate as a wrapper API that can be placed around any iterable, rather than being baked in as an intrinsic property of generators. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 13 October 2017 at 10:56, Guido van Rossum <gu...@python.org> wrote: > I'm out of energy to debate every point (Steve said it well -- that > decimal/generator example is too contrived), but I found one nit in Nick's > email that I wish to correct. > > On Wed, Oct 11, 2017 at 1:28 AM, Nick Coghlan <ncogh...@gmail.com> wrote: >> >> As a less-contrived example, consider context managers implemented as >> generators. >> >> We want those to run with the execution context that's active when >> they're used in a with statement, not the one that's active when they're >> created (the fact that generator-based context managers can only be used >> once mitigates the risk of creation time context capture causing problems, >> but the implications would still be weird enough to be worth avoiding). >> > > Here I think we're in agreement about the desired semantics, but IMO all > this requires is some special casing for @contextlib.contextmanager. To me > this is the exception, not the rule -- in most *other* places I would want > the yield to switch away from the caller's context. > > >> For native coroutines, we want them to run with the execution context >> that's active when they're awaited or when they're prepared for submission >> to an event loop, not the one that's active when they're created. >> > > This caught my eye as wrong. Considering that asyncio's tasks (as well as > curio's and trio's) *are* native coroutines, we want complete isolation > between the context active when `await` is called and the context active > inside the `async def` function. > The rationale for this behaviour *does* arise from a refactoring argument: async def original_async_function(): with some_context(): do_some_setup() raw_data = await some_operation() data = do_some_postprocessing(raw_data) Refactored: async def async_helper_function(): do_some_setup() raw_data = await some_operation() return do_some_postprocessing(raw_data) async def refactored_async_function(): with some_context(): data = await async_helper_function() However, considering that coroutines are almost always instantiated at the point where they're awaited, I do concede that creation time context capture would likely also work out OK for the coroutine case, which would leave contextlib.contextmanager as the only special case (and it would turn off both creation-time context capture *and* context isolation). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 11 October 2017 at 02:52, Guido van Rossum <gu...@python.org> wrote: > I think we really need to do more soul-searching before we decide that a > much more complex semantics and implementation is worth it to maintain > backwards compatibility for leaking in via next(). > As a less-contrived example, consider context managers implemented as generators. We want those to run with the execution context that's active when they're used in a with statement, not the one that's active when they're created (the fact that generator-based context managers can only be used once mitigates the risk of creation time context capture causing problems, but the implications would still be weird enough to be worth avoiding). For native coroutines, we want them to run with the execution context that's active when they're awaited or when they're prepared for submission to an event loop, not the one that's active when they're created. For generators-as-coroutines, we want them to be like their native coroutine counterparts: run with the execution context that's active when they're passed to "yield from" or prepared for submission to an event loop. It's only for generators-as-iterators that the question of what behaviour we want even really arises, as it's less clear cut whether we'd be better off overall if they behaved more like an eagerly populated container (and hence always ran with the execution context that's active when they're created), or more like the way they do now (where retrieval of the next value from a generator is treated like any other method call). That combination of use cases across context managers, native coroutines, top level event loop tasks, and generator-based coroutines mean we already need to support both execution models regardless, so the choice of default behaviour for generator-iterators won't make much difference to the overall complexity of the PEP. However, having generator-iterators default to *not* capturing their creation context makes them more consistent with the other lazy evaluation constructs, and also makes the default ContextVar semantics more consistent with thread local storage semantics. The flipside of that argument would be: * the choice doesn't matter if there aren't any context changes between creation & use * having generators capture their context by default may ease future migrations from eager container creation to generators in cases that involve context-dependent calculations * decorators clearing the implicitly captured context from the generator-iterator when appropriate is simpler than writing a custom iterator wrapper to handle the capturing I just don't find that counterargument compelling when we have specific use cases that definitely benefit from the proposed default behaviour (contextlib.contextmanager, asyncio.coroutine), but no concrete use cases for the proposed alternative that couldn't be addressed by a combination of map(), functools.partial(), and contextvars.run_in_execution_context(). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 10 October 2017 at 22:51, Koos Zevenhoven <k7ho...@gmail.com> wrote: > I see no reason why these two should be equivalent. > There is no "should" about it: it's a brute fact that the two forms *are* currently equivalent for lazy iterators (including generators), and both different from the form that uses eager evaluation of the values before the context change. Where should enters into the picture is by way of PEP 550 saying that they should *remain* equivalent because we don't have an adequately compelling justification for changing the runtime semantics. That is, given the following code: itr = make_iter() with decimal.localcontext() as ctx: ctc.prex = 30 for i in itr: pass Right now, today, in 3.6. the calculations in the iterator will use the modified decimal context, *not* the context that applied when the iterator was created. If you want to ensure that isn't the case, you have to force eager evaluation before the context change. What PEP 550 is proposing is that, by default, *nothing changes*: the lazy iteration in the above will continue to use the updated decimal context by default. However, people *will* gain a new option for avoiding that: instead of forcing eager evaluation, they'll be able to capture the creation context instead, and switching back to that each time the iterator needs to calculate a new value. If PEP 555 proposes that we should instead make lazy iteration match eager evaluation semantics by *default*, then that's going to be a much harder case to make because it's a gratuitous compatibility break - code that currently works one way will suddenly start doing something different, and end users will have difficulty getting it to behave the same way on 3.7 as it does on earlier versions. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 10 October 2017 at 22:34, Koos Zevenhoven <k7ho...@gmail.com> wrote: > Really, it was my mistake to ever make you think that > context_var.assign(42).__enter__() can be compared to .set(42) in PEP > 550. I'll say it once more: PEP 555 context arguments have no equivalent of > the PEP-550 .set(..). > Then your alternate PEP can't work, since it won't be useful to extension modules. Context managers are merely syntactic sugar for try/finally statements, so you can't wave your hands and say a context manager is the only supported API: you *have* to break the semantics down and explain what the try/finally equivalent looks like. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 10 October 2017 at 01:24, Guido van Rossum <gu...@python.org> wrote: > On Sun, Oct 8, 2017 at 11:46 PM, Nick Coghlan <ncogh...@gmail.com> wrote: > >> On 8 October 2017 at 08:40, Koos Zevenhoven <k7ho...@gmail.com> wrote: >> >>> I do remember Yury mentioning that the first draft of PEP 550 captured >>> something when the generator function was called. I think I started reading >>> the discussions after that had already been removed, so I don't know >>> exactly what it was. But I doubt that it was *exactly* the above, because >>> PEP 550 uses set and get operations instead of "assignment contexts" like >>> PEP 555 (this one) does. >>> >> >> We didn't forget it, we just don't think it's very useful. >> > > I'm not sure I agree on the usefulness. Certainly a lot of the complexity > of PEP 550 exists just to cater to Nathaniel's desire to influence what a > generator sees via the context of the send()/next() call. I'm still not > sure that's worth it. In 550 v1 there's no need for chained lookups. > The compatibility concern is that we want developers of existing libraries to be able to transparently switch from using thread local storage to context local storage, and the way thread locals interact with generators means that decimal (et al) currently use the thread local state at the time when next() is called, *not* when the generator is created. I like Yury's example for this, which is that the following two examples are currently semantically equivalent, and we want to preserve that equivalence: with decimal.localcontext() as ctx: ctc.prex = 30 for i in gen(): pass g = gen() with decimal.localcontext() as ctx: ctc.prex = 30 for i in g: pass The easiest way to maintain that equivalence is to say that even though preventing state changes leaking *out* of generators is considered a desirable change, we see preventing them leaking *in* as a gratuitous backwards compatibility break. This does mean that *neither* form is semantically equivalent to eager extraction of the generator values before the decimal context is changed, but that's the status quo, and we don't have a compelling justification for changing it. If folks subsequently decide that they *do* want "capture on creation" or "capture on first iteration" semantics for their generators, those are easy enough to add as wrappers on top of the initial thread-local-compatible base by using the same building blocks as are being added to help event loops manage context snapshots for coroutine execution. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On 8 October 2017 at 08:40, Koos Zevenhoven <k7ho...@gmail.com> wrote: > On Sun, Oct 8, 2017 at 12:16 AM, Nathaniel Smith <n...@pobox.com> wrote: > >> On Oct 7, 2017 12:20, "Koos Zevenhoven" <k7ho...@gmail.com> wrote: >> >> >> Unfortunately, we actually need a third kind of generator semantics, >> something like this: >> >> @contextvars.caller_context >> def genfunc(): >> assert cvar.value is the_value >> yield >> assert cvar.value is the_value >> >> with cvar.assign(the_value): >> gen = genfunc() >> >> next(gen) >> >> with cvar.assign(1234567890): >> try: >> next(gen) >> except StopIteration: >> pass >> >> Nick, Yury and I (and Nathaniel, Guido, Jim, ...?) somehow just narrowly >> missed the reasons for this in discussions related to PEP 550. Perhaps >> because we had mostly been looking at it from an async angle. >> >> >> That's certainly a semantics that one can write down (and it's what the >> very first version of PEP 550 did), >> > > I do remember Yury mentioning that the first draft of PEP 550 captured > something when the generator function was called. I think I started reading > the discussions after that had already been removed, so I don't know > exactly what it was. But I doubt that it was *exactly* the above, because > PEP 550 uses set and get operations instead of "assignment contexts" like > PEP 555 (this one) does. > We didn't forget it, we just don't think it's very useful. However, if you really want those semantics under PEP 550, you can do something like this: def use_creation_context(g): @functools.wraps(g) def make_generator_wrapper(*args, **kwds): gi = g(*args, **kwds) return _GeneratorWithCapturedEC(gi) return make_generator_wrapper class _GeneratorWithCapturedEC: def __init__(self, gi): self._gi = gi self._ec = contextvars.get_execution_context() def __next__(self): return self.send(None) def send(self, value): return contextvars.run_with_execution_context(self.ec, self._gi.send, value) def throw(self, *exc_details): return contextvars.run_with_execution_context(self.ec, self._gi.throw, *exc_details) def close(self): return self.throw(GeneratorExit) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add pop_callback to deque signature
On 4 October 2017 at 04:24, Chris Barker <chris.bar...@noaa.gov> wrote: > [Antoine] >> The problem as >> I understand it is that you need a Windows machine (or VM) together with >> the required set of compilers, and have to take the time to run the >> builds. > > Yup — still the case. Also with Mac and OS-X. Distributing a package with a > compiled component is still a lot more work. The broad availability & popularity of AppVeyor's free tier is another relevant change compared to a few years ago, and https://github.com/joerick/cibuildwheel is designed to work with that to help projects automate their artifact builds without need to maintain their own cross-platform build infrastructure. So yeah, we're definitely to a point where adding new data structures to the standard library, or new features to existing data structures, is mainly going to be driven by standard library use cases that benefit from them, rather than "binary dependencies are still too hard to publish & manage". (For example, __missing__ made defaultdict easy to implement). For deque specifically, I like Steven D'Aprano's suggestion of a "__dropped__" or "__discard__" subclassing API that makes it straightforward to change the way that queue overruns are handled (especially if raising an exception from the new subclass method can prevent the collection modification entirely - that way you could readily change the deque semantics in a subclass such that if the queue fills up, submitters start getting errors instead of silently discarding older messages, allowing backpressure to be more easily propagated through a system of queues). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Changes to the existing optimization levels
On 4 October 2017 at 01:42, Diana Clarke <diana.joan.cla...@gmail.com> wrote: > On Sun, Oct 1, 2017 at 6:19 AM, Antoine Pitrou <solip...@pitrou.net> wrote: >> Well, this is not really a bitfield, but a bitfield plus some irregular >> hardcoded values. Therefore I don't think it brings much in the way of >> discoverability / understandability. >> >> That said, perhaps it makes implementation easier on the C side... > > I think I'm coming to the same conclusion: using bitwise operations > for the optimization levels seems to just boil down to a more cryptic > version of the simple "level 3" solution, with public-facing impacts > to the pycache and existing interfaces etc that I don't think are > worth it in this case. > > My only other thought at the moment, would be to use the existing -X > option to achieve something similar to what I did with the new -N > option, but then just quickly map that back to an integer under the > hood. That is, "-X opt-nodebug -X opt-noassert" would just become > "level 3" internally so that the various interfaces wouldn't have to > change. Sorry, I don't think I was entirely clear as to what my suggestion actually was: * Switch to your suggested "set-of-strings" API at the Python level, with the Python level integer interface retained only for backwards compatibility * Keep the current integer-based *C* optimization API, but redefine the way that value is interpreted, rather than passing Python sets around The Python APIs would then convert the Python level sets to the bitfield representation almost immediately for internal use, but you wouldn't need to mess about with the bitfield yourself when calling the Python APIs. The difference I see relates to the fact that in Python: * sets of strings are easier to work with than integer bitfields * adding a new keyword-only argument to existing APIs is straightforward While in C: * integer bitfields are easier to work with than Python sets of Python strings * supporting a new argument would mean defining a whole new parallel set of APIs Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
