Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Tue, Nov 07, 2017 at 05:28:24PM +1000, Nick Coghlan wrote:
> On 7 November 2017 at 16:21, Steven D'Aprano  wrote:
> > On Mon, Nov 06, 2017 at 08:05:07PM -0800, David Mertz wrote:
> >> Maybe OrderedDict can be
> >> rewritten to use the dict implementation. But the evidence that all
> >> implementations will always be fine with this restraint feels poor,
> >
> > I think you have a different definition of "poor" to me :-)
> 
> While I think "poor" is understating the case, I think "excellent"
> (which you use later on) is overstating it. My own characterisation
> would be "at least arguably good enough".

Fair enough, and thanks for elaborating.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 7 November 2017 at 16:21, Steven D'Aprano  wrote:
> On Mon, Nov 06, 2017 at 08:05:07PM -0800, David Mertz wrote:
>> Maybe OrderedDict can be
>> rewritten to use the dict implementation. But the evidence that all
>> implementations will always be fine with this restraint feels poor,
>
> I think you have a different definition of "poor" to me :-)

While I think "poor" is understating the case, I think "excellent"
(which you use later on) is overstating it. My own characterisation
would be "at least arguably good enough".

> Nick has already done a survey of PyPy (which already has insertion-
> order preserving dicts), Jython, VOC, and Batavia, and they don't have
> any problem with this.

For these, my research only showed that their respective platforms
have an order-preserving hashmap implementation available.

What's entirely unclear at this point is how switching wholesale to
that may impact the *performance* of these implementations (both in
terms of speed and memory usage), and how much code churn would be
involved in actually making the change.

Making builtin dict() order-preserving may also still impose an
ongoing complexity cost on these implementations if they end up having
to split "the mapping we use for code execution namespaces" away from
"the mapping we provide as the builtin dict".

(That said, I think at least Jython already makes that distinction - I
believe their string-only namespace dict is a separate type, whereas
CPython plays dynamic optimisation games inside the regular dict
type).

So Barry's suggestion of providing an explicit
"collections.UnorderedDict" as a consistent spelling for "an
associative mapping without any ordering guarantees" is a reasonable
one, even if it's primary usage in CPython ends up being to ensure
algorithms are compatible with collections that don't provide an
inherently stable iteration order, and any associated performance
benefits are mostly seen on other implementations. (As Paul S notes,
such a data type would also serve a pedagogical purpose when teaching
computer science principles)

> IronPython is built on C#, which has order-
> preserving mappings.

I couldn't actually find a clearly suitable existing collection type
in the .NET CLR - the one I linked was just the one commonly
referenced as "good enough for most purposes". It had some constraints
that meant it may not be suitable as a builtin dict type in a Python
implementation (e.g. it looked to have a 32-bit length limit).

> Nuitka is built on C++, and if C++ can't implement
> an order-preserving mapping, there is something terribly wrong with the
> world. Cython (I believe) uses CPython's implementation, as does
> Stackless.

Right, the other C/C++ implementations that also target environments
with at least 128 MiB+ RAM (and typically more) can reasonably be
expected to tolerate similar space/speed trade-offs to those that
CPython itself makes (and that's assuming they aren't just using
CPython's data type implementations in the first place).

> The only well-known implementation that may have trouble with this is
> MicroPython, but it already changes the functionality of a lot of
> builtins and core language features, e.g. it uses a different method
> resolution order (so multiple inheritence won't work right), some
> builtins don't support slicing with three arguments, etc.
>
> I think the evidence is excellent that other implementations shouldn't
> have a problem with this, unless (like MicroPython) they are targetting
> machines with tiny memory resources. µPy runs on the PyBoard, which I
> believe has under 200K of memory. I think we can all forgive µPy if it
> only *approximately* matches Python semantics.

It runs on the ESP8266 as well, and that only has 96 kiB data memory.
This means we're already talking 3-4 orders of magnitude difference in
memory capacity and typical data set sizes between CPython and
MicroPython use cases, and that's only accounting for the *low* end of
CPython's use cases - once you expand out to multi-terabyte data sets
(the very upper end of what a single x86-64 server can handle if you
can afford the RAM for it), then we're talking 9-10 orders of
magnitude between CPython's high end and MicroPython's low end.

So for CPython's target use cases algorithmic efficiency dominates
performance, and we afford to invest extra memory usage and startup
overhead in service to more efficient data access algorithms.
MicroPython's the opposite - you're going to run out of memory for
data storage long before algorithmic inefficiency becomes your biggest
problem, but wasted bookkeeping memory and startup overhead can cause
problems even with small data sets.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.c

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 7 November 2017 at 09:23, Chris Barker  wrote:
> in short -- we don't have a choice (unless we add an explicit randomization
> as some suggested -- but that just seems perverse...)

And this is the key point for me: "choosing not to choose" is
effectively the same as standardising the feature, as enough Python
code will come to rely on CPython's behaviour that most alternative
implementations will feel obliged to start behaving the same way
CPython does (with MicroPython being the potential exception due to
memory usage constraints always winning over algorithmic efficiency
concerns in that context).

We added ResourceWarning a while back to help discourage reliance on
CPython promptly calling __del__ methods when dropping the last
reference to an object.

An equivalent for this case would be for dict objects to randomize
iteration (ala Go), once again requiring folks to opt-in via
collections.OrderedDict to get guaranteed ordering (perhaps with a
"o{}" dict display as new syntactic sugar).

But unless someone actually writes a PEP and implementation for that
in the next 12 weeks (and Guido accepts it), then we'll have 2
releases and 3 years of CPython working a particular way increasing
the inertia against making such a change in 3.8 (and beyond that, I'd
say we'd be well and truly into de facto standardisation territory,
and the chances of ever introducing deliberate perturbation of dict
iteration order would drop to nil).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 10:17:23PM -0200, Joao S. O. Bueno wrote:

> And also, forgot along the discussion, is the big disadvantage that
> other Python implementations would have a quite
> significant overhead on mandatory ordered dicts.

I don't think that is correct. Nick already did a survey, and found that
C# (IronPython), Java (Jython and VOC) and Javascript (Batavia) all have
acceptable insertion-order preserving mappings. C++ (Nuitka) surely
won't have any problem with this (if C++ cannot implement an efficient
order-preserving map, there is something terribly wrong with the world).

As for other languages that somebody might choose to build Python on 
(the Parrot VM, Haskell, D, Rust, etc) surely we shouldn't be limiting 
what Python does for the sake of hypothetical implementations in 
"underpowered" languages?

I don't mean to imply that any of those examples are necessarily
underpowered, but if language Foo is incapable of supporting an
efficient ordered map, then language Foo is simply not good enough for a
serious Python implementation. We shouldn't allow Python's evolution to
be hamstrung by the requirement to support arbitrarily weak
implementation languages.


> One that was mentioned along the way is transpilers, with
> Brython as an example - but there might be others.

Since Brython transpiles to Javascript, couldn't it use the standard
Map object, which preserves insertion order?

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map

Quote:

Description
A Map object iterates its elements in insertion order

The EMCAScript 6 standard specifies that Map.prototype.forEach operates
over the key/value pairs in insertion order:

https://tc39.github.io/ecma262/#sec-map-objects



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 06:35:48PM +0200, Paul Sokolovsky wrote:

> For MicroPython, it would lead to quite an overhead to make
> dictionary items be in insertion order. As I mentioned, MicroPython
> optimizes for very low bookkeeping memory overhead, so lookups are
> effectively O(n), but orderedness will increase constant factor
> significantly, perhaps 5x.

Paul, it would be good if you could respond to Raymond's earlier 
comments where he wrote:

I've just looked at the MicroPython dictionary implementation and 
think they won't have a problem implementing O(1) compact dicts with 
ordering.

The likely reason for the confusion is that they are already have an 
option for an "ordered array" dict variant that does a brute-force 
linear search.  However, their normal hashed lookup is very similar 
to ours and is easily amenable to being compact and ordered.

See:  
https://github.com/micropython/micropython/blob/77a48e8cd493c0b0e0ca2d2ad58a110a23c6a232/py/map.c#L139

Raymond has also volunteered to assist with this.


> Also, arguably any algorithm which would *maintain* insertion order
> over mutating operations would be more complex and/or require more
> memory that one which doesn't.

I think it would be reasonable to say that builtin dicts only maintain 
insertion order for insertions, lookups, and changing the value. Any 
mutation which deletes keys may arbitrarily re-order the dict.

If the user wants a stronger guarantee, then they should use 
OrderedDict.




-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 08:05:07PM -0800, David Mertz wrote:
> I strongly opposed adding an ordered guarantee to regular dicts. If the
> implementation happens to keep that, great. 

That's the worst of both worlds. The status quo is that unless we 
deliberately perturb the dictionary order, developers will come to rely 
on implementation order (because that's what the CPython reference 
implementation actually offers, regardless of what the docs say). 
Consequently:

- people will be writing non-portable code, whether they know it or not;

- CPython won't be able to change the implementation, because it will 
break too much code;

- other implementations will be pressured to match CPython's 
implementation.

The only difference is that on the one hand we are honest and up-front 
about requiring order-preserving dicts, and on the other we still 
require it, but pretend that we don't.

And frankly, it seems rather perverse to intentionally perturb 
dictionary order just to keep our options open that someday there might 
be some algorithm which offers sufficiently better performance but 
doesn't preserve order. Preserving order is useful, desirable, often 
requested functionality, and now that we have it, it would have to be 
one hell of an optimization to justify dropping it again.

(It is like Timsort and stability. How much faster sorting would it 
have taken to justify giving up sort stability? 50% faster? 100%? We 
wouldn't have done it for a 1% speedup.)

It would be better to relax the requirement that builtin dict is used 
for those things that would benefit from improved performance. Is there 
any need for globals() to be the same mapping type as dict? Probably 
not. If somebody comes up with a much more efficient, non-order- 
preserving map ideal for globals, it would be better to change globals 
than dict. In my opinion.


> Maybe OrderedDict can be
> rewritten to use the dict implementation. But the evidence that all
> implementations will always be fine with this restraint feels poor,

I think you have a different definition of "poor" to me :-)

Nick has already done a survey of PyPy (which already has insertion- 
order preserving dicts), Jython, VOC, and Batavia, and they don't have 
any problem with this. IronPython is built on C#, which has order- 
preserving mappings. Nuitka is built on C++, and if C++ can't implement 
an order-preserving mapping, there is something terribly wrong with the 
world. Cython (I believe) uses CPython's implementation, as does 
Stackless.

The only well-known implementation that may have trouble with this is 
MicroPython, but it already changes the functionality of a lot of 
builtins and core language features, e.g. it uses a different method 
resolution order (so multiple inheritence won't work right), some 
builtins don't support slicing with three arguments, etc.

I think the evidence is excellent that other implementations shouldn't 
have a problem with this, unless (like MicroPython) they are targetting 
machines with tiny memory resources. µPy runs on the PyBoard, which I 
believe has under 200K of memory. I think we can all forgive µPy if it 
only *approximately* matches Python semantics.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 7 November 2017 at 03:27, Chris Jerdonek  wrote:
> On Mon, Nov 6, 2017 at 4:11 AM Nick Coghlan  wrote:
>> Getting from the "Works on CPython 3.6+ but is technically
>> non-portable" state to a fully portable correct implementation that
>> ensures a particular key order in the JSON file thus currently
>> requires the following changes:
>
> Nick, it seems like this is more complicated than it needs to be. You can
> just pass sort_keys=True to json.dump() / json.dumps(). I use it for tests
> and human-readability all the time.

sort_keys is only equivalent to order preservation if the key order
you want *is* alphabetical order. While that's typically a good enough
assumption for JSON, it's not the case for things like CSV column
order, TOML or ini-file setting order, etc.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 7 November 2017 at 09:20, Lukasz Langa  wrote:
>
>
>> On Nov 5, 2017, at 11:28 PM, Nick Coghlan  wrote:
>>
>> On 6 November 2017 at 16:36, Lukasz Langa  wrote:
>>
>> - compile annotations like a small nested class body (but returning
>> the expression result, rather than None)
>> - emit MAKE_THUNK instead of the expression's opcodes
>> - emit STORE_ANNOTATION as usual
>>
>
> Is the motivation behind creating thunks vs. reusing lambdas just the 
> difference in handling class-level scope? If so, would it be possible to just 
> modify lambdas to behave thunk-like there? It sounds like this would strictly 
> broaden the functionality of lambdas, in other words, wouldn't create 
> backwards incompatibility for existing code.
>
> Reusing lambdas (with extending them to support class-level scoping) would be 
> a less scary endeavor than introducing a brand new language construct.

I want to say "yes", but it's more "sort of", and at least arguably
"no" (while you'd be using building blocks that already exist inside
the compiler and eval loop, you'd be putting them together in a
slightly new way that's a hybrid of the way lambdas work and the way
class bodies work).

For the code execution part, class creation currently uses
MAKE_FUNCTION (just like lambdas and def statements), and is able to
inject the namespace returned by __prepare__ as the code execution
namespace due to differences in the way the function's code object
gets compiled. These are *exactly* the semantics you'd want for
deferred annotations, but the way they're currently structured
internally is inconvenient for your use case.

Compilation details here:
https://github.com/python/cpython/blob/master/Python/compile.c#L1895
Code execution details here:
https://github.com/python/cpython/blob/master/Python/bltinmodule.c#L167

It's the combination of using COMPILER_SCOPE_CLASS at compilation time
with the direct call to "PyEval_EvalCodeEx" in __build_class__ that
means you can't just treat annotations as a regular lambda function -
lambdas are compiled with CO_OPTIMIZED set, and that means they won't
read local variable values from the provided locals namespace, which
in turn means you wouldn't be able to easily inject "vars(cls)" to
handle the method annotation case.

So that's where the idea of finally adding a "thunk" object came from:
it would be to a class body as a lambda expression is to a function
body, except that instead of relying on a custom call to
PyEval_EvalCodeEx in __build_class__ the way class bodies do, it would
instead define a suitable implementation of tp_call that accepted the
locals namespace to use as a parameter.

The nice part of this approach is that even though it would
technically be a new execution primitive, it's still one with
well-established name resolution semantics: the behaviour we already
use for class bodies.

> With my current understanding I still think stringification is both easier to 
> implement and understand by end users.

No matter how you slice it, PEP 563 *is* defining a new delayed
execution primitive as part of the core language syntax. The question
is whether we define it as a fully compiler integrated primitive, with
clearly specified lexical name resolution semantics that align with
other existing constructs, or something bolted on to the side of the
language without integrating it properly, which we then have to live
with forever.

"This isn't visibly quoted, but it's a string anyway, so the compiler
won't check it for syntax errors" isn't easy to understand. Neither is
the complex set of rules you're proposing for what people will need to
do in order to actually evaluate those strings and turn them back into
runtime objects.

By contrast, "parameter: annotation" can be explained as "it's like
'lambda: expression', but instead of being a regular function with an
explicit parameter list, the annotation is a deferred expression that
accepts a locals namespace to use when called".

> The main usability win of thunks/lambdas is not very significant: evaluating 
> them is as easy as calling them whereas strings require 
> typing.get_type_hints(). I still think being able to access function-local 
> state at time of definition is only theoretically useful.

Your current proposal means that this code will work:

class C:
field = 1
def method(a: C.field):
pass

But this will fail:

def make_class():
class C:
field = 1
def method(a: C.field):
pass

Dropping the "C." prefix would make the single class case work, but
wouldn't help with the nested classes case:

def make_class():
class C:
field = 1
class D:
def method(a: C.field):
pass

Confusingly, though, this would still work:

def make_class():
class C:
field = 1
class D:
field2 = C.field
def method(a: field2):

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[INADA Naoki]

I agree with Raymond.  dict ordered by default makes better developer
experience.

So, my concern is how "language spec" is important for minor (sorry about my
bad vocabulary) implementation?
What's difference between "MicroPython is 100% compatible with
language spec" and
"MicroPython is almost compatible with Python language spec, but has
some restriction"?

If it's very important, how about "strong recommendation for
implementations" instead of
"language spec"?
Users who don't care implementations other than CPython and PyPy can rely on
it's usability.

Regards,
INADA Naoki  


On Tue, Nov 7, 2017 at 2:11 PM, Raymond Hettinger
 wrote:
>
>> On Nov 6, 2017, at 8:05 PM, David Mertz  wrote:
>>
>> I strongly opposed adding an ordered guarantee to regular dicts. If the 
>> implementation happens to keep that, great. Maybe OrderedDict can be 
>> rewritten to use the dict implementation. But the evidence that all 
>> implementations will always be fine with this restraint feels poor, and we 
>> have a perfectly good explicit OrderedDict for those who want that.
>
> I think this post is dismissive of the value that users would get from having 
> reliable ordering by default.
>
> Having worked with Python 3.6 for a while, it is repeatedly delightful to 
> encounter the effects of ordering.  When debugging, it is a pleasure to be 
> able to easily see what has changed in a dictionary.  When creating XML, it 
> is joy to see the attribs show in the same order you added them.  When 
> reading a configuration, modifying it, and writing it back out, it is a 
> godsend to have it written out in about the same order you originally typed 
> it in.  The same applies to reading and writing JSON.  When adding a VIA 
> header in a HTTP proxy, it is nice to not permute the order of the other 
> headers. When generating url query strings for REST APIs, it is nice have the 
> parameter order match documented examples.
>
> We've lived without order for so long that it seems that some of us now think 
> data scrambling is a virtue.  But it isn't.  Scrambled data is the opposite 
> of human friendly.
>
>
> Raymond
>
>
> P.S. Especially during debugging, it is often inconvenient, difficult, or 
> impossible to bring in an OrderedDict after the fact or to inject one into 
> third-party code that is returning regular dicts.  Just because we have 
> OrderedDict in collections doesn't mean that we always get to take advantage 
> of it.  Plain dicts get served to us whether we want them or not.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Ethan Smith]

> Beyond the API already proposed in PEP 557, this would mean adding:
>
> * dataclasses.ClassVar (as proposed above)
> * dataclasses.Any (probably just set to the literal string
> "dataclasses.Any")
> * dataclasses.NamedTuple (as a replacement for typing.NamedTuple)
> * potentially dataclasses.is_class_var (instead of dataclasses
> implicitly snooping in sys.modules looking for a "typing" module)
>
> In effect, where we now have "typing" and "typing_extensions", we'd
> instead have "dataclasses" in the standard library (with the subset of
> annotations needed to define data record classes), and "typing" as an
> independently versioned module on PyPI.
>
>
I'm not so keen on this because I think some things in typing (such as
NamedTuple) probably deserve to be in the collections module. And some of
the ABCs could probably also be merged with collections.abc but doing this
correctly and not all at once would be quite difficult.
I do think the typing concepts should be better integrated into the
standard library. However, a fair amount of the clases you list (such as
NamedTuple) are in of themselves dependent on parts of typing.

Cheers,
Ethan

I think such an approach could address a few problems:
>
> - the useful-at-runtime concepts (in dataclasses) would be clearly
> separated from the static-type-analysis enablers (in typing)
> - type hints would still have a clear presence in the regular standard
> library, but the more complex parts would be opt-in (similar to
> statistics vs SciPy, secrets vs cryptography, etc)
> - as various concepts in typing matured and became genuinely stable,
> they could potentially "graduate" into the standard library's
> dataclasses module
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Raymond Hettinger]

> On Nov 6, 2017, at 8:05 PM, David Mertz  wrote:
> 
> I strongly opposed adding an ordered guarantee to regular dicts. If the 
> implementation happens to keep that, great. Maybe OrderedDict can be 
> rewritten to use the dict implementation. But the evidence that all 
> implementations will always be fine with this restraint feels poor, and we 
> have a perfectly good explicit OrderedDict for those who want that.

I think this post is dismissive of the value that users would get from having 
reliable ordering by default.

Having worked with Python 3.6 for a while, it is repeatedly delightful to 
encounter the effects of ordering.  When debugging, it is a pleasure to be able 
to easily see what has changed in a dictionary.  When creating XML, it is joy 
to see the attribs show in the same order you added them.  When reading a 
configuration, modifying it, and writing it back out, it is a godsend to have 
it written out in about the same order you originally typed it in.  The same 
applies to reading and writing JSON.  When adding a VIA header in a HTTP proxy, 
it is nice to not permute the order of the other headers. When generating url 
query strings for REST APIs, it is nice have the parameter order match 
documented examples.

We've lived without order for so long that it seems that some of us now think 
data scrambling is a virtue.  But it isn't.  Scrambled data is the opposite of 
human friendly.


Raymond


P.S. Especially during debugging, it is often inconvenient, difficult, or 
impossible to bring in an OrderedDict after the fact or to inject one into 
third-party code that is returning regular dicts.  Just because we have 
OrderedDict in collections doesn't mean that we always get to take advantage of 
it.  Plain dicts get served to us whether we want them or not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 11:33:10AM -0800, Barry Warsaw wrote:

> If we did make the change, it’s possible we would need a way to 
> explicit say that order is not preserved.  That seems a little weird 
> to me, but I suppose it could be useful. 

Useful for what?

Given that we will hypothetically have order-preserving dicts that 
perform no worse than unordered dicts, I'm struggling to think of a 
reason (apart from performance) why somebody would intentionally use a 
non-ordered dict. If performance was an issue, sure, it makes sense to 
have a non-ordered dict for when you don't want to pay the cost of 
keeping insertion order. But performance seems to be a non-issue.

I can see people wanting a SortedDict which automatically sorts the keys 
into some specified order. If I really work at it, I can imagine that 
there might even be a use-case for randomizing the key order (like 
calling random.shuffle on the keys). But if you are willing to use a 
dict with arbitrary order, that means that *you don't care* what order 
the keys are in. If you don't care, then insertion order should be no 
better or worse than any other implementation-defined arbitrary order.


> I like the idea previously 
> brought up that iteration order be deliberately randomized in that 
> case, but we’d still need a good way to spell that.

That would only be in the scenario that we decide *not* to guarantee 
insertion-order preserving semantics for dicts, in order to prevent 
users from relying on an implementation feature that isn't a language 
guarantee.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 7 November 2017 at 03:42, Simon Cross  wrote:
> Maybe there are ways around these things, but I'm not really seeing
> what's wrong with the current situation that can't be fixed with
> slightly better CI setups (which are good for other reasons too).

Given the status quo, how do educators learn that the examples they're
teaching to their students are using deprecated APIs?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[David Mertz]

I strongly opposed adding an ordered guarantee to regular dicts. If the
implementation happens to keep that, great. Maybe OrderedDict can be
rewritten to use the dict implementation. But the evidence that all
implementations will always be fine with this restraint feels poor, and we
have a perfectly good explicit OrderedDict for those who want that.

On Nov 6, 2017 7:39 PM, "Brett Cannon"  wrote:

>
>
> On Mon, 6 Nov 2017 at 11:08 Paul Sokolovsky  wrote:
>
>> Hello,
>>
>> On Mon, 06 Nov 2017 17:58:47 +
>> Brett Cannon  wrote:
>>
>> []
>>
>> > > Why suddenly once in 25 years there's a need to do something to
>> > > dict's, violating computer science background behind them (one of
>> > > the reason enough people loved Python comparing to other "practical
>> > > hack" languages)?
>> >
>> > I don't understand what "computer science background" is being
>> > violated?
>>
>> I tried to explain that in the previous mail, can try a different
>> angle. So, please open you favorite CS book (better few) and look up
>> "abstract data types", then "mapping/associative array" and "list". We
>> can use Wikipedia too: https://en.wikipedia.org/wiki/Associative_array.
>> So, please look up: "Operations associated with this data type allow".
>> And you'll see, that there're no "ordering" related operations are
>> defined. Vice versa, looking at "sequence" operations, there will be
>> "prev/next", maybe "get n'th" element operations, implying ordering.
>>
>
> I don't think you meant for this to come off as insulting, but telling me
> how to look up the definition of an associative array or map feels like
> you're putting me down. I also have a Ph.D. in computer science so I'm
> aware of the academic definitions of these data structures.
>
>
>>
>> Python used to be a perfect application of these principles. Its dict
>> was a perfect CS implementation of an abstract associative array, and
>> list - of "sequence" abstract type (with additional guarantee of O(1)
>> random element access).
>
>
>> People knew and rejoiced that Python is built on solid science
>> principles, or could *learn* them from it.
>
> That no longer will be true,
>> with a sound concept being replaced with on-the-spot practical hack,
>> choosing properties of a random associative array algorithm
>> implementation over properties of a superset of such algorithms (many
>> of which are again don't offer any orderness guarantees).
>>
>>
> I don't think it's fair to call the current dict implementation a hack.
> It's a sound design that has a certain property that we are discussing the
> masking of. As I said previously, I think this discussion comes down to
> whether we think there are pragmatic benefits to exposing the ordered
> aspects to the general developer versus not.
>
> -Brett
>
>
>>
>>
>> I know though what will be replied (based on the replies below): "all
>> these are implementation details" - no, orderness vs non-orderness of a
>> mapping algorithm is an implementation detail; "users shouldn't know all
>> that" - they should, that's the real knowledge, and up until now, they
>> could learn that from *Python docs*, "we can't predict future" - we
>> don't need, we just need to know the past (25 years in our case), and
>> understand why it was done like that, I don't think Guido couldn't code
>> it ordered in 1991, it's just not natural for a mapping type to be so,
>> and in 2017, it's not more natural than it was in 1991.
>
>
>>
>> MicroPython in particular appeared because Python offered all the
>> CS-sound properties and freedom and alternative choices for
>> implementation (more so than any other scripting language). It's losing
>> it, and not just for MicroPython's surprise.
>>
>>
>> []
>>
>>
>> --
>> Best regards,
>>  Paul  mailto:pmis...@gmail.com
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mertz%40gnosis.cx
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Nick Coghlan]

On 6 November 2017 at 12:18, Nick Coghlan  wrote:
> That particular dependency could also be avoided by defining an
> "is_class_var(annotation)" generic function and a "ClassVar" helper
> object in the dataclasses module. For example:
>
> class _ClassVar:
> def __init__(self, annotation):
> self.annotation = annotation
>
> class _MakeClassVar:
> def __getitem__(self, key):
> return _ClassVar(key)
>
> ClassVar = _MakeClassVar()
>
> @functools.singledispatch
> def is_class_var(annotation):
> return isinstance(annotation, _ClassVar)
>
> It would put the burden on static analysers and the typing module to
> understand that `dataclasses.ClassVar` meant the same thing
> conceptually as `typing.ClassVar`, but I think that's OK.

Eric filed a new issue for this idea here:
https://github.com/ericvsmith/dataclasses/issues/61

As I indicated in my comment there, I'm now wondering if there might
be an opportunity here whereby we could use the *dataclasses* module
to define a stable non-provisional syntactically compatible subset of
the typing module, and require folks to start explicitly depending on
the typing module (as a regular PyPI dependency) if they want anything
more sophisticated than that (Unions, Generics, TypeVars, etc).

Unlike the typing module, dataclasses *wouldn't* attempt to give
annotations any meaningful runtime semantics, they'd just be ordinary
class instances (hence no complex metaclasses required, hence a
relatively fast import time).

Beyond the API already proposed in PEP 557, this would mean adding:

* dataclasses.ClassVar (as proposed above)
* dataclasses.Any (probably just set to the literal string "dataclasses.Any")
* dataclasses.NamedTuple (as a replacement for typing.NamedTuple)
* potentially dataclasses.is_class_var (instead of dataclasses
implicitly snooping in sys.modules looking for a "typing" module)

In effect, where we now have "typing" and "typing_extensions", we'd
instead have "dataclasses" in the standard library (with the subset of
annotations needed to define data record classes), and "typing" as an
independently versioned module on PyPI.

I think such an approach could address a few problems:

- the useful-at-runtime concepts (in dataclasses) would be clearly
separated from the static-type-analysis enablers (in typing)
- type hints would still have a clear presence in the regular standard
library, but the more complex parts would be opt-in (similar to
statistics vs SciPy, secrets vs cryptography, etc)
- as various concepts in typing matured and became genuinely stable,
they could potentially "graduate" into the standard library's
dataclasses module

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 12:18:17PM +0200, Paul Sokolovsky wrote:

> > I don't think that situation should change the decision, 
> 
> Indeed, it shouldn't. What may change it is the simple and obvious fact
> that there's no need to change anything, as proven by the 25-year
> history of the language.

I disagree -- the history of Python shows that having dicts be unordered 
is a PITA for many Python programmers. Python eventually gained an 
ordered dict because it provides useful functionality that developers 
demand.

Every new generation of Python programmers comes along and gets confused 
by why dicts mysteriously change their order from how they were entered, 
why doctests involving dicts break, why keyword arguments lose their 
order, why they have to import a module to get ordered dicts instead of 
having it be built-in, etc. Historically, we had things like 
ConfigParser reordering ini files when you write them.

Having dicts be unordered is not a positive virtue, it is a limitation. 
Up until now, it was the price we've paid for having fast, O(1) dicts. 
Now we have a dict implementation which is fast, O(1) and ordered. Why 
pretend that we don't? This is a long-requested feature, and the cost 
appears to be small: by specifying this, all we do is rule out some, but 
not all, hypothetical future optimizations.

Unordered dicts served CPython well for 20+ years, but I doubt many 
people will miss them.


> What happens now borders on technologic surrealism - the CPython, after
> many years of persuasion, switched its dict algorithm, rather
> inefficient in terms of memory, to something else, less inefficient
> (still quite inefficient, taking "no overhead" as the baseline).

Trading off space for time is a very common practice. You said that 
lookups on MicroPython's dicts are O(N). How efficient is µPy when doing 
a lookup of a dict with ten million keys?

µPy has chosen to optimize for space, rather than time. That's great. 
But I don't think you should sneer at CPython's choice to optimize for 
time instead.

And given that µPy's dicts already fail to meet the expected O(1) dict 
behviour, and the already large number of functional differences (not 
just performance differences) between µPy and Python:

http://docs.micropython.org/en/latest/pyboard/genrst/index.html

I don't think that this will make much practical difference. MicroPython 
users already cannot expect to run arbitrary Python code that works in 
other implementations: the Python community is fragmented between µPy 
code written for tiny machines, and Python code for machines with lots 
of memory.


> That
> algorithm randomly had another property. Now there's a seemingly
> serious talk of letting that property leak into the *language spec*,

It will no more be a "leak" than any other deliberate design choice.


> despite the fact that there can be unlimited number of dictionary
> algorithms, most of them not having that property. 

Sure. So what? There's an unlimited number of algorithms that don't 
provide the functionality that we want. There are an unlimited number of 
sort algorithms, but Python guarantees that we're only going to use 
those that are stable. Similar applies for method resolution (which µPy 
already violates), strings, etc.


> What it will lead to is further fragmentation of the community.

Aren't you concerned about fragmenting the community because of the 
functional differences between MicroPython and the specs?

Sometimes a small amount of fragmentation is unavoidable, and not 
necessarily a bad thing.


> > P.S. If anyone does want to explore MicroPython's dict implementation,
> > and see if there might be an alternate implementation strategy that
> > offers both O(1) lookup and guaranteed ordering without using
> > additional memory
> 
> That would be the first programmer in the history to have a cake and
> eat it too. Memory efficiency, runtime efficiency, sorted order: choose
> 2 of 3.

Given that you state that µPy dicts are O(N) and unordered, does that 
mean you picked only 1 out of 3?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Brett Cannon]

On Mon, 6 Nov 2017 at 11:08 Paul Sokolovsky  wrote:

> Hello,
>
> On Mon, 06 Nov 2017 17:58:47 +
> Brett Cannon  wrote:
>
> []
>
> > > Why suddenly once in 25 years there's a need to do something to
> > > dict's, violating computer science background behind them (one of
> > > the reason enough people loved Python comparing to other "practical
> > > hack" languages)?
> >
> > I don't understand what "computer science background" is being
> > violated?
>
> I tried to explain that in the previous mail, can try a different
> angle. So, please open you favorite CS book (better few) and look up
> "abstract data types", then "mapping/associative array" and "list". We
> can use Wikipedia too: https://en.wikipedia.org/wiki/Associative_array.
> So, please look up: "Operations associated with this data type allow".
> And you'll see, that there're no "ordering" related operations are
> defined. Vice versa, looking at "sequence" operations, there will be
> "prev/next", maybe "get n'th" element operations, implying ordering.
>

I don't think you meant for this to come off as insulting, but telling me
how to look up the definition of an associative array or map feels like
you're putting me down. I also have a Ph.D. in computer science so I'm
aware of the academic definitions of these data structures.


>
> Python used to be a perfect application of these principles. Its dict
> was a perfect CS implementation of an abstract associative array, and
> list - of "sequence" abstract type (with additional guarantee of O(1)
> random element access).


> People knew and rejoiced that Python is built on solid science
> principles, or could *learn* them from it.

That no longer will be true,
> with a sound concept being replaced with on-the-spot practical hack,
> choosing properties of a random associative array algorithm
> implementation over properties of a superset of such algorithms (many
> of which are again don't offer any orderness guarantees).
>
>
I don't think it's fair to call the current dict implementation a hack.
It's a sound design that has a certain property that we are discussing the
masking of. As I said previously, I think this discussion comes down to
whether we think there are pragmatic benefits to exposing the ordered
aspects to the general developer versus not.

-Brett


>
>
> I know though what will be replied (based on the replies below): "all
> these are implementation details" - no, orderness vs non-orderness of a
> mapping algorithm is an implementation detail; "users shouldn't know all
> that" - they should, that's the real knowledge, and up until now, they
> could learn that from *Python docs*, "we can't predict future" - we
> don't need, we just need to know the past (25 years in our case), and
> understand why it was done like that, I don't think Guido couldn't code
> it ordered in 1991, it's just not natural for a mapping type to be so,
> and in 2017, it's not more natural than it was in 1991.


>
> MicroPython in particular appeared because Python offered all the
> CS-sound properties and freedom and alternative choices for
> implementation (more so than any other scripting language). It's losing
> it, and not just for MicroPython's surprise.
>
>
> []
>
>
> --
> Best regards,
>  Paul  mailto:pmis...@gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Terry Reedy]

On 11/6/2017 9:47 PM, Nick Coghlan wrote:

On 7 November 2017 at 05:00, Alex Gaynor  wrote:

I also feel this decision was a mistake. If there's a consensus to revert,
I'm happy to draft a PEP.


Even without consensus to revert, I think it would be great to have a
PEP summarising some of the trade-offs between the different optio
I hope that there is consensus that neither the old nor current default 
is necessarily the best we can do.



And I do think there's openness to improving the situation, it's just
not clear yet that simple reversion of the previous change is the
right approach to attaining that improvement. In particular, the point
Georg Brandl raised about the poor interaction between the "-W"
command line option and Python code that calls
"warnings.filterwarnings" is still valid, and still a cause for
concern (whereby if code disables deprecation warnings, there's no
command line option or environment variable that can be used to
override that setting and force all the warnings to be shown despite
what the code says).

More generally, we seem to have (at least) 4 different usage models desired:

- "show my users nothing about legacy calls" (the app dev with a user
provided Python use case)


I believe that this is the current default.  But in practice, it often 
also means 'show the developer nothing about legacy calls', and therein 
lies the problem with the current default.



- "show me all legacy calls" (the testing use case, needs to be a
command line/environment setting)


I believe that this was the old default.  And I understand that it is 
the default when running the CPython test suite.  For testing the 
stdlib, it works because we control all the dependencies of each module. 
 But for other apps, the problem for both users and developers being 
overwhelmed with warnings from sources not in one's control.



- "only show me legacy calls in *my* code" (the "I trust my deps to
take care of themselves" use case)


Perhaps this should be the new default, where 'my code' means everything 
under the directory containing the startup file.  If an app developer 
either fixes or suppresses warnings from app code when they first 
appear, then users will seldom or never see warnings.  So for users, 
this would then be close to the current default.


Even for unmaintained code, the noise should be someone limited in most 
cases.



- "show me all legacy calls into a particular dependency" (the "I'm
considering upgrading this" use case, handled by the warn module on
PyPI)


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[INADA Naoki]

As memory footprint and import time point of view, I prefer string to thunk.

We can intern strings, but not lambda.
Dict containing only strings is not tracked by GC,
dict containing lambdas is tracked by GC.
INADA Naoki  


On Tue, Nov 7, 2017 at 8:20 AM, Lukasz Langa  wrote:
>
>
>> On Nov 5, 2017, at 11:28 PM, Nick Coghlan  wrote:
>>
>> On 6 November 2017 at 16:36, Lukasz Langa  wrote:
>>
>> - compile annotations like a small nested class body (but returning
>> the expression result, rather than None)
>> - emit MAKE_THUNK instead of the expression's opcodes
>> - emit STORE_ANNOTATION as usual
>>
>
> Is the motivation behind creating thunks vs. reusing lambdas just the 
> difference in handling class-level scope? If so, would it be possible to just 
> modify lambdas to behave thunk-like there? It sounds like this would strictly 
> broaden the functionality of lambdas, in other words, wouldn't create 
> backwards incompatibility for existing code.
>
> Reusing lambdas (with extending them to support class-level scoping) would be 
> a less scary endeavor than introducing a brand new language construct.
>
> With my current understanding I still think stringification is both easier to 
> implement and understand by end users. The main usability win of 
> thunks/lambdas is not very significant: evaluating them is as easy as calling 
> them whereas strings require typing.get_type_hints(). I still think being 
> able to access function-local state at time of definition is only 
> theoretically useful.
>
> What would be significant though is if thunk/lambdas helped fixing forward 
> references in general. But I can't really see how that could work.
>
> - Ł
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Nick Coghlan]

On 7 November 2017 at 05:00, Alex Gaynor  wrote:
> I also feel this decision was a mistake. If there's a consensus to revert,
> I'm happy to draft a PEP.

Even without consensus to revert, I think it would be great to have a
PEP summarising some of the trade-offs between the different options.

And I do think there's openness to improving the situation, it's just
not clear yet that simple reversion of the previous change is the
right approach to attaining that improvement. In particular, the point
Georg Brandl raised about the poor interaction between the "-W"
command line option and Python code that calls
"warnings.filterwarnings" is still valid, and still a cause for
concern (whereby if code disables deprecation warnings, there's no
command line option or environment variable that can be used to
override that setting and force all the warnings to be shown despite
what the code says).

More generally, we seem to have (at least) 4 different usage models desired:

- "show my users nothing about legacy calls" (the app dev with a user
provided Python use case)
- "show me all legacy calls" (the testing use case, needs to be a
command line/environment setting)
- "only show me legacy calls in *my* code" (the "I trust my deps to
take care of themselves" use case)
- "show me all legacy calls into a particular dependency" (the "I'm
considering upgrading this" use case, handled by the warn module on
PyPI)

(Tangent: as indicated by my phrasing above, I'm wondering if "legacy
calls"/"calls to legacy APIs" might be better user-centric terminology
here, since those legacy calls are what we're actually trying to
enable people to find and change - deprecation warnings are merely a
tool designed to help serve that purpose)

One of the biggest sources of tension is that silent-by-default is
actually a good option when dependency upgrades are entirely in the
hands of the application developer - if you're using something like
pip-tools or pipenv to pin your dependencies to particular versions,
and bundling a specific Python runtime with your application, and
running your software through pre-merge CI, then "I am going to
upgrade my dependencies now" is just another source code change, and
your CI should pick up most major compatibility issues. In those
scenarios, runtime deprecation warnings really are noise most of the
time, since you already have a defined process for dealing with
potential API breakages in dependencies.

Where problems arise is when the decision on when to upgrade a
dependency *isn't* in the hands of the application developer (e.g.
folks using the system Python in a Linux distro, or an embedded Python
scripting engine in a larger application). In those cases, the runtime
deprecation warnings are important to notify users and maintainers
that the next version upgrade for the underlying platform may break
their code. This applies at higher levels, not just at the base Python
platform layer. For example, if Fedora rebases matplotlib, then
there's a chance that may require updating pitivi as well.

Given that tension, it may be that this is an area where it makes
sense for us to say "CPython uses [these warnings filters] by default,
but redistributors may set the default warnings filter differently.
Always use -W or PYTHONWARNINGS if you want to ensure a particular set
of default filters are active."

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Eric V. Smith]

On 11/6/2017 1:12 PM, Barry Warsaw wrote:

On Nov 5, 2017, at 20:47, Nick Coghlan  wrote:


warnings.silence_deprecations()
python -X silence-deprecations
PYTHONSILENCEDEPRECATIONS=x


It could be interesting to combine this with Tim's suggestion of
putting an upper version limit on the silencing, so the above may look
like:

warnings.ignore_deprecations((3, 7))
python -X ignore-deprecations=3.7
PYTHONIGNOREDEPRECATIONS=3.7


That could be cool as long as we also support wildcards, e.g. defaults along 
the lines of my suggestions above to ignore everything.


I'd like to see a command line or environment variable that says: "turn 
on deprecation warnings (and/or pending deprecation warnings), but do 
not show warnings for this list of modules (possibly regex's)".


Like:
PYTHONDEPRECATIONWARNINGSEXCEPTFOR=PIL,requests.*

Then I'd just turn it on for all modules (empty string?), and when I got 
something that was flooding me with output I'd add it to the list.


Eric.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Victor Stinner]

2017-11-07 0:41 GMT+01:00 Serhiy Storchaka :
> Several month ago there was a couple of buildbots including NetBSD and
> OpenBSD. What happened to them? Was the support of NetBSD and OpenBSD
> officially stopped as well as the support of OpenIndiana?

While I don't recall seeing any NetBSD buildbot, I recall that we had
a OpenBSD buildbot but it was red as far as I recall. Many tests were
failing on OpenBSD: test_crypt, test_socket, and many others.

Since we have much more "green" buildbot workers nowadays, I would
prefer to wait until almost all tests pass on a "new" platform, before
plugging a new buildbot. For example, I am now subscribed to the
buildbot-status@ mailing list which may be flooded by failures until a
buildbot becomes stable.

I'm ok with skipping enough tests for a "new" platforms to get a
"green" test suite, and fix skipped tests later. In my experience,
keeping tests which fail forever is annoying and hides new
regressions.

Even if a buildbot which always fail shouldn't send email
notifications, in practice, buildbot likes to send emails for various
reasons, even if the previous build was already a failure and the new
build is still a failure :-) But maybe we can adjust the buildbot
configuration to not send notifications on some buildbot workers.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Donald Stufft]

> On Nov 6, 2017, at 4:35 AM, Paul Moore  wrote:
> 
> 1. Without typing available, some programs using type annotations
> won't run. That is, using type annotations (a
> test-time/development-time feature) introduces a runtime dependency on
> typing, and hence introduces an extra deployment concern (unlike other
> development-type features like test frameworks).
> 2. For some people, if something isn't in the Python standard library
> (technically, in a standard install), it's not available (without
> significant effort, or possibly not at all). For those people, a
> runtime dependency on a non-stdlib typing module means "no use of type
> annotations allowed".
> 3. Virtual environments typically only include the stdlib, and "use
> system site-packages" has affects more than just a single module, so
> bundling still has issues for virtualenvs - and in the packaging
> tutorials, we're actively encouraging people to use virtualenvs. We
> (python-dev) can work around this issue for venv by auto-installing
> typing, but that still leaves virtualenv (which is critically short of
> resources, and needs to support older versions of Python, so a major
> change like bundling typing is going to be a struggle to get
> implemented).


Maybe we just need to fully flesh out the idea of a "Python Core" (What exists 
now as “Python”) and a “Python Platform” (Python Core + A select set of 
preinstalled libraries). Then typing can just be part of the Python Platform, 
and gets installed as part of your typical installation, but is otherwise an 
independent piece of code.___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Joao S. O. Bueno]

On 6 November 2017 at 17:23, Paul G  wrote:
> Is there a major objection to just adding in explicit syntax for 
> order-preserving dictionaries? To some extent that seems like a reasonable 
> compromise position in an "explicit is better than implicit" sense. A whole 
> lot of code is out there that doesn't require or expect order-preserving 
> dictionaries - it would be nice to be able to differentiate out the parts 
> where order actually *does* matter.
>
> (Of course, given that CPython's implementation is order-preserving, a bunch 
> of code is probably now being written that implicitly requires on this 
> detail, but at least having syntax that makes that clear would give people 
> the *option* to make the assumption explicit).

I think the additional syntax have the added benefit of preventing
code that relies on the ordering of dict literals to be run on older
versions, therefore triggering subtle bugs that had already being
mentioned.

And also, forgot along the discussion, is the big disadvantage that
other Python implementations would have a quite
significant overhead on mandatory ordered dicts. One that was
mentioned along the way is transpilers, with
Brython as an example - but there might be others. MircoPython is far
from being the only implementation affected.

  js
-><-

>
> On 11/06/2017 01:19 PM, Barry Warsaw wrote:
>> On Nov 6, 2017, at 02:18, Paul Sokolovsky  wrote:
>>
>>> What it will lead to is further fragmentation of the community. Python2
>>> vs Python3 split is far from being over, and now there're splits
>>> between:
>>>
>>> * people who use "yield from" vs "await"
>>> * people who use f-strings vs who don't
>>> * people who rely on sorted nature of dict's vs who don't
>>
>> This is the classic argument of, do you proceed conservatively and use the 
>> lowest-common denominator that makes your code work with the widest range of 
>> versions, or do you ride the wave and adopt the latest and greatest features 
>> as soon as they’re available?
>>
>> Neither answer is wrong or right… for everyone.  It’s also a debate as old 
>> as the software industry. Every package, project, company, developer, 
>> community will have to decide for themselves.  Once you realize you can’t 
>> win, you’ve won! :)
>>
>> -Barry
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
>>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

Nick Coghlan wrote:
> On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
> mistakenly though the async/await deprecation warnings were missing
> from 3.6.

Sometimes the universe just throws synchronicity right in your face.

I'm working on building an internal tool against Python 3.7 to take
advantage of the very cool -X importtime feature.  It's been a fun
challenge, but mostly because of our external dependencies.  For
example, PyThreadState renamed its structure members, so both Cython and
lxml needed new releases to adjust for this.  That's the "easy" part;
they've done it and those fixes work great.

We also depend on ldap3 .  Suddenly we
get a SyntaxError because ldap3 has a module ldap3/strategy/async.py.  I
say "suddenly" because of course *if* DeprecationWarnings had been
enabled by default, I'm sure someone would have noticed that those
imports were telling the developers about the impending problem in
Python 3.6.

https://github.com/cannatag/ldap3/issues/428

This just reinforces my opinion that even though printing
DeprecationWarning by default *can* be a hardship in some environments,
it is on the whole a positive beneficial indicator that gives developers
some runway to fix such problems.  These types of apparently sudden
breakages are the worse of all worlds.

Cheers,
-Barry


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Serhiy Storchaka]

06.11.17 19:41, Victor Stinner пише:

Example of platforms: MinGW, Cygwin, OpenBSD, NetBSD, xWorks RTOS, etc.

But the way, is there an exhaustive list of platforms "officially"
supported by CPython?


Several month ago there was a couple of buildbots including NetBSD and 
OpenBSD. What happened to them? Was the support of NetBSD and OpenBSD 
officially stopped as well as the support of OpenIndiana?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Serhiy Storchaka]

06.11.17 23:24, Antoine Pitrou пише:

On Mon, 6 Nov 2017 18:41:30 +0100
Victor Stinner  wrote:


Example of platforms: MinGW, Cygwin, OpenBSD, NetBSD, xWorks RTOS, etc.


We support POSIX-compatible platforms.  Do OpenBSD and NetBSD need
special care?


Yes, because our support is GNU/Linux-centric. Features not supported on 
Linux degrade with time.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

On Mon, Nov 6, 2017 at 11:23 AM, Paul G  wrote:

> (Of course, given that CPython's implementation is order-preserving, a
> bunch of code is probably now being written that implicitly requires on
> this detail, but at least having syntax that makes that clear would give
> people the *option* to make the assumption explicit).


This is a really key point -- a heck of a lot more people use cPython than
read the language spec. And a great deal of code is developed with a
certain level of ignorance -- try something, if it works, and your test
pass (if there are any), then you are done.

So right now, there is more an more code out there that relies on a regular
old dcit being ordered.

I've been struggling with teaching this right now -- my
written-a-couple-years ago materials talk about dicts being arbitrary
order, and then have a little demo of that fact. Now I'm teaching with
Python 3.6, and I had to add in something like:

   cPython does, in fact, preserve order with dicts, but it should be
considered an implementation detail, and not counted on ... (and by the
say, so does PyPy, and )"

I don't know, but I'm going to guess about 0% of my class is going to
remember that...

And if we added o{,,,} syntax it would be even worse, 'cause folks would
forget to use it, as their code wouldn't behave differently (kind of like
the 'b' flag on unix text files, or the u"string" where everything is ascii
in that string...)

in short -- we don't have a choice (unless we add an explicit randomization
as some suggested -- but that just seems perverse...)

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On Nov 5, 2017, at 11:28 PM, Nick Coghlan  wrote:
> 
> On 6 November 2017 at 16:36, Lukasz Langa  wrote:
> 
> - compile annotations like a small nested class body (but returning
> the expression result, rather than None)
> - emit MAKE_THUNK instead of the expression's opcodes
> - emit STORE_ANNOTATION as usual
> 

Is the motivation behind creating thunks vs. reusing lambdas just the 
difference in handling class-level scope? If so, would it be possible to just 
modify lambdas to behave thunk-like there? It sounds like this would strictly 
broaden the functionality of lambdas, in other words, wouldn't create backwards 
incompatibility for existing code.

Reusing lambdas (with extending them to support class-level scoping) would be a 
less scary endeavor than introducing a brand new language construct.

With my current understanding I still think stringification is both easier to 
implement and understand by end users. The main usability win of thunks/lambdas 
is not very significant: evaluating them is as easy as calling them whereas 
strings require typing.get_type_hints(). I still think being able to access 
function-local state at time of definition is only theoretically useful.

What would be significant though is if thunk/lambdas helped fixing forward 
references in general. But I can't really see how that could work.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Victor Stinner]

2017-11-06 23:07 GMT+01:00 Antoine Pitrou :
> The reason that testing on
> them is interesting, IMHO, is to chase potential Linux-isms in our code
> base.  Circumventing {Free,Open,Net}BSD-specific deficiences is not.

Serhiy found at least an interesting issue thanks to OpenBSD, a bug in
memory debug hooks:

   https://bugs.python.org/issue31626

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Antoine Pitrou]

On Mon, 6 Nov 2017 22:49:06 +0100
Victor Stinner  wrote:
> 
> CPython already contais 11 "#ifdef (...) __OpenBSD__" in C and 11
> sys.platform.startswith('openbsd') in Python. Supporting a "POSIX"
> platform requires some changes :-)

Yes... So, the question is: does OpenBSD already maintain a Python
port?  If so, then we should have them contribute their patches instead
of fixing issues ourselves.  If they don't want too, then we have no
obligation to maintain them.

OTOH, if you and Serhiy want to maintain them, you can do so.  But we
shouldn't promise anything for the future.  The reason that testing on
them is interesting, IMHO, is to chase potential Linux-isms in our code
base.  Circumventing {Free,Open,Net}BSD-specific deficiences is not.

To put things differently: if a commit breaks {Free,Open,Net}BSD but
leaves other platforms intact, I don't think it should be reverted.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Victor Stinner]

2017-11-06 22:16 GMT+01:00 Steve Dower :
> I don't believe CPython *partially* supports any platforms - either they are
> fully supported or they are not supported.

Ok. So there are two questions:

* Where is the list of platforms "endorsed" by CPython ("fully supported")

* How can we decide when to start to support support a new platform?

Usually, developers begin with a nice looking change: short, not
intrusive. But later, the real change comes, and it's much larger and
uglier :-) And the rationale for the first change is "come on, it's
just a few lines!". And I have troubles to justify to reject such
patch. Slowly, I'm pointing people to the right section of the PEP 11.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Victor Stinner]

2017-11-06 22:24 GMT+01:00 Antoine Pitrou :
> We support POSIX-compatible platforms.  Do OpenBSD and NetBSD need
> special care?

Aha, "POSIX", you are funny Antoine :-D

If it was a single #ifdef in the whole code base, I wouldn't have to
start such thread on python-dev :-)


Open issues with "OpenBSD" in the title:

31630: math.tan has poor accuracy near pi/2 on OpenBSD and NetBSD
31631: test_c_locale_coercion fails on OpenBSD
31635: test_strptime failure on OpenBSD
31636: test_locale failure on OpenBSD
25342: test_json segfault on OpenBSD
25191: test_getsetlocale_issue1813 failed on OpenBSD
23470: OpenBSD buildbot uses wrong stdlib
12588: test_capi.test_subinterps() failed on OpenBSD (powerpc)
12589: test_long.test_nan_inf() failed on OpenBSD (powerpc)

CPython already contais 11 "#ifdef (...) __OpenBSD__" in C and 11
sys.platform.startswith('openbsd') in Python. Supporting a "POSIX"
platform requires some changes :-)


Open issues with "NetBSD" in the title:

31925: [NetBSD] test_socket creates too many locks
30512: CAN Socket support for NetBSD
31927: Fix compiling the socket module on NetBSD 8 and other issues
31630: math.tan has poor accuracy near pi/2 on OpenBSD and NetBSD
31894: test_timestamp_naive failed on NetBSD

CPython already contains 17 "#ifdef (...) __NetBSD__" and 7
"platform.startswith('netbsd')" in Python.


MinGW:

... hum, there are 57 open MinGW issues, use the bug tracker to list them ;-)

Cygwin:

... hum, there are 23 open Cygwin issues, use the bug tracker to list them ;-)

etc.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Antoine Pitrou]

On Mon, 6 Nov 2017 18:41:30 +0100
Victor Stinner  wrote:
> 
> Example of platforms: MinGW, Cygwin, OpenBSD, NetBSD, xWorks RTOS, etc.

We support POSIX-compatible platforms.  Do OpenBSD and NetBSD need
special care?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Steve Dower]

On 06Nov2017 0941, Victor Stinner wrote:

[SNIP]

But the question here is more about "partial" support.

While changes are usually short, I dislike applying them to Python 2.7
and/or Python 3.6, until a platform is fully support. I prefer to
first see a platform fully supported to see how much changes are
required and to make sure that we get someone involved to maintain the
code (handle new issues).

Example of platforms: MinGW, Cygwin, OpenBSD, NetBSD, xWorks RTOS, etc.


I appreciate the desire for changes to be made upstream, especially on 
code that changes frequently enough to make it difficult to patch 
reliably (this is basically the entire premise behind my PEP 551). At 
the same time, I don't like carrying the burden of code for platforms we 
do not support (hence PEP 551 doesn't really add any interesting code). 
There is a balance to be found here, though.


I don't believe CPython *partially* supports any platforms - either they 
are fully supported or they are not supported.


However, we can and should do things that help other people support 
their platforms, such as making sure build options can be specified by 
environment variables. At the same time, we can and should *exclude* 
things that require platform-specific testing in core (for example, 
predefined options for building for a specific platform).


Similarly, if someone wanted to add specific platform support to a 
stdlib module via "if sys.platform ...", I'd be hesitant. However, if 
they refactored it such that it was easier *for a custom build of 
CPython* to provide/omit certain features (e.g. 
https://github.com/python/cpython/blob/30f4fa456ef626ad7a92759f492ec7a268f7af4e/Lib/threading.py#L1290-L1296 
) then I'd be more inclined to accept it - but only if there was no 
behavioural change on supported platforms.


Does that make sense? I'm not sure whether we ought to capture some 
general guidelines somewhere on how to make decisions around this, but 
we should certainly have the discussion here first anyway.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Serhiy Storchaka]

06.11.17 21:56, Paul Sokolovsky пише:

Btw, in all this discussion, I don't remember anyone mentioning sets. I
don't recall the way they're implemented in CPython, but they have
strong conceptual and semantic resemblance to dict's. So, what about
them, do they become ordered too?


No, sets are not ordered, and seems they will not be ordered at least in 
several next years.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Mon, 6 Nov 2017 11:33:10 -0800
Barry Warsaw  wrote:

[]

> If we did make the change, it’s possible we would need a way to
> explicit say that order is not preserved.  That seems a little weird

I recently was working on a more or less complex dataflow propagation
problem. It should converge to a fixed point, and it did, but on
different runs, to different ones. So, I know that I'm a bad
programmer, need to do more of my homework and grow. I know, that if I
rewrite it in C++ or C, it'll work unstable the same way, because it's
buggy. (Heck, over these years, I learned that I don't need to rewrite
things in C/C++, because Python is the *real* language, which works the
way computers do, without sugaring that up).

I need to remember that, because with Python 3.7, I may become a
good-programmer-in-a-ponyland-of-ordered-dicts.


Btw, in all this discussion, I don't remember anyone mentioning sets. I
don't recall the way they're implemented in CPython, but they have
strong conceptual and semantic resemblance to dict's. So, what about
them, do they become ordered too?


> 
> Cheers,
> -Barry
> 



-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 6, 2017, at 11:23, Paul G  wrote:
> 
> Is there a major objection to just adding in explicit syntax for 
> order-preserving dictionaries?

I don’t think new syntax is necessary.  We already have OrderedDict, which to 
me is a perfectly sensible way to spell “I need a mapping that preserves 
insertion order”, and the extra import doesn’t bother me.

I’m not saying whether or not to make the language guarantee that built-in dict 
preserves order.  I’m just saying that if we don’t make that language change, 
we already have everything we need to support both use cases.

If we did make the change, it’s possible we would need a way to explicit say 
that order is not preserved.  That seems a little weird to me, but I suppose it 
could be useful.  I like the idea previously brought up that iteration order be 
deliberately randomized in that case, but we’d still need a good way to spell 
that.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul G]

Is there a major objection to just adding in explicit syntax for 
order-preserving dictionaries? To some extent that seems like a reasonable 
compromise position in an "explicit is better than implicit" sense. A whole lot 
of code is out there that doesn't require or expect order-preserving 
dictionaries - it would be nice to be able to differentiate out the parts where 
order actually *does* matter.

(Of course, given that CPython's implementation is order-preserving, a bunch of 
code is probably now being written that implicitly requires on this detail, but 
at least having syntax that makes that clear would give people the *option* to 
make the assumption explicit).

On 11/06/2017 01:19 PM, Barry Warsaw wrote:
> On Nov 6, 2017, at 02:18, Paul Sokolovsky  wrote:
> 
>> What it will lead to is further fragmentation of the community. Python2
>> vs Python3 split is far from being over, and now there're splits
>> between:
>>
>> * people who use "yield from" vs "await"
>> * people who use f-strings vs who don't
>> * people who rely on sorted nature of dict's vs who don't
> 
> This is the classic argument of, do you proceed conservatively and use the 
> lowest-common denominator that makes your code work with the widest range of 
> versions, or do you ride the wave and adopt the latest and greatest features 
> as soon as they’re available?
> 
> Neither answer is wrong or right… for everyone.  It’s also a debate as old as 
> the software industry. Every package, project, company, developer, community 
> will have to decide for themselves.  Once you realize you can’t win, you’ve 
> won! :)
> 
> -Barry
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
> 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Eric V. Smith]

On 11/6/2017 1:40 PM, Barry Warsaw wrote:

On Nov 6, 2017, at 08:02, Victor Stinner  wrote:


While discussions on the typing module are still hot, what do you
think of allowing annotations in the standard libraries, but limited
to a few basic types:


I’m still -1 on adding annotations to the stdlib, despite their increasing use 
out in the wild, for the reasons that Steve and David have pointed out.  (Let’s 
let Eric be the one that breaks the mold with data classes.  Then we can blame 
him!)


Thanks for volunteering me!

Note that dataclasses completely ignores the annotations (they could all 
be None, for all I care), except for the one specific case of ClassVar. 
And that's still up for discussion, see 
https://github.com/ericvsmith/dataclasses/issues/61.


Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Tres Seaver]

On 11/06/2017 01:25 PM, R. David Murray wrote:
> Maybe I'm being a curmudgeon standing in the way of progress, but I'm
> pretty sure there are a number of people in my camp :)

I'm definitely there: anything which optimizes machine-readabilty over
readability for the Mark 1 eyeball is a lose.


Tres.
-- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Mon, 06 Nov 2017 17:58:47 +
Brett Cannon  wrote:

[]

> > Why suddenly once in 25 years there's a need to do something to
> > dict's, violating computer science background behind them (one of
> > the reason enough people loved Python comparing to other "practical
> > hack" languages)?
> 
> I don't understand what "computer science background" is being
> violated?

I tried to explain that in the previous mail, can try a different
angle. So, please open you favorite CS book (better few) and look up
"abstract data types", then "mapping/associative array" and "list". We
can use Wikipedia too: https://en.wikipedia.org/wiki/Associative_array.
So, please look up: "Operations associated with this data type allow".
And you'll see, that there're no "ordering" related operations are
defined. Vice versa, looking at "sequence" operations, there will be
"prev/next", maybe "get n'th" element operations, implying ordering.

Python used to be a perfect application of these principles. Its dict
was a perfect CS implementation of an abstract associative array, and
list - of "sequence" abstract type (with additional guarantee of O(1)
random element access).

People knew and rejoiced that Python is built on solid science
principles, or could *learn* them from it. That no longer will be true,
with a sound concept being replaced with on-the-spot practical hack,
choosing properties of a random associative array algorithm
implementation over properties of a superset of such algorithms (many
of which are again don't offer any orderness guarantees).



I know though what will be replied (based on the replies below): "all
these are implementation details" - no, orderness vs non-orderness of a
mapping algorithm is an implementation detail; "users shouldn't know all
that" - they should, that's the real knowledge, and up until now, they
could learn that from *Python docs*, "we can't predict future" - we
don't need, we just need to know the past (25 years in our case), and
understand why it was done like that, I don't think Guido couldn't code
it ordered in 1991, it's just not natural for a mapping type to be so,
and in 2017, it's not more natural than it was in 1991.


MicroPython in particular appeared because Python offered all the
CS-sound properties and freedom and alternative choices for
implementation (more so than any other scripting language). It's losing
it, and not just for MicroPython's surprise.


[]


-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Enabling depreciation warnings feature code cutoff

[Neil Schemenauer]

On 2017-11-06, Nick Coghlan wrote:
> Gah, seven years on from Python 2.7's release, I still get caught by
> that. I'm tempted to propose we reverse that decision and go back to
> enabling them by default :P

Either enable them by default or make them really easy to enable for
development evironments.  I think some setting of the PYTHONWARNINGS
evironment variable should do it.  It is not obvious to me how to do
it though.  Maybe there should be an environment variable that does
it more directly.  E.g.

PYTHONWARNDEPRECATED=1

Another idea is to have venv to turn them on by default or, based on
a command-line option, do it.  Or, maybe the unit testing frameworks
should turn on the warnings when they run.

The current "disabled by default" behavior is obviously not working
very well.  I had them turned on for a while and found quite a
number of warnings in what are otherwise high-quality Python
packages.  Obviously the vast majority of developers don't have them
turned on.

Regards,

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Alex Gaynor]

I also feel this decision was a mistake. If there's a consensus to revert,
I'm happy to draft a PEP.

Alex

On Nov 6, 2017 1:58 PM, "Neil Schemenauer"  wrote:

> On 2017-11-06, Nick Coghlan wrote:
> > Gah, seven years on from Python 2.7's release, I still get caught by
> > that. I'm tempted to propose we reverse that decision and go back to
> > enabling them by default :P
>
> Either enable them by default or make them really easy to enable for
> development evironments.  I think some setting of the PYTHONWARNINGS
> evironment variable should do it.  It is not obvious to me how to do
> it though.  Maybe there should be an environment variable that does
> it more directly.  E.g.
>
> PYTHONWARNDEPRECATED=1
>
> Another idea is to have venv to turn them on by default or, based on
> a command-line option, do it.  Or, maybe the unit testing frameworks
> should turn on the warnings when they run.
>
> The current "disabled by default" behavior is obviously not working
> very well.  I had them turned on for a while and found quite a
> number of warnings in what are otherwise high-quality Python
> packages.  Obviously the vast majority of developers don't have them
> turned on.
>
> Regards,
>
>   Neil
> ___
> python-committers mailing list
> python-committ...@python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Skip Montanaro]

> The PEP 11 has a nice description to get a *full* support of a new platform:
> https://www.python.org/dev/peps/pep-0011/

PEP 11 defines the endpoint, full support, and several requirements to
call a platform fully supported. It would be nice if a process was
defined for getting from "no support" to "full support." I think that
to be as supportive as possible (no pun intended), it would make sense
to grant people check-in privileges to a branch (if that's possible or
desirable), or, if not, list forks which are intended to add support
for various platforms which are moving toward full support. I don't
know if PEP 11 is the right place to track such activity, relatively
few updates should be required. I doubt it will be an onerous burden.
Something like:

* ButterflyOS - Victor Stinner (victor.stin...@gmail.com) is working
to add CPython support for this platform on this Git branch:
https://github.com/python/Butterfly
* MothOS - Skip Montanaro (skip.montan...@gmail.com) is working to add
CPython support for this platform on this Git branch:
https://github.com/smontanaro/Moth

Interested parties would be directed to contact the pilots of those
branches if they wanted to contribute.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Barry Warsaw]

On Nov 6, 2017, at 08:02, Victor Stinner  wrote:
> 
> While discussions on the typing module are still hot, what do you
> think of allowing annotations in the standard libraries, but limited
> to a few basic types:

I’m still -1 on adding annotations to the stdlib, despite their increasing use 
out in the wild, for the reasons that Steve and David have pointed out.  (Let’s 
let Eric be the one that breaks the mold with data classes.  Then we can blame 
him!)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Brett Cannon]

On Mon, Nov 6, 2017, 10:27 R. David Murray,  wrote:

> I agree with Steve.  There is *cognitive* overhead to type annotations.
> I find that they make Python code harder to read and understand.  So I
> object to them in the documentation and docstrings as well.  (Note:
> while I agree that the notation is compact for the simple types, the
> fact that it would appear for some signatures and not for others is a
> show stopper from my point of view...consistency is important to reducing
> the cognitive overhead of reading the docs.)
>
> I'm dealing with the spread of annotations on my current project,
> having to ask programmers on the team to delete annotations that they've
> "helpfully" added that to my mind serve no purpose on a project of the
> size we're developing, where we aren't using static analysis for anything.
>

I think this is the key point in your situation if the project is private.


> Maybe I'm being a curmudgeon standing in the way of progress, but I'm
> pretty sure there are a number of people in my camp :)
>

The key thing here is there are people like me who are using your analyzers
(and you are as well indirectly since the CLA bot is fully type hinted 😁).
I think the key question is whether we expect typeshed to keep up with
richer annotations using typing or are basic ones in the stdlib going ot
leave less of a gap in support in the long-term?

To be honest, I suspect most Python code in the stdlib would require
protocols to be accurate (C code is another matter), but return type hints
could be reasonably accurate.

-Brett


> On Mon, 06 Nov 2017 16:22:23 +, Steve Holden 
> wrote:
> > While I appreciate the value of annotations I think that *any* addition
> of
> > them to the stdlib would complicate an important learning resource
> > unnecessarily. S
> >
> > Steve Holden
> >
> > On Mon, Nov 6, 2017 at 4:07 PM, Victor Stinner  >
> > wrote:
> >
> > > Related to annotations, are you ok to annotate basic types in the
> > > *documentation* and/or *docstrings* of the standard library?
> > >
> > > For example, I chose to document the return type of time.time() (int)
> > > and time.time_ns() (float). It's short and I like how it's formatted.
> > > See the current rendered documentation:
> > >
> > > https://docs.python.org/dev/library/time.html#time.time
> > >
> > > "Annotations" in the documentation and docstrings have no impact on
> > > Python runtime performance. Annotations in docstrings makes them a few
> > > characters longer and so impact the memory footprint, but I consider
> > > that the overhead is negligible, especially when using python3 -OO.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[R. David Murray]

I agree with Steve.  There is *cognitive* overhead to type annotations.
I find that they make Python code harder to read and understand.  So I
object to them in the documentation and docstrings as well.  (Note:
while I agree that the notation is compact for the simple types, the
fact that it would appear for some signatures and not for others is a
show stopper from my point of view...consistency is important to reducing
the cognitive overhead of reading the docs.)

I'm dealing with the spread of annotations on my current project,
having to ask programmers on the team to delete annotations that they've
"helpfully" added that to my mind serve no purpose on a project of the
size we're developing, where we aren't using static analysis for anything.

Maybe I'm being a curmudgeon standing in the way of progress, but I'm
pretty sure there are a number of people in my camp :)

On Mon, 06 Nov 2017 16:22:23 +, Steve Holden  wrote:
> While I appreciate the value of annotations I think that *any* addition of
> them to the stdlib would complicate an important learning resource
> unnecessarily. S
> 
> Steve Holden
> 
> On Mon, Nov 6, 2017 at 4:07 PM, Victor Stinner 
> wrote:
> 
> > Related to annotations, are you ok to annotate basic types in the
> > *documentation* and/or *docstrings* of the standard library?
> >
> > For example, I chose to document the return type of time.time() (int)
> > and time.time_ns() (float). It's short and I like how it's formatted.
> > See the current rendered documentation:
> >
> > https://docs.python.org/dev/library/time.html#time.time
> >
> > "Annotations" in the documentation and docstrings have no impact on
> > Python runtime performance. Annotations in docstrings makes them a few
> > characters longer and so impact the memory footprint, but I consider
> > that the overhead is negligible, especially when using python3 -OO.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 6, 2017, at 02:18, Paul Sokolovsky  wrote:

> What it will lead to is further fragmentation of the community. Python2
> vs Python3 split is far from being over, and now there're splits
> between:
> 
> * people who use "yield from" vs "await"
> * people who use f-strings vs who don't
> * people who rely on sorted nature of dict's vs who don't

This is the classic argument of, do you proceed conservatively and use the 
lowest-common denominator that makes your code work with the widest range of 
versions, or do you ride the wave and adopt the latest and greatest features as 
soon as they’re available?

Neither answer is wrong or right… for everyone.  It’s also a debate as old as 
the software industry. Every package, project, company, developer, community 
will have to decide for themselves.  Once you realize you can’t win, you’ve 
won! :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Brett Cannon]

On Mon, 6 Nov 2017 at 00:30 Victor Stinner  wrote:

> 2017-11-06 8:47 GMT+01:00 Serhiy Storchaka :
> > 06.11.17 09:09, Guido van Rossum пише:
> >>
> >> I still find this unfriendly to users of Python scripts and small apps
> who
> >> are not the developers of those scripts. (Large apps tend to spit out so
> >> much logging it doesn't really matter.)
> >>
> >> Isn't there a better heuristic we can come up with so that the warnings
> >> tend to be on for developers but off for end users?
> >
> > There was a proposition to make deprecation warnings visible by default
> in
> > debug build and interactive interpreter.
>
> The problem is that outside CPython core developers, I expect that
> almost nobody runs a Python compiled in debug mode. We should provide
> debug features in the release build. For example, in Python 3.6, I
> added debug hooks on memory allocation in release mode using
> PYTHONMALLOC=debug. These hooks were already enabled by default in
> debug mode.
>
> Moreover, applications are not developed nor tested in the REPL.
>
> Last year, I proposed a global "developer mode". The idea is to
> provide the same experience than a Python debug build, but on a Python
> release build:
>
>   python3 -X dev script.py
> or
>   PYTHONDEV=1 python3 script.py
> behaves as
>   PYTHONMALLOC=debug python3 -Wd -b -X faulthandler script.py
>
> * Show DeprecationWarning and ResourceWarning warnings: python -Wd
> * Show BytesWarning warning: python -b
> * Enable Python assertions (assert) and set __debug__ to True: remove
> (or just ignore) -O or -OO command line arguments
> * faulthandler to get a Python traceback on segfault and fatal errors:
> python -X faulthandler
> * Debug hooks on Python memory allocators: PYTHONMALLOC=debug
>
> If you don't follow the CPython development, it's hard to be aware of
> "new" options like -X faulthandler (Python 3.3) or PYTHONMALLOC=debug
> (Python 3.6). And it's easy to forget an option like -b.
>
>
> Maybe we even a need -X dev=strict which would be stricter:
>
> * use -Werror instead of -Wd: raise an exception when a warning is emitted
> * use -bb instead of -b: get BytesWarning exceptions
> * Replace "inconsistent use of tabs and spaces in indentation" warning
> with an error in the Python parser
> * etc.
>

I like this idea and would argue that `-X dev` should encompass what's
proposed for `-X dev=strict` and just have it be strict to begin with. Then
we can add an equivalent environment variable and push people who use CI to
just blindly set the environment variable in their tests.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 5, 2017, at 20:47, Nick Coghlan  wrote:

>> warnings.silence_deprecations()
>> python -X silence-deprecations
>> PYTHONSILENCEDEPRECATIONS=x
> 
> It could be interesting to combine this with Tim's suggestion of
> putting an upper version limit on the silencing, so the above may look
> like:
> 
>warnings.ignore_deprecations((3, 7))
>python -X ignore-deprecations=3.7
>PYTHONIGNOREDEPRECATIONS=3.7

That could be cool as long as we also support wildcards, e.g. defaults along 
the lines of my suggestions above to ignore everything.

-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 5, 2017, at 23:08, Serhiy Storchaka  wrote:

> Following issues on GitHub related to new Python releases I have found that 
> many projects try to fix deprecation warning, but there are projects that are 
> surprised by ending of deprecation periods and removing features.

Like others here, I’ve also been bitten by silently ignored 
DeprecationWarnings.  We had some admittedly dodgy code in a corner of Mailman 
that we could have fixed earlier if we’d seen the warnings.  But we never did, 
so the first indication of a problem was when code actually *broke* with the 
new version of Python.  The problem was compounded because it wasn’t us that 
saw it first, it was a user, so now they had a broken installation and we had 
to issue a hot fix.  If we’d seen the DeprecationWarnings in the previous 
version of Python, we would have fixed them and all would have been good.

It’s true that those warnings can cause problems though.  There are certain 
build/CI environments, e.g. in Ubuntu, that fail when they see unexpected 
stderr output.  So when we start seeing new deprecations, we got build(-ish) 
time failures.  I still think that’s a minor price to pay for projects that 
*want* to do the right thing but don’t because those warnings are essentially 
hidden until they are breakages.  We have tools to help, so let’s use them.

Staying current and code clean is hard and never ending.  Welcome to software 
development!

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Brett Cannon]

On Mon, 6 Nov 2017 at 08:36 Paul Sokolovsky  wrote:

> Hello,
>
> On Mon, 6 Nov 2017 14:30:45 +0100
> Antoine Pitrou  wrote:
>
> > On Tue, 7 Nov 2017 00:14:35 +1100
> > Steven D'Aprano  wrote:
> > > On Mon, Nov 06, 2017 at 12:27:54PM +0100, Antoine Pitrou wrote:
> > >
> > > > The ordered-ness of dicts could instead become one of those stable
> > > > CPython implementation details, such as the fact that resources
> > > > are cleaned up timely by reference counting, that people
> > > > nevertheless should not rely on if they're writing portable
> > > > code.
> > >
> > > Given that (according to others) none of IronPython, Jython,
> > > Batavia, Nuitka, or even MicroPython, should have trouble
> > > implementing an insertion-order preserving dict, and that PyPy
> > > already has, why should we say it is a CPython implementation
> > > detail?
> >
> > That's not what I'm taking away from Paul Sokolovsky's message I was
> > responding to.  If you think otherwise then please expand and/or
> > contact Paul so that things are made clearer one way or the other.
>
> For MicroPython, it would lead to quite an overhead to make
> dictionary items be in insertion order. As I mentioned, MicroPython
> optimizes for very low bookkeeping memory overhead, so lookups are
> effectively O(n), but orderedness will increase constant factor
> significantly, perhaps 5x.
>
> Also, arguably any algorithm which would *maintain* insertion order
> over mutating operations would be more complex and/or require more
> memory that one which doesn't. (This is based on the idea that
> this problem is equivalent to maintaining a sorted data structure, where
> sorting key is "insertion order").
>
> CPython 3.6 gives **kwargs in the key specification order? That's fine
> if that's all that it says (note that it doesn't say what happens if
> someone takes **kwargs and starts to "del []" or "[] =" it). That
> allows different ways to implement it, per particular implementation's
> choice. That's even implementable in MicroPython. Like, lookups will be
> less efficient, but nobody realistically passes hundreds of kwargs.
>
> It's pretty different to go from that to dictionaries in general.
>
> Saying that a general dict always maintains insertion order is saying
> that "in Python, you have to use complex, memory hungry algorithms for
> a simple mapping type".
>

But that's an implementation detail that most folks won't care about.


>
> Saying something like "dict maintains insertion order until first
> modification" is going down the rabbit hole, making it all confusing,
> hard to remember, crazy-sounding to novices.
>
>
> Why all that, if, since the beginning of times, Python offered a
> structure with guaranteed ordering - list, and structure for efficient
> mapping one values into other - dict. Then even marriage between the
> two - OrderedDict.
>
> Why suddenly once in 25 years there's a need to do something to dict's,
> violating computer science background behind them (one of the reason
> enough people loved Python comparing to other "practical hack"
> languages)?
>

I don't understand what "computer science background" is being violated?


>
> Alternatives were already presented on the thread - if people want more
> and easier ordered dictionaries, it calls to add a special literal
> initializer for them ("o{}" was proposed).
>

I don't see that happening. If OrderedDict didn't lead to syntax then I
don't see why that's necessary now.

I think worrying about future optimizations that order preservation might
prevent is a premature optimization/red herring. None of us can predict the
future so worrying about some algorithmic breakthrough that will suddenly
materialize on how to implement dictionaries seems unnecessary. That line
of thought suggests we should guarantee as little as possible and just make
most stuff undefined behaviour.

I also think Paul S. has made it clear that MicroPython won't be
implementing order-preserving dicts due to their memory constraints. That's
fine, but I think we also need to realize that MicroPython is already a
slight hybrid by being based on Python 3.4 but with a backport of
async/await and a subset of the stdlib.

>From what I have read, this argument breaks down to this:

Standardizing order preserving and ...

   - MicroPython won't implement it, but all other major implementations
   will
   - Those that have order preservation will implement PEPs 468 and 520 for
   free
   - Some minor practical benefits in terms of avoiding accidental reliance
   on ordering

If we don't promise order preservation ...

   - CPython should probably add some perturbing to the iteration order
   - All implementations can handle this without issue or major
   implementation costs


So to me this sounds like a decision between the pragmatism of choosing
semantics that all Python implementations can handle or the developer
benefits of an order-preserving dict everywhere (all the other arguments I
think are minor compared to 

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Simon Cross]

I'm -1 on turning this on by default.

As a Python developer, I want to be aware of when deprecations are
introduced, but I don't want the users of my library or application to
care or know if I don't address those deprecation warnings for a few
months or a year. The right solution for me here seems to enable the
warnings in CI pipelines / tests.

As an end user, if I see deprecation warnings there's nothing I can
really do to make them go away straight away except run Python with
warnings turned off which seems to defeat the point of turning them on
by default. The right solution here seems to be for authors to test
their software before releasing.

I'm -2 on a complicated rule for when warnings are on because I'm
going to forget the rule a week from now and probably no one I work
with on a day to day basis will even know what the rule was to start
with.

Maybe there are ways around these things, but I'm not really seeing
what's wrong with the current situation that can't be fixed with
slightly better CI setups (which are good for other reasons too).

Schiavo
Simon
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Partial support of a platform

[Victor Stinner]

Hi,

I see more and more proposed changes to fix some parts of the code to
"partially" support a platform.

I remember that 5 years ago, the CPython code was "full" of #ifdef and
other conditional code to support various platforms, and I was happy
when we succeeded to remove support for all these old platforms like
OS/2, DOS or VMS.

The PEP 11 has a nice description to get a *full* support of a new platform:
https://www.python.org/dev/peps/pep-0011/

But the question here is more about "partial" support.

While changes are usually short, I dislike applying them to Python 2.7
and/or Python 3.6, until a platform is fully support. I prefer to
first see a platform fully supported to see how much changes are
required and to make sure that we get someone involved to maintain the
code (handle new issues).

Example of platforms: MinGW, Cygwin, OpenBSD, NetBSD, xWorks RTOS, etc.

But the way, is there an exhaustive list of platforms "officially"
supported by CPython?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Jerdonek]

On Mon, Nov 6, 2017 at 4:11 AM Nick Coghlan  wrote:

> Here's a more-complicated-than-a-doctest-for-a-dict-repo, but still
> fairly straightforward, example regarding the "insertion ordering
> dictionaries are easier to use correctly" argument:
>
> import json
> data = {"a":1, "b":2, "c":3}
> rendered = json.dumps(data)
> data2 = json.loads(rendered)
> rendered2 = json.dumps(data2)
> # JSON round trip
> assert data == data2, "JSON round trip failed"
> # Dict round trip
> assert rendered == rendered2, "dict round trip failed"
>
> Both of those assertions will always pass in CPython 3.6, as well as
> in PyPy, because their dict implementations are insertion ordered,
> which means the iteration order on the dictionaries is always "a",
> "b", "c".
>
> If you try it on 3.5 though, you should fairly consistently see that
> last assertion fail, since there's nothing in 3.5 that ensures that
> data and data2 will iterate over their keys in the same order.
>
> You can make that code implementation independent (and sufficiently
> version dependent to pass both assertions) by using OrderedDict:
>
> from collections import OrderedDict
> import json
> data = OrderedDict(a=1, b=2, c=3)
> rendered = json.dumps(data)
> data2 = json.loads(rendered, object_pairs_hook=OrderedDict)
> rendered2 = json.dumps(data2)
> # JSON round trip
> assert data == data2, "JSON round trip failed"
> # Dict round trip
> assert rendered == rendered2, "dict round trip failed"
>
> However, despite the way this code looks, the serialised key order
> *might not* be "a, b, c" on 3.5 and earlier (it will be on 3.6+, since
> that already requires that kwarg order be preserved).
>
> So the formally correct version independent code that reliably ensures
> that the key order in the JSON file is always "a, b, c" looks like
> this:
>
> from collections import OrderedDict
> import json
> data = OrderedDict((("a",1), ("b",2), ("c",3)))
> rendered = json.dumps(data)
> data2 = json.loads(rendered, object_pairs_hook=OrderedDict)
> rendered2 = json.dumps(data2)
> # JSON round trip
> assert data == data2, "JSON round trip failed"
> # Dict round trip
> assert rendered == rendered2, "dict round trip failed"
> # Key order
> assert "".join(data) == "".join(data2) == "abc", "key order failed"
>
> Getting from the "Works on CPython 3.6+ but is technically
> non-portable" state to a fully portable correct implementation that
> ensures a particular key order in the JSON file thus currently
> requires the following changes:


Nick, it seems like this is more complicated than it needs to be. You can
just pass sort_keys=True to json.dump() / json.dumps(). I use it for tests
and human-readability all the time.

—Chris


>
> - don't use a dict display, use collections.OrderedDict
> - make sure to set object_pairs_hook when using json.loads
> - don't use kwargs to OrderedDict, use a sequence of 2-tuples
>
> For 3.6, we've already said that we want the last constraint to age
> out, such that the middle version of the code also ensures a
> particular key order.
>
> The proposal is that in 3.7 we retroactively declare that the first,
> most obvious, version of this code should in fact reliably pass all
> three assertions.
>
> Failing that, the proposal is that we instead change the dict
> iteration implementation such that the dict round trip will start
> failing reasonably consistently again (the same as it did in 3.5), so
> that folks realise almost immediately that they still need
> collections.OrderedDict instead of the builtin dict.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Mon, 6 Nov 2017 14:30:45 +0100
Antoine Pitrou  wrote:

> On Tue, 7 Nov 2017 00:14:35 +1100
> Steven D'Aprano  wrote:
> > On Mon, Nov 06, 2017 at 12:27:54PM +0100, Antoine Pitrou wrote:
> >   
> > > The ordered-ness of dicts could instead become one of those stable
> > > CPython implementation details, such as the fact that resources
> > > are cleaned up timely by reference counting, that people
> > > nevertheless should not rely on if they're writing portable
> > > code.
> > 
> > Given that (according to others) none of IronPython, Jython,
> > Batavia, Nuitka, or even MicroPython, should have trouble
> > implementing an insertion-order preserving dict, and that PyPy
> > already has, why should we say it is a CPython implementation
> > detail?  
> 
> That's not what I'm taking away from Paul Sokolovsky's message I was
> responding to.  If you think otherwise then please expand and/or
> contact Paul so that things are made clearer one way or the other.

For MicroPython, it would lead to quite an overhead to make
dictionary items be in insertion order. As I mentioned, MicroPython
optimizes for very low bookkeeping memory overhead, so lookups are
effectively O(n), but orderedness will increase constant factor
significantly, perhaps 5x.

Also, arguably any algorithm which would *maintain* insertion order
over mutating operations would be more complex and/or require more
memory that one which doesn't. (This is based on the idea that
this problem is equivalent to maintaining a sorted data structure, where
sorting key is "insertion order").

CPython 3.6 gives **kwargs in the key specification order? That's fine
if that's all that it says (note that it doesn't say what happens if
someone takes **kwargs and starts to "del []" or "[] =" it). That
allows different ways to implement it, per particular implementation's
choice. That's even implementable in MicroPython. Like, lookups will be
less efficient, but nobody realistically passes hundreds of kwargs.

It's pretty different to go from that to dictionaries in general.

Saying that a general dict always maintains insertion order is saying
that "in Python, you have to use complex, memory hungry algorithms for
a simple mapping type".

Saying something like "dict maintains insertion order until first
modification" is going down the rabbit hole, making it all confusing,
hard to remember, crazy-sounding to novices. 


Why all that, if, since the beginning of times, Python offered a
structure with guaranteed ordering - list, and structure for efficient
mapping one values into other - dict. Then even marriage between the
two - OrderedDict.

Why suddenly once in 25 years there's a need to do something to dict's,
violating computer science background behind them (one of the reason
enough people loved Python comparing to other "practical hack"
languages)?

Alternatives were already presented on the thread - if people want more
and easier ordered dictionaries, it calls to add a special literal
initializer for them ("o{}" was proposed).


-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Steve Holden]

While I appreciate the value of annotations I think that *any* addition of
them to the stdlib would complicate an important learning resource
unnecessarily. S

Steve Holden

On Mon, Nov 6, 2017 at 4:07 PM, Victor Stinner 
wrote:

> Related to annotations, are you ok to annotate basic types in the
> *documentation* and/or *docstrings* of the standard library?
>
> For example, I chose to document the return type of time.time() (int)
> and time.time_ns() (float). It's short and I like how it's formatted.
> See the current rendered documentation:
>
> https://docs.python.org/dev/library/time.html#time.time
>
> "Annotations" in the documentation and docstrings have no impact on
> Python runtime performance. Annotations in docstrings makes them a few
> characters longer and so impact the memory footprint, but I consider
> that the overhead is negligible, especially when using python3 -OO.
>
> Victor
>
> 2017-11-06 17:02 GMT+01:00 Victor Stinner :
> > Hi,
> >
> > While discussions on the typing module are still hot, what do you
> > think of allowing annotations in the standard libraries, but limited
> > to a few basic types:
> >
> > * None
> > * bool, int, float, complex
> > * bytes, bytearray
> > * str
> >
> > I'm not sure about container types like tuple, list, dict, set,
> > frozenset. If we allow them, some developers may want to describe the
> > container content, like List[int] or Dict[int, str].
> >
> > My intent is to enhance the builtin documentation of functions of the
> > standard library including functions implemented in C. For example,
> > it's well known that id(obj) returns an integer. So I expect a
> > signature like:
> >
> >   id(obj) -> int
> >
> >
> > Context: Tal Einat proposed a change to convert the select module to
> > Argument Clinic:
> >
> >https://bugs.python.org/issue31938
> >https://github.com/python/cpython/pull/4265
> >
> > The docstring currently documents the return like:
> > ---
> > haypo@selma$ pydoc3 select.epoll.fileno|cat
> > Help on method_descriptor in select.epoll:
> >
> > select.epoll.fileno = fileno(...)
> > fileno() -> int
> >
> > Return the epoll control file descriptor.
> > ---
> >
> > I'm talking about "-> int", nice way to document that the function
> > returns an integer.
> >
> > Problem: even if Argument Clinic supports "return converters" like
> > "int", it doesn't generate a docstring with the return type. So I
> > created the issue:
> >
> > "Support return annotation in signature for Argument Clinic"
> > https://bugs.python.org/issue31939
> >
> > But now I am confused between docstrings, Argument Clinic, "return
> > converters", "signature" and "annotations"...
> >
> > R. David Murray reminded me the current policy:
> >
> > "the python standard library will not include type annotations, that
> > those are provided by typeshed."
> >
> > While we are discussing removing (or not) typing from the stdlib (!),
> > I propose to allow annotations in the stdlib, *but* only limited to
> > the most basic types.
> >
> > Such annotations *shouldn't* have a significant impact on performances
> > (startup time) or memory footprint.
> >
> > The expected drawback is that users can be surprised that some
> > functions get annotations, while others don't. For example,
> > os.fspath() requires a complex annotation which needs the typing
> > module and is currently done in typeshed, whereas id(obj) can get its
> > return type documented ("-> int").
> >
> > What do you think?
> >
> > Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Victor Stinner]

Related to annotations, are you ok to annotate basic types in the
*documentation* and/or *docstrings* of the standard library?

For example, I chose to document the return type of time.time() (int)
and time.time_ns() (float). It's short and I like how it's formatted.
See the current rendered documentation:

https://docs.python.org/dev/library/time.html#time.time

"Annotations" in the documentation and docstrings have no impact on
Python runtime performance. Annotations in docstrings makes them a few
characters longer and so impact the memory footprint, but I consider
that the overhead is negligible, especially when using python3 -OO.

Victor

2017-11-06 17:02 GMT+01:00 Victor Stinner :
> Hi,
>
> While discussions on the typing module are still hot, what do you
> think of allowing annotations in the standard libraries, but limited
> to a few basic types:
>
> * None
> * bool, int, float, complex
> * bytes, bytearray
> * str
>
> I'm not sure about container types like tuple, list, dict, set,
> frozenset. If we allow them, some developers may want to describe the
> container content, like List[int] or Dict[int, str].
>
> My intent is to enhance the builtin documentation of functions of the
> standard library including functions implemented in C. For example,
> it's well known that id(obj) returns an integer. So I expect a
> signature like:
>
>   id(obj) -> int
>
>
> Context: Tal Einat proposed a change to convert the select module to
> Argument Clinic:
>
>https://bugs.python.org/issue31938
>https://github.com/python/cpython/pull/4265
>
> The docstring currently documents the return like:
> ---
> haypo@selma$ pydoc3 select.epoll.fileno|cat
> Help on method_descriptor in select.epoll:
>
> select.epoll.fileno = fileno(...)
> fileno() -> int
>
> Return the epoll control file descriptor.
> ---
>
> I'm talking about "-> int", nice way to document that the function
> returns an integer.
>
> Problem: even if Argument Clinic supports "return converters" like
> "int", it doesn't generate a docstring with the return type. So I
> created the issue:
>
> "Support return annotation in signature for Argument Clinic"
> https://bugs.python.org/issue31939
>
> But now I am confused between docstrings, Argument Clinic, "return
> converters", "signature" and "annotations"...
>
> R. David Murray reminded me the current policy:
>
> "the python standard library will not include type annotations, that
> those are provided by typeshed."
>
> While we are discussing removing (or not) typing from the stdlib (!),
> I propose to allow annotations in the stdlib, *but* only limited to
> the most basic types.
>
> Such annotations *shouldn't* have a significant impact on performances
> (startup time) or memory footprint.
>
> The expected drawback is that users can be surprised that some
> functions get annotations, while others don't. For example,
> os.fspath() requires a complex annotation which needs the typing
> module and is currently done in typeshed, whereas id(obj) can get its
> return type documented ("-> int").
>
> What do you think?
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Allow annotations using basic types in the stdlib?

[Victor Stinner]

Hi,

While discussions on the typing module are still hot, what do you
think of allowing annotations in the standard libraries, but limited
to a few basic types:

* None
* bool, int, float, complex
* bytes, bytearray
* str

I'm not sure about container types like tuple, list, dict, set,
frozenset. If we allow them, some developers may want to describe the
container content, like List[int] or Dict[int, str].

My intent is to enhance the builtin documentation of functions of the
standard library including functions implemented in C. For example,
it's well known that id(obj) returns an integer. So I expect a
signature like:

  id(obj) -> int


Context: Tal Einat proposed a change to convert the select module to
Argument Clinic:

   https://bugs.python.org/issue31938
   https://github.com/python/cpython/pull/4265

The docstring currently documents the return like:
---
haypo@selma$ pydoc3 select.epoll.fileno|cat
Help on method_descriptor in select.epoll:

select.epoll.fileno = fileno(...)
fileno() -> int

Return the epoll control file descriptor.
---

I'm talking about "-> int", nice way to document that the function
returns an integer.

Problem: even if Argument Clinic supports "return converters" like
"int", it doesn't generate a docstring with the return type. So I
created the issue:

"Support return annotation in signature for Argument Clinic"
https://bugs.python.org/issue31939

But now I am confused between docstrings, Argument Clinic, "return
converters", "signature" and "annotations"...

R. David Murray reminded me the current policy:

"the python standard library will not include type annotations, that
those are provided by typeshed."

While we are discussing removing (or not) typing from the stdlib (!),
I propose to allow annotations in the stdlib, *but* only limited to
the most basic types.

Such annotations *shouldn't* have a significant impact on performances
(startup time) or memory footprint.

The expected drawback is that users can be surprised that some
functions get annotations, while others don't. For example,
os.fspath() requires a complex annotation which needs the typing
module and is currently done in typeshed, whereas id(obj) can get its
return type documented ("-> int").

What do you think?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Paul Moore]

On 6 November 2017 at 14:16, Gustavo Carneiro  wrote:
> But eventually, the warnings in 3rd-party Python modules gradually increased
> because, since warnings are disabled by default, authors of command-line
> tools, or even python modules,  don't even realise they are triggering the
> warning.
>
> So developers stop fixing warnings because they are hidden.  Things get
> worse and worse over the years.  Eventually I got fed up and removed the
> PYTHONWARNINGS env var.

Maybe it's worth running the test suites of a number of major packages
like pip, requests, django, ... with warnings switched on, to see what
the likely impact of making warnings display by default would be on
those projects? Hopefully, it's zero, but hard data is always better
than speculation :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Philipp A.]

Hi! Just this minute I ran across a case where I’d want DeprecationWarnings
on by default

(We want to rename a property in an API I’m co-developing. I has mainly
scientists as target audience, so end users, not developers)

Gustavo Carneiro  schrieb am Mo., 6. Nov. 2017 um
15:19 Uhr:

> Big +1 to turning warnings on by default again.
>
> When this behaviour first started, first I was surprised, then annoyed
> that warnings were being suppressed.  For a few years I learned to have
> `export PYTHONWARNINGS=default` in my .bashrc.
>
> But eventually, the warnings in 3rd-party Python modules gradually
> increased because, since warnings are disabled by default, authors of
> command-line tools, or even python modules,  don't even realise they are
> triggering the warning.
>
> So developers stop fixing warnings because they are hidden.  Things get
> worse and worse over the years.  Eventually I got fed up and removed the
> PYTHONWARNINGS env var.
>
> Showing warnings by default is good:
>  1. End users who don't understand what those warnings are are unlikely to
> see them since they don't command-line tools at all;
>  2. The users that do see them are sufficiently proficient to be able to
> submit a bug report;
>  3. If you file a bug report in tool that uses a 3rd party module, the
> author of that tool should open a corresponding bug report on the 3rd party
> module that actually caused the warning;
>  4. Given time, all the bug reports trickle down and create pressure on
> module maintainers to fix warnings;
>  5. If a module is being used and has no maintainer, it's a good
> indication it is time to fork it or find an alternative.
>
> Not fixing warnings is a form of technical debt that we should not
> encourage.  It is not the Python way.
>
>
> On 6 November 2017 at 02:05, Nick Coghlan  wrote:
>
>> On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
>> mistakenly though the async/await deprecation warnings were missing
>> from 3.6.
>>
>> They weren't missing, we'd just both forgotten those warnings were off
>> by default (7 years after the change to the default settings in 2.7 &
>> 3.2).
>>
>> So my proposal is simple (and not really new): let's revert back to
>> the way things were in 2.6 and earlier, with DeprecationWarning being
>> visible by default, and app devs having to silence it explicitly
>> during application startup (before they start importing third party
>> modules) if they don't want their users seeing it when running on the
>> latest Python version (e.g. this would be suitable for open source
>> apps that get integrated into Linux distros and use the system Python
>> there).
>>
>> This will also restore the previously clear semantic and behavioural
>> different between PendingDeprecationWarning (hidden by default) and
>> DeprecationWarning (visible by default).
>>
>> As part of this though, I'd suggest amending the documentation for
>> DeprecationWarning [1] to specifically cover how to turn it off
>> programmatically (`warnings.simplefilter("ignore",
>> DeprecationWarning)`), at the command line (`python -W
>> ignore::DeprecationWarning ...`), and via the environment
>> (`PYTHONWARNINGS=ignore::DeprecationWarning`).
>>
>> (Structurally, I'd probably put that at the end of the warnings
>> listing as a short introduction to warnings management, with links out
>> to the relevant sections of the documentation, and just use
>> DeprecationWarning as the specific example)
>>
>> Cheers,
>> Nick.
>>
>> [1] https://docs.python.org/3/library/exceptions.html#DeprecationWarning
>>
>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>>
>
>
>
> --
> Gustavo J. A. M. Carneiro
> Gambit Research
> "The universe is always one step beyond logic." -- Frank Herbert
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Gustavo Carneiro]

Big +1 to turning warnings on by default again.

When this behaviour first started, first I was surprised, then annoyed that
warnings were being suppressed.  For a few years I learned to have `export
PYTHONWARNINGS=default` in my .bashrc.

But eventually, the warnings in 3rd-party Python modules gradually
increased because, since warnings are disabled by default, authors of
command-line tools, or even python modules,  don't even realise they are
triggering the warning.

So developers stop fixing warnings because they are hidden.  Things get
worse and worse over the years.  Eventually I got fed up and removed the
PYTHONWARNINGS env var.

Showing warnings by default is good:
 1. End users who don't understand what those warnings are are unlikely to
see them since they don't command-line tools at all;
 2. The users that do see them are sufficiently proficient to be able to
submit a bug report;
 3. If you file a bug report in tool that uses a 3rd party module, the
author of that tool should open a corresponding bug report on the 3rd party
module that actually caused the warning;
 4. Given time, all the bug reports trickle down and create pressure on
module maintainers to fix warnings;
 5. If a module is being used and has no maintainer, it's a good indication
it is time to fork it or find an alternative.

Not fixing warnings is a form of technical debt that we should not
encourage.  It is not the Python way.


On 6 November 2017 at 02:05, Nick Coghlan  wrote:

> On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
> mistakenly though the async/await deprecation warnings were missing
> from 3.6.
>
> They weren't missing, we'd just both forgotten those warnings were off
> by default (7 years after the change to the default settings in 2.7 &
> 3.2).
>
> So my proposal is simple (and not really new): let's revert back to
> the way things were in 2.6 and earlier, with DeprecationWarning being
> visible by default, and app devs having to silence it explicitly
> during application startup (before they start importing third party
> modules) if they don't want their users seeing it when running on the
> latest Python version (e.g. this would be suitable for open source
> apps that get integrated into Linux distros and use the system Python
> there).
>
> This will also restore the previously clear semantic and behavioural
> different between PendingDeprecationWarning (hidden by default) and
> DeprecationWarning (visible by default).
>
> As part of this though, I'd suggest amending the documentation for
> DeprecationWarning [1] to specifically cover how to turn it off
> programmatically (`warnings.simplefilter("ignore",
> DeprecationWarning)`), at the command line (`python -W
> ignore::DeprecationWarning ...`), and via the environment
> (`PYTHONWARNINGS=ignore::DeprecationWarning`).
>
> (Structurally, I'd probably put that at the end of the warnings
> listing as a short introduction to warnings management, with links out
> to the relevant sections of the documentation, and just use
> DeprecationWarning as the specific example)
>
> Cheers,
> Nick.
>
> [1] https://docs.python.org/3/library/exceptions.html#DeprecationWarning
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> gjcarneiro%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Wolfgang]

Hi,

Could be missed by me but the
guarantee that dict literals are ordered implies not
that dict must be ordered in all cases.

The dict literal:

d = {"a": 1, "b": 2}

will keep the order of "a" and "b" because it is specified
as a dict literal.

But

d["c"] = 3

can change this order and it is allowed by the specification
of guaranteed ordered dict literals.

Please correct me if I am wrong.

In Python 3.6 it does not because dict
is implemented ordered and the insertion order is preserved.

Also I think we should give the whole thing more time and wait
with this guarantee. There are valid concerns against this.

Personally I like the ordering but if I need an ordered dict
it is ok for me to write this explicitly.

The **kwargs are already guaranteed to be ordered and this was
useful because the OrderedDict constructor benefits and for
other places it is useful.
But making all dict's per default ordered is another level.


Regards,

Wolfgang
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Antoine Pitrou]

On Mon, 6 Nov 2017 23:23:25 +1000
Nick Coghlan  wrote:
> On 6 November 2017 at 21:58, Antoine Pitrou  wrote:
> > I guess my takeaway point is that many situations are complicated, and
> > many third-party library developers are much less disciplined than what
> > some of us would idealistically expect them to be (those developers
> > probably often have good reasons for that).  For someone who takes care
> > to only use selected third-party libraries of high maintenance quality,
> > I'm very +1 on your proposal.  For the more murky (but rather common)
> > cases of relying on average quality third-party libraries, I'm +0.  
> 
> Agreed, and I'm thinking there could be a lot of value in the variant
> of the idea that says:
> 
> - tweak the default warning filters to turn DeprecationWarning back on
> for __main__ only

Thats sounds error-prone.  I'd rather have them on by default
everywhere.

> - add a new warnings module API specifically for managing deprecation warnings

+1

And I think we need to handle two different use cases:

- silencing warnings *emitted by* a certain module (e.g. a widely-used
  module which recently introduced major API changes)

- silencing warnings *reported in* a certain module (e.g. a
  sporadically-maintained library whose usage frequently emits
  deprecation warnings coming from other libraries)

Ideally, we also need a CLI switch (or environment variable) to override
these settings, so that one can run in "dev mode" and see all
problematic usage accross their library, application and third-party
dependencies.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5.4 doesn't appear in changelog

[Ned Deily]

On Nov 6, 2017, at 13:07, Antoine Pitrou  wrote:
> Is there a known reason why 3.5.4 doesn't appear in the changelogs at
> the bottom of https://docs.python.org/3.7/whatsnew/index.html ?
> 
> (all releases until 3.5.3 do)

As things stand now, the changelogs from maintenance releases have to be 
manually merged into feature release trees (e.g. master).  That's something 
I've done close to final feature release.  With the switch to blurb, it would 
be nice to better automate the sharing of changelogs between branches but I 
don't think anyone has tried to tackle that yet.  Patches welome.

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Antoine Pitrou]

On Tue, 7 Nov 2017 00:14:35 +1100
Steven D'Aprano  wrote:
> On Mon, Nov 06, 2017 at 12:27:54PM +0100, Antoine Pitrou wrote:
> 
> > The ordered-ness of dicts could instead become one of those stable
> > CPython implementation details, such as the fact that resources are
> > cleaned up timely by reference counting, that people nevertheless
> > should not rely on if they're writing portable code.  
> 
> Given that (according to others) none of IronPython, Jython, Batavia, 
> Nuitka, or even MicroPython, should have trouble implementing an 
> insertion-order preserving dict, and that PyPy already has, why should 
> we say it is a CPython implementation detail?

That's not what I'm taking away from Paul Sokolovsky's message I was
responding to.  If you think otherwise then please expand and/or contact
Paul so that things are made clearer one way or the other.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Stefan Krah]

On Mon, Nov 06, 2017 at 01:03:05PM +0200, Paul Sokolovsky wrote:
> > $ time ./micropython xxx.py 
> > $ time ../../cpython/python xxx.py 
> 
> > 
> > Congratulations ...
> 
> That's why I wrote "Python 3.3", it's hard to argue CPython doesn't do
> anything about the "Python is slow" proverb. It's still shouldn't be
> hard to construct a testcase where MicroPython is faster (by not
> implementing features not needed by 90% of Python programs of course,
> not "for free"). 

Sorry, that was a slightly mischievous benchmark indeed. -- Whether the proposal
is surreal or not depends on the likelihood that a) a substantially faster dict
algorithm will emerge and b) CPython, PyPy and Jython will switch to it.


My proposal was based on the fact that for almost two release cycles the
ordering implementation detail hasn't changed.




Stefan Krah




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 6 November 2017 at 21:58, Antoine Pitrou  wrote:
> I guess my takeaway point is that many situations are complicated, and
> many third-party library developers are much less disciplined than what
> some of us would idealistically expect them to be (those developers
> probably often have good reasons for that).  For someone who takes care
> to only use selected third-party libraries of high maintenance quality,
> I'm very +1 on your proposal.  For the more murky (but rather common)
> cases of relying on average quality third-party libraries, I'm +0.

Agreed, and I'm thinking there could be a lot of value in the variant
of the idea that says:

- tweak the default warning filters to turn DeprecationWarning back on
for __main__ only
- add a new warnings module API specifically for managing deprecation warnings

The first change would restore DeprecationWarning-by-default for:

- ad hoc single file scripts (potentially including Jupyter notebooks,
depending on which execution namespace kernels use)
- ad hoc experimentation at the REPL
- working through outdated examples at the REPL

For installed applications using setuptools (or similar), "__main__"
is the script wrapper, not any of the application code, and those have
been getting more minimal over time (and when they do have non-trivial
code in them, it's calling into setuptools/pkg_resources rather than
the standard library).

The second change would be designed around making it easier for app
developers to say "Always emit DeprecationWarnings for my own code,
don't worry about anything else".

With DeprecationWarning still off by default, that might look like:

warnings.reportdeprecations("myproject")

Cheers,
Nick.

P.S. For those interested, the issue where we switched to the current
behaviour is https://bugs.python.org/issue7319

And the related stdlib-sig thread is
https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html

That was apparently in the long gone era when I still read every
python-checkins message, so there's also a very short thread on
python-dev after it landed in SVN:
https://mail.python.org/pipermail/python-dev/2010-January/097178.html

The primary argument for the change in the stlib-sig thread is
definitely "App devs don't hide deprecation warnings, and then their
users complain about seeing them". Guido even goes so far to describe
app developers using the warnings filter interface as designed to
manage what they emit on stderr as "a special hack".

Later in the thread Georg Brandl brought up a fairly compelling
argument that Guido was right about that, which is that programmatic
filters to manage warnings currently don't compose well with the
command line and environment variable settings, since there's only one
list of warning filters, which means there's no way to say "put this
*before* the default filters, but *after* any filters that were
specified explicitly with -W or PYTHONWARNINGS". Instead, your options
are limited to prepending (the default behaviour) which overrides both
the defaults and any specific settings, or appending, which means you
can't even override the defaults.

When DeprecationWarnings were enabled by default, this meant there was
no native way to override application level filters that ignored them
in order to turn on DeprecationWarnings when running your test suite.

By contrast, having them be off by default with runtime programmatic
filter manipulation being rare offers a simple way to turn them on
globally, via "-Wd". Adding "-W once::DeprecationWarning:__main__" to
the default filter list doesn't change that substantially.

That said, this does make me wonder whether the warnings module should
either place a sentinel marker in its warning filter list to mark
where the default filters start (and adjust the append mode to insert
filters there), or else provide a new option for programmatic
configuration that's "higher priority than the defaults, lower
priority than the explicit configuration settings".

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 12:27:54PM +0100, Antoine Pitrou wrote:

> The ordered-ness of dicts could instead become one of those stable
> CPython implementation details, such as the fact that resources are
> cleaned up timely by reference counting, that people nevertheless
> should not rely on if they're writing portable code.

Given that (according to others) none of IronPython, Jython, Batavia, 
Nuitka, or even MicroPython, should have trouble implementing an 
insertion-order preserving dict, and that PyPy already has, why should 
we say it is a CPython implementation detail?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 3.5.4 doesn't appear in changelog

[Antoine Pitrou]

Hello,

Is there a known reason why 3.5.4 doesn't appear in the changelogs at
the bottom of https://docs.python.org/3.7/whatsnew/index.html ?

(all releases until 3.5.3 do)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 6 November 2017 at 21:18, Steve Holden  wrote:
> I have to agree: I find the elevation of a CPython implementation detail to
> a language feature somewhat hard to comprehend. Maybe it's more to do with
> the way it's been presented, but this is hardly an enhancement the language
> has been screaming for for years.
>
> Presumably there is little concern that algorithms that rely on this
> behaviour will be perfectly syntactically conformant with earlier versions
> but will fail subtly and without explanation? It's a small concern, but a
> real one - particularly for learners.

A similar concern existed when we elevated sort stability to being a
language requirement - if you relied on that guarantee, your code was
technically buggy on versions prior to 2.3, but eventually 2.2 and
earlier aged out of general use, allowing such code to become correct
in general.

So the current discussion is mainly about deciding where we want the
compatibility burden to fall in relation to dict insertion ordering:

1. Do we deliberately revert CPython back to being harder to use
correctly for the sake of making Python easier to implement?
2. Do we make Python harder to implement for the sake of making it
easier to use?
3. Do we choose not to choose, thus implicitly choosing "2" by default
due to the fact that Python is defined by a language spec and a
reference implementation, rather than *just* a language spec?

Here's a more-complicated-than-a-doctest-for-a-dict-repo, but still
fairly straightforward, example regarding the "insertion ordering
dictionaries are easier to use correctly" argument:

import json
data = {"a":1, "b":2, "c":3}
rendered = json.dumps(data)
data2 = json.loads(rendered)
rendered2 = json.dumps(data2)
# JSON round trip
assert data == data2, "JSON round trip failed"
# Dict round trip
assert rendered == rendered2, "dict round trip failed"

Both of those assertions will always pass in CPython 3.6, as well as
in PyPy, because their dict implementations are insertion ordered,
which means the iteration order on the dictionaries is always "a",
"b", "c".

If you try it on 3.5 though, you should fairly consistently see that
last assertion fail, since there's nothing in 3.5 that ensures that
data and data2 will iterate over their keys in the same order.

You can make that code implementation independent (and sufficiently
version dependent to pass both assertions) by using OrderedDict:

from collections import OrderedDict
import json
data = OrderedDict(a=1, b=2, c=3)
rendered = json.dumps(data)
data2 = json.loads(rendered, object_pairs_hook=OrderedDict)
rendered2 = json.dumps(data2)
# JSON round trip
assert data == data2, "JSON round trip failed"
# Dict round trip
assert rendered == rendered2, "dict round trip failed"

However, despite the way this code looks, the serialised key order
*might not* be "a, b, c" on 3.5 and earlier (it will be on 3.6+, since
that already requires that kwarg order be preserved).

So the formally correct version independent code that reliably ensures
that the key order in the JSON file is always "a, b, c" looks like
this:

from collections import OrderedDict
import json
data = OrderedDict((("a",1), ("b",2), ("c",3)))
rendered = json.dumps(data)
data2 = json.loads(rendered, object_pairs_hook=OrderedDict)
rendered2 = json.dumps(data2)
# JSON round trip
assert data == data2, "JSON round trip failed"
# Dict round trip
assert rendered == rendered2, "dict round trip failed"
# Key order
assert "".join(data) == "".join(data2) == "abc", "key order failed"

Getting from the "Works on CPython 3.6+ but is technically
non-portable" state to a fully portable correct implementation that
ensures a particular key order in the JSON file thus currently
requires the following changes:

- don't use a dict display, use collections.OrderedDict
- make sure to set object_pairs_hook when using json.loads
- don't use kwargs to OrderedDict, use a sequence of 2-tuples

For 3.6, we've already said that we want the last constraint to age
out, such that the middle version of the code also ensures a
particular key order.

The proposal is that in 3.7 we retroactively declare that the first,
most obvious, version of this code should in fact reliably pass all
three assertions.

Failing that, the proposal is that we instead change the dict
iteration implementation such that the dict round trip will start
failing reasonably consistently again (the same as it did in 3.5), so
that folks realise almost immediately that they still need
collections.OrderedDict instead of the builtin dict.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Antoine Pitrou]

On Mon, 6 Nov 2017 12:45:27 +0100
Antoine Pitrou  wrote:
> 
> I'm on the fence on this.
> 
> I was part of the minority who opposed the original decision.  So I
> really appreciate your sentiment.  Since then, I had to deal with a lot
> of very diverse third-party libraries, and I learned that:
> 
> - most third-party libraries don't ever emit PendingDeprecationWarning;
>   they only emit DeprecationWarning. So all their warnings would now be
>   visible by default. (1)
> 
> - release cycles are much shorter on third-party libraries, so it's
>   easier not to notice that one of your dependencies has started
>   changing some of its APIs - maybe you'll notice in 3 months.  Also,
>   perhaps you need a compatibility fallback anyway instead of
>   unconditionally switching to the new version of the API, which adds
>   to the maintenance cost.

Of course, there's also the case where it's own of your dependency's
dependencies, or your dependency's dependency's dependencies, or your
dependency's dependency's dependency's dependencies, that has started
to emit such warning because of how your depencency, or your
dependency's dependency, or your dependency's dependency's dependency,
calls into that library.

Now if you have several such dependencies (or dependencies'
dependencies, etc.), it becomes both likely and annoying to solve /
workaround.


I guess my takeaway point is that many situations are complicated, and
many third-party library developers are much less disciplined than what
some of us would idealistically expect them to be (those developers
probably often have good reasons for that).  For someone who takes care
to only use selected third-party libraries of high maintenance quality,
I'm very +1 on your proposal.  For the more murky (but rather common)
cases of relying on average quality third-party libraries, I'm +0.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Antoine Pitrou]

On Mon, 6 Nov 2017 12:05:07 +1000
Nick Coghlan  wrote:
> 
> So my proposal is simple (and not really new): let's revert back to
> the way things were in 2.6 and earlier, with DeprecationWarning being
> visible by default, and app devs having to silence it explicitly
> during application startup (before they start importing third party
> modules) if they don't want their users seeing it when running on the
> latest Python version (e.g. this would be suitable for open source
> apps that get integrated into Linux distros and use the system Python
> there).
> 
> This will also restore the previously clear semantic and behavioural
> different between PendingDeprecationWarning (hidden by default) and
> DeprecationWarning (visible by default).

I'm on the fence on this.

I was part of the minority who opposed the original decision.  So I
really appreciate your sentiment.  Since then, I had to deal with a lot
of very diverse third-party libraries, and I learned that:

- most third-party libraries don't ever emit PendingDeprecationWarning;
  they only emit DeprecationWarning. So all their warnings would now be
  visible by default. (1)

- release cycles are much shorter on third-party libraries, so it's
  easier not to notice that one of your dependencies has started
  changing some of its APIs - maybe you'll notice in 3 months.  Also,
  perhaps you need a compatibility fallback anyway instead of
  unconditionally switching to the new version of the API, which adds
  to the maintenance cost.

- depending on not-well-maintained third-party libraries is a fact of
  life; these libraries may induce a lot of DeprecationWarnings on
  their dependencies, and still work fine until some maintainer
  comes out from the grave (or steps briefly into it before
  returning to their normal non-programming life) to apply a proper fix
  and make a release.


The one part where I think your proposal is good (apart from making
things a bit simpler for developers) is that I also noticed some
authors of third-party libraries don't notice until late that their
code emits DeprecationWarnings in dependencies.  By knowing earlier
(and having their users affected) they may be enticed to fix those
issues earlier.  But that's only true for third-party libraries with an
active enough maintainer, and a tight enough release schedule.


As for why (1) happens, I think it's partly because changing from one
warning to another is cumbersome; partly because many libraries don't
want to be constrained by a long deprecation cycle.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Antoine Pitrou]

I think that Paul has a point.  Interestingly, at the same time we're
talking about guaranteeing the order of dicts, we're talking about
using another, unordered, data structure (hash array mapped tries) to
improve the performance of something that resembles a namespace.  It
seems the "unordered" part will be visible through
ExecutionContext.vars().

https://www.python.org/dev/peps/pep-0550/#enumerating-context-vars

The ordered-ness of dicts could instead become one of those stable
CPython implementation details, such as the fact that resources are
cleaned up timely by reference counting, that people nevertheless
should not rely on if they're writing portable code.

Regards

Antoine.


On Mon, 6 Nov 2017 12:18:17 +0200
Paul Sokolovsky  wrote:
> []
> 
> > I don't think that situation should change the decision,   
> 
> Indeed, it shouldn't. What may change it is the simple and obvious fact
> that there's no need to change anything, as proven by the 25-year
> history of the language.
> 
> What happens now borders on technologic surrealism - the CPython, after
> many years of persuasion, switched its dict algorithm, rather
> inefficient in terms of memory, to something else, less inefficient
> (still quite inefficient, taking "no overhead" as the baseline). That
> algorithm randomly had another property. Now there's a seemingly
> serious talk of letting that property leak into the *language spec*,
> despite the fact that there can be unlimited number of dictionary
> algorithms, most of them not having that property. 


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steve Holden]

On Mon, Nov 6, 2017 at 10:18 AM, Paul Sokolovsky  wrote:

> Hello,
>
> What happens now borders on technologic surrealism - the CPython, after
> many years of persuasion, switched its dict algorithm, rather
> inefficient in terms of memory, to something else, less inefficient
> (still quite inefficient, taking "no overhead" as the baseline). That
> algorithm randomly had another property. Now there's a seemingly
> serious talk of letting that property leak into the *language spec*,
> despite the fact that there can be unlimited number of dictionary
> algorithms, most of them not having that property.
>
> ​
I have to agree: I find the elevation of a CPython implementation detail to
a language feature ​somewhat hard to comprehend. Maybe it's more to do with
the way it's been presented, but this is hardly an enhancement the language
has been screaming for for years.

​Presumably there is little concern that algorithms that rely on this
behaviour will be perfectly syntactically conformant with earlier versions​
but will fail subtly and without explanation? It's a small concern, but a
real one - particularly for learners.

regards
 Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Antoine Pitrou]

On Sun, 5 Nov 2017 20:18:07 -0800
Lukasz Langa  wrote:
> 
> Interestingly enough, at Facebook we found out that using f-strings is 
> *faster* at runtime than the lazy form of logging.log("format with %s and 
> %d", arg1, arg2), including for cases when the log message is not emitted.

I suspect this depends on how complex your f-strings are (or the
interpolated data).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Mon, 6 Nov 2017 11:36:59 +0100
Stefan Krah  wrote:

> On Mon, Nov 06, 2017 at 12:18:17PM +0200, Paul Sokolovsky wrote:
> > MicroPython hashmap implementation is effectively O(n) (average and
> > worst case) due to the algorithm parameters chosen (like the load
> > factor of 1). Of course, parameters could be tweaked, but the ones
> > chosen are so because the memory usage is far more important for
> > MicroPython systems than performance characteristics (all due to
> > small amounts of memory). Like, MicroPython was twice as fast than
> > Python 3.3 on average, and 1000 times more efficient in the memory
> > usage.  
> 

[]

> 
> $ time ./micropython xxx.py 
> $ time ../../cpython/python xxx.py 

> 
> Congratulations ...

That's why I wrote "Python 3.3", it's hard to argue CPython doesn't do
anything about the "Python is slow" proverb. It's still shouldn't be
hard to construct a testcase where MicroPython is faster (by not
implementing features not needed by 90% of Python programs of course,
not "for free"). 

Anyway, where're memory measurements?

(This is offtopic, so I shouldn't have replied.)

> Stefan Krah

-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Angelico]

On Mon, Nov 6, 2017 at 9:36 PM, Stefan Krah  wrote:
> On Mon, Nov 06, 2017 at 12:18:17PM +0200, Paul Sokolovsky wrote:
>> MicroPython hashmap implementation is effectively O(n) (average and
>> worst case) due to the algorithm parameters chosen (like the load factor
>> of 1). Of course, parameters could be tweaked, but the ones chosen are
>> so because the memory usage is far more important for MicroPython
>> systems than performance characteristics (all due to small amounts of
>> memory). Like, MicroPython was twice as fast than Python 3.3 on
>> average, and 1000 times more efficient in the memory usage.
>
> $ cat xxx.py
>
> def pi_float():
> """native float"""
> lasts, t, s, n, na, d, da = 0, 3.0, 3, 1, 0, 0, 24
> while s != lasts:
> lasts = s
> n, na = n+na, na+8
> d, da = d+da, da+32
> t = (t * n) / d
> s += t
> return s
>
> for i in range(10):
> x = pi_float()
>
> $ time ./micropython xxx.py
>
> real0m4.424s
> user0m4.406s
> sys 0m0.016s
> $
> $ time ../../cpython/python xxx.py
>
> real0m1.066s
> user0m1.056s
> sys 0m0.010s
>
>
>
> Congratulations ...

Maybe I'm misreading your demo, but I fail to see what this has to do
with dict performance?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Stefan Krah]

On Mon, Nov 06, 2017 at 12:18:17PM +0200, Paul Sokolovsky wrote:
> MicroPython hashmap implementation is effectively O(n) (average and
> worst case) due to the algorithm parameters chosen (like the load factor
> of 1). Of course, parameters could be tweaked, but the ones chosen are
> so because the memory usage is far more important for MicroPython
> systems than performance characteristics (all due to small amounts of
> memory). Like, MicroPython was twice as fast than Python 3.3 on
> average, and 1000 times more efficient in the memory usage.

$ cat xxx.py 

def pi_float():
"""native float"""
lasts, t, s, n, na, d, da = 0, 3.0, 3, 1, 0, 0, 24
while s != lasts:
lasts = s
n, na = n+na, na+8
d, da = d+da, da+32
t = (t * n) / d
s += t
return s

for i in range(10):
x = pi_float()

$ time ./micropython xxx.py 

real0m4.424s
user0m4.406s
sys 0m0.016s
$ 
$ time ../../cpython/python xxx.py 

real0m1.066s
user0m1.056s
sys 0m0.010s



Congratulations ...



Stefan Krah




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 6 November 2017 at 20:21, Paul Moore  wrote:
> On 6 November 2017 at 03:38, Nick Coghlan  wrote:
>> - if we ever write "import foo" ourselves, then we're a Python
>> developer, and it's our responsibility to work out how to manage
>> DeprecationWarning when it gets raised by either our own code, or the
>> libraries and frameworks that we use
>
> As someone who was bitten by this when deprecation warnings were
> displayed by default, what's the process for suppressing deprecation
> warnings in modules that I import (and hence have no control over)
> *without* also suppressing them for my code (where I do want to fix
> them, so that my users don't have a problem)?
>
> That's the complicated bit that needs to be in the docs - more so than
> a simple pointer to how to suppress the warning altogether.

For "top level" deprecation warnings in the libraries you use (i.e.
those where the specific API you're calling from your code is either
the one that calls warnings.warn, or else it adjusts the stack level
argument so it acts that way), the warnings configuration looks like:

warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("once", category=DeprecationWarning,
module="myproject.*")
warnings.filterwarnings("once", category=DeprecationWarning,
module="__main__")

So that could stand to be made cleaner in a few specific ways:

1. Provide a dedicated API for configuring the deprecation warnings filtering
2. When given a module name, also enable warnings for submodules of that module

Given those design guidelines, an improvement may look like:

warnings.ignoredeprecations(except_for=["myproject","__main__"])

A middle ground between the status quo and full re-enablement of
deprecation warnings would also be to add the following to the default
filter set used when neither -W nor PYTHONWARNINGS is set:

warnings.filterwarnings("once", category=DeprecationWarning,
module="__main__")

That way, warnings would be emitted by default for the REPL and
top-level scripts, but getting them for imported libraries would
continue to be opt-in.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Paul Moore]

On 6 November 2017 at 03:38, Nick Coghlan  wrote:
> - if we ever write "import foo" ourselves, then we're a Python
> developer, and it's our responsibility to work out how to manage
> DeprecationWarning when it gets raised by either our own code, or the
> libraries and frameworks that we use

As someone who was bitten by this when deprecation warnings were
displayed by default, what's the process for suppressing deprecation
warnings in modules that I import (and hence have no control over)
*without* also suppressing them for my code (where I do want to fix
them, so that my users don't have a problem)?

That's the complicated bit that needs to be in the docs - more so than
a simple pointer to how to suppress the warning altogether.

On 6 November 2017 at 06:38, Nick Coghlan  wrote:
> Put "PYTHONWARNINGS=ignore::DeprecationWarning" before whatever
> command is giving them the warnings.
>
> Even on Windows, you can put that in a batch file with the actual
> command you want to run and silence the warnings that way.

Batch files do not behave the same in Windows as standard executables.
Having to wrap a "normal application" (for example, a script wrapper
installed via "pip install package" in a bat file is (a) messy for
inexperienced users, and (b) likely to cause weird errors (for example
nesting bat files is broken, so you can't use a "wrapped" command
transparently in another bat file without silent errors).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 6 November 2017 at 19:14, Steven D'Aprano  wrote:
> On Mon, Nov 06, 2017 at 01:07:51PM +1000, Nick Coghlan wrote:
>> When we did the "insertion ordered hash map" availability review, the
>> main implementations we were checking on behalf of were Jython & VOC
>> (JVM implementations), Batavia (JavaScript implementation), and
>> MicroPython (C implementation). Adding IronPython (C# implementation)
>> to the mix gives:
>
> Shouldn't we check with Nuitka (C++) and Cython as well?

If you're using dicts as dicts, Cython just uses the CPython ones via
the C API, rather than defining its own.

I didn't do any specific research for C++ (since it's essentially the
king of fast low level data structure design, hence its popularity in
graphics programming), but did see a few references to C++ ordered
hash map implementations while looking into the available options for
other runtimes.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Sun, 5 Nov 2017 12:04:41 +1000
Nick Coghlan  wrote:

> On 5 November 2017 at 04:35, Guido van Rossum 
> wrote:
> > This sounds reasonable -- I think when we introduced this in 3.6 we
> > were worried that other implementations (e.g. Jython) would have a
> > problem with this, but AFAIK they've reported back that they can do
> > this just fine. So let's just document this as a language
> > guarantee.  
> 
> When I asked Damien George about this for MicroPython, he indicated
> that they'd have to choose between guaranteed order and O(1) lookups
> given their current dict implementation. That surprised me a bit

MicroPython hashmap implementation is effectively O(n) (average and
worst case) due to the algorithm parameters chosen (like the load factor
of 1). Of course, parameters could be tweaked, but the ones chosen are
so because the memory usage is far more important for MicroPython
systems than performance characteristics (all due to small amounts of
memory). Like, MicroPython was twice as fast than Python 3.3 on
average, and 1000 times more efficient in the memory usage.

> (since PyPy and CPython both *saved* memory by switching to their
> guaranteed order implementations, hence the name "compact dict

There's nothing to save in MicroPython's dict implementation, simply
because it doesn't waste anything in the first place - uPy's dict entry
is just (key, val) (2 machine words), and load factor of the dict can
reach 1.0 as mentioned.

[]

> I don't think that situation should change the decision, 

Indeed, it shouldn't. What may change it is the simple and obvious fact
that there's no need to change anything, as proven by the 25-year
history of the language.

What happens now borders on technologic surrealism - the CPython, after
many years of persuasion, switched its dict algorithm, rather
inefficient in terms of memory, to something else, less inefficient
(still quite inefficient, taking "no overhead" as the baseline). That
algorithm randomly had another property. Now there's a seemingly
serious talk of letting that property leak into the *language spec*,
despite the fact that there can be unlimited number of dictionary
algorithms, most of them not having that property. 

What it will lead to is further fragmentation of the community. Python2
vs Python3 split is far from being over, and now there're splits
between:

* people who use "yield from" vs "await"
* people who use f-strings vs who don't
* people who rely on sorted nature of dict's vs who don't

etc.

[]

> P.S. If anyone does want to explore MicroPython's dict implementation,
> and see if there might be an alternate implementation strategy that
> offers both O(1) lookup and guaranteed ordering without using
> additional memory

That would be the first programmer in the history to have a cake and
eat it too. Memory efficiency, runtime efficiency, sorted order: choose
2 of 3.


-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Paul Moore]

On 6 November 2017 at 03:41, Lukasz Langa  wrote:
>
>> On 4 Nov, 2017, at 3:39 AM, Paul Moore  wrote:
>>
>> Lukasz Langa said:
>>> So, the difference is in perceived usability. It's psychological.
>>
>> Please, let's not start the "not in the stdlib isn't an issue" debate
>> again. If I concede it's a psychological issue, will you concede that
>> the fact that it's psychological doesn't mean that it's not a real,
>> difficult to solve, problem for some people? I'm also willing to
>> concede that it's a *minority* problem, if that helps. But can we stop
>> dismissing it as a non-existent problem?
>
> Paul, if you read the words I wrote in my e-mail verbatim, you will note that 
> I am not saying it's not real or it's not important. Quite the opposite. Can 
> you elaborate what made you think that my assertion that the issue is 
> psychological made you think I'm being dismissive? To me it looks like you're 
> aggressively agreeing with me, so I'd like to understand what caused your 
> reaction.

Apologies. On rereading your email, I can see how you meant that.
However, it didn't come across that way to me on first reading. There
have been a few other threads on various lists I've been involved in
recently where it's been really hard to get anyone to see that there's
any practical issues for people if a module is on PyPI rather than
being in the stdlib. So I'm afraid I'd got pretty tired of arguing the
same points over and over again, and over-reacted to what I thought
you said.

To explain my actual point a little more clearly:

1. Without typing available, some programs using type annotations
won't run. That is, using type annotations (a
test-time/development-time feature) introduces a runtime dependency on
typing, and hence introduces an extra deployment concern (unlike other
development-type features like test frameworks).
2. For some people, if something isn't in the Python standard library
(technically, in a standard install), it's not available (without
significant effort, or possibly not at all). For those people, a
runtime dependency on a non-stdlib typing module means "no use of type
annotations allowed".
3. Virtual environments typically only include the stdlib, and "use
system site-packages" has affects more than just a single module, so
bundling still has issues for virtualenvs - and in the packaging
tutorials, we're actively encouraging people to use virtualenvs. We
(python-dev) can work around this issue for venv by auto-installing
typing, but that still leaves virtualenv (which is critically short of
resources, and needs to support older versions of Python, so a major
change like bundling typing is going to be a struggle to get
implemented).

None of these problems are really addressed by simply saying "pip
install typing". That's not for psychological reasons, there are
genuine barriers to having that work. However, it's not at all clear
how many people are affected by these issues. My personal feeling is
that the group of people participating in open source development is
biased towards environments where it's not a problem, but that's more
gut feeling than hard facts. The place where barriers are
perceived/psychological is when we try to discuss workarounds or
solutions - because there's a lot of guesswork about what people's
environments are like, there's a tendency to assume scenarios that
support our preferred solution, rather than those that don't.

Personally, I like to think that my arguments are "giving a voice to
people in constrained corporate environments like mine, who are
under-represented in open source environments". But it's very easy for
a view like that to turn into "banging on about a minor problem as if
it were a crisis", and I'm probably guilty of doing that. Worse still,
that results in me getting frustrated and then over-reacting further -
which is where we came in.

So again, apologies. I misunderstood what you were saying, but more as
a result of my personal biases than because you weren't sufficiently
clear.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steven D'Aprano]

On Mon, Nov 06, 2017 at 01:07:51PM +1000, Nick Coghlan wrote:

> That means our choices for 3.7 boil down to:
> 
> * make this a language level guarantee that Python devs can reasonably rely on
> * deliberately perturb dict iteration in CPython the same way the
> default Go runtime does [1]

I agree with this choice.

My preference is for the first: having dicts be unordered has never been 
a positive virtue in itself, but always the cost we paid for fast O(1) 
access. Now what we have fast O(1) access *without* dicts being 
unordered, we should make it a language guarantee.

Provided of course that we can be reasonable certain that other 
implementations can do the same. And it looks like we can.

But if we wanted to still keep our options open, how about weakening the 
requirement that globals() and object __dicts__ be specifically the same 
type as builtin dict? That way if we discover a super-fast and compact 
dict implementation (maybe one that allows only string keys?) that is 
unordered, we can use it for object namespaces without affecting the 
builtin dict.


> When we did the "insertion ordered hash map" availability review, the
> main implementations we were checking on behalf of were Jython & VOC
> (JVM implementations), Batavia (JavaScript implementation), and
> MicroPython (C implementation). Adding IronPython (C# implementation)
> to the mix gives:

Shouldn't we check with Nuitka (C++) and Cython as well?

I'd be surprised if this is a problem for either of them, but we should 
ask.


> Since the round-trip behaviour that comes from guaranteed order
> preservation is genuinely useful, and we're comfortable with folks
> switching to more specialised containers when they need different
> performance characteristics from what the builtins provide, elevating
> insertion order preservation to a language level requirements makes
> sense.

+1


OrderedDict could then become a thin wrapper around regular dicts.

-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 6 November 2017 at 18:46, Victor Stinner  wrote:
> 2017-11-05 18:50 GMT+01:00 Guido van Rossum :
>> I don't see this as a reason to not put this into the language spec at 3.7.
>
> It can prevent some kinds of optimizations. Dictionaries are used
> "everywhere" in Python, so they are very important for performance.
>
> I would prefer to only keep the ordering warranty for function keyword
> arguments and class members, and use explicitly an ordered dictionary
> when needed.
>
> Sorry, I don't have any example right now of a concrete optimization
> which would not be possible with ordered dictionary. But Serhiy
> mentioned the performance impact of ordering in Python 3.6 dictionary
> on deletion.

Note that I *don't* think we should mandate that regular dictionaries
be synonymous with collections.OrderedDict - I think it's fine to say
that regular dicts are insertion ordered *until* a key is deleted, and
after that their ordering is arbitrary.
Insertion-ordered-until-the-first-delete is sufficient to guarantee
serialisation round trips, dict display ordering, and keyword order
preservation in the dict constructor (it's not sufficient to guarantee
ordering for class namespaces, as those may contain "del" statements,
but class bodies are also permitted to use a non-default mapping
type).

However, if we decide *not* to require that dictionaries be insertion
ordered, then I think we should do the same thing Go did, and have
dict iterators start at a random offset in the item sequence (that's
not actually my idea - Greg Smith suggested it at the core dev
sprints).

Otherwise we'll end up standardising the 3.6 behaviour by default, as
folks come to rely on it, and decline to support Python
implementations that don't provide insertion ordering semantics.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Victor Stinner]

2017-11-05 18:50 GMT+01:00 Guido van Rossum :
> I don't see this as a reason to not put this into the language spec at 3.7.

It can prevent some kinds of optimizations. Dictionaries are used
"everywhere" in Python, so they are very important for performance.

I would prefer to only keep the ordering warranty for function keyword
arguments and class members, and use explicitly an ordered dictionary
when needed.

Sorry, I don't have any example right now of a concrete optimization
which would not be possible with ordered dictionary. But Serhiy
mentioned the performance impact of ordering in Python 3.6 dictionary
on deletion.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Victor Stinner]

2017-11-06 8:47 GMT+01:00 Serhiy Storchaka :
> 06.11.17 09:09, Guido van Rossum пише:
>>
>> I still find this unfriendly to users of Python scripts and small apps who
>> are not the developers of those scripts. (Large apps tend to spit out so
>> much logging it doesn't really matter.)
>>
>> Isn't there a better heuristic we can come up with so that the warnings
>> tend to be on for developers but off for end users?
>
> There was a proposition to make deprecation warnings visible by default in
> debug build and interactive interpreter.

The problem is that outside CPython core developers, I expect that
almost nobody runs a Python compiled in debug mode. We should provide
debug features in the release build. For example, in Python 3.6, I
added debug hooks on memory allocation in release mode using
PYTHONMALLOC=debug. These hooks were already enabled by default in
debug mode.

Moreover, applications are not developed nor tested in the REPL.

Last year, I proposed a global "developer mode". The idea is to
provide the same experience than a Python debug build, but on a Python
release build:

  python3 -X dev script.py
or
  PYTHONDEV=1 python3 script.py
behaves as
  PYTHONMALLOC=debug python3 -Wd -b -X faulthandler script.py

* Show DeprecationWarning and ResourceWarning warnings: python -Wd
* Show BytesWarning warning: python -b
* Enable Python assertions (assert) and set __debug__ to True: remove
(or just ignore) -O or -OO command line arguments
* faulthandler to get a Python traceback on segfault and fatal errors:
python -X faulthandler
* Debug hooks on Python memory allocators: PYTHONMALLOC=debug

If you don't follow the CPython development, it's hard to be aware of
"new" options like -X faulthandler (Python 3.3) or PYTHONMALLOC=debug
(Python 3.6). And it's easy to forget an option like -b.


Maybe we even a need -X dev=strict which would be stricter:

* use -Werror instead of -Wd: raise an exception when a warning is emitted
* use -bb instead of -b: get BytesWarning exceptions
* Replace "inconsistent use of tabs and spaces in indentation" warning
with an error in the Python parser
* etc.

https://mail.python.org/pipermail/python-ideas/2016-March/039314.html

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nathaniel Smith]

On Sun, Nov 5, 2017 at 9:38 PM, Nick Coghlan  wrote:
> We've been running the current experiment for 7 years, and the main
> observable outcome has been folks getting surprised by breaking
> changes in CPython releases, especially folks that primarily use
> Python interactively (e.g. for data analysis), or as a scripting
> engine (e.g. for systems administration).

It's also caused lots of projects to switch to using their own ad hoc
warning types for deprecations, e.g. off the top of my head:

https://github.com/matplotlib/matplotlib/blob/6c51037864f9a4ca816b68ede78207f7ecec656c/lib/matplotlib/cbook/deprecation.py#L5
https://github.com/numpy/numpy/blob/d75b86c0c49f7eb3ec60564c2e23b3ff237082a2/numpy/_globals.py#L45
https://github.com/python-trio/trio/blob/f50aa8e00c29c7f2953b7bad38afc620772dca74/trio/_deprecate.py#L16

So in some ways the change has actually made it *harder* for end-user
applications/scripts to hide all deprecation warnings, because for
each package you use you have to somehow figure out which
idiosyncratic type it uses, and filter them each separately.

(In any changes though please do keep in mind that Python itself is
not the only one issuing deprecation warnings. I'm thinking in
particular of the filter-based-on-Python-version idea. Maybe you could
have subclasses like Py35DeprecationWarning and filter on those?)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 6 November 2017 at 17:09, Guido van Rossum  wrote:
> I still find this unfriendly to users of Python scripts and small apps who
> are not the developers of those scripts.

At a distro level, these warnings being off by default has actually
turned out to be a problem, as it's precisely those users of Python
scripts and small apps running in the system Python that don't find
out they're at risk of a future distro upgrade breaking their tools
until they hit the release where they actually break. They then go to
the developers of either the distro or those tools saying "Everything
is broken, now what do I do?", rather than less urgently asking "Hey,
what's up with this weird warning I'm getting now?".

So compared to that current experience of "My distro upgrade broke my
stuff", getting back to the occasional "After my distro upgrade, a
bunch of my stuff is now emitting messages I don't understand on
stderr" sounds likes a positive improvement to me :)

> Isn't there a better heuristic we can come up with so that the warnings tend
> to be on for developers but off for end users?

That depends on where you're drawing the line between "developer" and
"end user". Someone working on a new version of Django, for example,
would probably qualify as an end user from our perspective, while
they'd be a framework developer from the point of view of someone
building a website.

If we're talking about "Doesn't even know what Python is" end users,
then the most reliable way for devs to ensure they never see a
deprecation warning is to bundle Python with the application, instead
of expecting end users to handle the task of integrating the two
together.

If we're talking about library and frameworks developers, then the
only reliable way to distinguish between deprecation warnings that are
encountered because a dependency has a future compatibility problem
and those that are internal to the application is to use module
filtering:

warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("once", category=DeprecationWarning,
module="myproject.*")
warnings.filterwarnings("once", category=DeprecationWarning,
module="__main__.*")

This model allows folks to more selectively opt-in to getting warnings
from their direct dependencies, while ignoring warnings from further
down their dependency stack.

As things currently stand though, there's little inherent incentive
for new Python users to start learning how to do any of this -
instead, the default behaviour for the last several years has been
"Breaking API changes just happen sometimes without any prior
warning", and you have to go find out how to say "Please tell me when
breaking changes are coming" (and remember to opt in to that every
time you start Python) before you get any prior notification.

I do like Barry's suggestion of introducing a gentler API specifically
for filtering deprecations warnings, though, as I find the warnings
filtering system to be a bit like logging, in that it's sufficiently
powerful and flexible that getting started with it can be genuinely
confusing and intimidating.

In relation to that, the "warn" module README at
https://pypi.python.org/pypi/warn/ provides some additional examples
of how it can currently be difficult to craft a good definition of
which deprecation warnings someone actually wants to see.

Cheers,
Nick.

P.S. That README also points out another problem with the status quo:
DeprecationWarning still gets silenced by default when encountered in
third party modules as well, meaning that also shows up as an abrupt
compatibility break for anyone that didn't already know they needed to
opt-in to get deprecation warnings.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com