Re: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4)
On 1 February 2014 16:04, Ethan Furman wrote: > On 01/31/2014 07:23 PM, Larry Hastings wrote: >> Python is the language that cares about backwards-compatibility--bugs and >> all. If your code runs on version X.Y, it >> should run without modification on version X.(Y+Z) where Z is a positive >> integer. > > > So we only fix bugs that don't work at all? By which I mean, if the > interpreter doesn't crash, we don't fix it? No, we make a judgment call based on the severity and likelihood of encountering the bug, the consequences of encountering it, and the difficulty of working around it after you *do* encounter it. In this case: * Likelihood: low (using keyword arguments with simple APIs like this is not a common idiom, and you have to specifically pass -1 to trigger misbehaviour) * Consequence: application hang. Minimal chance of silent data corruption, high chance of being caught in testing if it's going to be encountered at all. Effectively equivalent impact arises when passing large integer values. * Security impact: negligible. If you're passing untrusted input to the second argument of repeat() at all, you're already exposed, independent of this bug. * Workaround: don't use keyword arguments or else prevalidate the input (the latter can also handle the large integer problem) * Potential backwards compatibility issue?: Yes, as users could be relying on this as a data driven trigger for infinite repetition and the most convenient currently available alternative spelling is rather awkward (involving *args). * Requires a new feature to fix properly?: Yes, as "None" should be added as a replacement, less error prone, data driven trigger for infinite repetition for deprecating or removing the current behaviour. Assessment: * this is a likely rare, high impact, easy to detect, easy to workaround failure where fixing it involves both adding a new feature and potentially breaking currently working code Conclusion: * add the new, supported feature that provides equivalent functionality (accepting None) in 3.5 * deprecate the problematic behaviour in 3.5 and then start treating it as equivalent to the positional argument handling in 3.6+ It's a complex balance between wanting to fix things that are broken in the interpreter and standard library and not breaking currently working end user code. As general rules of thumb: - for maintenance releases and a new feature release approaching release candidate status, err on the side of "preserving backwards compatibility is paramount" and "no new user visible features allowed" - for a new feature release after feature freeze, "no new user visible features allowed" still applies, but there is still scope to make the case for new deprecations and potentially even arguable backwards compatibility breaks that involve adding a note to the porting guide in the "What's New" document - for a new feature release before feature freeze, new features are allowed, as are new deprecations and notes in the porting guide - security issues that are deemed to be the interpreter's responsibility to resolve can trump all of this and lead to the inclusion of new features in maintenance releases. However, even then the new behaviour is likely to be opt-in in the old releases and opt-out in the next feature release (for example, hash randomisation). The "Porting to Python X.Y" guides can be a useful source of examples of previous changes that have been judged as posing too great a risk of breaking end user code to implement in a maintenance release, even when they're to resolve bugs. This is the one for 3.4: http://docs.python.org/dev/whatsnew/3.4.html#porting-to-python-3-4 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add PyType_GetSlot
Am 29.01.14 03:46, schrieb Larry Hastings: > So this would be a new public ABI function? Correct. > Would it be 100% new code, or would you need to refactor code internally > to achieve it? See the patch - it's new code. > Also, just curious: what is typeslots.h used for? I tried searching for > a couple of those macros, and their only appearance in trunk was their > definition. It's meant for use by extension modules. See xxlimited.c:Xxo_Type_slots for an application. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4)
On 02/01/2014 04:20 AM, Nick Coghlan wrote: No, we make a judgment call based on the severity and likelihood of encountering the bug, the consequences of encountering it, and the difficulty of working around it after you *do* encounter it. Thanks for the explanation. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4)
On 02/01/2014 04:20 AM, Nick Coghlan wrote: [snip very nice summary] So you agree that the bug should be fixed. So do I. My disagreement with Larry is that he would leave the bug in until Py4k. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: whatsnew: move of reload, update new windows-only ssl functions entry.
On Sat, Feb 1, 2014 at 12:27 PM, r.david.murray wrote: > http://hg.python.org/cpython/rev/b3f034f5000f > changeset: 4:b3f034f5000f > parent: 2:19d81cc213d7 > user:R David Murray > date:Sat Feb 01 12:27:07 2014 -0500 > summary: > whatsnew: move of reload, update new windows-only ssl functions entry. > > files: > Doc/whatsnew/3.4.rst | 10 -- > 1 files changed, 8 insertions(+), 2 deletions(-) > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -765,6 +765,10 @@ > it will normally be desirable to override the default implementation > for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) > > +The :func:`~importlib.reload` function has been moved from :mod:`imp` > +to :mod:`importlib`. The :func:`mod.reload` name is retained for > +backward compatibility, but is deprecated. > + > That wording seems confusing to me. It makes it seem like importlib.reload is deprecated when in fact it's the imp module itself. -Brett > > inspect > --- > @@ -1062,8 +1066,10 @@ > list of the loaded ``CA`` certificates. (Contributed by Christian Heimes > in > and :issue:`18147`.) > > -Add :func:`ssl.enum_cert_store` to retrieve certificates and CRL from > Windows' > -cert store. (Contributed by Christian Heimes in :issue:`17134`.) > +Two new windows-only functions, :func:`~ssl.enum_certificates` and > +:func:`~ssl.enum_crls` provide the ability to retrieve certificates, > +certificate information, and CRLs from the Windows cert store. > (Contributed > +by Christian Heimes in :issue:`17134`.) > > Support for server-side SNI using the new > :meth:`ssl.SSLContext.set_servername_callback` method. > > -- > Repository URL: http://hg.python.org/cpython > > ___ > Python-checkins mailing list > python-check...@python.org > https://mail.python.org/mailman/listinfo/python-checkins > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4)
On Sat, Feb 01, 2014 at 10:20:24PM +1000, Nick Coghlan wrote:
> On 1 February 2014 16:04, Ethan Furman wrote:
> > So we only fix bugs that don't work at all? By which I mean, if the
> > interpreter doesn't crash, we don't fix it?
>
> No, we make a judgment call based on the severity and likelihood of
> encountering the bug, the consequences of encountering it, and the
> difficulty of working around it after you *do* encounter it.
Nice summary Nick, however there is one slight error:
> In this case:
>
> * Likelihood: low (using keyword arguments with simple APIs like this
> is not a common idiom, and you have to specifically pass -1 to trigger
> misbehaviour)
It's not just -1, its any negative value:
py> from itertools import repeat
py> it = repeat('a', -17) # Correct.
py> next(it)
Traceback (most recent call last):
File "", line 1, in
StopIteration
py> it = repeat('a', times=-17) # Bug, repeats forever.
py> next(it)
'a'
[...]
> Conclusion:
>
> * add the new, supported feature that provides equivalent
> functionality (accepting None) in 3.5
>
> * deprecate the problematic behaviour in 3.5 and then start treating
> it as equivalent to the positional argument handling in 3.6+
Seems reasonable to me.
Thanks again for the nice summary.
--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Guidance regarding what counts as breaking backwards compatibility
Hi all,
Over on the Python-ideas list, there's a thread about the new statistics
module, and as the author of that module, I'm looking for a bit of
guidance regarding backwards compatibility. Specifically two issues:
(1) With numeric code, what happens if the module become more[1]
accurate in the future? Does that count as breaking backwards
compatibility?
E.g. Currently I use a particular algorithm for calculating variance.
Suppose that for a particular data set, this algorithm is accurate to
(say) seven decimal places:
# Python 3.4
variance(some_data) == 1.2345671
Later, I find a better algorithm, which improves the accuracy of the
result:
# Python 3.5 or 3.6
variance(some_data) == 1.23456789001
Would this count as breaking backwards compatibility? If so, how should
I handle this? I don't claim that the current implementation of the
statistics module is optimal, as far as precision and accuracy is
concerned. It may improve in the future.
Or would that count as a bug-fix? "Variance function was inaccurate, now
less wrong", perhaps.
I suppose the math module has the same issue, except that it just wraps
the C libraries, which are mature and stable and unlikely to change.
The random module has a similar issue:
http://docs.python.org/3/library/random.html#notes-on-reproducibility
(2) Mappings[2] are iterable. That means that functions which expect
sequences or iterators may also operate on mappings by accident. For
example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the
opportunity to handle mappings specifically in the future, without being
locked in by backwards-compatibility, how should one handle it?
a) document that behaviour with mappings is unsupported and may
change in the future;
b) raise a warning when passed a mapping, but still iterate over it;
c) raise an exception and refuse to iterate over the mapping;
d) something else?
Question (2) is of course a specific example of a more general
question, to what degree is the library author responsible for keeping
backwards compatibility under circumstances which are not part of the
intended API, but just work by accident?
[1] Or, for the sake of the argument, less accurate.
[2] And sets.
--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility
On 2/1/2014 8:06 PM, Steven D'Aprano wrote:
Hi all,
Over on the Python-ideas list, there's a thread about the new statistics
module, and as the author of that module, I'm looking for a bit of
guidance regarding backwards compatibility. Specifically two issues:
(1) With numeric code, what happens if the module become more[1]
accurate in the future? Does that count as breaking backwards
compatibility?
E.g. Currently I use a particular algorithm for calculating variance.
Suppose that for a particular data set, this algorithm is accurate to
(say) seven decimal places:
# Python 3.4
variance(some_data) == 1.2345671
Later, I find a better algorithm, which improves the accuracy of the
result:
# Python 3.5 or 3.6
variance(some_data) == 1.23456789001
Would this count as breaking backwards compatibility? If so, how should
I handle this? I don't claim that the current implementation of the
statistics module is optimal, as far as precision and accuracy is
concerned. It may improve in the future.
Or would that count as a bug-fix? "Variance function was inaccurate, now
less wrong", perhaps.
That is my inclination.
I suppose the math module has the same issue, except that it just wraps
the C libraries, which are mature and stable and unlikely to change.
Because C libraries differ, math results differ even in the same
version, so they can certainly change (hopefully improve) in future
versions. I think the better analogy is cmath, which I believe is more
than just a wrapper.
The random module has a similar issue:
http://docs.python.org/3/library/random.html#notes-on-reproducibility
(2) Mappings[2] are iterable. That means that functions which expect
sequences or iterators may also operate on mappings by accident.
I think 'accident' is the key. (Working with sets is not an accident.)
Anyone who really wants the mean of keys should be explicit:
mean(d.keys())
example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the
opportunity to handle mappings specifically in the future, without being
locked in by backwards-compatibility, how should one handle it?
a) document that behaviour with mappings is unsupported and may
change in the future;
I think the doc should in any case specify the proper domain. In this
case, I think it should exclude mappings: 'non-empty non-mapping
iterable of numbers', or 'an iterable of numbers that is neither empty
nor a mapping'. That makes the behavior at best undefined and subject to
change. There should also be a caveat about mixing types, especially
Decimals, if not one already. Perhaps rewrite the above as 'an iterable
that is neither empty nor a mapping of numbers that are mutually summable'.
b) raise a warning when passed a mapping, but still iterate over it;
c) raise an exception and refuse to iterate over the mapping;
This, if possible. An empty iterable will raise at '/ 0'. Most anything
that is not an iterable of number will eventually raise at '/ n'
Testing both that an exception is raised and that it is one we want is
why why unittest has assertRaises.
Question (2) is of course a specific example of a more general
question, to what degree is the library author responsible for keeping
backwards compatibility under circumstances which are not part of the
intended API, but just work by accident?
[1] Or, for the sake of the argument, less accurate.
[2] And sets.
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility
On 2 February 2014 11:06, Steven D'Aprano wrote:
> Hi all,
>
> Over on the Python-ideas list, there's a thread about the new statistics
> module, and as the author of that module, I'm looking for a bit of
> guidance regarding backwards compatibility. Specifically two issues:
>
>
> (1) With numeric code, what happens if the module become more[1]
> accurate in the future? Does that count as breaking backwards
> compatibility?
>
> E.g. Currently I use a particular algorithm for calculating variance.
> Suppose that for a particular data set, this algorithm is accurate to
> (say) seven decimal places:
>
> # Python 3.4
> variance(some_data) == 1.2345671
>
> Later, I find a better algorithm, which improves the accuracy of the
> result:
>
> # Python 3.5 or 3.6
> variance(some_data) == 1.23456789001
>
>
> Would this count as breaking backwards compatibility? If so, how should
> I handle this? I don't claim that the current implementation of the
> statistics module is optimal, as far as precision and accuracy is
> concerned. It may improve in the future.
For this kind of case, we tend to cover it in the "Porting to Python
X.Y" section of the What's New guide. User code *shouldn't* care about
this kind of change, but it *might*, so we split the difference and
say "It's OK in a feature release, but not in a maintenance release".
There have been multiple changes along these lines in our floating
handling as Tim Peters, Mark Dickinson et al have made various
improvements to reduce platform dependent behaviour (especially around
overflow handling, numeric precision, infinity and NaN handling, etc).
However, we also sometimes have module specific disclaimers - the
decimal module, for example, has an explicit caveat that updates to
the General Decimal Arithmetic Specification will be treated as bug
fixes, even if they would normally not be allowed in maintenance
releases.
For a non-math related example, a comment from Michael Foord at the
PyCon US 2013 sprints made me realise that the implementation of
setting the __wrapped__ attribute in functools was just flat out
broken - when applied multiple times it was supposed to create a chain
of references that eventually terminated in a callable without the
attribute set, but due to the bug every layer actually referred
directly to the innermost callable (the one without the attribute
set). Unfortunately, the docs I wrote for it were also ambiguous, so a
lot of folks (including Michael) assumed it was working as intended. I
have fixed the bug in 3.4, but there *is* a chance it will break
introspection code that assumed the old behaviour was intentional and
doesn't correctly unravel __wrapped__ chains.
> Or would that count as a bug-fix? "Variance function was inaccurate, now
> less wrong", perhaps.
>
> I suppose the math module has the same issue, except that it just wraps
> the C libraries, which are mature and stable and unlikely to change.
They may look that way *now*, but that's only after Tim, Mark et al
did a lot of work on avoiding platform specific issues and
inconsistencies
> The random module has a similar issue:
>
> http://docs.python.org/3/library/random.html#notes-on-reproducibility
I think a disclaimer in the statistics module similar to the ones in
the math module and this one in the random module would be appropriate
- one of the key purposes of the library/language reference is to let
us distinguish between "guaranteed behaviour user code can rely on"
and "implementation details that user code should not assume will
remain unchanged forever".
In this case, it would likely be appropriate to point out that the
algorithms used internally may change over time, thus potentially
changing the error bounds in the module output.
> (2) Mappings[2] are iterable. That means that functions which expect
> sequences or iterators may also operate on mappings by accident. For
> example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the
> opportunity to handle mappings specifically in the future, without being
> locked in by backwards-compatibility, how should one handle it?
>
> a) document that behaviour with mappings is unsupported and may
>change in the future;
>
> b) raise a warning when passed a mapping, but still iterate over it;
>
> c) raise an exception and refuse to iterate over the mapping;
>
> d) something else?
>
>
> Question (2) is of course a specific example of a more general
> question, to what degree is the library author responsible for keeping
> backwards compatibility under circumstances which are not part of the
> intended API, but just work by accident?
In this particular case, I consider having a single API treat mappings
differently from other iterables is a surprising anti-pattern and
providing a separate API specifically for mappings is clearer (cf
format() vs format_map()).
However, if you want to preserve maximum flexibility, the best near
term option is typically c (just disallow the input you haven't
decided how to han
