On Sun, Jul 10, 2016 at 10:15:33AM +1000, Nick Coghlan wrote:
> On 10 July 2016 at 05:10, Steven D'Aprano <st...@pearwood.info> wrote:
> > The other side of the issue is that requiring exact correspondence is
> > considerably more difficult and may be over-kill for some uses.
> > Hypothetically speaking, if I have an object that only supports pickling
> > by accident, and I replace it with one that doesn't support pickling,
> > shouldn't it be my decision whether that counts as a functional
> > regression (a bug) or a reliance on an undocumented and accidental
> > implementation detail?
>
> That's the proposed policy change and the reason I figured it needed a
> python-dev discussion, as currently it's up to folks adding the Python
> equivalents (or the C accelerators) to decide on a case by case basis
> whether or not to care about compatibility for:
>
> - string representations
> - pickling (and hence multiprocessing support)
> - subclassing
> - isinstance checks
> - descriptor behaviour
Right... and that's what I'm saying *ought* to be the decision of the
maintainer. Do I understand that you agree with this, but you want to
ensure that such decisions are made up front rather than when and if
discrepencies are noticed?
> The main way for such discrepancies to arise is for the Python
> implementation to be a function (or closure), while the C
> implementation is a custom stateful callable.
>
> The problem with the current "those are just technical implementation
> details" approach is that a lot of Pythonistas learn standard library
> API behaviour and capabilities through a mix of experimentation and
> introspection rather than reading the documentation,
Indeed.
> so if CPython
> uses the accelerated version by default, then most folks aren't going
> to notice the discrepancies until they (or their users) are trying to
> debug a problem like "my library works fine in CPython, but breaks
> when used with multiprocessing on PyPy" or "my doctests fail when
> running under MicroPython".
Yes, and that's a problem, but is it a big enough problem to justify a
policy change and pre-emptive effort to prevent it?
The great majority of people are never going to run their Python code on
anything other than CPython, and while I always encourage people to
write in the most platform-independent fashion possible, I am also
realistic enough to recognise that platform independence is an ideal
that many people will fail to meet. (I'm sure that I've written code
that isn't as platform independent as I hope.)
My two core questions are:
(1) How much extra effort are we going to *mandate* that core devs put
in to hide the differences between C and Python code, for the benefit of
a small minority that will notice them?
(2) When should that effort be done? Upfront, or when and as problems
are reported or noticed?
My preference for answers will be, (1) not much, and (2) when problems
are reported. In other words, close to the status quo.
I can't speak for others, but I have a tendency towards over-analysing
my code, trying to pre-emptively spot and avoid even really obscure
failure modes before they occur. That's a trap: it makes it hard to
finish (as much as any code is finished) and harder to meet deadlines.
It's taken me a lot of effort, and much influence from TDD, to realise
that it's okay to release code with a bug you didn't spot. You can
always fix it in the next release.
I think the same applies here. I'm okay with something close to the
status quo: if a C accelerator doesn't quite have the same undocumented
interface as the pure Python one, then its a bug in one or the other,
in which case its okay to fix it when somebody notices.
But I don't think I'm okay to make it mandatory that we prevent such
possible incompatibilities ahead of time.
If we do make this mandatory, how is it going to be enforced and
checked? The normal way to enforce that accelerated code has the same
behaviour as Python code is to see that they both pass the same tests.
But this can only check for features where a test has been written. If
you don't think of an incompatibility ahead of time, how do you write a
test for it?
I appreciate that the standard library should be held up to a higher
level of professionalism than external code, but I don't think that
*all* the burden should fall on the core developers. Reliance on
undocumented features is always a dubious thing to do. We all do it, and
when it turns out that the feature can't be counted on (because it
changes from one version to another, or isn't available on some
platforms), who is to blame for our application breaking?
As the programmer who relied on a promise that was never made, surely I
must take at least a bit of responsibility? Its not like the docs are
locked up in a filing cabinent in the basement behind a door with a sign
saying "Beware of the leopard".
I'm just not comfortable with mandating that core devs must do even more
work to protect programmers (including myself) from our own failures to
code defensively and read the docs, with respect to this specific issue.
We have to draw the line somewhere. I've seen people write code that
relies on the exact wording of error messages. I've even done that
myself, a long time ago. I hope that we would all agree that mirroring
error messages is crossing the line. But beyond that, I'm not sure where
the line lies, and I'd rather put off dealing with it until necessary,
and on a case-by-case basis.
> For non-standard library code, those kinds of latent compatibility
> defects are fine, but PEP 399 is specifically about setting design &
> development policy for the *standard library* to help improve
> cross-implementation compatibility not just of the standard library
> itself, but of code *using* the standard library in ways that work on
> CPython in its default configuration.
PEP 399 already raises this issue. Quote:
Any new accelerated code must act as a drop-in replacement as
close to the pure Python implementation as reasonable. Technical
details of the VM providing the accelerated code are allowed to
differ as necessary
Are you saying that's not good enough? If so, what's your proposed
wording?
--
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
