Hi Victor, I think this is a nice idea, but I do wonder how this would be used in practice:
The problem is that Python projects typically use lots of 3rd party modules and if a few decide to use/rely on the feature, this may create a situation where packages then compete over which compatibility mode to enable. In such situations, I typically prefer failing early, since detecting whether a package requires an older implementation of a Python API is hard from the outside. Regardless of the outcome of the PEP, what I think would be a great addition is a place where to find the the list of incompatible changes per Python release in a single document - much like you have done in the PEP. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ On 18.10.2019 03:02, Victor Stinner wrote: > Hi, > > I propose the following PEP to add > sys.set_python_compat_version(version) to introduce a partial > compatibility with Python 3.8 in the next Python 3.9 version. > > I also propose this PEP in a pull request: > https://github.com/python/peps/pull/1209 > (Please avoid using the temporary PEP number until this PR is merged: > officially, the PEP has no number yet.) > > If possible, please try to read the whole PEP before replying. I would > prefer to avoid knee-jerk reactions :-) The backward compatibility is > complex topic where things are not black or white: it's more a > grayscale. > > IMHO with the incoming end of Python 2 support, it's the right time to > propose this PEP! > > Victor > > > PEP: xxx > Title: Python Compatibility Version > Author: Victor Stinner <vstin...@python.org> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 18-Oct-2019 > Python-Version: 3.9 > > > Abstract > ======== > > Add ``sys.set_python_compat_version(version)`` to enable partial > compatibility with requested Python version. Add > ``sys.get_python_compat_version()``. > > Modify a few functions of the standard library to implement a partial > compatibility with Python 3.8. > > Add ``sys.set_python_min_compat_version(version)`` to deny backward > compatibility with Python older than *version*. > > Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` > command line options. Add ``PYTHONCOMPATVERSION`` and > ``PYTHONCOMPATMINVERSION`` environment variables. > > > Rationale > ========= > > The need to evolve frequently > ----------------------------- > > To remain relevant and useful, Python has to evolve frequently. Some > enhancements require incompatible changes. Any incompatible change can > break an unknown number of Python projects. Developers can decide to > not implement a feature because of that. > > Users want to get the latest Python version to get new features and > better performance. A few incompatible changes prevent them to use their > applications on the latest Python version. > > This PEP proposes to add a partial compatibility with old Python > versions as a tradeoff to fit both use cases. > > The main issue with the migration from Python 2 to Python 3 is not that > Python 3 is backward incompatible, but how incompatible changes were > introduced. > > > Partial compatibility to minimize the Python maintenance burden > --------------------------------------------------------------- > > While technically it would be possible to provide a full compatibility > with old Python versions, this PEP proposes to minimize the number of > functions handling backward compatibility to reduce the maintenance > burden of the Python project ("CPython"). > > Each change introducing backport compatibility to a function should be > properly discussed to estimate the maintenance cost in the long-term. > > Backward compatibility code will be dropped at each Python release, on a > case by case basis. Each compatibility function can be supported for a > different number of Python releases depending on its maintenance cost > and the estimated risk (number of broken projects) if it's removed. > > The maintenance cost does not only come from the code implementing the > backward compatibility, but come also from additional tests. > > > Cases excluded from backward compatibility > ------------------------------------------ > > The performance overhead of a compatibility code must be low when > ``sys.set_python_compat_version()`` is not called. > > The C API is out of the scope of this PEP: ``Py_LIMITED_API`` macro and > the stable ABI are solving this problem differently, see the `PEP 384: > Defining a Stable ABI <https://www.python.org/dev/peps/pep-0384/>`_. > > Security fixes which break the backward compatibility on purpose will > not get a compatibility layer. Security matters more than compatibility. > For example, ``http.client.HTTPSConnection`` was modified in Python > 3.4.3 to performs all the necessary certificate and hostname checks by > default. It was a deliberate change motivated by the `PEP 476: Enabling > certificate verification by default for stdlib http clients > <https://www.python.org/dev/peps/pep-0476/>`_ (`bpo-22417 > <https://bugs.python.org/issue22417>`_). > > The Python language does not provide backward compatibility. > > Changes which are not clearly incompatible are not covered by this PEP. > For example, Python 3.9 changed the default protocol in the ``pickle`` > module to Protocol 4 which was first introduced in Python 3.4. This > change is backward compatible up to Python 3.4. There is no need to use > the Protocol 3 by default when compatibility with Python 3.8 is > requested. > > New ``DeprecationWarning`` and ``PendingDeprecatingWarning`` warnings > of Python 3.9 will not be disabled in Python 3.8 compatibility mode. > If a project runs its test suite using ``-Werror`` (treat any warning as > an error), these warnings must be fixed, or specific deprecation > warnings must be ignored on a case by case basis. > > > Upgrade a project to a newer Python > ----------------------------------- > > Without backward compatibility, all incompatible changes must be fixed > at once, which can be a blocker issue. It is even worse when a project > is upgraded to a newer Python which is separated by multiple releases > from the old Python. > > Postponing an upgrade only makes things worse: each skipped release adds > more incompatible changes. The technical debt is only steadily > increasing. > > With backward compatibility, it becomes possible to upgrade Python > increamentally in a project, without having to fix all issues at once. > > The "all-or-nothing" is a showstopper to port large Python 2 code bases > to Python 3. The list of incompatible changes between Python 2 and > Python 3 is long, and it's getting longer at each Python 3.x release. > > > Cleaning up Python and DeprecationWarning > ----------------------------------------- > > One of the `Zen of Python (PEP 20) > <https://www.python.org/dev/peps/pep-0020/>`_ motto is: > > There should be one-- and preferably only one --obvious way to do > it. > > When Python evolves, new ways emerge inevitably. ``DeprecationWarning`` > are emitted to suggest to use the new way, but many developers ignore > these warnings, which are silent by default (except in the ``__main__`` > module: see the `PEP 565 <https://www.python.org/dev/peps/pep-0565/>_`). > Some developers simply ignore all warnings since they are too many > warnings, and so only bother with exceptions when deprecated code is > removed. > > Sometimes, supporting both ways has a minor maintenance cost, but > developers prefer to drop the old way to clean up the code. Such kind of > change is backward incompatible. > > Some developers can take the end of the Python 2 support as an > opportunity to push even more incompatible changes than usual. > > Adding backward compatibility as an opt-in prevents to break > applications and allows developers to continue to do such cleanup. > > > Redistribute the maintenance burden > ----------------------------------- > > The backward compatibility involves authors of backward incompatible > changes more in the upgrade path. > > > Examples of backward compatibility > ================================== > > collections ABC aliases > ----------------------- > > ``collections.abc`` aliases to ABC classes have been removed from the > ``collections`` module in Python 3.9, after being deprecated since > Python 3.3. For example, ``collections.Mapping`` no longer exists. > > In Python 3.6, aliases were created in ``collections/__init__.py`` by > ``from _collections_abc import *``. > > In Python 3.7, a ``__getattr__()`` has been added to the ``collections`` > module to emit a DeprecationWarning at the first access to an > attribute:: > > def __getattr__(name): > # For backwards compatibility, continue to make the collections ABCs > # through Python 3.6 available through the collections module. > # Note, no new collections ABCs were added in Python 3.7 > if name in _collections_abc.__all__: > obj = getattr(_collections_abc, name) > import warnings > warnings.warn("Using or importing the ABCs from > 'collections' instead " > "of from 'collections.abc' is deprecated > since Python 3.3, " > "and in 3.9 it will stop working", > DeprecationWarning, stacklevel=2) > globals()[name] = obj > return obj > raise AttributeError(f'module {__name__!r} has no attribute {name!r}') > > Compatibility with Python 3.8 can be restored in Python 3.9 by adding > back the ``__getattr__()`` function, but only when backward > compatibility is requested:: > > def __getattr__(name): > if (sys.get_python_compat_version() < (3, 9) > and name in _collections_abc.__all__): > ... > raise AttributeError(f'module {__name__!r} has no attribute {name!r}') > > > Deprecated open() "U" mode > -------------------------- > > The "U" mode of ``open()`` is deprecated since Python 3.4 and emits a > ``DeprecationWarning``. The `bpo-37330 > <https://bugs.python.org/issue37330>`_ proposes to drop this mode: > ``open()`` would raise an exception if ``U`` mode is used. > > This change falls into the "cleanup" category: it is not required to > implement a feature. > > A backward compatibility mode would be trivial to implement and would be > welcomed here by users. > > > Specification > ============= > > sys functions > ------------- > > Add 3 functions to the ``sys`` module: > > * ``sys.set_python_compat_version(version)``: set the Python > compatibility version. If it has been called previously, use the > minimum of requested versions. Raise an exception if > ``sys.set_python_min_compat_version(min_version)`` has been called and > ``version < min_version``. > *version* must be greater than or equal to ``(3, 0)``. > > * ``sys.set_python_min_compat_version(min_version)``: set the > **minimum** compatibility version. Raise an exception if > ``sys.set_python_compat_version(old_version)`` has been called > previously and ``old_version < min_version``. > *min_version* must be greater than or equal to ``(3, 0)``. > > * ``sys.get_python_compat_version()``: get the Python compatibility > version. Return a ``tuple`` of 3 integers. > > A *version* must a tuple of 2 or 3 integers. ``(major, minor)`` version > is equivalent to ``(major, minor, 0)``. > > By default, ``sys.get_python_compat_version()`` returns the current > Python version. > > Example to request compatibility with Python 3.8.0:: > > import collections > > sys.set_python_compat_version((3, 8)) > > # collections.Mapping alias, removed from Python 3.9, is available > # again, even if collections has been imported before calling > # set_python_compat_version(). > parent = collections.Mapping > > Obviously, calling ``sys.set_python_compat_version(version)`` has no > effect on code executed before the call. Use ``-X > compat_version=VERSION`` command line option or > ``PYTHONCOMPATVERSIONVERSION=VERSION`` environment variable to set the > compatibility version at Python startup. > > Command line > ------------ > > Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` > command line options: call respectivelly > ``sys.set_python_compat_version()`` and > ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string > with 2 or 3 numbers (``major.minor.micro`` or ``major.minor``). For > example, ``-X compat_version=3.8`` calls > ``sys.set_python_compat_version((3, 8))``. > > Add ``PYTHONCOMPATVERSIONVERSION=VERSION`` and > ``PYTHONCOMPATMINVERSION=VERSION=VERSION`` environment variables: call > respectivelly ``sys.set_python_compat_version()`` and > ``sys.set_python_min_compat_version()``. ``VERSION`` is a version > string with the same format that the command line options. > > > Backwards Compatibility > ======================= > > Introducing ``sys.set_python_compat_version()`` function means that an > application will behave differently depending on the compatibility > version. Moreover, since the version can be decreased multiple times, > the application can behave differently depending on the import order. > > Python 3.9 with ``sys.set_python_compat_version((3, 8))`` is not fully > compatible with Python 3.8: the compatibility is only partial. > > > Security Implications > ===================== > > ``sys.set_python_compat_version()`` must not disable security fixes. > > > Alternatives > ============ > > Provide a workaround for each incompatible change > ------------------------------------------------- > > An application can works around most of the incompatible changes which > impacts it. > > For example, ``collections`` aliases can be added again using:: > > import collections.abc > collections.Mapping = collections.abc.Mapping > collections.Sequence = collections.abc.Sequence > > Handle backward compatibility in the parser > ------------------------------------------- > > The parser is modified to support multiple versions of the Python > language (grammar). > > The current Python parser cannot be easily modified for that. AST and > grammar are hardcoded to a single Python version. > > In Python 3.8, ``compile()`` has an undocumented > ``_feature_version`` to not consider ``async`` and ``await`` as > keywords. > > The latest major language backward incompatible change was Python 3.7 > which made ``async`` and ``await`` real keywords. It seems like Twisted > was the only affected project, and Twisted had a single affected > function (it used a parameter called ``async``). > > Handling backward compatibility in the parser seems quite complex, not > only to modify the parser, but also for developers who have to check > which version of the Python language is used. > > from __future__ import python38_syntax > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Add ``pythonXY_syntax`` to the ``__future__`` module. It would enable > backward compatibility with Python X.Y syntax, but only for the current > file. > > With this option, there is no need to change > ``sys.implementation.cache_tag`` to use a different ``.pyc`` filename, > since the parser would always produce the same output for the same input > (except of the optimization level). > > Example:: > > from __future__ import python35_syntax > > async = 1 > await = 2 > > Update cache_tag > ^^^^^^^^^^^^^^^^ > > Modify the parser to use ``sys.get_python_compat_version()`` to choose > the version of the Python language. > > ``sys.set_python_compat_version()`` updates > ``sys.implementation.cache_tag`` to include the compatibility version > without the micro version as a suffix. For example, Python 3.9 uses > ``'cpython-39'`` by default, but > ``sys.set_python_compat_version((3, 7, 2))`` sets ``cache_tag`` to > ``'cpython-39-37'``. Changes of the Python language are now allowed > in micro releases. > > One problem is that ``import asyncio`` is likely to fail if > ``sys.set_python_compat_version((3, 6))`` has been called previously. > The code of the ``asyncio`` module requires ``async`` and ``await`` to > be real keywords (change done in Python 3.7). > > Another problem is that regular users cannot write ``.pyc`` files into > system directories, and so cannot create them on demand. It means that > ``.pyc`` optimization cannot be used in the backward compatibility mode. > > One solution for that is to modify the Python installer and Python > package installers to precompile ``.pyc`` files not only for the current > Python version, but also for multiple older Python versions (up to > Python 3.0?). > > Each ``.py`` file would have 3n ``.pyc`` files (3 optimization levels), > where ``n`` is the number of supported Python versions. For example, it > means 6 ``.pyc`` files, instead of 3, to support Python 3.8 and Python > 3.9. > > > Temporary moratorium on incompatible changes > -------------------------------------------- > > In 2009, the PEP 3003 "Python Language Moratorium" proposed to a > temporary moratorium (suspension) of all changes to the Python language > syntax, semantics, and built-ins for Python 3.1 and Python 3.2. > > In May 2018, during PEP 572 discussions, it was also proposed to slow > down Python changes: see the python-dev thread `Slow down... > <https://mail.python.org/archives/list/python-...@python.org/thread/HHKRXOMRJQH75VNM3JMSQIOOU6MIUB24/#PHA35EAPNONZMTOYBINGFR6XXNMCDPFQ>`_ > > `Barry Warsaw's call on this > <https://mail.python.org/archives/list/python-...@python.org/message/XR7IF2OB3S72KBP3PEQ3IKBOERE4FV2I/>`_: > > I don’t believe that the way for Python to remain relevant and > useful for the next 10 years is to cease all language evolution. > Who knows what the computing landscape will look like in 5 years, > let alone 10? Something as arbitrary as a 10 year moratorium is > (again, IMHO) a death sentence for the language. > > PEP 387 > ------- > > `PEP 387 -- Backwards Compatibility Policy > <https://www.python.org/dev/peps/pep-0387/>`_ proposes a process to make > incompatible changes. The main point is the 4th step of the process: > > See if there's any feedback. Users not involved in the original > discussions may comment now after seeing the warning. Perhaps > reconsider. > > PEP 497 > ------- > > `PEP 497 -- A standard mechanism for backward compatibility > <https://www.python.org/dev/peps/pep-0497/>`_ proposes different > solutions to provide backward compatibility. > > Except of the ``__past__`` mechanism idea, the PEP 497 does not propose > concrete solutions: > > When an incompatible change to core language syntax or semantics is > being made, Python-dev's policy is to prefer and expect that, > wherever possible, a mechanism for backward compatibility be > considered and provided for future Python versions after the > breaking change is adopted by default, in addition to any mechanisms > proposed for forward compatibility such as new future_statements. > > > Examples of incompatible changes > ================================ > > Python 3.8 > ---------- > > Examples of Python 3.8 incompatible changes: > > * (During beta phase) ``PyCode_New()`` required a new parameter: it > broke all Cython extensions (all projects distributing precompiled > Cython code). This change has been reverted during the 3.8 beta phase > and a new ``PyCode_NewWithPosOnlyArgs()`` function was added instead. > > * ``types.CodeType`` requires an additional mandatory parameter. > The ``CodeType.replace()`` function was added to help projects to no > longer depend on the exact signature of the ``CodeType`` constructor. > > * C extensions are no longer linked to libpython. > > * ``sys.abiflags`` changed from ``'m'`` to an empty string. > For example, ``python3.8m`` program is gone. > > * The C structure ``PyInterpreterState`` was made opaque. > > * Blender: > > * https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6 > * https://developer.blender.org/D6038 > > * XML attribute order: `bpo-34160 > <https://bugs.python.org/issue34160>`_. Broken projects: > > * `coverage <https://bugs.python.org/issue34160#msg329612>`_ > * `docutils <https://sourceforge.net/p/docutils/bugs/359/>`_ > * `pcs <https://bugzilla.redhat.com/show_bug.cgi?id=1705475>`_ > * `python-glyphsLib > <https://bugzilla.redhat.com/show_bug.cgi?id=1705391>`_ > > Backward compatibility cannot be added for all these changes. For > example, changes in the C API and in the build system are out of the > scope of this PEP. > > See `What’s New In Python 3.8: API and Feature Removals > <https://docs.python.org/dev/whatsnew/3.8.html#api-and-feature-removals>`_ > for all changes. > > See also the `Porting to Python 3.8 > <https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8>`_ > section of What’s New In Python 3.8. > > > Python 3.7 > ---------- > > Examples of Python 3.7 incompatible changes: > > * ``async`` and ``await`` are now reserved keywords. > * Several undocumented internal imports were removed. One example is > that ``os.errno`` is no longer available; use ``import errno`` > directly instead. Note that such undocumented internal imports may be > removed any time without notice, even in micro version releases. > * Unknown escapes consisting of ``'\'`` and an ASCII letter in > replacement templates for ``re.sub()`` were deprecated in Python 3.5, > and will now cause an error. > * The ``asyncio.windows_utils.socketpair()`` function has been removed: > it was an alias to ``socket.socketpair()``. > * ``asyncio`` no longer exports the ``selectors`` and ``_overlapped`` > modules as ``asyncio.selectors`` and ``asyncio._overlapped``. Replace > ``from asyncio import selectors`` with ``import selectors``. > * PEP 479 is enabled for all code in Python 3.7, meaning that > ``StopIteration`` exceptions raised directly or indirectly in > coroutines and generators are transformed into ``RuntimeError`` > exceptions. > * ``socketserver.ThreadingMixIn.server_close()`` now waits until all > non-daemon threads complete. Set the new ``block_on_close`` class > attribute to ``False`` to get the pre-3.7 behaviour. > * The ``struct.Struct.format`` type is now ``str`` instead of > ``bytes``. > * ``repr`` for ``datetime.timedelta`` has changed to include the keyword > arguments in the output. > * ``tracemalloc.Traceback`` frames are now sorted from oldest to most > recent to be more consistent with ``traceback``. > > Adding backward compatibility for most of these changes would be easy. > > See also the `Porting to Python 3.7 > <https://docs.python.org/dev/whatsnew/3.7.html#porting-to-python-3-7>`_ > section of What’s New In Python 3.7. > > > Micro releases > -------------- > > Sometimes, incompatible changes are introduced in micro releases > (``micro`` in ``major.minor.micro``) to fix bugs or security > vulnerabilities. Examples: > > * Python 3.7.2, ``compileall`` and ``py_compile`` module: the > *invalidation_mode* parameter's default value is updated to ``None``; > the ``SOURCE_DATE_EPOCH`` environment variable no longer > overrides the value of the *invalidation_mode* argument, and > determines its default value instead. > > * Python 3.7.1, ``xml`` modules: the SAX parser no longer processes > general external entities by default to increase security by default. > > * Python 3.5.2, ``os.urandom()``: on Linux, if the ``getrandom()`` > syscall blocks (the urandom entropy pool is not initialized yet), fall > back on reading ``/dev/urandom``. > > * Python 3.5.1, ``sys.setrecursionlimit()``: a ``RecursionError`` > exception is now raised if the new limit is too low at the current > recursion depth. > > * Python 3.4.4, ``ssl.create_default_context()``: RC4 was dropped from > the default cipher string. > > * Python 3.4.3, ``http.client``: ``HTTPSConnection`` now performs all > the necessary certificate and hostname checks by default. > > * Python 3.4.2, ``email.message``: ``EmailMessage.is_attachment()`` is > now a method instead of a property, for consistency with > ``Message.is_multipart()``. > > * Python 3.4.1, ``os.makedirs(name, mode=0o777, exist_ok=False)``: > Before Python 3.4.1, if *exist_ok* was ``True`` and the directory > existed, ``makedirs()`` would still raise an error if *mode* did not > match the mode of the existing directory. Since this behavior was > impossible to implement safely, it was removed in Python 3.4.1 > (`bpo-21082 <https://bugs.python.org/issue21082>`_). > > Examples of changes made in micro releases which are not backward > incompatible: > > * ``ssl.OP_NO_TLSv1_3`` constant was added to 2.7.15, 3.6.3 and 3.7.0 > for backwards compatibility with OpenSSL 1.0.2. > * ``typing.AsyncContextManager`` was added to Python 3.6.2. > * The ``zipfile`` module accepts a path-like object since Python 3.6.2. > * ``loop.create_future()`` was added to Python 3.5.2 in the ``asyncio`` > module. > > No backward compatibility code is needed for such kind of changes. > > > References > ========== > > Accepted PEPs: > > * `PEP 5 -- Guidelines for Language Evolution > <https://www.python.org/dev/peps/pep-0005/>`_ > * `PEP 236 -- Back to the __future__ > <https://www.python.org/dev/peps/pep-0236/>`_ > * `PEP 411 -- Provisional packages in the Python standard library > <https://www.python.org/dev/peps/pep-0411/>`_ > * `PEP 3002 -- Procedure for Backwards-Incompatible Changes > <https://www.python.org/dev/peps/pep-3002/>`_ > > Draft PEPs: > > * `PEP 602 -- Annual Release Cycle for Python > <https://www.python.org/dev/peps/pep-0602/>`_ > * `PEP 605 -- A rolling feature release stream for CPython > <https://www.python.org/dev/peps/pep-0605/>`_ > * See also withdrawn `PEP 598 -- Introducing incremental feature > releases <https://www.python.org/dev/peps/pep-0598/>`_ > > > Copyright > ========= > > This document is placed in the public domain or under the > CC0-1.0-Universal license, whichever is more permissive. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MJEU4GRYR5SFAPMGHUZJTLUO6KG3VQAV/ Code of Conduct: http://python.org/psf/codeofconduct/
