[Python-Dev] Usefulness of binary compatibility accross Python versions?
Hello, Nowadays we have an official mechanism for third-party C extensions to be binary-compatible accross feature releases of Python: the stable ABI. But, for non-stable ABI-using C extensions, there are also mechanisms in place to *try* and ensure binary compatibility. One of them is the way in which we add tp_ slots to the PyTypeObject structure. Typically, when adding a tp_XXX slot, you also need to add a Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures that have been compiled against a recent enough PyTypeObject definition. This way, extensions compiled against Python N-1 are supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, the core Python runtime won't try to access the (non-existing) tp_XXX member. However, beside internal code complication, it means you need to add a new Py_TPFLAGS_HAVE_XXX each time we add a slot. Since we have only 32 such bits available (many of them already taken), it is a very limited resource. Is it worth it? (*) Can an extension compiled against Python N-1 really claim to be compatible with Python N, despite other possible differences? (*) we can't extend the tp_flags field to 64 bits, precisely because of the binary compatibility problem... Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
I think it's more acceptable to require matching versions now than it was 10 years ago -- people are much more likely to use installer tools like pip and conda that can check version compatibility. I think I'd be okay with dropping the flag-based mechanism you describe if we were to introduce a clear mechanism that always rejected a dynamically loaded module if it was compiled for a different Python version. This should happen without any cooperation from the module. Perhaps in Python.h we can introduce a reference to a variable whose name varies by version (major.minor, I think) and which is defined only by the interpreter itself. Or perhaps the version should be based on a separate ABI version. On Sat, Dec 16, 2017 at 5:22 AM, Antoine Pitrou wrote: > > Hello, > > Nowadays we have an official mechanism for third-party C extensions to > be binary-compatible accross feature releases of Python: the stable ABI. > > But, for non-stable ABI-using C extensions, there are also mechanisms > in place to *try* and ensure binary compatibility. One of them is the > way in which we add tp_ slots to the PyTypeObject structure. > > Typically, when adding a tp_XXX slot, you also need to add a > Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures > that have been compiled against a recent enough PyTypeObject > definition. This way, extensions compiled against Python N-1 are > supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, > the core Python runtime won't try to access the (non-existing) tp_XXX > member. > > However, beside internal code complication, it means you need to add a > new Py_TPFLAGS_HAVE_XXX each time we add a slot. Since we have only 32 > such bits available (many of them already taken), it is a very limited > resource. Is it worth it? (*) Can an extension compiled against Python > N-1 really claim to be compatible with Python N, despite other possible > differences? > > (*) we can't extend the tp_flags field to 64 bits, precisely because of > the binary compatibility problem... > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
On Sat, 16 Dec 2017 09:34:02 -0800
Guido van Rossum wrote:
> I think it's more acceptable to require matching versions now than it was
> 10 years ago -- people are much more likely to use installer tools like pip
> and conda that can check version compatibility.
>
> I think I'd be okay with dropping the flag-based mechanism you describe if
> we were to introduce a clear mechanism that always rejected a dynamically
> loaded module if it was compiled for a different Python version. This
> should happen without any cooperation from the module. Perhaps in Python.h
> we can introduce a reference to a variable whose name varies by version
> (major.minor, I think) and which is defined only by the interpreter itself.
> Or perhaps the version should be based on a separate ABI version.
Interestingly, Python 2 had such an API version check (though it would
only emit a warning), it was removed as part of PEP 3121:
https://github.com/python/cpython/commit/1a21451b1d73b65af949193208372e86bf308411#diff-4664e4ea04dc636b18070ba01cf42d06L39
I haven't been able to find the pre-approval discussion around PEP
3121, so I'm not sure why the API check was removed. The PEP (quite
short!) also says nothing about it.
Currently, you can pass a `module_api_version` to PyModule_Create2(),
but that function is for specialists only :-)
("""Most uses of this function should be using PyModule_Create()
instead; only use this if you are sure you need it.""")
And the new multi-phase initialization API doesn't seem to support
passing an API version:
https://docs.python.org/3/c-api/module.html#multi-phase-initialization
Fortunately, nowadays all major platforms (Windows, Linux, macOS) tag
C extension filenames with the interpreter version (*), e.g.:
- "XXX.cpython-35m-darwin.so" on macOS
- "XXX?.cp35-win32.pyd" on Windows
- "XXX?.cpython-37dm-x86_64-linux-gnu.so" on Linux and others
So in practice there should be little potential for confusion, except
when renaming the extension file?
(*) references:
https://bugs.python.org/issue22980
https://docs.python.org/3.7/whatsnew/3.5.html#build-and-c-api-changes
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
On Sat, 16 Dec 2017 19:37:54 +0100
Antoine Pitrou wrote:
>
> Currently, you can pass a `module_api_version` to PyModule_Create2(),
> but that function is for specialists only :-)
>
> ("""Most uses of this function should be using PyModule_Create()
> instead; only use this if you are sure you need it.""")
Ah, it turns out I misunderstood that piece of documentation and also
what PEP 3121 really did w.r.t the module API check.
PyModule_Create() is actually a *macro* calling PyModule_Create2() with
the version number is was compiled against!
#ifdef Py_LIMITED_API
#define PyModule_Create(module) \
PyModule_Create2(module, PYTHON_ABI_VERSION)
#else
#define PyModule_Create(module) \
PyModule_Create2(module, PYTHON_API_VERSION)
#endif
And there's already a check for that version number in moduleobject.c:
https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114
That check is always invoked when calling PyModule_Create() and
PyModule_Create2(). Currently it merely invokes a warning, but we can
easily turn that into an error.
(with apologies to Martin von Löwis for not fully understanding what he
did at the time :-))
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
On Sat, Dec 16, 2017 at 11:14 AM, Antoine Pitrou
wrote:
> On Sat, 16 Dec 2017 19:37:54 +0100
> Antoine Pitrou wrote:
> >
> > Currently, you can pass a `module_api_version` to PyModule_Create2(),
> > but that function is for specialists only :-)
> >
> > ("""Most uses of this function should be using PyModule_Create()
> > instead; only use this if you are sure you need it.""")
>
> Ah, it turns out I misunderstood that piece of documentation and also
> what PEP 3121 really did w.r.t the module API check.
>
> PyModule_Create() is actually a *macro* calling PyModule_Create2() with
> the version number is was compiled against!
>
> #ifdef Py_LIMITED_API
> #define PyModule_Create(module) \
> PyModule_Create2(module, PYTHON_ABI_VERSION)
> #else
> #define PyModule_Create(module) \
> PyModule_Create2(module, PYTHON_API_VERSION)
> #endif
>
> And there's already a check for that version number in moduleobject.c:
> https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114
>
> That check is always invoked when calling PyModule_Create() and
> PyModule_Create2(). Currently it merely invokes a warning, but we can
> easily turn that into an error.
>
> (with apologies to Martin von Löwis for not fully understanding what he
> did at the time :-))
>
If it's only a warning, I worry that if we stop checking the flag bits it
can cause wild pointer following. This sounds like it would be a potential
security issue (load a module, ignore the warning, try to use a certain API
on a class it defines, boom). Also, could there still be 3rd party modules
out there that haven't been recompiled in a really long time and use some
older backwards compatible module initialization API? (I guess we could
stop supporting that and let them fail hard.)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
On Sat, 16 Dec 2017 11:42:15 -0800 Guido van Rossum wrote: > > If it's only a warning, I worry that if we stop checking the flag bits it > can cause wild pointer following. This sounds like it would be a potential > security issue (load a module, ignore the warning, try to use a certain API > on a class it defines, boom). Also, could there still be 3rd party modules > out there that haven't been recompiled in a really long time and use some > older backwards compatible module initialization API? (I guess we could > stop supporting that and let them fail hard.) As far as I can tell, all the legacy APIs were removed when PEP 3121 was implemented (Python 3 allowed us to do a clean break here). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
Well then, maybe you can propose some specific set of changes? (I'm about to go on vacation and I'd like to focus on other things for the next two weeks though, so don't count on me too much.) On Sat, Dec 16, 2017 at 3:49 PM, Antoine Pitrou wrote: > On Sat, 16 Dec 2017 11:42:15 -0800 > Guido van Rossum wrote: > > > > If it's only a warning, I worry that if we stop checking the flag bits it > > can cause wild pointer following. This sounds like it would be a > potential > > security issue (load a module, ignore the warning, try to use a certain > API > > on a class it defines, boom). Also, could there still be 3rd party > modules > > out there that haven't been recompiled in a really long time and use some > > older backwards compatible module initialization API? (I guess we could > > stop supporting that and let them fail hard.) > > As far as I can tell, all the legacy APIs were removed when PEP 3121 > was implemented (Python 3 allowed us to do a clean break here). > > Regards > > Antoine. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Decision of having a deprecation period or not for changing csv.DictReader returning type.
Hi, guys In https://github.com/python/cpython/pull/4904, I made csv.DictReader returning regular dict instead of OrderedDict. But this code could break existing code that relied on methods like move_to_end() which are present in OrderedDict() but not in dict(). As rhettinger suggested, such code is either unlikely or rare, so it would be net less disruptive for users to just go forward with this patch. So, would we just go forward with this patch or having a deprecation period before this patch? Any help is big thanks. -- Best Regards . shangdahao ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
