[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

Thanks everybody for your help. It is now documented.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

Ned Deily:
> My preference remains to not make the flags change at all

Well, "m" in ABI flags became useless around Python 3.4. I don't see the point 
of pretending that the "m" ABI is different and then provide two names for the 
same thing:

python3.8 and python3.8m
python3.8-config and python3.8m-config
man python3.8 and man python3.8m
etc.

I'm not sure why, but on Fedora /usr/bin/python3.7 and /usr/bin/python3.7m are 
two separated files... with the same content (ex: same MD5 sum). Well, it's 
just 17 KiB each :-) /usr/bin/python3.7-config is a symlink to 
python3.7m-config.

The change itself is painful for distributors like Red Hat (Fedora, RHEL), but 
things should be simpler once the change is done.

My long-term goal is to get a single ABI for everything: CPython, PyPy, 
RustPython, MicroPython, etc. :-) Maybe it's not possible, but at least I would 
like to move towards this goal!

Overall project:

* 
https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
* https://pythoncapi.readthedocs.io/

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[miss-islington]

miss-islington  added the comment:


New changeset 3fde750cc4e4057076650a92946ec1d492464799 by Miss Islington (bot) 
in branch '3.8':
bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
https://github.com/python/cpython/commit/3fde750cc4e4057076650a92946ec1d492464799


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[miss-islington]

Change by miss-islington :


--
pull_requests: +13953
pull_request: https://github.com/python/cpython/pull/14097

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 7efc526e5cfb929a79c192ac2dcf7eb78d3a4401 by Victor Stinner in 
branch 'master':
bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
https://github.com/python/cpython/commit/7efc526e5cfb929a79c192ac2dcf7eb78d3a4401


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13946
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/14090

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Kubilay Kocak]

Kubilay Kocak  added the comment:

Those script names would already have code to handle name changes based on 
existing abiflags/soabi string (as FreeBSD does), and would be trivial to 
modify for 'm' removal.

For third party stuff: we use setuptools --record for 'everything', so as long 
as filenames are 'correct' for the files that are actually installed, and 
downstreams are clear what the scope of the changes are, 'm' removal is not an 
issue when it come so packaging third party packages/extensions.

A new file extension format or substantially more complex magic would be, on 
the other hand.

A new minor is a good time to make these changes, and there's nothing really 
blocking about a new python38 port/package, and nothing consumes it yet in 
downstream OS's

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Ned Deily]

Ned Deily  added the comment:

Another user-visible difference:

$ ls /tmp/py37/bin/python*
/tmp/py37/bin/python3/tmp/py37/bin/python3.7-config  
/tmp/py37/bin/python3.7m-config
/tmp/py37/bin/python3.7  /tmp/py37/bin/python3.7m
/tmp/py37/bin/python3-config
$ ls /tmp/py38/bin/python*
/tmp/py38/bin/python3/tmp/py38/bin/python3.8-config
/tmp/py38/bin/python3.8  /tmp/py38/bin/python3-config

Granted, these differences aren't the end of the world but:

"But I'm not sure of what are the things impacting by ABIFLAGS="" rather than 
ABIFLAGS="m"."

Not to beat a dead horse but - we really should not make changes like this 
without first understanding what the impact will be.  As it stands, this change 
potentially affects end-users, third-party distributors of Python, and package 
developers.  And, for what reason?  I don't wsee how this change provides any 
real value for anyone and it does cause people to have to change things, like 
creating conditional code for 3.7- and 3.8+.  If we were starting from scratch 
today, sure - "m" is not needed.  But just because we can change something 
doesn't mean we should.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Ned Deily]

Ned Deily  added the comment:

> One question I have is, can/does SOABI flag removal affect any third party 
> package/extension builds in any way, particularly affecting the names of 
> files they produce?

Yep, the default file names of C extension modules differ on most/all? Unix-y 
platforms (certainly vanilla Linux and macOS) because the SOABI is included in 
the file name.

$ /tmp/py38/bin/python3.7 -m pip install --no-binary :all: psutil
$ ls /tmp/py37/lib/python3.7/site-packages/psutil/*.so
/tmp/py37/lib/python3.7/site-packages/psutil/_psutil_linux.cpython-37m-i386-linux-gnu.so
/tmp/py37/lib/python3.7/site-packages/psutil/_psutil_posix.cpython-37m-i386-linux-gnu.so

$ /tmp/py38/bin/python3.8 -m pip install --no-binary :all: psutil
$ ls /tmp/py38/lib/python3.8/site-packages/psutil/*.so
/tmp/py38/lib/python3.8/site-packages/psutil/_psutil_linux.cpython-38-i386-linux-gnu.so
/tmp/py38/lib/python3.8/site-packages/psutil/_psutil_posix.cpython-38-i386-linux-gnu.so

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Kubilay Kocak]

Kubilay Kocak  added the comment:

For FreeBSD Python language ports, the change doesn't have a big negative 
impact for the Python language/interpreter ports themselves

We have the following block in lang/python3? ports:

.if ${PORT_OPTIONS:MPYMALLOC}
ABIFLAGS:=  m${ABIFLAGS}
.endif

.if ${PORT_OPTIONS:MDEBUG}
ABIFLAGS:=  d${ABIFLAGS}
.endif


We then use the variable to substitute the correct suffix into the pkg-plist 
items:

.if !empty(ABIFLAGS)
PLIST_FILES+=   bin/python${PYTHON_VER}${ABIFLAGS} \
bin/python${PYTHON_VER}${ABIFLAGS}-config \
libdata/pkgconfig/python-${PYTHON_VER}${ABIFLAGS}.pc

${INSTALL_DATA} ${WRKSRC}/Tools/gdb/libpython.py \

${STAGEDIR}${PREFIX}/lib/libpython${PYTHON_VER}${ABIFLAGS}.so.1.0-gdb.py

One question I have is, can/does SOABI flag removal affect any third party 
package/extension builds in any way, particularly affecting the names of files 
they produce?

We have a ton of python packages/extensions in the ports tree, so anything that 
affected their builds, particularly when it comes to setuptools --record output 
changes, would be a blocker for supporting 3.8

For example, we had to produce custom ports framework code to account for 
PEP488 [1] and PEP3147 [2] filename suffixes, for these filename differences, 
which was painful at the time:

# PEP 0488 (https://www.python.org/dev/peps/pep-0488/)
.if ${PYTHON_REL} < 3500
PYTHON_PYOEXTENSION=pyo
.else
PYTHON_PYOEXTENSION=opt-1.pyc
.endif


.if ${PYTHON_REL} >= 3200 && defined(_PYTHON_FEATURE_PY3KPLIST)
# When Python version is 3.2+ we rewrite all the filenames
# of TMPPLIST that end with .py[co], so that they conform
# to PEP 3147 (see https://www.python.org/dev/peps/pep-3147/)
PYMAGICTAG= ${PYTHON_CMD} -c 'import sys; 
print(sys.implementation.cache_tag)'
_USES_stage+=   935:add-plist-python
add-plist-python:
@${AWK} '\
/\.py[co]$$/ && !($$0 ~ "/" pc "/") {id = match($$0, 
/\/[^\/]+\.py[co]$$/); if (id != 0) {d = substr($$0, 1, RSTART - 1); dirs[d] = 
1}; sub(/\.pyc$$/,  "." mt "&"); sub(/\.pyo$$/, "." mt "." pyo); 
sub(/[^\/]+\.py[co]$$/, pc "/&"); print; next} \
/^@dirrm / {d = substr($$0, 8); if (d in dirs) {print $$0 "/" 
pc}; print $$0; next} \
/^@dirrmtry / {d = substr($$0, 11); if (d in dirs) {print $$0 
"/" pc}; print $$0; next} \
{print} \
' \
pc="__pycache__" mt="$$(${PYMAGICTAG})" pyo="opt-1.pyc" \
${TMPPLIST} > ${TMPPLIST}.pyc_tmp
@${MV} ${TMPPLIST}.pyc_tmp ${TMPPLIST}


[1] https://www.python.org/dev/peps/pep-0488/
[2] https://www.python.org/dev/peps/pep-3147/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Miro Hrončok]

Miro Hrončok  added the comment:

I'm not sure either. So far most affected buildscripts are confused by:

/usr/include/python3.Xm -> /usr/include/python3.X
/usr/lib64/libpython3.Xm -> /usr/lib64/libpython3.X
extension.cpython-3Xm-arch-linux-gnu.so -> 
extension.cpython-3X-arch-linux-gnu.so

However this is pretty much Linux specific, so I wouldn't know how to explain 
this in an OS agnostic way.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

> It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe 
> enough.  However, this change still is not documented anywhere in 
> https://docs.python.org/dev/whatsnew/3.8.html

I concur that it should be documented. It's on my TODO list. But I'm not sure 
of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m". 
Someone has to propose a PR to document that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Miro Hrončok]

Miro Hrončok  added the comment:

It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough. 
 However, this change still is not documented anywhere in 
https://docs.python.org/dev/whatsnew/3.8.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Miro Hrončok]

Miro Hrončok  added the comment:

> But the impact of the change should probably also be discussed with at least 
> some of the large distributors.

Adapting the Fedora package. Will try to mass rebuild our packages to see what 
breaks.

--
nosy: +hroncok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

> No, you didn't document all the changed file names,  See, for instance, 
> /usr/bin/python3* for a start :)

Oh, I see. I was thinking at something else. First I proposed to drop the "d" 
from the SOABI for debug build, but then I changed my mind. So yeah, I didn't 
change the SOABI for:
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

And this change should be documented elsewhere.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Ned Deily]

Ned Deily  added the comment:

No, you didn't document all the changed file names,  See, for instance, 
/usr/bin/python3* for a start :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

About the PEP, I asked Barry to review the change and he approved it.

About the doc, I didn't know that anyone rely on the exact filename. I 
documented all changes in a top-level section of the What's New in Python 3.8:
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

Sorry for thr annoyance. I didn't know that the macOS installer rely on these 
filenames.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Ned Deily]

Change by Ned Deily :


--
assignee:  -> vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Ned Deily]

Ned Deily  added the comment:

This change has an impact on downstream packagers and users of Python 3.8 
because the SOABI flag values are also used to construct a number of file names 
and directories names in an Unix or macOS framework install besides the 
extension module (*.so) file names.  Suggest comparing the directory/file names 
installed to an alternate prefix (./configure --prefix=/tmp/3x ; make install) 
in 3.7 versus master with this PR.  Also suggest reading PEP 3149, which 
introduced the ABI-specific file names, and the various issues in the bug 
tracker that reference it.  At the very least, these changes need to be 
documented in the What's New for 3.8.  But the impact of the change should 
probably also be discussed with at least some of the large distributors.  (The 
change already inadvertently broke the macOS installer build - fixing.)

--
nosy: +ned.deily
priority: normal -> deferred blocker
resolution: fixed -> 
stage: resolved -> needs patch
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 6c44fde3e03079e0c69f823dafbe04af50b5bd0d by Victor Stinner in 
branch 'master':
bpo-36707: Remove the "m" flag (pymalloc) from SOABI (GH-12931)
https://github.com/python/cpython/commit/6c44fde3e03079e0c69f823dafbe04af50b5bd0d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

Change by STINNER Victor :


--
components: +Build
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Kubilay Kocak]

Change by Kubilay Kocak :


--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

pymalloc is enabled by default. --without-pymalloc allows to build Python
without pymalloc and use malloc by default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

If --with-pymalloc does not impact the API, why we need this option?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

STINNER Victor  added the comment:

I modified PyObject_MALLOC and other macros when I implemented the PEP 445 "Add 
new APIs to customize Python memory allocators":
https://www.python.org/dev/peps/pep-0445/

ABI tags are defined by the PEP 3149:
https://www.python.org/dev/peps/pep-3149/#proposal
"--with-pymalloc (flag: m)"

The PEP 393 already made "--with-wide-unicode (flag: u)" useless in Python 3.3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +12856
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36707] The "m" ABI flag of SOABI for pymalloc is no longer needed

[STINNER Victor]

New submission from STINNER Victor :

In Python 2.7, the WITH_PYMALLOC define (defined by ./configure --with-pymalloc 
which is the default) modified the Python API. For example, PyObject_MALLOC() 
is a macro replaced with PyObject_Malloc() with pymalloc but replaced with 
PyMem_MALLOC() without pymalloc.

In Python 3.8, it's no longer the case, PyObject_MALLOC is no longer modified 
by WITH_PYMALLOC, it's always an alias to PyObject_Malloc() for backward 
compatibility.

#define PyObject_MALLOC   PyObject_Malloc

More generally, WITH_PYMALLOC has no impact on the Python headers nor on the 
ABI in any way. Memory allocators can be switched at runtime using PYTHONMALLOC 
environment variable and -X dev command line option. For example, 
PYTHONMALLOC=malloc disables pymalloc (forces the usage of malloc() of the C 
library).

I propose attached PR which removes the "m" flag from SOABI in Python 3.8 when 
pymalloc is enabled.

--
messages: 340748
nosy: vstinner
priority: normal
severity: normal
status: open
title: The "m" ABI flag of SOABI for pymalloc is no longer needed
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com