https://fedoraproject.org/wiki/Changes/Python_Extension_Flags
== Summary ==
The build flags (<code>CFLAGS</code>, <code>CXXFLAGS</code> and
<code>LDFLAGS</code>) saved in the Python's distutils module for
building extension modules are switched from:
* <code>%{build_cflags}</code>,
* <code>%{build_cxxflags}</code> and
* <code>%{build_ldflags}</code>
to:
* <code>%{extension_cflags}</code>,
* <code>%{extension_cxxflags}</code> and
* <code>%{extension_ldflags}</code>.
This
[https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders
currently means] that no GCC plugins (such as annobin) are activated
and no GCC spec files (<code>-specs=</code> arguments) are used by
default when building Python extension modules (e.g. with
<code>python3 setup.py build</code>).
The {{package|python3-devel}} package will lose its runtime dependency
on {{package|redhat-rpm-config}} (which was only required for annobin
support and GCC spec files).
The change will affect building extension modules by users, outside of
the RPM environment. The Python standard library and Fedora's Python 3
RPM packages are still built with the "traditional" set of flags
(<code>%{build_cflags}</code> and friends), unless the package uses
nonstandard methods to build the extensions.
Only Python 3.7 ({{package|python3}}) and 3.6 ({{package|python36}})
will be changed.
== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Cstratak|Charalampos
Stratakis]]
* Email: python-ma...@redhat.com
=== The Problem ===
When Python is built, it saves the flags (<code>CFLAGS</code>,
<code>CXXFLAGS</code> and <code>LDFLAGS</code>) for further use when
building extension modules into a designated <code>Makefile</code>.
The distutils module (a component responsible for building Python
packages and extension modules) then reads the file and applies the
flags. You can see the file at
<code>/usr/lib64/python3.7/config-3.7m-x86_64-linux-gnu/Makefile</code>
in {{package|python3-libs}}. This is mostly done to make user-built
extension modules binary compatible with the Python interpreter they
are being built for.
Traditionally (=before this change), the {{package|python3}} package
was created in a way that it simply saved the same set of flags used
for building itself.
This proved problematic as the flags used to build Fedora packages
grew specific things (not actually needed for binary compatibility of
the extension modules) and several workarounds needed to be made, most
specifically the {{package|python3-devel}} package got a runtime
dependency on {{package|redhat-rpm-config}}:
* https://bugzilla.redhat.com/show_bug.cgi?id=1217376
* https://bugzilla.redhat.com/show_bug.cgi?id=1496757
* https://bugzilla.redhat.com/show_bug.cgi?id=1218294
The problematic flags are GCC plugins (such as annobin) and GCC spec
files (<code>-specs=</code> arguments).
Example: Any Python developer using Fedora automatically builds Python
extension modules with annobin and hardening flags by default even if
they don't need that. They might build the extension on Fedora, test
it and later ship it and build it on a CI that is not based on Fedora
and get different results.
=== The Solution ===
The solution is not to save the problematic flags, but only the flags
needed. Until recently, this would be hackish, but
[https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders
a designated set of flags] was created in Fedora 30, supposed to be
used by extension builders (such as the Python's distutil module):
<code>%{extension_cflags}</code>, <code>%{extension_cxxflags}</code>
and <code>%{extension_ldflags}</code>.
The
[https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders
documentation for the flags] currently reads:
<blockquote>
The current set of differences are:
* No GCC plugins (such as annobin) are activated.
* No GCC spec files (<code>-specs=</code> arguments) are used.
Additional flags may be removed in the future if they prove to be
incompatible with alternative toolchains.
</blockquote>
Python already had (incomplete) support for specifying different sets
of flags for itself and for the extensions. In Python 3.7.2 and 3.6.8
the ability was completed by providing a way to do this for
<code>LDFAGS</code> (this was actually driven by Fedora's needs).
Currently we are able to set a different set of flags for building
Python and for building user extension modules. The code implementing
this change can be inspected at
[https://src.fedoraproject.org/rpms/python3/pull-request/75 Fedora
python3 PR #75] and [https://github.com/python/cpython/pull/10900
CPython PR #10900].
=== Impact on users building extension modules ===
When an user compiles a Python extension module outside of RPMBuild
and does not specify any flags explicitly, this is what they used to
have:
$ python3 setup.py build
running build
running build_ext
building 'demo' extension
creating build
creating build/temp.linux-x86_64-3.7
gcc -pthread -Wno-unused-result -Wsign-compare
-DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c demo.c
-o build/temp.linux-x86_64-3.7/demo.o
creating build/lib.linux-x86_64-3.7
gcc -pthread -shared -Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -g
build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o
build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so
After the change, they will get:
$ python3 setup.py build
running build
running build_ext
building 'demo' extension
creating build
creating build/temp.linux-x86_64-3.7
gcc -pthread -Wno-unused-result -Wsign-compare
-DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv
-fPIC -I/usr/include/python3.7m -c demo.c -o
build/temp.linux-x86_64-3.7/demo.o
creating build/lib.linux-x86_64-3.7
gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g
build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o
build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so
Users are able to provide their own flags by setting the
<code>CFLAGS</code>/<code>CXXFLAGS</code> and <code>LDFLAGS</code>
environment variables. They can, for example, opt-in for annobin if
they wish to do so.
=== Impact on RPM packages building extension modules ===
When the spec file of the package sets the
<code>CFLAGS</code>/<code>CXXFLAGS</code> and <code>LDFLAGS</code>
variables to the expected Fedora values (e.g.
<code>CFLAGS=${RPM_OPT_FLAGS} LDFLAGS=${RPM_LD_FLAGS}</code> or
<code>%{build_cflags}</code>, <code>%{build_cxxflags}</code>,
<code>%{build_ldflags}</code>, <code>%{optflags}</code> etc.)
everything works as expected. The currently documented way of building
Python packages (<code>%py3_build</code> and
<code>%py3_install</code>)
[https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.python3
does this] for you, we recommend using it.
There might be some differences in the log, because the flags
concatenate. Here is the example from the {{package|python-psycopg2 }}
build.log.
Before this change:
gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG '''-O2 -g -pipe
-Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection''' -D_GNU_SOURCE -fPIC -fwrapv '''-O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection''' -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1
-DPSYCOPG_VERSION=2.7.5 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110000
-DHAVE_LO64=1 -I/usr/include/python3.7m -I. -I/usr/include
-I/usr/include/pgsql/server -c psycopg/psycopgmodule.c -o
build/temp.linux-x86_64-3.7/psycopg/psycopgmodule.o
-Wdeclaration-after-statement
After this change:
gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG '''-O2 -g -pipe
-Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection''' -D_GNU_SOURCE -fPIC
-fwrapv '''-O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection''' -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1
-DPSYCOPG_VERSION=2.7.5 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110000
-DHAVE_LO64=1 -I/usr/include/python3.7m -I. -I/usr/include
-I/usr/include/pgsql/server -c psycopg/psycopgmodule.c -o
build/temp.linux-x86_64-3.7/psycopg/psycopgmodule.o
-Wdeclaration-after-statement
If the package does <code>python3 setup.py build</code> to build the
extension modules manually without setting the flags, the flags are
not set properly. This is probably very rare. Even before the
<code>%py3_build</code> and <code>%py3_install</code> macros, the
preferred way was to call:
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build # from EL4
However, <code>LDFLAGS</code> are not set in that example.
There are 485 packages that have the <code>%{python3_sitearch}</code>
macro in them and hence are most likely to build Python extension
modules.
Out of those, 173 have <code>setup.py</code> directly in the spec. 65
have <code>setup.py build</code>. Change owners will go trough the
list and provide fixes. Packages are encouraged to switch to the
<code>%py3_build</code> macro themselves.
Worst case: Some package won't have annobin.
=== Python interpreters affected by this change ===
This change is fully possible (without backporting) in Python 3.6.8 /
3.7.2 or newer. Hence we'll only change the {{package|python3}} and
{{package|python36}} packages.
We'll keep older Pythons intact and let them finish their lifetime as
they are. Newer Python versions will inherit this change.
PyPy 3 ({{package|pypy3}}) might eventually be updated to a version
that supports this fully. Since no packages in Fedora use PyPy 3,
we'll update it then without a change proposal and we'll document the
fact on this Change page. (If any package starts using PyPy 3 untill
then, we'll coordinate with the maintainer/s).
== Benefit to Fedora ==
Python developers will get more upstream-like experience when building
Python extension modules.
Python developers won't need {{package|redhat-rpm-macros}} (and dozens
of other language-specific packages that go with that) installed just
to do so.
New decisions made about the distro packages won't necessarily affect
Python developers building their extension modules.
== Scope ==
* Proposal owners:
** Review, merge and build the
[https://src.fedoraproject.org/rpms/python3/pull-request/75 pull
request with the implementation].
** Go through the packages manually invoking <code>setup.py</code> to
build extension modules and provide fixes. Update to
<code>%py3_build</code> where possible, set the flags manually
otherwise.
* Other developers: Are encouraged to switch to
<code>%py3_build</code> (but they don't need to do anything, this is
not a System Wide Change).
* Release engineering: [https://pagure.io/releng/issue/8027 #8027]
(mass rebuild not needed, no releng impact anticipated).
* Policies and guidelines: already in place.
== Upgrade/compatibility impact ==
Not anticipated. Extension modules (built for the same Python version)
are compatible with the interpreter with or without the removed flags
back and forth.
== How To Test ==
=== For users (Python developers) ===
# build your favorite Python extension module in venv or outside venv
with <code>setup.py build</code> or <code>setup.py build_ext</code>
# observe the used flags and check annobin and <code>-specs</code>
'''are not there''', report bugs for {{package|python3}} otherwise
(and block our tracking bug)
# check if the extension works as expected
=== For packagers (Fedora contributors) ===
# build your favorite RPM package with Python extension module
# observe the used flags and check annobin and <code>-specs</code>
'''are there''', report bugs for that package otherwise (and block our
tracking bug)
# check if the package works as expected
== User Experience ==
See '''Benefit to Fedora''' above.
== Dependencies ==
Changes in {{package|redhat-rpm-config}} are already done.
We will check dependent Fedora packages.
== Contingency Plan ==
* Contingency mechanism: Change owners can revert the change at any
point. If removing the {{package|redhat-rpm-config}} dependency turns
out to be problematic, it can be added back until the problem is
fixed.
* Contingency deadline: final freeze
* Blocks release? No
* Blocks product? None
== Documentation ==
This page is the documentation.
== Release Notes ==
TBD.
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
_______________________________________________
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
