[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 7e44dc0ba768451f287a541cd1c85f7d87a41561 by Christian Heimes in 
branch 'main':
bpo-45573: Move mandatory core modules to Modules/Setup.bootstrap (GH-29616)
https://github.com/python/cpython/commit/7e44dc0ba768451f287a541cd1c85f7d87a41561


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:

Erlend implemented pkg-config for sqlite3 in bpo-45774. Thanks to his tireless 
effort we now have a blue print how to port other checks to optional pkg-config 
lookup.

--
superseder:  -> Detect SQLite in configure.ac

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:


New changeset e4bb22fabbea72776b377733047e0f32e0388059 by Christian Heimes in 
branch 'main':
bpo-45573: Add Modules/Setup.stdlib with conditional modules (GH-29615)
https://github.com/python/cpython/commit/e4bb22fabbea72776b377733047e0f32e0388059


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +27852
pull_request: https://github.com/python/cpython/pull/29616

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +27851
pull_request: https://github.com/python/cpython/pull/29615

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 5275e59c0c1b26226608e6c7c2548c26192d6575 by Christian Heimes in 
branch 'main':
bpo-45573: check for ossaudiodev in configure (GH-29614)
https://github.com/python/cpython/commit/5275e59c0c1b26226608e6c7c2548c26192d6575


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +27850
pull_request: https://github.com/python/cpython/pull/29614

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 25ecc040d007a55e4b5c30fa739054b52c1aacac by Christian Heimes in 
branch 'main':
bpo-45573: Introduce extension module flags in Makefile (GH-29594)
https://github.com/python/cpython/commit/25ecc040d007a55e4b5c30fa739054b52c1aacac


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +27837
pull_request: https://github.com/python/cpython/pull/29594

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Brett Cannon]

Brett Cannon  added the comment:

> Is traditional cross-compiling work still useful these days anyway?

Yes for things like WebAssembly where there is no equivalent of an emulated CPU 
under QEMU.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:


New changeset c3997865f24d9491318b401ae8e46d27332aca25 by Christian Heimes in 
branch 'main':
bpo-45573: Use Makefile's dependencies in setup.py (GH-29559)
https://github.com/python/cpython/commit/c3997865f24d9491318b401ae8e46d27332aca25


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +27807
pull_request: https://github.com/python/cpython/pull/29559

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +27784
pull_request: https://github.com/python/cpython/pull/29534

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:

gdbmmodule and dbmmodule need special treatment anyway. macOS has dbm-API build 
into libc. Linux has either libgdbm_compat, libndbm, or libdb. The 
--with-dbmliborder makes it even more interesting. Users can override in which 
order they want to probe for gdbm, ndbm, and libdb. We have to keep the 
"manual" library and header checks.

Cross-compilation with pkg-config uses a trivial wrapper script, 
https://autotools.io/pkgconfig/cross-compiling.html . The build system has to 
create and provide a script with correct sysroot setting.

Is traditional cross-compiling work still useful these days anyway? Emulation 
has made big leaps in the last decade. Emulated cross compiling with qemu has 
become widespread. It also has the big advantage that you can run the test 
suite on the emulated hardware and verify that your binaries work. AFAIK 
Fedora's build system uses qemu for a bunch of hardware targets.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Ned Deily]

Ned Deily  added the comment:

SGTM,2

This all sounds great. I think a goal here should be to remove all header and 
lib file searching from setup.py as that has always been a bug magnet. Perhaps 
one workaround for those libs that don't (yet) provide .pc files would be for 
us to supply reasonable default .pc for them, if possible, so that most 
builders wouldn't have to set the MODULE_xxx_*FLAGS variables while still 
removing the header/lib file searching from setup.py? Also it would be great to 
document how using pkg-config works in generic cross-compiling cases; we should 
strive to make at least the most common cases just work with minimal tweaking 
of configure arguments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Brett Cannon]

Brett Cannon  added the comment:

SGTM!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Erlend E. Aasland]

Change by Erlend E. Aasland :


--
nosy: +erlendaasland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:

We can detect majority of our dependencies with pkg-config. The use of 
pkg-config has some benefits:

* Distro's provide the .pc files in their -dev / -devel packages. The presence 
of a .pc file indicates that all development dependencies are available.

* pkg-config does the right thing for non-standard include and libraries 
directories as well as multiarch builds. In case a library or header is not on 
the default search path, pkg-config returns necessary -I and -L flags.

* At least the pkgconf implementation of pkg-config standard search /usr/local 
and ~/.local/ directories for .pc files. Cases like 
https://github.com/python/cpython/pull/29507 are handled correctly. On FreeBSD 
"pkgconf sqlite3 --cflags --libs" returns "-I/usr/local/include 
-L/usr/local/lib -lsqlite3".

* pkg-config understands dependencies. For example "pkg-config --libs tk" 
returns linker flags for TK *and* TCL.

* pkg-config can check for module version, e.g. "pkg-config sqlite3 
--atleast-version=3.7.15"


pkg-config modules:

  readline, libedit
  ncursesw, ncurses, panel, tinfo
  sqlite3
  zlib
  bzip2
  liblzma
  expat
  uuid (Linux's util-linux uuid)
  libffi
  libnsl, libtirpc
  libcrypt
  tcl, tk
  openssl, libssl, libcrypto
  
modules / libraries without pkg-config modules:

  decimal: libmpdec
  gdbm: gdbm
  dbm: gdbm_compat, ndbm, libdb (bdb)


To simplify use of flags in Modules/Setup, I propose to add two make variables 
for each module that needs cflags and ldflags:

  f"MODULE_{ext.name.upper()}_CFLAGS"
  f"MODULE_{ext.name.upper()}_LDFLAGS"

e.g. for the _ssl module:

  MODULE__SSL_CFLAGS=
  MODULE__SSL_LDFLAGS=-lssl -lcrypto

Then use the flags from Makefile in setup.py:

def update_extension_flags(self, ext):
name = ext.name.upper()
cflags = sysconfig.get_config_var(f"MODULE_{name}_CFLAGS")
if cflags:
ext.extra_compile_args.extend(shlex.split(cflags))
ldflags = sysconfig.get_config_var(f"MODULE_{name}_LDFLAGS")
if ldflags:
ext.extra_link_args.extend(shlex.split(ldflags))
return ext

Finally update Modules/makesetup to use the new variables, too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Ned Deily]

Change by Ned Deily :


--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:

$ find build/lib.linux-x86_64-3.11/ -name '*.so' | sort
build/lib.linux-x86_64-3.11/_ctypes.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_ctypes_test.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_dbm.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_decimal.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_gdbm.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_multiprocessing.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_posixshmem.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_sqlite3.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_testcapi.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_tkinter.cpython-311-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.11/_uuid.cpython-311-x86_64-linux-gnu.so

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Christian Heimes  added the comment:

I added some whitespace to Modules/Setup. All lines starting with ``#([a-z_])`` 
compile cleanly on Fedora 34 with all dependencies available. dbm, tkinter, and 
sqlite are missing or untested.


$ sed -E -i 's/^#([a-z_])/\1/g' Modules/Setup
$ make
...
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc   _asyncio   _bisect 
_blake2_bz2   _codecs_cn  
_codecs_hk _codecs_iso2022_codecs_jp  
_codecs_kr _codecs_tw _contextvars
_crypt _csv   _curses 
_curses_panel  _datetime  _elementtree
_hashlib   _heapq _json   
_lsprof_lzma  _md5
_multibytecodec_opcode_pickle 
_posixsubprocess   _queue _random 
_sha1  _sha256_sha3   
_sha512_socket_ssl
_statistics_struct_testbuffer 
_testimportmultiple   _testinternalcapi   _testmultiphase 
_typing_xxsubinterpreters   _xxtestfuzz 
_zoneinfo  array  audioop 
binascii   cmath  fcntl   
grpmath   mmap
nisossaudiodevpwd 
pyexpatreadline   resource
select spwd   syslog  
termiostime   unicodedata 
xxlimited  xxlimited_35   zlib   

$  ./python -c "import sys; print(sys.builtin_module_names)"
('_abc', '_ast', '_asyncio', '_bisect', '_blake2', '_bz2', '_codecs', 
'_codecs_cn', '_codecs_hk', '_codecs_iso2022', '_codecs_jp', '_codecs_kr', 
'_codecs_tw', '_collections', '_contextvars', '_crypt', '_csv', '_curses', 
'_curses_panel', '_datetime', '_elementtree', '_functools', '_hashlib', 
'_heapq', '_imp', '_io', '_json', '_locale', '_lsprof', '_lzma', '_md5', 
'_multibytecodec', '_opcode', '_operator', '_pickle', '_posixsubprocess', 
'_queue', '_random', '_sha1', '_sha256', '_sha3', '_sha512', '_signal', 
'_socket', '_sre', '_ssl', '_stat', '_statistics', '_string', '_struct', 
'_symtable', '_testbuffer', '_testimportmultiple', '_testinternalcapi', 
'_testmultiphase', '_thread', '_tokenize', '_tracemalloc', '_typing', 
'_warnings', '_weakref', '_xxsubinterpreters', '_xxtestfuzz', '_zoneinfo', 
'array', 'atexit', 'audioop', 'binascii', 'builtins', 'cmath', 'errno', 
'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'mmap', 
'nis', 'ossaudiodev', 'posix', 'pwd', 'pyexpat'
 , 'readline', 'resource', 'select', 'spwd', 'sys', 'syslog', 'termios', 
'time', 'unicodedata', 'xx', 'xxlimited', 'xxlimited_35', 'xxsubtype', 'zlib')
$ ldd python
linux-vdso.so.1 (0x7fffa1f4)
libpthread.so.0 => /lib64/libpthread.so.0 (0x7f6032fe2000)
libdl.so.2 => /lib64/libdl.so.2 (0x7f6032fdb000)
libutil.so.1 => /lib64/libutil.so.1 (0x7f6032fd6000)
libm.so.6 => /lib64/libm.so.6 (0x7f6032e92000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x7f6032e58000)
libnsl.so.2 => /lib64/libnsl.so.2 (0x7f6032e3c000)
libtirpc.so.3 => /lib64/libtirpc.so.3 (0x7f6032e0a000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x7f6032df7000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x7f6032dcb000)
libz.so.1 => /lib64/libz.so.1 (0x7f6032db1000)
libreadline.so.8 => /lib64/libreadline.so.8 (0x7f6032d59000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x7f6032cbc000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x7f60329cc000)
libncursesw.so.6 => /lib64/libncursesw.so.6 (0x7f603298d000)
libtinfo.so.6 => /lib64/libtinfo.so.6 (0x7f603295e000)
libpanel.so.6 => /lib64/libpanel.so.6 (0x7f6032958000)
libc.so.6 => /lib64/libc.so.6 (0x7f6032789000)
/lib64/ld-linux-x86-64.so.2 (0x7f6033029000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x7f6032732000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x7f6032652000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x7f603263a000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x7f6032633000)
libncurses.so.6 => /lib64/libncurses.so.6 (0x7f6032606000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x7f60325f5000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x7f60325ec000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x7f60325d2000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x7f60325a6000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x7f603250f000)

--
nosy: +brett.cannon, twouters

___
Python tracker 

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

Change by Christian Heimes :


--
keywords: +patch
pull_requests: +27438
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/29164

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45573] Use pkg-config autoconf macros to detect flags for Modules/Setup

[Christian Heimes]

New submission from Christian Heimes :

pkg-config [1] is a standard tool on Linux and other platforms to detect 
presence of dependencies as well as to figure out which compiler and linker 
flags they require. Development packages provide a .pc file, e.g. ncurses 
provides a ncursesw.pc file.

$ pkg-config --libs --cflags ncursesw
-D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -lncursesw -ltinfo

I propose to use a modified version of pkg-config's 
PKG_HAVE_DEFINE_WITH_MODULES macro in our configure script. On succss the 
modified macro defines:

* HAVE_FOO=1
* FOO_CFLAGS="some compile flags"
* FOO_LIBS="some library flags for linker"

On error, it sets nothing and does not cause configure to stop with an error.

The macro also allows users to override flags by setting FOO_CFLAGS and 
FOO_LIBS env vars. HAVE_FOO is added to pyconfig.h. The FOO_CFLAGS/LIBS are 
added to Makefile, from where it can be consumed by Modules/Setup.

Eventually Python could use the flags in setup.py, too. For now I would like to 
start with Modules/Setup. It is only used by some power users and has less risk 
of breaking the setup of beginners.

[1] https://www.freedesktop.org/wiki/Software/pkg-config/

--
assignee: christian.heimes
components: Build
messages: 404781
nosy: christian.heimes
priority: normal
severity: normal
status: open
title: Use pkg-config autoconf macros to detect flags for Modules/Setup
type: enhancement
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com