Re: python debugging on trunk
On Mon, Sep 26, 2022, at 12:23 PM, Karl Berry wrote: > Is anyone up for debugging some Python-related test failures on > RHEL-based systems? I have access to a RHEL7 system, I know Python, and this sounds much less unpleasant than everything else I'm supposed to be doing today. > I have a suspicion the problem is that on RHEL systems, "python" invokes > "python2" (because that's what the Python maintainers recommend, as I > understand it), but the tests are assuming it invokes "python3". Indeed: $ lsb_release -d Description: Red Hat Enterprise Linux Workstation release 7.9 (Maipo) $ python --version Python 2.7.5 $ python3 --version Python 3.6.8 I have very strong opinions on this topic, having had to deal with a lot of badly-written Python 2 code that, in the worst case, silently corrupts data when run under Python 3. IMNSHO, the bare command name "python" and the absolute pathname "/usr/bin/python" MUST[rfc2119] be *permanently reserved* for the Python 2 interpreter. It's fine for them not to exist at all, but if they exist and they invoke Python 3, that is a *bug* in the distribution. (Yes, there's a PEP that says something different -- IMO it is wrong, its authors probably hadn't encountered the *really* troublesome Python 2 code that I have.) --- RHEL7, unsurprisingly, ships with Autoconf 2.69. Before attempting to build Automake from Git, I built Autoconf 2.71 from the tarball release, installed it in a subdirectory of my homedir, and made that directory come first on PATH. There were no unexpected test failures in Autoconf's testsuite. Other relevant tool versions include: $ perl --version This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi (with 44 registered patches, see perl -V for more detail) $ m4 --version m4 (GNU M4) 1.4.16 $ /bin/sh --version GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu) $ make --version GNU Make 3.82 $ sort --version sort (GNU coreutils) 8.22 $ gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) $ libtool --version libtool (GNU libtool) 2.4.2 I see these Automake testsuite failures: Testsuite summary for GNU Automake 1.16i # TOTAL: 2988 # PASS: 2877 # SKIP: 54 # XFAIL: 39 # FAIL: 18 # XPASS: 0 # ERROR: 0 FAIL: t/canon7 FAIL: t/fort5 FAIL: t/instmany-python FAIL: t/ltinit FAIL: t/nobase-python FAIL: t/pr401b FAIL: t/py-compile-basedir FAIL: t/py-compile-basic FAIL: t/py-compile-destdir FAIL: t/py-compile-option-terminate FAIL: t/python-pr10995 FAIL: t/python-prefix FAIL: t/python-vars FAIL: t/python10 FAIL: t/python12 FAIL: t/python3 FAIL: t/subobj9 FAIL: t/suffix5 That's _most_ of the failures you got, plus some more: +FAIL: t/canon7 +FAIL: t/fort5 +FAIL: t/ltinit +FAIL: t/pr401b +FAIL: t/python10 +FAIL: t/subobj9 +FAIL: t/suffix5 -FAIL: t/subobj-pr13928-more-langs There are only two failure modes: canon7, fort5, ltinit, pr401b, subobj9, suffix5 ERROR: files left in build directory after distclean: ./.__afs I did the build in an AFS volume; these are the AFS equivalent of the NFS "silly rename" hack (apparently it can happen with SMB as well). "make distcleancheck" should ignore these. instmany-python, nobase-python, py-compile-basic, py-compile-destdir, py-compile-option-terminate, pyghon-pr10995, python-prefix, python-vars, python10, python12, python3 AttributeError: 'module' object has no attribute 'implementation' This is coming from this code in lib/py-compile (there are three copies of it): if hasattr(sys.implementation, 'cache_tag'): py_compile.compile(filepath, importlib.util.cache_from_source(filepath), path) else: py_compile.compile(filepath, filepath + 'c', path) sys.implementation was added in Python 3.3 and importlib.util.cache_from_source was added in 3.4. Python 2.7's importlib _only_ has 'import_module'. The appended patch should address both issues. Note that I have only tested it with Python 2.7 and 3.6. It _may_ not be correct for Python in the 3.0--3.3 (inclusive) range; I cannot conveniently test that. zw * lib/am/distdir.am (distcleancheck_listfiles): Filter out "silly rename" files, unavoidably created by deleting files that are still open in some process on various network file systems. * lib/py-compile: Test directly for availability of importlib.util.cache_from_source. Untangle logic for when to generate -O and -OO bytecode. Reformat embedded Python fragments. diff --git a/lib/am/distdir.am b/lib/am/distdir.am index 0f591266d..b8d90e572 100644 --- a/lib/am/distdir.am +++ b/lib/am/distdir.am @@ -568,7 +568,9 @@ distuninstallcheck: ## Define distcleancheck_listfiles and distcleancheck separately ## from distcheck, so that they can be overridden by the user. .PHONY: distcleancheck -distcleancheck_listfiles = find . -type f -print +distcleancheck_listfiles = \ + find . \( -type f -a \! \ + \( -name .nfs* -o -name .smb* -o -name .__afs* \) \) -print distcleancheck:
bug#58102: [PATCH] Do not use -Q when calculating Emacs site-lisp directory
This change ensures consistency and compatibility for both GNU Emacs
and XEmacs because:
* XEmacs doesn't support -Q and ignores it; and
* the different releases of GNU Emacs handle it differently, or not at
all.
With GNU Emacs, '-Q' was added in 22.1, and specifying it on earlier
versions causes an immediate abort. Also, the meaning of '-Q' changes
in 24.1 where it also implies/uses a new '--no-site-lisp' option.
This new option means it won't add the site-lisp directories to
`load-path', but they need to be there because `am_cv_lispdir' is
looking for them.
Tested on GNU Emacs 21.4, 24.3, 27.2, 28.2, and XEmacs 21.5.
From this testing I think the existing usage of '--no-site-file'
is good to stay as it's the one mentioned in the current GNU Emacs
manual and recognized back to at least 19.34. Also, whilst the
XEmacs manual and '--help' mention "-no-site-file" it also
supports "--no-site-file".From 38e5c77356f56ac1c1bd50c9ced5bb7607c3ebbf Mon Sep 17 00:00:00 2001
From: Richard Hopkins <>
Date: Mon, 26 Sep 2022 21:25:06 +0100
Subject: [PATCH] Do not use -Q when calculating Emacs site-lisp directory
* m4/lispdir.m4 (am_cv_lispdir): Remove '-Q' argument.
* doc/automake.texi (Hard-Coded Install Paths): Remove '-Q'
argument.
This change ensures consistency and compatibility for both GNU Emacs
and XEmacs because:
* XEmacs doesn't support -Q and ignores it; and
* the different releases of GNU Emacs handle it differently, or not at
all.
With GNU Emacs, '-Q' was added in 22.1, and specifying it on earlier
versions causes an immediate abort. Also, the meaning of '-Q' changes
in 24.1 where it also implies/uses a new '--no-site-lisp' option.
This new option means it won't add the site-lisp directories to
`load-path', but they need to be there because `am_cv_lispdir' is
looking for them.
---
doc/automake.texi | 2 +-
m4/lispdir.m4 | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/automake.texi b/doc/automake.texi
index 21c49645e..444740bb6 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -13350,7 +13350,7 @@ instance, here is how @code{AM_PATH_LISPDIR} (@pxref{Emacs Lisp})
computes @samp{$(lispdir)}:
@example
-$EMACS -batch -Q --no-site-file -eval '(while load-path
+$EMACS -batch --no-site-file -eval '(while load-path
(princ (concat (car load-path) "\n"))
(setq load-path (cdr load-path)))' >conftest.out
lispdir=`sed -n
diff --git a/m4/lispdir.m4 b/m4/lispdir.m4
index 6f9b89d2c..cec4b3554 100644
--- a/m4/lispdir.m4
+++ b/m4/lispdir.m4
@@ -34,7 +34,7 @@ AC_DEFUN([AM_PATH_LISPDIR],
# which is non-obvious for non-emacs users.
# Redirecting /dev/null should help a bit; pity we can't detect "broken"
# emacsen earlier and avoid running this altogether.
- AC_RUN_LOG([$EMACS -batch -Q --no-site-file -eval '(while load-path (princ (concat (car load-path) "\n")) (setq load-path (cdr load-path)))' conftest.out])
+ AC_RUN_LOG([$EMACS -batch --no-site-file -eval '(while load-path (princ (concat (car load-path) "\n")) (setq load-path (cdr load-path)))' conftest.out])
am_cv_lispdir=`sed -n \
-e 's,/$,,' \
-e '/.*\/lib\/x*emacs\/site-lisp$/{s,.*/lib/\(x*emacs/site-lisp\)$,${libdir}/\1,;p;q;}' \
--
2.37.3
bug#58025: [PATCH] Ensure `byte-compile-dest-file-function' is used
Hi Richard - I installed the -l bytecomp patch you sent (copied below). It seems safe, and good in any case. Thanks. If there are changes to make in the -Q / -q area, let's address those separately. (I'll close this bug, I guess, but fine to keep discussing wherever.) I don't think Automake uses -Q now? Since the release of 1.16.5, there has been one change already, to pass --no-site-file (as you can see below; I'll attach the current lisp.am for possible convenience). From your research, I guess that should be changed to -no-site-file (one hyphen)? Where is it that the double-hyphen --no... fails? In general, it is definitely necessary to support Emacs 21 (I use it :). I'm not sure if we absolutely have to support 18 or 19, but when possible, it is certainly desirable. --thanks, karl. * lib/am/lisp.am (.el.elc): Require the bytecomp library so byte-compile-dest-file-function can be used when available. diff --git a/lib/am/lisp.am b/lib/am/lisp.am index 6395ef389..500e2c530 100644 --- a/lib/am/lisp.am +++ b/lib/am/lisp.am @@ -41,6 +41,7 @@ endif %?INSTALL% $(EMACS) --batch --no-site-file \ $(AM_ELCFLAGS) $(ELCFLAGS) \ $$am__subdir_includes -L $(builddir) -L $(srcdir) \ + -l bytecomp \ --eval '$(am__emacs_byte_compile_setup)' \ -f batch-byte-compile '$<'; \ else :; fi lisp.am Description: Binary data
bug#58026: [PATCH] Emacs .elc byte compilation now respects silent rules
Emacs byte compilation now respects silent build rules (e.g. make V=0). i.e. use AM_V_GEN to show this instead GEN bar.elc Thanks. I applied the patch.
python debugging on trunk
Is anyone up for debugging some Python-related test failures on RHEL-based systems? Mike Vapier from gentoo made many improvements to the Python support. (Mike, if you're still out there, would love to hear back.) Unfortunately, the end result is that 13 tests (listed below) now fail for me on Alma Linux 8 (and, presumably, Rocky 8; and I believe CentOS 7) with the trunk automake. My appetite for (and experience in) debugging Python problems is pretty low. But we can't make another release until this is fixed. So, I wonder if there's anyone else who's interested in figuring this out. I can probably figure out access to a system where it fails, if that's a problem. I have a suspicion the problem is that on RHEL systems, "python" invokes "python2" (because that's what the Python maintainers recommend, as I understand it), but the tests are assuming it invokes "python3". Knowing Python, I feel pretty sure that there will be many complications in dealing with this. Thanks, Karl FAIL: t/instmany-python.sh FAIL: t/nobase-python.sh FAIL: t/py-compile-basic.sh FAIL: t/py-compile-basedir.sh FAIL: t/py-compile-destdir.sh FAIL: t/py-compile-option-terminate.sh FAIL: t/python3.sh FAIL: t/python12.sh FAIL: t/python-prefix.sh FAIL: t/python-pr10995.sh FAIL: t/python-vars.sh FAIL: t/subobj-pr13928-more-langs.sh For example, test-suite.log shows this for the nobase-python.sh failure: .. Byte-compiling python modules... one.py Traceback (most recent call last): File "", line 13, in AttributeError: 'module' object has no attribute 'implementation' make: *** [Makefile:304: install-myPYTHON] Error 1
bug#58025: [PATCH] Ensure `byte-compile-dest-file-function' is used
On Sat, Sep 24, 2022, at 5:45 AM, Richard Hopkins wrote: > On 2022-09-23 16:15, Zack Weinberg wrote: >> Thank you for the patch. Are you able to test it with a version of >> GNU Emacs older than 23.2? I see that you tested it with XEmacs 21, >> but as I recall there were quite substantial differences between >> XEmacs 21 and GNU Emacs of similar vintage. > > I've now managed to test this on Emacs 21.4.1 (Slackware 12.0) and > the byte compilation works - loading bytecomp is fine, and > `byte-compile-dest-file' is defined as expected. The other patch > to respect silent rules also works on 21.4.1. > > If we do need to support that far back I will investigate the "-Q" > / "--no-site-file" handling to improve compatibility across the > board. I'm not an official maintainer for Automake, but I think we probably don't have to worry about Emacs any older than v21. Would anyone else like to express an opinion? > "-Q" will error before GNU Emacs 22, and is ignored on XEmacs. > > "-no-site-file" (single hypen) should be used instead of > "--no-site-file" as it works on all of them. > > "-Q" also shouldn't be specified on later GNU Emacs as it affects > the result of `am_cv_lispdir' calculation due to excluding > site lisp directories from `load-path' which it's trying to find. > This is because "-Q" also adds "--no-site-lisp" in later GNU Emacs. > > So, the plan will be to not use "-Q" and to use "-q -no-site-file" > instead. This sounds like it would be a worthwhile change regardless of where we decide to draw the line on supporting old versions of Emacs. zw
