Re: gnulib-tool sh+py mismatch when updating Emacs

[Collin Funk]

On 5/4/24 10:54 PM, Paul Eggert wrote:
>> I was using Autoconf 2.72 on my system.
>
> Ah, I am using Autoconf 2.71, which is what's in /usr/bin/autoconf on Fedora 
> 40. It comes from the autoconf-2.71-10.fc40.noarch package, which is the 
> current version for Fedora 40.
> 
> Occasionlly I use bleeding-edge autoconf (even past 2.72) but that's only to 
> test Autoconf, not other packages.

When Bruno was creating the test suite for gnulib-tool.py we had some
issues with our gperf and bison --version's being different leading to
different files being generated.

I probably installed Autoconf 2.72 because I expected different
versions there to cause issues.

I found the commit where the sorting in that file changed:


https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=c2ab755698db245898a4cc89149eb5df256e4bd0

>> -diff -r $diff_options --exclude=__pycache__ -q . "$tmp" >/dev/null 
>> ||
>> +diff -r $diff_options --exclude=__pycache__ 
>> --exclude=autom4te.cache -q . "$tmp" >/dev/null ||
> 
> Looks good to me, thanks. (I didn't bother testing it on my old slow machine.)

I've applied the attached patch.

CollinFrom 987535a15d4d2902818661feb6d6b363e4d7af2b Mon Sep 17 00:00:00 2001
From: Collin Funk 
Date: Sat, 4 May 2024 23:46:02 -0700
Subject: [PATCH] gnulib-tool: Ignore autom4te.cache when using
 GNULIB_TOOL_IMPL=sh+py.

Reported by Paul Eggert in:
.

* gnulib-tool: Don't compare the autom4te.cache directory since requests
are not sorted in Autoconf version 2.71 and below.
---
 ChangeLog   | 8 
 gnulib-tool | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index 4b5b9bad06..f8e20e06c6 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2024-05-04  Collin Funk  
+
+	gnulib-tool: Ignore autom4te.cache when using GNULIB_TOOL_IMPL=sh+py.
+	Reported by Paul Eggert in:
+	.
+	* gnulib-tool: Don't compare the autom4te.cache directory since requests
+	are not sorted in Autoconf version 2.71 and below.
+
 2024-05-04  Bruno Haible  
 
 	readutmp, boot-time: Work around a Cygwin 3.5.3 bug.
diff --git a/gnulib-tool b/gnulib-tool
index 56c4473318..789fe916a8 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -209,7 +209,7 @@ case "$GNULIB_TOOL_IMPL" in
 else
   diff_options=
 fi
-diff -r $diff_options --exclude=__pycache__ -q . "$tmp" >/dev/null ||
+diff -r $diff_options --exclude=__pycache__ --exclude=autom4te.cache -q . "$tmp" >/dev/null ||
   func_fatal_error "gnulib-tool.py produced different files than gnulib-tool.sh! Compare `pwd` and $tmp."
 # Compare the two outputs.
 diff -q "$tmp-sh-out" "$tmp-py-out" >/dev/null ||
-- 
2.45.0

Re: gnulib-tool sh+py mismatch when updating Emacs

[Paul Eggert]

On 2024-05-04 16:03, Collin Funk wrote:


I noticed in your emacs/output.0 it says "Generated by GNU Autoconf 2.71".

I was using Autoconf 2.72 on my system.


Ah, I am using Autoconf 2.71, which is what's in /usr/bin/autoconf on 
Fedora 40. It comes from the autoconf-2.71-10.fc40.noarch package, which 
is the current version for Fedora 40.


Occasionlly I use bleeding-edge autoconf (even past 2.72) but that's 
only to test Autoconf, not other packages.




-diff -r $diff_options --exclude=__pycache__ -q . "$tmp" >/dev/null ||
+diff -r $diff_options --exclude=__pycache__ --exclude=autom4te.cache -q . 
"$tmp" >/dev/null ||


Looks good to me, thanks. (I didn't bother testing it on my old slow 
machine.)

Re: gnulib-tool sh+py mismatch when updating Emacs

[Collin Funk]

Hi Paul,

On 5/4/24 3:02 PM, Paul Eggert wrote:
> On 2024-05-04 13:45, Collin Funk wrote:
>> I can't reproduce this (using Fedora 40).
> 
> That's odd, as I just now reproduced it on Fedora 40 x86-64 using the 
> following from-scratch recipe:

I noticed in your emacs/output.0 it says "Generated by GNU Autoconf 2.71".

I was using Autoconf 2.72 on my system. Using my Emacs directory and
Autoconf ./configure'd from the v2.71 tag:

$ LC_ALL=en_US.utf8 GNULIB_TOOL_IMPL=sh+py admin/merge-gnulib
../gnulib/gnulib-tool: *** gnulib-tool.py produced different files than 
gnulib-tool.sh! Compare /home/collin/.local/src/emacs and 
/home/collin/.local/src/glpyR1z7xq.
../gnulib/gnulib-tool: *** Stop.
$ diff -ru /home/collin/.local/src/emacs /home/collin/.local/src/glpyR1z7xq
diff -ru /home/collin/.local/src/emacs/autom4te.cache/requests 
/home/collin/.local/src/glpyR1z7xq/autom4te.cache/requests
--- /home/collin/.local/src/emacs/autom4te.cache/requests   2024-05-04 
15:56:55.496035451 -0700
+++ /home/collin/.local/src/glpyR1z7xq/autom4te.cache/requests  
2024-05-04 15:56:25.973984808 -0700
  @@ -15,69 +15,69 @@
   'configure.ac'
 ],
 {
  -'AM_PROG_CC_C_O' => 1,
  -'AM_PROG_AR' => 1,
  -'AC_CANONICAL_BUILD' => 1,

I'm assuming sometime between 2.71 and 2.72 that file started getting
sorted when it previously didn't?

I think this change should be the proper fix:

diff --git a/gnulib-tool b/gnulib-tool
index 56c4473318..789fe916a8 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -209,7 +209,7 @@ case "$GNULIB_TOOL_IMPL" in
 else
   diff_options=
 fi
-diff -r $diff_options --exclude=__pycache__ -q . "$tmp" >/dev/null ||
+diff -r $diff_options --exclude=__pycache__ --exclude=autom4te.cache 
-q . "$tmp" >/dev/null ||
   func_fatal_error "gnulib-tool.py produced different files than 
gnulib-tool.sh! Compare `pwd` and $tmp."
 # Compare the two outputs.
 diff -q "$tmp-sh-out" "$tmp-py-out" >/dev/null ||

Can you confirm before I apply it?

Collin

Re: gnulib-tool sh+py mismatch when updating Emacs

[Collin Funk]

On 5/4/24 3:02 PM, Paul Eggert wrote:
> The attached build log shows what I got. The last 'grep' command shows the 
> bug, as lib/gnulib.mk.in uses HAVE_OFF64_T without defining it.
> 
> This invocation of admin/merge-gnulib uses a nearly-empty environment and an 
> empty home directory, to lessen any local changes I might have.
> 
> If it helps to debug, you can get a tarball of all the resulting Emacs 
> directory (sans .git subdirectory) temporarily, here:

Thanks for the details!

I'll have a look at it. I downloaded the tarball so no worries here if
you remove it.

Collin

Re: gnulib-tool sh+py mismatch when updating Emacs

[Paul Eggert]

On 2024-05-04 13:45, Collin Funk wrote:

I can't reproduce this (using Fedora 40).


That's odd, as I just now reproduced it on Fedora 40 x86-64 using the 
following from-scratch recipe:


  mkdir new empty 

  empty_home=$PWD/empty 

  cd new 

  git clone git://git.sv.gnu.org/emacs.git 

  (cd emacs && git checkout fd859fbea2e9d13e76db1c5295d9ddd1c5955d83) 

  git clone git://git.sv.gnu.org/gnulib.git 

  (cd gnulib && git checkout fde88b711c9b1df5b142444ac7b0bc2aa8892d3a) 

  cd emacs 

  env -i HOME="$empty_home" PATH=/usr/bin admin/merge-gnulib 



The attached build log shows what I got. The last 'grep' command shows 
the bug, as lib/gnulib.mk.in uses HAVE_OFF64_T without defining it.


This invocation of admin/merge-gnulib uses a nearly-empty environment 
and an empty home directory, to lessen any local changes I might have.


If it helps to debug, you can get a tarball of all the resulting Emacs 
directory (sans .git subdirectory) temporarily, here:


https://www.cs.ucla.edu/~eggert/emacs.tgz$ mkdir new empty
$ empty_home=$PWD/empty
$ cd new
$ git clone git://git.sv.gnu.org/emacs.git
Cloning into 'emacs'...
remote: Counting objects: 1102975, done.
remote: Compressing objects: 100% (198635/198635), done.
remote: Total 1102975 (delta 903465), reused 1098398 (delta 899028)
Receiving objects: 100% (1102975/1102975), 413.08 MiB | 21.93 MiB/s, done.
Resolving deltas: 100% (903465/903465), done.
Updating files: 100% (5266/5266), done.
$ (cd emacs && git checkout fd859fbea2e9d13e76db1c5295d9ddd1c5955d83)
Note: switching to 'fd859fbea2e9d13e76db1c5295d9ddd1c5955d83'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c 

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at fd859fbea2e Allow `letrec` binding without init expression
$ git clone git://git.sv.gnu.org/gnulib.git
Cloning into 'gnulib'...
remote: Counting objects: 295108, done.
remote: Compressing objects: 100% (35985/35985), done.
remote: Total 295108 (delta 261344), reused 292549 (delta 259017)
Receiving objects: 100% (295108/295108), 74.69 MiB | 11.59 MiB/s, done.
Resolving deltas: 100% (261344/261344), done.
$ (cd gnulib && git checkout fde88b711c9b1df5b142444ac7b0bc2aa8892d3a)
Note: switching to 'fde88b711c9b1df5b142444ac7b0bc2aa8892d3a'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c 

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at fde88b711c readutmp, boot-time: Work around a Cygwin 3.5.3 bug.
$ cd emacs
$ env -i HOME="$empty_home" PATH=/usr/bin admin/merge-gnulib
Checking whether you have the necessary tools...
(Read INSTALL.REPO for more details on building Emacs)
Checking for autoconf (need at least version 2.65) ... ok
Your system has the required tools.
Building aclocal.m4 ...
Running 'autoreconf -fi -I m4' ...
Building 'aclocal.m4' in exec ...
Running 'autoreconf -fi' in exec ...
Configuring local git repository...
'.git/config' -> '.git/config.~1~'
git config transfer.fsckObjects 'true'
git config diff.cpp.xfuncname '!^[ 
\t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])
^((::[[:space:]]*)?[A-Za-z_][A-Za-z_0-9]*[[:space:]]*\(.*)$
^((#define[[:space:]]|DEFUN).*)$'
git config diff.elisp.xfuncname 
'^\([^[:space:]]*def[^[:space:]]+[[:space:]]+([^()[:space:]]+)'
git config diff.m4.xfuncname '^((m4_)?define|A._DEFUN(_ONCE)?)\([^),]*'
git config diff.make.xfuncname 
'^([$.[:alnum:]_].*:|[[:alnum:]_]+[[:space:]]*([*:+]?[:?]?|!?)=|define .*)'
git config diff.shell.xfuncname 
'^([[:space:]]*[[:alpha:]_][[:alnum:]_]*[[:space:]]*\(\)|[[:alpha:]_][[:alnum:]_]*=)'
git config diff.texinfo.xfuncname '^@node[[:space:]]+([^,[:space:]][^,]+)'
Installing git hooks...
'build-aux/git-hooks/commit-msg' -> '.git/hooks/commit-msg'
'build-aux/git-hooks/pre-commit' -> '.git/hooks/pre-commit'
'build-aux/git-hooks/prepare-commit-msg' -> '.git/hooks/prepare-commit-msg'
'build-aux/git-hooks/post-commit' -> '.git/hooks/post-commit'
'build-aux/git-hooks/pre-push' -> '.git/hooks/pre-push'
'build-aux/git-hooks/commit-msg-files.awk' -> '.git/hooks/commit-msg-files.awk'
'.git/hooks/applypatch-msg.sample' -> '.git/hooks/applypatch-msg'
'.git/hooks/pre-applypatch.sample' -> '.git/hooks/pre-applypatch'
You can now run './configure'.
Module list with included 

Re: gnulib-tool sh+py mismatch when updating Emacs

[Collin Funk]

Hi Paul,

On 5/4/24 9:03 AM, Paul Eggert wrote:
> Some sort of locale problem? The issue seems to be benign, except that it's a 
> false positive when testing.
> 
> To reproduce it, clone current gnulib (commit 
> fde88b711c9b1df5b142444ac7b0bc2aa8892d3a) and current emacs (commit 
> fd859fbea2e9d13e76db1c5295d9ddd1c5955d83) under the current directory, and 
> then run:

I can't reproduce this (using Fedora 40).

For what it's worth I used Emacs admin/merge-gnulib frequently when I
started working on gnulib-tool.py. Bruno then added it as a test case
in maint-tools. So I can almost guarantee it is safe to use there.

Could the autogen.sh script be messing it up perhaps? Or old build
artifacts? My Emacs directory was clean during run.

Collin

gnulib-tool sh+py mismatch when updating Emacs

[Paul Eggert]

Some sort of locale problem? The issue seems to be benign, except that 
it's a false positive when testing.


To reproduce it, clone current gnulib (commit 
fde88b711c9b1df5b142444ac7b0bc2aa8892d3a) and current emacs (commit 
fd859fbea2e9d13e76db1c5295d9ddd1c5955d83) under the current directory, 
and then run:


cd emacs
LC_ALL=en_US.utf8 GNULIB_TOOL_IMPL=sh+py admin/merge-gnulib

Here's what happens on Ubuntu 23.10 x86-64:

$ GNULIB_TOOL_IMPL=sh+py admin/merge-gnulib
Checking whether you have the necessary tools...
(Read INSTALL.REPO for more details on building Emacs)
Checking for autoconf (need at least version 2.65) ... ok
Your system has the required tools.
Building aclocal.m4 ...
Running 'autoreconf -fi -I m4' ...
Building 'aclocal.m4' in exec ...
Running 'autoreconf -fi' in exec ...
Configuring local git repository...
'.git/config' -> '.git/config.~1~'
git config transfer.fsckObjects 'true'
git config diff.cpp.xfuncname '!^[ 
]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])

^((::[[:space:]]*)?[A-Za-z_][A-Za-z_0-9]*[[:space:]]*\(.*)$
^((#define[[:space:]]|DEFUN).*)$'
git config diff.elisp.xfuncname 
'^\([^[:space:]]*def[^[:space:]]+[[:space:]]+([^()[:space:]]+)'

git config diff.m4.xfuncname '^((m4_)?define|A._DEFUN(_ONCE)?)\([^),]*'
git config diff.make.xfuncname 
'^([$.[:alnum:]_].*:|[[:alnum:]_]+[[:space:]]*([*:+]?[:?]?|!?)=|define .*)'
git config diff.shell.xfuncname 
'^([[:space:]]*[[:alpha:]_][[:alnum:]_]*[[:space:]]*\(\)|[[:alpha:]_][[:alnum:]_]*=)'

git config diff.texinfo.xfuncname '^@node[[:space:]]+([^,[:space:]][^,]+)'
Installing git hooks...
'build-aux/git-hooks/commit-msg' -> '.git/hooks/commit-msg'
'build-aux/git-hooks/pre-commit' -> '.git/hooks/pre-commit'
'build-aux/git-hooks/prepare-commit-msg' -> '.git/hooks/prepare-commit-msg'
'build-aux/git-hooks/post-commit' -> '.git/hooks/post-commit'
'build-aux/git-hooks/pre-push' -> '.git/hooks/pre-push'
'build-aux/git-hooks/commit-msg-files.awk' -> 
'.git/hooks/commit-msg-files.awk'

'.git/hooks/applypatch-msg.sample' -> '.git/hooks/applypatch-msg'
'.git/hooks/pre-applypatch.sample' -> '.git/hooks/pre-applypatch'
You can now run './configure'.
../gnulib/gnulib-tool: *** gnulib-tool.py produced different files than 
gnulib-tool.sh! Compare /home/eggert/src/gnu/d/emacs and 
/home/eggert/src/gnu/d/glpyD3FBYw.

../gnulib/gnulib-tool: *** Stop.


The difference is between autom4te.cache/requests and 
../glpyD3FBYw/autom4te.cache/requests. They're sorted differently:


$ diff -ur ./autom4te.cache/requests ../glpyD3FBYw/autom4te.cache/requests
--- ./autom4te.cache/requests   2024-05-04 08:47:45.397566582 -0700
+++ ../glpyD3FBYw/autom4te.cache/requests	2024-05-04 08:46:41.307448492 
-0700

@@ -16,69 +16,69 @@
 'configure.ac'
   ],
   {
-'_LT_AC_TAGCONFIG' => 1,
-'LT_INIT' => 1,
-'AM_XGETTEXT_OPTION' => 1,
-'m4_sinclude' => 1,
-'_AM_COND_IF' => 1,
-'AM_PROG_MOC' => 1,
+'AM_GNU_GETTEXT' => 1,
+'AC_CANONICAL_BUILD' => 1,
 'AC_CONFIG_AUX_DIR' => 1,
+'AC_PROG_LIBTOOL' => 1,
+'m4_include' => 1,
+'AM_CONDITIONAL' => 1,
+'AM_PROG_MKDIR_P' => 1,
+'_AM_SUBST_NOTMAKE' => 1,
+'AM_EXTRA_RECURSIVE_TARGETS' => 1,
+'AC_DEFINE_TRACE_LITERAL' => 1,
+'AC_FC_FREEFORM' => 1,
+'m4_pattern_forbid' => 1,
+'AM_INIT_AUTOMAKE' => 1,
 'LT_CONFIG_LTDL_DIR' => 1,
 'AC_CANONICAL_SYSTEM' => 1,
-'AC_CANONICAL_BUILD' => 1,
-'_AM_COND_ELSE' => 1,
+'AC_FC_PP_DEFINE' => 1,
+'LT_INIT' => 1,
+'AC_FC_SRCEXT' => 1,
+'AC_LIBSOURCE' => 1,
 'AC_CONFIG_LIBOBJ_DIR' => 1,
-'AM_PROG_CXX_C_O' => 1,
+'AM_PROG_FC_C_O' => 1,
 'AC_CONFIG_HEADERS' => 1,
-'AM_GNU_GETTEXT' => 1,
+'_AM_COND_ELSE' => 1,
+'AC_CONFIG_SUBDIRS' => 1,
+'AM_PROG_CXX_C_O' => 1,
+'m4_sinclude' => 1,
+'AM_MAKEFILE_INCLUDE' => 1,
 'AM_PROG_LIBTOOL' => 1,
+'AC_INIT' => 1,
+'include' => 1,
+'_AM_COND_ENDIF' => 1,
+'AM_POT_TOOLS' => 1,
+'AM_MAINTAINER_MODE' => 1,
+'_LT_AC_TAGCONFIG' => 1,
+'_AM_MAKEFILE_INCLUDE'