Re: gnulib-tool problem with off64_t and Emacs

[Bruno Haible]

Paul Eggert wrote:
> I had problems updating Emacs to use gnulib commit 
> fde88b711c9b1df5b142444ac7b0bc2aa8892d3a along with emacs commit 
> fd859fbea2e9d13e76db1c5295d9ddd1c5955d83 (these are the same commits as 
> I mentioned earlier today). I reproduced it like this:
> 
> cd emacs
> admin/merge-gnulib
> 
> The resulting lib/gnulib.mk.in had a line:
> 
> -e 's|@''HAVE_OFF64_T''@|$(HAVE_OFF64_T)|g' \
> 
> but there was no "HAVE_OFF64_T = @HAVE_OFF64_T@" line

I reproduce it.

But when I run
  admin/merge-gnulib
a second time, it adds the line:

$ diff -u lib/gnulib.mk.in~ lib/gnulib.mk.in
--- lib/gnulib.mk.in~   2024-05-05 15:17:51.168952913 +0200
+++ lib/gnulib.mk.in2024-05-05 15:28:29.973871297 +0200
@@ -808,6 +808,7 @@
 HAVE_MODULES = @HAVE_MODULES@
 HAVE_NANOSLEEP = @HAVE_NANOSLEEP@
 HAVE_NATIVE_COMP = @HAVE_NATIVE_COMP@
+HAVE_OFF64_T = @HAVE_OFF64_T@
 HAVE_OPENAT = @HAVE_OPENAT@
 HAVE_OPENDIR = @HAVE_OPENDIR@
 HAVE_OS_H = @HAVE_OS_H@

The order in which gnulib-tool creates the files seems to be correct:
lib/gnulib.mk.in is generated last.

  ...
  Replacing file m4/nstrftime.m4 (non-gnulib code backed up in 
m4/nstrftime.m4~) !!
  Copying file m4/off64_t.m4
  Replacing file m4/off_t.m4 (non-gnulib code backed up in m4/off_t.m4~) !!
  ...
  Replacing file m4/xattr.m4 (non-gnulib code backed up in m4/xattr.m4~) !!
  Replacing file m4/zzgnulib.m4 (non-gnulib code backed up in m4/zzgnulib.m4~) 
!!
  Creating m4/gnulib-cache.m4
  Updating m4/gnulib-comp.m4 (backup in m4/gnulib-comp.m4~)
  Updating lib/gnulib.mk.in (backup in lib/gnulib.mk.in~)
  Finished.

The problem is that the new file m4/off64_t.m4 is not yet reflected
in aclocal.m4 at the moment
  autoconf -t ...
is run. This patch fixes it.

I'm not including the fix in the old gnulib-tool.sh. Please document in
emacs/admin/merge-gnulib that Python 3 is now needed as a prerequisite.


2024-05-05  Bruno Haible  

gnulib-tool.py: Regenerate aclocal.m4 before using 'autoconf -t ...'.
Reported by Paul Eggert in
.
* pygnulib/GLImport.py (GLImport): New field m4dirs.
(GLImport.__init__): Accept an additional m4dirs argument.
(GLImport.execute): Regenerate aclocal.m4 before creating the library
Makefile.
* pygnulib/main.py (main): Pass the guessed_m4dirs to GLImport.

diff --git a/pygnulib/GLImport.py b/pygnulib/GLImport.py
index a11da0e63d..e67cbbe5a2 100644
--- a/pygnulib/GLImport.py
+++ b/pygnulib/GLImport.py
@@ -23,6 +23,7 @@
 import subprocess as sp
 from .constants import (
 DIRS,
+UTILS,
 MODES,
 TESTS,
 cleaner,
@@ -61,6 +62,7 @@ class GLImport:
 is a very good choice.'''
 
 mode: int
+m4dirs: list[str]
 config: GLConfig
 cache: GLConfig
 emitter: GLEmiter
@@ -68,19 +70,23 @@ class GLImport:
 moduletable: GLModuleTable
 makefiletable: GLMakefileTable
 
-def __init__(self, config: GLConfig, mode: int) -> None:
+def __init__(self, config: GLConfig, mode: int, m4dirs: list[str]) -> None:
 '''Create GLImport instance.
-The first variable, mode, must be one of the values of the MODES dict
-object, which is accessible from constants module. The second one, 
config,
-must be a GLConfig object.'''
+config - must be a GLConfig object.
+mode - must be one of the values of the constants.MODES values.
+m4dirs - list of all directories that contain relevant .m4 files.'''
 if type(config) is not GLConfig:
 raise TypeError('config must have GLConfig type, not %s'
 % repr(config))
-if type(mode) is int and MODES['import'] <= mode <= MODES['update']:
-self.mode = mode
-else:  # if mode is not int or is not 0-3
+if not (type(mode) is int and MODES['import'] <= mode <= 
MODES['update']):
 raise TypeError('mode must be 0 <= mode <= 3, not %s'
 % repr(mode))
+if type(m4dirs) is not list:
+raise TypeError('m4dirs must be a list of strings, not %s'
+% repr(m4dirs))
+
+self.mode = mode
+self.m4dirs = m4dirs
 
 # config contains the configuration, as specified through command-line
 # parameters.
@@ -1209,6 +1215,17 @@ def execute(self, filetable: GLFileTable, transformers: 
dict[str, tuple[re.Patte
 if os.path.isfile(tmpfile):
 os.remove(tmpfile)
 
+if self.config['gnu_make']:
+# Regenerate aclocal.m4.
+# This is needed because the next step may run 'autoconf -t' and
+# the preceding steps may have added new *.m4 files (which need to
+# be reflected in aclocal.m4 before 'autoconf -t' is run).
+aclocal_args = []
+for dir in self.m4dirs:
+aclocal_args.append('-I')
+

Re: gnulib-tool problem with off64_t and Emacs

[Paul Eggert]

On 2024-05-04 10:43, Bruno Haible wrote:

but there was no "HAVE_OFF64_T = @HAVE_OFF64_T@" line

Strange. The AC_SUBST([HAVE_OFF64_T]) in m4/off64_t.m4 should produce
such a line.


My thought as well.



The fragile hackery is only needed because you've hit a bug


Yes. To be fair it could well be an Emacs bug. Whatever it is, it's 
currently a hassle.




Do you see the 'sys_types' module enabled? Or maybe it is only
conditionally enabled?


Yes, there is a sys_types section in lib/gnulib.mk.in, and yes, it is 
conditionally enabled. It's preceded by this:


  ## begin gnulib module sys_types
  ifeq (,$(OMIT_GNULIB_MODULE_sys_types))

The ifeq succeeds and sys/types.h is being built.

I can reproduce the problem in fresh emacs and gnulib checkout siblings 
(same commits as before) by running the following in the emacs 
directory. This invokes gnulib-tool directly instead of indirectly via 
admin/merge-gnulib:


  ../gnulib/gnulib-tool --dir= --conditional-dependencies --import 
--no-changelog --no-vc-files --gnu-make --makefile-name=gnulib.mk.in 
--avoid=access --avoid=btowc --avoid=chmod --avoid=close 
--avoid=crypto/af_alg --avoid=dup --avoid=fchdir --avoid=fstat 
--avoid=iswblank --avoid=iswctype --avoid=iswdigit --avoid=iswxdigit 
--avoid=langinfo --avoid=localename-unsafe-limited --avoid=lock 
--avoid=mbrtowc --avoid=mbsinit --avoid=memchr --avoid=mkdir 
--avoid=msvc-inval --avoid=msvc-nothrow --avoid=nl_langinfo 
--avoid=openat-die --avoid=opendir --avoid=pthread-h --avoid=raise 
--avoid=save-cwd --avoid=select --avoid=setenv --avoid=sigprocmask 
--avoid=stat --avoid=stdarg --avoid=threadlib --avoid=tzset 
--avoid=unsetenv --avoid=utime --avoid=utime-h --avoid=wchar 
--avoid=wcrtomb --avoid=wctype --avoid=wctype-h alignasof alloca-opt 
binary-io boot-time byteswap c-ctype c-strcase canonicalize-lgpl 
careadlinkat close-stream copy-file-range count-leading-zeros 
count-one-bits count-trailing-zeros crypto/md5 crypto/md5-buffer 
crypto/sha1-buffer crypto/sha256-buffer crypto/sha512-buffer d-type 
diffseq double-slash-root dtoastr dtotimespec dup2 environ execinfo 
faccessat fchmodat fcntl fcntl-h fdopendir file-has-acl filemode 
filename filevercmp flexmember fpieee free-posix fstatat fsusage fsync 
futimens getline getloadavg getopt-gnu getrandom gettime gettimeofday 
gitlog-to-changelog ieee754-h ignore-value intprops largefile libgmp 
lstat manywarnings memmem-simple mempcpy memrchr memset_explicit minmax 
mkostemp mktime nanosleep nproc nstrftime pathmax pipe2 pselect 
pthread_sigmask qcopy-acl readlink readlinkat regex sig2str sigdescr_np 
socklen stat-time std-gnu11 stdbool stdckdint stddef stdio stpcpy 
strnlen strnlen strtoimax symlink sys_stat sys_time tempname time-h 
time_r time_rz timegm timer-time timespec-add timespec-sub 
update-copyright unlocked-io utimensat vla warnings year2038

Re: gnulib-tool problem with off64_t and Emacs

[Bruno Haible]

Hi Paul,

> but there was no "HAVE_OFF64_T = @HAVE_OFF64_T@" line

Strange. The AC_SUBST([HAVE_OFF64_T]) in m4/off64_t.m4 should produce
such a line.

Do you see the 'sys_types' module enabled? Or maybe it is only
conditionally enabled?

> It strikes me that apps like Emacs and coreutils don't need (and 
> shouldn't use) off64_t which was merely intended as a transitional type 
> in the 1990s, so there should be some way for them to disable the 
> off64_t business

off64_t is defined on glibc in , , ,
, and since one of Gnulib's principles is to provide a glibc-like
API portably, we now ensure that off64_t is defined in these 4 header files
— regardless whether the future z*printf functions in  will be
requested by the package.

I hope that coreutils will make good use of the z*printf functions once
they exist.

> without the abovementioned sort of fragile hackery.

The fragile hackery is only needed because you've hit a bug, that I haven't
had time to investigate so far.

Bruno

gnulib-tool problem with off64_t and Emacs

[Paul Eggert]

I had problems updating Emacs to use gnulib commit 
fde88b711c9b1df5b142444ac7b0bc2aa8892d3a along with emacs commit 
fd859fbea2e9d13e76db1c5295d9ddd1c5955d83 (these are the same commits as 
I mentioned earlier today). I reproduced it like this:


cd emacs
admin/merge-gnulib

The resulting lib/gnulib.mk.in had a line:

  -e 's|@''HAVE_OFF64_T''@|$(HAVE_OFF64_T)|g' \

but there was no "HAVE_OFF64_T = @HAVE_OFF64_T@" line, and this caused 
the "#if !@HAVE_OFF64_T@" line in lib/sys_types.in.h to become "#if !" 
in lib/sys/types.h which is a syntax error in C.


I plan to work around the problem in Emacs by changing 
admin/merge-gnulib to remove m4/off64_t.m4, and putting the following 
into configure.ac instead:


  # Emacs does not need to check for off64_t. 

  AC_DEFUN([gl_TYPE_OFF64_T], 

[HAVE_OFF64_T=1 


 AC_SUBST([HAVE_OFF64_T])])

This is of course fragile.

It strikes me that apps like Emacs and coreutils don't need (and 
shouldn't use) off64_t which was merely intended as a transitional type 
in the 1990s, so there should be some way for them to disable the 
off64_t business without the abovementioned sort of fragile hackery. A 
documented way to disable off64_t discovery should simplify 
configuration and make it less likely to run into glitches such as the 
one I ran into with Emacs.