Re: How can Autoconf help with the transition to stricter compilation defaults?

[Paul Eggert]

On 2022-11-11 07:11, Aaron Ballman wrote:

Clang doesn't require such a linker (we work with various system linkers).


As long as the system linkers continue to work as they have 
traditionally worked, we're fine.



the frontend perspective, we can't tell the difference between
"trust me this is safe because it never gets executed" and "this is a
CVE"


If some system linker ever attempts to reject links with mismatched 
signatures, Autoconf-generated code will need to have a way to shut that 
off. I hope Clang maintainers can be cajoled into supporting that, if 
the time comes. Perhaps there can be a #pragma, or a compile-time 
option, to do that.

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Paul Eggert]

On 2022-11-11 07:11, Aaron Ballman wrote:

We believe the runtime behavior is sufficiently dangerous to
warrant a conservative view that any call to a function will be a call
that gets executed at runtime, hence a definitive signature mismatch
is something we feel comfortable diagnosing (in some form) by default.


As long as these diagnostics by default do not cause the compiler to 
exit with nonzero status, we should be OK with Autoconf-generated 
'configure' scripts. Although there will be problems with people who run 
"./configure CFLAGS='-Werror'", that sort of usage has always been 
problematic and unsupported by Autoconf, so we can simply continue to 
tell people "don't do that".

Re: On time64 and Large File Support

[Russ Allbery]

Wookey  writes:

> Now, I'm not yet sure if just having autoconf 2.72 will actually break
> things. AIUI, these changes only apply where LFS
> (-D_FILE_OFFSET_BITS=64) is turned on, so in Debian at least, where that
> is not the default on 32bit arches, maybe this is OK. But probably quite
> a lot of packages already enable LFS so they are suddenly going to get a
> new ABI if they expose timet anywhere?
> https://codesearch.debian.net/search?q=AC_SYS_LARGEFILE&perpkg=1 shows
> 163 pages of hits, and a quick peruse suggsts that AC_SYS_LARGEFILE is
> used by a lot of packages (as you might expect - this transition has
> been going on for many years). And just having that macro in
> configure.(in|ac) will turn 64-bit timet on if you autoreconf with
> 2.72. Right?

If indeed pre-existing use of AC_SYS_LARGEFILES would suddenly enable
64-bit time_t on autoreconf, I can name two packages just off the top of
my head that this change to Autoconf will immediately break if their
Debian packages are rebuilt with a newer version of Autoconf, creating
severe bugs.

libremctl will have its ABI changed without any coordination or versioning
(which I will be doing, moving forward, but have not started tackling yet
in part because I was waiting to see what the plan would be and whether
there will be some coordinated change to SONAMEs, a new architecture, or
what).  And INN, which admittedly is a disaster about things like this for
lots of historical reasons, will have its *on-disk file format* changed
without notice in a way that will cause serious failure and possibly data
corruption on upgrades.

This is just wildly backward-incompatible and seems like an awful idea.
If we're going to throw a big switch and rebuild everything, it needs to
be done at a distro-wide level.  I believe the only safe thing for
Autoconf to do is to provide an opt-in facility, similar to what was done
for AC_SYS_LARGEFILE, and then leave deciding whether to opt in to
higher-level machinery.

> However my limited understanding as of right now says that autoconf 2.72
> tying 64bit time_t to use of AC_SYS_LARGEFILE means that 2.72 can't be
> used in debian yet. So I currently favour not tying them together in
> this release.

That's also my understanding from the thread so far, although I'm not sure
that I'm following all of the subtleties.

> People have been using AC_SYS_LARGEFILE without 64bit time_t for many
> years now so it's not yet clear to me why that cannot continue.

And these are conceptually not at all the same thing.  I saw Paul's
explanation for why he views them as fundamentally the same because of
their effect on system calls like stat, but I certainly don't think of
them that way and I am quite dubious many other people will either.  The
set of things that I have to check to ensure that time_t is handled
correctly is totally different than the set of things I thought about when
enabling AC_SYS_LARGEFILE many years in the past.

I recognize that there will be overlap once file timestamps are past 2038
and that will happen sooner than anyone plans for, but it's still true
that this has *not* happened right now and this therefore is not currently
creating many bugs, whereas this switch in this way will create many, very
serious bugs immediately.

-- 
Russ Allbery (ea...@eyrie.org) 

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-12 12:23, Wookey wrote:

we can't just have everyone who enabled LFS sometime in the
last 20 years suddenly being forced into the time_t change (can we?)


We've done it in the past.

AC_SYS_LARGEFILE originally was in Gnulib, before it migrated to 
Autoconf. Originally it affected only off_t; its effect on other types 
like ino_t came later. Hence people who initially used AC_SYS_LARGEFILE 
were later "suddenly forced" into an ino_t width change. Similarly for 
other non-off_t types that AC_SYS_LARGEFILE affects.


So there's longstanding precedent for AC_SYS_LARGEFILE changing the 
width of system types that were formerly unaffected. The difference is 
that time_t is more widely used than ino_t etc., and people are 
understandably more concerned about time_t changes.



Because of the concerns raised in this thread it's become clear that 
what's in Autoconf now is too drastic, and I've proposed (though not yet 
implemented) a change that will cause AC_SYS_LARGEFILE to continue to 
not affect time_t unless there's an affirmative request like 
"./configure --enable-year2038". I am waiting for feedback on that from 
Zack and others, and am hoping for quick turnaround on this because we 
do need a new Autoconf release.

Re: On time64 and Large File Support

[Wookey]

On 2022-11-12 11:15 -0800, Paul Eggert wrote:
> On 2022-11-12 10:50, Bruno Haible wrote:
> > I'm saying
> > "tiny" because we are still 15 years away, and new releases of the (not
> > many) affected packages are likely to come in the next 1-2 years.
> 
> Not so "tiny", I'm afraid. My department is still running a server with
> libraries and executables that are over 17 years old.

Nobody disagrees with the idea that 2038 fixes are now fairly urgent
if we want to avoid real-world stuff breaking in 15 years time. That
has been increasingly clear for the last few years.

But the point here is that we need a bit of co-ordination and a plan,
particularly for binary distros. This isn't a minor change that should
just happen to people because there was an autoconf upgrade. Or at
least I don't think it is. My assumption is that if you just turned it
on in random packages as they were uploaded, there would be very
serious (unacceptably bad) breakage - we need to co-ordinate sets of
matching-ABI packages to upgrade together, and the worry is that the
matching set is 'most of the distro'.

Now, I'm not yet sure if just having autoconf 2.72 will actually break
things. AIUI, these changes only apply where LFS
(-D_FILE_OFFSET_BITS=64) is turned on, so in Debian at least, where
that is not the default on 32bit arches, maybe this is OK. But
probably quite a lot of packages already enable LFS so they are
suddenly going to get a new ABI if they expose timet anywhere?
https://codesearch.debian.net/search?q=AC_SYS_LARGEFILE&perpkg=1 shows
163 pages of hits, and a quick peruse suggsts that AC_SYS_LARGEFILE is
used by a lot of packages (as you might expect - this transition has
been going on for many years). And just having that macro in
configure.(in|ac) will turn 64-bit timet on if you autoreconf with
2.72. Right?

We can't embark on that until we decide whether this transition will
be done within the existing arch/triplet or with a new one. I agree we
should decide that pronto. And I think that 'we' is bigger than just Debian.

If new autoconf (and gnulib) just turns this on within the existing
arch/triplet then we, the distro, can't use those new versions unless
we back out those changes, or until we decide that we are going to
attempt this ABI transition within the existing arch/triplet.

I'm OK with this being done either way (complicated transition within
arch/triplet, or whole-world rebuild with new arch/triplet) once the
size of the problem (and thus the amount of work) is reasonably
understood and there is some consensus amongst distros. TTBOMK we need
some more research before we can make that choice (although I realise
some others are further down this road than I am, and yes I did plan
to get to this point about a year ago, so apologies for
slowness). Hence my suggestion of a full discussion on the
cross-distro list. It is overdue.

However my limited understanding as of right now says that autoconf
2.72 tying 64bit time_t to use of AC_SYS_LARGEFILE means that 2.72
can't be used in debian yet. So I currently favour not tying them
together in this release.

People have been using AC_SYS_LARGEFILE without 64bit time_t for many
years now so it's not yet clear to me why that cannot continue. And
even if it is a good idea to complete both transitions in the same
upheaval we can't just have everyone who enabled LFS sometime in the
last 20 years suddenly being forced into the time_t change (can we?).

Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-12 10:50, Bruno Haible wrote:

I'm saying
"tiny" because we are still 15 years away, and new releases of the (not
many) affected packages are likely to come in the next 1-2 years.


Not so "tiny", I'm afraid. My department is still running a server with 
libraries and executables that are over 17 years old. I have asked for 
it to be decommissioned, but there are backward compatibility concerns. 
(The OS is still supported by its supplier, and we install security 
patches, most recently the day before yesterday.)


Admittedly my situation differs from embedded environments where 
_TIME_BITS=64 is likely to make a difference 15 years from now. 
Unfortunately the situation in embedded environments is often worse.




My assessment is based on the understanding that Zack's proposed change
is essentially [a small change to the code]


Yes, that's my understanding too. Unfortunately the Autoconf change 
would have to be more complicated than that, since documentation and 
comments would have to change accordingly. And the change to Gnulib code 
would be considerably more complicated; this inevitably follows from any 
significant change we make in this area to Autoconf on Savannah. I would 
rather we spent our limited resources moving forward not backward.

Re: On time64 and Large File Support

[Bruno Haible]

Paul Eggert wrote:
> On 2022-11-12 06:16, Zack Weinberg wrote:
> > I am going to go ahead and do this if nobody raises a concrete objection
> > within the next 24 hours.
> 
> I object to a simple reversion, as this will cause problems downstream 
> with Gnulib-using applications, several of which have already been 
> released assuming the current Autoconf git will go into the next 
> release, and which will stop working if built from Git with an Autoconf 
> 2.72 where the AC_SYS_LARGEFILE changes have been reverted.

Paul, my assessment of Zack's proposed change is different. Things will
not "stop working". But rather, for packages
  * that have been released in the last three months, with the then-newest
Gnulib,
  * only when this package is reconfigured with Autoconf 2.72
then
  (1) a portability problem to HP-UX/ia64 in 32-bit mode will appear,
  (2) year 2038 compliance measures will not be enabled.

(1) is not something we need to worry about, since HP-UX/ia64 has so few
users, and why should them run 'autoreconf'.

(2) is a tiny setback on our journey to year 2038 compliance. I'm saying
"tiny" because we are still 15 years away, and new releases of the (not
many) affected packages are likely to come in the next 1-2 years.

My assessment is based on the understanding that Zack's proposed change
is essentially

diff --git a/lib/autoconf/specific.m4 b/lib/autoconf/specific.m4
index 0a9adba5..649d2d17 100644
--- a/lib/autoconf/specific.m4
+++ b/lib/autoconf/specific.m4
@@ -278,7 +278,7 @@ AS_IF([test "$enable_largefile" != no],
  [Define for large files, on AIX-style hosts.],
  [_AC_SYS_LARGEFILE_TEST_INCLUDES])],
 [64],
-  [_AC_SYS_YEAR2038()])])
+  [])])
 
   AH_VERBATIM([__MINGW_USE_VC2005_COMPAT],
 [#if !defined __MINGW_USE_VC2005_COMPAT && defined __MINGW32__

and upon inspection of the largefile.m4 part of this Gnulib commit:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=b9bf95fd0a6ab666b484b1e224321664f051f7fa

Bruno

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-11 18:20, Zack Weinberg wrote:


I don’t think Paul considered the new behavior of AC_SYS_LARGEFILE to be
overriding the glibc maintainers. Rather, I think he was only thinking
about applications, not libraries, and only about source distribution.


No, I was thinking about libraries as well. Although there are problems 
with libraries and time_t, for decades we've had the same problems with 
AC_SYS_LARGEFILE and off_t, rlim_t, ino_t, etc. Why should time_t be 
treated differently from the other types?




(It’s unfortunate, IMNSHO, that glibc chose not to provide a time_t
equivalent of _LARGEFILE64_SOURCE, as this means it’s much more
difficult for any shared library other than glibc to offer *both* time_t
and time64_t binary interfaces.)


I can easily understand why glibc didn't take that approach. The LFS API 
designed in the mid-1990s to support both off_t and off64_t in 
user-level code was supposed to be a "transitional API", but we're still 
stuck with it nearly 30 years later, even though almost nobody uses it 
(and the few who do often don't use it correctly). Even if we weren't 
dissuaded by the problems of the LFS API, we won't have a luxury of a 
30-year transition for time_t, as we're only 15 years away from 2038.


More generally, the LFS API approach doesn't scale, as the complexity of 
the implementation would grow exponentially with the number of features 
involved. Nobody wants to implement or support that sort of thing. This 
is why glibc put rlimt_t, ino_t, etc. under the off_t umbrella.




I am honestly not sure what to do about this in the long term, but for
the proposed “this weekend, just bugfixes” Autoconf 2.72, I do think it
makes sense to back out change #2, only — that is, AC_SYS_YEAR2038 will
exist, but AC_SYS_LARGEFILE will *not* imply AC_SYS_YEAR2038. 


AC_SYS_LARGEFILE doesn't imply AC_SYS_YEAR2038 now, in Autoconf git. 
Things are a bit more complicated, unfortunately.

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-12 06:16, Zack Weinberg wrote:

I am going to go ahead and do this if nobody raises a concrete objection
within the next 24 hours.


I object to a simple reversion, as this will cause problems downstream 
with Gnulib-using applications, several of which have already been 
released assuming the current Autoconf git will go into the next 
release, and which will stop working if built from Git with an Autoconf 
2.72 where the AC_SYS_LARGEFILE changes have been reverted. Current 
Gnulib assumes that AC_SYS_LARGEFILE will behave in Autoconf 2.72 like 
what's in Git now. If we change AC_SYS_LARGFILE back, we will need to 
change Gnulib too, and update downstream apps accordingly.


Instead, I suggest a partial reversion, in which AC_SYS_LARGEFILE goes 
back to its previous meaning by default, but there is a well-documented 
way to get the current in-Git meaning, a way that Gnulib can recognize 
and use. For example, you'd get the new meaning if you configure with 
--enable-year2038, or if configure.ac uses a new macro 
AC_SYS_LARGER_FILE that supersedes AC_SYS_LARGEFILE. This will also 
require some Gnulib changes, but they'll be more reliable and stable 
than whipsaw changes required by reverting then whatever future changes 
Autoconf makes.


This proposal would resolve the backward-compatibility concerns for 
people who want to delay worrying about year-2038 issues, while also 
resolving the compatibility concerns of Gnulib. It would also provide a 
better-documented way for distros that want to switch to 64-bit time_t.


Of course this sort of thing cannot be written and tested in an hour. 
But reverting is not that simple either, and would be a less efficient 
use of everybody's time.

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Zack Weinberg]

Demi Marie Obenour  writes:
> On 11/10/22 15:19, Paul Eggert wrote:
>> Here's the main Autoconf issue issue with bool. Traditionally, Autoconf 
>> supported K&R C, C89, C99, etc. At some point (I'm not sure when), 
>> Autoconf started requiring C89 or later. Is it now OK for Autoconf to 
>> require C99 or later, as far as bool is concerned? If so, that'll 
>> considerably simplify the ongoing maintenance hassle for bool.
>>
>> Requiring C99-or-later bool is the option that Gnulib has taken. Its 
>> 'stdbool' module and its gl_C_BOOL macro assumes C99 or later, and as 
>> far as I know no Gnulib-using package is using Gnulib's 'stdbool-c99' 
>> module which we kept around in case somebody still needed bool to work 
>> atop a C89 system. (We considered supporting C23 bool atop C89 but it 
>> was too painful.)
>
> I am fine with this option.

On the whole I’m also fine with this, but I think we need to consider,
separately, two kinds of Autoconf-using packages and three kinds of
compilers.

I think it’s definitely fine if Autoconf-using packages that request
support for ‘bool’, using either AC_C_BOOL or gl_C_BOOL, start requiring
a C99 compiler as of 2.72 (but see below).  I suspect there are existing
packages (Kermit comes to mind) that intend to maintain compatibility
with C89 compilers *forever*, and will choose *not* to use AC_C_BOOL,
and will be annoyed if AC_PROG_CC by itself starts rejecting C89 compilers.

(We may eventually decide we don’t support C89 compilers *at all*
anymore but that should be an independent discussion and not driven by
‘bool’.)

Then, on the compiler side of things, there’s compilers that have
complete support for ‘bool’ as it was specified in C99 (i.e. both the
‘_Bool’ keyword is recognized and a useful ‘stdbool.h’); there’s
compilers that have ‘_Bool’ but *don’t* have a useful ‘stdbool.h’; and
there’s compilers that don’t have any “true Boolean type” (as I put it
in the manual) at all.  In earlier discussions, IIRC, we determined that
compilers in all three of these categories do exist.

I suggest that what we mean by “Packages that use AC_C_BOOL require a
C99 compiler” is precisely “When @code{AC_C_BOOL} is used,
@command{configure} will error out if the C compiler does not recognize
either @samp{bool var;} or @samp{_Bool var;} as a valid complete
translation unit.”  In other words, the third category of compilers, the
ones that don’t have any “true Boolean type” (whose name we know), are
rejected.  However, compilers in the second category (_Bool available,
stdbool.h either unavailable or does not work) will be accepted and
config.h will perform the #defines that stdbool.h ought to have
performed.

> I just checked and both GCC 12.2 and clang 14 support  in C89
> mode.  I do get a -Wc99-extensions warning from clang but that can easily
> be suppressed with -Wno-c99-extensions.

Good to know, thanks.

zw

Re: On time64 and Large File Support

[Zack Weinberg]

Sam James  writes:

>> On 12 Nov 2022, at 02:20, Zack Weinberg via Libc-alpha 
>>  wrote:
>> I am honestly not sure what to do about this in the long term, but for
>> the proposed “this weekend, just bugfixes” Autoconf 2.72, I do think it
>> makes sense to back out change #2, only — that is, AC_SYS_YEAR2038 will
>> exist, but AC_SYS_LARGEFILE will *not* imply AC_SYS_YEAR2038.  That will
>> limit the impact of AC_SYS_YEAR2038 to packages that have explicitly
>> added it, and should make it safe for Fedora and Gentoo to drop in 2.72
>> in order to unblock C23 testing — am I correct?  It doesn’t resolve the
>> larger issue, but it gives us more time to think about what the
>> resolution ought to be.
>> 
>> What do you think?
>
> This is really I think the best option while allowing us time & space
> to complete the larger discussion.
[…]

I am going to go ahead and do this if nobody raises a concrete objection
within the next 24 hours.

zw