Re: On time64 and Large File Support

[Sam James]

> On 12 Nov 2022, at 04:56, Wookey  wrote:
> 
> On 2022-11-12 04:28 +, Sam James wrote:
>> 
>> 
>>> On 12 Nov 2022, at 04:20, Wookey  wrote:
>>> 
>>> Distros need to co-ordinate on this. If there are going to be new
>>> triplets for the 'LFS and 64_bit timet' ABI(s) then we should agree on
>>> them and use them. If distros are happy to migrate to these ABIs
>>> within the existing arm-linux-gnueabihf and i386-linux-gnu (or
>>> i686-linux-gnu) then we should do that.
>>> 
>>> If half the distros migrate within the existing triplet and the rest use
>>> a new one, that sounds like a recipie for much confusion.
>>> 
>> 
>> 100%. And also on sharing patches and known problems
>> and experience with the migration. All of it!
> 
> OK. It seems that the time is right to have this conversation. So we should 
> agree on a place to do it.
> 
> We have used the linaro cross-distro list in the past as it is
> intended for this sort of cross-cutting discussions. I can't think of
> a better place?
> https://lists.linaro.org/mailman3/lists/cross-distro.lists.linaro.org/
> 
> Some of the right people are probably already there, but we should
> encourage others with relevant expertise to join.
> 

That list is new to me (sub'd now) but I'm fine with using it as long
as linaro are still OK with us using it and them maintaining it.

Thanks for taking the initiative!

If they aren't, we could request a list from lists.linux.dev
(Linux Foundation).

Best,
sam


signature.asc
Description: Message signed with OpenPGP

Re: On time64 and Large File Support

[Wookey]

On 2022-11-12 04:28 +, Sam James wrote:
> 
> 
> > On 12 Nov 2022, at 04:20, Wookey  wrote:
> > 
> > Distros need to co-ordinate on this. If there are going to be new
> > triplets for the 'LFS and 64_bit timet' ABI(s) then we should agree on
> > them and use them. If distros are happy to migrate to these ABIs
> > within the existing arm-linux-gnueabihf and i386-linux-gnu (or
> > i686-linux-gnu) then we should do that.
> > 
> > If half the distros migrate within the existing triplet and the rest use
> > a new one, that sounds like a recipie for much confusion.
> > 
> 
> 100%. And also on sharing patches and known problems
> and experience with the migration. All of it!

OK. It seems that the time is right to have this conversation. So we should 
agree on a place to do it.

We have used the linaro cross-distro list in the past as it is
intended for this sort of cross-cutting discussions. I can't think of
a better place?
https://lists.linaro.org/mailman3/lists/cross-distro.lists.linaro.org/

Some of the right people are probably already there, but we should
encourage others with relevant expertise to join.


Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: On time64 and Large File Support

[Sam James]

> On 12 Nov 2022, at 04:20, Wookey  wrote:
> 
> On 2022-11-11 10:19 +0100, Florian Weimer wrote:
> 
> Hi. I've started looking into the 64-bit time_t transition for 32-bit armhf
> in Debian.  We are currently doing a preliminary bootstrap to see what
> breaks. We strongly suspect that only a wholesale rebuild for the new
> ABI (i.e a new Debian architecture) is practical, but have not yet
> entirely ruled out attempting a migration within the existing armhf
> arch.
> 
> [snip]
> 
>> * Sam James
>> 
>>> In Gentoo, we've been planning out what we should do for time64 on
>>> glibc [0] and concluded that we need some support in glibc for a newer
>>> option. I'll outline why below.
>>> 
>>> Proposal: glibc gains two new build-time configure options:
>>> * --enable-hard-time64
>>> * --enable-hard-lfs
> 
> I don't quite follow the logic of this. glibc already has build-time macros 
> to set these two things:
> _TIME_BITS=64
> _FILE_OFFSET_BITS=64
> 
> why do we need configure options too?

How do you make sure that every program built uses it? Not every
program respects CPPFLAGS and even in CFLAGS, it's a bit
of a nuisance.

If you patch GCC, you don't cover Clang. If you patch system
compilers, that's messy but also doesn't help with custom-built programs.

Of course, we could just patch glibc and cheerily jam it in the headers,
but we run into the kind of problems that Joseph Myers mentions then,
I think (basically I'd want to make sure we do it right.)

> 
>>> We're now (possibly) on the eve of an autoconf 2.72 release which contains 
>>> two changes
>>> of note [2][3]
>>> 1. addition of a new AC_SYS_YEAR2038 macro;
>>> 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
> 
> Which is the opposite way round to glibc, where _TIME_BITS=64 requires
> _FILE_OFFSET_BITS=64, but not the other way round
> (_FILE_OFFSET_BITS=64, can be set on its own). Am I misunderstanding 
> something here?
> 

I wonder the same. I don't think it's obvious, and it may not be obvious
to people writing software using autoconf either...

> It doesn't seem right to me that AC_SYS_LARGEFILE should imply
> AC_SYS_YEAR2038. What is the reasoning behind that?
> 
>> I really wish the rest of GNU would talk to glibc maintainers before
>> overriding glibc maintainer decisions.  If we cannot revert this in
>> autoconf (and gnulib), this will very much endanger the Fedora i386
>> port.  Debian will probably be impacted in the same way.
> 
> I need to read around all this as I have only just become aware that
> the LFS thing is entangled with the timet_64 thing. Is there a good
> place to read _why_ one implies the other? It definitely complicates
> matters.

time64 has to imply LFS because of some structures like stat including
both off_t (LFS) and st_atim (time64), I think. Some of it is internal too.

Or do you mean LFS => time64? I have no idea for why that's
entangled in autoconf and gnulib.

> 
>>> On reflection and after extensive discussion within Gentoo (although
>>> I don't seek to speak for everybody there) - with special thanks to
>>> David Seifert and Arsen Arsenović for tolerating my bikesheds on this,
>>> we don't think it's feasible to handle this in a piecemeal fashion -
>>> at the very least not without spending a significant & for some,
>>> undesirable amount of time on supporting "obsolete" 32-bit platforms.
> 
> Distros need to co-ordinate on this. If there are going to be new
> triplets for the 'LFS and 64_bit timet' ABI(s) then we should agree on
> them and use them. If distros are happy to migrate to these ABIs
> within the existing arm-linux-gnueabihf and i386-linux-gnu (or
> i686-linux-gnu) then we should do that.
> 
> If half the distros migrate within the existing triplet and the rest use
> a new one, that sounds like a recipie for much confusion.
> 

100%. And also on sharing patches and known problems
and experience with the migration. All of it!

> I could write more, but I'll swot up a bit first :-)

It's not easy to find much about all of this! I almost
felt like I was missing something at first. :)

Best,
sam


signature.asc
Description: Message signed with OpenPGP

Re: On time64 and Large File Support

[Wookey]

On 2022-11-11 10:19 +0100, Florian Weimer wrote:

Hi. I've started looking into the 64-bit time_t transition for 32-bit armhf
in Debian.  We are currently doing a preliminary bootstrap to see what
breaks. We strongly suspect that only a wholesale rebuild for the new
ABI (i.e a new Debian architecture) is practical, but have not yet
entirely ruled out attempting a migration within the existing armhf
arch.

A test of this in 2020 by Arnd Bergman found that too much stuff was broken.
https://lists.debian.org/debian-arm/2020/02/msg00024.html
Things now look much more mature as some others have already fixed various 
things.
https://lists.debian.org/debian-arm/2022/09/msg00012.html
https://lists.debian.org/debian-arm/2022/10/msg00020.html

I've not got far with bootstrapping but had got as for as noting that
LFS and 64bit timet are tied together in glibc (setting _TIME_BITS=64
requires setting _FILE_OFFSET_BITS=64). So that's two lots of
interacting ABI changes, which doesn't make things any simpler.

> * Sam James
> 
> > In Gentoo, we've been planning out what we should do for time64 on
> > glibc [0] and concluded that we need some support in glibc for a newer
> > option. I'll outline why below.
> >
> > Proposal: glibc gains two new build-time configure options:
> > * --enable-hard-time64
> > * --enable-hard-lfs

I don't quite follow the logic of this. glibc already has build-time macros to 
set these two things:
_TIME_BITS=64
_FILE_OFFSET_BITS=64

why do we need configure options too?

> We should define new target triplets for this if it's really required.

We have been coming to the conclusion that this is necessary. If it's
not feasible to migrate with the existing armhf (and maybe i386)
architectures, then we need a new triplet to define this ABI and (in
debian, match the dkpg arch name).

> We need to support legacy binaries on i386.  Few libraries are
> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
> to LFS or time64 needs to be evaluated on a per-library basis.  For most
> distributions, no one is going to do that work, and we have to stick to
> whathever we are building today.

Well Debian has started this work.

> > These would hard-enable the relevant #defines within glibc's headers
> > and ensure that any binaries built with such a glibc have both Large
> > File Support (LFS) and time64 support.
> >
> > I've come to the conclusion it's infeasible to try handle the
> > migration piecemeal. Various mismatches can and will occur (and while
> > it's more likely with time64, it's possible with LFS too) [1].

Right. This is definitely a problem, as both items will be in structs
that are exposed in ABIs in various places. What I've not yet got a
handle on is just how big a problem. Debian has done large ABI
transitions before within an architecture (glibc5->6), (GCC 4.0 C++
ABI), and (long-double transition (from 64 to 128 bits), on a subset of 
arches), so it is
possible. (That long-double transition is the most like this
transition, because it involves types that can appear in C and C++
ABI). https://lists.debian.org/debian-devel/2007/05/msg01173.html


> > We're now (possibly) on the eve of an autoconf 2.72 release which contains 
> > two changes
> > of note [2][3]
> > 1. addition of a new AC_SYS_YEAR2038 macro;
> > 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.

Which is the opposite way round to glibc, where _TIME_BITS=64 requires
_FILE_OFFSET_BITS=64, but not the other way round
(_FILE_OFFSET_BITS=64, can be set on its own). Am I misunderstanding something 
here?

It doesn't seem right to me that AC_SYS_LARGEFILE should imply
AC_SYS_YEAR2038. What is the reasoning behind that?

> I really wish the rest of GNU would talk to glibc maintainers before
> overriding glibc maintainer decisions.  If we cannot revert this in
> autoconf (and gnulib), this will very much endanger the Fedora i386
> port.  Debian will probably be impacted in the same way.

I need to read around all this as I have only just become aware that
the LFS thing is entangled with the timet_64 thing. Is there a good
place to read _why_ one implies the other? It definitely complicates
matters.

> > On reflection and after extensive discussion within Gentoo (although
> > I don't seek to speak for everybody there) - with special thanks to
> > David Seifert and Arsen Arsenović for tolerating my bikesheds on this,
> > we don't think it's feasible to handle this in a piecemeal fashion -
> > at the very least not without spending a significant & for some,
> > undesirable amount of time on supporting "obsolete" 32-bit platforms.

Distros need to co-ordinate on this. If there are going to be new
triplets for the 'LFS and 64_bit timet' ABI(s) then we should agree on
them and use them. If distros are happy to migrate to these ABIs
within the existing arm-linux-gnueabihf and i386-linux-gnu (or
i686-linux-gnu) then we should do that.

If half the distros migrate within the existing triplet and the 

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Sam James]

> On 12 Nov 2022, at 00:53, Paul Eggert  wrote:
> 
> On 2022-11-11 15:25, Sam James wrote:
>> That's not a judgement on whether the changes will ultimately remain in 
>> autoconf, I'm just
>> hesitant to allow a discussion I've kicked off to derail something that we 
>> were planning
>> on doing anyway.
>> What do you think?
> 
> I'm hesitant to do that partly because the changes to _TIME_BITS are already 
> released in multiple packages and need to be dealt with, regardless of 
> whether they're backed out of Autoconf. This is because they've been in 
> Gnulib since July and several packages based on these Gnulib changes have 
> been released since then. Current Gnulib assumes these changes will appear in 
> the next Autoconf release; if that's not true, we'll need to upgrade Gnulib 
> and in the meantime the other packages released since July would still have 
> the changes whatever we do with Gnulib and/or Autoconf.
> 
> Since distros need to deal with the issue anyway, regardless of what Autoconf 
> and/or Gnulib does, I don't see why backing the changes out of Autoconf will 
> help all that much.
> 
> It should pretty easy for a distro to say "hold on, I don't want 64-bit 
> time_t yet" without changing either Autoconf or Gnulib so if you want to go 
> that route please feel free to do so.

The fact it's already shipped in gnulib & that the "real problem" is in glibc 
IMO means that I don't feel
strongly about reverting it.

You're right that distros have the toggle so as long as we mention this 
prominently enough in NEWS,
I don't have a strong objection to master being released as-is.


signature.asc
Description: Message signed with OpenPGP

Re: On time64 and Large File Support

[Sam James]

> On 12 Nov 2022, at 02:20, Zack Weinberg via Libc-alpha 
>  wrote:
> 
> Sam James  writes:
>>> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
>>> We need to support legacy binaries on i386.  Few libraries are
>>> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
>>> to LFS or time64 needs to be evaluated on a per-library basis.  For most
>>> distributions, no one is going to do that work, and we have to stick to
>>> whathever we are building today.
>> 
>> While I agree, I don't think it's as well-known that it should be that
>> these are ABI breaking and require inspection. It's being done ad-hoc
>> or in many cases, not at all.
> …
 We're now (possibly) on the eve of an autoconf 2.72 release which
 contains two changes of note [2][3]
 1. addition of a new AC_SYS_YEAR2038 macro;
 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
 
 Indeed, the gnulib version of change #2 is exactly how we ended up with
 wget/gnutls breaking [1]. I feel this shows that the only approach
 "supported" by glibc right now is untenable.
>>> 
>>> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
>>> for Fedora unfortunately.
>>> 
>>> I thought the gnulib change has been reverted?
>>> 
>>> I really wish the rest of GNU would talk to glibc maintainers before
>>> overriding glibc maintainer decisions.
> 
> There seems to be a disconnect between Paul Eggert’s position on these
> changes and everyone else on this thread’s position.
> 
> I don’t think Paul considered the new behavior of AC_SYS_LARGEFILE to be
> overriding the glibc maintainers. Rather, I think he was only thinking
> about applications, not libraries, and only about source distribution.
> If an application is being built on the machine where it’ll be used, and
> both it and the system support building it with 64-bit time_t and off_t,
> *why wouldn’t you*?  It has no impact on anything else to do that, and
> it future-proofs your installataion.
> 
> But everyone else is worrying about cases where, either, an application
> is built with 64-bit time_t and/or off_t and then copied to a different
> system where the underlying libraries *don’t* support that, or a *shared
> library* is rebuilt with 64-bit time_t and/or off_t and that silently
> changes its ABI out from under programs that use it.
> 

Precisely.

> (It’s unfortunate, IMNSHO, that glibc chose not to provide a time_t
> equivalent of _LARGEFILE64_SOURCE, as this means it’s much more
> difficult for any shared library other than glibc to offer *both* time_t
> and time64_t binary interfaces.)
> 
> (It’s also unfortunate that, 20+ years after the invention of ELF symbol
> versioning, it’s still ridiculously difficult.  So many macros and
> assembly hacks and fighting with libtool.  But I digress.)
> 
> I am honestly not sure what to do about this in the long term, but for
> the proposed “this weekend, just bugfixes” Autoconf 2.72, I do think it
> makes sense to back out change #2, only — that is, AC_SYS_YEAR2038 will
> exist, but AC_SYS_LARGEFILE will *not* imply AC_SYS_YEAR2038.  That will
> limit the impact of AC_SYS_YEAR2038 to packages that have explicitly
> added it, and should make it safe for Fedora and Gentoo to drop in 2.72
> in order to unblock C23 testing — am I correct?  It doesn’t resolve the
> larger issue, but it gives us more time to think about what the
> resolution ought to be.
> 
> What do you think?

This is really I think the best option while allowing us time & space
to complete the larger discussion.

We keep AC_SYS_YEAR2038 which lets upstreams opt in,
but it avoids the risk of sudden LFS-into-time64 rollover.

Year 2038 work is really important (which is why I brought
up the topic) but I don't want confusion around it to make
people avoid adopting 2.72 or leave us without a 2.72
at all.

I do sympathise with Paul's position, but I'm not
a believer in piecemeal anyway now and it's fair to say there's no
consensus on this yet.

Paul's point Is fair that distros who wish can set the cache var to
ff, so *if* you and Paul want to keep it in.

I could live with a bigger "please check this!" in NEWS. I don't think it's a
total disaster as long as we brief distros first, as we're already
doing it for gnuilb & upstreams may well end up doing themselves,
so the train is on its way out of the station anyway.

(I think we need to keep the discussion going on that front,
but I think it'd be wrong to let it block this release, as it's
not a simple problem and I'm still unsure how I fully feel
about the issues at its core.)

I don't think it's conservative to make a macro suddenly
do a new thing which might require testing on the part
of projects using autoconf. Even if I get the aim.

And if we want to change it later, we still can. If we all
magically reach consensus in a month, there's nothing
stopping that being pursued.

So let's go for keep AC_SYS_YEAR2038 only?

Best,
sam


signature.asc
Description: 

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Zack Weinberg]

Bob Friesenhahn  writes:
> On Thu, 10 Nov 2022, Zack Weinberg wrote:
>> The biggest remaining (potential) problem, that I’m aware of, is that
>> AC_CHECK_FUNC unconditionally declares the function we’re probing for
>> as ‘char NAME (void)’
>
> This does seem like the biggest problem since it would appear to
> require that new macros are used and that the configure.ac developer
> would need to supply additional syntax (where syntax does not
> currently exist) in order to succeed with linking.  It is not possible
> for Autoconf itself to anticipate the function names used by libraries
> and assume some specific prototype.

For anything specified by either ISO C or POSIX/XSI, I think Autoconf
probably *can* bake in an expected function prototype.  It might be
worth doing the same for common-but-not-standardized C library
extensions (for concreteness, let’s say anything that exists in both
glibc and at least two BSDs is a candidate for this treatment).

We could go on doing what we do today for any unrecognized symbol, but
issue a warning, recommending the script be updated to supply the
expected prototype and headers.

zw

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Joseph Myers]

On Fri, 11 Nov 2022, Zack Weinberg via Gcc wrote:

> These are also a trip hazard for novices, and the only way to turn them
> off is with -std=cXX, which also turns another trip hazard (trigraphs)
> *on*… so yeah, anything you can do to help speed up their removal, I
> think it’d be worthwhile.

As of GCC 13, -std=c2x will disable trigraphs, since they've been removed 
from C2x.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Sam James]

> On 12 Nov 2022, at 03:40, Zack Weinberg  wrote:
> 
> Florian Weimer  writes:
>> based on a limited attempt to get this fixed about three years
>> ago, I expect that many of the problematic packages have not had their
>> configure scripts regenerated using autoconf for a decade or more.  This
>> means that as an autoconf maintainer, you unfortunately won't be able to
>> help us much.
> 
> I’m sadly not surprised.
> 
> This is definitely more work than I can see myself doing on a volunteer
> basis, but a 2.69.1 patch release — nothing that’s not already on trunk,
> cherry pick the changes needed to support the newer compilers (and
> also newer Perl and Bash and M4) is a thing that could happen.

I didn't want to ask you to do this because I felt fortunate enough
you were volunteering to handle 2.72, but this would indeed be a help,
because then I won't have to try persuade people they should totally upgrade,
and it should happen naturally enough with distro upgrades.

If you are willing, that would be welcome.

Of course, we'll have to go lobby them, but that is what it is :)



signature.asc
Description: Message signed with OpenPGP

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Zack Weinberg]

Florian Weimer  writes:
> based on a limited attempt to get this fixed about three years
> ago, I expect that many of the problematic packages have not had their
> configure scripts regenerated using autoconf for a decade or more.  This
> means that as an autoconf maintainer, you unfortunately won't be able to
> help us much.

I’m sadly not surprised.

This is definitely more work than I can see myself doing on a volunteer
basis, but a 2.69.1 patch release — nothing that’s not already on trunk,
cherry pick the changes needed to support the newer compilers (and
also newer Perl and Bash and M4) is a thing that could happen.

> Thanks, these changes are going to be helpful to get a clean run from
> our Fedora tester.

Autoconf’s own test suite is sadly not very thorough.  If you find more
problems I will prioritize them.

> Once you include the header, you also need to know function parameters,
> otherwise you won't be able to form a valid call.

You can assign to a function pointer variable if you know the complete
type signature, which is desirable for other reasons (see reply to Rich).
Needing to know how to form argument *values* could be much more trouble,
but I don’t think it should be necessary.

>> p.s. GCC and Clang folks: As long as you’re changing the defaults out
>> from under people,
>
> Hmph, I wouldn't frame it this way.  We are aware of GCC's special role
> as the system compiler.  We're trying to upstream the changes to sources
> before flipping the compiler default.  (The burden of being a system
> compiler and all that.)  A 25-year transition period evidently wasn't
> enough, so some effort is still needed.  We may conclude that removing
> these extensions is too costly even in 2024.

I didn’t mean to imply that I disliked any of the changes.  In fact,
with my day job (CS professor) hat on, I am quite looking forward to not
having to warn the kids about these legacy features anymore (we don’t
_teach_ them, but they inevitably use them by accident, particularly
implicit function declarations, and then get confused because ‘cc’ with
no -W options doesn’t catch the mistake).

>> can you please also remove the last few predefined
>> user-namespace macros (-Dlinux, -Dunix, -Darm, etc) from all the
>> -std=gnuXX modes?
>
> That's a good point, I'll think about how we can instrument GCC to
> support tracking that.  We won't be able help with -Darm on the Fedora
> side (the AArch64 port doesn't have that, and there's no longer a Fedora
> 32-bit Arm port), but -Dlinux and -Dunix we can help with.

These are also a trip hazard for novices, and the only way to turn them
off is with -std=cXX, which also turns another trip hazard (trigraphs)
*on*… so yeah, anything you can do to help speed up their removal, I
think it’d be worthwhile.

zw

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Zack Weinberg]

Rich Felker  writes:
> On Thu, Nov 10, 2022 at 12:16:20PM -0500, Zack Weinberg wrote:
>> The biggest remaining (potential) problem, that I’m aware of, is that
>> AC_CHECK_FUNC unconditionally declares the function we’re probing for
>> as ‘char NAME (void)’, and asks the compiler to call it with no
>> arguments, regardless of what its prototype actually is.
…
> Thanks for bringing this up. It is very important and I am very much
> in favor of making these changes and doing it in a way that existing
> broken and unmaintained software can be made to work just by
> re-generating configure scripts with up-to-date autoconf, even if that
> means hard-coding a list of headers needed to get the right
> declarations and automatically pulling them in.

Right.  In principle, I think AC_CHECK_FUNCS([symbol]), where ‘symbol’ is
in either ISO C or in XSI, could be made to do a check of the form you
suggest at the end of https://ewontfix.com/13/

#include 
#include 
int main()
{
double (*p)(const char *, char **, locale_t) = strtod_l;
}

It’s “just” a matter of compiling that big list of headers and expected
function signatures.  I’d also want to do something to ensure that this
assignment is not optimized out, so the linker has to process an
undefined reference to the symbol.

For symbols that are not covered by the built-in list, we could require
people to indicate the necessary headers and signature somehow.
Hypothetical notation

AC_CHECK_FUNCS([
[argp_parse, [argp.h],
   [error_t], [const struct argp *, int, char **,
   unsigned int, int *, void *]]
])

Note that this still isn’t perfect; if some system provides a function
with an identical type signature, but different *semantics*, to the one
the application wants, no compilation test can detect that.  Autoconf’s
not going to step away from its “avoid compile-and-run tests, that
breaks cross compilation” stance.

> I've been writing/complaining about autoconf doing this wrong for
> decades, with the best writeup around 9 years ago at
> https://ewontfix.com/13/. Part of the reason is that this has bitten
> musl libc users over and over again due to configure finding symbols
> that were intended only as ABI-compat and trying to use them (without
> declarations) at the source level

I vaguely recall some cases where this bit glibc and Apple’s libc as
well.

In principle, you’re supposed to be able to declare some ISO C functions
yourself, e.g.

extern int printf(const char *, ...);
int main(void) {
printf("hello world\n");
}

is strictly conforming per C99, but this bypasses any symbol renaming
applied by stdio.h.

> What I'd like to see happen is complete deprecation of the autoconf
> link-only tests

Do you have a concrete list of documented Autoconf macros that you would
like to see deprecated for this reason?

zw

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Zack Weinberg]

Nick Bowler  writes:
> My gut feeling is that Autoconf should just determine the necessary
> options to get compatible behaviour out of these modern compilers, at
> least for the purpose of running configure tests.  For example, Autoconf
> should probably build the AC_CHECK_FUNC programs using gcc's
> -fno-builtin option

I fear this will cause more problems than it solves.  Messing with
compiler options inside a configure script has a track record of
clashing with “outer” build tools that expect to be able to dictate the
options.

> It saddens me to see so much breakage happening in "modern C", a
> language that has (until now) a long history of new language features
> being carefully introduced to avoid these sort of problems.

I don’t exactly _disagree_ with this.  Quite a few of the compatibility-
breaking changes going into C2x (promoting ‘bool’ to a true keyword, for
instance, and changing the meaning of an empty argument list in a
function declaration) strike me as unnecessary churn.  However, the
specific set of changes that are under discussion right now—removal of
implicit function declarations, implicit int, and old-style function
definitions from the _default_ language accepted by C compilers—I’m very
much in favor of, because they make life significantly easier for people
writing _new_ code.  It’s not healthy for a language to always
prioritize old code over new code.

(Yes, you _can_ opt in to all three of those changes now, but you have
to type a bunch of -W options.  With my day job hat on, I am very much
looking forward to a day where ‘cc test.c’ errors out on implicit
function declarations, because then I won’t have to _explain_ implicit
function declarations, and why they are dangerous, to my students
anymore.)

>> p.s. GCC and Clang folks: As long as you’re changing the defaults out
>> from under people, can you please also remove the last few predefined
>> user-namespace macros (-Dlinux, -Dunix, -Darm, etc) from all the
>> -std=gnuXX modes?
>
> Meh, even though these macros are a small thing I don't accept the
> "things are breaking anyway so let's break even more things" attitude.

Getting rid of these is another change that will make life easier for
people writing new code.

zw

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Demi Marie Obenour]

On 11/10/22 15:19, Paul Eggert wrote:
> On 2022-11-10 09:16, Zack Weinberg wrote:
>> Changes to handle C23 built-in ‘bool’ better are under development but
>> the design has not yet been finalized.
> 
> [I'm cc'ing this to bug-gnulib too.]
> 
> To my mind this is the biggest outstanding issue in Autoconf as far as 
> C23 goes, as the upgrade path for Autoconf's existing bool support is 
> not entirely clear. As Florian mentioned, distros can't assume Autoconf 
> upgrades when building other packages; that being said, we should get 
> Autoconf's bool support fixed sooner rather than later as bool hassles 
> will remain until Autoconf is fixed and these fixes are propagated to 
> Autoconf's users.
> 
> Here's the main Autoconf issue issue with bool. Traditionally, Autoconf 
> supported K C, C89, C99, etc. At some point (I'm not sure when), 
> Autoconf started requiring C89 or later. Is it now OK for Autoconf to 
> require C99 or later, as far as bool is concerned? If so, that'll 
> considerably simplify the ongoing maintenance hassle for bool.
> 
> Requiring C99-or-later bool is the option that Gnulib has taken. Its 
> 'stdbool' module and its gl_C_BOOL macro assumes C99 or later, and as 
> far as I know no Gnulib-using package is using Gnulib's 'stdbool-c99' 
> module which we kept around in case somebody still needed bool to work 
> atop a C89 system. (We considered supporting C23 bool atop C89 but it 
> was too painful.)

I am fine with this option.

> If we follow Gnulib's lead, Autoconf will generate a config.h that does 
> "#include " on pre-C23 systems

At least in the case of config.h files that could be included in external
programs, I recommend unconditionally generating “#include ”.
That will ensure pre-C23 code that includes these headers builds okay.
I believe that in C23 and above this is a harmless no-op, as it is in C++.

> and this config.h will not 
> not work on pre-C99 systems. This of course could in theory break some 
> programs, just as compiling them with C23 can break them. But I don't 
> see any better option at this point. And besides, what package outside 
> of a museum still requires C89 and don't work with C99?

I just checked and both GCC 12.2 and clang 14 support  in C89
mode.  I do get a -Wc99-extensions warning from clang but that can easily
be suppressed with -Wno-c99-extensions.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)


OpenPGP_0xB288B55FFF9C22C1.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: On time64 and Large File Support

[Zack Weinberg]

Sam James  writes:
>> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
>> We need to support legacy binaries on i386.  Few libraries are
>> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
>> to LFS or time64 needs to be evaluated on a per-library basis.  For most
>> distributions, no one is going to do that work, and we have to stick to
>> whathever we are building today.
>
> While I agree, I don't think it's as well-known that it should be that
> these are ABI breaking and require inspection. It's being done ad-hoc
> or in many cases, not at all.
…
>>> We're now (possibly) on the eve of an autoconf 2.72 release which
>>> contains two changes of note [2][3]
>>> 1. addition of a new AC_SYS_YEAR2038 macro;
>>> 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
>>> 
>>> Indeed, the gnulib version of change #2 is exactly how we ended up with
>>> wget/gnutls breaking [1]. I feel this shows that the only approach
>>> "supported" by glibc right now is untenable.
>> 
>> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
>> for Fedora unfortunately.
>> 
>> I thought the gnulib change has been reverted?
>> 
>> I really wish the rest of GNU would talk to glibc maintainers before
>> overriding glibc maintainer decisions.

There seems to be a disconnect between Paul Eggert’s position on these
changes and everyone else on this thread’s position.

I don’t think Paul considered the new behavior of AC_SYS_LARGEFILE to be
overriding the glibc maintainers. Rather, I think he was only thinking
about applications, not libraries, and only about source distribution.
If an application is being built on the machine where it’ll be used, and
both it and the system support building it with 64-bit time_t and off_t,
*why wouldn’t you*?  It has no impact on anything else to do that, and
it future-proofs your installataion.

But everyone else is worrying about cases where, either, an application
is built with 64-bit time_t and/or off_t and then copied to a different
system where the underlying libraries *don’t* support that, or a *shared
library* is rebuilt with 64-bit time_t and/or off_t and that silently
changes its ABI out from under programs that use it.

(It’s unfortunate, IMNSHO, that glibc chose not to provide a time_t
equivalent of _LARGEFILE64_SOURCE, as this means it’s much more
difficult for any shared library other than glibc to offer *both* time_t
and time64_t binary interfaces.)

(It’s also unfortunate that, 20+ years after the invention of ELF symbol
versioning, it’s still ridiculously difficult.  So many macros and
assembly hacks and fighting with libtool.  But I digress.)

I am honestly not sure what to do about this in the long term, but for
the proposed “this weekend, just bugfixes” Autoconf 2.72, I do think it
makes sense to back out change #2, only — that is, AC_SYS_YEAR2038 will
exist, but AC_SYS_LARGEFILE will *not* imply AC_SYS_YEAR2038.  That will
limit the impact of AC_SYS_YEAR2038 to packages that have explicitly
added it, and should make it safe for Fedora and Gentoo to drop in 2.72
in order to unblock C23 testing — am I correct?  It doesn’t resolve the
larger issue, but it gives us more time to think about what the
resolution ought to be.

What do you think?
zw

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Paul Eggert]

On 2022-11-11 15:25, Sam James wrote:

That's not a judgement on whether the changes will ultimately remain in 
autoconf, I'm just
hesitant to allow a discussion I've kicked off to derail something that we were 
planning
on doing anyway.

What do you think?


I'm hesitant to do that partly because the changes to _TIME_BITS are 
already released in multiple packages and need to be dealt with, 
regardless of whether they're backed out of Autoconf. This is because 
they've been in Gnulib since July and several packages based on these 
Gnulib changes have been released since then. Current Gnulib assumes 
these changes will appear in the next Autoconf release; if that's not 
true, we'll need to upgrade Gnulib and in the meantime the other 
packages released since July would still have the changes whatever we do 
with Gnulib and/or Autoconf.


Since distros need to deal with the issue anyway, regardless of what 
Autoconf and/or Gnulib does, I don't see why backing the changes out of 
Autoconf will help all that much.


It should pretty easy for a distro to say "hold on, I don't want 64-bit 
time_t yet" without changing either Autoconf or Gnulib so if you want to 
go that route please feel free to do so.

Re: On time64 and Large File Support

[Joseph Myers]

On Fri, 11 Nov 2022, Paul Eggert wrote:

> One possible way forward would be for glibc to change its defaults for
> _FILE_OFFSET_BITS and _TIME_BITS to be 64, in the next major release. This

See my notes at 
 on 
what would be involved in changing the default *just of _FILE_OFFSET_BITS* 
for user programs, depending on whether the mode used to build glibc tests 
is also changed (but without changing the settings used for building glibc 
itself, which is much more complicated).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-11 03:38, Florian Weimer wrote:

But that said, these binaries are broken anyway in 2038?

No, I expect users to run them in time-shifted VMs or containers.


That's reasonable for systems where accurate timestamps are not 
important and where time_t width mismatches would just get in the way.


However, if this is the expected approach, I suggest having a standard 
and well-documented way to timeshift VMs and containers, and unless you 
want to cede the field entirely to other platforms this documentation 
effort and testing should be done now rather than in the year 2037. 
(Another thing to add to your bulging to-do list)

Re: On time64 and Large File Support

[Palmer Dabbelt]

On Fri, 11 Nov 2022 11:01:50 PST (-0800), libc-al...@sourceware.org wrote:

>> We need to support legacy binaries on i386.  Few libraries are
>> explicitly dual-ABI.  Whether it's safe to switch libraries above
>> glibc to LFS or time64 needs to be evaluated on a per-library
>> basis.  For most distributions, no one is going to do that work,
>> and we have to stick to whathever we are building today.
>
> ... since for Debian the libraries with different ABI end up in different
> multiarch paths then.

I didn't expect co-installability as a requirement.  But yes, if that's
the goal, we need non-overlapping paths.


Doesn't that requirement come automatically with "we need to support legacy
binaries"? How else would that work?


> Anyone with a more, ahem, standard filesystem arrangement has to find
> a different solution for the problem of legacy binaries.

We can have lib, lib64, libx32, and lib32t quite easily, that's not the
problem.  What's missing is ldconfig support.  The previous three x86
architectures have ELF-level selectors; we might need something special
there as well.


Yup. I was thinking of lib32n (which won't collide with anything out
there), but the selector problem remains.

[Apart from all further fun problems with library paths unexpected by
unwary upstreams... riscv64 (lib64/lp64d, lib64/lp64, lib32/ilp32d, 
lib32/ilp32) and mips64 (lib, lib32, lib64) send their regards.]


I don't want to derail the thread, but sorry again for the RISC-V bits 
there.  IMO we can fix it without an ABI break via adding some new 
paths, I just don't know what they should be.  Happy to hear if anyone 
has suggestions...

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-11 03:22, Florian Weimer wrote:


Fortunately, any issues should be quite visibile in
the distribution DWARF data.  If we put the time32 switch in place, we
should be able to tell whether it's effective enough in practice.


OK, thanks; if problems turn up in that area please let the 
Autoconf/Gnulib people know what we can do to help address them.




LFS issues have been with us for a long time, and packages and
distributions have workarounds for it (e.g., using off64_t or long long
in public headers).  What we have today mostly works.  But it's unknown
whether existing AC_SYS_LARGEFILE users require any additional work for
time64 changes.


If what we have today mostly works, then it works for blkcnt_t, dev_t, 
ino_t, off_t and rlim_t, all of whose widths differ on 32-bit x86 glibc 
depending on whether one compiles with -D_FILE_OFFSET_BITS=64. So 
there's some precedent for hoping that what we have today will mostly 
work for time_t and -D_TIME_BITS=64 as well.


In hindsight, perhaps it would have been better to have 
_FILE_OFFSET_BITS=64 also control time_t width, since that would have 
made for one less configure-time option to worry about. (After all 
rlim_t can hold 64-bit counts of seconds, so why not time_t?) I suppose 
it's too late for that now, though.

Re: On time64 and Large File Support

[Andreas K. Huettel]

> >> We need to support legacy binaries on i386.  Few libraries are
> >> explicitly dual-ABI.  Whether it's safe to switch libraries above
> >> glibc to LFS or time64 needs to be evaluated on a per-library
> >> basis.  For most distributions, no one is going to do that work,
> >> and we have to stick to whathever we are building today.
> >
> > ... since for Debian the libraries with different ABI end up in different
> > multiarch paths then.
> 
> I didn't expect co-installability as a requirement.  But yes, if that's
> the goal, we need non-overlapping paths.

Doesn't that requirement come automatically with "we need to support legacy
binaries"? How else would that work?

> > Anyone with a more, ahem, standard filesystem arrangement has to find
> > a different solution for the problem of legacy binaries.
> 
> We can have lib, lib64, libx32, and lib32t quite easily, that's not the
> problem.  What's missing is ldconfig support.  The previous three x86
> architectures have ELF-level selectors; we might need something special
> there as well.

Yup. I was thinking of lib32n (which won't collide with anything out
there), but the selector problem remains.

[Apart from all further fun problems with library paths unexpected by
unwary upstreams... riscv64 (lib64/lp64d, lib64/lp64, lib32/ilp32d, 
lib32/ilp32) and mips64 (lib, lib32, lib64) send their regards.]


-- 
Andreas K. Hüttel
dilfri...@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Aaron Ballman]

On Thu, Nov 10, 2022 at 4:05 PM Paul Eggert  wrote:
>
> On 2022-11-10 10:19, Aaron Ballman wrote:
> > In terms of the Clang side of things, I don't think we've formed any
> > sort of official stance on how to handle that yet. It's UB (you can
> > declare the C standard library interface without UB but calling any
> > function with a mismatched signature is UB)
>
> The Autoconf-generated code is never executed, so this is not a runtime
> issue; it's merely an issue of undefined behavior during translation.

FWIW, the only thing the (Clang) compiler is aware of is translation.
So from the frontend perspective, we can't tell the difference between
"trust me this is safe because it never gets executed" and "this is a
CVE". We believe the runtime behavior is sufficiently dangerous to
warrant a conservative view that any call to a function will be a call
that gets executed at runtime, hence a definitive signature mismatch
is something we feel comfortable diagnosing (in some form) by default.

> A
> problem could occur with a picky compiler or linker that rejects modules
> with mismatched function type declarations. Does Clang do that, or
> require or use such a linker? If not, there's no practical problem here.
> If so, it'd be helpful if Clang continued to support its traditional
> behavior that doesn't reject Autoconf's test cases, and for this to be
> the default.

Clang doesn't require such a linker (we work with various system linkers).

> Autoconf arose because one cannot ask something like "Can I call the
> renameat2 function with its usual signature?" in standard C code and one
> must escape into something like the shell to handle such questions. C23
> and GCC and Clang have added a few features to answer such questions,
> such as __has_include. But these features don't go nearly far enough.
> For example, __has_include tells me only whether the include file
>  exists; it won't tell me whether I can successfully include
> , or whether  will declare the function 'bar', or whether
> 'bar' will have a signature compatible with my code's calls to 'bar'.
>
> If I could request a single thing from the C23/GCC/Clang side, I'd ask
> for better facilities to be able to ask such questions within C code,
> without using the shell. Then a good chunk of Autoconf could dry up and
> blow away.
>
> I realize that I'm asking for a lot. For example, a traditional
> implementation cannot answer the renameat2 question without consulting
> the linker. That being said, this ability is essential for modular
> programming at the low level, and if compilers don't provide this
> ability Autoconf will simply have to do the best it can, regardless of
> whether it generates source code that relies on undefined behavior.

This would be challenging for an implementation like Clang where we
work with an arbitrary C runtime library (which may be dynamically
loaded on the target machine) and an arbitrary linker.

~Aaron

Re: On time64 and Large File Support

[Andreas K. Huettel]

> >
> > Proposal: glibc gains two new build-time configure options:
> > * --enable-hard-time64
> > * --enable-hard-lfs
> 
> We should define new target triplets for this if it's really required.
> 

That doesn't really help anyone *but* Debian ...

> We need to support legacy binaries on i386.  Few libraries are
> explicitly dual-ABI.  Whether it's safe to switch libraries above
> glibc to LFS or time64 needs to be evaluated on a per-library
> basis.  For most distributions, no one is going to do that work,
> and we have to stick to whathever we are building today.

... since for Debian the libraries with different ABI end up in different
multiarch paths then.

Anyone with a more, ahem, standard filesystem arrangement has to find
a different solution for the problem of legacy binaries.

-- 
Andreas K. Hüttel
dilfri...@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

signature.asc
Description: This is a digitally signed message part.

Re: On time64 and Large File Support

[Florian Weimer]

* Sam James:

>> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
>> 
>> * Sam James:
>> 
>>> In Gentoo, we've been planning out what we should do for time64 on
>>> glibc [0] and concluded that we need some support in glibc for a newer
>>> option. I'll outline why below.
>>> 
>>> Proposal: glibc gains two new build-time configure options:
>>> * --enable-hard-time64
>>> * --enable-hard-lfs
>> 
>> We should define new target triplets for this if it's really required.
>
> I hadn't considered that option. I'll reflect on it. Please let me know
> if you have further thoughts on this.
>
> But that said, these binaries are broken anyway in 2038?

No, I expect users to run them in time-shifted VMs or containers.
Wrong dates are better than no ability to run anything at all.

And whoever can recompile to switch to time64 can just recompile for a
64-bit target.  There are some embedded users that stick to 32-bit for
cost savings, but I think the cost allocation is quite wrong: They save
a bit on per-unit costs, but they do not really contribute back to GNU
(and most don't even use glibc, although there is some use out there).

>> We need to support legacy binaries on i386.  Few libraries are
>> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
>> to LFS or time64 needs to be evaluated on a per-library basis.  For most
>> distributions, no one is going to do that work, and we have to stick to
>> whathever we are building today.
>
> While I agree, I don't think it's as well-known that it should be that
> these are ABI breaking and require inspection. It's being done ad-hoc
> or in many cases, not at all.
>
> Part of the use of this thread is, if nothing else, we can show upstreams
> and other distros It if they're confused.
>
> It's very easy to miss that a package has started enabling LFS
> and then your opportunity to catch the ABI breakage is gone.
>
> It doesn't help that I (and I suspect most distribution maintainers)
> do all development on x86_64 and hence even ABI diffing isn't
> going to notice. You have to specifically diff the build system, which I
> do, but it's not easy if it's buried deep within gnulib or something.

I really assumed that setting the default in glibc would solve this for
everyone: binary distributions keep using time32, and source-based
embedded distributions can switch to time64 if they want to. *sigh*

I mean, we have things like more compact stack usage through certain
ABI-breaking GCC options.  The kernel can use those safely, but few
people attempt to apply them to userspace.  There, having the right
default in the toolchain is sufficient.  I didn't expect time64 to be
different.

Thanks,
Florian

Re: On time64 and Large File Support

[Florian Weimer]

* Andreas K. Huettel:

>> >
>> > Proposal: glibc gains two new build-time configure options:
>> > * --enable-hard-time64
>> > * --enable-hard-lfs
>> 
>> We should define new target triplets for this if it's really required.
>> 
>
> That doesn't really help anyone *but* Debian ...
>
>> We need to support legacy binaries on i386.  Few libraries are
>> explicitly dual-ABI.  Whether it's safe to switch libraries above
>> glibc to LFS or time64 needs to be evaluated on a per-library
>> basis.  For most distributions, no one is going to do that work,
>> and we have to stick to whathever we are building today.
>
> ... since for Debian the libraries with different ABI end up in different
> multiarch paths then.

I didn't expect co-installability as a requirement.  But yes, if that's
the goal, we need non-overlapping paths.

> Anyone with a more, ahem, standard filesystem arrangement has to find
> a different solution for the problem of legacy binaries.

We can have lib, lib64, libx32, and lib32t quite easily, that's not the
problem.  What's missing is ldconfig support.  The previous three x86
architectures have ELF-level selectors; we might need something special
there as well.

Debian does not have a multi-arch ldconfig, either.  Their path layout
isn't really ideal for that because it lacks a marker directory like
glibc-hwcaps that would allow ldconfig to build the cache from file
system content without knowledge of the exact architecture list.

Maybe I can get justification for upstreaming some form of multi-arch
support in the toolchain.  But I find it difficult to make this a top
priority.  (We currently use the upstream path layout in our
distributions.)

Thanks,
Florian

Re: On time64 and Large File Support

[Florian Weimer]

* Paul Eggert:

> On 2022-11-11 01:19, Florian Weimer wrote:
>
>> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
>> for Fedora unfortunately.
>> I thought the gnulib change has been reverted?
>
> I'm not sure where you got that impression. Bleeding-edge (unreleased)
> Autoconf AC_SYS_LARGEFILE, along with Gnulib AS_SYS_LARGEFILE as
> shipped in several GNU apps already, default to 64-bit time_t on
> platforms where setting _TIME_BITS=64 changes time_t from 32 to 64
> bits.  (This is not the same as defaulting to AC_SYS_YEAR2038 which
> *requires* wider-than-32-bit time_t, but I expect the difference
> doesn't matter here.)

Ugh.

>> I really wish the rest of GNU would talk to glibc maintainers before
>> overriding glibc maintainer decisions.  If we cannot revert this in
>> autoconf (and gnulib), this will very much endanger the Fedora i386
>> port.  Debian will probably be impacted in the same way.
>
> I'm not sure what is meant by "overriding", as Autoconf etc. are
> merely using a documented glibc feature. Also, the apps in question
> can be configured to stick with 32-bit time_t by using "./configure 
> --disable-year2038" and/or setting the corresponding cache variables,
>   so distros continue to have a choice about which time_t flavor they
>  prefer.

There are many documented toolchain features that change ABI.  It's up
to developers to determine whether it's safe to enable them.  For Y2038
support, that's obviously quite desirable in principle.  But we (glibc
upstream) looked at it and concluded that we just can't introduce it by
default and maintain backwards compatibility with existing binaries,
which is of paramount importance for i386.

I never expected GNU to switch i386 to time64 by default.  Attempting
this switch is a huge distraction and prevents many of us from working
on more important matters that benefit GNU in much more direct ways.

> What I'm gathering from your email is that 32-bit Fedora x86 needs an
> easy way to say "hold on, I want 32-bit time_t to be the default for
> all 'configure' runs". If the --disable-year2038 option of 'configure'
> isn't enough, and/or setting the appropriate cache variables isn't
> enough, what other configuration method would you like?

I don't know yet.  Fortunately, any issues should be quite visibile in
the distribution DWARF data.  If we put the time32 switch in place, we
should be able to tell whether it's effective enough in practice.

> Also, how does this issue with 64-bit time_t differ from the
> decades-old issue with 64-bit off_t? AC_SYS_LARGEFILE has long
> defaulted to 64-bit off_t, and Autoconf-generated configure scripts
> have long had --disable-largefile options and related cache variables
> much the same way that they're now dealing with 64-bit time_t. Is the
> difference merely that time_t is more widely used than off_t, so the
> ABI problems are more likely now?

LFS issues have been with us for a long time, and packages and
distributions have workarounds for it (e.g., using off64_t or long long
in public headers).  What we have today mostly works.  But it's unknown
whether existing AC_SYS_LARGEFILE users require any additional work for
time64 changes.  It's also not clear how to approach this from an
individual upstream perspective if different distributions have
different requirements.  I just don't see many libraries adopting a
dual-ABI approach like glibc did, or spending much work on this.

Thanks,
Florian

Re: On time64 and Large File Support

[Richard Purdie]

On Fri, 2022-11-11 at 08:38 +, Sam James wrote:
> In Gentoo, we've been planning out what we should do for time64 on glibc [0]
> and concluded that we need some support in glibc for a newer option. I'll 
> outline
> why below.
> 
> Proposal: glibc gains two new build-time configure options:
> * --enable-hard-time64
> * --enable-hard-lfs
> 
> These would hard-enable the relevant #defines within glibc's headers and 
> ensure that any
> binaries built with such a glibc have both Large File Support (LFS) and 
> time64 support.
> 
> I've come to the conclusion it's infeasible to try handle the migration 
> piecemeal. Various
> mismatches can and will occur (and while it's more likely with time64, it's 
> possible with LFS
> too) [1].

As a data point, Yocto Project has been debating this a bit. Based upon
what I've seen so far, I reached a similar conclusion, that we probably
needed to "just switch". As a source based system, the situation is
quite different to distros with binary feeds though.

My questions are around how much warning we can give people where code
is doing something it shouldn't and I don't have a good feel for how
much risk we're putting onto users with existing runtimes.

Cheers,

Richard

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-11 01:19, Florian Weimer wrote:


AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
for Fedora unfortunately.

I thought the gnulib change has been reverted?


I'm not sure where you got that impression. Bleeding-edge (unreleased) 
Autoconf AC_SYS_LARGEFILE, along with Gnulib AS_SYS_LARGEFILE as shipped 
in several GNU apps already, default to 64-bit time_t on platforms where 
setting _TIME_BITS=64 changes time_t from 32 to 64 bits.  (This is not 
the same as defaulting to AC_SYS_YEAR2038 which *requires* 
wider-than-32-bit time_t, but I expect the difference doesn't matter here.)




I really wish the rest of GNU would talk to glibc maintainers before
overriding glibc maintainer decisions.  If we cannot revert this in
autoconf (and gnulib), this will very much endanger the Fedora i386
port.  Debian will probably be impacted in the same way.


I'm not sure what is meant by "overriding", as Autoconf etc. are merely 
using a documented glibc feature. Also, the apps in question can be 
configured to stick with 32-bit time_t by using "./configure 
--disable-year2038" and/or setting the corresponding cache variables, so 
distros continue to have a choice about which time_t flavor they prefer.


What I'm gathering from your email is that 32-bit Fedora x86 needs an 
easy way to say "hold on, I want 32-bit time_t to be the default for all 
'configure' runs". If the --disable-year2038 option of 'configure' isn't 
enough, and/or setting the appropriate cache variables isn't enough, 
what other configuration method would you like?


Also, how does this issue with 64-bit time_t differ from the decades-old 
issue with 64-bit off_t? AC_SYS_LARGEFILE has long defaulted to 64-bit 
off_t, and Autoconf-generated configure scripts have long had 
--disable-largefile options and related cache variables much the same 
way that they're now dealing with 64-bit time_t. Is the difference 
merely that time_t is more widely used than off_t, so the ABI problems 
are more likely now?

Re: On time64 and Large File Support

[Sam James]

> On 11 Nov 2022, at 09:19, Florian Weimer  wrote:
> 
> * Sam James:
> 
>> In Gentoo, we've been planning out what we should do for time64 on
>> glibc [0] and concluded that we need some support in glibc for a newer
>> option. I'll outline why below.
>> 
>> Proposal: glibc gains two new build-time configure options:
>> * --enable-hard-time64
>> * --enable-hard-lfs
> 
> We should define new target triplets for this if it's really required.

I hadn't considered that option. I'll reflect on it. Please let me know
if you have further thoughts on this.

But that said, these binaries are broken anyway in 2038?

> We need to support legacy binaries on i386.  Few libraries are
> explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
> to LFS or time64 needs to be evaluated on a per-library basis.  For most
> distributions, no one is going to do that work, and we have to stick to
> whathever we are building today.

While I agree, I don't think it's as well-known that it should be that
these are ABI breaking and require inspection. It's being done ad-hoc
or in many cases, not at all.

Part of the use of this thread is, if nothing else, we can show upstreams
and other distros It if they're confused.

It's very easy to miss that a package has started enabling LFS
and then your opportunity to catch the ABI breakage is gone.

It doesn't help that I (and I suspect most distribution maintainers)
do all development on x86_64 and hence even ABI diffing isn't
going to notice. You have to specifically diff the build system, which I
do, but it's not easy if it's buried deep within gnulib or something.

> 
>> These would hard-enable the relevant #defines within glibc's headers
>> and ensure that any binaries built with such a glibc have both Large
>> File Support (LFS) and time64 support.
>> 
>> I've come to the conclusion it's infeasible to try handle the
>> migration piecemeal. Various mismatches can and will occur (and while
>> it's more likely with time64, it's possible with LFS too) [1].
>> 
>> We're now (possibly) on the eve of an autoconf 2.72 release which contains 
>> two changes
>> of note [2][3]
>> 1. addition of a new AC_SYS_YEAR2038 macro;
>> 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
>> 
>> Indeed, the gnulib version of change #2 is exactly how we ended up with
>> wget/gnutls breaking [1]. I feel this shows that the only approach
>> "supported" by glibc right now is untenable.
> 
> AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
> for Fedora unfortunately.
> 
> I thought the gnulib change has been reverted?
> 
> I really wish the rest of GNU would talk to glibc maintainers before
> overriding glibc maintainer decisions.  If we cannot revert this in
> autoconf (and gnulib), this will very much endanger the Fedora i386
> port.  Debian will probably be impacted in the same way.
> 

Right, we need to be unified on this, because it's confusing enough
without unilateralism.



signature.asc
Description: Message signed with OpenPGP

Re: On time64 and Large File Support

[Sam James]

> On 11 Nov 2022, at 09:16, Paul Eggert  wrote:
> 
> On 2022-11-11 00:38, Sam James wrote:
>> All that to say, I don't propose making these options unconditional,
>> because I think the boat has sailed as of glibc-2.34 [4], and I think
>> it's fair that autoconf keep the behaviour as described in git master
>> right now given the situation with glibc, but I don't think it's
>> a wise path for most distributions to follow.
> As a practical matter, I expect that each distro will have to do a big-bang 
> move, assuming the distro want to support traditional 32-bit platforms at 
> all. It makes little sense to try to have some programs and libraries with 
> 32-bit time_t, while others have 64-bit time_t. Just switch to 64-bit time_t 
> everywhere.
> 
> This is not something distros can put off for long. We're only 15 years from 
> when 32-bit time_t stops working. If distros plan to to support traditional 
> 32-bit platforms, they really need to do the big-bang move soon. They can do 
> it by predefining _TIME_BITS=64 and _FILE_OFFSET_BITS=64, or by using the new 
> Autoconf macros, or whatever.
> 
> One possible way forward would be for glibc to change its defaults for 
> _FILE_OFFSET_BITS and _TIME_BITS to be 64, in the next major release. This 
> would make it easier to do a big-bang switch, which is what people need to do 
> anyway. The backward-compatibility argument for defaulting these to 32 is 
> making less and less sense as time goes on.

+1.

I completely agree and I've reached the same conclusion. My suggestion for 
configure args
was to be a bit more gentle and avoid controversy, but really, your suggestion 
would work and would
force distros to handle it at the same time, which is best for users.

(And I really did try to make piecemeal work, but I've decided it can't.)

I think we're at risk of distros either putting this off or equivocating which
just harms our users. I should've spoken up at the time of 2.34.

FWIW, musl did this and it really was for the best: 
https://musl.libc.org/time64.html.



signature.asc
Description: Message signed with OpenPGP

Re: On time64 and Large File Support

[Florian Weimer]

* Sam James:

> In Gentoo, we've been planning out what we should do for time64 on
> glibc [0] and concluded that we need some support in glibc for a newer
> option. I'll outline why below.
>
> Proposal: glibc gains two new build-time configure options:
> * --enable-hard-time64
> * --enable-hard-lfs

We should define new target triplets for this if it's really required.

We need to support legacy binaries on i386.  Few libraries are
explicitly dual-ABI.  Whether it's safe to switch libraries above glibc
to LFS or time64 needs to be evaluated on a per-library basis.  For most
distributions, no one is going to do that work, and we have to stick to
whathever we are building today.

> These would hard-enable the relevant #defines within glibc's headers
> and ensure that any binaries built with such a glibc have both Large
> File Support (LFS) and time64 support.
>
> I've come to the conclusion it's infeasible to try handle the
> migration piecemeal. Various mismatches can and will occur (and while
> it's more likely with time64, it's possible with LFS too) [1].
>
> We're now (possibly) on the eve of an autoconf 2.72 release which contains 
> two changes
> of note [2][3]
> 1. addition of a new AC_SYS_YEAR2038 macro;
> 2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.
>
> Indeed, the gnulib version of change #2 is exactly how we ended up with
> wget/gnutls breaking [1]. I feel this shows that the only approach
> "supported" by glibc right now is untenable.

AC_SYS_LARGEFILE defaulting to AC_SYS_YEAR2038 is extremely destructive
for Fedora unfortunately.

I thought the gnulib change has been reverted?

I really wish the rest of GNU would talk to glibc maintainers before
overriding glibc maintainer decisions.  If we cannot revert this in
autoconf (and gnulib), this will very much endanger the Fedora i386
port.  Debian will probably be impacted in the same way.

> On reflection and after extensive discussion within Gentoo (although
> I don't seek to speak for everybody there) - with special thanks to
> David Seifert and Arsen Arsenović for tolerating my bikesheds on this,
> we don't think it's feasible to handle this in a piecemeal fashion -
> at the very least not without spending a significant & for some,
> undesirable amount of time on supporting "obsolete" 32-bit platforms.
>
> Right now, we're forcing the year2038 autoconf cache variable off
> to avoid more gnutls/wget instances and plan to do a hard-rebuild (new
> set of "profiles" in Gentoo parlance) for 32-bit systems that
> enable this. This will be difficult to do properly without having
> significant stragglers without some support in glibc as proposed.

Fedora will have to implement some sort of override as well.  We can't
do a hard rebuild because of compatibility requirements.  There are old
binaries people still use, and contemporary binaries built on relatively
current distributions (distributions that will never switch i386 ABI).

> All that to say, I don't propose making these options unconditional,
> because I think the boat has sailed as of glibc-2.34 [4], and I think
> it's fair that autoconf keep the behaviour as described in git master
> right now given the situation with glibc, but I don't think it's
> a wise path for most distributions to follow.

Hopefully the autoconf change can still be reverted.

Thanks,
Florian

Re: On time64 and Large File Support

[Paul Eggert]

On 2022-11-11 00:38, Sam James wrote:

All that to say, I don't propose making these options unconditional,
because I think the boat has sailed as of glibc-2.34 [4], and I think
it's fair that autoconf keep the behaviour as described in git master
right now given the situation with glibc, but I don't think it's
a wise path for most distributions to follow.
As a practical matter, I expect that each distro will have to do a 
big-bang move, assuming the distro want to support traditional 32-bit 
platforms at all. It makes little sense to try to have some programs and 
libraries with 32-bit time_t, while others have 64-bit time_t. Just 
switch to 64-bit time_t everywhere.


This is not something distros can put off for long. We're only 15 years 
from when 32-bit time_t stops working. If distros plan to to support 
traditional 32-bit platforms, they really need to do the big-bang move 
soon. They can do it by predefining _TIME_BITS=64 and 
_FILE_OFFSET_BITS=64, or by using the new Autoconf macros, or whatever.


One possible way forward would be for glibc to change its defaults for 
_FILE_OFFSET_BITS and _TIME_BITS to be 64, in the next major release. 
This would make it easier to do a big-bang switch, which is what people 
need to do anyway. The backward-compatibility argument for defaulting 
these to 32 is making less and less sense as time goes on.

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Sam James]

> On 10 Nov 2022, at 17:16, Zack Weinberg  wrote:
> 
> I’m the closest thing Autoconf has to a lead maintainer at present.
> 
> It’s come to my attention (via https://lwn.net/Articles/913505/ and
> https://fedoraproject.org/wiki/Changes/PortingToModernC) that GCC and
> Clang both plan to disable several “legacy” C language features by
> default in a near-future release (GCC 14, Clang 16) (see the Fedora
> wiki link for a list).  I understand that this change potentially
> breaks a lot of old dusty code, and in particular that
> Autoconf-generated configure scripts use constructs that may *silently
> give the wrong answer to a probe* when a stricter compiler is in use.

Thank you for asking. The fixes in git get us there, I think, as far
as you can, with the exception of the stuff you (and I) mention below.

> 
> [...]
> 
> The biggest remaining (potential) problem, that I’m aware of, is that
> AC_CHECK_FUNC unconditionally declares the function we’re probing for
> as ‘char NAME (void)’, and asks the compiler to call it with no
> arguments, regardless of what its prototype actually is.  It is not
> clear to me whether this will still work with the planned changes to
> the compilers.  Both GCC 12 and Clang 14 have on-by-default warnings
> triggered by ‘extern char memcpy(void);’ (or any other standard
> library function whose prototype is coded into the compiler) and this
> already causes problems for people who run configure scripts with
> CC='cc -Werror'.  Unfortunately this is very hard to fix — we would
> have to build a comprehensive list of library functions into Autoconf,
> mapping each to either its documented prototype or to a header where
> it ought to be declared; in the latter case we would also have to make
> e.g. AC_CHECK_FUNCS([getaddrinfo]) imply AC_CHECK_HEADERS([sys/types.h
> sys/socket.h netdb.h]) which might mess up configure scripts that
> aren’t expecting headers to be probed at that point.
> 
> How important do you think it is for this to be fixed?
> 
> Are there any other changes you would like to see in a near-future
> Autoconf 2.72 in order to make this transition easier?

This might be a WONTFIX but let me mention it just for
the record:
1. AC_CHECK_FUNCS is, as you've noticed, very noisy.

I would support having a hardcoded list for certain CHOSTs
as Rich suggests to reduce noise, but I don't think we can
do this accurately very quickly.

I think for Gentoo's efforts, I might just build up a set
of cache variables for glibc/musl on each arch to
reduce the noise in logs. But that's time consuming
and brittle still, so I'm not sure.

(Note that Gentoo and Fedora are taking two complementary
but different approaches here:
we're diffing config.logs and other compiler
output, in addition to build logs, while Florian for Fedora
Is building a list of functions which *we know* are available
In a specific environment and patching gcc to spit out
logs when something in said list is missing. This mitigates
noise for things like functions in libbsd, which I'm finding
a bit of a pain.)

I'll see what others say.

2. I often have to set the following cache variables to
reduce noise in logs:
* ac_cv_c_undeclared_builtin_options="none needed"
* ac_cv_header_sys_types_h_makedev=no
* gl_cv_compiler_check_decl_option="-Werror=implicit-function-declaration" 
(obviously this is gnulib)
* gl_cv_minmax_in_limits_h=no

I don't know if we can do anything to make these tests smarter
or just leave it as-is. It's fine if we can't, as exporting the cache
vars is not a bad workaround for us when doing testing.

> 
> zw
> 
> p.s. GCC and Clang folks: As long as you’re changing the defaults out
> from under people, can you please also remove the last few predefined
> user-namespace macros (-Dlinux, -Dunix, -Darm, etc) from all the
> -std=gnuXX modes?

I support this as well. This kind of thing has led to endless
bugs in userland, see https://reviews.llvm.org/D137511.



signature.asc
Description: Message signed with OpenPGP

Re: How can Autoconf help with the transition to stricter compilation defaults?

[Sam James]

> On 11 Nov 2022, at 03:33, Zack Weinberg  wrote:
> 
> On Thu, Nov 10, 2022, at 10:08 PM, Sam James wrote:
>>> On 10 Nov 2022, at 21:10, Michael Orlitzky  wrote:
>>> While everyone else is discussing big ideas, it would be helpful for me
>>> personally if autoconf just made a release with the latest bugfixes.
>> 
>> Before I dive into the rest of this thread: yes, this is one of
>> my main thoughts on the matter. Autoconf has a huge network
>> effect problem and letting the existing fixes start to propagate
>> would be most helpful.
> 
> It would be relatively easy for me to take a couple hours this weekend and 
> put out a 2.72 release with everything that's already in trunk and nothing 
> else.  Anyone have reasons I _shouldn't_ do that?
> [...]
> 
> I have not been following the y2038 work closely.  Is it going to affect 
> things in a good way or a bad way??

I've started a discussion on libc-alpha about this, but I think it depends on 
how you view
the migration. I've come to the conclusion it's probably good but only after 
thinking
about it a lot. I wish it'd been discussed on the mailing lists first, as it's 
not obvious
that it's okay, and I'm not sure if others will even share my view.

Let's have the conversation there as it'll be easier to track.

Thanks for prompting me to write this up finally.


signature.asc
Description: Message signed with OpenPGP

On time64 and Large File Support

[Sam James]

Hi all,

In Gentoo, we've been planning out what we should do for time64 on glibc [0]
and concluded that we need some support in glibc for a newer option. I'll 
outline
why below.

Proposal: glibc gains two new build-time configure options:
* --enable-hard-time64
* --enable-hard-lfs

These would hard-enable the relevant #defines within glibc's headers and ensure 
that any
binaries built with such a glibc have both Large File Support (LFS) and time64 
support.

I've come to the conclusion it's infeasible to try handle the migration 
piecemeal. Various
mismatches can and will occur (and while it's more likely with time64, it's 
possible with LFS
too) [1].

We're now (possibly) on the eve of an autoconf 2.72 release which contains two 
changes
of note [2][3]
1. addition of a new AC_SYS_YEAR2038 macro;
2. making AC_SYS_LARGEFILE change behaviour to imply AC_SYS_YEAR2038.

Indeed, the gnulib version of change #2 is exactly how we ended up with
wget/gnutls breaking [1]. I feel this shows that the only approach
"supported" by glibc right now is untenable.

On reflection and after extensive discussion within Gentoo (although
I don't seek to speak for everybody there) - with special thanks to
David Seifert and Arsen Arsenović for tolerating my bikesheds on this,
we don't think it's feasible to handle this in a piecemeal fashion -
at the very least not without spending a significant & for some,
undesirable amount of time on supporting "obsolete" 32-bit platforms.

Right now, we're forcing the year2038 autoconf cache variable off
to avoid more gnutls/wget instances and plan to do a hard-rebuild (new
set of "profiles" in Gentoo parlance) for 32-bit systems that
enable this. This will be difficult to do properly without having
significant stragglers without some support in glibc as proposed.

All that to say, I don't propose making these options unconditional,
because I think the boat has sailed as of glibc-2.34 [4], and I think
it's fair that autoconf keep the behaviour as described in git master
right now given the situation with glibc, but I don't think it's
a wise path for most distributions to follow.

See also a previous discussion on libc-alpha [5].

What do you think?

[0] https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration
[1] https://bugs.gentoo.org/828001
[2] 
https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=f6657256a37da44c987c04bf9cd75575dfca3b60
[3] 
https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=bc66c26f488975ea9ad22033b9fa28809f4bf65e
[4] https://sourceware.org/pipermail/libc-alpha/2021-August/129718.html
[5] https://sourceware.org/pipermail/libc-alpha/2022-January/134985.html

best,
sam


signature.asc
Description: Message signed with OpenPGP