Re: rhel8 test failure confirmation?
Mike Frysinger wrote:
On 02 Dec 2023 17:07, Jacob Bachmeyer wrote:
Mike Frysinger wrote:
On 06 Apr 2023 21:29, Jacob Bachmeyer wrote:
Karl Berry wrote:
jb> a more thorough test would locate the autom4te script and grep it
for the perllibdir that was substituted when autoconf was
configured.
I guess that would work.
Challenge accepted. Here's a refined version: (lines \-folded for email)
if $PERL -I${autom4te_perllibdir:-$(sed -n \
'/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
<$(command -v autom4te))} -MAutom4te::FileUtils \
-e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
# autom4te uses Time::HiRes
else
# autom4te does not use Time::HiRes
fi
this doesn't work on systems that wrap `autom4te`. [...]
[...]
so i don't know why we would need to set/export autom4te_perllibdir in our
wrapper. we've been doing this for over 20 years without ever setting that
var (or any other internal autoconf/automake var), so i'm pretty confident
our approach is OK.
No, not in the wrapper---in the automake ebuild script that runs
configure to match the autom4te that the wrapper will run. That test I
proposed checks for autom4te_perllibdir in the environment before
extracting it from autom4te precisely so distributions like Gentoo would
have a knob to turn if their packaging breaks that test.
That said, it turns out that this whole line of discussion is now a red
herring; see below.
[...]
i'm not aware of anything loading the Autom4te perl modules outside of the
autoconf projects. does that actually happen ? i don't think we want to
have automake start loading autoconf perl modules directly. going through
the CLI interface seems better at this point.
Autoconf and Automake are very closely associated; there is even some
shared code that is synchronized between them. Autom4te::FileUtils is
also Automake::FileUtils, for example.
The test we are discussing here was intended for Automake's configure
script to use to check if the installed Autoconf has high-resolution
timestamp support. It turned out that the test I wrote can give a false
positive, as some versions of other dependencies of Autom4te::FileUtils
/also/ use Time::HiRes, causing the test to correctly report that
Time::HiRes was loaded, but Autom4te::FileUtils nonetheless does not
actually use it. The test could probably be improved to fix the false
positives, but that would be getting into deep magic in Perl that might
not be fully reliable across Perl versions. (It would be necessary to
determine if (a) Time::HiRes::stat exists and (b)
Autom4te::FileUtils::stat is an alias to it. Having configure build a
special XSUB just to check this is well into "ridiculous" territory.)
As such, the Automake maintainers replaced this particular test with a
simpler test that just locates Autom4te/FileUtils.pm and greps it for
"Time::HiRes", thus the error message you received, which initially had
me confused because the test I proposed cannot produce that message as
it does not use grep.
An important bit of context to keep in mind is that I am not certain
that timestamp resolution is still a problem outside of the Automake and
Autoconf testsuites, since Autoconf and Automake now require cache files
to actually be newer than their sources and consider the cache files
stale if the timestamps are equal. This is a problem for the testsuite
because the testsuite is supposed to actually exercise the caching
mechanism, and higher-resolution timestamps can significantly reduce the
amount of time required to run the tests by reducing the delays needed
to ensure the caches will be valid.
-- Jacob
Re: rhel8 test failure confirmation?
On Sat, Dec 2, 2023, at 8:58 PM, Jacob Bachmeyer wrote: > Zack Weinberg wrote: >> Either way is no problem from my end, but it would be more work >> for automake (parsing --version output, instead of just checking the >> exit status of autom4te --assert-high-resolution-timestamp-support) >> Karl, do you have a preference here? I can make whatever you decide >> on happen, in the next couple of days. > > There would not need to be much parsing, just "automake --version | grep > HiRes" in that case, or "case `automake --version` in *HiRes*) ...;; > easc" to avoid running grep if you want. I specifically want to hear what Karl thinks. zw
Re: rhel8 test failure confirmation?
Zack Weinberg wrote: On Sat, Dec 2, 2023, at 7:33 PM, Jacob Bachmeyer wrote: Zack Weinberg wrote: Would it help if we added a command line option to autom4te that made it report whether it thought it could use high resolution timestamps? Versions of autom4te that didn't recognize this option should be conservatively assumed not to support them. Why not just add that information to the --version message? Add a "(HiRes)" tag somewhere if Time::HiRes is available? Either way is no problem from my end, but it would be more work for automake (parsing --version output, instead of just checking the exit status of autom4te --assert-high-resolution-timestamp-support) Karl, do you have a preference here? I can make whatever you decide on happen, in the next couple of days. There would not need to be much parsing, just "automake --version | grep HiRes" in that case, or "case `automake --version` in *HiRes*) ...;; easc" to avoid running grep if you want. -- Jacob
Re: rhel8 test failure confirmation?
On 02 Dec 2023 17:07, Jacob Bachmeyer wrote:
> Mike Frysinger wrote:
> > On 06 Apr 2023 21:29, Jacob Bachmeyer wrote:
> >> Karl Berry wrote:
> >>> jb> a more thorough test would locate the autom4te script and grep it
> >>> for the perllibdir that was substituted when autoconf was
> >>> configured.
> >>>
> >>> I guess that would work.
> >>>
> >> Challenge accepted. Here's a refined version: (lines \-folded for email)
> >>
> >> if $PERL -I${autom4te_perllibdir:-$(sed -n \
> >> '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}'
> >> \
> >> <$(command -v autom4te))} -MAutom4te::FileUtils \
> >> -e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
> >># autom4te uses Time::HiRes
> >> else
> >># autom4te does not use Time::HiRes
> >> fi
> >>
> >
> > this doesn't work on systems that wrap `autom4te`. Gentoo for example wraps
> > all autoconf & automake scripts to support parallel installs of different
> > versions. this way we can easily have access to every autoconf version. we
> > got this idea from Mandrake, so we aren't the only ones ;).
>
> If you install a wrapper script, (instead of, for example, making
> autom4te, etc. easily-repointable symlinks), then you must also set
symlinks are vastly inferior. they require privilege to modify, are not atomic,
and bleed into the rest of the system and other users. the wrappers we have in
Gentoo on the otherhand respect env vars (so you can e.g. WANT_AUTOCONF=2.53),
and by default scan the current set of generated files to see what was used, and
then see what's installed to automatically match. so if you're running automake
in a project built with automake 1.11 and you have that installed, we will use
it automatically for you. or you can force WANT_AUTOMAKE=latest to always use
the latest version.
> autom4te_perllibdir in the environment to the appropriate directory when
> building autoconf/automake. This (with the Gentoo-specific knowledge of
> where the active autom4te is actually located) should be easy to add to
> the ebuild.
>
> If autom4te_perllibdir is set in the environment, its value will be used
> instead of extracting that information from the autom4te script.
when e.g. autoconf 2.69 is built to use versioned paths at configure, compile,
and install time, there is no need to muck around with internal vars like the
autom4te_perllibdir. `autoconf-2.69` & `autom4te-2.69` always invokes exactly
the 2.69 version regardless of what `autoconf` does.
so i don't know why we would need to set/export autom4te_perllibdir in our
wrapper. we've been doing this for over 20 years without ever setting that
var (or any other internal autoconf/automake var), so i'm pretty confident
our approach is OK.
> > seems like the only reliable option is to invoke autom4te.
> > am_autom4te_ver=`$AUTOM4TE --version | sed -n '1{s:.*) ::;p}'
> > AS_CASE([$am_autom4te_ver],
> > ... do the matching ...
> >
> > what is the first autoconf release that has the fix ?
>
> The problem with testing autoconf versions for this is that Time::HiRes
> is an *optional* module in Perl. It was available from CPAN before it
> was bundled with Perl, and distributions technically can *unbundle* it
> from later Perl releases if they want. The only reliable way to know if
> Time::HiRes is available (without effectively reimplementing Perl's
> module search) is to try to load it. Autom4te now (correctly) uses
> Time::HiRes if it is available and falls back to Perl builtins if not,
> for any version of Perl. The best way to check if high-resolution
> timestamps are available to autom4te is to have perl load
> Autom4te::FileUtils and check if that also loaded Time::HiRes.
i'm not aware of anything loading the Autom4te perl modules outside of the
autoconf projects. does that actually happen ? i don't think we want to
have automake start loading autoconf perl modules directly. going through
the CLI interface seems better at this point.
-mike
signature.asc
Description: PGP signature
Re: rhel8 test failure confirmation?
On 02 Dec 2023 18:33, Jacob Bachmeyer wrote: > Zack Weinberg wrote: > > On Sat, Dec 2, 2023, at 6:37 PM, Karl Berry wrote: > >> The best way to check if high-resolution > >> timestamps are available to autom4te is to have perl load > >> Autom4te::FileUtils and check if that also loaded Time::HiRes. > >> > >> The problem with that turned out to be that Time::HiRes got loaded from > >> other system modules, resulting in the test thinking that autom4te used > >> it when that wasn't actually the case. That's what happened in practice > >> with your patch. > > > > Would it help if we added a command line option to autom4te that made it > > report whether it thought it could use high resolution timestamps? Versions > > of autom4te that didn't recognize this option should be conservatively > > assumed not to support them. > > Why not just add that information to the --version message? Add a > "(HiRes)" tag somewhere if Time::HiRes is available? All versions that > know to check if Time::HiRes is loaded will also know how to use it, > unlike the earlier test. parsing out the exact version from --version is already a pita. trying it freeform text that we also want tools to do substring searching on sounds extremely fragile and the opposite of future-proof. libtool at least has a --config option that dumps shell-link syntax. -mike signature.asc Description: PGP signature
Re: rhel8 test failure confirmation?
On 02 Dec 2023 19:17, Zack Weinberg wrote: > On Sat, Dec 2, 2023, at 6:37 PM, Karl Berry wrote: > > The best way to check if high-resolution > > timestamps are available to autom4te is to have perl load > > Autom4te::FileUtils and check if that also loaded Time::HiRes. > > > > The problem with that turned out to be that Time::HiRes got loaded from > > other system modules, resulting in the test thinking that autom4te used > > it when that wasn't actually the case. That's what happened in practice > > with your patch. > > Would it help if we added a command line option to autom4te that made it > report whether it thought it could use high resolution timestamps? Versions > of autom4te that didn't recognize this option should be conservatively > assumed not to support them. upthread somewhere Karl (iirc) threw out a bikeshed idea like --has=. i don't know if any of the autotools or GNU tools have such things that we could duplicate to avoid bikeshedding too much. that would make it work in this case easily, and setup framework for future checks. -mike signature.asc Description: PGP signature
Re: rhel8 test failure confirmation?
On Sat, Dec 2, 2023, at 7:33 PM, Jacob Bachmeyer wrote: > Zack Weinberg wrote: >> Would it help if we added a command line option to autom4te that made >> it report whether it thought it could use high resolution timestamps? >> Versions of autom4te that didn't recognize this option should be >> conservatively assumed not to support them. > > Why not just add that information to the --version message? Add a > "(HiRes)" tag somewhere if Time::HiRes is available? Either way is no problem from my end, but it would be more work for automake (parsing --version output, instead of just checking the exit status of autom4te --assert-high-resolution-timestamp-support) Karl, do you have a preference here? I can make whatever you decide on happen, in the next couple of days. >> (Of course there's the additional wrinkle that whether high >> resolution timestamps *work* depends on what filesystem >> autom4te.cache is stored in > > Is this actually still a problem (other than for ensuring the cache is > used in the testsuite) I don't *think* so but I don't understand the problem 100% so I could be missing something. zw
Re: rhel8 test failure confirmation?
Zack Weinberg wrote: On Sat, Dec 2, 2023, at 6:37 PM, Karl Berry wrote: The best way to check if high-resolution timestamps are available to autom4te is to have perl load Autom4te::FileUtils and check if that also loaded Time::HiRes. The problem with that turned out to be that Time::HiRes got loaded from other system modules, resulting in the test thinking that autom4te used it when that wasn't actually the case. That's what happened in practice with your patch. Would it help if we added a command line option to autom4te that made it report whether it thought it could use high resolution timestamps? Versions of autom4te that didn't recognize this option should be conservatively assumed not to support them. Why not just add that information to the --version message? Add a "(HiRes)" tag somewhere if Time::HiRes is available? All versions that know to check if Time::HiRes is loaded will also know how to use it, unlike the earlier test. (Of course there's the additional wrinkle that whether high resolution timestamps *work* depends on what filesystem autom4te.cache is stored in, but that's even harder to probe... one problem at a time?) Yes; even standard-resolution timestamps might not be "all there" with FAT and its infamous 2-second timestamp resolution. Is this actually still a problem (other than for ensuring the cache is used in the testsuite) after Bogdan's patches to require that cache files be strictly newer than their source files? -- Jacob
Re: rhel8 test failure confirmation?
On Sat, Dec 2, 2023, at 6:37 PM, Karl Berry wrote: > The best way to check if high-resolution > timestamps are available to autom4te is to have perl load > Autom4te::FileUtils and check if that also loaded Time::HiRes. > > The problem with that turned out to be that Time::HiRes got loaded from > other system modules, resulting in the test thinking that autom4te used > it when that wasn't actually the case. That's what happened in practice > with your patch. Would it help if we added a command line option to autom4te that made it report whether it thought it could use high resolution timestamps? Versions of autom4te that didn't recognize this option should be conservatively assumed not to support them. (Of course there's the additional wrinkle that whether high resolution timestamps *work* depends on what filesystem autom4te.cache is stored in, but that's even harder to probe... one problem at a time?) zw
Re: rhel8 test failure confirmation?
The best way to check if high-resolution timestamps are available to autom4te is to have perl load Autom4te::FileUtils and check if that also loaded Time::HiRes. The problem with that turned out to be that Time::HiRes got loaded from other system modules, resulting in the test thinking that autom4te used it when that wasn't actually the case. That's what happened in practice with your patch. Hence the current test that merely greps the autom4te source for HiRes, which is not sufficient, granted. But we have to do something. Distributions unbundling Time::HiRes ... at some point I don't mind saying that people have to set am_cv_sleep_fractional_seconds=false by hand. karl
Re: rhel8 test failure confirmation?
Mike Frysinger wrote:
On 06 Apr 2023 21:29, Jacob Bachmeyer wrote:
Karl Berry wrote:
jb> a more thorough test would locate the autom4te script and grep it
for the perllibdir that was substituted when autoconf was
configured.
I guess that would work.
Challenge accepted. Here's a refined version: (lines \-folded for email)
if $PERL -I${autom4te_perllibdir:-$(sed -n \
'/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
<$(command -v autom4te))} -MAutom4te::FileUtils \
-e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
# autom4te uses Time::HiRes
else
# autom4te does not use Time::HiRes
fi
this doesn't work on systems that wrap `autom4te`. Gentoo for example wraps
all autoconf & automake scripts to support parallel installs of different
versions. this way we can easily have access to every autoconf version. we
got this idea from Mandrake, so we aren't the only ones ;).
If you install a wrapper script, (instead of, for example, making
autom4te, etc. easily-repointable symlinks), then you must also set
autom4te_perllibdir in the environment to the appropriate directory when
building autoconf/automake. This (with the Gentoo-specific knowledge of
where the active autom4te is actually located) should be easy to add to
the ebuild.
If autom4te_perllibdir is set in the environment, its value will be used
instead of extracting that information from the autom4te script.
[...]
seems like the only reliable option is to invoke autom4te.
am_autom4te_ver=`$AUTOM4TE --version | sed -n '1{s:.*) ::;p}'
AS_CASE([$am_autom4te_ver],
... do the matching ...
what is the first autoconf release that has the fix ?
The problem with testing autoconf versions for this is that Time::HiRes
is an *optional* module in Perl. It was available from CPAN before it
was bundled with Perl, and distributions technically can *unbundle* it
from later Perl releases if they want. The only reliable way to know if
Time::HiRes is available (without effectively reimplementing Perl's
module search) is to try to load it. Autom4te now (correctly) uses
Time::HiRes if it is available and falls back to Perl builtins if not,
for any version of Perl. The best way to check if high-resolution
timestamps are available to autom4te is to have perl load
Autom4te::FileUtils and check if that also loaded Time::HiRes.
-- Jacob
Re: rhel8 test failure confirmation?
On Sat, Dec 2, 2023, at 6:58 AM, Mike Frysinger wrote: > On 06 Apr 2023 21:29, Jacob Bachmeyer wrote: >> Karl Berry wrote: >> > jb> a more thorough test would locate the autom4te script and grep it >> > for the perllibdir that was substituted when autoconf was >> > configured. >> > >> > I guess that would work. >> >> Challenge accepted. Here's a refined version: > > this doesn't work on systems that wrap `autom4te`. Gentoo for example wraps > all autoconf & automake scripts to support parallel installs of different > versions. this way we can easily have access to every autoconf version. we > got this idea from Mandrake, so we aren't the only ones ;). ... > what is the first autoconf release that has the fix ? > -mike 2.71 does not have the fix. Can you please test the 2.72d beta that I just put out (see https://lists.gnu.org/archive/html/autoconf/2023-11/msg0.html)? (yes, the version numbering for autoconf betas looks strange, it's a convention that predates semver and I don't have time to argue with anyone about it) zw
Re: rhel8 test failure confirmation?
On 06 Apr 2023 21:29, Jacob Bachmeyer wrote:
> Karl Berry wrote:
> > jb> a more thorough test would locate the autom4te script and grep it
> > for the perllibdir that was substituted when autoconf was
> > configured.
> >
> > I guess that would work.
>
> Challenge accepted. Here's a refined version: (lines \-folded for email)
>
> if $PERL -I${autom4te_perllibdir:-$(sed -n \
> '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
> <$(command -v autom4te))} -MAutom4te::FileUtils \
> -e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
># autom4te uses Time::HiRes
> else
># autom4te does not use Time::HiRes
> fi
this doesn't work on systems that wrap `autom4te`. Gentoo for example wraps
all autoconf & automake scripts to support parallel installs of different
versions. this way we can easily have access to every autoconf version. we
got this idea from Mandrake, so we aren't the only ones ;).
as such, running git configure produces errors:
checking for autom4te... /usr/bin/autom4te
grep: /Autom4te/FileUtils.pm: No such file or directory
checking the filesystem timestamp resolution... 2
and then breaks some tests:
$ grep ^FAIL: test-suite.log
FAIL: t/depcomp2
FAIL: t/distcheck-no-prefix-or-srcdir-override
t/depcomp2.log showing:
+ cat stderr
grep: /Autom4te/FileUtils.pm: No such file or directory
+ sed /rm:.*conftest\.dSYM/d stderr
+ test -s stderr2
+ cat stderr2
grep: /Autom4te/FileUtils.pm: No such file or directory
+ _am_exit 1
seems like the only reliable option is to invoke autom4te.
am_autom4te_ver=`$AUTOM4TE --version | sed -n '1{s:.*) ::;p}'
AS_CASE([$am_autom4te_ver],
... do the matching ...
what is the first autoconf release that has the fix ?
-mike
signature.asc
Description: PGP signature
