bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Mathieu Lirzin]

Hello,

Mathieu Lirzin  writes:

>>From 83d5d37bc8f0adb0e20a6fe7ab68029d2479dd32 Mon Sep 17 00:00:00 2001
> From: Mathieu Lirzin 
> Date: Thu, 18 Jan 2018 11:19:13 +0100
> Subject: [PATCH] tests: Don't check 'Getopt::Long' corner cases
>
> Depending on the installed 'Getopt::Long' perl module, command-line
> handling may vary a bit.  As a consequence we prefer not to check
> command-line corners cases.  This change fixes automake bug#29638.
>
> * t/aclocal.sh (am_create_testdir): Don't expect "--versi" to be
> interpreted as "--version".
> * t/automake-cmdline.tap: Don't expect "--vers" to be interpreted as
> "--version" and things after "--" to be interpreted as file arguments.
> (do_check): Display the actual command output.
> * t/maken3.sh (check_targets): "--force" is not a documented option, so
> don't use it.
> ---

I have pushed this in commit 903a80e0def90b88c1e4eead353af126a31a5422.

Feel free to reopen this bug if the problem persists.

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Mathieu Lirzin]

Hello,

Mathieu Lirzin  writes:

> Dennis Clarke  writes:
>
>> The five failed tests are :
>>
>> FAIL: t/aclocal.sh
>> FAIL: t/automake-cmdline.tap 4 - list of options terminated by '--' (stderr)
>> FAIL: t/automake-cmdline.tap 17 - unambiguous incomplete long option
>> FAIL: t/maken3.sh
>> FAIL: t/maken3-w.sh
>
> My impression is that those failing tests are checking the edge cases
> of Getopt::Long which is system dependent and not the functional
> behavior of Automake.  As a consequence it seems reasonable to narrow
> the tests to more conservative Getopt::Long behaviors.

Here is a patch that should fix this issue.

>From 83d5d37bc8f0adb0e20a6fe7ab68029d2479dd32 Mon Sep 17 00:00:00 2001
From: Mathieu Lirzin 
Date: Thu, 18 Jan 2018 11:19:13 +0100
Subject: [PATCH] tests: Don't check 'Getopt::Long' corner cases

Depending on the installed 'Getopt::Long' perl module, command-line
handling may vary a bit.  As a consequence we prefer not to check
command-line corners cases.  This change fixes automake bug#29638.

* t/aclocal.sh (am_create_testdir): Don't expect "--versi" to be
interpreted as "--version".
* t/automake-cmdline.tap: Don't expect "--vers" to be interpreted as
"--version" and things after "--" to be interpreted as file arguments.
(do_check): Display the actual command output.
* t/maken3.sh (check_targets): "--force" is not a documented option, so
don't use it.
---
 t/aclocal.sh   |  2 --
 t/automake-cmdline.tap | 13 ++---
 t/maken3.sh|  2 +-
 3 files changed, 3 insertions(+), 14 deletions(-)

diff --git a/t/aclocal.sh b/t/aclocal.sh
index 8cc8d5cc3..008493d5d 100644
--- a/t/aclocal.sh
+++ b/t/aclocal.sh
@@ -58,6 +58,4 @@ cat stderr >&2
 grep 'unrecognized option.*--ver' stderr
 grep '[Tt]ry.*--help.*for more information' stderr
 
-$ACLOCAL --versi
-
 :
diff --git a/t/automake-cmdline.tap b/t/automake-cmdline.tap
index c4441efe6..306231faa 100644
--- a/t/automake-cmdline.tap
+++ b/t/automake-cmdline.tap
@@ -18,7 +18,7 @@
 
 . test-init.sh
 
-plan_ 17
+plan_ 14
 
 # Usage: bad_cmdline DESCRIPTION REGEX-FOR-STDERR [ARGS-FOR-AUTOMAKE...]
 do_check ()
@@ -28,18 +28,11 @@ do_check ()
   regex=$1; shift
   AUTOMAKE_fails -d "$desc (run)" -- "$@"
   command_ok_ "$desc (stderr)" grep "$regex" stderr
+  cat stderr
 }
 
 do_check 'invalid long option' 'unrecognized option.*--voo' --voo
 
-# Older perl has a buggy Getopt::Long which makes this fail.
-if $PERL -e 'require 5.8.2;'; then
-  do_check "list of options terminated by '--'" \
-   'input file.*--voo' -- --voo
-else
-  skip_row_ 2 -r "older perl with buggy Getopt::Long"
-fi
-
 do_check "empty argument" \
  'empty argument' ''
 
@@ -58,6 +51,4 @@ do_check "'--help' as option argument" \
 do_check "ambiguous incomplete option" \
  'unrecognized option.*--ver' --ver
 
-command_ok_ "unambiguous incomplete long option" $AUTOMAKE --vers
-
 :
diff --git a/t/maken3.sh b/t/maken3.sh
index c37743cb7..8fe1d3269 100644
--- a/t/maken3.sh
+++ b/t/maken3.sh
@@ -181,7 +181,7 @@ check_targets || exit 1
 # TODO: add BUILT_SOURCES to sub2, fix fallout.
 sed 's/##//' < Makefile.am > t
 mv -f t Makefile.am
-$AUTOMAKE -Wno-override --force Makefile
+$AUTOMAKE -Wno-override Makefile
 ./configure
 check_targets || exit 1
 
-- 
2.15.1


Can you confirm this works on your system?

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Eric Blake]

On 01/04/2018 09:08 PM, Mathieu Lirzin wrote:
>> If I understand GNU Coding Standards, we really do want to make sure
>> unambiguous abbreviations of long options work.  
> 
> I am unaware of such GCS recommandation.  Do you have a pointer to the
> part of the standards suggesting that?

Hmm. I just re-read
https://www.gnu.org/prep/standards/standards.html#Command_002dLine-Interfaces,
and all I can see is that it recommends:

"Please define long-named options that are equivalent to the
single-letter Unix-style options. We hope to make GNU more user friendly
this way. This is easy to do with the GNU function getopt_long."

and then I extrapolated that since getopt_long() recognizes unambiguous
abbreviations, anything else used instead of getopt_long() should do
likewise.  But you're right that it does not seem to be an explicit
requirement, so much as an ease-of-use and consistency issue.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Mathieu Lirzin]

Eric Blake  writes:

> On 01/04/2018 07:49 PM, Mathieu Lirzin wrote:
>
>> for example from Automake 1.15.1 build directory the following command
>> is supposed to work:
>> 
>> --8<---cut here---start->8---
>> $ t/wrap/automake-1.15 --vers
>> automake (GNU automake) 1.15.1
>> Copyright (C) 2017 Free Software Foundation, Inc.
>> License GPLv2+: GNU GPL version 2 or later 
>> 
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.
>> 
>> Written by Tom Tromey 
>>and Alexandre Duret-Lutz .
>> --8<---cut here---end--->8---
>> 
>> According to your logs this doesn't work on your system.  My impression
>> is that those failing tests are checking the edge cases of Getopt::Long
>> which is system dependent and not the functional behavior of Automake.
>> As a consequence it seems reasonable to narrow the tests to more
>> conservative Getopt::Long behaviors.
>> 
>> WDYT?
>
> If I understand GNU Coding Standards, we really do want to make sure
> unambiguous abbreviations of long options work.  

I am unaware of such GCS recommandation.  Do you have a pointer to the
part of the standards suggesting that?

> I'd argue that if not all versions of perl Getopt::Long are working
> the way the testsuite currently expects, that we should instead keep
> the test unchanged and find ways to work around the broken perl module
> versions (perhaps by manually specifying all abbreviations as explicit
> options ourselves, rather than relying on Getopt::Long to do it for
> us).

This could indeed be done, however I am not convinced by the usefulness
of such workaround.

> At the same time, once we do ascertain which version of
> Getopt::Long you are using, it may be worth reporting the flaw in that
> version to your distro vendor, as Automake is not the only software
> that would have to work around that particular weakness.

Agreed.

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Eric Blake]

On 01/04/2018 07:49 PM, Mathieu Lirzin wrote:

> for example from Automake 1.15.1 build directory the following command
> is supposed to work:
> 
> --8<---cut here---start->8---
> $ t/wrap/automake-1.15 --vers
> automake (GNU automake) 1.15.1
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv2+: GNU GPL version 2 or later 
> 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Written by Tom Tromey 
>and Alexandre Duret-Lutz .
> --8<---cut here---end--->8---
> 
> According to your logs this doesn't work on your system.  My impression
> is that those failing tests are checking the edge cases of Getopt::Long
> which is system dependent and not the functional behavior of Automake.
> As a consequence it seems reasonable to narrow the tests to more
> conservative Getopt::Long behaviors.
> 
> WDYT?

If I understand GNU Coding Standards, we really do want to make sure
unambiguous abbreviations of long options work.  I'd argue that if not
all versions of perl Getopt::Long are working the way the testsuite
currently expects, that we should instead keep the test unchanged and
find ways to work around the broken perl module versions (perhaps by
manually specifying all abbreviations as explicit options ourselves,
rather than relying on Getopt::Long to do it for us).  At the same time,
once we do ascertain which version of Getopt::Long you are using, it may
be worth reporting the flaw in that version to your distro vendor, as
Automake is not the only software that would have to work around that
particular weakness.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Mathieu Lirzin]

Hello,

Dennis Clarke  writes:

> The same five tests fail for automake 1.15 and 1.15.1 on RHEL 7.4. Also
> the testsuite itself reports mysterious "Error 1" and "Error 2" upon
> termination and that looks questionable :
>
> 
> Testsuite summary for GNU Automake 1.15
> 
> # TOTAL: 2693
> # PASS:  2422
> # SKIP:  226
> # XFAIL: 40
> # FAIL:  5
> # XPASS: 0
> # ERROR: 0
> 
> See ./test-suite.log
> Please report to bug-automake@gnu.org
> 
> gmake[2]: *** [Makefile:3027: test-suite.log] Error 1
> gmake[2]: Leaving directory
> '/usr/local/build/automake-1.15_3.10.0-693.11.1.el7.x86_64.001'
> gmake[1]: *** [Makefile:3135: check-TESTS] Error 2
> gmake[1]: Leaving directory
> '/usr/local/build/automake-1.15_3.10.0-693.11.1.el7.x86_64.001'
> gmake: *** [Makefile:3366: check-am] Error 2
> admsys@sedna$
>
>
> The five failed tests are :
>
> FAIL: t/aclocal.sh
> FAIL: t/automake-cmdline.tap 4 - list of options terminated by '--' (stderr)
> FAIL: t/automake-cmdline.tap 17 - unambiguous incomplete long option
> FAIL: t/maken3.sh
> FAIL: t/maken3-w.sh

The common characteristic of those failures is the command line
handling.

Automake use Perl provided Getopt::Long module to parse the command line
arguments and the version you are using seems to behave differently than
usual.

for example from Automake 1.15.1 build directory the following command
is supposed to work:

--8<---cut here---start->8---
$ t/wrap/automake-1.15 --vers
automake (GNU automake) 1.15.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later 

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Tom Tromey 
   and Alexandre Duret-Lutz .
--8<---cut here---end--->8---

According to your logs this doesn't work on your system.  My impression
is that those failing tests are checking the edge cases of Getopt::Long
which is system dependent and not the functional behavior of Automake.
As a consequence it seems reasonable to narrow the tests to more
conservative Getopt::Long behaviors.

WDYT?

Sorry for the delay.

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#29638: Same five tests fail with 1.15 on RHEL 7.4

[Dennis Clarke]

The same five tests fail for automake 1.15 and 1.15.1 on RHEL 7.4. Also
the testsuite itself reports mysterious "Error 1" and "Error 2" upon
termination and that looks questionable :


Testsuite summary for GNU Automake 1.15

# TOTAL: 2693
# PASS:  2422
# SKIP:  226
# XFAIL: 40
# FAIL:  5
# XPASS: 0
# ERROR: 0

See ./test-suite.log
Please report to bug-automake@gnu.org

gmake[2]: *** [Makefile:3027: test-suite.log] Error 1
gmake[2]: Leaving directory 
'/usr/local/build/automake-1.15_3.10.0-693.11.1.el7.x86_64.001'

gmake[1]: *** [Makefile:3135: check-TESTS] Error 2
gmake[1]: Leaving directory 
'/usr/local/build/automake-1.15_3.10.0-693.11.1.el7.x86_64.001'

gmake: *** [Makefile:3366: check-am] Error 2
admsys@sedna$


The five failed tests are :

FAIL: t/aclocal.sh
FAIL: t/automake-cmdline.tap 4 - list of options terminated by '--' (stderr)
FAIL: t/automake-cmdline.tap 17 - unambiguous incomplete long option
FAIL: t/maken3.sh
FAIL: t/maken3-w.sh

The test suite log is attached as 
test-suite_gnu_automake_1.15_rhel7.4_kern_3.10.0-693.11.1.el7.x86_64.log.xz


Dennis Clarke





test-suite_gnu_automake_1.15_rhel7.4_kern_3.10.0-693.11.1.el7.x86_64.log.xz
Description: application/xz