bug#7849: new instspc* test failures

[Stefano Lattarini]

tags 7849 patch
close 7849
thanks

On 01/05/2012 10:53 AM, Peter Rosin wrote:
> Stefano Lattarini skrev 2012-01-05 09:38:
>> Hi Peter, thanks for the patch.  Looks good, modulo a couple of nits
>> below.  Feel free to push to master when they have been addressed.
> 
> Pushed with nits addressed.  I'm not sure about the strange/strangely
> thing either, but strangely seems safer so...
> 
> The test no longer errors out on MSYS, it's all PASS, XFAIL and SKIPs.
>
Very good!  Then I'm closing the bug report.

Thank you very much for working on this,
  Stefano

bug#7849: new instspc* test failures

[Peter Rosin]

Stefano Lattarini skrev 2012-01-05 09:38:
> Hi Peter, thanks for the patch.  Looks good, modulo a couple of nits
> below.  Feel free to push to master when they have been addressed.

Pushed with nits addressed.  I'm not sure about the strange/strangely
thing either, but strangely seems safer so...

The test no longer errors out on MSYS, it's all PASS, XFAIL and SKIPs.

Cheers,
Peter

bug#7849: new instspc* test failures

[Stefano Lattarini]

Hi Peter, thanks for the patch.  Looks good, modulo a couple of nits
below.  Feel free to push to master when they have been addressed.

On 01/05/2012 01:35 AM, Peter Rosin wrote:
>
> From 27100f0b94f8e38e8bd30c27277d7ad4e9f4dd1a Mon Sep 17 00:00:00 2001
> From: Peter Rosin 
> Date: Thu, 5 Jan 2012 01:29:27 +0100
> Subject: [PATCH] tests: work around strangeness in MSYS
> 
> MSYS mishandles carriage returns and behaves very strange for
>
s/strange/strangely/?  Not a rhetorical question -- "strangely" seems
more correct to me, but I'm not sure it really is.

> directories with colon in them. It seems that colon-directories are
> somehow mixed up with drive letters.
> 
> Fixes automake bug#7849.
> 
> * tests/instspc.tap: Skip instead of erroring out when $test_string
> is empty for the carriageret case, as that is expected on MSYS. Also,
> for similar reasons, skip instead of erroring out when it is not
> possible to cd into the just created directory, and the directory
> name contains a colon.
>
I assume that, after your changes, this test doesn't error out anymore
and doesn't experience any failure even on MSYS.  Right?  If there are
still failures, it might be worth stating it in the commit message.

> ---
>  tests/instspc.tap |   28 +++-
>  1 files changed, 27 insertions(+), 1 deletions(-)
> 
> diff --git a/tests/instspc.tap b/tests/instspc.tap
> index 9eb145f..2e9641c 100755
> --- a/tests/instspc.tap
> +++ b/tests/instspc.tap
>
The copyright years should be updated (the `update-copyright' script
from gnulib can help you with this chore).

> [SNIP]

Thanks,
  Stefano

bug#7849: new instspc* test failures

[Peter Rosin]

Stefano Lattarini skrev 2012-01-04 21:43:
> Hi Peter, thanks for the super-quick feedback.
> 
> On 01/04/2012 03:17 PM, Peter Rosin wrote:
>> Stefano Lattarini skrev 2012-01-04 13:24:
>>> (Me rummaging in older bug reports ...)
>>>
>>> Reference:
>>>   
>>>
>>> On 01/16/2011 09:46 AM, Ralf Wildenhues wrote:
 There are a number of new failures since the splitup of instspc.test.
 Most seem to stem from increased coverage.  Rather than putting each
 failure in a different PR, I'm listing a number of them here; we can
 still split things out further later if that proves necessary.

 On MinGW, instspc-carriageret-* fail because the CR character is not
 treated correctly by the shell and/or other tools like sed.  I don't
 think this is fixable without fixing the shell.

 On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
 binary in a directory named '...' fails (causing the "compiler works"
 test to fail at configure time):

 $ mkdir ...
 $ cd ...
 $ cp /usr/bin/ls.exe .
 $ ./ls.exe
 bash: ./ls.exe: No such file or directory

>>> Can anyone still reproduce such failures with the latest version of
>>> `instspc.tap' from master?  If yes, can anyone suggest some workarounds
>>> for failures that are due to bugs/limitations in Cygwin and MSYS rather
>>> than to automake bugs?
>>
>> Cygwin runs the test ok these days, but not MSYS/MinGW.
>> MSYS still stumbles on carriageret and the only workaround I could
>> think of is to check if the current test is carriageret and
>> skip instead of fail if $test_string is empty in that case.
>> But that's quite ugly,
>>
> Ah, but no more ugly than many other workarounds we have to avoid spurious
> testsuite failure (if you want to see real horrors, take a look at, e.g.,
> 'uninstall-fail.test', or at 'tap-signal.tap' before commit d47115).

I'd rather not... :-)

>>eval "test_string=\${instspc__$test_name}" \
>>  && test x"$test_string" != x \
>> -|| fatal_ "invalid test name: '$test_name'"
>> +|| {
>> +  test x"$test_name" != xcarriageret \
>> +&& fatal_ "invalid test name: '$test_name'"
>> +  skip_ "CR probably treated as NL: '$test_name'"
>>
> You need to skip *two* tests here, otherwise the driver will complain that the
> number of tests run doesn't match the TAP plan.  Also, no need to relax the

Ah, right.

> sanity check on the `eval' (that should always work also on  MinGW).  I'd go
> for something like this in the end (untested!):
> 
>eval "test_string=\${instspc__$test_name}" \
>  || fatal_ "invalid test name: '$test_name'"
> 
>if test x"$test_string" != x; then
>  if test x"$test_name" != xcarriageret; then
>fatal_ "invalid test name: '$test_name'"
>  else
># MinGW/MSYS version x.y still mishandles carriage returns; see
># automake bug#7849.
>skip_ -r "carriage-return treated as null char" "$test_name in 
> builddir"
>skip_ -r "carriage-return treated as null char" "$test_name in 
> destddir"
>continue
>  fi
>fi

Right, but I fixed a couple of things.

>> and doesn't really bring us much as the test falls over again on
>> dosdrive aka 'a:'.
>>
> Well, it will bring us something once we've fixed also this error :-)
> 
>> On MSYS, "mkdir ./a:" creates
>> the directory "a", w/o the trailing colon. Marvelous.
>>
> Indeed.
> 
>> So, next (ugly as hell) thing to do is to try to create a dir with
>> only a colon if there is any colon in $test_string and skip if that
>> fails.
>>
> Hmm...  what about this simpler sanity check instead?
> 
>   -mkdir "./$test_string" || {
>   +# On MSYS, "mkdir ./a:" creates the directory "a", without the trailing
>   +# colon.  Marvelous.  So, even if mkdir succeeds, we must still check that
>   +# the create directory has the expected name.
>   +mkdir "./$test_string" && test -d "./$test-string" || {
>  skip_ -r "mkdir failed" "$test_name in builddir"
>  skip_ -r "mkdir failed" "$test_name in destdir"
>   ...
> 
> Would this work too?

No, "test -d ..." succeeds. New attempt attached, see below.

>> If you're not busy puking all over the place after reading the
>> patch, you might be able to convince me to write a commit message
>> for it...
>>
> Pretty please? ;-)
> 
>> The test no longer "fatals" on MSYS/MingW with these changes
>> (instspc: exit 0).
>>
> Probably you already know, but I'd rather stress this anyway, just to be sure:
> the TAP-based test scripts do *not* report failure/success through their final
> exit status (any non-zero exit status will be flagged as an hard-error in 
> fact);
> the report of test results will be done with special "directives" emitted on
> the stdout.  So the above result doesn't mean that `instspc.tap' is now 
> passing,
> but only that it is not encountering unexpected errors anymore.
> 
> To correctly execute a TAP-based script and get its res

bug#7849: new instspc* test failures

[Stefano Lattarini]

Hi Peter, thanks for the super-quick feedback.

On 01/04/2012 03:17 PM, Peter Rosin wrote:
> Stefano Lattarini skrev 2012-01-04 13:24:
>> (Me rummaging in older bug reports ...)
>>
>> Reference:
>>   
>>
>> On 01/16/2011 09:46 AM, Ralf Wildenhues wrote:
>>> There are a number of new failures since the splitup of instspc.test.
>>> Most seem to stem from increased coverage.  Rather than putting each
>>> failure in a different PR, I'm listing a number of them here; we can
>>> still split things out further later if that proves necessary.
>>>
>>> On MinGW, instspc-carriageret-* fail because the CR character is not
>>> treated correctly by the shell and/or other tools like sed.  I don't
>>> think this is fixable without fixing the shell.
>>>
>>> On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
>>> binary in a directory named '...' fails (causing the "compiler works"
>>> test to fail at configure time):
>>>
>>> $ mkdir ...
>>> $ cd ...
>>> $ cp /usr/bin/ls.exe .
>>> $ ./ls.exe
>>> bash: ./ls.exe: No such file or directory
>>>
>> Can anyone still reproduce such failures with the latest version of
>> `instspc.tap' from master?  If yes, can anyone suggest some workarounds
>> for failures that are due to bugs/limitations in Cygwin and MSYS rather
>> than to automake bugs?
> 
> Cygwin runs the test ok these days, but not MSYS/MinGW.
> MSYS still stumbles on carriageret and the only workaround I could
> think of is to check if the current test is carriageret and
> skip instead of fail if $test_string is empty in that case.
> But that's quite ugly,
>
Ah, but no more ugly than many other workarounds we have to avoid spurious
testsuite failure (if you want to see real horrors, take a look at, e.g.,
'uninstall-fail.test', or at 'tap-signal.tap' before commit d47115).

>eval "test_string=\${instspc__$test_name}" \
>  && test x"$test_string" != x \
> -|| fatal_ "invalid test name: '$test_name'"
> +|| {
> +  test x"$test_name" != xcarriageret \
> +&& fatal_ "invalid test name: '$test_name'"
> +  skip_ "CR probably treated as NL: '$test_name'"
>
You need to skip *two* tests here, otherwise the driver will complain that the
number of tests run doesn't match the TAP plan.  Also, no need to relax the
sanity check on the `eval' (that should always work also on  MinGW).  I'd go
for something like this in the end (untested!):

   eval "test_string=\${instspc__$test_name}" \
 || fatal_ "invalid test name: '$test_name'"

   if test x"$test_string" != x; then
 if test x"$test_name" != xcarriageret; then
   fatal_ "invalid test name: '$test_name'"
 else
   # MinGW/MSYS version x.y still mishandles carriage returns; see
   # automake bug#7849.
   skip_ -r "carriage-return treated as null char" "$test_name in builddir"
   skip_ -r "carriage-return treated as null char" "$test_name in destddir"
   continue
 fi
   fi

> and doesn't really bring us much as the test falls over again on
> dosdrive aka 'a:'.
>
Well, it will bring us something once we've fixed also this error :-)

> On MSYS, "mkdir ./a:" creates
> the directory "a", w/o the trailing colon. Marvelous.
>
Indeed.

> So, next (ugly as hell) thing to do is to try to create a dir with
> only a colon if there is any colon in $test_string and skip if that
> fails.
>
Hmm...  what about this simpler sanity check instead?

  -mkdir "./$test_string" || {
  +# On MSYS, "mkdir ./a:" creates the directory "a", without the trailing
  +# colon.  Marvelous.  So, even if mkdir succeeds, we must still check that
  +# the create directory has the expected name.
  +mkdir "./$test_string" && test -d "./$test-string" || {
 skip_ -r "mkdir failed" "$test_name in builddir"
 skip_ -r "mkdir failed" "$test_name in destdir"
  ...

Would this work too?

> If you're not busy puking all over the place after reading the
> patch, you might be able to convince me to write a commit message
> for it...
>
Pretty please? ;-)

> The test no longer "fatals" on MSYS/MingW with these changes
> (instspc: exit 0).
>
Probably you already know, but I'd rather stress this anyway, just to be sure:
the TAP-based test scripts do *not* report failure/success through their final
exit status (any non-zero exit status will be flagged as an hard-error in fact);
the report of test results will be done with special "directives" emitted on
the stdout.  So the above result doesn't mean that `instspc.tap' is now passing,
but only that it is not encountering unexpected errors anymore.

To correctly execute a TAP-based script and get its results, you must either
run it through the automake-provided harness:

  $ make check TESTS=instspc.tap

or use a proper third-party TAP runner:

  $ prove --merge ./instspc.tap


Thanks,
  Stefano

bug#7849: new instspc* test failures

[Stefano Lattarini]

On 01/04/2012 03:35 PM, Peter Rosin wrote:
> Peter Rosin skrev 2012-01-04 15:17:
>>mkdir "./$test_string" || {
>>  skip_ -r "mkdir failed" "$test_name in builddir"
>>  skip_ -r "mkdir failed" "$test_name in destdir"
> 
> BTW, what's with the -r in the above "skip_"s?
> 
It is to give the reason of the SKIP (the other argument being the name
of the test); note that this makes sense only for TAP based tests.  For
example, on a system that cannot handle a `|' in a test name, the above
will give lines like this in the "make check" output:

  ...
  SKIP: instspc.tap 31 - pipe in builddir # SKIP mkdir failed
  SKIP: instspc.tap 32 - pipe in destdir # SKIP mkdir failed
  ...

The exact details are only in the code and comments in
`tests/tap-functions.sh'; sorry.

Regards,
  Stefano

bug#7849: new instspc* test failures

[Peter Rosin]

Peter Rosin skrev 2012-01-04 15:17:
> Stefano Lattarini skrev 2012-01-04 13:24:
>> (Me rummaging in older bug reports ...)
>>
>> Reference:
>>   
>>
>> On 01/16/2011 09:46 AM, Ralf Wildenhues wrote:
>>> There are a number of new failures since the splitup of instspc.test.
>>> Most seem to stem from increased coverage.  Rather than putting each
>>> failure in a different PR, I'm listing a number of them here; we can
>>> still split things out further later if that proves necessary.
>>>
>>> On MinGW, instspc-carriageret-* fail because the CR character is not
>>> treated correctly by the shell and/or other tools like sed.  I don't
>>> think this is fixable without fixing the shell.
>>>
>>> On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
>>> binary in a directory named '...' fails (causing the "compiler works"
>>> test to fail at configure time):
>>>
>>> $ mkdir ...
>>> $ cd ...
>>> $ cp /usr/bin/ls.exe .
>>> $ ./ls.exe
>>> bash: ./ls.exe: No such file or directory
>>>
>> Can anyone still reproduce such failures with the latest version of
>> `instspc.tap' from master?  If yes, can anyone suggest some workarounds
>> for failures that are due to bugs/limitations in Cygwin and MSYS rather
>> than to automake bugs?
> 
> Cygwin runs the test ok these days, but not MSYS/MinGW.
> MSYS still stumbles on carriageret and the only workaround I could
> think of is to check if the current test is carriageret and
> skip instead of fail if $test_string is empty in that case.
> But that's quite ugly, and doesn't really bring us much as the test
> falls over again on dosdrive aka 'a:'. On MSYS, "mkdir ./a:" creates
> the directory "a", w/o the trailing colon. Marvelous.
> So, next (ugly as hell) thing to do is to try to create a dir with
> only a colon if there is any colon in $test_string and skip if that
> fails.
> 
> If you're not busy puking all over the place after reading the
> patch, you might be able to convince me to write a commit message
> for it...
> 
> The test no longer "fatals" on MSYS/MingW with these changes
> (instspc: exit 0).
> 
> Cheers,
> Peter
> 
> diff --git a/tests/instspc.tap b/tests/instspc.tap
> index 9eb145f..b408c16 100755
> --- a/tests/instspc.tap
> +++ b/tests/instspc.tap
> @@ -236,11 +236,26 @@ for test_name in $test_names_list; do
> 
>eval "test_string=\${instspc__$test_name}" \
>  && test x"$test_string" != x \
> -|| fatal_ "invalid test name: '$test_name'"
> +|| {
> +  test x"$test_name" != xcarriageret \
> +&& fatal_ "invalid test name: '$test_name'"
> +  skip_ "CR probably treated as NL: '$test_name'"
> +  continue
> +}
> 
># Skip the next checks if this system doesn't support the required
># characters in file names.
> 
> +  case $test_string in
> +  *:*)
> +if mkdir "./:"; then
> +  rmdir "./:"
> +else
> +  test_string=':'
> +fi
> +;;
> +  esac
> +
>mkdir "./$test_string" || {
>  skip_ -r "mkdir failed" "$test_name in builddir"
>  skip_ -r "mkdir failed" "$test_name in destdir"

BTW, what's with the -r in the above "skip_"s?

Cheers,
Peter

bug#7849: new instspc* test failures

[Peter Rosin]

Stefano Lattarini skrev 2012-01-04 13:24:
> (Me rummaging in older bug reports ...)
> 
> Reference:
>   
> 
> On 01/16/2011 09:46 AM, Ralf Wildenhues wrote:
>> There are a number of new failures since the splitup of instspc.test.
>> Most seem to stem from increased coverage.  Rather than putting each
>> failure in a different PR, I'm listing a number of them here; we can
>> still split things out further later if that proves necessary.
>>
>> On MinGW, instspc-carriageret-* fail because the CR character is not
>> treated correctly by the shell and/or other tools like sed.  I don't
>> think this is fixable without fixing the shell.
>>
>> On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
>> binary in a directory named '...' fails (causing the "compiler works"
>> test to fail at configure time):
>>
>> $ mkdir ...
>> $ cd ...
>> $ cp /usr/bin/ls.exe .
>> $ ./ls.exe
>> bash: ./ls.exe: No such file or directory
>>
> Can anyone still reproduce such failures with the latest version of
> `instspc.tap' from master?  If yes, can anyone suggest some workarounds
> for failures that are due to bugs/limitations in Cygwin and MSYS rather
> than to automake bugs?

Cygwin runs the test ok these days, but not MSYS/MinGW.
MSYS still stumbles on carriageret and the only workaround I could
think of is to check if the current test is carriageret and
skip instead of fail if $test_string is empty in that case.
But that's quite ugly, and doesn't really bring us much as the test
falls over again on dosdrive aka 'a:'. On MSYS, "mkdir ./a:" creates
the directory "a", w/o the trailing colon. Marvelous.
So, next (ugly as hell) thing to do is to try to create a dir with
only a colon if there is any colon in $test_string and skip if that
fails.

If you're not busy puking all over the place after reading the
patch, you might be able to convince me to write a commit message
for it...

The test no longer "fatals" on MSYS/MingW with these changes
(instspc: exit 0).

Cheers,
Peter

diff --git a/tests/instspc.tap b/tests/instspc.tap
index 9eb145f..b408c16 100755
--- a/tests/instspc.tap
+++ b/tests/instspc.tap
@@ -236,11 +236,26 @@ for test_name in $test_names_list; do

   eval "test_string=\${instspc__$test_name}" \
 && test x"$test_string" != x \
-|| fatal_ "invalid test name: '$test_name'"
+|| {
+  test x"$test_name" != xcarriageret \
+&& fatal_ "invalid test name: '$test_name'"
+  skip_ "CR probably treated as NL: '$test_name'"
+  continue
+}

   # Skip the next checks if this system doesn't support the required
   # characters in file names.

+  case $test_string in
+  *:*)
+if mkdir "./:"; then
+  rmdir "./:"
+else
+  test_string=':'
+fi
+;;
+  esac
+
   mkdir "./$test_string" || {
 skip_ -r "mkdir failed" "$test_name in builddir"
 skip_ -r "mkdir failed" "$test_name in destdir"

bug#7849: new instspc* test failures

[Stefano Lattarini]

(Me rummaging in older bug reports ...)

Reference:
  

On 01/16/2011 09:46 AM, Ralf Wildenhues wrote:
> There are a number of new failures since the splitup of instspc.test.
> Most seem to stem from increased coverage.  Rather than putting each
> failure in a different PR, I'm listing a number of them here; we can
> still split things out further later if that proves necessary.
> 
> On MinGW, instspc-carriageret-* fail because the CR character is not
> treated correctly by the shell and/or other tools like sed.  I don't
> think this is fixable without fixing the shell.
> 
> On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
> binary in a directory named '...' fails (causing the "compiler works"
> test to fail at configure time):
> 
> $ mkdir ...
> $ cd ...
> $ cp /usr/bin/ls.exe .
> $ ./ls.exe
> bash: ./ls.exe: No such file or directory
> 
Can anyone still reproduce such failures with the latest version of
`instspc.tap' from master?  If yes, can anyone suggest some workarounds
for failures that are due to bugs/limitations in Cygwin and MSYS rather
than to automake bugs?

Thanks,
  Stefano

bug#7849: new instspc* test failures

[Ralf Wildenhues]

* Ralf Wildenhues wrote on Sun, Jan 16, 2011 at 09:46:26AM CET:
> On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
> binary in a directory named '...' fails (causing the "compiler works"
> test to fail at configure time):
> 
> $ mkdir ...
> $ cd ...
> $ cp /usr/bin/ls.exe .
> $ ./ls.exe
> bash: ./ls.exe: No such file or directory

This has now been reported upstream:
http://thread.gmane.org/gmane.os.cygwin/124707

bug#7849: new instspc* test failures

[Ralf Wildenhues]

There are a number of new failures since the splitup of instspc.test.
Most seem to stem from increased coverage.  Rather than putting each
failure in a different PR, I'm listing a number of them here; we can
still split things out further later if that proves necessary.


On MinGW, instspc-carriageret-* fail because the CR character is not
treated correctly by the shell and/or other tools like sed.  I don't
think this is fixable without fixing the shell.


On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
binary in a directory named '...' fails (causing the "compiler works"
test to fail at configure time):

$ mkdir ...
$ cd ...
$ cp /usr/bin/ls.exe .
$ ./ls.exe
bash: ./ls.exe: No such file or directory

I still need to report this upstream.


Cheers,
Ralf