Re: improved Automake test for file names with funny characters

[Paul Eggert]

Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

>  PE> What changes in mkinstalldirs need to be merged into install-sh?
>  PE> (Presumably these can be folded in after the changes below are 
> installed.)
>
> I'll have to check.  I think I noticed that mkinstalldirs was
> trying `mkdir -p' before constructing the subdirectories one by
> one, while install-sh wasn't even attempting `mkdir -p'.  But your
> patch addresses this already, doesn't it?

Yes, install-sh tests whether mkdir -p works before using it.

>  PE> In the revised patch enclosed below, I'll add some commentary about
>  PE> this to help explain it.
>
> Thanks all lot!  Could you remember to bump install-sh's
> $scriptversion before you commit it ?

You're welcome.  I did that, and installed the patch into the trunk.
Thanks for looking over it.

Re: improved Automake test for file names with funny characters

[Alexandre Duret-Lutz]

>>> "PE" == Paul Eggert <[EMAIL PROTECTED]> writes:

 PE> Following up on your response to
 PE> :

 PE> Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

 >> I think some of these changes need to be synced with
 >> mkinstalldirs and vice-versa.

 PE> I thought mkinstalldirs was obsolete now?  Do we need to
 PE> continue to maintain it?

gettext still uses it.  But I agree: let's forget about it, and let
the remaining users to fix the problems they encounter if they can't switch
to `install -d' for some reason.

 PE> What changes in mkinstalldirs need to be merged into install-sh?
 PE> (Presumably these can be folded in after the changes below are installed.)

I'll have to check.  I think I noticed that mkinstalldirs was
trying `mkdir -p' before constructing the subdirectories one by
one, while install-sh wasn't even attempting `mkdir -p'.  But your
patch addresses this already, doesn't it?

[...]

 PE> In the revised patch enclosed below, I'll add some commentary about
 PE> this to help explain it.

Thanks all lot!  Could you remember to bump install-sh's
$scriptversion before you commit it ?

 >> It does not seem to be fixed on platforms where "mkdir -p" does
 >> not work.  (I'm concerned by the spurious report about test
 >> suite failures more than by the missing feature on these
 >> system.)

 PE> That should be fixed now; please see below.

 >> Does Stephan's patch for quoting $@ before eval look right to you?

 PE> No, because echo mishandles some path names.  However, there is
 PE> a better fix for the IFS problem in general, as discussed in
 PE> .
 PE> The revised patch below incorporates this idea into install-sh.

Thanks for the link.  I didn't know about `set -f'.
-- 
Alexandre Duret-Lutz

Re: improved Automake test for file names with funny characters

[Stepan Kasal]

Hello,

On Tue, Sep 06, 2005 at 12:16:31PM -0700, Paul Eggert wrote:
> Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:
> > Does Stepan's patch for quoting $@ before eval look right to you?
> 
> No, because echo mishandles some path names.  [...]

and what about this:

   d=`sed "s,',''',g;s,/,' ',g;s,^,',;s,\\\$,'," 

Re: improved Automake test for file names with funny characters

[Paul Eggert]

Following up on your response to
:

Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

> I think some of these changes need to be synced with
> mkinstalldirs and vice-versa.

I thought mkinstalldirs was obsolete now?  Do we need to continue to
maintain it?

What changes in mkinstalldirs need to be merged into install-sh?
(Presumably these can be folded in after the changes below are installed.)

> But it's not clear to me because you seems to tune it much more than
> the ChangeLog accounts for.  What's the story with all these
> test_mode/umask things ?

In the revised patch enclosed below, I'll add some commentary about
this to help explain it.

> It does not seem to be fixed on platforms where "mkdir -p" does
> not work.  (I'm concerned by the spurious report about test
> suite failures more than by the missing feature on these
> system.)

That should be fixed now; please see below.

> Does Stephan's patch for quoting $@ before eval look right to you?

No, because echo mishandles some path names.  However, there is
a better fix for the IFS problem in general, as discussed in
.
The revised patch below incorporates this idea into install-sh.

> On second though I don't buy it [the "sleep 1; test -d"]: it only
> works if both "mkdir -p" invocations are creating the same directory
> path.  It doesn't work if one create "a/b/", the other "a/c/", and
> both race on "a" as described in mkinstalldirs.

Good catch.  The enclosed revised patch addresses that as well, by
falling back on the slow mkdir method if mkdir -p fails.

This patch also fixes a typo in the doc patch I submitted last time
("length file names" -> "lengthy file names").

2005-09-06  Paul Eggert  <[EMAIL PROTECTED]>

* doc/automake.texi (limitations on file names): New section.
* lib/install-sh: Rewrite to support '*' in file names.
Also, tune so that we don't invoke so many commands in the usual case.
* tests/instspc.test: The "*" test is now fixed.

Index: doc/automake.texi
===
RCS file: /cvs/automake/automake/doc/automake.texi,v
retrieving revision 1.121
diff -p -u -r1.121 automake.texi
--- doc/automake.texi   27 Jul 2005 19:05:21 -  1.121
+++ doc/automake.texi   6 Sep 2005 19:14:46 -
@@ -253,6 +253,7 @@ Frequently Asked Questions about Automak
 * CVS:: CVS and generated files
 * maintainer-mode:: missing and AM_MAINTAINER_MODE
 * wildcards::   Why doesn't Automake support wildcards?
+* limitations on file names::   Limitations on source and installed file names
 * distcleancheck::  Files left in build directory after distclean
 * Flag Variables Ordering:: CFLAGS vs.@: AM_CFLAGS vs.@: mumble_CFLAGS
 * renamed objects:: Why are object files sometimes renamed?
@@ -8031,6 +8032,7 @@ lists.
 * CVS:: CVS and generated files
 * maintainer-mode:: missing and AM_MAINTAINER_MODE
 * wildcards::   Why doesn't Automake support wildcards?
+* limitations on file names::   Limitations on source and installed file names
 * distcleancheck::  Files left in build directory after distclean
 * Flag Variables Ordering:: CFLAGS vs.@: AM_CFLAGS vs.@: mumble_CFLAGS
 * renamed objects:: Why are object files sometimes renamed?
@@ -8357,6 +8359,69 @@ variables as far Automake is concerned.
 You can get warnings about @samp{$(wildcard ...}) constructs using the
 @option{-Wportability} flag.
 
[EMAIL PROTECTED] limitations on file names
[EMAIL PROTECTED] Limitations on file names
[EMAIL PROTECTED] file names, limitations on
+
+Automake attempts to support all kinds of file names, even those that
+contain unusual characters or are unusually long.  However, some
+limitations are imposed by the underlying operating system and tools.
+
+Most operating systems prohibit the use of the null byte in file
+names, and reserve @samp{/} as a directory separator.  Also, they
+require that file names are properly encoded for the user's locale.
+Automake is subject to these limits.
+
+Portable packages should limit themselves to @acronym{POSIX} file
+names.  These can contain @acronym{ASCII} letters and digits,
[EMAIL PROTECTED], @samp{.}, and @samp{-}.  File names consist of components
+separated by @samp{/}.  File name components cannot begin with
[EMAIL PROTECTED]
+
+Portable POSIX file names cannot contain components that exceed a
+14-byte limit, but nowadays it's normally safe to assume the
+more-generous @acronym{XOPEN} limit of 255 bytes.  @acronym{POSIX}
+limits file names to 255 bytes (@acronym{XOPEN} allows 1023 bytes),
+but you may want to limit a source tarball to file names to 99 bytes
+to avoid interoperability problems with old versions of @command{tar}.
+
+If you 

Re: improved Automake test for file names with funny characters

[Alexandre Duret-Lutz]

>>> "adl" == Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

[...]

 adl> mkinstalldirs also uses IFS the split the directory in case `mkdir -p' 
 adl> does not exist, but its `mkdir -p' check is more restrictive.  (I buy
 adl> your "sleep 1; test -d" instead.)

On second though I don't buy it: it only works if both "mkdir
-p" invocations are creating the same directory path.  It doesn't work
if one create "a/b/", the other "a/c/", and both race on "a" as
described in mkinstalldirs.
-- 
Alexandre Duret-Lutz

Re: improved Automake test for file names with funny characters

[Alexandre Duret-Lutz]

>>> "Paul" == Paul Eggert <[EMAIL PROTECTED]> writes:

 Paul> Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:
 >> I'm reluctant to chop $destdir using sed in a loop, but at least
 >> that sounds less clumsy than going into an empty subdirectory to
 >> execute set :) Any other idea ?

 Paul> Here's a fix for that problem.  The basic idea is to drop the use
 Paul> of IFS, except on ancient platforms that don't support mkdir -p.

 Paul> Also, this fix documents Automake's current limitations in this area.

Excellent! Thanks!

 Paul> And it tunes install-sh a bit.

I think some of these changes need to be synced with
mkinstalldirs and vice-versa.  But it's not clear to me because
you seems to tune it much more than the ChangeLog accounts for.
What's the story with all these test_mode/umask things ?

mkinstalldirs also uses IFS the split the directory in case `mkdir -p' 
does not exist, but its `mkdir -p' check is more restrictive.  (I buy
your "sleep 1; test -d" instead.)

 Paul> By the way, I notice that we're now using install-sh even when GNU
 Paul> install is available.  Is there a reason for that?  There are some
 Paul> security issues involved here, I'm afraid, which would make it better
 Paul> to use GNU install if available.

Automake uses install-sh systematically for installing nobase_
targets (instspc.test tests these a lot for this reason).  It
will use the install command found AC_PROG_INSTALL for other
targets.  The reason is that nobase_ targets need to be
installed with the equivalent of GNU's "install -D", which isn't
supported by other install implementations.  Sure enough, we
could have an AC_PROG_INSTALL_D check and use "install -D" when
available.

 Paul> 2005-07-20  Paul Eggert  <[EMAIL PROTECTED]>

 Paul> * doc/automake.texi (limitations on file names): New section.
 Paul> * lib/install-sh: Rewrite to support '*' in file names.
 Paul> Also, tune so that we don't invoke so many commands in the usual case.
 Paul> * tests/instspc.test: The "*" test is now fixed.

It does not seem to be fixed on platforms where "mkdir -p" does
not work.  (I'm concerned by the spurious report about test
suite failures more than by the missing feature on these
system.)  Does Stephan's patch for quoting $@ before eval look
right to you?
-- 
Alexandre Duret-Lutz

Re: improved Automake test for file names with funny characters

[Stepan Kasal]

Hello,

On Wed, Jul 20, 2005 at 01:56:35PM -0700, Paul Eggert wrote:
> Here's a fix for that problem.  The basic idea is to drop the use
> of IFS, except on ancient platforms that don't support mkdir -p.

Automake doesn't use non-GNU mkdir -p; it says that Solaris' mkdir -p
is not thread safe.  See automake/m4/mkdirp.m4

That file also says that "mkdir -p -- ." might fail if current directory
is not writable.  The original bug report was:
http://lists.gnu.org/archive/html/bug-automake/2004-07/msg00052.html
(It speaks about a setu where root cannot write to a NFS share.)

On my GNU/Linux, "cd / && mkdir -p ." succeeds, under normal user.
It would be nice to know which platform has this behaviour.

> ... drop the use of IFS, except on ...

But there are still situations where IFS is used and the wildcard expansion
hits.  Perhaps my tiny patch should be also applied, after Paul's big
patch is resolved.

> By the way, I notice that we're now using install-sh even when GNU
> install is available.  Is there a reason for that?

"install-sh -d" (without any dirname) works, while "install -d" complains.

But perhaps Automake should use GNU install in other cases.

But at least for $(mkinstalldirs) you need "install-sh -d", not
"install -d".  (See automake.in for the definition of $(mkinstalldirs).)

Or perhaps
mkinstalldirs = install -d .
would do the trick.  Unless it has the same problems as "mkdir -p .", see
above.

Have a nice day,
Stepan

Re: improved Automake test for file names with funny characters

[Paul Eggert]

Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

> I'm reluctant to chop $destdir using sed in a loop, but at least
> that sounds less clumsy than going into an empty subdirectory to
> execute set :) Any other idea ?

Here's a fix for that problem.  The basic idea is to drop the use
of IFS, except on ancient platforms that don't support mkdir -p.

Also, this fix documents Automake's current limitations in this area.
And it tunes install-sh a bit.

By the way, I notice that we're now using install-sh even when GNU
install is available.  Is there a reason for that?  There are some
security issues involved here, I'm afraid, which would make it better
to use GNU install if available.

2005-07-20  Paul Eggert  <[EMAIL PROTECTED]>

* doc/automake.texi (limitations on file names): New section.
* lib/install-sh: Rewrite to support '*' in file names.
Also, tune so that we don't invoke so many commands in the usual case.
* tests/instspc.test: The "*" test is now fixed.

Index: doc/automake.texi
===
RCS file: /cvs/automake/automake/doc/automake.texi,v
retrieving revision 1.119
diff -p -c -r1.119 automake.texi
*** doc/automake.texi   9 Jul 2005 09:30:31 -   1.119
--- doc/automake.texi   20 Jul 2005 20:54:54 -
*** Frequently Asked Questions about Automak
*** 253,258 
--- 253,259 
  * CVS:: CVS and generated files
  * maintainer-mode:: missing and AM_MAINTAINER_MODE
  * wildcards::   Why doesn't Automake support wildcards?
+ * limitations on file names::   Limitations on source and installed file names
  * distcleancheck::  Files left in build directory after distclean
  * Flag Variables Ordering:: CFLAGS vs.@: AM_CFLAGS vs.@: mumble_CFLAGS
  * renamed objects:: Why are object files sometimes renamed?
*** lists.
*** 8016,8021 
--- 8017,8023 
  * CVS:: CVS and generated files
  * maintainer-mode:: missing and AM_MAINTAINER_MODE
  * wildcards::   Why doesn't Automake support wildcards?
+ * limitations on file names::   Limitations on source and installed file names
  * distcleancheck::  Files left in build directory after distclean
  * Flag Variables Ordering:: CFLAGS vs.@: AM_CFLAGS vs.@: mumble_CFLAGS
  * renamed objects:: Why are object files sometimes renamed?
*** variables as far Automake is concerned.
*** 8342,8347 
--- 8344,8412 
  You can get warnings about @samp{$(wildcard ...}) constructs using the
  @option{-Wportability} flag.
  
+ @node limitations on file names
+ @section Limitations on file names
+ @cindex file names, limitations on
+ 
+ Automake attempts to support all kinds of file names, even those that
+ contain unusual characters or are unusually long.  However, some
+ limitations are imposed by the underlying operating system and tools.
+ 
+ Most operating systems prohibit the use of the null byte in file
+ names, and reserve @samp{/} as a directory separator.  Also, they
+ require that file names are properly encoded for the user's locale.
+ Automake is subject to these limits.
+ 
+ Portable packages should limit themselves to @acronym{POSIX} file
+ names.  These can contain @acronym{ASCII} letters and digits,
+ @samp{_}, @samp{.}, and @samp{-}.  File names consist of components
+ separated by @samp{/}.  File name components cannot begin with
+ @samp{-}.
+ 
+ Portable POSIX file names cannot contain components that exceed a
+ 14-byte limit, but nowadays it's normally safe to assume the
+ more-generous @acronym{XOPEN} limit of 255 bytes.  @acronym{POSIX}
+ limits file names to 255 bytes (@acronym{XOPEN} allows 1023 bytes),
+ but you may want to limit a source tarball to file names to 99 bytes
+ to avoid interoperability problems with old versions of @command{tar}.
+ 
+ If you depart from these rules (e.g., by using [EMAIL PROTECTED]
+ characters in file names, or by using length file names), your
+ installers may have problems for reasons unrelated to Automake.
+ However, if this does not concern you, you should know about the
+ limitations imposed by Automake itself.  These limitations are
+ undesirable, but some of them seem to be inherent to underlying tools
+ like Autoconf, Make, M4, and the shell.  They fall into three
+ categories: install directories, build directories, and file names.
+ 
+ The following characters:
+ 
+ @example
+ @r{newline} " # $ ' `
+ @end example
+ 
+ should not appear in the names of install directories.  For example,
+ the operand of @command{configure}'s @option{--prefix} option should
+ not contain these characters.
+ 
+ Build directories suffer the same limitations as install directories,
+ and in addition should not contain the following characters:
+ 
+ @example
+ & @@ \
+ @end example
+ 
+ For example, the full name of the directory containin

Re: improved Automake test for file names with funny characters

[Stepan Kasal]

Hello Alexandre,

On Tue, Jul 19, 2005 at 10:08:45PM +0200, Alexandre Duret-Lutz wrote:
> I'm reluctant to chop $destdir using sed in a loop, [...]

I think there is a possibility to call the sed only once, before the set.
Please have a look at the attached patch.
I have made some experiments on my cmdline before writing it, but I
haven't tested the patch, sorry.

What would you think about it?

A note: I think that the \\\$ can be replaced by mere $
The reason is that "$," is not expanded, as comma is not valid shell
variable name.  Can we rely on this feature?

Have a nice day,
Stepan
2005-07-20  Stepan Kasal  <[EMAIL PROTECTED]>

* lib/install-sh: Use sed and eval instead of IFS=/, to avoid word
splitting.

Index: lib/install-sh
===
RCS file: /cvs/automake/automake/lib/install-sh,v
retrieving revision 1.25
diff -u -r1.25 install-sh
--- lib/install-sh  9 Jul 2005 10:21:12 -   1.25
+++ lib/install-sh  20 Jul 2005 08:31:05 -
@@ -224,11 +224,10 @@
   -*) pathcomp=./ ;;
   *)  pathcomp= ;;
 esac
-oIFS=$IFS
-IFS=/
-set fnord $dstdir
+# We cannot use IFS=/ because wildcards are expanded after word splitting.
+d=`echo "$dstdir"|sed "s,',''',g;s,/,' ',g;s,^,',;s,\\\$,',"`
+eval set fnord "$d"
 shift
-IFS=$oIFS
 
 for d
 do

Re: improved Automake test for file names with funny characters

[Alexandre Duret-Lutz]

>>> "Paul" == Paul Eggert <[EMAIL PROTECTED]> writes:

 Paul> After the recent Autoconf patch to allow | in file names,
 Paul> and some minor cleanups, here is a revised patch (I took your last
 Paul> version and ran with it):

I've installed this patch as follow.  (I just removed '%' and
'@%:@' from the expected failures, since they now pass.)

The failure for '*' comes from the following fragment in install-sh:
   
# emulate "mkdir -p $dstdir"
case $dstdir in
  /*) pathcomp=/ ;;
  -*) pathcomp=./ ;;
  *)  pathcomp= ;;
esac
oIFS=$IFS
IFS=/
set fnord $dstdir
shift
IFS=$oIFS

for d
do
  test "x$d" = x && continue

  pathcomp=$pathcomp$d
  if test ! -d "$pathcomp"; then
$mkdirprog "$pathcomp"
# mkdir can fail with a `File exist' error in case several
# install-sh are creating the directory concurrently.  This
# is OK.
test -d "$pathcomp" || exit 1
  fi
  pathcomp=$pathcomp/
done

Where $dstdir is a path such as /usr/bin that must be split as "usr" "bin".
Unfortunately filename expansion occurs after IFS splitting, so if $dstdir 
looks like /usr/*, the * will match anything in the current directory.

I'm reluctant to chop $destdir using sed in a loop, but at least
that sounds less clumsy than going into an empty subdirectory to
execute set :) Any other idea ?  (Or do we really care?
install-sh has used IFS here for years.)


2005-07-19  Paul Eggert  <[EMAIL PROTECTED]>

* tests/instspc.test: Major rewrite to test for many other
problematic file names, e.g., '$', '"', '('.  Automake and
Autoconf can't handle many of them, so do not report a failure
if the usual candidates show up.

Index: tests/instspc.test
===
RCS file: /cvs/automake/automake/tests/instspc.test,v
retrieving revision 1.5
diff -u -r1.5 instspc.test
--- tests/instspc.test  10 Jul 2005 19:09:52 -  1.5
+++ tests/instspc.test  19 Jul 2005 19:30:05 -
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2004  Free Software Foundation, Inc.
+# Copyright (C) 2004, 2005  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -18,12 +18,13 @@
 # the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
 # Boston, MA 02110-1301, USA.
 
-# Check that installation to directory with spaces succeed.
-# Report from James Amundson.
+# Check that installation to directory with shell metacharacters succeed.
+# Original report from James Amundson about file names with spaces.
+# Other characters added by Paul Eggert.
 
 # This is mostly the same input as nobase.test, but we do not use
 # libtool libraries, because Libtool does not preserve space in
-# filenames (Issue observed with ltmain.sh (GNU libtool) 1.5a (1.1323
+# file names (Issue observed with ltmain.sh (GNU libtool) 1.5a (1.1323
 # 2003/11/10 21:06:47))
 
 
@@ -32,8 +33,7 @@
 
 set -e
 
-# Make sure this system supports spaces in filenames.
-mkdir 'a  b' || exit 77
+# Set up files that won't change each time through the loop.
 
 cat >> configure.in <<'EOF'
 AC_PROG_CC
@@ -41,6 +41,24 @@
 AC_OUTPUT
 EOF
 
+mkdir sub
+
+: > sub/base.h
+: > sub/nobase.h
+: > sub/base.dat
+: > sub/nobase.dat
+: > sub/base.sh
+: > sub/nobase.sh
+
+cat >source.c <<'EOF'
+int
+main (int argc, char **argv)
+{
+  return 0;
+}
+EOF
+cp source.c source2.c
+
 cat > Makefile.am << 'EOF'
 foodir = $(prefix)/foo
 fooexecdir = $(prefix)/foo
@@ -64,50 +82,111 @@
 sub_libbase_a_SOURCES = source.c
 sub_libnobase_a_SOURCES = source.c
 
-test-install-space: install
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.h"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.h"
-   test   -f "$(DESTDIR)/more  space/foo/base.h"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.dat"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.dat"
-   test   -f "$(DESTDIR)/more  space/foo/base.dat"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.sh"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.sh"
-   test   -f "$(DESTDIR)/more  space/foo/base.sh"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase$(EXEEXT)"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/base$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/sub/libnobase.a"
-   test ! -f "$(DESTDIR)/more  space/foo/libnobase.a"
-   test   -f "$(DESTDIR)/more  space/foo/libbase.a"
+test-install-sep: install
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.h'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.h'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.h'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.dat'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.dat'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.dat'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.sh'
+   test 

Re: improved Automake test for file names with funny characters

[Paul Eggert]

Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

> $ac_abs_srcdir, $ac_top_srcdir, and $ac_abs_top_srcdir all
> contain a '&', so the matched pattern gets inserted back into
> the output.  Then sed processes the line again and loop
> infinitely.

Thanks for investigating it.  I installed the following patch into
Autoconf to fix that problem.  This doesn't solve the problem of
arbitrary chars in those strings, but at least '&' doesn't make
'configure' loop now.

2005-07-03  Paul Eggert  <[EMAIL PROTECTED]>

* lib/autoconf/general.m4 (AC_ARG_PROGRAM): Use &, not |, in
sed substitution command, so that we allow | in program prefixes
and program suffixes.  (& is a problem anyway; we're not fixing
that here.)
* lib/autoconf/status.m4 (AC_CONFIG_FILES): Likewise, for
configure_input, top_builddir, srcdir, etc.
* lib/autotest/general.m4 (AT_INIT): Likewise, for
PATH_SEPARATOR in AUTOTEST_PATH.

Index: lib/autoconf/general.m4
===
RCS file: /cvsroot/autoconf/autoconf/lib/autoconf/general.m4,v
retrieving revision 1.873
diff -p -u -r1.873 general.m4
--- lib/autoconf/general.m4 4 Jul 2005 05:22:35 -   1.873
+++ lib/autoconf/general.m4 4 Jul 2005 06:26:00 -
@@ -1501,10 +1501,10 @@ Program names:
   --program-transform-name=PROGRAM   run sed PROGRAM on installed program names
 m4_divert_pop([HELP_BEGIN])dnl
 test "$program_prefix" != NONE &&
-  program_transform_name="s|^|$program_prefix|;$program_transform_name"
+  program_transform_name="s&^&$program_prefix&;$program_transform_name"
 # Use a double $ so make ignores it.
 test "$program_suffix" != NONE &&
-  program_transform_name="s|\$|$program_suffix|;$program_transform_name"
+  program_transform_name="s&\$&$program_suffix&;$program_transform_name"
 # Double any \ or $.  echo might interpret backslashes.
 # By default was `s,x,x', remove it if useless.
 cat <<\_ACEOF >conftest.sed
Index: lib/autoconf/status.m4
===
RCS file: /cvsroot/autoconf/autoconf/lib/autoconf/status.m4,v
retrieving revision 1.67
diff -p -u -r1.67 status.m4
--- lib/autoconf/status.m4  1 Jul 2005 08:45:22 -   1.67
+++ lib/autoconf/status.m4  4 Jul 2005 06:26:00 -
@@ -547,13 +547,13 @@ cat >>$CONFIG_STATUS <<\_ACEOF
 :t
 [/@[a-zA-Z_][a-zA-Z_0-9]*@/!b]
 dnl configure_input is a somewhat special, so we don't call AC_SUBST_TRACE.
-s|@configure_input@|$configure_input|;t t
+s&@configure_input@&$configure_input&;t t
 dnl During the transition period, this is a special case:
-s|@top_builddir@|$ac_top_builddir_sub|;t t[]AC_SUBST_TRACE([top_builddir])
+s&@top_builddir@&$ac_top_builddir_sub&;t t[]AC_SUBST_TRACE([top_builddir])
 m4_foreach([_AC_Var], [srcdir, abs_srcdir, top_srcdir, abs_top_srcdir,
builddir, abs_builddir,
abs_top_builddir]AC_PROVIDE_IFELSE([AC_PROG_INSTALL], 
[[, INSTALL]]),
-  [s|@_AC_Var@|$ac_[]_AC_Var|;t t[]AC_SUBST_TRACE(_AC_Var)
+  [s&@_AC_Var@&$ac_[]_AC_Var&;t t[]AC_SUBST_TRACE(_AC_Var)
 ])dnl
 " $ac_file_inputs m4_defn([_AC_SED_CMDS])>$tmp/out
 m4_popdef([_AC_SED_CMDS])dnl
Index: lib/autotest/general.m4
===
RCS file: /cvsroot/autoconf/autoconf/lib/autotest/general.m4,v
retrieving revision 1.185
diff -p -u -r1.185 general.m4
--- lib/autotest/general.m4 21 Jun 2005 16:03:59 -  1.185
+++ lib/autotest/general.m4 4 Jul 2005 06:26:00 -
@@ -478,7 +478,7 @@ m4_divert_push([PREPARE_TESTS])dnl
 # For embedded test suites, AUTOTEST_PATH is relative to the top level
 # of the package.  Then expand it into build/src parts, since users
 # may create executables in both places.
-AUTOTEST_PATH=`echo $AUTOTEST_PATH | sed "s,:,$PATH_SEPARATOR,g"`
+AUTOTEST_PATH=`echo $AUTOTEST_PATH | sed "s&:&$PATH_SEPARATOR&g"`
 at_path=
 _AS_PATH_WALK([$AUTOTEST_PATH $PATH],
 [test -n "$at_path" && at_path=$at_path$PATH_SEPARATOR

Re: improved Automake test for file names with funny characters

[Paul Eggert]

Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

>   4) $ht and $sp do not fail for me.  I had to remove these from both
>  lists of expected failures.  Do they fail for you?

No, not now -- now that I'm using the Autoconf patch you just posted.
(I haven't investigated why.  If you think it important I could, I guess.)

> Below is the new version.

Thanks.  After the recent Autoconf patch to allow | in file names,
and some minor cleanups, here is a revised patch (I took your last
version and ran with it):

2005-07-03  Paul Eggert  <[EMAIL PROTECTED]>

* tests/instspc.test: Major rewrite to test for many other
problematic file names, e.g., '$', '"', '('.  Automake and
Autoconf can't handle many of them, so report an expected
failure if the usual candidates show up.

I'm attaching a patch, and the new version of the file, to avoid
problems with control characters in email.

--- instspc.test.~1.3.~ 2005-05-14 13:28:55.0 -0700
+++ instspc.test2005-07-03 23:51:01.0 -0700
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2004  Free Software Foundation, Inc.
+# Copyright (C) 2004, 2005  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -18,12 +18,13 @@
 # the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
 # Boston, MA 02110-1301, USA.
 
-# Check that installation to directory with spaces succeed.
-# Report from James Amundson.
+# Check that installation to directory with shell metacharacters succeed.
+# Original report from James Amundson about file names with spaces.
+# Other characters added by Paul Eggert.
 
 # This is mostly the same input as nobase.test, but we do not use
 # libtool libraries, because Libtool does not preserve space in
-# filenames (Issue observed with ltmain.sh (GNU libtool) 1.5a (1.1323
+# file names (Issue observed with ltmain.sh (GNU libtool) 1.5a (1.1323
 # 2003/11/10 21:06:47))
 
 
@@ -32,8 +33,7 @@ required='gcc'
 
 set -e
 
-# Make sure this system supports spaces in filenames.
-mkdir 'a  b' || exit 77
+# Set up files that won't change each time through the loop.
 
 cat >> configure.in <<'EOF'
 AC_PROG_CC
@@ -41,6 +41,24 @@ AC_PROG_RANLIB
 AC_OUTPUT
 EOF
 
+mkdir sub
+
+: > sub/base.h
+: > sub/nobase.h
+: > sub/base.dat
+: > sub/nobase.dat
+: > sub/base.sh
+: > sub/nobase.sh
+
+cat >source.c <<'EOF'
+int
+main (int argc, char **argv)
+{
+  return 0;
+}
+EOF
+cp source.c source2.c
+
 cat > Makefile.am << 'EOF'
 foodir = $(prefix)/foo
 fooexecdir = $(prefix)/foo
@@ -64,50 +82,115 @@ nobase_fooexec_LIBRARIES = sub/libnobase
 sub_libbase_a_SOURCES = source.c
 sub_libnobase_a_SOURCES = source.c
 
-test-install-space: install
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.h"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.h"
-   test   -f "$(DESTDIR)/more  space/foo/base.h"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.dat"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.dat"
-   test   -f "$(DESTDIR)/more  space/foo/base.dat"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.sh"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.sh"
-   test   -f "$(DESTDIR)/more  space/foo/base.sh"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase$(EXEEXT)"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/base$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/sub/libnobase.a"
-   test ! -f "$(DESTDIR)/more  space/foo/libnobase.a"
-   test   -f "$(DESTDIR)/more  space/foo/libbase.a"
+test-install-sep: install
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.h'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.h'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.h'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.dat'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.dat'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.dat'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.sh'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.sh'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.sh'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase$(EXEEXT)'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase$(EXEEXT)'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base$(EXEEXT)'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/libnobase.a'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/libnobase.a'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/libbase.a'
 EOF
 
-mkdir sub
-
-: > sub/base.h
-: > sub/nobase.h
-: > sub/base.dat
-: > sub/nobase.dat
-: > sub/base.sh
-: > sub/nobase.sh
-
-cat >source.c <<'EOF'
-int
-main (int argc, char *argv[])
-{
-  return 0;
-}
-EOF
-cp source.c source2.c
-
 $ACLOCAL
 $AUTOCONF
 $AUTOMAKE -a
 
-mkdir build
-cd build
+# Some control characters that are white space:
+# back space, carriage return, form feed, horizontal tab, line feed, space
+bs=''
+cr='
'
+ff=''
+ht='   '

Re: improved Automake test for file names with funny characters

[Alexandre Duret-Lutz]

Hi Paul,

This patch looks really great!  Thanks for doing that.

I've changed a couple of things:
  1) exit 77 as soon as one "mkdir $file" fails (perhaps this could be
 done in a more subtle way, but I'm not sure that is really important)
  2) exit 0 if no unexpected failure or success (was that exit 77 intended?)
  3) `make file="$file"' is written as `file=$file make -e' in all the
 test suite for portability to non-GNU makes in case there are recursive 
 make invocations (`make maintainer-check' will complain about the former
 syntax)
  4) $ht and $sp do not fail for me.  I had to remove these from both
 lists of expected failures.  Do they fail for you?
 [I think this is independent from the Autoconf patch I posted.  The
 errors messages while checking for config.site do not abort configure.]

Below is the new version.  But this last point worries me.

 Paul> The worst case are two tests that cause infinite loops.  Ouch!  The
 Paul> script doesn't attempt to run those two tests.  Look for 'infinite
 Paul> loop' in the patch below.

This seems to come from this bit of config.status:

  sed "

:t
/@[a-zA-Z_][a-zA-Z_0-9]*@/!b
s|@configure_input@|$configure_input|;t t
s|@top_builddir@|$ac_top_builddir_sub|;t t
s|@srcdir@|$ac_srcdir|;t t
s|@abs_srcdir@|$ac_abs_srcdir|;t t
s|@top_srcdir@|$ac_top_srcdir|;t t
s|@abs_top_srcdir@|$ac_abs_top_srcdir|;t t
s|@builddir@|$ac_builddir|;t t
s|@abs_builddir@|$ac_abs_builddir|;t t
s|@abs_top_builddir@|$ac_abs_top_builddir|;t t
s|@INSTALL@|$ac_INSTALL|;t t
" $ac_file_inputs | sed -f "$tmp/subs-1.sed" >$tmp/out


$ac_abs_srcdir, $ac_top_srcdir, and $ac_abs_top_srcdir all
contain a '&', so the matched pattern gets inserted back into
the output.  Then sed processes the line again and loop
infinitely.

 Paul> I suppose that these lists should either be shortened, or
 Paul> documented, or both.

Agreed, both :)

 Paul> 2005-07-01  Paul Eggert  <[EMAIL PROTECTED]>

 Paul> * tests/instspc.test: Major rewrite to test for many other
 Paul> problematic file names, e.g., '$', '"', '('.  Automake and
 Paul> Autoconf can't handle many of them, so report an expected
 Paul> failure if the usual candidates show up.


Index: tests/instspc.test
===
RCS file: /cvs/automake/automake/tests/instspc.test,v
retrieving revision 1.3
diff -u -r1.3 instspc.test
--- tests/instspc.test  14 May 2005 20:28:55 -  1.3
+++ tests/instspc.test  2 Jul 2005 22:14:40 -
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2004  Free Software Foundation, Inc.
+# Copyright (C) 2004, 2005  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -18,12 +18,13 @@
 # the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
 # Boston, MA 02110-1301, USA.
 
-# Check that installation to directory with spaces succeed.
-# Report from James Amundson.
+# Check that installation to directory with shell metacharacters succeed.
+# Original report from James Amundson about file names with spaces.
+# Other characters added by Paul Eggert.
 
 # This is mostly the same input as nobase.test, but we do not use
 # libtool libraries, because Libtool does not preserve space in
-# filenames (Issue observed with ltmain.sh (GNU libtool) 1.5a (1.1323
+# file names (Issue observed with ltmain.sh (GNU libtool) 1.5a (1.1323
 # 2003/11/10 21:06:47))
 
 
@@ -32,8 +33,7 @@
 
 set -e
 
-# Make sure this system supports spaces in filenames.
-mkdir 'a  b' || exit 77
+# Set up files that won't change each time through the loop.
 
 cat >> configure.in <<'EOF'
 AC_PROG_CC
@@ -41,6 +41,24 @@
 AC_OUTPUT
 EOF
 
+mkdir sub
+
+: > sub/base.h
+: > sub/nobase.h
+: > sub/base.dat
+: > sub/nobase.dat
+: > sub/base.sh
+: > sub/nobase.sh
+
+cat >source.c <<'EOF'
+int
+main (int argc, char **argv)
+{
+  return 0;
+}
+EOF
+cp source.c source2.c
+
 cat > Makefile.am << 'EOF'
 foodir = $(prefix)/foo
 fooexecdir = $(prefix)/foo
@@ -64,50 +82,127 @@
 sub_libbase_a_SOURCES = source.c
 sub_libnobase_a_SOURCES = source.c
 
-test-install-space: install
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.h"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.h"
-   test   -f "$(DESTDIR)/more  space/foo/base.h"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.dat"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.dat"
-   test   -f "$(DESTDIR)/more  space/foo/base.dat"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.sh"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.sh"
-   test   -f "$(DESTDIR)/more  space/foo/base.sh"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase$(EXEEXT)"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/base$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/sub/libnobase.a"
-   test ! -f "$(DESTDIR)/more  space/foo/libnobase.a"
-   test   -f "$(DESTDIR)/more  space/foo/libbase.a"
+test-install-sep:

improved Automake test for file names with funny characters

[Paul Eggert]

Following up on a discussion in the Autoconf and Automake mailing
lists, here's a proposed patch to Automake to have it test better for
file names with funny characters.  If you look for the
expected_build_failures and expected_install_failures variables,
you'll see the test cases that Automake/Autoconf/Make etc. cannot
handle now.  For example, file names containing '$'.  I suppose that
these lists should either be shortened, or documented, or both.

The worst case are two tests that cause infinite loops.  Ouch!  The
script doesn't attempt to run those two tests.  Look for 'infinite
loop' in the patch below.

Watch out: there are unusual control characters in this patch.  I'll
attach a copy of the patched file too, just in case.

2005-07-01  Paul Eggert  <[EMAIL PROTECTED]>

* tests/instspc.test: Major rewrite to test for many other
problematic file names, e.g., '$', '"', '('.  Automake and
Autoconf can't handle many of them, so report an expected
failure if the usual candidates show up.


Index: tests/instspc.test
===
RCS file: /cvs/automake/automake/tests/instspc.test,v
retrieving revision 1.3
diff -p -u -r1.3 instspc.test
--- tests/instspc.test  14 May 2005 20:28:55 -  1.3
+++ tests/instspc.test  1 Jul 2005 21:49:37 -
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2004  Free Software Foundation, Inc.
+# Copyright (C) 2004, 2005  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -18,8 +18,9 @@
 # the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
 # Boston, MA 02110-1301, USA.
 
-# Check that installation to directory with spaces succeed.
-# Report from James Amundson.
+# Check that installation to directory with shell metacharacters succeed.
+# Original report from James Amundson about file names with spaces.
+# Other characters added by Paul Eggert.
 
 # This is mostly the same input as nobase.test, but we do not use
 # libtool libraries, because Libtool does not preserve space in
@@ -32,8 +33,7 @@ required='gcc'
 
 set -e
 
-# Make sure this system supports spaces in filenames.
-mkdir 'a  b' || exit 77
+# Set up files that won't change each time through the loop.
 
 cat >> configure.in <<'EOF'
 AC_PROG_CC
@@ -41,6 +41,24 @@ AC_PROG_RANLIB
 AC_OUTPUT
 EOF
 
+mkdir sub
+
+: > sub/base.h
+: > sub/nobase.h
+: > sub/base.dat
+: > sub/nobase.dat
+: > sub/base.sh
+: > sub/nobase.sh
+
+cat >source.c <<'EOF'
+int
+main (int argc, char **argv)
+{
+  return 0;
+}
+EOF
+cp source.c source2.c
+
 cat > Makefile.am << 'EOF'
 foodir = $(prefix)/foo
 fooexecdir = $(prefix)/foo
@@ -64,50 +82,135 @@ nobase_fooexec_LIBRARIES = sub/libnobase
 sub_libbase_a_SOURCES = source.c
 sub_libnobase_a_SOURCES = source.c
 
-test-install-space: install
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.h"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.h"
-   test   -f "$(DESTDIR)/more  space/foo/base.h"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.dat"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.dat"
-   test   -f "$(DESTDIR)/more  space/foo/base.dat"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase.sh"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase.sh"
-   test   -f "$(DESTDIR)/more  space/foo/base.sh"
-   test   -f "$(DESTDIR)/more  space/foo/sub/nobase$(EXEEXT)"
-   test ! -f "$(DESTDIR)/more  space/foo/nobase$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/base$(EXEEXT)"
-   test   -f "$(DESTDIR)/more  space/foo/sub/libnobase.a"
-   test ! -f "$(DESTDIR)/more  space/foo/libnobase.a"
-   test   -f "$(DESTDIR)/more  space/foo/libbase.a"
+test-install-sep: install
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.h'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.h'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.h'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.dat'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.dat'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.dat'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase.sh'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase.sh'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base.sh'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/nobase$(EXEEXT)'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/nobase$(EXEEXT)'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/base$(EXEEXT)'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/sub/libnobase.a'
+   test ! -f '$(DESTDIR)/$(file)-prefix/foo/libnobase.a'
+   test   -f '$(DESTDIR)/$(file)-prefix/foo/libbase.a'
 EOF
 
-mkdir sub
-
-: > sub/base.h
-: > sub/nobase.h
-: > sub/base.dat
-: > sub/nobase.dat
-: > sub/base.sh
-: > sub/nobase.sh
-
-cat >source.c <<'EOF'
-int
-main (int argc, char *argv[])
-{
-  return 0;
-}
-EOF
-cp source.c source2.c
-
 $ACLOCAL
 $AUTOCONF
 $AUTOMAKE -a
 
-mkdir build
-cd build
+# Some control character