Re: AMTAR brokenness
Paul == Paul Eggert [EMAIL PROTECTED] writes: Paul Alexandre Duret-Lutz [EMAIL PROTECTED] writes: I've prepared this. However I'm not really proud of the name `filenames-99'. Maybe someone can suggest some nicer name? Paul I don't know what the rules are for Automake option spelling, but how Paul about filename-length-max=99? This is because the obvious Paul generalization is to be able to specify whatever length you like. Thanks for the suggestion. I'm checking this in and will close the PR. 2004-04-23 Alexandre Duret-Lutz [EMAIL PROTECTED] For PR automake/414: Introduce option filename-length-max=99. * doc/automake.texi (Options): Document it. * configure.ac (AM_INIT_AUTOMAKE): Use it. * lib/Automake/Options.pm (_process_option_list): Recognize it. * automake.in (handle_dist): Set FILENAME_FILTER. * lib/am/distdir.am (distdir) [FILENAME_FILTER]: Diagnose long filenames. * tests/fn99.test: New file. * tests/Makefile.am (TESTS): Add fn99.test. Index: NEWS === RCS file: /cvs/automake/automake/NEWS,v retrieving revision 1.268 diff -u -r1.268 NEWS --- NEWS19 Apr 2004 23:20:07 - 1.268 +++ NEWS23 Apr 2004 21:55:04 - @@ -45,8 +45,8 @@ * Diagnose AC_CONFIG_AUX_DIR calls following AM_INIT_AUTOMAKE. (PR/49) * Tar format can be chosen with the new options tar-v7, tar-ustar, and - tar-pax. - + tar-pax. The new option filename-length-max=99 helps diagnosing + filenames that are too long for tar-v7. New in 1.8: Index: automake.in === RCS file: /cvs/automake/automake/automake.in,v retrieving revision 1.1553 diff -u -r1.1553 automake.in --- automake.in 17 Apr 2004 18:46:05 - 1.1553 +++ automake.in 23 Apr 2004 21:55:08 - @@ -3498,9 +3498,13 @@ if rule 'dist-hook'; $transform{'DIST-TARGETS'} = join(' ', @dist_targets); + my $flm = option ('filename-length-max'); + my $filename_filter = $flm ? '.' x $flm-[1] : ''; + $output_rules .= file_contents ('distdir', new Automake::Location, - %transform); + %transform, + FILENAME_FILTER = $filename_filter); } Index: configure.ac === RCS file: /cvs/automake/automake/configure.ac,v retrieving revision 1.14 diff -u -r1.14 configure.ac --- configure.ac10 Apr 2004 17:18:01 - 1.14 +++ configure.ac23 Apr 2004 21:55:10 - @@ -1,6 +1,6 @@ # Process this file with autoconf to produce a configure script. -# Copyright (C) 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003 +# Copyright (C) 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004 # Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify @@ -36,7 +36,7 @@ # bothered by `missing'. am_AUTOCONF=${AUTOCONF-autoconf} -AM_INIT_AUTOMAKE([1.6 dist-bzip2]) +AM_INIT_AUTOMAKE([1.6 dist-bzip2 filename-length-max=99]) # The API version is the base version. We must guarantee # compatibility for all releases with the same API version. Index: doc/automake.texi === RCS file: /cvs/automake/automake/doc/automake.texi,v retrieving revision 1.32 diff -u -r1.32 automake.texi --- doc/automake.texi 19 Apr 2004 23:20:07 - 1.32 +++ doc/automake.texi 23 Apr 2004 21:55:16 - @@ -5812,6 +5812,16 @@ Hook @code{dist-tarZ} to @code{dist}. @trindex dist-tarZ [EMAIL PROTECTED] @code{filename-length-max=99} [EMAIL PROTECTED] Option, filename-length-max=99 [EMAIL PROTECTED] filename-length-max=99 +Abort if filenames longer than 99 characters are found during [EMAIL PROTECTED] dist}. Such long filenames are generally considered not to +be portable in tarballs. See the @code{tar-v7} and @code{tar-ustar} +options below. This option should be used in the top-level [EMAIL PROTECTED] or as an argument of @code{AM_INIT_AUTOMAKE} in [EMAIL PROTECTED], it will be ignored otherwise. + @item @code{no-define} @cindex Option, no-define This options is meaningful only when passed as an argument to @@ -5919,6 +5929,17 @@ @item @code{tar-v7} @itemx @code{tar-ustar} @itemx @code{tar-pax} [EMAIL PROTECTED] Option, tar-v7 [EMAIL PROTECTED] Option, tar-ustar [EMAIL PROTECTED] Option, tar-pax [EMAIL PROTECTED] tar formats [EMAIL PROTECTED] v7 tar format [EMAIL PROTECTED] ustar format [EMAIL PROTECTED] pax format [EMAIL PROTECTED] tar-v7 [EMAIL PROTECTED] tar-ustar [EMAIL PROTECTED] tar-pax + These three mutually exclusive options select the tar format to use when generating tarballs with @code{make dist}. (The tar file created is then compressed according to the set of @code{no-dist-gzip}, @@ -5935,7
Re: AMTAR brokenness
Paul == Paul Eggert [EMAIL PROTECTED] writes: [...] Paul Many tar implementations have trouble with path names longer than 99 Paul bytes. This includes the current GNU tar official latest non-alpha Paul release (which is buggy in this area). It would be reasonable to add Paul an automake option that checks for longer-than-99-byte file names, for Paul people who are worried about such things. But I don't think it needs Paul to be high priority. I've prepared this. However I'm not really proud of the name `filenames-99'. Maybe someone can suggest some nicer name? 2004-04-20 Alexandre Duret-Lutz [EMAIL PROTECTED] For PR automake/414: Introduce option filenames-99. * doc/automake.texi (Options): Document it. * configure.ac (AM_INIT_AUTOMAKE): Use it. * lib/Automake/Options.pm (_process_option_list): Recognize it. * automake.in (handle_dist): Set FILENAMES_99. * lib/am/distdir.am (distdir) [FILENAMES_99]: Diagnose long filenames. * tests/fn99.test: New file. * tests/Makefile.am (TESTS): Add fn99.test. Index: NEWS === RCS file: /cvs/automake/automake/NEWS,v retrieving revision 1.268 diff -u -r1.268 NEWS --- NEWS19 Apr 2004 23:20:07 - 1.268 +++ NEWS20 Apr 2004 20:30:56 - @@ -45,8 +45,8 @@ * Diagnose AC_CONFIG_AUX_DIR calls following AM_INIT_AUTOMAKE. (PR/49) * Tar format can be chosen with the new options tar-v7, tar-ustar, and - tar-pax. - + tar-pax. The new option filenames-99 helps diagnosing filenames + that are too long for tar-v7. New in 1.8: Index: automake.in === RCS file: /cvs/automake/automake/automake.in,v retrieving revision 1.1553 diff -u -r1.1553 automake.in --- automake.in 17 Apr 2004 18:46:05 - 1.1553 +++ automake.in 20 Apr 2004 20:31:00 - @@ -3500,7 +3500,8 @@ $output_rules .= file_contents ('distdir', new Automake::Location, - %transform); + %transform, + FILENAMES_99 = !! option 'filenames-99'); } Index: configure.ac === RCS file: /cvs/automake/automake/configure.ac,v retrieving revision 1.14 diff -u -r1.14 configure.ac --- configure.ac10 Apr 2004 17:18:01 - 1.14 +++ configure.ac20 Apr 2004 20:31:02 - @@ -1,6 +1,6 @@ # Process this file with autoconf to produce a configure script. -# Copyright (C) 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003 +# Copyright (C) 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004 # Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify @@ -36,7 +36,7 @@ # bothered by `missing'. am_AUTOCONF=${AUTOCONF-autoconf} -AM_INIT_AUTOMAKE([1.6 dist-bzip2]) +AM_INIT_AUTOMAKE([1.6 dist-bzip2 filenames-99]) # The API version is the base version. We must guarantee # compatibility for all releases with the same API version. Index: doc/automake.texi === RCS file: /cvs/automake/automake/doc/automake.texi,v retrieving revision 1.32 diff -u -r1.32 automake.texi --- doc/automake.texi 19 Apr 2004 23:20:07 - 1.32 +++ doc/automake.texi 20 Apr 2004 20:31:07 - @@ -5812,6 +5812,16 @@ Hook @code{dist-tarZ} to @code{dist}. @trindex dist-tarZ [EMAIL PROTECTED] @code{filenames-99} [EMAIL PROTECTED] Option, filenames-99 [EMAIL PROTECTED] filenames-99 +Abort if filenames longer than 99 characters are found during [EMAIL PROTECTED] dist}. Such long filenames are generally considered not to +be portable in tarballs. See the @code{tar-v7} and @code{tar-ustar} +options below. This option should be used in the top-level [EMAIL PROTECTED] or as an argument of @code{AM_INIT_AUTOMAKE} in [EMAIL PROTECTED], it will be ignored otherwise. + @item @code{no-define} @cindex Option, no-define This options is meaningful only when passed as an argument to @@ -5919,6 +5929,17 @@ @item @code{tar-v7} @itemx @code{tar-ustar} @itemx @code{tar-pax} [EMAIL PROTECTED] Option, tar-v7 [EMAIL PROTECTED] Option, tar-ustar [EMAIL PROTECTED] Option, tar-pax [EMAIL PROTECTED] tar formats [EMAIL PROTECTED] v7 tar format [EMAIL PROTECTED] ustar format [EMAIL PROTECTED] pax format [EMAIL PROTECTED] tar-v7 [EMAIL PROTECTED] tar-ustar [EMAIL PROTECTED] tar-pax + These three mutually exclusive options select the tar format to use when generating tarballs with @code{make dist}. (The tar file created is then compressed according to the set of @code{no-dist-gzip}, @@ -5935,7 +5956,8 @@ given longer filenames some tar implementations will diagnose the problem while other will generate broken tarballs or use non-portable extensions. Furthermore, the
Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] writes: I've prepared this. However I'm not really proud of the name `filenames-99'. Maybe someone can suggest some nicer name? I don't know what the rules are for Automake option spelling, but how about filename-length-max=99? This is because the obvious generalization is to be able to specify whatever length you like.
Re: [Bug-tar] Re: AMTAR brokenness
Thanks for writing that. Some minor points: Alexandre Duret-Lutz [EMAIL PROTECTED] writes: [EMAIL PROTECTED] selects the new pax format defined by POSIX +1003.1-2001. It supports filenames with up to 65535 characters. Hmm, where did that 65535 come from? I don't know of any limit of 65535 bytes (presumably you meant bytes, not characters) that is required by POSIX. Also, there are moves to change the pax format (so far in an upward-compatible way, but you never know). Perhaps you should mention that tar-pax is intended to be the most recent version of the pax interchange format, not necessarily the 2001 version. (This problem probably afflicts the GNU tar documentation as well) +# Substitute a variable $(am__untar) that exctract such exctract - extract + cpio) +am__tar='find $$tardir -print | cpio -H $1 -L -o' +am__tar_='find $tardir -print | cpio -H $1 -L -o' +am__untar='cpio -i' Wouldn't it be safer to use 'cpio -H tar -i' rather than 'cpio -i'? The latter autodetects, but I worry that autodetection isn't infallible. (I don't know about cpio to be sure here.)
Re: [Bug-tar] Re: AMTAR brokenness
Paul == Paul Eggert [EMAIL PROTECTED] writes: Paul Alexandre Duret-Lutz [EMAIL PROTECTED] writes: [EMAIL PROTECTED] selects the new pax format defined by POSIX +1003.1-2001. It supports filenames with up to 65535 characters. Paul Hmm, where did that 65535 come from? I don't know of any limit of Paul 65535 bytes (presumably you meant bytes, not characters) that is Paul required by POSIX. I read that in the Heirloom pax manpage http://heirloom.sourceforge.net/man/pax.1.html Maybe its maintainer can enlighten us. Paul Also, there are moves to change the pax format (so far in an Paul upward-compatible way, but you never know). Perhaps you should Paul mention that tar-pax is intended to be the most recent version of Paul the pax interchange format, not necessarily the 2001 version. (This Paul problem probably afflicts the GNU tar documentation as well) Will do. + cpio) +am__tar='find $$tardir -print | cpio -H $1 -L -o' +am__tar_='find $tardir -print | cpio -H $1 -L -o' +am__untar='cpio -i' Paul Wouldn't it be safer to use 'cpio -H tar -i' rather than 'cpio -i'? Paul The latter autodetects, but I worry that autodetection isn't infallible. Paul (I don't know about cpio to be sure here.) It shouldn't hurt. -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] selects the new pax format defined by POSIX +1003.1-2001. It supports filenames with up to 65535 characters. POSIX 1003.1-2001 specification does not impose any limit on lengths of files stored in PAX interchange format. Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] wrote: Paul == Paul Eggert [EMAIL PROTECTED] writes: Paul Alexandre Duret-Lutz [EMAIL PROTECTED] writes: [EMAIL PROTECTED] selects the new pax format defined by POSIX +1003.1-2001. It supports filenames with up to 65535 characters. Paul Hmm, where did that 65535 come from? I don't know of any limit of Paul 65535 bytes (presumably you meant bytes, not characters) that is Paul required by POSIX. I read that in the Heirloom pax manpage http://heirloom.sourceforge.net/man/pax.1.html Maybe its maintainer can enlighten us. The reason why I introduced this was the following: When cpio/pax encounters a broken archive with a very large number in the file name length field, it tries to allocate a lot of memory to store the file name; it will eventually fail to allocate that memory and terminate operation immediately. That's bad because if the header was just skipped, another valid header could follow (with -k/-K specified). Thus I introduced some sane limit here. Since cpio does not currently change its working directory, it is restriced to the open() limit of PATH_MAX anyway and will get ENAMETOOLONG, which is at 4096 on most systems. I chose 65535 because I've never seen a Unix system which even remotely supports names of such length for open(). But the value is arbitrary and could be raised to any value as long as enough memory is available to store a file name. Gunnar -- http://omnibus.ruf.uni-freiburg.de/~gritter
Re: [Bug-tar] Re: AMTAR brokenness
adl == Alexandre Duret-Lutz [EMAIL PROTECTED] writes: [...] Paul the existing code uses the -o option, which is [not portable] [...] Speaking about `o', I've just discovered that `missing' contains some magic to strip that flag when tar fails, and retry. It means that Roger's troubles should vanish once tar is changed to report filenames too long for `o' with $? != 0. -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
Here is the patch I'm installing. Besides the doc update, and the cpio -H $1 -i thing, I also changed the name of the cache variable to include the _AM_PROG_TAR argument (so that subpackages with different tar-xxx option do not share the same cache variable). As I've said, I'm also deliberately telling the tar2.test failing on system that cannot create pax archive to get some feedback from people using CVS Automake. (I tried Darwin and Solaris8, where this test fails.) I'll correct this before the release. 2004-04-20 Alexandre Duret-Lutz [EMAIL PROTECTED] For PR automake/414: Introduce options tar-v7, tar-ustar, and tar-pax to select tar format. * doc/automake.texi (Options): Document them. * lib/Automake/Options.pm (_process_option_list): Process these new options. * lib/am/distdir.am (dist-gzip, dist-bzip2, dist-tarZ, dist, distcheck): Adjust to use am__tar and am__untar. * m4/tar.m4: New file. * m4/Makefile.am (dist_m4data_DATA): Add tar.m4. * m4/init.m4 (AM_INIT_AUTOMAKE): Support the new options and call _AM_PROG_TAR. * tests/tar.test, tests/tar2.test, tests/tar3.test: New files. * tests/Makefile.am (TESTS): Add them. Index: NEWS === RCS file: /cvs/automake/automake/NEWS,v retrieving revision 1.267 diff -u -r1.267 NEWS --- NEWS17 Apr 2004 18:46:05 - 1.267 +++ NEWS19 Apr 2004 23:04:37 - @@ -44,6 +44,9 @@ * Diagnose AC_CONFIG_AUX_DIR calls following AM_INIT_AUTOMAKE. (PR/49) +* Tar format can be chosen with the new options tar-v7, tar-ustar, and + tar-pax. + New in 1.8: Index: doc/automake.texi === RCS file: /cvs/automake/automake/doc/automake.texi,v retrieving revision 1.31 diff -u -r1.31 automake.texi --- doc/automake.texi 2 Apr 2004 07:14:26 - 1.31 +++ doc/automake.texi 19 Apr 2004 23:04:44 - @@ -5916,6 +5916,46 @@ @file{subdir/file.cxx}, then the output file would be @file{subdir/file.o}. [EMAIL PROTECTED] @code{tar-v7} [EMAIL PROTECTED] @code{tar-ustar} [EMAIL PROTECTED] @code{tar-pax} +These three mutually exclusive options select the tar format to use +when generating tarballs with @code{make dist}. (The tar file created +is then compressed according to the set of @code{no-dist-gzip}, [EMAIL PROTECTED] and @code{dist-tarZ} options in use.) + +These options must be passed as argument to @code{AM_INIT_AUTOMAKE} +(@xref{Macros}) because they can causes new configure check to be +performed. Automake will complain if it sees such option in a [EMAIL PROTECTED] variable. + [EMAIL PROTECTED] selects the old V7 tar format. This is the historical +default. This antiquated format is understood by all tar +implementations and supports filenames with up to 99 characters. When +given longer filenames some tar implementations will diagnose the +problem while other will generate broken tarballs or use non-portable +extensions. Furthermore, the V7 format cannot store empty +directories. + [EMAIL PROTECTED] selects the ustar format defined by POSIX +1003.1-1988. This format is believed to be old enough to be portable. +It fully supports directories, and stores filenames with up to 255 +characters. However you may run against broken tar implementations +that incorrectly handle filenames longer than 99 characters (please +report them to @email{bug-automake@@gnu.org} so we can document this +accurately). + [EMAIL PROTECTED] selects the new pax interchange format defined by POSIX +1003.1-2001. It does not limit the length of filenames. However, +this format is very young and should probably be restricted to +packages which target only very modern platforms. There are moves to +change the pax format in an upward-compatible way, so this option may +refer to a more recent version in the future. + [EMAIL PROTECTED] knows several ways to construct these formats. It +will not abort if it cannot find a tool up to the task (so that the +package can still be built), but @code{make dist} will fail. + @item @var{version} @cindex Option, version A version number (e.g. @samp{0.30}) can be specified. If Automake is not Index: lib/Automake/Options.pm === RCS file: /cvs/automake/automake/lib/Automake/Options.pm,v retrieving revision 1.2 diff -u -r1.2 Options.pm --- lib/Automake/Options.pm 30 Sep 2003 19:05:54 - 1.2 +++ lib/Automake/Options.pm 19 Apr 2004 23:04:44 - @@ -1,4 +1,4 @@ -# Copyright (C) 2003 Free Software Foundation, Inc. +# Copyright (C) 2003, 2004 Free Software Foundation, Inc. # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -270,6 +270,22 @@ || $_ eq 'cygnus' || $_ eq 'no-dependencies') {
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] wrote: Sergey able to produce correct 'ustar' archives. Whatever Sergey options you give to previous versions of GNU tar, the Sergey produced archives will deviate from the standard. Ouch! Is that a `will always deviate' or a `can deviate in some conditions (like -o does)'? It is will always deviate. So as I sees it the question is: in this conditions, will old-GNU-tar-without-o deviate more from ustar than old-GNU-tar-with-o deviates from V7? The distance between the produced archive and the required format will be almost the same in both cases. The main difference when using -o is that archives produced with this option do not contain symbolic ownership information (usernames/group names). Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
Roger Leigh [EMAIL PROTECTED] wrote: I didn't realise it supported generating tar files. For the cpio I have (GNU cpio version 2.5): ustar The POSIX.1 tar format. Also recognizes GNU tar archives, which are similar but not identical. How does this differ from tar --format=posix. Isn't the POSIX tar format for the current GNU tar now the same thing? No, it isn't. The option --format=posix means POSIX.1-2001 format (a.k.a. pax format), whereas ustar is POSIX.1-1988. The former is atually a superset of the latter. Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
Paul Eggert [EMAIL PROTECTED] wrote: Given all the hassles that will accompany any change, perhaps we should give the maintainer a more gradual upgrade path. For example, we could add an automake macro AM_TAR_FORMAT. [...] That's a nice idea. I'd vote for it. I'd put cpio last on the list, with the lowest priority. I agree. Traditional UNIX cpio had serious bugs whenever two files had inode numbers that were the same modulo 2**16. This problem persisted for many years (up until I stopped using cpio -- perhaps it still has the bug for all I know). GNU cpio cannot yet handle large files or archives (i.e., larger than 2 GiB on typical x86 hosts). The CVS version is able to handle large files, however most distributions still ship cpio binaries compiled without large file support. Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
Paul == Paul Eggert [EMAIL PROTECTED] writes: [...] Paul Given all the hassles that will accompany any change, perhaps we Paul should give the maintainer a more gradual upgrade path. Paul For example, we could add an automake macro AM_TAR_FORMAT. Paul AM_TAR_FORMAT([v7]) would use the current behavior (flawed as it is -- Paul at least it's something people are familiar with). Paul AM_TAR_FORMAT([ustar]) would attempt to generate ustar format, using Paul the commands tar --format=ustar, tar (if not GNU tar), pax -x Paul ustar, and cpio -H ustar, in that order. Likewise for other Paul formats as we find the need to support them. [...] Sounds great. Here is a preliminary patch for this. I have still to test it more extensively, but that should be enough for comments. (BTW the real purpose of tar2.test is to get report from people on platforms where the pax format is not supported.) 2004-04-18 Alexandre Duret-Lutz [EMAIL PROTECTED] For PR automake/414: Introduce options tar-v7, tar-ustar, and tar-pax to select tar format. * doc/automake.texi (Options): Document them. * lib/Automake/Options.pm (_process_option_list): Process these new options. * lib/am/distdir.am (dist-gzip, dist-bzip2, dist-tarZ, dist, distcheck): Adjust to use am__tar and am__untar. * m4/tar.m4: New file. * m4/Makefile.am (dist_m4data_DATA): Add tar.m4. * m4/init.m4 (AM_INIT_AUTOMAKE): Support the new options and call _AM_PROG_TAR. * tests/tar.test, tests/tar2.test, tests/tar3.test: New files. * tests/Makefile.am (TESTS): Add them. Index: NEWS === RCS file: /cvs/automake/automake/NEWS,v retrieving revision 1.267 diff -u -r1.267 NEWS --- NEWS17 Apr 2004 18:46:05 - 1.267 +++ NEWS18 Apr 2004 21:12:28 - @@ -44,6 +44,9 @@ * Diagnose AC_CONFIG_AUX_DIR calls following AM_INIT_AUTOMAKE. (PR/49) +* Tar format can be chosen with the new options tar-v7, tar-ustar, and + tar-pax. + New in 1.8: Index: doc/automake.texi === RCS file: /cvs/automake/automake/doc/automake.texi,v retrieving revision 1.31 diff -u -r1.31 automake.texi --- doc/automake.texi 2 Apr 2004 07:14:26 - 1.31 +++ doc/automake.texi 18 Apr 2004 21:12:34 - @@ -5916,6 +5916,40 @@ @file{subdir/file.cxx}, then the output file would be @file{subdir/file.o}. [EMAIL PROTECTED] @code{tar-v7} [EMAIL PROTECTED] @code{tar-ustar} [EMAIL PROTECTED] @code{tar-pax} +These three mutually exclusive options select the tar format to use +when generating tarballs with @code{make dist}. (The tar file created +is then compressed according to the set of @code{no-dist-gzip}, [EMAIL PROTECTED] and @code{dist-tarZ} options in use.) + +These options must be passed as argument to @code{AM_INIT_AUTOMAKE} +(@xref{Macros}) because they can causes new configure check to be +performed. Automake will complain if it sees such option in a [EMAIL PROTECTED] variable. + [EMAIL PROTECTED] selects the old V7 tar format. This is the historical +default. This antiquated format is understood by all tar +implementations and supports filenames with up to 99 characters. When +given longer filenames some tar implementations will diagnose the +problem while other will generate broken tarballs or use non-portable +extensions. Furthermore, the V7 format cannot store empty +directories. + [EMAIL PROTECTED] selects the ustar format defined by POSIX +1003.1-1988. This format is believed to be old enough to be portable. +It fully supports directories, and stores filenames with up to 255 +characters. However you may run against broken tar implementations +that incorrectly handle filenames longer than 99 characters (please +report them to @email{bug-automake@@gnu.org} so we can document this +accurately). + [EMAIL PROTECTED] selects the new pax format defined by POSIX +1003.1-2001. It supports filenames with up to 65535 characters. +However this format is very young and should probably be avoided in +all packages that do not target only modern systems. + @item @var{version} @cindex Option, version A version number (e.g. @samp{0.30}) can be specified. If Automake is not Index: lib/Automake/Options.pm === RCS file: /cvs/automake/automake/lib/Automake/Options.pm,v retrieving revision 1.2 diff -u -r1.2 Options.pm --- lib/Automake/Options.pm 30 Sep 2003 19:05:54 - 1.2 +++ lib/Automake/Options.pm 18 Apr 2004 21:12:35 - @@ -1,4 +1,4 @@ -# Copyright (C) 2003 Free Software Foundation, Inc. +# Copyright (C) 2003, 2004 Free Software Foundation, Inc. # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -270,6 +270,22 @@ || $_ eq
Re: [Bug-tar] Re: AMTAR brokenness
On Sun, 18 Apr 2004 23:18:05 +0200, Alexandre Duret-Lutz wrote: [EMAIL PROTECTED] selects the new pax format defined by POSIX +1003.1-2001. It supports filenames with up to 65535 characters. +However this format is very young and should probably be avoided in +all packages that do not target only modern systems. Perhaps, a bit more idiomatic: However, this format is very young and should probably be restricted to packages which target only very modern platforms. -- ES
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] writes: How about this scheme: In the light of later discussion, how about this scheme instead? Use the first of the following commands that works: tar --format=ustar tar pax -x ustar I wouldn't bother with POSIXLY_CORRECT=1 or tar -o, as they don't really help much these days. On second though, shouldn't we try to use pax first? tar is no longer a POSIX requirement, right? The tar command has never been a POSIX requirement. However, pax has never caught on, for various reasons, and it hasn't been road-tested as much as tar has. It makes sense to use pax if tar is not available, but I wouldn't make pax my first choice. Also do we really need to try `-o'? I vaguely recall that RMS used to suggest -o in the GNU coding standards, for compatibility with v7 'tar', but that requirement is no longer necessary (it's not in the standards now, anyway). Besides GNU tar, is there some tar implementation that does not default to the ustar format? Many tar implementations have trouble with path names longer than 99 bytes. This includes the current GNU tar official latest non-alpha release (which is buggy in this area). It would be reasonable to add an automake option that checks for longer-than-99-byte file names, for people who are worried about such things. But I don't think it needs to be high priority.
Re: [Bug-tar] Re: AMTAR brokenness
Roger Leigh [EMAIL PROTECTED] writes: I'm using the patch I posted. If and when anyone complains, I'll instruct them to build and install the latest GNU tar (or build it for them). I require the functionality for long pathnames, and I don't want to have to support broken tools. For my needs, 99 chars is not at all sufficient. Have you ever considered using 'cpio -H ustar' instead of tar? -- Ralph
Re: [Bug-tar] Re: AMTAR brokenness
Paul == Paul Eggert [EMAIL PROTECTED] writes: Paul Alexandre Duret-Lutz [EMAIL PROTECTED] writes: How about this scheme: Paul In the light of later discussion, how about this scheme instead? Paul Use the first of the following commands that works: Paul tar --format=ustar Paul tar Paul pax -x ustar This looks satisfactory when GNU tar 1.13.93 is installed, or when GNU tar is not installed. However, on most GNU/Linux setups, which have GNU tar 1.13.25 installed, my understanding is that will cause GNU-tarballs to be produced. What if we try hard not to use such old GNU tar versions? Use `tar --format=ustar' if possible. If `tar' is GNU's: Use `pax -x ustar' if that works, or fall back to plain `tar'. Otherwise Use plain `tar', or fall back to `pax -x ustar'. [...] Paul The tar command has never been a POSIX requirement. However, pax Paul has never caught on, for various reasons, and it hasn't been Paul road-tested as much as tar has. It makes sense to use pax if Paul tar is not available, but I wouldn't make pax my first choice. I trust you. However my impression from Sergey comments is that older GNU tar versions should not be a choice at all when it comes to creating ustar archives :( What about Ralph's `cpio -H ustar' suggestion? I've never used cpio. It looks like the -H option is non-standard, but is supported by many implementations. Probably this is even less exercised than `pax -x ustar'. Paul Many tar implementations have trouble with path names Paul longer than 99 bytes. This includes the current GNU tar Paul official latest non-alpha release (which is buggy in this Paul area). It would be reasonable to add an automake option Paul that checks for longer-than-99-byte file names, for Paul people who are worried about such things. I agree. -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
Ralph Schleicher [EMAIL PROTECTED] writes: Roger Leigh [EMAIL PROTECTED] writes: I'm using the patch I posted. If and when anyone complains, I'll instruct them to build and install the latest GNU tar (or build it for them). I require the functionality for long pathnames, and I don't want to have to support broken tools. For my needs, 99 chars is not at all sufficient. Have you ever considered using 'cpio -H ustar' instead of tar? I didn't realise it supported generating tar files. For the cpio I have (GNU cpio version 2.5): ustar The POSIX.1 tar format. Also recognizes GNU tar archives, which are similar but not identical. How does this differ from tar --format=posix. Isn't the POSIX tar format for the current GNU tar now the same thing? -- Roger Leigh Printing on GNU/Linux? http://gimp-print.sourceforge.net/ GPG Public Key: 0x25BFB848. Please sign and encrypt your mail.
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] writes: Paul tar --format=ustar Paul tar Paul pax -x ustar This looks satisfactory when GNU tar 1.13.93 is installed, or when GNU tar is not installed. However, on most GNU/Linux setups, which have GNU tar 1.13.25 installed, my understanding is that will cause GNU-tarballs to be produced. OK, but hold on a second: what's the goal here? If the goal is to produce ustar format, then you're correct: we shouldn't try plain tar (without checking it, anyway). But if the goal is to produce tarballs that work, then the most likely consumer of the tarball right now is GNU tar 1.13.25 or earlier -- which chokes on some ustar tarballs. Given all the hassles that will accompany any change, perhaps we should give the maintainer a more gradual upgrade path. For example, we could add an automake macro AM_TAR_FORMAT. AM_TAR_FORMAT([v7]) would use the current behavior (flawed as it is -- at least it's something people are familiar with). AM_TAR_FORMAT([ustar]) would attempt to generate ustar format, using the commands tar --format=ustar, tar (if not GNU tar), pax -x ustar, and cpio -H ustar, in that order. Likewise for other formats as we find the need to support them. The default could be v7 for now, but we can warn people that it is subject to change: it could change to ustar, then posix, for example. That way, adventurous maintainers could use AM_TAR_FORMAT([ustar]) now, whereas more conservative types would avoid any change in behavior. What about Ralph's `cpio -H ustar' suggestion? I'd put cpio last on the list, with the lowest priority. Traditional UNIX cpio had serious bugs whenever two files had inode numbers that were the same modulo 2**16. This problem persisted for many years (up until I stopped using cpio -- perhaps it still has the bug for all I know). GNU cpio cannot yet handle large files or archives (i.e., larger than 2 GiB on typical x86 hosts).
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] writes: 1. The --format=posix options is a GNU Tar option, which means `make dist' can no longer be run with other make implementations. Presumably you meant tar implementations and not make implementations? But this is already the case, as the existing code uses the -o option, which is specific to GNU tar (the -o option has a different meaning with traditional tar, and traditional tar rejects -o when creating archives). 2. The --format option is a new option of tar 1.13.93, so the produced Makefile will not even run with widespread tar versions. True: any such patch along these lines would have to configure this, depending on whether tar supported the option in question. Come to think of it, while you're at it you could also see whether -o is supported, so that the makefile ports to non-GNU tar implementations. 3. The POSIX format is young Quite true. Few implementations support the POSIX 1003.1-2001 format. Forcing the ustar format might be a possibility (is it?), if that can be done portably (that probably involves a configure check). Yes, I think this is the best approach: that is, use --format=ustar if available, then -o if that works, otherwise don't use anything. Another idea is to arrange the dist rules so users have a way to override the tar options. That also might be helpful, though I think not as useful; most users won't want to futz with this.
Re: [Bug-tar] Re: AMTAR brokenness
Hello, Actually I was planning to address this issue a bit later, since I do not have a solution for automake yet. Anyway: Alexandre Duret-Lutz [EMAIL PROTECTED] wrote: Roger + $(AMTAR) chf - $(distdir) --format=posix | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz [...] Sorry for the late comment. I can see at least three reasons against this patch: I agree with all three. The format used for distribution tarballs should be a reasonably old one, so that any existing version of tar be able to cope with it. On the other hand, let me notice that the current practice of using (so called) V7 format for make dist falls within the first reason you mentioned: The --format=posix options is a GNU Tar option, which means `make dist' can no longer be run with other make implementations. Automake makefiles should be portable and this breaks it. The -o option used by automake is as well a GNU Tar option, and it has another meaning for other tar implementations. According to POSIX specs, -o stands for 'Assign to extracted files the user and group identifiers of the user running the program', i.e. it corresponds to GNU tar --no-same-owner option. Let's take as examples two other tar implementations: 1) Sun tar. Its manpage says: o Ownership. Assign to extracted files the user and group identifiers of the user running the program, rather than those on tarfile. [...] 2) Star: -nochown, -o Do not restore owner and group of files. This may be used if super user privileges are needed to overwrite existing files but the local ownership of the existing files should not change. Thus, using -o in Makefiles is at best irrelevant to creation of archives. Returning to the question which format should be chosen for distribution tarballs: I would recommend the 'ustar' format. It also has a limitation on the maximum length of stored pathnames, but the limitation is 256 characters, which is reasonably large. Besides, ustar archives can be read by any existing implementation of tar. Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
Paul == Paul Eggert [EMAIL PROTECTED] writes: Paul Alexandre Duret-Lutz [EMAIL PROTECTED] writes: 1. The --format=posix options is a GNU Tar option, which means `make dist' can no longer be run with other make implementations. Paul Presumably you meant tar implementations and not make Paul implementations? Yep. Paul But this is already the case, as the existing code uses Paul the -o option, which is specific to GNU tar (the -o Paul option has a different meaning with traditional tar, and Paul traditional tar rejects -o when creating archives). Thanks, I didn't know that. My comment was only based on the fact that the automake test suite uses it and nobody ever complained about it. But of course the test suite doesn't verify the output format. Looking into why I never noticed myself, I can see that Solaris 8's tar ignores `o' when creating archives (I guess it create a ustar file, though I do not know how to verify this). FreeBSD/NetBSD/OpenBSD/Darwin's tar understand `o' as a V7 request like GNU Tar. OSF1's tar understand's `o' as GNU tar does when creating archives, and as Solaris does when extracting archives. I cannot check more platform. But at least it's interesting that it works on more that just GNU tar. Forcing the ustar format might be a possibility (is it?), if that can be done portably (that probably involves a configure check). Paul Yes, I think this is the best approach: that is, use --format=ustar if Paul available, then -o if that works, otherwise don't use anything. Another idea is to arrange the dist rules so users have a way to override the tar options. Paul That also might be helpful, though I think not as useful; most users Paul won't want to futz with this. Ok. I'll try to do both of these this week-end and Cc the patch to bug-tar for comments. -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
Sergey == Sergey Poznyakoff [EMAIL PROTECTED] writes: [...] Sergey Returning to the question which format should be chosen for Sergey distribution tarballs: I would recommend the 'ustar' format. Sergey It also has a limitation on the maximum length of stored pathnames, Sergey but the limitation is 256 characters, which is reasonably large. Sergey Besides, ustar archives can be read by any existing implementation Sergey of tar. By the way, the pax manpage from the Heirloom Toolchest (http://heirloom.berlios.de/man/pax.1.html) mentions that due to implementation errors, file names longer than 99 characters can not considered to be generally portable when taking about the ustar format. I think it's the only place where I've read this. Has anybody seen such implementation? I'm tempted to think we shouldn't care. -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
adl == Alexandre Duret-Lutz [EMAIL PROTECTED] writes: Paul == Paul Eggert [EMAIL PROTECTED] writes: [...] Forcing the ustar format might be a possibility (is it?), if that can be done portably (that probably involves a configure check). Paul Yes, I think this is the best approach: that is, use --format=ustar if Paul available, then -o if that works, otherwise don't use anything. [...] adl Ok. I'll try to do both of these this week-end and Cc the patch adl to bug-tar for comments. On second though, shouldn't we try to use pax first? tar is no longer a POSIX requirement, right? Also do we really need to try `-o'? This sounds odd because we are trying to get rid of the 100-char limit, and using `-o' would means `make dist' will work for deep trees on some platform and not on some others. Besides GNU tar, is there some tar implementation that does not default to the ustar format? How about this scheme: Use `pax -x ustar' if it works. Otherwise try `tar --format=ustar'. Otherwise use `POSIXLY_CORRECT=1 tar'. (I'm assuming that POSIXLY_CORRECT=1 will coerce older tar versions into producing ustar format. Am I wrong, or is there another way?) -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] wrote: On second though, shouldn't we try to use pax first? tar is no longer a POSIX requirement, right? Right, but I'd say it's too early to relay on pax. Also do we really need to try `-o'? No, we do not. Besides GNU tar, is there some tar implementation that does not default to the ustar format? GNU tar 1.14 will be the last release that uses non-portable GNU format by default. Subsequent versions will default to 'ustar' format. How about this scheme: Use `pax -x ustar' if it works. Otherwise try `tar --format=ustar'. This looks OK (though see my comment above). Otherwise use `POSIXLY_CORRECT=1 tar'. (I'm assuming that POSIXLY_CORRECT=1 will coerce older tar versions into producing ustar format. Am I wrong, or is there another way?) No, it will not. Previous versions of GNU tar were not able to produce correct 'ustar' archives. Whatever options you give to previous versions of GNU tar, the produced archives will deviate from the standard. Using -o does not help much, either: if the file list to be archived contains long filenames or symbolic links, tar will create GNU extension headers for them, thus hurting the portability of the resulting archive. Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
Alexandre Duret-Lutz [EMAIL PROTECTED] wrote: I think it's the only place where I've read this. Has anybody seen such implementation? I'm tempted to think we shouldn't care. Agreed. Regards, Sergey
Re: [Bug-tar] Re: AMTAR brokenness
On Fri, 16 Apr 2004, Alexandre Duret-Lutz wrote: By the way, the pax manpage from the Heirloom Toolchest (http://heirloom.berlios.de/man/pax.1.html) mentions that due to implementation errors, file names longer than 99 characters can not considered to be generally portable when taking about the ustar format. I think it's the only place where I've read this. Has anybody seen such implementation? I'm tempted to think we shouldn't care. More than likely such implementations exist. Usually when tar programs break while extracting long file names, they continue on, leaving bad file names in their wake. Probably the user won't even notice until something doesn't work. Is there a reason to allow file names longer than 99 characters in a package? Clearly this is non-portable. Why not enforce a maximum file name length of 99 characters in Automake? One way to enforce this is to use sed to truncate file names longer than 99 characters before passing them to tar so that tar complains/fails during 'make dist'. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: [Bug-tar] Re: AMTAR brokenness
Sergey == Sergey Poznyakoff [EMAIL PROTECTED] writes: [...] Otherwise use `POSIXLY_CORRECT=1 tar'. (I'm assuming that POSIXLY_CORRECT=1 will coerce older tar versions into producing ustar format. Am I wrong, or is there another way?) Sergey No, it will not. Previous versions of GNU tar were not Sergey able to produce correct 'ustar' archives. Whatever Sergey options you give to previous versions of GNU tar, the Sergey produced archives will deviate from the standard. Ouch! Is that a `will always deviate' or a `can deviate in some conditions (like -o does)'? Sergey Using -o does not help much, either: if the file list Sergey to be archived contains long filenames or symbolic Sergey links, tar will create GNU extension headers for them, Sergey thus hurting the portability of the resulting archive. We don't have symbolic links. Only regular files and non-empty directories; furthermore ownership is not important. So as I sees it the question is: in this conditions, will old-GNU-tar-without-o deviate more from ustar than old-GNU-tar-with-o deviates from V7? If the answer is yes, then I guess we must keep using `o' with older GNU tar versions, and people with long filenames will have to get a recent tar version if they want portable tarballs. Otherwise it seems we simply do not need `o'. -- Alexandre Duret-Lutz
Re: [Bug-tar] Re: AMTAR brokenness
Bob Friesenhahn [EMAIL PROTECTED] writes: Is there a reason to allow file names longer than 99 characters in a package? Clearly this is non-portable. Why not enforce a maximum file name length of 99 characters in Automake? I experience the breakage when using Doxygen to generate a reference manual from C++ code. Some of the filenames it generates are over 80 chars, due to using namespaces, and long classnames etc. If the path is called libfoobar-baz-12.32.44/doc/libfoobar-baz/html/., that's an extra 46 chars in the path. The 99 char limit has been comfortably exceeded, and I had no control over this! [There are options to produce mangled 8.3 names, but I want them to be human readable.] One way to enforce this is to use sed to truncate file names longer than 99 characters before passing them to tar so that tar complains/fails during 'make dist'. What if the truncated names already exist? You will then silently produce a broken archive. Better to do find . | wc -L to get an accurate figure, and then bail out. I'm using the patch I posted. If and when anyone complains, I'll instruct them to build and install the latest GNU tar (or build it for them). I require the functionality for long pathnames, and I don't want to have to support broken tools. For my needs, 99 chars is not at all sufficient. Regards, Roger -- Roger Leigh Printing on GNU/Linux? http://gimp-print.sourceforge.net/ GPG Public Key: 0x25BFB848. Please sign and encrypt your mail.
Re: AMTAR brokenness
Roger == Roger Leigh [EMAIL PROTECTED] writes: [...] Roger - $(AMTAR) chof - $(distdir) | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz Roger + $(AMTAR) chf - $(distdir) --format=posix | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz [...] Sorry for the late comment. I can see at least three reasons against this patch: 1. The --format=posix options is a GNU Tar option, which means `make dist' can no longer be run with other make implementations. Automake makefiles should be portable and this breaks it. 2. The --format option is a new option of tar 1.13.93, so the produced Makefile will not even run with widespread tar versions. 3. The POSIX format is young and AFAICT many platforms out there cannot read it. (Sometimes tar won't read it but you can read it with the pax utility, sometimes neither work. E.g. such archives cause pax and tar to segfault on the OSF1 V4.0 box I have access to.) I'm not denying there is a problem to fix, but the above change really breaks more things than it cures. Without it, the only people annoyed are those with 100+-character-long filenames. With it _all_ packages produce non-portable tarballs using non-portable syntax. Forcing the ustar format might be a possibility (is it?), if that can be done portably (that probably involves a configure check). Another idea is to arrange the dist rules so users have a way to override the tar options. -- Alexandre Duret-Lutz
AMTAR brokenness
Recent versions of GNU tar correctly use the -o option. This breaks any project with path lengths 100. Previously GNU tar incorrectly produced a POSIX tar file with -o, but now it produces a crippled UNIX v7 tarfile. PR/414 has been filed against automake. A patch against automake 1.8.3 follows, which uses the correct --format=posix syntax. Both the current and patched forms do rely on GNU tar. A configure check to check for GNY tar might be better. Either way, GNU tar should *not* be given the -o option, since it causes horrible breakage. Regards, Roger --- /tmp/distdir.am 2004-04-01 12:32:29.0 +0100 +++ distdir.am 2004-04-01 12:30:26.0 +0100 @@ -238,19 +238,19 @@ GZIP_ENV = --best .PHONY: dist-gzip dist-gzip: distdir - $(AMTAR) chof - $(distdir) | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz + $(AMTAR) chf - $(distdir) --format=posix | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz $(am__remove_distdir) ?BZIP2?DIST_ARCHIVES += $(distdir).tar.bz2 .PHONY: dist-bzip2 dist-bzip2: distdir - $(AMTAR) chof - $(distdir) | bzip2 -9 -c $(distdir).tar.bz2 + $(AMTAR) chf - $(distdir) --format=posix | bzip2 -9 -c $(distdir).tar.bz2 $(am__remove_distdir) ?COMPRESS?DIST_ARCHIVES += $(distdir).tar.Z .PHONY: dist-tarZ dist-tarZ: distdir - $(AMTAR) chof - $(distdir) | compress -c $(distdir).tar.Z + $(AMTAR) chf - $(distdir) --format=posix | compress -c $(distdir).tar.Z $(am__remove_distdir) ?SHAR?DIST_ARCHIVES += $(distdir).shar.gz @@ -281,9 +281,9 @@ .PHONY: dist dist-all dist dist-all: distdir -?GZIP? $(AMTAR) chof - $(distdir) | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz -?BZIP2?$(AMTAR) chof - $(distdir) | bzip2 -9 -c $(distdir).tar.bz2 -?COMPRESS? $(AMTAR) chof - $(distdir) | compress -c $(distdir).tar.Z +?GZIP? $(AMTAR) chf - $(distdir) --format=posix | GZIP=$(GZIP_ENV) gzip -c $(distdir).tar.gz +?BZIP2?$(AMTAR) chf - $(distdir) --format=posix | bzip2 -9 -c $(distdir).tar.bz2 +?COMPRESS? $(AMTAR) chf - $(distdir) --format=posix | compress -c $(distdir).tar.Z ?SHAR? shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c $(distdir).shar.gz ?ZIP? -rm -f $(distdir).zip ?ZIP? zip -rq $(distdir).zip $(distdir) -- Roger Leigh Printing on GNU/Linux? http://gimp-print.sourceforge.net/ GPG Public Key: 0x25BFB848. Please sign and encrypt your mail.