[PATCH] {maint} Improve and extend tests `man*.test'.
Pinging this patch again, following this:
http://lists.gnu.org/archive/html/automake-patches/2010-11/msg3.html
Since I was at it, I rebased the patch, and squashed in some additional
improvements. The updated patch is attached; here is what I squashed in
(after the rebase):
diff --git a/ChangeLog b/ChangeLog
index 55e5a99..e39dc11 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,15 +1,16 @@
-2010-08-17 Stefano Lattarini stefano.lattar...@gmail.com
+2010-11-08 Stefano Lattarini stefano.lattar...@gmail.com
Improve and extend tests on man pages support.
* tests/man.test: Enable `errexit' shell flag, and related changes.
Make grepping of generated Makefile.in slighty stricter.
- * tests/man3.test: Add trailing `:' command.
+ * tests/man3.test: Add trailing `:' command.
* tests/man5.test: Prefer cat + here-doc over echo to append to
configure.in.
* tests/man2.test: Likewise, and add trailing `:' command.
* tests/man4.test: More thorough and consistent checking of make
- error messages. Move the checks using the real `help2man' program
- to ...
+ error messages. Place fake `help2man' program in a new `bin'
+ directory rather than in `.'. Move the checks using the real
+ `help2man' program to ...
* tests/man6.test: ... this new test, and extend them. This test
passes with GNU make and Solaris make, still fails with BSD make.
* tests/man7.test: New test, extracted from old man4.test, which
diff --git a/tests/man.test b/tests/man.test
index 7037330..1b287a8 100755
--- a/tests/man.test
+++ b/tests/man.test
@@ -31,6 +31,6 @@ END
$ACLOCAL
$AUTOMAKE
-grep '^MANS = ' Makefile.in
+grep '^MANS *=' Makefile.in
:
diff --git a/tests/man4.test b/tests/man4.test
index 643817c..3eff468 100755
--- a/tests/man4.test
+++ b/tests/man4.test
@@ -59,17 +59,17 @@ while test $# -gt 0; do
shift
done
END
-
cp foo bar
+chmod +x foo bar
-cat help2man 'END'
+mkdir bin
+cat bin/help2man 'END'
#! /bin/sh
# fake help2man script that lets `missing' think it is not installed
exit 127
END
-
-chmod +x foo bar help2man
-PATH=`pwd`$PATH_SEPARATOR$PATH
+chmod +x bin/help2man
+PATH=`pwd`/bin$PATH_SEPARATOR$PATH
grep_error_messages()
{
Regards,
Stefano
From cdd3cf335605dd91c65c4e6744efe9c94854e9ce Mon Sep 17 00:00:00 2001
From: Stefano Lattarini stefano.lattar...@gmail.com
Date: Mon, 12 Jul 2010 17:37:33 +0200
Subject: [PATCH] Improve and extend tests on man pages support.
* tests/man.test: Enable `errexit' shell flag, and related changes.
Make grepping of generated Makefile.in slighty stricter.
* tests/man3.test: Add trailing `:' command.
* tests/man5.test: Prefer cat + here-doc over echo to append to
configure.in.
* tests/man2.test: Likewise, and add trailing `:' command.
* tests/man4.test: More thorough and consistent checking of make
error messages. Place fake `help2man' program in a new `bin'
directory rather than in `.'. Move the checks using the real
`help2man' program to ...
* tests/man6.test: ... this new test, and extend them. This test
passes with GNU make and Solaris make, still fails with BSD make.
* tests/man7.test: New test, extracted from old man4.test, which
checks for a bug in maintainer-clean w.r.t. generated manpages.
* tests/man8.test: New test, extracted from old man4.test, which
checks for a bug in distcheck w.r.t. generated manpages. Passes
with GNU make and Solaris make, still fails with BSD make.
---
ChangeLog | 21
tests/man.test | 13 +--
tests/man2.test |8 +++-
tests/man3.test |4 ++-
tests/man4.test | 44 -
tests/man5.test |6 ++-
tests/man6.test | 99 +++
tests/man7.test | 61 ++
tests/man8.test | 58
9 files changed, 282 insertions(+), 32 deletions(-)
create mode 100755 tests/man6.test
create mode 100755 tests/man7.test
create mode 100755 tests/man8.test
diff --git a/ChangeLog b/ChangeLog
index a356958..e39dc11 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,24 @@
+2010-11-08 Stefano Lattarini stefano.lattar...@gmail.com
+
+ Improve and extend tests on man pages support.
+ * tests/man.test: Enable `errexit' shell flag, and related changes.
+ Make grepping of generated Makefile.in slighty stricter.
+ * tests/man3.test: Add trailing `:' command.
+ * tests/man5.test: Prefer cat + here-doc over echo to append to
+ configure.in.
+ * tests/man2.test: Likewise, and add trailing `:' command.
+ * tests/man4.test: More thorough and consistent checking of make
+ error messages. Place fake `help2man' program in a new `bin'
+ directory rather than in `.'. Move the checks using the real
+ `help2man' program to ...
+ *
Re: [PATCH] Minor improvements and extensions to various tests.
Pinging this patch again, following this: http://lists.gnu.org/archive/html/automake-patches/2010-11/msg3.html There should be no need to rebase the patch (which is based off of maint), so that I should just be able to merge it into master from the copy in my local automake repository. Regards, Stefano
Re: [Automake-commit] [SCM] GNU Automake branch, master, updated. v1.11-414-g3dffb01
[ moving from automake-commit ] Hi Stefano, * Stefano Lattarini wrote on Sun, Nov 07, 2010 at 04:37:15PM CET: This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project GNU Automake. [...] commit 1e460312d79fe15170e7519fb785e132ac7293fb Merge: 4555c81 657ba88 Author: Stefano Lattarini stefano.lattar...@gmail.com Date: Sun Nov 7 11:50:06 2010 +0100 Merge branch 'tests-misc-fixes-1' Conflicts: tests/Makefile.am tests/Makefile.in For what little it's worth, I tend to remove the Conflicts: section of the generated merge commit log entry when I have successfully dealt with any conflicts. It's no big deal either way really, but I find it more distracting than helpful. Cheers, Ralf
Re: [Automake-commit] [SCM] GNU Automake branch, master, updated. v1.11-414-g3dffb01
On Monday 08 November 2010, Ralf Wildenhues wrote: [ moving from automake-commit ] Hi Stefano, * Stefano Lattarini wrote on Sun, Nov 07, 2010 at 04:37:15PM CET: This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project GNU Automake. [...] commit 1e460312d79fe15170e7519fb785e132ac7293fb Merge: 4555c81 657ba88 Author: Stefano Lattarini stefano.lattar...@gmail.com Date: Sun Nov 7 11:50:06 2010 +0100 Merge branch 'tests-misc-fixes-1' Conflicts: tests/Makefile.am tests/Makefile.in For what little it's worth, I tend to remove the Conflicts: section of the generated merge commit log entry when I have successfully dealt with any conflicts. It's no big deal either way really, but I find it more distracting than helpful. OK, I'll try to remember that and follow your preference, as I have no strong opinion on the matter. Also, in this case, the conflict was a spurious one, and immediately solved, so removing the Conflicts: section would have probably been a good idea anyway. Regards, Stefano
execvp: /bin/sh: Argument list too long
Hi, I looked through the mailing list archives and found several questions and proposed solutions related to this problem: - Split up file lists so several rules are generated: http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html - What is your env's size? http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html - It will fail. In other words Sorry, can't fix? http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html My problem is that I can not make dist in one project and not make check in another project. Both have a large number of tests. The one where make check fails has a few more tests, but make dist will fail there, as well. What are the plans on fixing this? Are there any? Autoconf already checks for the maximum argument length, but since by that time, Makefile.in is already generated, it can't help. Maybe storing the list of dist files and check files in a different file and reading this with xargs $file would help. Any other ideas? -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: Copy files in VPATH build
On Mon, Nov 8, 2010 at 9:14 AM, Steffen Dettmer steffen.dett...@googlemail.com wrote: On Sun, Oct 31, 2010 at 1:09 PM, Jason Curl jcurln...@arcor.de wrote: * On Oct 30, 2010, at 5:17 PM, Jason Curljcurln...@arcor.de wrote: When building my package with ./configure everything works well as some datafiles that are required for tests are in the correct place (my tests don't need to be installed with make install). However, when I use VPATH builds with ../configure, I don't know how to get my data files copied from the source to the current build path. cpreadini: �...@srcdir@/cpfiles.sh @srcdir@/read.ini ./read.ini For some similar cases here we simply made some `read.ini.in' and used AC_CONFIG_FILES (of course requires not to have @token@ in the read.ini file), maybe not nice but pragmatic :) Why are the copies needed? Most of our tests working on input data files either create the files: check_PROGRAMS=readtest readtest_DEPENDENCIES=$(LIBDEPS) read.ini read.ini: Makefile echo ; autogen by Makefile.am read.ini �...@echo [test] read.ini �...@echo opt=val read.ini CLEANFILES=read.ini (BTW, is this right?) or have something like TESTS=readtest.sh with a readtest.sh (could even be autogenerated): test -z $srcdir srcdir=. ./readtest --ini-file=$srcdir/read.ini Is this done right and portable? oki, Steffen That's pretty much exactly how we have things for CouchDB tests, except that I wrote an Erlang source module that autoconf rewrites so it knows where the build and source directories. Then in test code, I just wrap file names with calls to source_file/1 and build_file/1 to get the correct paths. This means that when writing tests you only have to know if something is going to be part of the build/generated by tests or is part of the source tarball. I'm not exactly sure which copies you're referring to needing, but another pattern I've seen is to just have a build target that copies all files to the build directory so that the knowledge of paths doesn't need to be considered. You can just treat everything as relative to where your tests are run. HTH, Paul Davis
How to structure a GNU scientific project
I'm trying to organize the directory and file structure of my project and figure out how everything should be installed in a way that complies with the GCS and FHS. Currently, my project provides several command line binary executables that do some numerical number crunching using GNU GSL. The inputs to these binaries are a few human readable text files which specify some simulation parameters and settings. The outputs of the executables are a couple of data files (time histories, level curves, etc.) and a couple of text files that show the simulation settings that were actually used and specify the format of the output data. On top of this, I have some Python scripts which generate plots from the data files, and save them as pdfs. The python scripts call os.system to specify the simulation inputs, run the simulation, and process output data to make some plots. It also takes all inputs and outputs (input text files + output data + plots) and bundles them into a time-stamped tar.gz file so that multiple simulation runs don't overwrite each other, and to provide a way to go back and look at simulation results and know exactly what conditions created them. I have several sets of example input text files that allow a user to run the simulations with some default parameters. I'm not clear on where these files should be placed in the source distribution and where they should be installed during 'make install'. I would like a user to be able to easily find and open these text files so they can use them as templates for running simulations with different parameters. My questions are: 1) Where should I put these text simulation configuration files within my source distribution, and where should they be installed to by default? 2) How should I make my application and/or user aware of where they are installed? The way these files are used is by specifying a command line flag that directs the executable to parse a particular input file, so in order for this to be useful, the directory they are installed into must be known. 3) Should the python scripts go in site-packages, or would it make more sense for them to be installed alongside the binary executables? 4) I use the python scripts to save everything in a results/ folder (and I'm often working with my source directory, so this is src/results). It seems like this folder ought to be in the users home directory somewhere, but maybe there are other places it would make sense to put this type of output data? It seems like the text input files should go in a subdirectory of /usr/local/share, or maybe the whole project should go into a subdirectory of /opt. The way I'm using the tools is by having everything within a folder in my home directory, but this is probably not a good way to distribute. The basic source layout is: top-level -- has the standard GCS files and Autotools files src -- sources files which compile to executables, also has some python scripts for postprocessing the data src/common -- convenience libraries that are used by executables src/intialconditions -- text files for controlling initial simulation conditions src/parameters -- text files for specify different numerical values for simulation parameters src/integrationsettings -- text files for choosing different numerical integration settings There are anywhere from 5-20 text files in each of the src/initialconditions, src/parameters, src/integrationsettings folders that I want to distribute in some logical fashion. Any thoughts? Also, I'm using Autotools (Autoconf and Automake), so if there are good ways to do this automagically with these tools, that would be ideal. Thanks, ~Luke -- Dale L. Peterson Sports Biomechanics Lab, UC Davis http://dlpeterson.com/blog Office: +01 530-752-2163 Mobile: +01 805-698-6157
Re: execvp: /bin/sh: Argument list too long
Hello Pippijn, * Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 04:55:45PM CET: I looked through the mailing list archives and found several questions and proposed solutions related to this problem: - Split up file lists so several rules are generated: http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html - What is your env's size? http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html - It will fail. In other words Sorry, can't fix? http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html Also, we wrote a short section in the manual about this (in recent versions): info Automake Length Limitations My problem is that I can not make dist in one project and not make check in another project. Both have a large number of tests. The one where make check fails has a few more tests, but make dist will fail there, as well. One workaround is to put tests in more than one directory, with more than one Makefile.am. Yes, it is ugly, I know. Another is to require non-VPATH builds on those systems with the most severe restrictions, such as MSYS and IRIX (if you can't convince root to increase the limit). What are the plans on fixing this? We'll fix what we can fix. Without GNU make-specifics, that is hard. One thing I'd definitely like to see done is allowing multiple parallel-tests test suites in one Makefile.am, so you can still have a nonrecursive setup. That may mean that 'make dist' still fails, but my hope is that is not needed to work on every platform. Would that be sufficient for your needs? Are there any? Autoconf already checks for the maximum argument length, It's a configure check from a Libtool macro that does that, actually. but since by that time, Makefile.in is already generated, it can't help. Maybe storing the list of dist files and check files in a different file and reading this with xargs $file would help. One problem is that with many constructs, automake would like to know the exact set of files it deals with. We can try to deviate from that in some places but my guess is that will always look like a hack that avoids some but not all issues. One other strategy we maybe could implement is alternative GNU make-specific rules that avoid limitations by dealing with files with make functions. That is usually doable, but doing this consistently will be quite some work and cause quite some duplication in the Makefile.in. I don't have current plans to pursue this; you can see e.g., the write_entries_to_file macro in the gcc/Makefile.in file in the GCC tree for how this can be done in principle. Cheers, Ralf
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 08:03:06PM +0100, Ralf Wildenhues wrote:
* Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 04:55:45PM CET:
I looked through the mailing list archives and found several questions
and proposed solutions related to this problem:
- Split up file lists so several rules are generated:
http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html
- What is your env's size?
http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html
- It will fail. In other words Sorry, can't fix?
http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html
Also, we wrote a short section in the manual about this (in recent
versions):
info Automake Length Limitations
I hadn't seen that, before. Now I read it, I noticed something:
Automake itself employs a couple of strategies to avoid long command
lines. For example, when ‘${srcdir}/’ is prepended to file names, as
can happen with above $(data_DATA) lists, it limits the amount of
arguments passed to external commands.
Why doesn't it do that, always? Is the performance hit too large?
My problem is that I can not make dist in one project and not make check
in another project. Both have a large number of tests. The one where make
check fails has a few more tests, but make dist will fail there, as well.
One workaround is to put tests in more than one directory, with more
than one Makefile.am. Yes, it is ugly, I know. Another is to require
non-VPATH builds on those systems with the most severe restrictions,
such as MSYS and IRIX (if you can't convince root to increase the
limit).
I am root on my (Linux) system and I set the stack size to unlimited. The
libtool macro reported a few billion (or something other really large)
for maximum argument list length, bash also agreed (it easily executed
the distdir target when copied into a bash script), but make doesn't.
Both gnu make and pmake abort with the too long message.
What are the plans on fixing this?
We'll fix what we can fix. Without GNU make-specifics, that is hard.
One thing I'd definitely like to see done is allowing multiple
parallel-tests test suites in one Makefile.am, so you can still have a
nonrecursive setup. That may mean that 'make dist' still fails, but
my hope is that is not needed to work on every platform.
Would that be sufficient for your needs?
I'd be happy if make dist worked on my system, but it doesn't and I
didn't find a way to make it work, yet (any suggestions here?).
but since by that time, Makefile.in is
already generated, it can't help. Maybe storing the list of dist files
and check files in a different file and reading this with xargs $file
would help.
One problem is that with many constructs, automake would like to know
the exact set of files it deals with. We can try to deviate from that
in some places but my guess is that will always look like a hack that
avoids some but not all issues.
Automake can know the maximum set of files it deals with. Wouldn't that
help, already?
One other strategy we maybe could implement is alternative GNU
make-specific rules that avoid limitations by dealing with files with
make functions. That is usually doable, but doing this consistently
will be quite some work and cause quite some duplication in the
Makefile.in. I don't have current plans to pursue this;
I would definitely prefer no GNU make stuff in the generated makefiles.
you can see
e.g., the write_entries_to_file macro in the gcc/Makefile.in file in the
GCC tree for how this can be done in principle.
I'll take a look at it.
--
Pippijn van Steenhoven
signature.asc
Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote: I am root on my (Linux) system and I set the stack size to unlimited. The libtool macro reported a few billion (or something other really large) for maximum argument list length, bash also agreed (it easily executed the distdir target when copied into a bash script), but make doesn't. What version of GNU make are you using? There was an issue in GNU make 3.81 related to setting stack limits, that I believe is resolved in GNU make 3.82. I can't recall the details. You might try that if you're not already using it.
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 09:56:52PM +0100, Ralf Wildenhues wrote: [1] On some systems this is not true; e.g., GNU make evades the limit on MSYS for this particular point, IIRC using a response file. Can automake do something similar? The dist files are always the same (well, they should be), so they can be written to an extra file, which is then piped into xargs or something. I'd be happy if make dist worked on my system, but it doesn't and I didn't find a way to make it work, yet (any suggestions here?). You need to specify exactly where it fails if my previous suggestions were not enough. Showing the Makefile.am in question would help too. make distdir fails. The Makefile.am is not a single file. A combined file is here: http://paste.xinu.at/ou4jy/, but I'm not sure that is very useful. A tarball with all .am files is here: http://paste.xinu.at/TzadQ/ What version of GNU make are you using? There was an issue in GNU make 3.81 related to setting stack limits, that I believe is resolved in GNU make 3.82. I can't recall the details. I don't think it's that. It's that GNU make also requires the per-argument limit to be large, and IIRC that is still 128K on Linux (yes, speaking about the kernel only). I wrote about this before: http://thread.gmane.org/gmane.comp.gnu.make.bugs/4219 but my proposed hack back then was (understandably) not too popular, and I haven't gotten back to anything better yet. ;-) In other words, I can't make dist even with GNU make 3.82? What do you suggest, then? -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
* Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 09:01:12PM CET:
On Mon, Nov 08, 2010 at 08:03:06PM +0100, Ralf Wildenhues wrote:
Also, we wrote a short section in the manual about this (in recent
versions):
info Automake Length Limitations
I hadn't seen that, before. Now I read it, I noticed something:
Automake itself employs a couple of strategies to avoid long command
lines. For example, when ‘${srcdir}/’ is prepended to file names, as
can happen with above $(data_DATA) lists, it limits the amount of
arguments passed to external commands.
Why doesn't it do that, always? Is the performance hit too large?
That's not the point.
With code like this:
foo = some long list of files
target: $(foo)
$do_something_with $(foo)
the fork for the shell to $do_something_with may fail already if $(foo)
is too long[1]. The above strategy is only eligible for issues when
$(foo) itself is short enough to be expanded, but prepending $(srcdir)/
everywhere *will* make it go over the limit.
[1] On some systems this is not true; e.g., GNU make evades the limit on
MSYS for this particular point, IIRC using a response file.
What are the plans on fixing this?
We'll fix what we can fix. Without GNU make-specifics, that is hard.
One thing I'd definitely like to see done is allowing multiple
parallel-tests test suites in one Makefile.am, so you can still have a
nonrecursive setup. That may mean that 'make dist' still fails, but
my hope is that is not needed to work on every platform.
Would that be sufficient for your needs?
I'd be happy if make dist worked on my system, but it doesn't and I
didn't find a way to make it work, yet (any suggestions here?).
You need to specify exactly where it fails if my previous suggestions
were not enough. Showing the Makefile.am in question would help too.
but since by that time, Makefile.in is
already generated, it can't help. Maybe storing the list of dist files
and check files in a different file and reading this with xargs $file
would help.
One problem is that with many constructs, automake would like to know
the exact set of files it deals with. We can try to deviate from that
in some places but my guess is that will always look like a hack that
avoids some but not all issues.
Automake can know the maximum set of files it deals with. Wouldn't that
help, already?
Well, that's not the hard part. automake would need to rewrite long
lists of files into separate variables, and use them in separate
recipes. Generating sets of such recipes for several targets is
probably doable, but not straight-forward. Throw conditionals and
@substed@ parts into the mix, and it can easily get a mess.
* Paul Smith wrote on Mon, Nov 08, 2010 at 09:03:59PM CET:
On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote:
I am root on my (Linux) system and I set the stack size to unlimited.
The libtool macro reported a few billion (or something other really
large) for maximum argument list length, bash also agreed (it easily
executed the distdir target when copied into a bash script), but
make doesn't.
What version of GNU make are you using? There was an issue in GNU make
3.81 related to setting stack limits, that I believe is resolved in GNU
make 3.82. I can't recall the details.
I don't think it's that. It's that GNU make also requires the
per-argument limit to be large, and IIRC that is still 128K on Linux
(yes, speaking about the kernel only). I wrote about this before:
http://thread.gmane.org/gmane.comp.gnu.make.bugs/4219
but my proposed hack back then was (understandably) not too popular,
and I haven't gotten back to anything better yet. ;-)
Cheers,
Ralf
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 03:03:59PM -0500, Paul Smith wrote: On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote: I am root on my (Linux) system and I set the stack size to unlimited. The libtool macro reported a few billion (or something other really large) for maximum argument list length, bash also agreed (it easily executed the distdir target when copied into a bash script), but make doesn't. What version of GNU make are you using? There was an issue in GNU make 3.81 related to setting stack limits, that I believe is resolved in GNU make 3.82. I can't recall the details. You might try that if you're not already using it. Ah, I'm using 3.81. -- Pippijn van Steenhoven signature.asc Description: Digital signature
bug#7333: [PATCH] {maint} Fix a bug in variable concatanation with `+='.
merge 7333 7345 tags 7333 + patch close 7333 v1.11-222-g7a020d6 thanks * Stefano Lattarini wrote on Sun, Nov 07, 2010 at 03:26:52PM CET: Now, should I also try to close the bug #7333 on debbugs? The instruction at http://debbugs.gnu.org/Developer.html seems quite clear about how to do so, so I think I can manage to get it right. BTW, Ouch! I see that my previous reply presenting the patch has erroneously opended a new, spurious bug report (#7345) in the tracker! Ralf, could you please you close that report as invalid? Not sure how that happened, probably the Subject: change. The commands above (Bcc:ed to control at debbugs) should merge and close both bugs. Cheers, Ralf
