execvp: /bin/sh: Argument list too long
Hi, I looked through the mailing list archives and found several questions and proposed solutions related to this problem: - Split up file lists so several rules are generated: http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html - "What is your env's size?" http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html - It will fail. In other words "Sorry, can't fix"? http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html My problem is that I can not make dist in one project and not make check in another project. Both have a large number of tests. The one where make check fails has a few more tests, but make dist will fail there, as well. What are the plans on fixing this? Are there any? Autoconf already checks for the maximum argument length, but since by that time, Makefile.in is already generated, it can't help. Maybe storing the list of dist files and check files in a different file and reading this with xargs < $file would help. Any other ideas? -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
Hello Pippijn, * Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 04:55:45PM CET: > I looked through the mailing list archives and found several questions > and proposed solutions related to this problem: > > - Split up file lists so several rules are generated: > http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html > > - "What is your env's size?" > http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html > > - It will fail. In other words "Sorry, can't fix"? > http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html Also, we wrote a short section in the manual about this (in recent versions): info Automake "Length Limitations" > My problem is that I can not make dist in one project and not make check > in another project. Both have a large number of tests. The one where make > check fails has a few more tests, but make dist will fail there, as well. One workaround is to put tests in more than one directory, with more than one Makefile.am. Yes, it is ugly, I know. Another is to require non-VPATH builds on those systems with the most severe restrictions, such as MSYS and IRIX (if you can't convince root to increase the limit). > What are the plans on fixing this? We'll fix what we can fix. Without GNU make-specifics, that is hard. One thing I'd definitely like to see done is allowing multiple parallel-tests test suites in one Makefile.am, so you can still have a nonrecursive setup. That may mean that 'make dist' still fails, but my hope is that is not needed to work on every platform. Would that be sufficient for your needs? > Are there any? Autoconf already checks > for the maximum argument length, It's a configure check from a Libtool macro that does that, actually. > but since by that time, Makefile.in is > already generated, it can't help. Maybe storing the list of dist files > and check files in a different file and reading this with xargs < $file > would help. One problem is that with many constructs, automake would like to know the exact set of files it deals with. We can try to deviate from that in some places but my guess is that will always look like a hack that avoids some but not all issues. One other strategy we maybe could implement is alternative GNU make-specific rules that avoid limitations by dealing with files with make functions. That is usually doable, but doing this consistently will be quite some work and cause quite some duplication in the Makefile.in. I don't have current plans to pursue this; you can see e.g., the write_entries_to_file macro in the gcc/Makefile.in file in the GCC tree for how this can be done in principle. Cheers, Ralf
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 08:03:06PM +0100, Ralf Wildenhues wrote:
> * Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 04:55:45PM CET:
> > I looked through the mailing list archives and found several questions
> > and proposed solutions related to this problem:
> >
> > - Split up file lists so several rules are generated:
> > http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html
> >
> > - "What is your env's size?"
> > http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html
> >
> > - It will fail. In other words "Sorry, can't fix"?
> > http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html
>
> Also, we wrote a short section in the manual about this (in recent
> versions):
> info Automake "Length Limitations"
I hadn't seen that, before. Now I read it, I noticed something:
Automake itself employs a couple of strategies to avoid long command
lines. For example, when ‘${srcdir}/’ is prepended to file names, as
can happen with above $(data_DATA) lists, it limits the amount of
arguments passed to external commands.
Why doesn't it do that, always? Is the performance hit too large?
> > My problem is that I can not make dist in one project and not make check
> > in another project. Both have a large number of tests. The one where make
> > check fails has a few more tests, but make dist will fail there, as well.
>
> One workaround is to put tests in more than one directory, with more
> than one Makefile.am. Yes, it is ugly, I know. Another is to require
> non-VPATH builds on those systems with the most severe restrictions,
> such as MSYS and IRIX (if you can't convince root to increase the
> limit).
I am root on my (Linux) system and I set the stack size to unlimited. The
libtool macro reported a few billion (or something other really large)
for maximum argument list length, bash also agreed (it easily executed
the "distdir" target when copied into a bash script), but make doesn't.
Both gnu make and pmake abort with the "too long" message.
> > What are the plans on fixing this?
>
> We'll fix what we can fix. Without GNU make-specifics, that is hard.
> One thing I'd definitely like to see done is allowing multiple
> parallel-tests test suites in one Makefile.am, so you can still have a
> nonrecursive setup. That may mean that 'make dist' still fails, but
> my hope is that is not needed to work on every platform.
>
> Would that be sufficient for your needs?
I'd be happy if make dist worked on my system, but it doesn't and I
didn't find a way to make it work, yet (any suggestions here?).
> > but since by that time, Makefile.in is
> > already generated, it can't help. Maybe storing the list of dist files
> > and check files in a different file and reading this with xargs < $file
> > would help.
>
> One problem is that with many constructs, automake would like to know
> the exact set of files it deals with. We can try to deviate from that
> in some places but my guess is that will always look like a hack that
> avoids some but not all issues.
Automake can know the maximum set of files it deals with. Wouldn't that
help, already?
> One other strategy we maybe could implement is alternative GNU
> make-specific rules that avoid limitations by dealing with files with
> make functions. That is usually doable, but doing this consistently
> will be quite some work and cause quite some duplication in the
> Makefile.in. I don't have current plans to pursue this;
I would definitely prefer no GNU make stuff in the generated makefiles.
> you can see
> e.g., the write_entries_to_file macro in the gcc/Makefile.in file in the
> GCC tree for how this can be done in principle.
I'll take a look at it.
--
Pippijn van Steenhoven
signature.asc
Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote: > I am root on my (Linux) system and I set the stack size to unlimited. > The libtool macro reported a few billion (or something other really > large) for maximum argument list length, bash also agreed (it easily > executed the "distdir" target when copied into a bash script), but > make doesn't. What version of GNU make are you using? There was an issue in GNU make 3.81 related to setting stack limits, that I believe is resolved in GNU make 3.82. I can't recall the details. You might try that if you're not already using it.
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 09:56:52PM +0100, Ralf Wildenhues wrote: > [1] On some systems this is not true; e.g., GNU make evades the limit on > MSYS for this particular point, IIRC using a response file. Can automake do something similar? The dist files are always the same (well, they should be), so they can be written to an extra file, which is then piped into xargs or something. > > I'd be happy if make dist worked on my system, but it doesn't and I > > didn't find a way to make it work, yet (any suggestions here?). > > You need to specify exactly where it fails if my previous suggestions > were not enough. Showing the Makefile.am in question would help too. "make distdir" fails. The Makefile.am is not a single file. A combined file is here: http://paste.xinu.at/ou4jy/, but I'm not sure that is very useful. A tarball with all .am files is here: http://paste.xinu.at/TzadQ/ > > What version of GNU make are you using? There was an issue in GNU make > > 3.81 related to setting stack limits, that I believe is resolved in GNU > > make 3.82. I can't recall the details. > > I don't think it's that. It's that GNU make also requires the > per-argument limit to be large, and IIRC that is still 128K on Linux > (yes, speaking about the kernel only). I wrote about this before: > http://thread.gmane.org/gmane.comp.gnu.make.bugs/4219 > but my proposed hack back then was (understandably) not too popular, > and I haven't gotten back to anything better yet. ;-) In other words, I can't make dist even with GNU make 3.82? What do you suggest, then? -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
* Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 09:01:12PM CET:
> On Mon, Nov 08, 2010 at 08:03:06PM +0100, Ralf Wildenhues wrote:
> > Also, we wrote a short section in the manual about this (in recent
> > versions):
> > info Automake "Length Limitations"
>
> I hadn't seen that, before. Now I read it, I noticed something:
>
> Automake itself employs a couple of strategies to avoid long command
> lines. For example, when ‘${srcdir}/’ is prepended to file names, as
> can happen with above $(data_DATA) lists, it limits the amount of
> arguments passed to external commands.
>
> Why doesn't it do that, always? Is the performance hit too large?
That's not the point.
With code like this:
foo = some long list of files
target: $(foo)
$do_something_with $(foo)
the fork for the shell to $do_something_with may fail already if $(foo)
is too long[1]. The above strategy is only eligible for issues when
$(foo) itself is short enough to be expanded, but prepending $(srcdir)/
everywhere *will* make it go over the limit.
[1] On some systems this is not true; e.g., GNU make evades the limit on
MSYS for this particular point, IIRC using a response file.
> > > What are the plans on fixing this?
> >
> > We'll fix what we can fix. Without GNU make-specifics, that is hard.
> > One thing I'd definitely like to see done is allowing multiple
> > parallel-tests test suites in one Makefile.am, so you can still have a
> > nonrecursive setup. That may mean that 'make dist' still fails, but
> > my hope is that is not needed to work on every platform.
> >
> > Would that be sufficient for your needs?
>
> I'd be happy if make dist worked on my system, but it doesn't and I
> didn't find a way to make it work, yet (any suggestions here?).
You need to specify exactly where it fails if my previous suggestions
were not enough. Showing the Makefile.am in question would help too.
> > > but since by that time, Makefile.in is
> > > already generated, it can't help. Maybe storing the list of dist files
> > > and check files in a different file and reading this with xargs < $file
> > > would help.
> >
> > One problem is that with many constructs, automake would like to know
> > the exact set of files it deals with. We can try to deviate from that
> > in some places but my guess is that will always look like a hack that
> > avoids some but not all issues.
>
> Automake can know the maximum set of files it deals with. Wouldn't that
> help, already?
Well, that's not the hard part. automake would need to rewrite long
lists of files into separate variables, and use them in separate
recipes. Generating sets of such recipes for several targets is
probably doable, but not straight-forward. Throw conditionals and
@substed@ parts into the mix, and it can easily get a mess.
* Paul Smith wrote on Mon, Nov 08, 2010 at 09:03:59PM CET:
> On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote:
> > I am root on my (Linux) system and I set the stack size to unlimited.
> > The libtool macro reported a few billion (or something other really
> > large) for maximum argument list length, bash also agreed (it easily
> > executed the "distdir" target when copied into a bash script), but
> > make doesn't.
>
> What version of GNU make are you using? There was an issue in GNU make
> 3.81 related to setting stack limits, that I believe is resolved in GNU
> make 3.82. I can't recall the details.
I don't think it's that. It's that GNU make also requires the
per-argument limit to be large, and IIRC that is still 128K on Linux
(yes, speaking about the kernel only). I wrote about this before:
http://thread.gmane.org/gmane.comp.gnu.make.bugs/4219
but my proposed hack back then was (understandably) not too popular,
and I haven't gotten back to anything better yet. ;-)
Cheers,
Ralf
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 03:03:59PM -0500, Paul Smith wrote: > On Mon, 2010-11-08 at 21:01 +0100, Pippijn van Steenhoven wrote: > > I am root on my (Linux) system and I set the stack size to unlimited. > > The libtool macro reported a few billion (or something other really > > large) for maximum argument list length, bash also agreed (it easily > > executed the "distdir" target when copied into a bash script), but > > make doesn't. > > What version of GNU make are you using? There was an issue in GNU make > 3.81 related to setting stack limits, that I believe is resolved in GNU > make 3.82. I can't recall the details. > > You might try that if you're not already using it. Ah, I'm using 3.81. -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
Pippijn van Steenhoven wrote: > I am root on my (Linux) system and I set the stack size to unlimited. The > libtool macro reported a few billion (or something other really large) > for maximum argument list length, bash also agreed (it easily executed > the "distdir" target when copied into a bash script), but make doesn't. > Both gnu make and pmake abort with the "too long" message. What Linux kernel version are you using? Note that linux-2.6.23 and later kernels have changed how this is handled. http://www.in-ulm.de/~mascheck/various/argmax/ http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba http://www.gnu.org/software/coreutils/faq/#Argument-list-too-long Bob
Re: execvp: /bin/sh: Argument list too long
On Tue, Nov 09, 2010 at 10:37:24AM -0700, Bob Proulx wrote: > Pippijn van Steenhoven wrote: > > I am root on my (Linux) system and I set the stack size to unlimited. The > > libtool macro reported a few billion (or something other really large) > > for maximum argument list length, bash also agreed (it easily executed > > the "distdir" target when copied into a bash script), but make doesn't. > > Both gnu make and pmake abort with the "too long" message. > > What Linux kernel version are you using? Note that linux-2.6.23 and > later kernels have changed how this is handled. I'm using a new enough kernel. Like I said, copy/pasting the lines executed by "make distdir" into a shell script and executing that works fine, it's just in make that it doesn't work. To be more explicit: http://paste.xinu.at/WzKP/ works (shell script) http://paste.xinu.at/h7kj/ does not work (makefile) max_cmd_len is 3458764513820540925, which should suffice. It's just that linux still has a 128K per-argument limit, which is apparently what chokes make. -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
On Mon, Nov 08, 2010 at 10:11:20PM +0100, Pippijn van Steenhoven wrote: > "make distdir" fails. The Makefile.am is not a single file. A combined > file is here: http://paste.xinu.at/ou4jy/, but I'm not sure that is very > useful. A tarball with all .am files is here: http://paste.xinu.at/TzadQ/ In case the paste site will delete the tarball. Here it is again: http://xinutec.org/~pippijn/files/up/makefiles.tar.gz -- Pippijn van Steenhoven signature.asc Description: Digital signature
Re: execvp: /bin/sh: Argument list too long
Hello Pippijn, * Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 10:11:20PM CET: > On Mon, Nov 08, 2010 at 09:56:52PM +0100, Ralf Wildenhues wrote: > > [1] On some systems this is not true; e.g., GNU make evades the limit on > > MSYS for this particular point, IIRC using a response file. > > Can automake do something similar? The dist files are always the same > (well, they should be), so they can be written to an extra file, which is > then piped into xargs or something. In principle: yes, when the list of files is completely known at automake run time. This would mean, no @substituted@ bits in that macro anywhere, no conditional bits, and you don't get to override macros at make run time like make DISTFILES='foo bar baz' These new limitations would not seem very harsh for most packages; especially for the distdir target, most conditionals are ineffective, and users are not likely to want to override them at make run time, but disallowing subtitutions would be a real limitation. Even just reordering the list of files might introduce bugs with some (admittedly fragile) setups, with packages that rely on dependency order (thus of course disallowing parallel execution) or that contain e.g., both FILE and ../sub/FILE in the list somewhere. (We do have measures in place to cope with FILE and $(srcdir)/FILE so the latter point should be really rare.) > > > What version of GNU make are you using? There was an issue in GNU make > > > 3.81 related to setting stack limits, that I believe is resolved in GNU > > > make 3.82. I can't recall the details. > > > > I don't think it's that. It's that GNU make also requires the > > per-argument limit to be large, and IIRC that is still 128K on Linux > > (yes, speaking about the kernel only). I wrote about this before: > > http://thread.gmane.org/gmane.comp.gnu.make.bugs/4219 > > but my proposed hack back then was (understandably) not too popular, > > and I haven't gotten back to anything better yet. ;-) > > In other words, I can't make dist even with GNU make 3.82? What do you > suggest, then? Well, you could try my patch from http://article.gmane.org/gmane.comp.gnu.make.bugs/4219 (append /raw to the URL to get it unmangled). I still think letting GNU make avoid the issue on Linux systems would be the easiest way out here. As already written, alternative workarounds include reorganizing your code a bit to have a bit more recursive makefile setup, so that each makefiles' list would remain below the limit. Cheers, Ralf
Re: execvp: /bin/sh: Argument list too long
On Thu, Nov 11, 2010 at 09:54:35PM +0100, Ralf Wildenhues wrote: > > In other words, I can't make dist even with GNU make 3.82? What do you > > suggest, then? > > Well, you could try my patch from > http://article.gmane.org/gmane.comp.gnu.make.bugs/4219 > (append /raw to the URL to get it unmangled). > > I still think letting GNU make avoid the issue on Linux systems would be > the easiest way out here. > > As already written, alternative workarounds include reorganizing your > code a bit to have a bit more recursive makefile setup, so that each > makefiles' list would remain below the limit. Hi Ralf, I was able to reduce the number of files sufficiently so I am currently not hitting the limit. In the future, when more tests are added, I will go for your second suggestion. The testsuite/ directory won't suffer much from a recursive setup. I just need to move out the .as -> .c build rule to a common rules file included in all the testsuite makefiles as well as the main build makefile. Thanks, Pippijn signature.asc Description: Digital signature
