Hello bug-make readers, Xan Lopez brought up an interesting Automake issue about long commands that led us to an issue GNU make could be more helpful with[1][2]:
Linux >= 2.6.23 has removed the in-kernel hardwired command line length limit[3]. So, once stack size has been increased sufficiently with ulimit -s 32768 or so, one could hope to use longer commands in makefiles. However, there is also a new limit on the length of each individual command line argument[4], MAX_ARG_STRLEN (131072). This kills the advantage when GNU make uses sh -c 'very long shell command line now all in one arg' for the command, i.e., iff the command contains any shell active stuff. I don't like this inconsistency, I would like complex commands to have no smaller limit than simple ones. :-) 1) One possibility would be to drop -c and pipe the command line to sh on standard input. Just like parallel make, this would prevent using the standard input of the make process within the child command process. 2) Another possibility would be a hack like the following: split the long line into N substrings, each below the argument limit, and invoke sh -c 'eval "${1}${2}...${10}...${N}"' sh quoted-substring1 ... This changes the following user-observable semantics: - the positional parameters have different settings; this can be fixed with a 'set x; shift;' at the start of the eval'ed string, - LINENO is not reliable in eval with some shells. - ${10} and up are not portable to pre-Posix shells like Solaris sh and the heirloom shell under Linux (See autoconf.info 'Limitations of Builtins' for more details on these issues.) I don't think these cause big problems in practice, more so if we go this way only with command line lengths that would otherwise have failed anyway, and only on Linux systems. 3) Yet another possibility would be to ask the kernel developers to increase MAX_ARG_STRLEN. However, since there seems to be some safety issue involved in its value (see links referenced in [3]), I doubt that would be the most successful approach. What do you think? Below is an initial patch to do (2); the index computations probably still need fixes. There is the problem that the MAX_ARG_STRLEN value is obtained at compile time, and only if those headers are present, but 'make' may be deployed on a system with a different setting later. And I've probably failed to follow some coding guidelines for GNU make, and broken some unrelated platform. I've blissfully assumed the number of substrings would have at most 5 digits, and failed to produce test suite additions so far. The evaled string contains many more parameter expansions than are normally needed, this is just supposed to be an upper bound on them (any further parameters are empty anyway). Tested on GNU/Linux only, with the test suite and the example from [2], and several shells. Also tested with 'make SHELL=/bin/sh\ -x' to ensure that, when we set $0 the '-x' does not end up munged in $1 (this code probably has a bug if SHELL contains quotes or backslashes or so). For my system and with a stack size of 32M, the patch allows 2^6 times longer command lines in the makefile (roughly 7M), which should buy WebKitGTK+ some time until hitting the next limit. Thanks, Ralf [1] http://lists.gnu.org/archive/html/bug-autoconf/2009-07/msg00038.html [2] http://lists.gnu.org/archive/html/bug-autoconf/2009-07/msg00046.html [3] http://www.in-ulm.de/~mascheck/various/argmax/#linux [4] http://www.in-ulm.de/~mascheck/various/argmax/#maximum_number 2009-07-29 Ralf Wildenhues <ralf.wildenh...@gmx.de> * configure.in: Check for sys/user.h and linux/binfmts.h headers. * job.c: Include them if available. (construct_command_argv_internal): When constructing the command line with 'sh -c', use multiple arguments together with eval expansion to evade the Linux per-argument length limit MAX_ARG_STRLEN if it is defined. Problem reported against Automake by Xan Lopez <x...@gnome.org>. Index: configure.in =================================================================== RCS file: /cvsroot/make/make/configure.in,v retrieving revision 1.150 diff -u -r1.150 configure.in --- configure.in 4 Jun 2009 06:30:27 -0000 1.150 +++ configure.in 29 Jul 2009 19:08:55 -0000 @@ -64,7 +64,8 @@ AC_HEADER_STAT AC_HEADER_TIME AC_CHECK_HEADERS(stdlib.h locale.h unistd.h limits.h fcntl.h string.h \ - memory.h sys/param.h sys/resource.h sys/time.h sys/timeb.h) + memory.h sys/param.h sys/resource.h sys/time.h sys/timeb.h \ + sys/user.h linux/binfmts.h) # Set a flag if we have an ANSI C compiler if test "$ac_cv_prog_cc_stdc" != no; then Index: job.c =================================================================== RCS file: /cvsroot/make/make/job.c,v retrieving revision 1.193 diff -u -r1.193 job.c --- job.c 9 Jun 2009 15:35:38 -0000 1.193 +++ job.c 29 Jul 2009 20:44:04 -0000 @@ -29,6 +29,11 @@ #include <string.h> +#if defined (HAVE_LINUX_BINFMTS_H) && defined (HAVE_SYS_USER_H) +#include <sys/user.h> +#include <linux/binfmts.h> +#endif + /* Default shell to use. */ #ifdef WINDOWS32 #include <windows.h> @@ -2717,9 +2722,18 @@ #endif unsigned int line_len = strlen (line); +#ifdef MAX_ARG_STRLEN + static char eval_line[] = "eval\\ \\\"set\\ x\\;\\ shift\\;\\ "; +#define ARG_NUMBER_DIGITS 5 +#define EVAL_LEN (sizeof(eval_line)-1 + shell_len + 4 \ + + (7 + ARG_NUMBER_DIGITS) * 2 * line_len / (MAX_ARG_STRLEN - 2)) +#else +#define EVAL_LEN 0 +#endif char *new_line = alloca (shell_len + (sizeof (minus_c)-1) - + (line_len*2) + 1); + + (line_len*2) + 1 + EVAL_LEN); char *command_ptr = NULL; /* used for batch_mode_shell mode */ + char *args_ptr; # ifdef __EMX__ /* is this necessary? */ if (!unixy_shell) @@ -2732,6 +2746,30 @@ memcpy (ap, minus_c, sizeof (minus_c) - 1); ap += sizeof (minus_c) - 1; command_ptr = ap; + +#if !defined (WINDOWS32) && defined (MAX_ARG_STRLEN) + if (unixy_shell && line_len > MAX_ARG_STRLEN) + { + unsigned j; + memcpy (ap, eval_line, sizeof (eval_line) - 1); + ap += sizeof (eval_line) - 1; + for (j = 1; j <= 2 * line_len / (MAX_ARG_STRLEN - 2); j++) + ap += sprintf (ap, "\\$\\{%u\\}", j); + *ap++ = '\\'; + *ap++ = '"'; + *ap++ = ' '; + /* Copy only the first word of SHELL to $0. */ + for (p = shell; *p != '\0'; ++p) + { + if (isspace ((unsigned char)*p)) + break; + *ap++ = *p; + } + *ap++ = ' '; + } +#endif + args_ptr = ap; + for (p = line; *p != '\0'; ++p) { if (restp != NULL && *p == '\n') @@ -2779,6 +2817,14 @@ } #endif *ap++ = *p; + +#if !defined (WINDOWS32) && defined (MAX_ARG_STRLEN) + if (unixy_shell && line_len > MAX_ARG_STRLEN && (ap - args_ptr > MAX_ARG_STRLEN - 2)) + { + *ap++ = ' '; + args_ptr = ap; + } +#endif } if (ap == new_line + shell_len + sizeof (minus_c) - 1) /* Line was empty. */ _______________________________________________ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make