For Your Consideration ====================== Hello,
this is an implementation of a new option for Make. It controls the number of parallel jobs in regards to the amount of memory in use on a host. It allows one to use Make's parallel job execution on hosts with multiple CPU cores, but where memory is limited. It works similar to the -l, --load-average option and has to be used together with the -j, --jobs option. For example, assuming a host with 2GB of memory, the following options are equivalent: -u25%, -u0.25, -u512m, or -u0.5g In this example will Make stop creating new jobs when a host's memory use reaches 512MB (-j1), and will resume when it drops back below the threshold (-jN). A memory threshold can be specified as either an absolute value or as a percentage of the host's total memory. When the option is used in combination with -l, --load-average then Make stops creating new jobs when either the load threshold has been reached, or when the memory threshold has been reached. In addition will Make use the load and memory thresholds as a hint for a job's average memory use and scales the number of jobs (from -j1 to -jN). This reduces load and memory spikes near the thresholds and stabilizes the memory usage of a Make run. A value of 0 turns it off (-u0) and sets it to unlimited, which is also the default. The attached patch is against release 4.3 of Make. It implements the -u option for Linux using /proc/meminfo, and it enables the use of /proc/loadavg for the -l option. How well does it work? ====================== To show its effectiveness in practise did I create graphs for system loads (upper graphs) and memory usages (lower graphs) of different Make runs. In these graphs is an average shown (thicker line) together with the raw values (thinner lines, spikes). Because the memory use can vary between runs did I animate the graphs by using 5 Make runs. The "std-" labels in the legends mean that the standard Make 4.3 release was used. The "new-" labels mean the patched version was used. The following is a Linux kernel build on a 16-core CPU with 32GB of memory. The first graphs (black) show standard runs with only the -j option. The second graphs (red) show standard runs with -j and -l. The third graphs (green) show runs with -j and -l, using /proc/loadavg under Linux. The forth graphs (blue) show runs with -j, -l and -u combined, using /proc/loadavg and /proc/meminfo. The additional graphs show variations of the arguments to the -u option: https://zippyimage.com/images/2022/02/12/183964b428e4a0f0530c647c923924b9.gif The same kernel build on a 4-core SoC with 512MB: https://zippyimage.com/images/2022/02/12/aa05eb906c4bb1f6eb969b5bc29b7731.gif To test the limits did I use a Makefile from image processing. It consists of 3 rules and processes 30 files, leaving little room for Make to control its jobs. The jobs themselves use multiple threads, which inflate the system load, and use a fair amount of memory: https://zippyimage.com/images/2022/02/12/189b22aa75adc0a6b7a34b6b9cceabee.gif A more practical test is a build of LLVM/Clang-13 with itself, using link time optimization (LTO). The Makefiles are CMake-generated. Here a link job with LTO can use up to 24GB of memory. The host itself has got 32GB of RAM plus 16GB of ZRAM for swap, making it near impossible to build with a fixed number of parallel jobs. A "make -j8" will consistently run out of memory and out of swap space (red, failed). A "make -j8 -u30%" will build it 5 times without failing. Other values for -u work, too, however only builds, which succeeded at least 5 times and not failed once, are shown in the graphs. The combination of -l with -u together works better and allows to build with "-j16 -l16 -u90%" in under an hour: https://zippyimage.com/images/2022/02/12/9837a5551cc9350c17806c7b57efb991.gif The last test is a LLVM/Clang-13 LTO build with "-j16 -l16 -u90%" and CCACHE enabled. It speeds up the build time and brings the LTO link jobs closer together, causing them to further overlap. It builds 5 times without failing: https://zippyimage.com/images/2022/02/12/a55d2c3352a134edcb6af3feed8c2ff6.gif The patch is provided "as is". I hope you enjoy it and find it useful. Regards, Sven
diff -r -u make-4.3/src/job.c make-4.3.1/src/job.c --- make-4.3/src/job.c 2020-01-19 20:32:59.000000000 +0000 +++ make-4.3.1/src/job.c 2022-02-11 14:09:08.074210132 +0000 @@ -221,6 +221,7 @@ static void free_child (struct child *); static void start_job_command (struct child *child); static int load_too_high (void); +static int memory_too_high (void); static int job_next_command (struct child *); static int start_waiting_job (struct child *); @@ -1616,7 +1617,7 @@ /* If we are running at least one job already and the load average is too high, make this one wait. */ if (!c->remote - && ((job_slots_used > 0 && load_too_high ()) + && ((job_slots_used > 0 && (load_too_high () || memory_too_high ())) #ifdef WINDOWS32 || process_table_full () #endif @@ -2017,8 +2018,9 @@ OK, I'm not sure exactly how to handle that, but for sure we need to clamp this value at the number of cores before this can be enabled. */ -#define PROC_FD_INIT -1 - static int proc_fd = PROC_FD_INIT; +#ifdef linux + static int proc_fd = -2; +#endif double load, guess; time_t now; @@ -2032,6 +2034,7 @@ if (max_load_average < 0) return 0; +#ifdef linux /* If we haven't tried to open /proc/loadavg, try now. */ #define LOADAVG "/proc/loadavg" if (proc_fd == -2) @@ -2093,6 +2096,7 @@ close (proc_fd); proc_fd = -1; } +#endif /* linux */ /* Find the real system load average. */ make_access (); @@ -2138,6 +2142,111 @@ #endif } +static int +memory_too_high (void) +{ +#ifdef linux + static int proc_fd = -2; + + if (max_memory_used == 0.0) + return 0; + +#define MEMINFO "/proc/meminfo" + if (proc_fd == -2) + { + EINTRLOOP (proc_fd, open (MEMINFO, O_RDONLY)); + if (proc_fd < 0) + DB (DB_JOBS, ("Cannot use " MEMINFO " as memory use detection method.\n")); + else + { + DB (DB_JOBS, ("Using " MEMINFO " memory use detection method.\n")); + fd_noinherit (proc_fd); + } + } + + if (proc_fd >= 0) + { + int r; + + EINTRLOOP (r, lseek (proc_fd, 0, SEEK_SET)); + if (r >= 0) + { +#define PROC_MEMINFO_SIZE 256 + char mem[PROC_MEMINFO_SIZE+1]; + + EINTRLOOP (r, read (proc_fd, mem, PROC_MEMINFO_SIZE)); + if (r >= 0) + { + const char *p; + char *end; + double total = -1.0, avail = -1.0, used; + + /* The structure of /proc/meminfo is: + MemTotal: ... kB\n + MemFree: ... kB\n + MemAvailable: ... kB\n + ... + The difference between MemTotal and MemAvailable is + the amount of memory currently used. */ + mem[r] = '\0'; + p = strchr (mem, ':'); + if (p) + total = strtod (++p, &end); + if (end > p) + { + p = strchr (end, ':'); + if (p) + p = strchr (p+1, ':'); + if (p) + avail = strtod (p+1, &end); + } + used = total - avail; + if (total <= 0.0 || avail < 0.0 || used < 0.0) + { + /* The reported values in /proc/meminfo are invalid. */ + DB (DB_JOBS, ("Mem = %.3f kB, Avail = %.3f kB, Used = %.3f kB\n", + total, avail, used)); + return 1; + } + if (max_memory_used <= 1.0) + /* We are using percentages, so scale down. */ + used /= total; + if (used >= max_memory_used) + /* The limit has been reached, so no new jobs for now. */ + return 1; + if (max_load_average >= 1.0) + { + /* When a load average is specified together with a + memory limit do we scale the number of jobs with + the amount of memory used in order to stabilize + the memory use and to reduce load and memory + spikes. + + The function here creates a balance between speed + at a low memory use (creating slightly more jobs) + and linear scalability at a high memory use. + */ + double rt = used / max_memory_used; + + return job_slots_used + 1 >= + max_load_average * (1.0 - (2.0 - rt) * rt * rt); + } + return 0; + } + } + + /* If we got here, something went wrong. Give up on this method. */ + if (r < 0) + DB (DB_JOBS, ("Failed to read " MEMINFO ": %s\n", strerror (errno))); + + close (proc_fd); + proc_fd = -1; + } +#endif /* linux */ + + return 0; +} + /* Start jobs that are waiting for the load to be lower. */ void diff -r -u make-4.3/src/main.c make-4.3.1/src/main.c --- make-4.3/src/main.c 2020-01-19 20:32:59.000000000 +0000 +++ make-4.3.1/src/main.c 2022-02-11 14:20:07.372986002 +0000 @@ -36,6 +36,9 @@ #ifdef HAVE_STRINGS_H # include <strings.h> /* for strcasecmp */ #endif +#ifdef HAVE_STDLIB_H +# include <stdlib.h> /* for strtod */ +#endif # include "pathstuff.h" # include "sub_proc.h" # include "w32err.h" @@ -279,6 +282,13 @@ double max_load_average = -1.0; double default_load_average = -1.0; +/* Maximum memory usage upto which multiple jobs will be run. Zero + means unlimited, values between 0.0 and 1.0 are treated as a + percentage of total memory, greater values are treated as + absolute values in kilobytes. */ +const char *memory_used_arg = NULL; +double max_memory_used = 0.0; + /* List of directories given with -C switches. */ static struct stringlist *directories = 0; @@ -397,6 +407,9 @@ N_("\ --trace Print tracing information.\n"), N_("\ + -u [N], --memory-used=[N] Don't start multiple jobs when system memory use\n\ + is above N.\n"), + N_("\ -v, --version Print the version number of make and exit.\n"), N_("\ -w, --print-directory Print the current directory.\n"), @@ -451,6 +464,7 @@ &default_load_average, "load-average" }, { 'o', filename, &old_files, 0, 0, 0, 0, 0, "old-file" }, { 'O', string, &output_sync_option, 1, 1, 0, "target", 0, "output-sync" }, + { 'u', string, &memory_used_arg, 1, 1, 0, "0", "0", "memory-used" }, { 'W', filename, &new_files, 0, 0, 0, 0, 0, "what-if" }, /* These are long-style options. */ @@ -799,6 +813,58 @@ #endif } +/* Decode the -u, --memory-used argument. A number between 0.0 and 1.0 + is take as a percentage of the reported total memory. Numbers + greater 1.0 can be followed by a unit such as k, m, or g, or the + percent sign. Absolute values greater than 1.0 are represented in + kilobytes internally. */ + +static void +decode_memory_used_arg (void) +{ + if (memory_used_arg != NULL) + { + double value; + char *end; + + value = strtod (memory_used_arg, &end); + if (value > 0.0 && end != NULL && *end) + { + switch (tolower (*end)) + { + case 'k': /* kilo */ + break; + case 'm': /* mega */ + value *= (double) (1UL<<10); + break; + case 'g': /* giga */ + value *= (double) (1UL<<20); + break; + case 't': /* tera */ + value *= (double) (1UL<<30); + break; + case 'p': /* peta */ + value *= (double) (1UL<<40); + break; + case 'e': /* exa */ + value *= (double) (1UL<<50); + break; + case '%': + /* Treat percentage values greater 100% as invalid. */ + value *= (value <= 100.0) ? 0.01 : -1.0; + break; + default: + value /= 1024.0; + break; + } + } + if (value < 0.0) + OS (fatal, NILF, + _("Invalid system memory limit '%s'"), memory_used_arg); + max_memory_used = value; + } +} + #ifdef WINDOWS32 #ifndef NO_OUTPUT_SYNC @@ -3008,6 +3074,7 @@ /* If there are any options that need to be decoded do it now. */ decode_debug_flags (); decode_output_sync_flags (); + decode_memory_used_arg (); /* Perform any special switch handling. */ run_silent = silent_flag; diff -r -u make-4.3/src/makeint.h make-4.3.1/src/makeint.h --- make-4.3/src/makeint.h 2020-01-19 20:32:59.000000000 +0000 +++ make-4.3.1/src/makeint.h 2022-01-20 20:09:23.412527682 +0000 @@ -687,6 +687,7 @@ extern unsigned int job_slots; extern double max_load_average; +extern double max_memory_used; extern const char *program;