I'd like to suggest and implement a new feature in make to control parallelism. I'm basing my suggestion on an old post from Paul Smith:
http://lists.gnu.org/archive/html/bug-make/2012-07/msg00007.html I've implemented this in my own projects, though without the "uniqueness" of different lock-file names for multiple 'make'-s running in different sessions as I don't need that uniqueness feature at the moment. In the course of implementing the shell-based flock in a makefile, here's my rule, slightly sanitized for clarity: # Serialize the install targets: %/install: flock $(SOMEDIR)/install_lock $(MAKE) -C $(@D) $(notdir $(@)) This works as expected, and I like to credit Paul for his suggestion. Now, after observing this behavior at build time, it occurred to me that at least the theoretical implementation of an $(flock ...) function _inside_ make itself would provide a major additional benefit that it could release the job-slot while the $(MAKE) currently under execution was waiting for the lock, thus not holding an active job-slot hostage while it waited for the lock, unlike the shell-based flock solution. In watching my current (shell-based) implementation running with a -j8 or even a -j16, I have seen the make reduced to an effective -j1 when my jobs all hit the same flock. This is not ideal, and completely defeats the purpose of parallel builds for that particular target, since it consumes all the job slots but only one can be executing. This is not a "fault" of make, but make could offer a solution with an internal $(flock ...). So the advantage of an internal $(flock ...) implementation are several: 1) release job slot while waiting on lock, thus keeping the number of slots available for runnable jobs at the user-specified "-j X" level 2) one $(MAKE) (in a session) does not share locks with other $(MAKE)'s in a different session, thus solving an issue that Paul himself pointed out right after his post that I reference above. >From an implementation standpoint, the job-slot count manipulation is not a new concept, as sub-makes already exempt themselves and that mechanism should be usable (I'm just guessing here, but it seems logical) for a similar exemption, based upon state as follows: release job-slot while waiting for lock, once lock has been obtained transition to "runnable" job-state as would normally happen (so await available job-slot and continue execution as normal while holding lock), do nothing to job-slot when lock is released. So there are several design choices that occur to me, revolving around formatting of the command itself: $(flock lock-name) rest-of-target-rule This would make my code snippet above look like %/install: $(flock install_lock) $(MAKE) -C $(@D) $(notdir $(@)) another option is: $(flock lock-name command-to-run-while-holding-lock) yielding: %/install: $(flock install_lock $(MAKE) -C $(@D) $(notdir $(@))) and there may be other options. I'm open to suggestions, though I am leaning towards the first [$(flock lock-name) rest-of target-rule)] because it might be easier to parse, and we could treat the presence of an $(flock ...) function anywhere within a particular instance of a recipe to cause that single recipe to honor the lock request and then carry-on as normal with that whole recipe, sort of similar to how a $(MAKE) appearing anywhere within the recipe gets special treatment. And I do mean a single recipe, not all the recipes in a given rule. So that's my design plan, feel free to shoot at it. That which doesn't kill it will only make it stronger. And one more thing...just to keep to things sane and avoid the inevitable dead-lock scenarios, an $(flock lock-name) would only belong to a particular instance of $(MAKE), and could not be inherited by sub-make's nor shared with other sibling make' at the same level. It would be private and local to whatever makefile it's declared in, and travels no further. This might be a bit limiting, but the cost of implementing it should be bearable this way. I'm trying to keep this small, sane, and yet usable in the most common situations. Its not intended to provide a transactional database locking mechanism... Any thoughts, suggestions, or pointers (especially to where in the make source tree I should begin with...) are appreciated. Thanks, Zoltan -- ### Any similarity between my views and the truth is completely ### ### coincidental, except that they are endorsed by NO ONE ### _______________________________________________ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
