Re: Cooperation with distributed job processing systems
How do you think about to delegate such an implementation detail to a configurable job submission system? That could be done, but the current implementation of the jobserver software isn't really separable like that. It would be a nice thing to make a more abstract API which could be replaced with alternatives. Do you find information about the software Makeflow interesting for improved management of build work flows? http://www.cse.nd.edu/~ccl/software/makeflow/ Regards, Markus ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Cooperation with distributed job processing systems
On Sat, 2015-01-03 at 23:30 +0100, SF Markus Elfring wrote: There are programming interfaces available which provide support for submitting jobs to bigger and more powerful computer systems. I imagine that the software jobserver could be extended for the convenient reuse of such APIs. http://make.mad-scientist.net/papers/jobserver-implementation/ I find it highly unlikely that jobserver will be helpful directly. Jobserver is a method of communicating how many jobs make thinks are running between different instances (parent/child) of the make program itself, so it knows that no more than N jobs are invoked between all instances. What you seem to be suggesting is something different: a facility for make to start jobs on remote systems. There are various methods for this: the most common is to use distcc in conjunction with make: https://code.google.com/p/distcc/ although this is really restricted to building C/C++ programs so not generally applicable. There is a nascent facility in the GNU make code that provides for make invoking jobs through a separate facility: the remote capability. Currently there is a remote-stub.c which defines the interface, and a remote-cstms.c which attempts to use the Customs facility for starting remote jobs. However, this is obsolete and I'm not sure if anyone has tried to use it for years. Nevertheless if one were to try to add native facilities for starting jobs on remote systems into GNU make itself, rather than doing it through use of a special SHELL variable setting, etc. then remote would likely be the best way to do it, not jobserver. I urge you to explore these other methods first, though, as they're a lot less effort all the way around. If you can write a small program that will forward a command to a remote system, then you can just replace SHELL in your makefile with that command: SHELL = /bin/submit-job all: run-a-job now make will invoke: /bin/submit-job -c run-a-job (you can control the flags with the .SHELLFLAGS variable). ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Cooperation with distributed job processing systems
Jobserver is a method of communicating how many jobs make thinks are running between different instances (parent/child) of the make program itself, so it knows that no more than N jobs are invoked between all instances. How do you think about to delegate such an implementation detail to a configurable job submission system? What you seem to be suggesting is something different: a facility for make to start jobs on remote systems. Yes. - My suggestion goes into this direction. There are programming interfaces available which support transparent data exchange between local and remote information systems. There are various methods for this: the most common is to use distcc in conjunction with make: https://code.google.com/p/distcc/ although this is really restricted to building C/C++ programs It is nice when remote compilation can be used with dedicated tools already. so not generally applicable. I would appreciate when this limitation can be dropped eventually. There is a nascent facility in the GNU make code that provides for make invoking jobs through a separate facility: the remote capability. I am curious to know a bit more about such a possibility. Currently there is a remote-stub.c which defines the interface, and a remote-cstms.c which attempts to use the Customs facility for starting remote jobs. However, this is obsolete and I'm not sure if anyone has tried to use it for years. Interesting background information ... Nevertheless if one were to try to add native facilities for starting jobs on remote systems into GNU make itself, rather than doing it through use of a special SHELL variable setting, etc. then remote would likely be the best way to do it, not jobserver. Does the software use a clean internal API already? I urge you to explore these other methods first, though, as they're a lot less effort all the way around. If you can write a small program that will forward a command to a remote system, then you can just replace SHELL in your makefile with that command: SHELL = /bin/submit-job all: run-a-job I find that this approach would apply one step too late because the make program manages corresponding process parallelisation by the parameter -j. Is an alternative job submission needed earlier? Regards, Markus ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Cooperation with distributed job processing systems
On Mon, 2015-01-05 at 18:24 +0100, SF Markus Elfring wrote: Jobserver is a method of communicating how many jobs make thinks are running between different instances (parent/child) of the make program itself, so it knows that no more than N jobs are invoked between all instances. How do you think about to delegate such an implementation detail to a configurable job submission system? That could be done, but the current implementation of the jobserver software isn't really separable like that. It would be a nice thing to make a more abstract API which could be replaced with alternatives. Nevertheless if one were to try to add native facilities for starting jobs on remote systems into GNU make itself, rather than doing it through use of a special SHELL variable setting, etc. then remote would likely be the best way to do it, not jobserver. Does the software use a clean internal API already? As I said, the API is defined in remote-stub.c. It's relatively clean. I don't think anyone has used it for a very long time so it's quite possible it's bit-rotted. I urge you to explore these other methods first, though, as they're a lot less effort all the way around. If you can write a small program that will forward a command to a remote system, then you can just replace SHELL in your makefile with that command: SHELL = /bin/submit-job all: run-a-job I find that this approach would apply one step too late because the make program manages corresponding process parallelisation by the parameter -j. Correct. There are two completely separate things: one thing is make deciding how many jobs can be run in parallel and when more jobs can be started. That's controlled by the jobserver and currently there's no way to replace the built-in jobserver implementation with something different (such as allowing a remote facility to track this). The second thing is invoking a single job once make believes that it's OK to do so. The remote API can make it possible to build make so that it always uses this method, but I think it's become obsolete because it's simpler and more flexible to implement a separate program and reset SHELL, as I describe above, than implement a bunch of new code inside GNU make itself. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Cooperation with distributed job processing systems
There are two completely separate things: one thing is make deciding how many jobs can be run in parallel and when more jobs can be started. Thanks for your explanation. That's controlled by the jobserver and currently there's no way to replace the built-in jobserver implementation with something different (such as allowing a remote facility to track this). I hope that this situation can be improved somehow. The second thing is invoking a single job once make believes that it's OK to do so. That is reasonable. The remote API can make it possible to build make so that it always uses this method, This might be also interesting for specific software applications. but I think it's become obsolete The software status might evolve again a bit more. because it's simpler and more flexible to implement a separate program and reset SHELL, How many computer resources can be eventually saved with an alternative approach? as I describe above, than implement a bunch of new code inside GNU make itself. How are the chances to improve internal configuration interfaces? Regards, Markus ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
