Greetings! Thus spoke Alexander Inyukhin: > On Tue, Jan 08, 2019 at 06:08:00PM +0100, Davide Monsorno > wrote: > > I have tried to use task spooler in order to launch 2 tasks > > (a1,a2) in parallel, followed by 3 tasks (b1,b2,b3) in > > parallel. To this end, I have increased the maximum number > > of slots to 3, with the option -S [number of slots]. > > Task spooler has a limited support for dependencies. Probably, > an external logic is required to handle this case. > > [...] > > I could forward request upstream to support multiple > dependencies.
+1 for multiple dependency support. task-spooler 1.0-1 indeed sadly only supports single dependencies; it's baked into the "queue item" data structure it employs. I see two ways to implement the "external logic" you mention above, and both of them have drawbacks. The straight-forward way is to submit a task that first waits for all dependencies in turn and then runs the real task. This blocks at least one slot, and may block one of its own dependencies from actually running because there are no more free slots left, creating a livelock. The less straight-forward way is to wrap the task in a chain of task-spooler submissions, each with one dependency less than the previous link in the chain. For example, assume task D depends on C, B, and A. First, submit A as usual. Then create task B' which depends on A and whose action is to submit task C'. Task C' depends on B and its action is to submit task D'. Task D' depends on C and its action is to submit task D. Task D shall depend on no other tasks. Once A, B and C are done, then task D' will succeed in submitting task D, effectively making D depend on A, B and C. This approach does not unnecessarily block task-spooler slots since task-spooler is keeping track of when to actually run the respective tasks. But because it creates intermediate tasks it is difficult to use when A, B or C themselves have multiple dependencies and must be chained as well; the final task IDs of A, B and C are only known once the true A, B and C tasks are submitted. Having upstream support for multiple dependencies would move this logic into the task-spooler itself, where such dependency checks can be done without running a task too early or starting multiple intermediate tasks, and thus sidestepping the above-mentioned drawbacks. So yes, please do forward this request for support to upstream. Thank you, and kind regards, Marco
signature.asc
Description: PGP signature
