> On 24. Apr 2018, at 20:16, Santiago Bragagnolo <santiagobragagn...@gmail.com> > wrote: > > Hi Holger! > I respond in bold
hehe. And in the reply I am back to non rich text. Let me see if I quote it correctly. > > > > On Tue, 24 Apr 2018 at 12:00 Holger Freyther <hol...@freyther.de> wrote: > Hey! > > I wondered if somebody thought of remote task execution? > > *If you mean something else, I would need more information :). > When you do [ action ] schedule / [ action ] future, both created tasks are > scheduled into the default runner. The default runner is a working pool with > a default 'poolSizeMax' on 4, meaning, limit 4 processes working over the > tasks. (this is a dynamic configuration, you can change it by > TKTConfiguration runner poolMaxSize: 20. ) Yes. But with more work than the workers can handle the queue will grow. Which means the (median/max) latency of the system will monotonically increase.. to the point of the entire system failing (tasks handled after the external deadlines expired, effectively no work being done). For network connected systems I like to think in terms of "back pressure" (not read more from the socket than the image can handle, eventually leading to the TCP window shrinking) and one way of doing it is to have bounded queues (and/or sleep when scheduling work). I can see multiple parts of a solution (and they have different benefits and issues): * Be able to attach a deadline to a task (e.g. see context.Context in go) * Be able to have a "blocking until queue is less than X elements" schedule (but that is difficult as one task might be scheduled during the >>#value of another task). > Are there ideas how to add remote task scheduling? Maybe use Seamless for it? > Since you speak about seamless here, i suppose two different images, doesn't > matter where. > It's not a bad idea to go seamless, but i did not go through the first > restriction of remote executions (if the remote image can or not execute the > task and if both images share the same semantic for the execution), then i > did not yet checked on which communication platform to use for it Right it would need to be homogeneous images (and care taken that the external interface remains similar enough). > Have workers connect to the scheduler? Other ideas? > what do you mean by connection to the scheduler? The workers we use do not > know their pools, if that is what you mean. Let's assume scheduling a task is simple (read something from a socket) but the computation is expensive (database look-up, math, etc). Hopefully one will reach the point where one image can schedule more tasks than a single worker image can handle. At that point it could be neat to scale by just starting another image. By inverting the launch order (workers connect to the scheduler) scaling can become more easy. holger
