> On 24. Apr 2018, at 23:31, Santiago Bragagnolo <santiagobragagn...@gmail.com>
> wrote:
>
>
> Yes. But with more work than the workers can handle the queue will grow.
> Which means the (median/max) latency of the system will monotonically
> increase.. to the point of the entire system failing (tasks handled after the
> external deadlines expired, effectively no work being done).
>
>
> Normally the worker pool adjust to the minimal needed workers (there is a
> watch dog checking how much idle processes are there, or more workers are
> needed, and ensuring to spawn or stop process regarding to the state).
> So, the number poolMaxSize is just a maximal limit. This limit should be set
> for ensuring that the tasks that are running concurrently are not incurring
> into too much resource consumption or into too much overhead leading to kind
> of trashing.
> I am not really friend of setting only a number for such a complex
> problematic, but so far is the only approach I found that it does not lead to
> a complex design. If you have better ideas to discuss on this subject, i am
> completely open. (the same to deal with priorities by general system
> understanding rather than absolute numbers)
I think we might not talk about the same thing. Any system might end up being
driven close or above its limits. One question is if it can recover from it.
Let me try to give you a basic example (and if one changes from 'dev' to a
proper work pool one just needs to adjust timings to show the same problem).
The code schedules a block that on each invocation takes about one second to
execute. But the completion time is monotonically increasing.
| completions |
completions := OrderedCollection new.
1 to: 1000 do: [:each |
| start |
start := DateAndTime now.
[ (Delay forSeconds: 1) wait. completions add: (DateAndTime now -
start) ] schedule.
(Delay forMilliseconds: 200) wait.
].
completions
Now why is this a problem? It is a problem because once the system is in
overload it will never recover (unless tasks are being stopped). The question
is what can be done from a framework point to gracefully degrade? I am leaving
this here right now.
holger
