Simon Bence created NIFI-10073:
----------------------------------
Summary: Adding new load balancing strategy for better utilisation
and to avoid backpressure if possible
Key: NIFI-10073
URL: https://issues.apache.org/jira/browse/NIFI-10073
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Simon Bence
Assignee: Simon Bence
With the current load balancing for connections, NiFi might end up in a
situation where the utilisation of the nodes differ highly. In extreme cases
backpressure might be applied even if there are nodes might accept data.
In order to resolve this, a new load balancing strategy needs to be introduced
with the following characteristics:
- Work stealing should be possible, thus the strategy should try to keep all
nodes busy.
- If any node can accept data we should not apply backpressure
To reach this, when the newly introduced strategi is being applied, a shared
partition will be used by the remote partitons. When the flow files are sent to
a remote node, the decision must be made, how many of them should be sent.
Helping this decision the load balancing protocol needs to be modified in the
way, instead of sending back SPACE_AVAILABLE and QUEUE_FULL, the nodes should
provide a more detailed information.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
