Anton Kalashnikov created FLINK-23453:
-----------------------------------------
Summary: Dynamic calculation of the buffer size
Key: FLINK-23453
URL: https://issues.apache.org/jira/browse/FLINK-23453
Project: Flink
Issue Type: Sub-task
Reporter: Anton Kalashnikov
To calculate the desired buffer size we need to take into account the
throughput, configuration(timeInInputBuffer), and the actual number of buffers
in use. It makes sense to use EMA for this calculation to smoothen out
intermittent spikes.
The calculation based on the actual number of buffers in use helps to avoid
problems with the data skew (when only a couple of channels out of thousands
have any data). So the solution needs to reliably and efficiently calculate
either the estimated or an average number of buffers in use.
Buffer size can be erratic if it’s not trivial to make it stable in the MVP.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
