Gyula Fora created FLINK-2497:
---------------------------------
Summary: Rework streaming iteration flow control
Key: FLINK-2497
URL: https://issues.apache.org/jira/browse/FLINK-2497
Project: Flink
Issue Type: Improvement
Components: Streaming
Reporter: Gyula Fora
Currently streaming iterations use a simple Blocking queue to pass data from
the iteration tails to the iteration heads.
While this works most of the time, during increased load (or even short bursts)
the increased amount of records in transit inside the loop can cause the
limited number of input/output buffer-pools to fill up and deadlock the system.
(caused by the back-pressure mechanism)
One solution to this problem could be to remove back-pressure on the loop-edge
by replacing the Blocking queue with some soft spilling queue that will write
the records automatically to disk letting the iteration tail to progress with
its input.
For a more robust solution we need to implement more elaborate flow control in
streaming programs containing iterations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
