[
https://issues.apache.org/jira/browse/FLINK-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268923#comment-14268923
]
Gyula Fora commented on FLINK-1368:
-----------------------------------
I totally agree that in almost all cases when a streaming program spills to
disk it is useless.
But for windowing in case of some policies like time, a spike in the incoming
data rate should be handled without memory errors.
So for instance we are working on the last 10 minutes of data it might happen
that sometimes that window doesn't fit in memory and the buffer holding the
records should spill to disk if we assume that it will be able to catch up
afterwards. This is probably a feature that people expect for applications
where the data rate can vary much.
So probably disk spilling should only be considered for the windows.
> Change memory management settings for Streaming programs
> --------------------------------------------------------
>
> Key: FLINK-1368
> URL: https://issues.apache.org/jira/browse/FLINK-1368
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 0.9
> Reporter: Gyula Fora
>
> Currently streaming programs use the same memory management settings as a
> batch api by reserving a large share of the heap for the runtime, while not
> making any use of it.
> This should be disabled until the streaming runtime is reworked so it makes a
> good use of this reserved space. One part where this could be most relevant
> is windowing operators where a large time window could easily cause memory
> problems with the current implementation. For windowing disk spilling should
> be implemented to increase robustness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
