[ https://issues.apache.org/jira/browse/FLINK-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004341#comment-16004341 ]
ASF GitHub Bot commented on FLINK-6215: --------------------------------------- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3669 I think this would be crucial. It does not even work for any meaningful tests in the way it is in the code right now. +1 for putting this into 1.3 as a bugfix > Make the StatefulSequenceSource scalable. > ----------------------------------------- > > Key: FLINK-6215 > URL: https://issues.apache.org/jira/browse/FLINK-6215 > Project: Flink > Issue Type: Bug > Components: DataStream API > Affects Versions: 1.3.0 > Reporter: Kostas Kloudas > Assignee: Kostas Kloudas > Fix For: 1.3.0 > > > Currently the {{StatefulSequenceSource}} instantiates all the elements to > emit first and keeps them in memory. This is not scalable as for large > sequences of elements this can lead to out of memory exceptions. > To solve this, we can pre-partition the sequence of elements based on the > {{maxParallelism}} parameter, and just keep state (to checkpoint) per such > partition. -- This message was sent by Atlassian JIRA (v6.3.15#6346)