On Mon, Aug 15, 2016 at 8:52 PM, Maciek Próchniak <m...@touk.pl> wrote: > I know it's not really desired way of using flink and that it would be > better to keep data as state inside stream and have it updated by some join > operator, but for us it's a bit of overkill - what's more, we have many (not > so large) streams, it would be not really feasible to keep all of state > (which is the same) in each of them.
Hey Maciek! The points you raise all make sense and there is work in progress to provide better support for these use cases: https://issues.apache.org/jira/browse/FLINK-4391 The general idea would be to have something like a multi threaded flat map function that dispatches the requests to a thread pool (it's like "virtually" increasing the parallelism as you do now). This is pretty straight forward to implement if you don't need to worry about fault tolerance for now. Integrating this with checkpointing is a little more involved and will be addressed as part of the linked issue.
