[
https://issues.apache.org/jira/browse/FLINK-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643969#comment-15643969
]
ASF GitHub Bot commented on FLINK-4391:
---------------------------------------
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
The `notifyCheckpointComplete` will mitigate the problem of adding latency
a little bit but not get rid of it because the tasks are only notified about a
completed checkpoint if the whole checkpoint (all operators have finished their
checkpoints) has succeeded. This can take a noticeable amount of time if you
have a complex topology.
I really would not like to build in a hand tailored solution for the
`AsyncWaitOperator` into the `StreamTask`. This is imo not a good design,
because the `StreamTask` should not know anything about the operators which are
executed within it.
I still think that the checkpoint lock would be a better solution because
then the `Emitter` thread can continue emitting elements right after the
checkpoint (local) has completed and we don't have to introduce custom control
logic for the `AsyncWaitOperator`.
> Provide support for asynchronous operations over streams
> --------------------------------------------------------
>
> Key: FLINK-4391
> URL: https://issues.apache.org/jira/browse/FLINK-4391
> Project: Flink
> Issue Type: New Feature
> Components: DataStream API
> Reporter: Jamie Grier
> Assignee: david.wang
>
> Many Flink users need to do asynchronous processing driven by data from a
> DataStream. The classic example would be joining against an external
> database in order to enrich a stream with extra information.
> It would be nice to add general support for this type of operation in the
> Flink API. Ideally this could simply take the form of a new operator that
> manages async operations, keeps so many of them in flight, and then emits
> results to downstream operators as the async operations complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
