[
https://issues.apache.org/jira/browse/FLINK-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738318#comment-16738318
]
vinoyang commented on FLINK-11286:
----------------------------------
[~aljoscha]
I think it makes sense to allow watermarks to be assigned from the pipeline
(not just the source). It is not always the source that is suitable for
assigning watermarks. In many of our business, we need to go through the ETL to
parse the format or filter the data before we assign the watermark.
I hope that {{markAsTemporarilyIdle}} (or a similar mechanism) can be called
where the watermark could be assigned.
> Support to send StreamStatus.IDLE for non-source operators
> -----------------------------------------------------------
>
> Key: FLINK-11286
> URL: https://issues.apache.org/jira/browse/FLINK-11286
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: vinoyang
> Assignee: vinoyang
> Priority: Major
>
> Currently, only stream source tasks can be marked as temporary idle. But many
> times, this approach has limitations.
> Considering such a scenario, there is a DAG as follows:
> {{source->map->filter->flatmap->keyBy->window}}, with a degree of parallelism
> of 10. Among them, the watermark is not sent by the source operator, but is
> downstream, such as flatmap. Every source subtask will not be idle. However,
> after the filter, some pipelines generate "idle". For example, there are 3
> pipelines that will no longer have data sent downstream. At this time, we
> can't call the {{markAsTemporarilyIdle}} method to mark the current pipeline
> in the idle state. This will affect the downstream window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
