[
https://issues.apache.org/jira/browse/FLINK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lee Y S closed FLINK-29048.
---------------------------
Resolution: Duplicate
> WatermarksWithIdleness does not work with FLIP-27 sources
> ---------------------------------------------------------
>
> Key: FLINK-29048
> URL: https://issues.apache.org/jira/browse/FLINK-29048
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
> Affects Versions: 1.15.1
> Reporter: Lee Y S
> Priority: Major
>
> In org.apache.flink.streaming.api.operators.SourceOperator, there are
> separate instances of WatermarksWithIdleness created for each split output
> and the main output. There is multiplexing of watermarks between split
> outputs but no multiplexing between split output and main output in
> org.apache.flink.streaming.api.operators.source.ProgressiveTimestampsAndWatermarks.
>
> For a source such as org.apache.flink.connector.kafka.source.KafkaSource,
> {color:#353833}there is only output from splits and no output from main.
> Hence the main output will (after an initial timeout) be marked as
> idle.{color}
> {color:#353833} {color}
> {color:#353833}The implementation of {color}WatermarksWithIdleness is such
> that once an output is idle, it will periodically re-mark the output as idle.
> Since there is no multiplexing between split outputs and main output, the
> idle marks coming from main output will repeatedly set the output to idle
> even though there are events from the splits. Result is that the entire
> source is repeatedly marked as idle.
>
> One solution i can think of is to multiplex split and main output in
> org.apache.flink.streaming.api.operators.source.ProgressiveTimestampsAndWatermarks
> but I am not sure if there are other considerations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
