Gyula Fora created FLINK-35157: ---------------------------------- Summary: Sources with watermark alignment get stuck once some subtasks finish Key: FLINK-35157 URL: https://issues.apache.org/jira/browse/FLINK-35157 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.18.1, 1.19.0, 1.17.2 Reporter: Gyula Fora
The current watermark alignment logic can easily get stuck if some subtasks finish while others are still running. The reason is that once a source subtask finishes, the subtask is not excluded from alignment, effectively blocking the rest of the job to make progress beyond last wm + alignment time for the finished sources. This can be easily reproduced by the following simple pipeline: {noformat} StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(2); DataStream<Long> s = env.fromSource(new NumberSequenceSource(0, 100), WatermarkStrategy.<Long>forMonotonousTimestamps().withTimestampAssigner((SerializableTimestampAssigner<Long>) (aLong, l) -> aLong).withWatermarkAlignment("g1", Duration.ofMillis(10), Duration.ofSeconds(2)), "Sequence Source").filter((FilterFunction<Long>) aLong -> { Thread.sleep(200); return true; } ); s.print(); env.execute();{noformat} The solution could be to send out a max watermark event once the sources finish or to exclude them from the source coordinator -- This message was sent by Atlassian Jira (v8.20.10#820010)