Hi Flink community,
We observed a potential correctness issue with two-phase event-time window
aggregation in a managed deployment reporting Flink version 2.2.1.
Configuration:
- Kafka source with 2 partitions; source and aggregation parallelism both set
to 2.
- 10-second TUMBLE windows.
- Watermark: event_time - INTERVAL '3' SECOND.
- table.exec.source.idle-timeout = '15 s'.
- table.optimizer.agg-phase-strategy = 'TWO_PHASE'.
- The execution plan contains LocalWindowAggregate and GlobalWindowAggregate.
Reproduction:
1. Send four records with the same grouping key into the same window: three
records to partition P0, each with amount 10, and one record to P1 with amount
101.
2. Stop sending to P1. Continue sending records with advancing event timestamps
to P0, using a separate grouping key.
3. Allow P1 to become idle and observe the target window result.
Expected: COUNT = 4, SUM = 131.
Actual: COUNT = 3, SUM = 30.
The missing contribution is already absent before P1 resumes. Switching to
ONE_PHASE produces the correct result. A TWO_PHASE control run also produced
the correct result when a checkpoint occurred after the P1 record arrived but
before P1 became idle.
This suggests an interaction between idleness, watermark propagation, and
two-phase aggregation, but we have not confirmed the exact root cause.
Is this a known issue, and is there an existing fix we should evaluate? I can
share the SQL, input sequence, and additional test results.
I have been unable to complete Jira account registration, so I am reporting
this here first.
Thanks,
Yuquan Chen
