[
https://issues.apache.org/jira/browse/FLINK-33756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796533#comment-17796533
]
Jim Hughes commented on FLINK-33756:
------------------------------------
Hi [~jeyhunkarimov], nice analysis! I did see that there were two pairs of
Local-Global window aggregates when I very briefly looked initially; I totally
agree that has to be part of the issue.
Out of curiosity, how did you see the value coming out of the various windows?
Was it println debugging or something else?
I like your explanation about the order of `processWatermark` and
`processElement`; that explains the apparent flakiness.
Looks like the different orderings is coming from the exchanging / hashing
which is happening between the windows. Perhaps thinking about how timestamps
and the exchange operator will help us sort this out. (Along with your note
that we are "losing" the original timestamp in some sense.)
> Missing record with CUMULATE/HOP windows using an optimization
> --------------------------------------------------------------
>
> Key: FLINK-33756
> URL: https://issues.apache.org/jira/browse/FLINK-33756
> Project: Flink
> Issue Type: Bug
> Reporter: Jim Hughes
> Priority: Major
>
> I have seen an optimization cause a window fail to emit a record.
> With the optimization `TABLE_OPTIMIZER_DISTINCT_AGG_SPLIT_ENABLED` set to
> true,
> the configuration AggregatePhaseStrategy.TWO_PHASE set, using a HOP or
> CUMULATE window with an offset, a record can be sent which causes one of the
> multiple active windows to fail to emit a record.
> The linked code
> (https://github.com/jnh5y/flink/commit/ec90aa501d86f95559f8b22b0610e9fb786f05d4)
> modifies the `WindowAggregateJsonITCase` to demonstrate the case.
>
> The test `testDistinctSplitDisabled` shows the expected behavior. The test
> `testDistinctSplitEnabled` tests the above configurations and shows that one
> record is missing from the output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
