[
https://issues.apache.org/jira/browse/FLINK-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-18405:
-----------------------------------
Fix Version/s: (was: 1.17.0)
> Add watermark support for unaligned checkpoints
> -----------------------------------------------
>
> Key: FLINK-18405
> URL: https://issues.apache.org/jira/browse/FLINK-18405
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.12.0
> Reporter: Arvid Heise
> Priority: Not a Priority
> Labels: auto-deprioritized-major
>
> Currently, Flink generates the watermark as a first step of recovery instead
> of
> storing the latest watermark in the operators to ease rescaling. In unaligned
> checkpoints, that means on recovery, Flink generates watermarks after it
> restores in-flight data. If your pipeline uses an operator that applies the
> latest watermark on each record, it will produce incorrect results during
> recovery if the watermark is not directly or indirectly part of the operator
> state. Thus, SQL OVER operator should not be used with unaligned
> checkpoints, while window operators are safe to use.
> A possible solution is to store the watermark in the operator state. If
> rescaling may occur, watermarks should be stored per key-group in a
> union-state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
