[
https://issues.apache.org/jira/browse/FLINK-39680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gen Luo updated FLINK-39680:
----------------------------
Labels: (was: pull-request-available)
> Enable unaligned checkpoint for pointwise connection by FORCE_UNALIGNED
> -----------------------------------------------------------------------
>
> Key: FLINK-39680
> URL: https://issues.apache.org/jira/browse/FLINK-39680
> Project: Flink
> Issue Type: Improvement
> Reporter: Gen Luo
> Priority: Major
>
> Unaligned checkpoints on pointwise edges are currently prohibited by
> FLINK-21936 and FLINK-21945 to ensure data consistency during parallelism
> changes. This restriction is necessary when a keyed stream or keyed source
> feeds into a pointwise edge, as channel state is redistributed upon recovery
> with changed parallelism. This breaks the guarantee that records of the same
> key reach the same downstream task, potentially compromising logic that
> relies on key ordering.
> However, there are valid scenarios where key ordering is irrelevant, but
> unaligned checkpoints are essential. For instance, in bounded streaming jobs
> processing datasets like images, each record may take seconds or minutes to
> process. This can block checkpoint barriers, making aligned checkpoints
> infeasible within expected timeouts.
> Since Flink cannot automatically determine whether a job requires strict key
> ordering, we should preserve the current default behavior. Instead, I propose
> allowing users to explicitly opt-in to unaligned checkpoints for such cases.
> The existing configuration
> `execution.checkpointing.unaligned.forced`—originally introduced to support
> iterative jobs—can be leveraged for this purpose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
