[
https://issues.apache.org/jira/browse/FLINK-39680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080877#comment-18080877
]
Gen Luo commented on FLINK-39680:
---------------------------------
[~roman][~pnowojski][~zhuzh]
Hi, could you please take a look at this issue? I'd appreciate your perspective
on whether the current restriction is only for data consistency, or if there
are other underlying concerns. Additionally, I'm wondering if leveraging the
existing `execution.checkpointing.unaligned.forced` configuration is a suitable
approach here.
> Enable unaligned checkpoint for pointwise connection by FORCE_UNALIGNED
> -----------------------------------------------------------------------
>
> Key: FLINK-39680
> URL: https://issues.apache.org/jira/browse/FLINK-39680
> Project: Flink
> Issue Type: Improvement
> Reporter: Gen Luo
> Priority: Major
>
> Unaligned checkpoints on pointwise edges are currently prohibited by
> FLINK-21936 and FLINK-21945 to ensure data consistency during parallelism
> changes. This restriction is necessary when a keyed stream or keyed source
> feeds into a pointwise edge, as channel state is redistributed upon recovery
> with changed parallelism. This breaks the guarantee that records of the same
> key reach the same downstream task, potentially compromising logic that
> relies on key ordering.
> However, there are valid scenarios where key ordering is irrelevant, but
> unaligned checkpoints are essential. For instance, in bounded streaming jobs
> processing datasets like images, each record may take seconds or minutes to
> process. This can block checkpoint barriers, making aligned checkpoints
> infeasible within expected timeouts.
> Since Flink cannot automatically determine whether a job requires strict key
> ordering, we should preserve the current default behavior. Instead, I propose
> allowing users to explicitly opt-in to unaligned checkpoints for such cases.
> The existing configuration
> `execution.checkpointing.unaligned.forced`—originally introduced to support
> iterative jobs—can be leveraged for this purpose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
