[
https://issues.apache.org/jira/browse/FLINK-41009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Csaky reassigned FLINK-41009:
------------------------------------
Assignee: Yubiao Feng
> Use a single Pulsar transaction per checkpoint in the Pulsar sink
> -----------------------------------------------------------------
>
> Key: FLINK-41009
> URL: https://issues.apache.org/jira/browse/FLINK-41009
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
> Affects Versions: 2.0.0
> Reporter: Yubiao Feng
> Assignee: Yubiao Feng
> Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-5.0.0
>
>
>
> {{Currently, the Pulsar sink creates a separate Pulsar transaction for each
> target topic within the same Flink checkpoint.}}
> {{This weakens checkpoint-level atomicity: when committing a checkpoint, some
> topic transactions may be committed successfully while others fail. A later
> recovery may then encounter inconsistent transaction states, including
> transaction timeout or transaction-not-found errors.}}
> Pulsar transactions can span multiple topics and namespaces. The sink should
> therefore create and reuse one transaction for all records written during a
> single checkpoint, regardless of their target topics.
> At checkpoint completion, the sink should emit one committable and commit
> that single transaction. A new transaction should only be created after the
> previous checkpoint's transaction has been handed over for commit.
> {{Benefits:}}
> * Provide atomic commit semantics across all Pulsar topics written in one
> checkpoint.
> * Avoid partial transaction commits within a checkpoint.
> * Reduce transaction creation and commit overhead.
> * {{Avoid recovery failures caused by inconsistent per-topic transaction
> states. }}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
