[
https://issues.apache.org/jira/browse/FLINK-41009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yubiao Feng updated FLINK-41009:
--------------------------------
Description:
{{Currently, the Pulsar sink creates a separate Pulsar transaction for each
target topic within the same Flink checkpoint.}}
{{This weakens checkpoint-level atomicity: when committing a checkpoint, some
topic transactions may be committed successfully while others fail. A later
recovery may then encounter inconsistent transaction states, including
transaction timeout or transaction-not-found errors.}}
Pulsar transactions can span multiple topics and namespaces. The sink should
therefore create and reuse one transaction for all records written during a
single checkpoint, regardless of their target topics.
At checkpoint completion, the sink should emit one committable and commit that
single transaction. A new transaction should only be created after the previous
checkpoint's transaction has been handed over for commit.
{{Benefits:}}
* Provide atomic commit semantics across all Pulsar topics written in one
checkpoint.
* Avoid partial transaction commits within a checkpoint.
* Reduce transaction creation and commit overhead.
* {{Avoid recovery failures caused by inconsistent per-topic transaction
states. }}
was:
{{Currently, the Pulsar sink creates a separate Pulsar transaction for each
target topic within the same Flink checkpoint.}}
{{This weakens checkpoint-level atomicity: when committing a checkpoint, some
topic transactions may be committed successfully while others fail. A later
recovery may then encounter inconsistent transaction states, including
transaction timeout or transaction-not-found errors.}}
{{Pulsar transactions can span multiple topics and namespaces. The sink should
therefore create and reuse one transaction for all records written during a
single checkpoint, regardless of their target topics.
At checkpoint completion, the sink should emit one committable and commit that
single transaction. A new transaction should only be created after the previous
checkpoint's transaction has been handed over for commit.}}
{{}}
{{Benefits:}}
* {{Provide atomic commit semantics across all Pulsar topics written in one
checkpoint. }}
* {{Avoid partial transaction commits within a checkpoint. }}
* {{Reduce transaction creation and commit overhead. }}
* {{Avoid recovery failures caused by inconsistent per-topic transaction
states. }}
> Use a single Pulsar transaction per checkpoint in the Pulsar sink
> -----------------------------------------------------------------
>
> Key: FLINK-41009
> URL: https://issues.apache.org/jira/browse/FLINK-41009
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
> Affects Versions: 2.0.0
> Reporter: Yubiao Feng
> Priority: Major
>
>
> {{Currently, the Pulsar sink creates a separate Pulsar transaction for each
> target topic within the same Flink checkpoint.}}
> {{This weakens checkpoint-level atomicity: when committing a checkpoint, some
> topic transactions may be committed successfully while others fail. A later
> recovery may then encounter inconsistent transaction states, including
> transaction timeout or transaction-not-found errors.}}
> Pulsar transactions can span multiple topics and namespaces. The sink should
> therefore create and reuse one transaction for all records written during a
> single checkpoint, regardless of their target topics.
> At checkpoint completion, the sink should emit one committable and commit
> that single transaction. A new transaction should only be created after the
> previous checkpoint's transaction has been handed over for commit.
> {{Benefits:}}
> * Provide atomic commit semantics across all Pulsar topics written in one
> checkpoint.
> * Avoid partial transaction commits within a checkpoint.
> * Reduce transaction creation and commit overhead.
> * {{Avoid recovery failures caused by inconsistent per-topic transaction
> states. }}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
