[
https://issues.apache.org/jira/browse/FLINK-41060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130741#comment-18130741
]
Martijn Visser commented on FLINK-41060:
----------------------------------------
The writer now only prepares the transaction and hands its xid to the
committer, which commits it after the checkpoint completed. Two fixes came with
it: SimpleXaConnectionProvider.failAndRollback() now rolls back a prepared
transaction that a restored writer finds, and PoolingXaConnectionProvider
returns a connection to the pool once its transaction is prepared. MySQL needs
xa_detach_on_prepare=ON for the commit on another connection (the default since
8.0.29), and the docs say so now. Rolling back leftover transactions on restore
is still broken; that is FLINK-41101.
> Exactly-once JdbcSink commits its XA branch at the checkpoint barrier
> ---------------------------------------------------------------------
>
> Key: FLINK-41060
> URL: https://issues.apache.org/jira/browse/FLINK-41060
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
> Affects Versions: jdbc-3.2.0, jdbc-3.3.0, jdbc-4.0.0, jdbc-4.1.0,
> jdbc-3.4.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Critical
> Fix For: jdbc-5.0.0, jdbc-4.1.1
>
>
> With the exactly-once {{JdbcSink}}, {{JdbcWriter.prepareCommit()}} prepares
> the XA branch, resets {{pendingRecords}} and returns the committable, after
> which {{JdbcWriter.snapshotState()}} sees {{pendingRecords == false}} and
> calls {{commitTxUntil(lastCheckpointId)}}, which commits the branch that was
> just prepared. The data is committed at the barrier, before the checkpoint
> completes. A checkpoint that fails after the barrier passed the sink leaves
> its records committed, and the restart replays and writes them again.
> The committer does not compensate: {{JdbcCommitableSerializer}} only writes
> the {{Xid}}, so a deserialized committable carries no transaction and
> {{JdbcCommitter.commit()}} falls back to {{commitTx()}} on its own empty
> {{XaTransaction}}. With writer and committer in one chain the committable
> still carries the writer's transaction object, which is why
> {{ExactlyOnceJdbcSinkTest}} passes; with {{pipeline.operator-chaining=false}}
> the same job leaves its only branch in-doubt and commits no row (Derby: 0 of
> 10 rows, 1 in-doubt branch).
> We should let {{snapshotState}} commit only when {{prepareCommit}} found
> nothing to prepare, and let {{JdbcCommitter.commit}} commit the handed xid
> through {{xaConnectionProvider.commit(xid, true)}}.
> {{ExactlyOnceJdbcWriterTest.testCheckpoint}} asserts the barrier-time
> visibility today and needs to change with it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
