[
https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-15018.
-----------------------------------
Fix Version/s: 3.8.0
Resolution: Fixed
> Potential tombstone offsets corruption for exactly-once source connectors
> -------------------------------------------------------------------------
>
> Key: KAFKA-15018
> URL: https://issues.apache.org/jira/browse/KAFKA-15018
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Affects Versions: 3.3.0, 3.4.0, 3.3.1, 3.3.2, 3.5.0, 3.4.1
> Reporter: Chris Egerton
> Assignee: Sagar Rao
> Priority: Major
> Fix For: 3.8.0
>
>
> When exactly-once support is enabled for source connectors, source offsets
> can potentially be written to two different offsets topics: a topic specific
> to the connector, and the global offsets topic (which was used for all
> connectors prior to KIP-618 / version 3.3.0).
> Precedence is given to offsets in the per-connector offsets topic, but if
> none are found for a given partition, then the global offsets topic is used
> as a fallback.
> When committing offsets, a transaction is used to ensure that source records
> and source offsets are written to the Kafka cluster targeted by the source
> connector. This transaction only includes the connector-specific offsets
> topic. Writes to the global offsets topic take place after writes to the
> connector-specific offsets topic have completed successfully, and if they
> fail, a warning message is logged, but no other action is taken.
> Normally, this ensures that, for offsets committed by exactly-once-supported
> source connectors, the per-connector offsets topic is at least as up-to-date
> as the global offsets topic, and sometimes even ahead.
> However, for tombstone offsets, we lose that guarantee. If a tombstone offset
> is successfully written to the per-connector offsets topic, but cannot be
> written to the global offsets topic, then the global offsets topic will still
> contain that source offset, but the per-connector topic will not. Due to the
> fallback-on-global logic used by the worker, if a task requests offsets for
> one of the tombstoned partitions, the worker will provide it with the offsets
> present in the global offsets topic, instead of indicating to the task that
> no offsets can be found.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
