Chris Egerton created KAFKA-15018:
-------------------------------------
Summary: Potential tombstone offsets corruption for exactly-once
source connectors
Key: KAFKA-15018
URL: https://issues.apache.org/jira/browse/KAFKA-15018
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.3.2, 3.3.1, 3.4.0, 3.3.0, 3.5.0, 3.4.1
Reporter: Chris Egerton
When exactly-once support is enabled for source connectors, source offsets can
potentially be written to two different offsets topics: a topic specific to the
connector, and the global offsets topic (which was used for all connectors
prior to KIP-618 / version 3.3.0).
Precedence is given to offsets in the per-connector offsets topic, but if none
are found for a given partition, then the global offsets topic is used as a
fallback.
When committing offsets, a transaction is used to ensure that source records
and source offsets are written to the Kafka cluster targeted by the source
connector. This transaction only includes the connector-specific offsets topic.
Writes to the global offsets topic take place after writes to the
connector-specific offsets topic have completed successfully, and if they fail,
a warning message is logged, but no other action is taken.
Normally, this ensures that, for offsets committed by exactly-once-supported
source connectors, the per-connector offsets topic is at least as up-to-date as
the global offsets topic, and sometimes even ahead.
However, for tombstone offsets, we lose that guarantee. If a tombstone offset
is successfully written to the per-connector offsets topic, but cannot be
written to the global offsets topic, then the global offsets topic will still
contain that source offset, but the per-connector topic will not. Due to the
fallback-on-global logic used by the worker, if a task requests offsets for one
of the tombstoned partitions, the worker will provide it with the offsets
present in the global offsets topic, instead of indicating to the task that no
offsets can be found.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
