Diego Erdody created KAFKA-13431: ------------------------------------ Summary: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users) Key: KAFKA-13431 URL: https://issues.apache.org/jira/browse/KAFKA-13431 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Diego Erdody Assignee: Diego Erdody
There's currently an incompatibility between Sink connectors overriding the {{SinkTask.preCommit}} method (for asynchronous processing) and SMTs that mutate the topic field. The problem was present since the {{preCommit}} method inception and is rooted in a mismatch between the topic/partition that is passed to {{open/preCommit}} (the original topic and partition before applying any transformations) and the topic partition that is present in the SinkRecord that the {{SinkTask.put}} method receives (after transformations are applied). Since that's all the information the connector has to implement any kind of internal offset tracking, the topic/partitions it can return in preCommit will correspond to the transformed topic, when the framework actually expects it to be the original topic. -- This message was sent by Atlassian Jira (v8.3.4#803005)