Chris Egerton created KAFKA-14610:
-------------------------------------
Summary: Publish Mirror Maker 2 offset syncs in task commit method
Key: KAFKA-14610
URL: https://issues.apache.org/jira/browse/KAFKA-14610
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Reporter: Chris Egerton
Mirror Maker 2 periodically publishes offset sync messages to a Kafka topic
that contains the corresponding upstream and downstream offsets for a
replicated topic partition.
Currently, this publishing takes place inside the [commitRecord
method|https://github.com/apache/kafka/blob/e38526e375389868664c8977c7a2125e5da2388c/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L192],
which is invoked by the Kafka Connect framework after a source record has been
successfully sent by its producer (i.e., ack'd by the requested number of
brokers).
Mirror Maker 2 also has logic to limit the number of in-flight offset sync
messages. Once ten messages have been dispatched to the producer used for
offset syncs (which is a separate producer from the one that the Kafka Connect
framework uses for sending records received from the [poll
method|https://github.com/apache/kafka/blob/e38526e375389868664c8977c7a2125e5da2388c/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L134])
that have not yet been ack'd by the requested number of brokers, Mirror Maker
2 begins to skip sending offset sync messages, and will only resume sending
messages once the number of in-flight offset syncs goes below 10, and new calls
to the {{commitRecord}} method take place.
When bursts of throughput occur in replicated topic partitions, this can cause
offset syncs to be dropped for long periods of time if an offset sync is
skipped for some topic partition due to a high number of in-flight messages and
then no further messages are read from that same topic partition for a while.
Instead, the task should cache offset syncs in its {{{}commitRecord method{}}},
and only actually send offset sync messages in its [commit
method|https://github.com/apache/kafka/blob/e38526e375389868664c8977c7a2125e5da2388c/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L108],
which is invoked periodically by the Kafka Connect framework. Any offset syncs
that are skipped due to too many in-flight messages will then be automatically
retried later when {{commit}} is re-invoked, regardless of whether any more
records are read from the corresponding topic partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
