Greg Harris created KAFKA-15906:
-----------------------------------

             Summary: Emit offset syncs more often than offset.lag.max for 
low-throughput/finite partitions
                 Key: KAFKA-15906
                 URL: https://issues.apache.org/jira/browse/KAFKA-15906
             Project: Kafka
          Issue Type: Improvement
          Components: mirrormaker
            Reporter: Greg Harris


Right now, the offset.lag.max configuration limits the number of offset syncs 
are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore. 
After 100 records have been emitted for a partition, _and_ the semaphore is 
available, an offset sync can be emitted.

For low-volume topics, the `offset.lag.max` default of 100 is much more 
restrictive than the rate-limiting semaphore. For example, a topic which 
mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset 
sync. If the topic is actually finite, the last offset sync will never arrive, 
and the translation will have a persistent lag.

Instead, we can periodically flush the offset syncs for partitions that are 
under the offset.lag.max limit, but have not received an offset sync recently. 
This could be a new configuration, be a hard-coded time, or be based on the 
existing emit.checkpoints.interval.seconds and 
sync.group.offsets.interval.seconds configurations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to