[
https://issues.apache.org/jira/browse/KAFKA-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Harris updated KAFKA-15906:
--------------------------------
Description:
Right now, the offset.lag.max configuration limits the number of offset syncs
are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore.
After 100 records have been emitted for a partition, _and_ the semaphore is
available, an offset sync can be emitted.
For low-volume topics, the `offset.lag.max` default of 100 is much more
restrictive than the rate-limiting semaphore. For example, a topic which
mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset
sync. If the topic is actually finite, the last offset sync will never arrive,
and the translation will have a persistent lag.
Instead, we can periodically flush the offset syncs for partitions that are
under the offset.lag.max limit, but have not received an offset sync recently.
This could be a new configuration, be a hard-coded time, or be based on the
existing emit.checkpoints.interval.seconds and
sync.group.offsets.interval.seconds configurations.
Alternatively, we could decrease the default `offset.lag.max` value to 0, and
rely on the fair semaphore to limit the number of syncs emitted for
high-throughput partitions. The semaphore is not currently configurable, so
users wanting lower throughput on the offset-syncs topic will still need an
offset.lag.max > 0.
was:
Right now, the offset.lag.max configuration limits the number of offset syncs
are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore.
After 100 records have been emitted for a partition, _and_ the semaphore is
available, an offset sync can be emitted.
For low-volume topics, the `offset.lag.max` default of 100 is much more
restrictive than the rate-limiting semaphore. For example, a topic which
mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset
sync. If the topic is actually finite, the last offset sync will never arrive,
and the translation will have a persistent lag.
Instead, we can periodically flush the offset syncs for partitions that are
under the offset.lag.max limit, but have not received an offset sync recently.
This could be a new configuration, be a hard-coded time, or be based on the
existing emit.checkpoints.interval.seconds and
sync.group.offsets.interval.seconds configurations.
> Emit offset syncs more often than offset.lag.max for low-throughput/finite
> partitions
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-15906
> URL: https://issues.apache.org/jira/browse/KAFKA-15906
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Greg Harris
> Priority: Minor
>
> Right now, the offset.lag.max configuration limits the number of offset syncs
> are emitted by the MirrorSourceTask, along with a fair rate-limiting
> semaphore. After 100 records have been emitted for a partition, _and_ the
> semaphore is available, an offset sync can be emitted.
> For low-volume topics, the `offset.lag.max` default of 100 is much more
> restrictive than the rate-limiting semaphore. For example, a topic which
> mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset
> sync. If the topic is actually finite, the last offset sync will never
> arrive, and the translation will have a persistent lag.
> Instead, we can periodically flush the offset syncs for partitions that are
> under the offset.lag.max limit, but have not received an offset sync
> recently. This could be a new configuration, be a hard-coded time, or be
> based on the existing emit.checkpoints.interval.seconds and
> sync.group.offsets.interval.seconds configurations.
>
> Alternatively, we could decrease the default `offset.lag.max` value to 0, and
> rely on the fair semaphore to limit the number of syncs emitted for
> high-throughput partitions. The semaphore is not currently configurable, so
> users wanting lower throughput on the offset-syncs topic will still need an
> offset.lag.max > 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
