[
https://issues.apache.org/jira/browse/KAFKA-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030170#comment-17030170
]
Vinoth Chandar commented on KAFKA-9430:
---------------------------------------
The right way to compute the endOffsets would be following (I think the
positions reported are okay. they are the consume position on the source topic
anyway)
For active tasks, if the optimization is enabled and the source topic is being
reused as changelog
* (running state) endOffset = consumer position on the source topic partition
(indicating its fully caught up)
* (restoring state) endOffset = last committed consumer offset for the source
topic partition (Need to check restoration code for this path more)
For standby tasks, if the optimization is enabled and the source topic is being
reused as changelog
* (running state) endOffset = last commit consumer offset/offset limit tracked
by the standby
* (restoring state) same as above.
[~guozhang] you mentioned with some changes that landed recently, that handle
this? From the KafkaStreams class, I still see the values from the
`adminClient.listOffsets(...)` being used for endOffset.. Am I missing
something?
> Tighten up lag estimates when source topic optimization is on
> --------------------------------------------------------------
>
> Key: KAFKA-9430
> URL: https://issues.apache.org/jira/browse/KAFKA-9430
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.5.0
> Reporter: Vinoth Chandar
> Assignee: Vinoth Chandar
> Priority: Blocker
>
> Right now, we use _endOffsets_ of the source topic for the computation. For
> "optimized" changelogs, this will be wrong, strictly speaking, but it's an
> over-estimate (which seems better than an under-estimate), and it's also
> still an apples-to-apples comparison, since all replicas would use the same
> upper bound to compute their lags, so the "pick the freshest" replica is
> still going to pick the right one.
> The current implementation is technically correct, within the documented
> behavior that the result is an "estimate", but I marked it as a blocker to be
> sure that we revisit it after ongoing work to refactor the task management in
> Streams is complete. If it becomes straightforward to tighten up the
> estimate, we should go ahead and do it. Otherwise, we can downgrade the
> priority of the ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
