[
https://issues.apache.org/jira/browse/SAMZA-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366858#comment-14366858
]
Yan Fang commented on SAMZA-540:
--------------------------------
Hi [~renato2099], thanks for the patch. Looked at the patch, a few things:
1. when we say "maxOffset", we were meaning the "max/latest" offset in the
incoming stream. For example, the incoming stream has latest offset 1000 and we
read until 510 and checkpoint to offset 500. Here, the maxOffset is 1000, and
the checkpointed offset is 500. It's my bad that I did not explain very well.
In your patch, apparently, you are storing the 500 as the maxOffset and there
is no updating. After my second thought, there's no easy way for OffsetManager
to figure the maxOffset out from current implementation. So let's leave it
there. Only focus on "expose the checkpointed offset".
2. In terms of "expose the checkpointed offset", the idea is to put the
checkpointed offset into the metrics. You may want to add
[Gauge|http://samza.apache.org/learn/documentation/0.8/api/javadocs/org/apache/samza/metrics/Gauge.html]
for the OffsetManager. Can refer to
[TaskInstanceMetrics|https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/container/TaskInstanceMetrics.scala]
and
[KafkaSystemConsumerMetrics|https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumerMetrics.scala].
You can register metrics in the register method and update them in checkpoint
method.
3. no "print" in code (line 339 in your patch) :)
> Expose latency related metrics in OffsetManager
> -----------------------------------------------
>
> Key: SAMZA-540
> URL: https://issues.apache.org/jira/browse/SAMZA-540
> Project: Samza
> Issue Type: Improvement
> Components: metrics
> Affects Versions: 0.8.0
> Reporter: Yan Fang
> Labels: newbie
> Fix For: 0.10.0
>
> Attachments: SAMZA-540.1.patch
>
>
> Follow-up to SAMZA-503
> Expose checkpointed offset, max offset, and max offset - checkpointed offset
> metrics in OffsetManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
