Hi Sam, This sounds like a condition I fixed in https://github.com/apache/kafka/commit/ed51b2cdf5bdac210a6904bead1a2ca6e8411406#diff-8b364ed2d0abd8e8ae21f5d322db6564R221 . I realized that the prior code creates a new Meter, which uses a Total metric instead of a Count. But that would total all the values of the metric, when instead what we want is to "total" the number of measurements (aka count them).
I just peeked at the 1.1 branch, and it seems this change made it in after the 1.1 branch cut, so it would only be fixed in 2.0. Thanks, -John On Wed, Jun 20, 2018 at 8:44 PM Guozhang Wang <wangg...@gmail.com> wrote: > Thanks for reporting this Sam, could you check and confirm if this issue is > fixed in trunk? If not, we should file a JIRA. > > > Guozhang > > On Wed, Jun 20, 2018 at 6:41 PM, Sam Lendle <slen...@pandora.com> wrote: > > > It looks like there is indeed a bug in kafka-streams 1.1.0. I think what > > was happening was the time spent processing each record in ns was being > > added to the total metric instead of incrementing by 1 for each record. > > Looks like the implementation has been changed in trunk. I don't see any > > commit messages mentioning this particular issue, but hopefully the > change > > fixes it. > > ------------------------------ > > *From:* Sam Lendle > > *Sent:* Wednesday, June 20, 2018 6:10:03 PM > > *To:* users@kafka.apache.org > > *Subject:* Some Total and Rate metrics are not consistent > > > > > > I’m trying to use the total metrics introduced in KIP-187 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics) > > > > > > > > For some metrics, the total and rates are not consistent. In particular, > > for stream-processor-node-metrics, I’m seeing about 500-800 operations > per > > second in a particular streams thread/processor node as reported by the > > process-rate metric, but the process-total metric is increasing by about > > 100 million per second. See attached screenshot from VisualVM. > > > > > > > > Other metrics seem fine, for example forward-rate and forward-total > > metrics under stream-processor-node-metrics are consistent. > > > > > > > > Am I misunderstanding the interpretation of the –total metrics? If this > is > > a bug, can I do anything in addition to this email to report it? File a > > JIRA? > > > > > > Best, > > Sam > > > > > > > > > > > > > > > > > > > > > -- > -- Guozhang >
