[
https://issues.apache.org/jira/browse/KAFKA-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034367#comment-16034367
]
Hamidreza Afzali commented on KAFKA-5368:
-----------------------------------------
*Problem:*
Skipped records sensor is using a {{Rate}} of type {{Count}} for
skipped-records-rate metric.
In
{{org.apache.kafka.streams.processor.internals.StreamThread#addRecordsToTasks}}
the Count value is incremented by one regardless of the number of skipped
records, i.e. the value increments even if no record is skipped.
{code}
skippedRecordsSensor.add(metrics.metricName("skipped-records-rate",
this.groupName, "The average per-second number of skipped records.",
this.tags), new Rate(new Count()));
...
private void addRecordsToTasks(final ConsumerRecords<byte[], byte[]> records) {
if (records != null && !records.isEmpty()) {
...
streamsMetrics.skippedRecordsSensor.record(records.count() -
numAddedRecords, timerStartedMs);
}
}
{code}
{{org.apache.kafka.streams.processor.internals.StreamThread#addRecordsToTasks}}
is called in
{{org.apache.kafka.streams.processor.internals.StreamThread#runLoop}} after
each successful poll request.
{code}
private void runLoop() {
...
while (stillRunning()) {
...
final ConsumerRecords<byte[], byte[]> records =
pollRequests(pollTimeMs);
if (records != null && !records.isEmpty() && !activeTasks.isEmpty()) {
streamsMetrics.pollTimeSensor.record(computeLatency(),
timerStartedMs);
addRecordsToTasks(records);
...
}
...
}
...
}
{code}
This can explain why skipped-records-rate is equal to poll-rate.
*Solution:*
The sensor should keep a sum of all skipped records.
> Kafka Streams skipped-records-rate sensor produces nonzero values when the
> timestamps are valid
> -----------------------------------------------------------------------------------------------
>
> Key: KAFKA-5368
> URL: https://issues.apache.org/jira/browse/KAFKA-5368
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Hamidreza Afzali
> Assignee: Hamidreza Afzali
>
> Kafka Streams skipped-records-rate sensor produces nonzero values even when
> the timestamps are valid and records are processed. The values are equal to
> poll-rate.
> Related issue: KAFKA-5055
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
