[
https://issues.apache.org/jira/browse/NIFI-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-16380:
------------------------------------
Description:
Following the pattern of {{{}ConsumeKafka{}}}, the {{ConsumeKinesis}} Processor
should track lag and record metrics for each invocation. The Map of attributes
for Counters and Gauges should include the Kinesis Stream and Shard information.
The metrics should include the following:
* kinesis.bytes.consumed
** Type: Counter
* kinesis.records.consumed
** Type: Counter
* kinesis.consumer.milliseconds.behind
** Type: Gauge
* kinesis.records.parsed.errors
** Type: Counter
Kinesis supports the concept of
[MillisBehindLatest|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html#API_GetRecords_ResponseSyntax]
to indicate the number milliseconds behind the consumer is from the latest
record in the stream. This is a loose indicator of lag because it does not
provide the number of records and interpreting the number requires an
understanding of expected periodicity for records added to the Kinesis Stream.
was:
Following the pattern of {{ConsumeKafka}}, the {{ConsumeKinesis}} Processor
should track lag and record metrics for each invocation. The Map of attributes
for Counters and Gauges should include the Kinesis Stream and Shard information.
The metrics should include the following:
* kinesis.bytes.consumed
** Type: Counter
* kinesis.records.consumed
** Type: Counter
* kinesis.consumer.behind.ms
** Type: Gauge
* kinesis.records.parsed.errors
** Type: Counter
Kinesis supports the concept of
[MillisBehindLatest|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html#API_GetRecords_ResponseSyntax]
to indicate the number milliseconds behind the consumer is from the latest
record in the stream. This is a loose indicator of lag because it does not
provide the number of records and interpreting the number requires an
understanding of expected periodicity for records added to the Kinesis Stream.
> Add Records Metrics to ConsumeKinesis
> -------------------------------------
>
> Key: NIFI-16380
> URL: https://issues.apache.org/jira/browse/NIFI-16380
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: David Handermann
> Assignee: David Handermann
> Priority: Major
>
> Following the pattern of {{{}ConsumeKafka{}}}, the {{ConsumeKinesis}}
> Processor should track lag and record metrics for each invocation. The Map of
> attributes for Counters and Gauges should include the Kinesis Stream and
> Shard information.
> The metrics should include the following:
> * kinesis.bytes.consumed
> ** Type: Counter
> * kinesis.records.consumed
> ** Type: Counter
> * kinesis.consumer.milliseconds.behind
> ** Type: Gauge
> * kinesis.records.parsed.errors
> ** Type: Counter
> Kinesis supports the concept of
> [MillisBehindLatest|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html#API_GetRecords_ResponseSyntax]
> to indicate the number milliseconds behind the consumer is from the latest
> record in the stream. This is a loose indicator of lag because it does not
> provide the number of records and interpreting the number requires an
> understanding of expected periodicity for records added to the Kinesis Stream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
