[jira] [Created] (KAFKA-16129) Add integration test for KIP-977
Qichao Chu created KAFKA-16129:
--
Summary: Add integration test for KIP-977
Key: KAFKA-16129
URL: https://issues.apache.org/jira/browse/KAFKA-16129
Project: Kafka
Issue Type: Task
Reporter: Qichao Chu
*{{metrics.verbosity}}* will be a new dynamic config introduced to control the
verbosity(fan-out rate) of the metrics. It's a config with JSON format
specifying the condition controlling fan-out of the metrics. If the value
*{{high}}* is set for the *{{level}}* key of the configured JSON(see below for
example values), high fan-out tags(e.g. {*}{{partition}}{*})will be added to
metrics specified by the {{*name*}} filter and will apply to all the topics
that meet the conditions in the {{*filters*}} section. In the *{{low}}*
settings, these tags will be assigned with an empty value. We elected to make
it central so that this implementation can be generalized in the future either
into a library, or allow other means for centralized control.
More details:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
The following 3 tests will be done for common metrics collectors: JMX,
Prometheus, and OpenTelemetry.
# The partition tag can be observed from metrics if high verbosity is used
# The partition tag should result in an empty string or be filtered out by the
metrics collector if default verbosity is used
# Dynamically setting the verbosity can result in the behavior defined in the
above tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16128) Use metrics.verbosity in throughput metrics
Qichao Chu created KAFKA-16128:
--
Summary: Use metrics.verbosity in throughput metrics
Key: KAFKA-16128
URL: https://issues.apache.org/jira/browse/KAFKA-16128
Project: Kafka
Issue Type: Task
Reporter: Qichao Chu
*{{metrics.verbosity}}* will be a new dynamic config introduced to control the
verbosity(fan-out rate) of the metrics. It's a config with JSON format
specifying the condition controlling fan-out of the metrics.
This task will link it to the following throughput metrics. Specifically, in
the event level 'high' is used, the partition dimension should be added.
|Metrics Name|Meaning|
|{{MessagesInPerSec}}|Messages entered the partition, per second|
|{{BytesInPerSec}}|Bytes entered the partition, per second|
|{{BytesOutPerSec}}|Bytes retrieved from the partition, per second|
|{{BytesRejectedPerSec}}|Bytes exceeding max message size in a partition, per
second|
|{{TotalProduceRequestsPerSec}}|Produce request count for a partition, per
second|
|{{TotalFetchRequestsPerSec}}|Fetch request count for a partition, per second|
|{{FailedProduceRequestsPerSec}}|Failed to produce request count for a
partition, per second|
|{{FailedFetchRequestsPerSec}}|Failed to fetch request count for a partition,
per second|
|{{FetchMessageConversionsPerSec}}|Broker side conversions(de-compressions) for
a partition, per second|
|{{ProduceMessageConversionsPerSec}}|Broker side conversions(compressions) for
a partition, per second|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16127) Reset corresponding verbosity configs when topic is deleted
Qichao Chu created KAFKA-16127:
--
Summary: Reset corresponding verbosity configs when topic is
deleted
Key: KAFKA-16127
URL: https://issues.apache.org/jira/browse/KAFKA-16127
Project: Kafka
Issue Type: Task
Reporter: Qichao Chu
*{{metrics.verbosity}}* will be a new dynamic config introduced to control the
verbosity(fan-out rate) of the metrics. It's a config with JSON format
specifying the condition controlling fan-out of the metrics.
When the topic is deleted, the corresponding topic config should be reset to
default value because:
# If the topic is automatically recreated, the verbosity from the previous
generation may cause too much/too little metrics to be emitted
# Too many unused configurations may cause user error when configuring this
field. The typical workflow would be getting the value first, then adding or
modifying the config to reflect the latest requirement. If we don't delete
unused entries, it could only grow instead of evolve.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16125) Add parser for the metrics.verbosity config
Qichao Chu created KAFKA-16125:
--
Summary: Add parser for the metrics.verbosity config
Key: KAFKA-16125
URL: https://issues.apache.org/jira/browse/KAFKA-16125
Project: Kafka
Issue Type: Task
Reporter: Qichao Chu
Assignee: Qichao Chu
*{{metrics.verbosity}}* will be a new dynamic config introduced to control the
verbosity(fan-out rate) of the metrics. It's a config with JSON format
specifying the condition controlling fan-out of the metrics.
The parser will have the following validation responsibilities:
# Validate that the config conforms to JSON format.
# Validate that the level field is configured correctly with a set level
# Validate that the names field pattern is acceptable to
{{java.util.regex-compatible}}
# Validate that the filters field conforms to JSON map format
# Validate that all the filter patterns are acceptable to
{{java.util.regex-compatible}}
# Validate that no other field exists in the config
After parsing and validating the config, the parser should:
* Generate a config object which contains the level, names, and filter fields.
There should exist a default verbosity level too.
* Inside the object, the predicates should also be generated to allow fast
querying of the rules
* Overwrite/update the existing predicates if the config is updated
The config object should:
* Determine the verbosity level supplied with the topic name and the metrics
name
* If not defined, fall back to the default config level
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16124) Create metrics.verbosity dynamic config for controlling metrics verbosity
Qichao Chu created KAFKA-16124:
--
Summary: Create metrics.verbosity dynamic config for controlling
metrics verbosity
Key: KAFKA-16124
URL: https://issues.apache.org/jira/browse/KAFKA-16124
Project: Kafka
Issue Type: Task
Reporter: Qichao Chu
Assignee: Qichao Chu
*{{metrics.verbosity}}* will be a new dynamic config introduced to control the
verbosity(fan-out rate) of the metrics. It's a config with JSON format
specifying the condition controlling fan-out of the metrics.
More details:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15447) Partition Level Throughput Metrics
Qichao Chu created KAFKA-15447: -- Summary: Partition Level Throughput Metrics Key: KAFKA-15447 URL: https://issues.apache.org/jira/browse/KAFKA-15447 Project: Kafka Issue Type: Improvement Components: metrics Reporter: Qichao Chu Assignee: Qichao Chu Please find the details at [https://cwiki.apache.org/confluence/display/KAFKA/KIP-977+Partition-Level+Throughput+Metrics] -- This message was sent by Atlassian Jira (v8.20.10#820010)
