[jira] [Created] (KAFKA-7722) Increase ProducerPerformance precision by using nanoTime
Kevin Lu created KAFKA-7722: --- Summary: Increase ProducerPerformance precision by using nanoTime Key: KAFKA-7722 URL: https://issues.apache.org/jira/browse/KAFKA-7722 Project: Kafka Issue Type: New Feature Components: tools Reporter: Kevin Lu Assignee: Kevin Lu https://cwiki.apache.org/confluence/display/KAFKA/KIP-403%3A+Increase+ProducerPerformance+precision+by+using+nanoTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7904) Add AtMinIsr topic partition category
Kevin Lu created KAFKA-7904: --- Summary: Add AtMinIsr topic partition category Key: KAFKA-7904 URL: https://issues.apache.org/jira/browse/KAFKA-7904 Project: Kafka Issue Type: New Feature Reporter: Kevin Lu Assignee: Kevin Lu https://cwiki.apache.org/confluence/display/KAFKA/KIP-427%3A+Add+AtMinIsr+topic+partition+category -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name
[
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Lu resolved KAFKA-5991.
-
Resolution: Fixed
> Change Consumer per partition lag metrics to put topic-partition-id in tags
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
> Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
> brought per partition lag metrics to {{KafkaConsumer}}, but these metrics
> put the {{TOPIC-PARTITION_ID}} inside of the metric name itself. These
> metrics should instead utilize the tags and put {{key="topic-partition"}} and
> {{value=TOPIC-PARTITION_ID}}.
> Per-broker (node) and per-topic metrics utilize tags in this way by putting
> {{key="node/topic"}} and {{value=NODE_ID/TOPIC_NAME}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7207) Make rate & total metrics documentation consistent
Kevin Lu created KAFKA-7207: --- Summary: Make rate & total metrics documentation consistent Key: KAFKA-7207 URL: https://issues.apache.org/jira/browse/KAFKA-7207 Project: Kafka Issue Type: Improvement Components: documentation Reporter: Kevin Lu Assignee: Kevin Lu [KIP-187|https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics] added an additional cumulative total metric for each existing rate metric. In the [Monitoring|https://kafka.apache.org/documentation/#monitoring] section of the docs, there is a mix of some sections only listing out the rate metric, and some sections listing out both the total metric and the rate metric. We should make all the sections consistent and list out all of the total metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7218) Log thread names & ids when KafkaConsumer throws ConcurrentModificationException
Kevin Lu created KAFKA-7218: --- Summary: Log thread names & ids when KafkaConsumer throws ConcurrentModificationException Key: KAFKA-7218 URL: https://issues.apache.org/jira/browse/KAFKA-7218 Project: Kafka Issue Type: Improvement Components: consumer, log Reporter: Kevin Lu Assignee: Kevin Lu KafkaConsumer does not support multi-threaded usage and any access by a thread that does not have the lock will cause ConcurrentModificationException to be thrown. For some users leveraging frameworks on top of Kafka that abstract the actual KafkaConsumer class/calls, it can be hard to identify user and/or framework bugs when this exception is thrown. It would be incredibly helpful to log both the thread name and the thread ID of the thread that has acquired the lock, and the current thread that is attempting to obtain the lock in the exception message. KafkaConsumer currently only tracks the id of the thread that has acquired the lock. Additionally, we should also keep track of the thread name. An example of the exception message: "KafkaConsumer is not safe for multi-threaded access: acquiredThreadName=acquiredThread, acquiredThreadId=1, currentThreadName=rejectedThread, currentThreadId=2" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7236) Add --critical-partitions option to describe topics command
Kevin Lu created KAFKA-7236: --- Summary: Add --critical-partitions option to describe topics command Key: KAFKA-7236 URL: https://issues.apache.org/jira/browse/KAFKA-7236 Project: Kafka Issue Type: Improvement Components: tools Reporter: Kevin Lu Assignee: Kevin Lu A topic partition can be in one of four states (assuming replication factor of 3): (ISR = in sync replica) 3/3 ISRs: OK 2/3 ISRs: WARNING (under-replicated partition) 1/3 ISRs: CRITICAL (under-replicated partition) 0/3 ISRs: FATAL (offline/unavailable partition) TopicCommand already has the --under-replicated-partitions and --unavailable-partitions flags, but it would be beneficial to include an additional --critical-partitions option that specifically lists out partitions in CRITICAL state (only one remaining ISR left). With this new option, Kafka users can use this option to identify the exact topic partitions that are critical and need immediate repartitioning. Kafka users can also set up critical alerts to trigger when the output of this command contains partitions. A couple cases where identifying this CRITICAL state is useful in alerting: * Users that have a large amount of topics in a single cluster, making it incredibly hard to manually repartition all topics that have under-replicated partitions, so they only take action when it hits CRITICAL state * Users with a high replication-factor that can tolerate some broker failures and only take action when it hits CRITICAL state -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7300) Add Fetcher fetch-error-rate and fetch-error-total metrics
Kevin Lu created KAFKA-7300: --- Summary: Add Fetcher fetch-error-rate and fetch-error-total metrics Key: KAFKA-7300 URL: https://issues.apache.org/jira/browse/KAFKA-7300 Project: Kafka Issue Type: New Feature Components: clients, consumer, metrics Reporter: Kevin Lu Assignee: Kevin Lu The KafkaConsumer is a complex client that requires many different components to function properly. When a consumer fails, it can be difficult to identify the root cause and which component failed (ConsumerCoordinator, Fetcher, ConsumerNetworkClient, etc). This aims to improve the monitoring and detection of KafkaConsumer’s Fetcher component. Fetcher will send a fetch request for each node that the consumer has assigned partitions for. This fetch request may fail under the following cases: * Intermittent network issues (goes to onFailure) * Node sent an invalid full/incremental fetch response (FetchSessionHandler’s handleResponse returns false) * FetchSessionIdNotFound * InvalidFetchSessionEpochException These cases are logged, but it would be valuable to provide a corresponding metric that allows for monitoring and alerting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-5906) Change metric.reporters configuration key to metrics.reporters to be consistent
Kevin Lu created KAFKA-5906: --- Summary: Change metric.reporters configuration key to metrics.reporters to be consistent Key: KAFKA-5906 URL: https://issues.apache.org/jira/browse/KAFKA-5906 Project: Kafka Issue Type: Improvement Components: config, metrics Reporter: Kevin Lu Priority: Minor The "metric.reporters" configuration key should be consistent with the actual classes. Clients have a MetricsReporter.class while the broker has a KafkaMetricsReporter.class. We have seen quite a few people configure this field incorrectly by setting it as "metrics.reporters". The configuration key could be renamed to "metrics.reporters" to match the classes, or the classes can be renamed to MetricReporter.class and KafkaMetricReporter.class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name
Kevin Lu created KAFKA-5991: --- Summary: Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name Key: KAFKA-5991 URL: https://issues.apache.org/jira/browse/KAFKA-5991 Project: Kafka Issue Type: Improvement Affects Versions: 0.11.0.0, 0.10.2.0, 0.10.1.0, 0.10.0.1, 0.10.0.0 Reporter: Kevin Lu Priority: Minor A KIP will be needed for this (?) as this requires a change to the public API (metric). [KIP-92](https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer) brought per partition lag metrics to KafkaConsumer, but these metrics put the "TOPIC-PARTITION_ID" inside of the metric name itself. These metrics should instead utilize the tags and put key="topic-partition" and value="TOPIC-PARTITION_ID". Per-broker (node) and per-topic metrics utilize tags in this way by putting key="node/topic" and value="NODE_ID/TOPIC_NAME". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6123) MetricsReporter does not get auto-generated client.id
Kevin Lu created KAFKA-6123:
---
Summary: MetricsReporter does not get auto-generated client.id
Key: KAFKA-6123
URL: https://issues.apache.org/jira/browse/KAFKA-6123
Project: Kafka
Issue Type: Improvement
Components: clients, metrics
Affects Versions: 0.11.0.0
Reporter: Kevin Lu
Priority: Minor
When a {{MetricsReporter}} is configured for a client, it will receive the
user-specified configurations via {{Configurable.configure(Map
configs)}}. Likewise, {{ProducerInterceptor}} and {{ConsumerInterceptor}}
receive user-specified configurations in their configure methods.
The difference is when a user does not specify the {{client.id}} field, Kafka
will auto-generate client ids (producer-1, producer-2, consumer-1, consumer-2,
etc). This auto-generated {{client.id}} will be passed into the interceptors'
configure method, but it is not passed to the {{MetricsReporter}} configure
method.
This makes it harder to directly map {{MetricsReporter}} with the interceptors
for the client when users do not specify the {{client.id}} field. The
{{client.id}} can be determined from identifying a metric with the
{{client.id}} tag, but this is hacky and requires traversal.
It would be useful to have auto-generated {{client.id}} field also passed to
the {{MetricsReporter}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-8874) KIP-517: Add consumer metric indicating time between poll calls
Kevin Lu created KAFKA-8874: --- Summary: KIP-517: Add consumer metric indicating time between poll calls Key: KAFKA-8874 URL: https://issues.apache.org/jira/browse/KAFKA-8874 Project: Kafka Issue Type: New Feature Components: consumer, metrics Reporter: Kevin Lu Assignee: Kevin Lu [https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls] -- This message was sent by Atlassian Jira (v8.3.2#803003)
