[jira] [Assigned] (KAFKA-6263) Expose metric for group metadata loading duration
[ https://issues.apache.org/jira/browse/KAFKA-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Vela reassigned KAFKA-6263: - Assignee: Anastasia Vela > Expose metric for group metadata loading duration > - > > Key: KAFKA-6263 > URL: https://issues.apache.org/jira/browse/KAFKA-6263 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Anastasia Vela >Priority: Major > Labels: needs-kip > > We have seen in several cases where the log cleaner either wasn't enabled or > had experienced some failure that __consumer_offsets partitions can grow > excessively. When one of these partitions changes leadership, the new > coordinator must load the offset cache from the start of the log, which can > take arbitrarily long depending on how large the partition has grown (we have > seen cases where it took hours). Catching this problem is not always easy > because the condition is rare and the symptom just tends to be a long period > of inactivity in the consumer group which gradually gets worse over time. It > may therefore be useful to have a broker metric for the load time so that it > can be monitored and potentially alerted on. Same thing goes for the > transaction log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6263) Expose metric for group metadata loading duration
[ https://issues.apache.org/jira/browse/KAFKA-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Vela updated KAFKA-6263: -- Labels: (was: needs-kip) > Expose metric for group metadata loading duration > - > > Key: KAFKA-6263 > URL: https://issues.apache.org/jira/browse/KAFKA-6263 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Anastasia Vela >Priority: Major > > We have seen in several cases where the log cleaner either wasn't enabled or > had experienced some failure that __consumer_offsets partitions can grow > excessively. When one of these partitions changes leadership, the new > coordinator must load the offset cache from the start of the log, which can > take arbitrarily long depending on how large the partition has grown (we have > seen cases where it took hours). Catching this problem is not always easy > because the condition is rare and the symptom just tends to be a long period > of inactivity in the consumer group which gradually gets worse over time. It > may therefore be useful to have a broker metric for the load time so that it > can be monitored and potentially alerted on. Same thing goes for the > transaction log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6263) Expose metric for group metadata loading duration
[ https://issues.apache.org/jira/browse/KAFKA-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Vela updated KAFKA-6263: -- Component/s: core > Expose metric for group metadata loading duration > - > > Key: KAFKA-6263 > URL: https://issues.apache.org/jira/browse/KAFKA-6263 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jason Gustafson >Assignee: Anastasia Vela >Priority: Major > > We have seen in several cases where the log cleaner either wasn't enabled or > had experienced some failure that __consumer_offsets partitions can grow > excessively. When one of these partitions changes leadership, the new > coordinator must load the offset cache from the start of the log, which can > take arbitrarily long depending on how large the partition has grown (we have > seen cases where it took hours). Catching this problem is not always easy > because the condition is rare and the symptom just tends to be a long period > of inactivity in the consumer group which gradually gets worse over time. It > may therefore be useful to have a broker metric for the load time so that it > can be monitored and potentially alerted on. Same thing goes for the > transaction log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-10525) Emit JSONs with new auto-generated schema
Anastasia Vela created KAFKA-10525: -- Summary: Emit JSONs with new auto-generated schema Key: KAFKA-10525 URL: https://issues.apache.org/jira/browse/KAFKA-10525 Project: Kafka Issue Type: Improvement Components: core Reporter: Anastasia Vela Assignee: Anastasia Vela Kafka’s request and response traces currently output in a format that is JSON-like and are not easily parsable. These are currently emitted by RequestChannel when logging is turned on at DEBUG level. Structured logs will be easier to load and parse with other tools like jq, elasticsearch, druid or presto. There is a new auto-generated schema for each request type that supports outputting JSON payloads for request and response payloads. These can be adapted to provide structured request tracing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
