[jira] [Created] (KAFKA-16226) Performance regression in Trogdor benchmark with high partition counts
Mayank Shekhar Narula created KAFKA-16226: - Summary: Performance regression in Trogdor benchmark with high partition counts Key: KAFKA-16226 URL: https://issues.apache.org/jira/browse/KAFKA-16226 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0, 3.6.1 Right now in java-client, producer-batches backoff upto retry.backoff.ms(100ms by default). This Jira proposes that backoff should be skipped if client knows of a newer-leader for the partition in a sub-sequent retry(typically through refresh of parition-metadata via the Metadata RPC). This would help improve the latency of the produce-request around when partition leadership changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15970) KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva
[ https://issues.apache.org/jira/browse/KAFKA-15970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15970. --- Resolution: Fixed > KIP-951, port newly added tests in FetcherTest.java to > FetchRequestManagerTest.ajva > --- > > Key: KAFKA-15970 > URL: https://issues.apache.org/jira/browse/KAFKA-15970 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 3.7.0 >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Labels: kip-951 > > Java client changes for > https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15970) KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva
Mayank Shekhar Narula created KAFKA-15970: - Summary: KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva Key: KAFKA-15970 URL: https://issues.apache.org/jira/browse/KAFKA-15970 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Java client changes for https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15868) KIP-951 - Leader discovery optimisations for the client
[ https://issues.apache.org/jira/browse/KAFKA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15868. --- Resolution: Fixed > KIP-951 - Leader discovery optimisations for the client > --- > > Key: KAFKA-15868 > URL: https://issues.apache.org/jira/browse/KAFKA-15868 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 3.7.0 >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Fix For: 3.7.0 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15868) KIP-951 - Leader discovery optimisations for the client
Mayank Shekhar Narula created KAFKA-15868: - Summary: KIP-951 - Leader discovery optimisations for the client Key: KAFKA-15868 URL: https://issues.apache.org/jira/browse/KAFKA-15868 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0 https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15824) SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet
[ https://issues.apache.org/jira/browse/KAFKA-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15824. --- Resolution: Fixed > SubscriptionState's maybeValidatePositionForCurrentLeader should handle > partition which isn't subscribed yet > > > Key: KAFKA-15824 > URL: https://issues.apache.org/jira/browse/KAFKA-15824 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Fix For: 3.7.0 > > > As can be seen [here|#L453], maybeValidatePositionForCurrentLeader doesn't > check if partition is subscribed by checking TopicPartitionState cached is > null or not, as done by > [maybeCompleteValidation|[http://example.com|https://github.com/apache/kafka/blob/832627fc78484fdc7c8d6da8a2d20e7691dbf882/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java#L477]]. > So it throws IllegalStateException for a partition that is yet not > subscribed. > Lack of this check writing thread-safe code w.r.t SubscriptionState class is > awkward. This can be seen from the example code below. For example, at line 1 > partitionA would be in allCurrentlySubscribedTopics, but at line 2 it could > be removed from subscribed partitions(in a separate thread). So this forces > the user of this class to handle IllegalStateException which is awkward. > {code:java} > // Following is example code for the user of > SubscriptionState::maybeValidatePositionForCurrentLeader > Set allCurrentlySubscribedTopics = > subscriptionState.assignedPartitions(); // line 1 > if(allCurrentlySubscribedTopics.contains(tp)) { > ConsumerMetadata.LeaderAndEpoch leaderAndEpoch = > metadata.currentLeader(tp); > try() { > subscriptionState.maybeValidatePositionForCurrentLeader(apiVersions, tp, > leaderAndEpoch); // line 2 > } catch (IllegalStateException e) { >// recover from it. // line 3 > } > }{code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15824) SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet
Mayank Shekhar Narula created KAFKA-15824: - Summary: SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet Key: KAFKA-15824 URL: https://issues.apache.org/jira/browse/KAFKA-15824 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0, 3.6.1 Right now in java-client, producer-batches backoff upto retry.backoff.ms(100ms by default). This Jira proposes that backoff should be skipped if client knows of a newer-leader for the partition in a sub-sequent retry(typically through refresh of parition-metadata via the Metadata RPC). This would help improve the latency of the produce-request around when partition leadership changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15415) In Java-client, backoff should be skipped for retried producer-batch to a new leader
[ https://issues.apache.org/jira/browse/KAFKA-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15415. --- Resolution: Fixed > In Java-client, backoff should be skipped for retried producer-batch to a new > leader > > > Key: KAFKA-15415 > URL: https://issues.apache.org/jira/browse/KAFKA-15415 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Fix For: 3.7.0, 3.6.1 > > > Right now in java-client, producer-batches backoff upto > retry.backoff.ms(100ms by default). This Jira proposes that backoff should be > skipped if client knows of a newer-leader for the partition in a sub-sequent > retry(typically through refresh of parition-metadata via the Metadata RPC). > This would help improve the latency of the produce-request around when > partition leadership changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15627) KIP-951, Java client changes to incorporate the leader discovery optimisations
Mayank Shekhar Narula created KAFKA-15627: - Summary: KIP-951, Java client changes to incorporate the leader discovery optimisations Key: KAFKA-15627 URL: https://issues.apache.org/jira/browse/KAFKA-15627 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Java client changes for https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15415) In Java-client, backoff should be skipped for retried producer-batch for a new leader
Mayank Shekhar Narula created KAFKA-15415: - Summary: In Java-client, backoff should be skipped for retried producer-batch for a new leader Key: KAFKA-15415 URL: https://issues.apache.org/jira/browse/KAFKA-15415 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0 Right now in java-client, producer-batches backoff upto `retry.backoff.ms`(100ms by default). This Jira proposes that backoff should be skipped if in a sub-sequent retry, client knows of a newer-leader for the partition. This would help improve the latency of the produce-request around leadership changes of a partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)