[jira] [Created] (KAFKA-16226) Performance regression in Trogdor benchmark with high partition counts
Mayank Shekhar Narula created KAFKA-16226: - Summary: Performance regression in Trogdor benchmark with high partition counts Key: KAFKA-16226 URL: https://issues.apache.org/jira/browse/KAFKA-16226 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0, 3.6.1 Right now in java-client, producer-batches backoff upto retry.backoff.ms(100ms by default). This Jira proposes that backoff should be skipped if client knows of a newer-leader for the partition in a sub-sequent retry(typically through refresh of parition-metadata via the Metadata RPC). This would help improve the latency of the produce-request around when partition leadership changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15970) KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva
[ https://issues.apache.org/jira/browse/KAFKA-15970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15970. --- Resolution: Fixed > KIP-951, port newly added tests in FetcherTest.java to > FetchRequestManagerTest.ajva > --- > > Key: KAFKA-15970 > URL: https://issues.apache.org/jira/browse/KAFKA-15970 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 3.7.0 >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Labels: kip-951 > > Java client changes for > https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15970) KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva
Mayank Shekhar Narula created KAFKA-15970: - Summary: KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva Key: KAFKA-15970 URL: https://issues.apache.org/jira/browse/KAFKA-15970 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Java client changes for https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15868) KIP-951 - Leader discovery optimisations for the client
[ https://issues.apache.org/jira/browse/KAFKA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15868. --- Resolution: Fixed > KIP-951 - Leader discovery optimisations for the client > --- > > Key: KAFKA-15868 > URL: https://issues.apache.org/jira/browse/KAFKA-15868 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 3.7.0 >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Fix For: 3.7.0 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15868) KIP-951 - Leader discovery optimisations for the client
Mayank Shekhar Narula created KAFKA-15868: - Summary: KIP-951 - Leader discovery optimisations for the client Key: KAFKA-15868 URL: https://issues.apache.org/jira/browse/KAFKA-15868 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0 https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15824) SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet
[
https://issues.apache.org/jira/browse/KAFKA-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Shekhar Narula resolved KAFKA-15824.
---
Resolution: Fixed
> SubscriptionState's maybeValidatePositionForCurrentLeader should handle
> partition which isn't subscribed yet
>
>
> Key: KAFKA-15824
> URL: https://issues.apache.org/jira/browse/KAFKA-15824
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Mayank Shekhar Narula
>Assignee: Mayank Shekhar Narula
>Priority: Major
> Fix For: 3.7.0
>
>
> As can be seen [here|#L453], maybeValidatePositionForCurrentLeader doesn't
> check if partition is subscribed by checking TopicPartitionState cached is
> null or not, as done by
> [maybeCompleteValidation|[http://example.com|https://github.com/apache/kafka/blob/832627fc78484fdc7c8d6da8a2d20e7691dbf882/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java#L477]].
> So it throws IllegalStateException for a partition that is yet not
> subscribed.
> Lack of this check writing thread-safe code w.r.t SubscriptionState class is
> awkward. This can be seen from the example code below. For example, at line 1
> partitionA would be in allCurrentlySubscribedTopics, but at line 2 it could
> be removed from subscribed partitions(in a separate thread). So this forces
> the user of this class to handle IllegalStateException which is awkward.
> {code:java}
> // Following is example code for the user of
> SubscriptionState::maybeValidatePositionForCurrentLeader
> Set allCurrentlySubscribedTopics =
> subscriptionState.assignedPartitions(); // line 1
> if(allCurrentlySubscribedTopics.contains(tp)) {
> ConsumerMetadata.LeaderAndEpoch leaderAndEpoch =
> metadata.currentLeader(tp);
> try() {
> subscriptionState.maybeValidatePositionForCurrentLeader(apiVersions, tp,
> leaderAndEpoch); // line 2
> } catch (IllegalStateException e) {
>// recover from it. // line 3
> }
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15824) SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet
Mayank Shekhar Narula created KAFKA-15824: - Summary: SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet Key: KAFKA-15824 URL: https://issues.apache.org/jira/browse/KAFKA-15824 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0, 3.6.1 Right now in java-client, producer-batches backoff upto retry.backoff.ms(100ms by default). This Jira proposes that backoff should be skipped if client knows of a newer-leader for the partition in a sub-sequent retry(typically through refresh of parition-metadata via the Metadata RPC). This would help improve the latency of the produce-request around when partition leadership changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15415) In Java-client, backoff should be skipped for retried producer-batch to a new leader
[ https://issues.apache.org/jira/browse/KAFKA-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15415. --- Resolution: Fixed > In Java-client, backoff should be skipped for retried producer-batch to a new > leader > > > Key: KAFKA-15415 > URL: https://issues.apache.org/jira/browse/KAFKA-15415 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Mayank Shekhar Narula >Assignee: Mayank Shekhar Narula >Priority: Major > Fix For: 3.7.0, 3.6.1 > > > Right now in java-client, producer-batches backoff upto > retry.backoff.ms(100ms by default). This Jira proposes that backoff should be > skipped if client knows of a newer-leader for the partition in a sub-sequent > retry(typically through refresh of parition-metadata via the Metadata RPC). > This would help improve the latency of the produce-request around when > partition leadership changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15627) KIP-951, Java client changes to incorporate the leader discovery optimisations
Mayank Shekhar Narula created KAFKA-15627: - Summary: KIP-951, Java client changes to incorporate the leader discovery optimisations Key: KAFKA-15627 URL: https://issues.apache.org/jira/browse/KAFKA-15627 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Java client changes for https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15415) In Java-client, backoff should be skipped for retried producer-batch for a new leader
Mayank Shekhar Narula created KAFKA-15415: - Summary: In Java-client, backoff should be skipped for retried producer-batch for a new leader Key: KAFKA-15415 URL: https://issues.apache.org/jira/browse/KAFKA-15415 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayank Shekhar Narula Assignee: Mayank Shekhar Narula Fix For: 3.7.0 Right now in java-client, producer-batches backoff upto `retry.backoff.ms`(100ms by default). This Jira proposes that backoff should be skipped if in a sub-sequent retry, client knows of a newer-leader for the partition. This would help improve the latency of the produce-request around leadership changes of a partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
