[jira] [Assigned] (KAFKA-15908) Remove deprecated Consumer API poll(long timeout)
[
https://issues.apache.org/jira/browse/KAFKA-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield reassigned KAFKA-15908:
Assignee: Andrew Schofield
> Remove deprecated Consumer API poll(long timeout)
> -
>
> Key: KAFKA-15908
> URL: https://issues.apache.org/jira/browse/KAFKA-15908
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Andrew Schofield
>Priority: Critical
> Labels: consumer-threading-refactor, timeout
> Fix For: 4.0.0
>
>
> Per
> [KIP-266|https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior],
> the {{Consumer.poll(long timeout)}} method was deprecated back in 2.0.0.
> In 3.7, there are two implementations, each with different behavior:
> * The {{LegacyKafkaConsumer}} implementation will continue to work but will
> log a warning about its removal
> * The {{AsyncKafkaConsumer}} implementation will throw an error.
> In 4.0, the `poll` method that takes a single `long` timeout will be removed
> altogether.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-17500) NOT_LEADER_OR_FOLLOWER with metadata redirection in share group response
[ https://issues.apache.org/jira/browse/KAFKA-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17500. -- Resolution: Fixed > NOT_LEADER_OR_FOLLOWER with metadata redirection in share group response > > > Key: KAFKA-17500 > URL: https://issues.apache.org/jira/browse/KAFKA-17500 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > Implement KIP-951 for ShareFetch and ShareAcknowledge responses to make > consumers more responsive to leadership changes -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16733) Add support for formatting new records written to offsets topic in kafka-dump-log.sh
[ https://issues.apache.org/jira/browse/KAFKA-16733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16733. -- Fix Version/s: 4.0.0 Resolution: Fixed > Add support for formatting new records written to offsets topic in > kafka-dump-log.sh > > > Key: KAFKA-16733 > URL: https://issues.apache.org/jira/browse/KAFKA-16733 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17671) Create better documentation for transactions
[ https://issues.apache.org/jira/browse/KAFKA-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-17671: Assignee: Andrew Schofield > Create better documentation for transactions > > > Key: KAFKA-17671 > URL: https://issues.apache.org/jira/browse/KAFKA-17671 > Project: Kafka > Issue Type: Task >Reporter: Justine Olshan >Assignee: Andrew Schofield >Priority: Major > > There is not a single source of truth that explains how transactions should > work in kafka. > We should create a document that explains how they work, the guarantees, and > tips for how to design applications that use transactions. > > The topic came up when discussing > https://issues.apache.org/jira/browse/KAFKA-17582 and the expectations for > consumer offsets when transactions are aborted. This should be made clear in > the documentation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17663) Add Metadata caching in admin.internals.PartitionLeaderStrategy
[ https://issues.apache.org/jira/browse/KAFKA-17663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-17663: Assignee: Andrew Schofield > Add Metadata caching in admin.internals.PartitionLeaderStrategy > --- > > Key: KAFKA-17663 > URL: https://issues.apache.org/jira/browse/KAFKA-17663 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Nicolas Guyomar >Assignee: Andrew Schofield >Priority: Major > > Hi team, > To the best of my knowledge, there is no Metadata caching in AdminClient > admin.internals.PartitionLeaderStrategy, and for use cases that keep long > live AdminClient running with scheduled invocation of method such as the > listOffsets, we could keep reusing the same Metadata and not request fresh > one on each invocation > This is particularly true when the listOffsets is invoked for monitoring > purpose, often querying a high number of partitions at the same time, leading > to rather "large" Metadata requests to be processed on the cluster side > Thank you -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16734) Add support for formatting records written to share-group state topic in kafka-dump-log.sh
[ https://issues.apache.org/jira/browse/KAFKA-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885313#comment-17885313 ] Andrew Schofield commented on KAFKA-16734: -- This will take a tiny tweak to KIP-932 to add an appropriate flag to kafka-dump-log.sh. > Add support for formatting records written to share-group state topic in > kafka-dump-log.sh > -- > > Key: KAFKA-16734 > URL: https://issues.apache.org/jira/browse/KAFKA-16734 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Sushant Mahajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16733) Add support for formatting new records written to offsets topic in kafka-dump-log.sh
[ https://issues.apache.org/jira/browse/KAFKA-16733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16733: - Summary: Add support for formatting new records written to offsets topic in kafka-dump-log.sh (was: Add support for formatting new records written to offsets topic) > Add support for formatting new records written to offsets topic in > kafka-dump-log.sh > > > Key: KAFKA-16733 > URL: https://issues.apache.org/jira/browse/KAFKA-16733 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17634) Tighten up wakeup handling for share consumer
Andrew Schofield created KAFKA-17634: Summary: Tighten up wakeup handling for share consumer Key: KAFKA-17634 URL: https://issues.apache.org/jira/browse/KAFKA-17634 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16734) Add support for formatting records written to share-group state topic in kafka-dump-log.sh
[ https://issues.apache.org/jira/browse/KAFKA-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-16734: Assignee: Sushant Mahajan (was: Andrew Schofield) Summary: Add support for formatting records written to share-group state topic in kafka-dump-log.sh (was: Add support for formatting records written to share-group state topic) > Add support for formatting records written to share-group state topic in > kafka-dump-log.sh > -- > > Key: KAFKA-16734 > URL: https://issues.apache.org/jira/browse/KAFKA-16734 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Sushant Mahajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17516) Client metrics configuration resources default configs not shown by kafka-configs.sh
[
https://issues.apache.org/jira/browse/KAFKA-17516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881472#comment-17881472
]
Andrew Schofield edited comment on KAFKA-17516 at 9/24/24 2:07 PM:
---
Looking at
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration,]
it seems that the following output would be sensible.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --alter --entity-type
client-metrics --entity-name CM1 --add-config
metrics=org.apache.kafka,interval.ms=1000
This sets the configs metrics and interval.ms, but leaves match unset.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --describe
--entity-type client-metrics --entity-name CM1
Dynamic configs for client-metric CM1 are:
interval.ms=1000 sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=1000}
metrics=org.apache.kafka sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=org.apache.kafka}
This doesn't show the value of the config match because it's not set and will
take the default value. However, you can see the default with this command.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --describe
--entity-type client-metrics --entity-name CM1
Dynamic configs for client-metric CM1 are:
interval.ms=1000 sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=1000}
metrics=org.apache.kafka sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=org.apache.kafka}
match= sensitive=false synonyms={}
Using the kafka-client-metrics.sh tool instead, the output always shows the
defaults giving a quick way to see the effective settings.
$ bin/kafka-client-metrics.sh --bootstrap-server localhost:9092 --describe
--name CM1
Client metrics configs for CM1 are:
metrics=org.apache.kafka
interval.ms=30
match=
Now it's possible to work out what the interval duration is without having to
remember the default value.
was (Author: schofielaj):
Looking at
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration,]
it seems that the following output would be sensible.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --alter --entity-type
client-metrics --entity-name CM1 --add-config
metrics=org.apache.kafka,interval.ms=1000
This sets the configs metrics and interval.ms, but leaves match unset.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --describe
--entity-type client-metrics --entity-name CM1
Dynamic configs for client-metric CM1 are:
interval.ms=1000 sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=1000, DEFAULT_CONFIG=30}
metrics=org.apache.kafka sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=org.apache.kafka, DEFAULT_CONFIG=}
This doesn't show the value of the config match because it's not set and will
take the default value. However, you can see the default with this command.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --describe
--entity-type client-metrics --entity-name CM1
Dynamic configs for client-metric CM1 are:
interval.ms=1000 sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=1000, DEFAULT_CONFIG=30}
metrics=org.apache.kafka sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=org.apache.kafka, DEFAULT_CONFIG=}
match= sensitive=false synonyms=\{DEFAULT_CONFIG=}
Using the kafka-client-metrics.sh tool instead, the output always shows the
defaults giving a quick way to see the effective settings.
$ bin/kafka-client-metrics.sh --bootstrap-server localhost:9092 --describe
--name CM1
Client metrics configs for CM1 are:
metrics=org.apache.kafka
interval.ms=30
match=
Now it's possible to work out what the interval duration is without having to
remember the default value.
> Client metrics configuration resources default configs not shown by
> kafka-configs.sh
>
>
> Key: KAFKA-17516
> URL: https://issues.apache.org/jira/browse/KAFKA-17516
> Project: Kafka
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Fix For: 4.0.0
>
>
> KIP-714 introduces client metrics configuration resources for configuring the
> set of metrics pushed by clients. The kafka-configs.sh and
> kafka-client-metrics.sh tools are used to manipulate them.
> When kafka-configs.sh is used to manipulate configurations for other resource
> types, such as topics and groups, configuration for which there is no
> explicit dynamic value specified have a default which is either part of the
> definition of the definition of each config, or inherited from another config
> (usually f
[jira] [Created] (KAFKA-17549) Add INCONSISTENT_GROUP_TYPE to ShareGroupDescribe and kafka-share-groups.sh
Andrew Schofield created KAFKA-17549: Summary: Add INCONSISTENT_GROUP_TYPE to ShareGroupDescribe and kafka-share-groups.sh Key: KAFKA-17549 URL: https://issues.apache.org/jira/browse/KAFKA-17549 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17550) Add INCONSISTENT_GROUP_TYPE to ConsumerGroupDescribe and kafka-consumer-groups.sh
Andrew Schofield created KAFKA-17550: Summary: Add INCONSISTENT_GROUP_TYPE to ConsumerGroupDescribe and kafka-consumer-groups.sh Key: KAFKA-17550 URL: https://issues.apache.org/jira/browse/KAFKA-17550 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17548) Create admin client integration test with a mixture of consumer and share groups
Andrew Schofield created KAFKA-17548: Summary: Create admin client integration test with a mixture of consumer and share groups Key: KAFKA-17548 URL: https://issues.apache.org/jira/browse/KAFKA-17548 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Enhance PlaintextAdminIntegrationTest.scala with a test of Admin.listGroups in the presence of a mixture of group types. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17502) Modify handling of commitSync() in ShareConsumeRequestManager
[ https://issues.apache.org/jira/browse/KAFKA-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17502. -- Resolution: Fixed > Modify handling of commitSync() in ShareConsumeRequestManager > - > > Key: KAFKA-17502 > URL: https://issues.apache.org/jira/browse/KAFKA-17502 > Project: Kafka > Issue Type: Sub-task >Reporter: Shivsundar R >Assignee: Shivsundar R >Priority: Major > > Currently the code in ShareConsumeRequestManager works on the basis that > there can only be one commitSync()/close() at a time. But there is a chance > these calls timeout on the application thread, but are still sent later on > the background thread. > To cover this case, we will now have a list of AcknowledgeRequestStates to > store the commitSyncs() and a separate requestState to store the close(). > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17546) Create kafka-groups.sh tool
Andrew Schofield created KAFKA-17546: Summary: Create kafka-groups.sh tool Key: KAFKA-17546 URL: https://issues.apache.org/jira/browse/KAFKA-17546 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16891) KIP-1043: Administration of groups
[ https://issues.apache.org/jira/browse/KAFKA-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-16891: Assignee: Andrew Schofield > KIP-1043: Administration of groups > -- > > Key: KAFKA-16891 > URL: https://issues.apache.org/jira/browse/KAFKA-16891 > Project: Kafka > Issue Type: New Feature >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > > This issue tracks the development of KIP-1043: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1043%3A+Administration+of+groups. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16732) Investigate missing values for share-coordinator-metrics and share-group-metrics in the broker
[ https://issues.apache.org/jira/browse/KAFKA-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16732: - Summary: Investigate missing values for share-coordinator-metrics and share-group-metrics in the broker (was: Support for share-coordinator-metrics in the broker) > Investigate missing values for share-coordinator-metrics and > share-group-metrics in the broker > -- > > Key: KAFKA-16732 > URL: https://issues.apache.org/jira/browse/KAFKA-16732 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17544) Add a log message for KafkaShareConsumer early access
Andrew Schofield created KAFKA-17544: Summary: Add a log message for KafkaShareConsumer early access Key: KAFKA-17544 URL: https://issues.apache.org/jira/browse/KAFKA-17544 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield KafkaShareConsumer should issue a log message when it is instantiated and still in Early Access. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17516) Client metrics configuration resources default configs not shown by kafka-configs.sh
[
https://issues.apache.org/jira/browse/KAFKA-17516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881472#comment-17881472
]
Andrew Schofield commented on KAFKA-17516:
--
Looking at
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration,]
it seems that the following output would be sensible.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --alter --entity-type
client-metrics --entity-name CM1 --add-config
metrics=org.apache.kafka,interval.ms=1000
This sets the configs metrics and interval.ms, but leaves match unset.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --describe
--entity-type client-metrics --entity-name CM1
Dynamic configs for client-metric CM1 are:
interval.ms=1000 sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=1000, DEFAULT_CONFIG=30}
metrics=org.apache.kafka sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=org.apache.kafka, DEFAULT_CONFIG=}
This doesn't show the value of the config match because it's not set and will
take the default value. However, you can see the default with this command.
$ bin/kafka-config.sh --bootstrap-server localhost:9092 --describe
--entity-type client-metrics --entity-name CM1
Dynamic configs for client-metric CM1 are:
interval.ms=1000 sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=1000, DEFAULT_CONFIG=30}
metrics=org.apache.kafka sensitive=false
synonyms=\{DYNAMIC_CLIENT_METRICS_CONFIG=org.apache.kafka, DEFAULT_CONFIG=}
match= sensitive=false synonyms=\{DEFAULT_CONFIG=}
Using the kafka-client-metrics.sh tool instead, the output always shows the
defaults giving a quick way to see the effective settings.
$ bin/kafka-client-metrics.sh --bootstrap-server localhost:9092 --describe
--name CM1
Client metrics configs for CM1 are:
metrics=org.apache.kafka
interval.ms=30
match=
Now it's possible to work out what the interval duration is without having to
remember the default value.
> Client metrics configuration resources default configs not shown by
> kafka-configs.sh
>
>
> Key: KAFKA-17516
> URL: https://issues.apache.org/jira/browse/KAFKA-17516
> Project: Kafka
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Fix For: 4.0.0
>
>
> KIP-714 introduces client metrics configuration resources for configuring the
> set of metrics pushed by clients. The kafka-configs.sh and
> kafka-client-metrics.sh tools are used to manipulate them.
> When kafka-configs.sh is used to manipulate configurations for other resource
> types, such as topics and groups, configuration for which there is no
> explicit dynamic value specified have a default which is either part of the
> definition of the definition of each config, or inherited from another config
> (usually from the broker).
> This is not the case currently for client metrics resources, and this is not
> intentional. There are defaults for the configurations and they are applied.
> It's just that they're not visible using kafka-configs.sh --describe --all
> and not returned by the admin API. They should be.
> This issue adjusts the handling of the default values so that they are
> displayed by this tool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17541) Improve handling of delivery count
Andrew Schofield created KAFKA-17541: Summary: Improve handling of delivery count Key: KAFKA-17541 URL: https://issues.apache.org/jira/browse/KAFKA-17541 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield There are two situations in which the delivery count handling needs to be more intelligent. First, for records which are automatically released as a result of closing a share session normally, the delivery count should not be incremented. These records were fetched but they were not actually delivered to the client since the disposition of the delivery records is carried in the ShareAcknowledge which closes the share session. Any remaining records were not delivered, only fetched. Second, for records which have a delivery count which is more than 1 or 2, there is a suspicion that the records are not being delivered due to a problem rather than just natural retrying. The batching of these records should be reduced, even down to a single record as a time so we do not have the failure to deliver a poisoned record actually causing adjacent records to be considered unsuccessful and potentially reach the delivery count limit without proper reason. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17510) Refactor code between SharePartitionManager and DelayedShareFetch for share partition initialization
[ https://issues.apache.org/jira/browse/KAFKA-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-17510: Assignee: Apoorv Mittal > Refactor code between SharePartitionManager and DelayedShareFetch for share > partition initialization > > > Key: KAFKA-17510 > URL: https://issues.apache.org/jira/browse/KAFKA-17510 > Project: Kafka > Issue Type: Sub-task >Reporter: Abhinav Dixit >Assignee: Apoorv Mittal >Priority: Major > > In reference to comment > [https://github.com/apache/kafka/pull/16969#discussion_r1750754052] , we need > to refactor code between SharePartitionManager and DelayedShareFetch so that > the initialization error of share partition gets considered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17539) Implement registerMetricsForSubscription
Andrew Schofield created KAFKA-17539: Summary: Implement registerMetricsForSubscription Key: KAFKA-17539 URL: https://issues.apache.org/jira/browse/KAFKA-17539 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 Add the equivalent of KIP-1076 to KIP-932 and implement the new methods. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17516) Client metrics configuration resources default configs not shown by kafka-configs.sh
[ https://issues.apache.org/jira/browse/KAFKA-17516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17516: - Summary: Client metrics configuration resources default configs not shown by kafka-configs.sh (was: Client metrics configuration resources default configs not shown by Kafka-config.sh) > Client metrics configuration resources default configs not shown by > kafka-configs.sh > > > Key: KAFKA-17516 > URL: https://issues.apache.org/jira/browse/KAFKA-17516 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 3.7.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > KIP-714 introduces client metrics configuration resources for configuring the > set of metrics pushed by clients. The kafka-configs.sh and > kafka-client-metrics.sh tools are used to manipulate them. > When kafka-configs.sh is used to manipulate configurations for other resource > types, such as topics and groups, configuration for which there is no > explicit dynamic value specified have a default which is either part of the > definition of the definition of each config, or inherited from another config > (usually from the broker). > This is not the case currently for client metrics resources, and this is not > intentional. There are defaults for the configurations and they are applied. > It's just that they're not visible using kafka-configs.sh --describe --all > and not returned by the admin API. They should be. > This issue adjusts the handling of the default values so that they are > displayed by this tool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17516) Client metrics configuration resources default configs not shown by Kafka-config.sh
Andrew Schofield created KAFKA-17516: Summary: Client metrics configuration resources default configs not shown by Kafka-config.sh Key: KAFKA-17516 URL: https://issues.apache.org/jira/browse/KAFKA-17516 Project: Kafka Issue Type: Bug Components: admin Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 KIP-714 introduces client metrics configuration resources for configuring the set of metrics pushed by clients. The kafka-configs.sh and kafka-client-metrics.sh tools are used to manipulate them. When kafka-configs.sh is used to manipulate configurations for other resource types, such as topics and groups, configuration for which there is no explicit dynamic value specified have a default which is either part of the definition of the definition of each config, or inherited from another config (usually from the broker). This is not the case currently for client metrics resources, and this is not intentional. There are defaults for the configurations and they are applied. It's just that they're not visible using kafka-configs.sh --describe --all and not returned by the admin API. They should be. This issue adjusts the handling of the default values so that they are displayed by this tool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17400) Introduce a purgatory to deal with share fetch requests that cannot be completed instantaneously
[ https://issues.apache.org/jira/browse/KAFKA-17400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17400: - Summary: Introduce a purgatory to deal with share fetch requests that cannot be completed instantaneously (was: Intorduce a purgatory to deal with share fetch requests that cannot be completed instantaneously) > Introduce a purgatory to deal with share fetch requests that cannot be > completed instantaneously > > > Key: KAFKA-17400 > URL: https://issues.apache.org/jira/browse/KAFKA-17400 > Project: Kafka > Issue Type: Sub-task >Reporter: Abhinav Dixit >Assignee: Abhinav Dixit >Priority: Major > > h1. When record lock partition limit is reached, the ShareFetch should wait > for up to MaxWaitMs for records to be released -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17443) Create a tool to manipulate share-group state snapshots for recovery purposes
Andrew Schofield created KAFKA-17443: Summary: Create a tool to manipulate share-group state snapshots for recovery purposes Key: KAFKA-17443 URL: https://issues.apache.org/jira/browse/KAFKA-17443 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17347) Add omitted --client-metrics option to kafka-configs.sh
[ https://issues.apache.org/jira/browse/KAFKA-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17347: - Description: KIP-714 introduced client metrics resources to kafka-configs.sh. The option --entity-type client-metrics was added, and a shorthand of "client-metrics" was also included in the comments. However, the "-client-metrics" option whose syntax matches all of the other entity types was omitted. This corrects that omission. (was: KIP-714 introduced client metrics resources to kafka-configs.sh. The option `--entity-type client-metrics` was added, and a shorthand of `--client-metrics` was also included in the comments. However, the `--client-metrics` option whose syntax matches all of the other entity types was omitted. This corrects that omission.) > Add omitted --client-metrics option to kafka-configs.sh > --- > > Key: KAFKA-17347 > URL: https://issues.apache.org/jira/browse/KAFKA-17347 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 3.7.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > KIP-714 introduced client metrics resources to kafka-configs.sh. The option > --entity-type client-metrics was added, and a shorthand of "client-metrics" > was also included in the comments. However, the "-client-metrics" option > whose syntax matches all of the other entity types was omitted. This corrects > that omission. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17347) Add omitted --client-metrics option to kafka-configs.sh
[ https://issues.apache.org/jira/browse/KAFKA-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877004#comment-17877004 ] Andrew Schofield commented on KAFKA-17347: -- Hi [~isding_l], I would prefer to do it because it's my mistake in the first place. I missed in when I did that part of KIP-714/KIP-1000. I would appreciate your review of the PR when I'm ready though. > Add omitted --client-metrics option to kafka-configs.sh > --- > > Key: KAFKA-17347 > URL: https://issues.apache.org/jira/browse/KAFKA-17347 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 3.7.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > KIP-714 introduced client metrics resources to kafka-configs.sh. The option > `--entity-type client-metrics` was added, and a shorthand of > `--client-metrics` was also included in the comments. However, the > `--client-metrics` option whose syntax matches all of the other entity types > was omitted. This corrects that omission. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17425) Improve coexistence of consumer groups and share groups
Andrew Schofield created KAFKA-17425: Summary: Improve coexistence of consumer groups and share groups Key: KAFKA-17425 URL: https://issues.apache.org/jira/browse/KAFKA-17425 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 Investigate all of the situations in which consumer group and share group operations can be used on the wrong type of group and ensure that the result is good. Currently, there is at least one ClassCastException in the group coordinator related to this, so this task will resolve that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17291) Add integration test for share group list and describe admin calls
[ https://issues.apache.org/jira/browse/KAFKA-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17291. -- Resolution: Fixed > Add integration test for share group list and describe admin calls > -- > > Key: KAFKA-17291 > URL: https://issues.apache.org/jira/browse/KAFKA-17291 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17378) Initial performance testing fixes
Andrew Schofield created KAFKA-17378: Summary: Initial performance testing fixes Key: KAFKA-17378 URL: https://issues.apache.org/jira/browse/KAFKA-17378 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 Collect together some fixes for problems identified during initial performance testing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17350) Improve kafka-share-groups.sh --describe output for empty groups
[ https://issues.apache.org/jira/browse/KAFKA-17350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17350. -- Resolution: Fixed > Improve kafka-share-groups.sh --describe output for empty groups > > > Key: KAFKA-17350 > URL: https://issues.apache.org/jira/browse/KAFKA-17350 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > When you use `kafka-share-groups.sh --describe` for an empty group, it prints > an empty table without any indication that this is expected. > `kafka-consumer-groups.sh` summarises the group status to make the output > more informative. This is a simple improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16727) Add dynamic group configuration for record lock duration
[ https://issues.apache.org/jira/browse/KAFKA-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-16727: Assignee: Chirag Wadhwa (was: Andrew Schofield) > Add dynamic group configuration for record lock duration > > > Key: KAFKA-16727 > URL: https://issues.apache.org/jira/browse/KAFKA-16727 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Chirag Wadhwa >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17368) Add delivery count to kafka-console-share-consumer.sh
Andrew Schofield created KAFKA-17368: Summary: Add delivery count to kafka-console-share-consumer.sh Key: KAFKA-17368 URL: https://issues.apache.org/jira/browse/KAFKA-17368 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Andrew Schofield Assignee: Andrew Schofield Now that ConsumerRecord.deliveryCount() exists, enhance kafka-console-share-consumer.sh to exploit it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17350) Improve kafka-share-groups.sh --describe output for empty groups
Andrew Schofield created KAFKA-17350: Summary: Improve kafka-share-groups.sh --describe output for empty groups Key: KAFKA-17350 URL: https://issues.apache.org/jira/browse/KAFKA-17350 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 When you use `kafka-share-groups.sh --describe` for an empty group, it prints an empty table without any indication that this is expected. `kafka-consumer-groups.sh` summarises the group status to make the output more informative. This is a simple improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17347) Add omitted --client-metrics option to kafka-configs.sh
Andrew Schofield created KAFKA-17347: Summary: Add omitted --client-metrics option to kafka-configs.sh Key: KAFKA-17347 URL: https://issues.apache.org/jira/browse/KAFKA-17347 Project: Kafka Issue Type: Bug Components: tools Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 KIP-714 introduced client metrics resources to kafka-configs.sh. The option `--entity-type client-metrics` was added, and a shorthand of `--client-metrics` was also included in the comments. However, the `--client-metrics` option whose syntax matches all of the other entity types was omitted. This corrects that omission. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17341) Refactor consumer heartbeat managers
[ https://issues.apache.org/jira/browse/KAFKA-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17341: - Parent: KAFKA-16092 Issue Type: Sub-task (was: Task) > Refactor consumer heartbeat managers > > > Key: KAFKA-17341 > URL: https://issues.apache.org/jira/browse/KAFKA-17341 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > HeartbeatRequestManager and ShareHeartbeatRequestManager are very closely > related and there's a lot of code duplication. They can be refactored to > share the majority of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17341) Refactor consumer heartbeat managers
[ https://issues.apache.org/jira/browse/KAFKA-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17341: - Issue Type: Task (was: Improvement) > Refactor consumer heartbeat managers > > > Key: KAFKA-17341 > URL: https://issues.apache.org/jira/browse/KAFKA-17341 > Project: Kafka > Issue Type: Task > Components: clients >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > HeartbeatRequestManager and ShareHeartbeatRequestManager are very closely > related and there's a lot of code duplication. They can be refactored to > share the majority of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17341) Refactor consumer heartbeat managers
Andrew Schofield created KAFKA-17341: Summary: Refactor consumer heartbeat managers Key: KAFKA-17341 URL: https://issues.apache.org/jira/browse/KAFKA-17341 Project: Kafka Issue Type: Improvement Components: clients Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 HeartbeatRequestManager and ShareHeartbeatRequestManager are very closely related and there's a lot of code duplication. They can be refactored to share the majority of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16714) kafka-share-groups.sh supporting list and describe
[ https://issues.apache.org/jira/browse/KAFKA-16714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16714. -- Resolution: Fixed > kafka-share-groups.sh supporting list and describe > -- > > Key: KAFKA-16714 > URL: https://issues.apache.org/jira/browse/KAFKA-16714 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17292) Add SHARE to group.coordinator.rebalance.protocols
[ https://issues.apache.org/jira/browse/KAFKA-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17292. -- Resolution: Fixed > Add SHARE to group.coordinator.rebalance.protocols > -- > > Key: KAFKA-17292 > URL: https://issues.apache.org/jira/browse/KAFKA-17292 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17225) Refactor consumer membership managers
[ https://issues.apache.org/jira/browse/KAFKA-17225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17225. -- Resolution: Fixed > Refactor consumer membership managers > - > > Key: KAFKA-17225 > URL: https://issues.apache.org/jira/browse/KAFKA-17225 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > The initial drop of ShareMembershipManager contained a lot of code duplicated > from MembershipManagerImpl. The plan was always to share as much code as > possible between the membership managers for consumer groups and share > groups. This issue refactors the membership managers so that almost all of > the code is in common. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17318) Introduce ConsumerRecord.deliveryCount()
Andrew Schofield created KAFKA-17318: Summary: Introduce ConsumerRecord.deliveryCount() Key: KAFKA-17318 URL: https://issues.apache.org/jira/browse/KAFKA-17318 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17289) Add integration test for ShareGroupDescribe requests
[ https://issues.apache.org/jira/browse/KAFKA-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17289. -- Fix Version/s: 4.0.0 Resolution: Fixed > Add integration test for ShareGroupDescribe requests > > > Key: KAFKA-17289 > URL: https://issues.apache.org/jira/browse/KAFKA-17289 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17217) Clients : Optimise batching of requests per node in ShareConsumeRequestManager
[ https://issues.apache.org/jira/browse/KAFKA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17217. -- Resolution: Fixed > Clients : Optimise batching of requests per node in ShareConsumeRequestManager > -- > > Key: KAFKA-17217 > URL: https://issues.apache.org/jira/browse/KAFKA-17217 > Project: Kafka > Issue Type: Sub-task >Reporter: Shivsundar R >Assignee: Shivsundar R >Priority: Major > > In ShareConsumeRequestManager, currently every time we perform a commitSync > or commitAsync, we create one ShareAcknowledge RPC for the same. Here, we can > optimise the number of RPC calls by batching the acknowledgements before the > next poll is invoked per node. > This will ensure that between 2 calls, the acknowledgements are accumulated > in one request per node and then sent during poll, resulting in lesser RPC > calls. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16716) Add AdminClient.describeShareGroups and AdminClient.listShareGroups
[ https://issues.apache.org/jira/browse/KAFKA-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16716. -- Resolution: Fixed > Add AdminClient.describeShareGroups and AdminClient.listShareGroups > --- > > Key: KAFKA-16716 > URL: https://issues.apache.org/jira/browse/KAFKA-16716 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17292) Add SHARE to group.coordinator.rebalance.protocols
Andrew Schofield created KAFKA-17292: Summary: Add SHARE to group.coordinator.rebalance.protocols Key: KAFKA-17292 URL: https://issues.apache.org/jira/browse/KAFKA-17292 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17291) Add integration test for share group list and describe admin calls
Andrew Schofield created KAFKA-17291: Summary: Add integration test for share group list and describe admin calls Key: KAFKA-17291 URL: https://issues.apache.org/jira/browse/KAFKA-17291 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17290) Add integration test for ShareGroupFetch/Acknowledge requests
Andrew Schofield created KAFKA-17290: Summary: Add integration test for ShareGroupFetch/Acknowledge requests Key: KAFKA-17290 URL: https://issues.apache.org/jira/browse/KAFKA-17290 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17289) Add integration test for ShareGroupDescribe requests
Andrew Schofield created KAFKA-17289: Summary: Add integration test for ShareGroupDescribe requests Key: KAFKA-17289 URL: https://issues.apache.org/jira/browse/KAFKA-17289 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17287) Add integration test for KafkaShareConsumer
Andrew Schofield created KAFKA-17287: Summary: Add integration test for KafkaShareConsumer Key: KAFKA-17287 URL: https://issues.apache.org/jira/browse/KAFKA-17287 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Shivsundar R Add an integration test suite for testing KafkaShareConsumer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16723) Add kafka-console-share-consumer.sh
[ https://issues.apache.org/jira/browse/KAFKA-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-16723: Assignee: Shivsundar R (was: Andrew Schofield) > Add kafka-console-share-consumer.sh > --- > > Key: KAFKA-16723 > URL: https://issues.apache.org/jira/browse/KAFKA-16723 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Shivsundar R >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17262) kafka-topics.sh usage message is confusing
Andrew Schofield created KAFKA-17262: Summary: kafka-topics.sh usage message is confusing Key: KAFKA-17262 URL: https://issues.apache.org/jira/browse/KAFKA-17262 Project: Kafka Issue Type: Bug Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 There is a lot of historical cruft in the usage message for kafka-topics.sh. For example, the --bootstrap-server option is required nowadays, but the usage message is written in some cases as if it was still optional. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17247) Revised share group record schemas
Andrew Schofield created KAFKA-17247: Summary: Revised share group record schemas Key: KAFKA-17247 URL: https://issues.apache.org/jira/browse/KAFKA-17247 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 In KIP-932, the group coordinator does not persist assignments for share groups. While this sounds like a good idea in terms of minimising overhead for data which doesn't strictly need to be recoverable, it significantly adds to the complexity of working with the coordinator framework. This issue revised the definitions of the share group record schemas following more closely the schemas used for consumer groups, and eliminating the need to maintain soft state alongside the group coordinator's timeline structure. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17231) Share consumer node latency metrics
[ https://issues.apache.org/jira/browse/KAFKA-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17231: - Description: This is the share consumer equivalent of https://github.com/apache/kafka/pull/16755. Summary: Share consumer node latency metrics (was: Sn) > Share consumer node latency metrics > --- > > Key: KAFKA-17231 > URL: https://issues.apache.org/jira/browse/KAFKA-17231 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Priority: Major > > This is the share consumer equivalent of > https://github.com/apache/kafka/pull/16755. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17231) Share consumer node latency metrics
[ https://issues.apache.org/jira/browse/KAFKA-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-17231: Assignee: Andrew Schofield > Share consumer node latency metrics > --- > > Key: KAFKA-17231 > URL: https://issues.apache.org/jira/browse/KAFKA-17231 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > > This is the share consumer equivalent of > https://github.com/apache/kafka/pull/16755. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17231) Sn
Andrew Schofield created KAFKA-17231: Summary: Sn Key: KAFKA-17231 URL: https://issues.apache.org/jira/browse/KAFKA-17231 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17225) Refactor consumer membership managers
Andrew Schofield created KAFKA-17225: Summary: Refactor consumer membership managers Key: KAFKA-17225 URL: https://issues.apache.org/jira/browse/KAFKA-17225 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 The initial drop of ShareMembershipManager contained a lot of code duplicated from MembershipManagerImpl. The plan was always to share as much code as possible between the membership managers for consumer groups and share groups. This issue refactors the membership managers so that almost all of the code is in common. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17144) Retract v6 of ListGroups RPC added in error
Andrew Schofield created KAFKA-17144: Summary: Retract v6 of ListGroups RPC added in error Key: KAFKA-17144 URL: https://issues.apache.org/jira/browse/KAFKA-17144 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield As agreed in the latter stages of the review for KIP-932, the AdminClient.listGroups() and ListGroups v6 RPC were to be removed. However, I forgot to remove the RPC definition from the KIP and then merged the v6 RPC in AK. We need to retract this unused RPC version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16730) Initial code for share-group consumer
[ https://issues.apache.org/jira/browse/KAFKA-16730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16730. -- Resolution: Fixed > Initial code for share-group consumer > - > > Key: KAFKA-16730 > URL: https://issues.apache.org/jira/browse/KAFKA-16730 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17093. -- Resolution: Not A Problem > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864674#comment-17864674 ] Andrew Schofield commented on KAFKA-17093: -- [~tkalmijn] You're welcome. It was definitely worth seeing whether there was something to fix as we approach the next release cutoff. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864599#comment-17864599 ] Andrew Schofield commented on KAFKA-17093: -- *I do think that seekToEnd works as designed and as documented.* The difference between the isolation levels is that read_committed consumers are not prepared to read uncommitted data. KafkaConsumer.poll() and KafkaConsumer.seekToEnd() both stop before revealing the existence of any records from open transactions. The consumer's position is the next offset that will be requested when fetching records from the broker. At the end of a topic-partition, it points past the last record so that fetching sees new records as they turn up. The record prior to that might be a control record. It might even have been removed by the log cleaner for a compacted topic. The position automatically "jumps" over gaps and control records. These kinds of factors are why it's tricky to be concrete about the "last message". KafkaAdminClient.listOffsets is your best bet to discover offset information, but it behaves as you'd expect :) . For background, there is a KIP in this area [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1021%3A+Allow+to+get+last+stable+offset+%28LSO%29+in+kafka-get-offsets.sh] which is not yet voted. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864284#comment-17864284 ] Andrew Schofield commented on KAFKA-17093: -- [~tkalmijn] I think there's a path forward here. The problem you have is that you're struggling to deal with unknown numbers of control records when you're trying to read all of the records. I think the following works: * Call KafkaConsumer.seekToEnd() and then KafkaConsumer.position() - this will skip over the control records and position the consumer right after all of the records which it can read. * Seek back to wherever you want to consume from. * Consume using KafkaConsumer.poll(). When the offset reaches the end offset, or KafkaConsumer.position() reaches that point, you've consumed all of the records. I've added a new version of the test app. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17093: - Attachment: Kafka17093-v3.java > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16741) Add ShareGroupHeartbeat API support in GroupCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16741. -- Resolution: Fixed > Add ShareGroupHeartbeat API support in GroupCoordinator > --- > > Key: KAFKA-16741 > URL: https://issues.apache.org/jira/browse/KAFKA-16741 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16731) Support for share-group-metrics in the broker
[ https://issues.apache.org/jira/browse/KAFKA-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16731. -- Resolution: Fixed > Support for share-group-metrics in the broker > - > > Key: KAFKA-16731 > URL: https://issues.apache.org/jira/browse/KAFKA-16731 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Sushant Mahajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864118#comment-17864118 ] Andrew Schofield commented on KAFKA-17093: -- [~tkalmijn] Client application attached. On AK trunk, it works as expected. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-17093: - Attachment: Kafka17093.java > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864081#comment-17864081 ] Andrew Schofield commented on KAFKA-17093: -- [~tkalmijn] I'll work on writing a simple Kafka client application that very carefully lines up the topic with a predictable set of records and then uses the consumer methods in question. I'll share the code once I've finished the investigations. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-17093: Assignee: Andrew Schofield > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16731) Support for share-group-metrics in the broker
[ https://issues.apache.org/jira/browse/KAFKA-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-16731: Assignee: Sushant Mahajan (was: Andrew Schofield) > Support for share-group-metrics in the broker > - > > Key: KAFKA-16731 > URL: https://issues.apache.org/jira/browse/KAFKA-16731 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Sushant Mahajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16731) Support for share-group-metrics in the broker
[ https://issues.apache.org/jira/browse/KAFKA-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863158#comment-17863158 ] Andrew Schofield commented on KAFKA-16731: -- https://github.com/apache/kafka/pull/16488 > Support for share-group-metrics in the broker > - > > Key: KAFKA-16731 > URL: https://issues.apache.org/jira/browse/KAFKA-16731 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17028) FindCoordinator v6 initial implementation
[ https://issues.apache.org/jira/browse/KAFKA-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17028. -- Fix Version/s: 3.9.0 Resolution: Fixed > FindCoordinator v6 initial implementation > - > > Key: KAFKA-17028 > URL: https://issues.apache.org/jira/browse/KAFKA-17028 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17028) FindCoordinator v6 initial implementation
Andrew Schofield created KAFKA-17028: Summary: FindCoordinator v6 initial implementation Key: KAFKA-17028 URL: https://issues.apache.org/jira/browse/KAFKA-17028 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16725) Add broker configurations
[ https://issues.apache.org/jira/browse/KAFKA-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16725. -- Resolution: Fixed > Add broker configurations > - > > Key: KAFKA-16725 > URL: https://issues.apache.org/jira/browse/KAFKA-16725 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh
[ https://issues.apache.org/jira/browse/KAFKA-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16724: - Fix Version/s: 3.9.0 > Add new options for kafka-producer-perf-test.sh > --- > > Key: KAFKA-16724 > URL: https://issues.apache.org/jira/browse/KAFKA-16724 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Shivsundar R >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16721) Add exceptions for the new error codes
[ https://issues.apache.org/jira/browse/KAFKA-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16721: - Fix Version/s: 3.9.0 > Add exceptions for the new error codes > -- > > Key: KAFKA-16721 > URL: https://issues.apache.org/jira/browse/KAFKA-16721 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16713) Add new RPC definitions
[ https://issues.apache.org/jira/browse/KAFKA-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16713: - Fix Version/s: 3.9.0 > Add new RPC definitions > --- > > Key: KAFKA-16713 > URL: https://issues.apache.org/jira/browse/KAFKA-16713 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16950) Define Persister and Share Coordinator RPCs
[
https://issues.apache.org/jira/browse/KAFKA-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-16950.
--
Resolution: Fixed
> Define Persister and Share Coordinator RPCs
> ---
>
> Key: KAFKA-16950
> URL: https://issues.apache.org/jira/browse/KAFKA-16950
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Andrew Schofield
>Priority: Major
>
> Add Persister interface with schemas for RPCs. The classes which are needed
> by SharePartition to integrate are below, note some of them results from the
> generated json schema classes.
>
>
> {code:java}
> import org.apache.kafka.server.group.share.GroupTopicPartitionData;
> import org.apache.kafka.server.group.share.PartitionAllData;
> import org.apache.kafka.server.group.share.PartitionErrorData;
> import org.apache.kafka.server.group.share.PartitionFactory;
> import org.apache.kafka.server.group.share.PartitionIdLeaderEpochData;
> import org.apache.kafka.server.group.share.PartitionStateBatchData;
> import org.apache.kafka.server.group.share.Persister;
> import org.apache.kafka.server.group.share.PersisterStateBatch;
> import org.apache.kafka.server.group.share.ReadShareGroupStateParameters;
> import org.apache.kafka.server.group.share.ReadShareGroupStateResult;
> import org.apache.kafka.server.group.share.TopicData;
> import org.apache.kafka.server.group.share.WriteShareGroupStateParameters;
> import org.apache.kafka.server.group.share.WriteShareGroupStateResult; {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14511) Extend AlterIncrementalConfigs API to support group config
[ https://issues.apache.org/jira/browse/KAFKA-14511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855076#comment-17855076 ] Andrew Schofield commented on KAFKA-14511: -- Hi [~isding_l] I'm happy to take this one over and get it finished for the next AK release. OK with you? > Extend AlterIncrementalConfigs API to support group config > -- > > Key: KAFKA-14511 > URL: https://issues.apache.org/jira/browse/KAFKA-14511 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Lan Ding >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16950) Define Persister and Share Coordinator RPCs
[
https://issues.apache.org/jira/browse/KAFKA-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield reassigned KAFKA-16950:
Assignee: Andrew Schofield
> Define Persister and Share Coordinator RPCs
> ---
>
> Key: KAFKA-16950
> URL: https://issues.apache.org/jira/browse/KAFKA-16950
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Andrew Schofield
>Priority: Major
>
> Add Persister interface with schemas for RPCs. The classes which are needed
> by SharePartition to integrate are below, note some of them results from the
> generated json schema classes.
>
>
> {code:java}
> import org.apache.kafka.server.group.share.GroupTopicPartitionData;
> import org.apache.kafka.server.group.share.PartitionAllData;
> import org.apache.kafka.server.group.share.PartitionErrorData;
> import org.apache.kafka.server.group.share.PartitionFactory;
> import org.apache.kafka.server.group.share.PartitionIdLeaderEpochData;
> import org.apache.kafka.server.group.share.PartitionStateBatchData;
> import org.apache.kafka.server.group.share.Persister;
> import org.apache.kafka.server.group.share.PersisterStateBatch;
> import org.apache.kafka.server.group.share.ReadShareGroupStateParameters;
> import org.apache.kafka.server.group.share.ReadShareGroupStateResult;
> import org.apache.kafka.server.group.share.TopicData;
> import org.apache.kafka.server.group.share.WriteShareGroupStateParameters;
> import org.apache.kafka.server.group.share.WriteShareGroupStateResult; {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh
[ https://issues.apache.org/jira/browse/KAFKA-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16724. -- Resolution: Fixed > Add new options for kafka-producer-perf-test.sh > --- > > Key: KAFKA-16724 > URL: https://issues.apache.org/jira/browse/KAFKA-16724 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Shivsundar R >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16914) Add Share Group dynamic and broker configs
[ https://issues.apache.org/jira/browse/KAFKA-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16914: - Summary: Add Share Group dynamic and broker configs (was: Add Shage Group dynamic and broker configs) > Add Share Group dynamic and broker configs > -- > > Key: KAFKA-16914 > URL: https://issues.apache.org/jira/browse/KAFKA-16914 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Apoorv Mittal >Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0 > > > Add the configs required for share group in KafkaConfig or equivalent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh
[ https://issues.apache.org/jira/browse/KAFKA-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-16724: Assignee: Shivsundar R (was: Andrew Schofield) > Add new options for kafka-producer-perf-test.sh > --- > > Key: KAFKA-16724 > URL: https://issues.apache.org/jira/browse/KAFKA-16724 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Shivsundar R >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16894) Define group.version=2
Andrew Schofield created KAFKA-16894: Summary: Define group.version=2 Key: KAFKA-16894 URL: https://issues.apache.org/jira/browse/KAFKA-16894 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Create group.version=2 as the switch to enable share groups in the broker. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16740) Define skeleton for SharePartitionManager and SharePartition
[ https://issues.apache.org/jira/browse/KAFKA-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16740. -- Resolution: Fixed > Define skeleton for SharePartitionManager and SharePartition > > > Key: KAFKA-16740 > URL: https://issues.apache.org/jira/browse/KAFKA-16740 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > > Add high level design for broker side implementation for fetching and > acknowledging messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16891) KIP-1043: Administration of groups
Andrew Schofield created KAFKA-16891: Summary: KIP-1043: Administration of groups Key: KAFKA-16891 URL: https://issues.apache.org/jira/browse/KAFKA-16891 Project: Kafka Issue Type: New Feature Reporter: Andrew Schofield This issue tracks the development of KIP-1043: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1043%3A+Administration+of+groups. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16715) Create KafkaShareConsumer interface
[ https://issues.apache.org/jira/browse/KAFKA-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16715. -- Resolution: Fixed > Create KafkaShareConsumer interface > --- > > Key: KAFKA-16715 > URL: https://issues.apache.org/jira/browse/KAFKA-16715 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16721) Add exceptions for the new error codes
[ https://issues.apache.org/jira/browse/KAFKA-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16721. -- Resolution: Fixed Delivered as part of https://github.com/apache/kafka/pull/16022. > Add exceptions for the new error codes > -- > > Key: KAFKA-16721 > URL: https://issues.apache.org/jira/browse/KAFKA-16721 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16713) Add new RPC definitions
[ https://issues.apache.org/jira/browse/KAFKA-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16713. -- Resolution: Fixed ShareGroupHeartbeat, ShareGroupDescribe, ShareFetch and ShareAcknowledge RPCs have been delivered. > Add new RPC definitions > --- > > Key: KAFKA-16713 > URL: https://issues.apache.org/jira/browse/KAFKA-16713 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16835) Add Support for consumer to read in commit order.
[ https://issues.apache.org/jira/browse/KAFKA-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850425#comment-17850425 ] Andrew Schofield commented on KAFKA-16835: -- This is a very significant change to how Kafka works. Kafka does actually know the commit order so it is technically possible to understand the commit order of records with some processing. But... Because records are added to topic-partitions in produce order, not commit order, and the offsets are assigned when the records are added, commit-order consumption would not deliver records across transactions in offset order. It seems very tricky to me to understand how to track consumer position (committed offsets) when the records are being delivered out of order. The combination of commit-order delivery and consumer groups seems incompatible with the current client API behaviour. In summary, this would be a big piece of work that would likely require deep changes in the broker as well as the client. Not a small effort. > Add Support for consumer to read in commit order. > - > > Key: KAFKA-16835 > URL: https://issues.apache.org/jira/browse/KAFKA-16835 > Project: Kafka > Issue Type: New Feature > Components: clients, consumer, offset manager >Reporter: Manjunath >Priority: Critical > > Currently consumer supports offset order to receive messages.There are some > cases where commit order is very important.For example assume case where > PostgreSQL-14 randomly streams multiple in-progress large transactions to > some intermediate client which starts transactional producer instances for > multiple in-progress transactions,using this producer instances client pushes > data to kafka. Now consumer should strictly read messages based on commit > order. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16759) Invalid client telemetry transition on consumer close
[
https://issues.apache.org/jira/browse/KAFKA-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-16759.
--
Resolution: Fixed
> Invalid client telemetry transition on consumer close
> -
>
> Key: KAFKA-16759
> URL: https://issues.apache.org/jira/browse/KAFKA-16759
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Minor
> Fix For: 3.8.0
>
>
> Using the console consumer with client telemetry enabled, I hit an invalid
> state transition when closing the consumer with CTRL-C. The consumer sends a
> final "terminating" telemetry push which puts the client telemetry reporter
> into TERMINATING_PUSH_IN_PROGRESS state. When it receives a response in this
> state, it attempts an invalid state transition.
>
> {noformat}
> [2024-05-13 19:19:35,804] WARN Error updating client telemetry state,
> disabled telemetry
> (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
> java.lang.IllegalStateException: Invalid telemetry state transition from
> TERMINATING_PUSH_IN_PROGRESS to PUSH_NEEDED; the valid telemetry state
> transitions from TERMINATING_PUSH_IN_PROGRESS are: TERMINATED
> at
> org.apache.kafka.common.telemetry.ClientTelemetryState.validateTransition(ClientTelemetryState.java:163)
> at
> org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.maybeSetState(ClientTelemetryReporter.java:827)
> at
> org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.handleResponse(ClientTelemetryReporter.java:520)
> at
> org.apache.kafka.clients.NetworkClient$TelemetrySender.handleResponse(NetworkClient.java:1321)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:948)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:594)
> at
> org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.poll(NetworkClientDelegate.java:130)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.sendUnsentRequests(ConsumerNetworkThread.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.cleanup(ConsumerNetworkThread.java:275)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:95)
> [2024-05-13 19:19:35,805] WARN Unable to transition state after successful
> push telemetry from state TERMINATING_PUSH_IN_PROGRESS
> (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16741) Add ShareGroupHeartbeat API support in GroupCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16741: - Summary: Add ShareGroupHeartbeat API support in GroupCoordinator (was: Add SharGroupHeartbeat API support in GroupCoordinator) > Add ShareGroupHeartbeat API support in GroupCoordinator > --- > > Key: KAFKA-16741 > URL: https://issues.apache.org/jira/browse/KAFKA-16741 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16722) Add ConsumerGroupPartitionAssignor interface
[ https://issues.apache.org/jira/browse/KAFKA-16722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16722: - Description: Adds the interface `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` as described in KIP-932. (was: Adds the interfaces `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` and `org.apache.kafka.coordinator.group.assignor.ShareGroupPartitionAssignor` as described in KIP-932.) > Add ConsumerGroupPartitionAssignor interface > > > Key: KAFKA-16722 > URL: https://issues.apache.org/jira/browse/KAFKA-16722 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.8.0 > > > Adds the interface > `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` > as described in KIP-932. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16722) Add ConsumerGroupPartitionAssignor interface
[ https://issues.apache.org/jira/browse/KAFKA-16722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16722: - Summary: Add ConsumerGroupPartitionAssignor interface (was: Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor) > Add ConsumerGroupPartitionAssignor interface > > > Key: KAFKA-16722 > URL: https://issues.apache.org/jira/browse/KAFKA-16722 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.8.0 > > > Adds the interfaces > `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` > and `org.apache.kafka.coordinator.group.assignor.ShareGroupPartitionAssignor` > as described in KIP-932. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16722) Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor
[ https://issues.apache.org/jira/browse/KAFKA-16722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16722: - Fix Version/s: 3.8.0 > Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor > -- > > Key: KAFKA-16722 > URL: https://issues.apache.org/jira/browse/KAFKA-16722 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.8.0 > > > Adds the interfaces > `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` > and `org.apache.kafka.coordinator.group.assignor.ShareGroupPartitionAssignor` > as described in KIP-932. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16722) Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor
[ https://issues.apache.org/jira/browse/KAFKA-16722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield updated KAFKA-16722: - Description: Adds the interfaces `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` and `org.apache.kafka.coordinator.group.assignor.ShareGroupPartitionAssignor` as described in KIP-932. > Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor > -- > > Key: KAFKA-16722 > URL: https://issues.apache.org/jira/browse/KAFKA-16722 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > > Adds the interfaces > `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` > and `org.apache.kafka.coordinator.group.assignor.ShareGroupPartitionAssignor` > as described in KIP-932. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16759) Invalid client telemetry transition on consumer close
[
https://issues.apache.org/jira/browse/KAFKA-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846360#comment-17846360
]
Andrew Schofield commented on KAFKA-16759:
--
Analysing the transitions, the invalid transition is real.
{noformat}
+++ SUBSCRIPTION_NEEDED -->> SUBSCRIPTION_IN_PROGRESS
+++ SUBSCRIPTION_IN_PROGRESS -->> PUSH_NEEDED
^C+++ PUSH_NEEDED -->> TERMINATING_PUSH_NEEDED
+++ TERMINATING_PUSH_NEEDED -->> TERMINATING_PUSH_IN_PROGRESS
+++ TERMINATING_PUSH_IN_PROGRESS -->> PUSH_NEEDED
[2024-05-14 16:48:05,043] WARN Error updating client telemetry state, disabled
telemetry (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
java.lang.IllegalStateException: Invalid telemetry state transition from
TERMINATING_PUSH_IN_PROGRESS to PUSH_NEEDED; the valid telemetry state
transitions from TERMINATING_PUSH_IN_PROGRESS are: TERMINATED
at
org.apache.kafka.common.telemetry.ClientTelemetryState.validateTransition(ClientTelemetryState.java:163)
at
org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.maybeSetState(ClientTelemetryReporter.java:828)
at
org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.handleResponse(ClientTelemetryReporter.java:520)
at
org.apache.kafka.clients.NetworkClient$TelemetrySender.handleResponse(NetworkClient.java:1321)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:948)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:594)
at
org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.poll(NetworkClientDelegate.java:130)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.runOnce(ConsumerNetworkThread.java:140)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:88)
[2024-05-14 16:48:05,044] WARN Unable to transition state after successful push
telemetry from state TERMINATING_PUSH_IN_PROGRESS
(org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
+++ TERMINATING_PUSH_IN_PROGRESS -->> TERMINATED{noformat}
> Invalid client telemetry transition on consumer close
> -
>
> Key: KAFKA-16759
> URL: https://issues.apache.org/jira/browse/KAFKA-16759
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Minor
> Fix For: 3.8.0
>
>
> Using the console consumer with client telemetry enabled, I hit an invalid
> state transition when closing the consumer with CTRL-C. The consumer sends a
> final "terminating" telemetry push which puts the client telemetry reporter
> into TERMINATING_PUSH_IN_PROGRESS state. When it receives a response in this
> state, it attempts an invalid state transition.
>
> {noformat}
> [2024-05-13 19:19:35,804] WARN Error updating client telemetry state,
> disabled telemetry
> (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
> java.lang.IllegalStateException: Invalid telemetry state transition from
> TERMINATING_PUSH_IN_PROGRESS to PUSH_NEEDED; the valid telemetry state
> transitions from TERMINATING_PUSH_IN_PROGRESS are: TERMINATED
> at
> org.apache.kafka.common.telemetry.ClientTelemetryState.validateTransition(ClientTelemetryState.java:163)
> at
> org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.maybeSetState(ClientTelemetryReporter.java:827)
> at
> org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.handleResponse(ClientTelemetryReporter.java:520)
> at
> org.apache.kafka.clients.NetworkClient$TelemetrySender.handleResponse(NetworkClient.java:1321)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:948)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:594)
> at
> org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.poll(NetworkClientDelegate.java:130)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.sendUnsentRequests(ConsumerNetworkThread.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.cleanup(ConsumerNetworkThread.java:275)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:95)
> [2024-05-13 19:19:35,805] WARN Unable to transition state after successful
> push telemetry from state TERMINATING_PUSH_IN_PROGRESS
> (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16759) Invalid client telemetry transition on consumer close
Andrew Schofield created KAFKA-16759:
Summary: Invalid client telemetry transition on consumer close
Key: KAFKA-16759
URL: https://issues.apache.org/jira/browse/KAFKA-16759
Project: Kafka
Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Assignee: Andrew Schofield
Fix For: 3.8.0
Using the console consumer with client telemetry enabled, I hit an invalid
state transition when closing the consumer with CTRL-C. The consumer sends a
final "terminating" telemetry push which puts the client telemetry reporter
into TERMINATING_PUSH_IN_PROGRESS state. When it receives a response in this
state, it attempts an invalid state transition.
{noformat}
[2024-05-13 19:19:35,804] WARN Error updating client telemetry state, disabled
telemetry (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
java.lang.IllegalStateException: Invalid telemetry state transition from
TERMINATING_PUSH_IN_PROGRESS to PUSH_NEEDED; the valid telemetry state
transitions from TERMINATING_PUSH_IN_PROGRESS are: TERMINATED
at
org.apache.kafka.common.telemetry.ClientTelemetryState.validateTransition(ClientTelemetryState.java:163)
at
org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.maybeSetState(ClientTelemetryReporter.java:827)
at
org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.handleResponse(ClientTelemetryReporter.java:520)
at
org.apache.kafka.clients.NetworkClient$TelemetrySender.handleResponse(NetworkClient.java:1321)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:948)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:594)
at
org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.poll(NetworkClientDelegate.java:130)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.sendUnsentRequests(ConsumerNetworkThread.java:262)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.cleanup(ConsumerNetworkThread.java:275)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:95)
[2024-05-13 19:19:35,805] WARN Unable to transition state after successful push
telemetry from state TERMINATING_PUSH_IN_PROGRESS
(org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
