[jira] [Resolved] (KAFKA-17306) Soften the validation when replaying tombstones
[ https://issues.apache.org/jira/browse/KAFKA-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17306. - Fix Version/s: 4.0.0 Resolution: Fixed > Soften the validation when replaying tombstones > --- > > Key: KAFKA-17306 > URL: https://issues.apache.org/jira/browse/KAFKA-17306 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: David Jacot >Priority: Major > Fix For: 4.0.0 > > > At present, replaying the tombstones requires the deleted entity to exist. > However the record to create the entity can be removed and only leave the > tombstone after compaction. > This can cause error when a group coordinator loads a new __consumer_offsets > partition, as some entities can be removed without creation during replay. As > a result, we should soften the validation when replaying tombstones to avoid > it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15756) Migrate existing integration tests to run old protocol in new coordinator
[
https://issues.apache.org/jira/browse/KAFKA-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-15756.
-
Resolution: Won't Do
The new group coordinator is now used by default so all the tests use it by
default too unless specified otherwise.
> Migrate existing integration tests to run old protocol in new coordinator
> -
>
> Key: KAFKA-15756
> URL: https://issues.apache.org/jira/browse/KAFKA-15756
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
>
> There is one flaky test left, we need to figure out how to reduce the
> flakiness.
> {code:java}
> testConsumptionWithBrokerFailures{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15621) Add histogram metrics to GroupCoordinatorRuntimeMetrics
[ https://issues.apache.org/jira/browse/KAFKA-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15621. - Fix Version/s: 4.0.0 Resolution: Fixed > Add histogram metrics to GroupCoordinatorRuntimeMetrics > --- > > Key: KAFKA-15621 > URL: https://issues.apache.org/jira/browse/KAFKA-15621 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Jeff Kim >Priority: Major > Fix For: 4.0.0 > > > We will add new histograms to Kafka Metrics library soon. When we have it, we > can start using it in GroupCoordinatorRuntimeMetrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17413) Re-introduce `group.version` feature flag
[ https://issues.apache.org/jira/browse/KAFKA-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17413. - Resolution: Fixed > Re-introduce `group.version` feature flag > - > > Key: KAFKA-17413 > URL: https://issues.apache.org/jira/browse/KAFKA-17413 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Blocker > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17327) Add support of group in kafka-configs.sh
[ https://issues.apache.org/jira/browse/KAFKA-17327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17327. - Resolution: Fixed > Add support of group in kafka-configs.sh > > > Key: KAFKA-17327 > URL: https://issues.apache.org/jira/browse/KAFKA-17327 > Project: Kafka > Issue Type: Improvement >Reporter: Lan Ding >Assignee: Lan Ding >Priority: Major > Fix For: 4.0.0 > > > Add support of group in kafka-configs.sh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17376) Use the new group coordinator by default in 4.0
[ https://issues.apache.org/jira/browse/KAFKA-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17376. - Resolution: Fixed > Use the new group coordinator by default in 4.0 > --- > > Key: KAFKA-17376 > URL: https://issues.apache.org/jira/browse/KAFKA-17376 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Blocker > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-14048) The Next Generation of the Consumer Rebalance Protocol
[ https://issues.apache.org/jira/browse/KAFKA-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot reopened KAFKA-14048: - > The Next Generation of the Consumer Rebalance Protocol > -- > > Key: KAFKA-14048 > URL: https://issues.apache.org/jira/browse/KAFKA-14048 > Project: Kafka > Issue Type: Improvement >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 4.0.0 > > > This Jira tracks the development of KIP-848: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14048) The Next Generation of the Consumer Rebalance Protocol
[ https://issues.apache.org/jira/browse/KAFKA-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14048. - Fix Version/s: 4.0.0 Resolution: Fixed > The Next Generation of the Consumer Rebalance Protocol > -- > > Key: KAFKA-14048 > URL: https://issues.apache.org/jira/browse/KAFKA-14048 > Project: Kafka > Issue Type: Improvement >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 4.0.0 > > > This Jira tracks the development of KIP-848: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17413) Re-introduce `group.version` feature flag
David Jacot created KAFKA-17413: --- Summary: Re-introduce `group.version` feature flag Key: KAFKA-17413 URL: https://issues.apache.org/jira/browse/KAFKA-17413 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot Fix For: 4.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16379) Coordinator flush time and event purgatory time metrics
[ https://issues.apache.org/jira/browse/KAFKA-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16379. - Fix Version/s: 4.0.0 Resolution: Fixed > Coordinator flush time and event purgatory time metrics > --- > > Key: KAFKA-16379 > URL: https://issues.apache.org/jira/browse/KAFKA-16379 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17279) Handle retriable errors from offset fetches in ConsumerCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-17279.
-
Fix Version/s: 3.9.0
Resolution: Fixed
> Handle retriable errors from offset fetches in ConsumerCoordinator
> --
>
> Key: KAFKA-17279
> URL: https://issues.apache.org/jira/browse/KAFKA-17279
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
>Reporter: Sean Quah
>Assignee: Sean Quah
>Priority: Minor
> Fix For: 3.9.0
>
>
> Currently {{ConsumerCoordinator}}'s {{OffsetFetchResponseHandler}} only
> retries on {{COORDINATOR_LOAD_IN_PROGRESS}} and {{NOT_COORDINATOR}} errors.
> The error handling should be expanded to retry on all retriable errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17383) Update upgrade notes about removal of `offsets.commit.required.acks`
David Jacot created KAFKA-17383: --- Summary: Update upgrade notes about removal of `offsets.commit.required.acks` Key: KAFKA-17383 URL: https://issues.apache.org/jira/browse/KAFKA-17383 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot Fix For: 4.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16503) getOrMaybeCreateClassicGroup should not thrown GroupIdNotFoundException
[ https://issues.apache.org/jira/browse/KAFKA-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16503. - Resolution: Fixed Addressed by https://github.com/apache/kafka/pull/16919. > getOrMaybeCreateClassicGroup should not thrown GroupIdNotFoundException > --- > > Key: KAFKA-16503 > URL: https://issues.apache.org/jira/browse/KAFKA-16503 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > > It looks like `getOrMaybeCreateClassicGroup` method throws an > `GroupIdNotFoundException` error when the group exists but with the wrong > type. As `getOrMaybeCreateClassicGroup` is mainly used on the > join-group/sync-group APIs, this seems incorrect. We need to double check and > fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17376) Use the new group coordinator by default in 4.0
David Jacot created KAFKA-17376: --- Summary: Use the new group coordinator by default in 4.0 Key: KAFKA-17376 URL: https://issues.apache.org/jira/browse/KAFKA-17376 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot Fix For: 4.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17295) New consumer fails with assert in consumer_test.py’s test_fencing_static_consumer system test
[
https://issues.apache.org/jira/browse/KAFKA-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-17295.
-
Assignee: Dongnuo Lyu
Resolution: Fixed
Fixed by https://github.com/apache/kafka/pull/16845.
> New consumer fails with assert in consumer_test.py’s
> test_fencing_static_consumer system test
> -
>
> Key: KAFKA-17295
> URL: https://issues.apache.org/jira/browse/KAFKA-17295
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, system tests
>Affects Versions: 3.8.0
>Reporter: Kirk True
>Assignee: Dongnuo Lyu
>Priority: Blocker
> Labels: kip-848-client-support, system-tests
> Fix For: 4.0.0
>
>
> I'm occasionally seeing this error in {{test_fencing_static_consumer}}:
> {code}
> AssertionError('Static consumers attempt to join with instance id in use
> should not cause a rebalance')
> Traceback (most recent call last):
> File
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
> line 184, in _do_run
> data = self.run_test()
> File
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
> line 262, in run_test
> return self.test_context.function(self.test)
> File
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
> line 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/client/consumer_test.py",
> line 366, in test_fencing_static_consumer
> assert num_rebalances == consumer.num_rebalances(), "Static consumers
> attempt to join with instance id in use should not cause a rebalance"
> AssertionError: Static consumers attempt to join with instance id in use
> should not cause a rebalance{code}
> The parameters to the test were:
> * {{num_conflict_consumers}}: {{1}}
> * {{fencing_stage}}: {{stable}}
> * {{metadata_quorum}}: {{ISOLATED_KRAFT}}
> * {{use_new_coordinator}}: {{True}}
> * {{group_protocol}}: {{consumer}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-17219) Adjust system test framework for new protocol consumer
[
https://issues.apache.org/jira/browse/KAFKA-17219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-17219.
-
Assignee: Dongnuo Lyu
Resolution: Fixed
Fixed by https://github.com/apache/kafka/pull/16845.
> Adjust system test framework for new protocol consumer
> --
>
> Key: KAFKA-17219
> URL: https://issues.apache.org/jira/browse/KAFKA-17219
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, system tests
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Blocker
> Labels: kip-848-client-support, system-tests
> Fix For: 4.0.0
>
>
> The current test framework doesn't work well with the existing tests using
> the new consumer protocol. There are two main issues I've seen.
>
> First, we sometimes assume there is no rebalance triggered, for instance in
> {{consumer_test.py::test_consumer_failure}}
> {code:java}
> verify that there were no rebalances on failover
> assert num_rebalances == consumer.num_rebalances(), "Broker failure should
> not cause a rebalance"{code}
> The current frame work calculates {{num_rebalances}} by increment by one
> every time a new assignment is received, so if a reconciliation happened
> during the failover, {{num_rebalances}} will also be incremented. For new
> protocol we need a new way to update {{{}num_rebalances{}}}.
>
> Second, for the new protocol, we need a way to make sure all members have
> joined {*}and stablized{*}. Currently we only make sure all members have
> joined (the event handlers are all in Joined state), where some partitions
> haven't been assigned and more time is needed for reconciliation. The issue
> can cause failure in assertions like timeout waiting for consumption and
> {code:java}
> partition_owner = consumer.owner(partition)
> assert partition_owner is not None {code}
>
> For a short term solution, we can make the tests pass by bypassing with
> adding {{time.sleep}} or skip checking {{{}num_rebalance{}}}. To truly fix
> them, we should adjust
> {{tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java}} to
> work well with the new protocol.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16576) New consumer fails with assert in consumer_test.py’s test_consumer_failure system test
[
https://issues.apache.org/jira/browse/KAFKA-16576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16576.
-
Assignee: Dongnuo Lyu
Resolution: Fixed
Fixed by https://github.com/apache/kafka/pull/16845.
> New consumer fails with assert in consumer_test.py’s test_consumer_failure
> system test
> --
>
> Key: KAFKA-16576
> URL: https://issues.apache.org/jira/browse/KAFKA-16576
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, system tests
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Dongnuo Lyu
>Priority: Blocker
> Labels: flaky-test, kip-848-client-support, system-tests
> Fix For: 4.0.0
>
>
> The {{consumer_test.py}} system test intermittently fails with the following
> error:
> {code}
> test_id:
> kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_failure.clean_shutdown=True.enable_autocommit=True.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
> status: FAIL
> run time: 42.582 seconds
> AssertionError()
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
> line 184, in _do_run
> data = self.run_test()
> File
> "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
> line 262, in run_test
> return self.test_context.function(self.test)
> File
> "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py",
> line 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/consumer_test.py",
> line 399, in test_consumer_failure
> assert partition_owner is not None
> AssertionError
> Notify
> {code}
> Affected tests:
> * {{test_consumer_failure}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14510) Extend DescribeConfigs API to support group configs
[ https://issues.apache.org/jira/browse/KAFKA-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14510. - Fix Version/s: 4.0.0 Resolution: Fixed > Extend DescribeConfigs API to support group configs > --- > > Key: KAFKA-14510 > URL: https://issues.apache.org/jira/browse/KAFKA-14510 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Lan Ding >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14511) Extend AlterIncrementalConfigs API to support group config
[ https://issues.apache.org/jira/browse/KAFKA-14511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14511. - Fix Version/s: 4.0.0 Resolution: Fixed > Extend AlterIncrementalConfigs API to support group config > -- > > Key: KAFKA-14511 > URL: https://issues.apache.org/jira/browse/KAFKA-14511 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Lan Ding >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17228) Static member using new protocol should always replace the one using the old protocol
[
https://issues.apache.org/jira/browse/KAFKA-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-17228.
-
Fix Version/s: 4.0.0
Reviewer: David Jacot
Resolution: Fixed
> Static member using new protocol should always replace the one using the old
> protocol
> -
>
> Key: KAFKA-17228
> URL: https://issues.apache.org/jira/browse/KAFKA-17228
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
> Fix For: 4.0.0
>
>
> {color:#172b4d}In the old protocol, when a static consumer shuts down, it
> [won't send explicit LeaveGroup
> request|https://github.com/apache/kafka/blob/010ab19b724ae011e85686ce47320f4f85d9a11f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1158-L1164].
> It's okay because the old protocol replaces the existing member whenever a
> new member with the same instance id joins.{color}
> {color:#172b4d}However in the new protocol, we [requires the existing member
> to send leave
> group|https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java#L2236-L2238]
> for a new static member to replace the existing one. The gap causes the
> upgraded new consumer unable to join the group in both online/offline
> upgrade.{color}
> {color:#172b4d}We should make the static member using new protocol replace
> the static member using old protocol regardless of whether the latter has
> left the group.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-17267) New group coordinator can return REQUEST_TIMED_OUT for OFFSET_FETCHes
[
https://issues.apache.org/jira/browse/KAFKA-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-17267.
-
Fix Version/s: 4.0.0
Reviewer: David Jacot
Assignee: Sean Quah
Resolution: Fixed
> New group coordinator can return REQUEST_TIMED_OUT for OFFSET_FETCHes
> -
>
> Key: KAFKA-17267
> URL: https://issues.apache.org/jira/browse/KAFKA-17267
> Project: Kafka
> Issue Type: Bug
> Components: group-coordinator
>Reporter: Sean Quah
>Assignee: Sean Quah
>Priority: Minor
> Fix For: 4.0.0
>
>
> Under some circumstances, the new group coordinator can return
> {{REQUEST_TIMED_OUT}} errors in response to {{OFFSET_FETCH}} requests.
> However, the client (ConsumerCoordinator) does not handle this error code and
> treats it as non-retryable. For compatibility with older clients, we can map
> {{REQUEST_TIMED_OUT}} to {{{}NOT_COORDINATOR{}}}.
>
> Similar to KAFKA-16386.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17298) Update upgrade notes for 4.0
David Jacot created KAFKA-17298: --- Summary: Update upgrade notes for 4.0 Key: KAFKA-17298 URL: https://issues.apache.org/jira/browse/KAFKA-17298 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot Fix For: 4.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16944) Range assignor doesn't co-partition with stickiness
[
https://issues.apache.org/jira/browse/KAFKA-16944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16944.
-
Fix Version/s: 3.9.0
Resolution: Fixed
> Range assignor doesn't co-partition with stickiness
> ---
>
> Key: KAFKA-16944
> URL: https://issues.apache.org/jira/browse/KAFKA-16944
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Ritika Reddy
>Assignee: Ritika Reddy
>Priority: Major
> Fix For: 3.9.0
>
>
> When stickiness is considered during range assignments, it is possible that
> in certain cases where co-partitioning is guaranteed we fail.
> An example would be:
> Consider two topics T1, T2 with 3 partitions each and three members A, B, C.
> Let's say the existing assignment (for whatever reason) is:
> {quote}A -> T1P0 || B -> T1P1, T2P0, T2P1, T2P2 || C -> T1P2
> {quote}
> Now we trigger a rebalance with the following subscriptions where all members
> are subscribed to both topics everything else is the same
> {quote}A -> T1, T2 || B -> T1, T2 || C -> T1, T2
> {quote}
> Since all the topics have an equal number of partitions and all the members
> are subscribed to the same set of topics we would expect co-partitioning
> right so would we want the final assignment returned to be
> {quote}A -> T1P0, T2P0 || B -> T1P1, T2P1 || C -> T1P2, T2P2
> {quote}
> SO currently the client side assignor returns the following but it's because
> they don't assign sticky partitions
> {{{}C=[topic1-2, topic2-2], B=[topic1-1, topic2-1], A=[topic1-0,
> topic2-0]{}}}Our
>
> Server side assignor returns:
> (The partitions in bold are the sticky partitions)
> {{{}A=MemberAssignment(targetPartitions={topic2=[1],
> }}\{{{}{*}topic1=[0]{*}{}}}{{{}}),
> B=MemberAssignment(targetPartitions={{}}}{{{}*topic2=[0]*{}}}{{{},
> {{{{{}*topic1=[1]*{}}}{{{}}),
> C=MemberAssignment(targetPartitions={topic2=[2], {{{{{}*topic1=[2]*{}}}
> *As seen above co-partitioning is expected but not returned.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-17058) Extend CoordinatorRuntime to support non-atomic writes
[ https://issues.apache.org/jira/browse/KAFKA-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17058. - Fix Version/s: 3.9.0 Resolution: Fixed > Extend CoordinatorRuntime to support non-atomic writes > -- > > Key: KAFKA-17058 > URL: https://issues.apache.org/jira/browse/KAFKA-17058 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17047) Refactor Consumer group and shared classes with Share to modern package
[ https://issues.apache.org/jira/browse/KAFKA-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17047. - Fix Version/s: 3.9.0 Resolution: Fixed > Refactor Consumer group and shared classes with Share to modern package > --- > > Key: KAFKA-17047 > URL: https://issues.apache.org/jira/browse/KAFKA-17047 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17050) Revert group.version for 3.8 and 3.9
[ https://issues.apache.org/jira/browse/KAFKA-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17050. - Fix Version/s: 3.8.0 Resolution: Fixed > Revert group.version for 3.8 and 3.9 > > > Key: KAFKA-17050 > URL: https://issues.apache.org/jira/browse/KAFKA-17050 > Project: Kafka > Issue Type: Task >Affects Versions: 3.8.0, 3.9.0 >Reporter: Justine Olshan >Assignee: Justine Olshan >Priority: Major > Fix For: 3.8.0 > > > After much discussion for KAFKA-17011, we decided it would be best for 3.8 to > just remove the group version feature for 3.8. > As for 3.9, [~dajac] said it would be easier for EA users of the group > coordinator to have a single way to configure. For 4.0 we can reintroduce it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17058) Extend CoordinatorRuntime to support non-atomic writes
David Jacot created KAFKA-17058: --- Summary: Extend CoordinatorRuntime to support non-atomic writes Key: KAFKA-17058 URL: https://issues.apache.org/jira/browse/KAFKA-17058 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16822) Abstract consumer group in coordinator to share functionality with share group
[ https://issues.apache.org/jira/browse/KAFKA-16822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16822. - Fix Version/s: 3.9.0 Resolution: Fixed > Abstract consumer group in coordinator to share functionality with share group > -- > > Key: KAFKA-16822 > URL: https://issues.apache.org/jira/browse/KAFKA-16822 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16973) Fix caught-up condition
[ https://issues.apache.org/jira/browse/KAFKA-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16973. - Fix Version/s: 3.9.0 Resolution: Fixed > Fix caught-up condition > --- > > Key: KAFKA-16973 > URL: https://issues.apache.org/jira/browse/KAFKA-16973 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.9.0 > > > When a write operation does not have any records, the coordinator runtime > checked whether the state machine is caught-up to decide whether the > operation should wait until the state machine is committed up to the > operation point or the operation should be completed. The current > implementation assumes that there will always be a pending write operation > waiting in the deferred queue when the state machine is not fully caught-up > yet. This is true except when the state machine is just loaded and not > caught-up yet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16673) Optimize toTopicPartitions with ConsumerProtocolSubscription
[ https://issues.apache.org/jira/browse/KAFKA-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16673. - Fix Version/s: 3.9.0 Resolution: Fixed > Optimize toTopicPartitions with ConsumerProtocolSubscription > > > Key: KAFKA-16673 > URL: https://issues.apache.org/jira/browse/KAFKA-16673 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.9.0 > > > https://github.com/apache/kafka/pull/15798#discussion_r1582981154 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16973) Fix caught-up condition
David Jacot created KAFKA-16973: --- Summary: Fix caught-up condition Key: KAFKA-16973 URL: https://issues.apache.org/jira/browse/KAFKA-16973 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot When a write operation does not have any records, the coordinator runtime checked whether the state machine is caught-up to decide whether the operation should wait until the state machine is committed up to the operation point or the operation should be completed. The current implementation assumes that there will always be a pending write operation waiting in the deferred queue when the state machine is not fully caught-up yet. This is true except when the state machine is just loaded and not caught-up yet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16317) Add event rate in GroupCoordinatorRuntimeMetrics
[ https://issues.apache.org/jira/browse/KAFKA-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16317. - Resolution: Won't Do > Add event rate in GroupCoordinatorRuntimeMetrics > > > Key: KAFKA-16317 > URL: https://issues.apache.org/jira/browse/KAFKA-16317 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > > We want a sensor to record every time we process a new event in the > coordinator runtime. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16674) Adjust classicGroupJoinToConsumerGroup to add subscription model
[ https://issues.apache.org/jira/browse/KAFKA-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16674. - Resolution: Fixed This was done in https://github.com/apache/kafka/pull/15785. > Adjust classicGroupJoinToConsumerGroup to add subscription model > > > Key: KAFKA-16674 > URL: https://issues.apache.org/jira/browse/KAFKA-16674 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > > [https://github.com/apache/kafka/pull/15785] adds subscription model to the > group state that affects `classicGroupJoinToConsumerGroup`. We'll need to > make adjustment to comply with the change once #15785 is merged. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-16673) Optimize toTopicPartitions with ConsumerProtocolSubscription
[ https://issues.apache.org/jira/browse/KAFKA-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot reopened KAFKA-16673: - > Optimize toTopicPartitions with ConsumerProtocolSubscription > > > Key: KAFKA-16673 > URL: https://issues.apache.org/jira/browse/KAFKA-16673 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > > https://github.com/apache/kafka/pull/15798#discussion_r1582981154 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16673) Optimize toTopicPartitions with ConsumerProtocolSubscription
[ https://issues.apache.org/jira/browse/KAFKA-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16673. - Fix Version/s: 3.8.0 Resolution: Duplicate This was done as part of https://github.com/apache/kafka/pull/15785. > Optimize toTopicPartitions with ConsumerProtocolSubscription > > > Key: KAFKA-16673 > URL: https://issues.apache.org/jira/browse/KAFKA-16673 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > > https://github.com/apache/kafka/pull/15798#discussion_r1582981154 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16770) Coalesce records into bigger batches
[ https://issues.apache.org/jira/browse/KAFKA-16770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16770. - Resolution: Fixed > Coalesce records into bigger batches > > > Key: KAFKA-16770 > URL: https://issues.apache.org/jira/browse/KAFKA-16770 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Blocker > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16930) UniformHeterogeneousAssignmentBuilder throws NPE when member has not subscriptions
[
https://issues.apache.org/jira/browse/KAFKA-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16930.
-
Resolution: Fixed
> UniformHeterogeneousAssignmentBuilder throws NPE when member has not
> subscriptions
> --
>
> Key: KAFKA-16930
> URL: https://issues.apache.org/jira/browse/KAFKA-16930
> Project: Kafka
> Issue Type: Sub-task
>Affects Versions: 3.7.0
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Blocker
> Fix For: 3.8.0
>
>
> {code:java}
> java.lang.NullPointerException: Cannot invoke
> "org.apache.kafka.coordinator.group.assignor.MemberAssignment.targetPartitions()"
> because the return value of "java.util.Map.get(Object)" is null
> at
> org.apache.kafka.coordinator.group.assignor.GeneralUniformAssignmentBuilder.canMemberParticipateInReassignment(GeneralUniformAssignmentBuilder.java:248)
> at
> org.apache.kafka.coordinator.group.assignor.GeneralUniformAssignmentBuilder.balance(GeneralUniformAssignmentBuilder.java:336)
> at
> org.apache.kafka.coordinator.group.assignor.GeneralUniformAssignmentBuilder.buildAssignment(GeneralUniformAssignmentBuilder.java:157)
> at
> org.apache.kafka.coordinator.group.assignor.UniformAssignor.assign(UniformAssignor.java:84)
> at
> org.apache.kafka.coordinator.group.consumer.TargetAssignmentBuilder.build(TargetAssignmentBuilder.java:302)
> at
> org.apache.kafka.coordinator.group.GroupMetadataManager.updateTargetAssignment(GroupMetadataManager.java:1913)
> at
> org.apache.kafka.coordinator.group.GroupMetadataManager.consumerGroupHeartbeat(GroupMetadataManager.java:1518)
> at
> org.apache.kafka.coordinator.group.GroupMetadataManager.consumerGroupHeartbeat(GroupMetadataManager.java:2254)
> at
> org.apache.kafka.coordinator.group.GroupCoordinatorShard.consumerGroupHeartbeat(GroupCoordinatorShard.java:308)
> at
> org.apache.kafka.coordinator.group.GroupCoordinatorService.lambda$consumerGroupHeartbeat$0(GroupCoordinatorService.java:298)
> at
> org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime$CoordinatorWriteEvent.lambda$run$0(CoordinatorRuntime.java:769)
> at
> org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime.withActiveContextOrThrow(CoordinatorRuntime.java:1582)
> at
> org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime.access$1400(CoordinatorRuntime.java:96)
> at
> org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime$CoordinatorWriteEvent.run(CoordinatorRuntime.java:767)
> at
> org.apache.kafka.coordinator.group.runtime.MultiThreadedEventProcessor$EventProcessorThread.handleEvents(MultiThreadedEventProcessor.java:144)
> at
> org.apache.kafka.coordinator.group.runtime.MultiThreadedEventProcessor$EventProcessorThread.run(MultiThreadedEventProcessor.java:176)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16930) UniformHeterogeneousAssignmentBuilder throws NPE when member has not subscriptions
David Jacot created KAFKA-16930:
---
Summary: UniformHeterogeneousAssignmentBuilder throws NPE when
member has not subscriptions
Key: KAFKA-16930
URL: https://issues.apache.org/jira/browse/KAFKA-16930
Project: Kafka
Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot
{code:java}
java.lang.NullPointerException: Cannot invoke
"org.apache.kafka.coordinator.group.assignor.MemberAssignment.targetPartitions()"
because the return value of "java.util.Map.get(Object)" is null
at
org.apache.kafka.coordinator.group.assignor.GeneralUniformAssignmentBuilder.canMemberParticipateInReassignment(GeneralUniformAssignmentBuilder.java:248)
at
org.apache.kafka.coordinator.group.assignor.GeneralUniformAssignmentBuilder.balance(GeneralUniformAssignmentBuilder.java:336)
at
org.apache.kafka.coordinator.group.assignor.GeneralUniformAssignmentBuilder.buildAssignment(GeneralUniformAssignmentBuilder.java:157)
at
org.apache.kafka.coordinator.group.assignor.UniformAssignor.assign(UniformAssignor.java:84)
at
org.apache.kafka.coordinator.group.consumer.TargetAssignmentBuilder.build(TargetAssignmentBuilder.java:302)
at
org.apache.kafka.coordinator.group.GroupMetadataManager.updateTargetAssignment(GroupMetadataManager.java:1913)
at
org.apache.kafka.coordinator.group.GroupMetadataManager.consumerGroupHeartbeat(GroupMetadataManager.java:1518)
at
org.apache.kafka.coordinator.group.GroupMetadataManager.consumerGroupHeartbeat(GroupMetadataManager.java:2254)
at
org.apache.kafka.coordinator.group.GroupCoordinatorShard.consumerGroupHeartbeat(GroupCoordinatorShard.java:308)
at
org.apache.kafka.coordinator.group.GroupCoordinatorService.lambda$consumerGroupHeartbeat$0(GroupCoordinatorService.java:298)
at
org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime$CoordinatorWriteEvent.lambda$run$0(CoordinatorRuntime.java:769)
at
org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime.withActiveContextOrThrow(CoordinatorRuntime.java:1582)
at
org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime.access$1400(CoordinatorRuntime.java:96)
at
org.apache.kafka.coordinator.group.runtime.CoordinatorRuntime$CoordinatorWriteEvent.run(CoordinatorRuntime.java:767)
at
org.apache.kafka.coordinator.group.runtime.MultiThreadedEventProcessor$EventProcessorThread.handleEvents(MultiThreadedEventProcessor.java:144)
at
org.apache.kafka.coordinator.group.runtime.MultiThreadedEventProcessor$EventProcessorThread.run(MultiThreadedEventProcessor.java:176)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16821) Create a new interface to store member metadata
[ https://issues.apache.org/jira/browse/KAFKA-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16821. - Fix Version/s: 3.8.0 Resolution: Fixed > Create a new interface to store member metadata > --- > > Key: KAFKA-16821 > URL: https://issues.apache.org/jira/browse/KAFKA-16821 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Major > Fix For: 3.8.0 > > Attachments: Screenshot 2024-05-14 at 11.03.10 AM.png > > > !Screenshot 2024-05-14 at 11.03.10 AM.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14509) Add ConsumerGroupDescribe API
[ https://issues.apache.org/jira/browse/KAFKA-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14509. - Fix Version/s: 3.8.0 Resolution: Fixed > Add ConsumerGroupDescribe API > - > > Key: KAFKA-14509 > URL: https://issues.apache.org/jira/browse/KAFKA-14509 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Max Riedel >Priority: Major > Labels: kip-848-preview > Fix For: 3.8.0 > > > The goal of this task is to implement the ConsumerGroupDescribe API as > described > [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupDescribeAPI]; > and to implement the related changes in the admin client as described > [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-Admin#describeConsumerGroups]. > On the server side, this mainly requires the following steps: > # The request/response schemas must be defined (see > ListGroupsRequest/Response.json for an example); > # Request/response classes must be defined (see > ListGroupsRequest/Response.java for an example); > # The API must be defined in KafkaApis (see > KafkaApis#handleDescribeGroupsRequest for an example); > # The GroupCoordinator interface (java file) must be extended for the new > operations. > # The new operation must be implemented in GroupCoordinatorService (new > coordinator in Java) whereas the GroupCoordinatorAdapter (old coordinator in > Scala) should just reject the request. > We could probably do 1) and 2) in one pull request and the remaining ones in > another. > On the admin client side, this mainly requires the followings steps: > * Define all the new java classes as defined in the KIP. > * Add the new API to KafkaAdminClient class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14701) More broker side partition assignor to common
[ https://issues.apache.org/jira/browse/KAFKA-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14701. - Fix Version/s: 3.8.0 Resolution: Fixed > More broker side partition assignor to common > - > > Key: KAFKA-14701 > URL: https://issues.apache.org/jira/browse/KAFKA-14701 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 3.8.0 >Reporter: David Jacot >Assignee: David Jacot >Priority: Blocker > Fix For: 3.8.0 > > > Before releasing KIP-848, we need to move the server side partition assignor > to its final location in common. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16664) Re-add EventAccumulator.take(timeout)
[ https://issues.apache.org/jira/browse/KAFKA-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16664. - Fix Version/s: 3.8.0 Resolution: Fixed > Re-add EventAccumulator.take(timeout) > - > > Key: KAFKA-16664 > URL: https://issues.apache.org/jira/browse/KAFKA-16664 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > Fix For: 3.8.0 > > > [https://github.com/apache/kafka/pull/15835] should be used with a timeout in > EventAccumulator#take. We added a commit to remove the timeout, we should > revert it -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16861) Don't convert to group to classic if the size is larger than group max size
[ https://issues.apache.org/jira/browse/KAFKA-16861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16861. - Fix Version/s: 3.8.0 Resolution: Fixed > Don't convert to group to classic if the size is larger than group max size > --- > > Key: KAFKA-16861 > URL: https://issues.apache.org/jira/browse/KAFKA-16861 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: TengYao Chi >Priority: Major > Fix For: 3.8.0 > > > It should be one-line fix [0] > [0] > https://github.com/apache/kafka/blob/2d9994e0de915037525f041ff9a9b9325f838938/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java#L810 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16864) Copy on write in the Optimized Uniform Assignor
[ https://issues.apache.org/jira/browse/KAFKA-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16864. - Fix Version/s: 3.8.0 Resolution: Fixed > Copy on write in the Optimized Uniform Assignor > --- > > Key: KAFKA-16864 > URL: https://issues.apache.org/jira/browse/KAFKA-16864 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > > An optimization for the uniform (homogenous) assignor by avoiding creating a > copy of all the assignments. Instead, the assignor creates a copy only if the > assignment is updated. It is a sort of copy-on-write. This change reduces the > overhead of the TargetAssignmentBuilder when ran with the uniform > (homogenous) assignor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16860) Introduce `group.version` feature flag
[ https://issues.apache.org/jira/browse/KAFKA-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16860. - Resolution: Fixed > Introduce `group.version` feature flag > -- > > Key: KAFKA-16860 > URL: https://issues.apache.org/jira/browse/KAFKA-16860 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Blocker > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16722) Add ConsumerGroupPartitionAssignor interface
[ https://issues.apache.org/jira/browse/KAFKA-16722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16722. - Reviewer: David Jacot Resolution: Fixed > Add ConsumerGroupPartitionAssignor interface > > > Key: KAFKA-16722 > URL: https://issues.apache.org/jira/browse/KAFKA-16722 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.8.0 > > > Adds the interface > `org.apache.kafka.coordinator.group.assignor.ConsumerGroupPartitionAssignor` > as described in KIP-932. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16569) Target Assignment Format Change
[ https://issues.apache.org/jira/browse/KAFKA-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16569. - Resolution: Won't Do > Target Assignment Format Change > --- > > Key: KAFKA-16569 > URL: https://issues.apache.org/jira/browse/KAFKA-16569 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Major > > Currently the assignment is stored as Map>, we > want to change it to a list > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16832) LeaveGroup API for upgrading ConsumerGroup
[ https://issues.apache.org/jira/browse/KAFKA-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16832. - Fix Version/s: 3.8.0 Resolution: Fixed > LeaveGroup API for upgrading ConsumerGroup > -- > > Key: KAFKA-16832 > URL: https://issues.apache.org/jira/browse/KAFKA-16832 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16860) Introduce `group.version` feature flag
David Jacot created KAFKA-16860: --- Summary: Introduce `group.version` feature flag Key: KAFKA-16860 URL: https://issues.apache.org/jira/browse/KAFKA-16860 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot Fix For: 3.8.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16371) Unstable committed offsets after triggering commits where metadata for some partitions are over the limit
[
https://issues.apache.org/jira/browse/KAFKA-16371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16371.
-
Fix Version/s: 3.8.0
3.7.1
Assignee: David Jacot
Resolution: Fixed
> Unstable committed offsets after triggering commits where metadata for some
> partitions are over the limit
> -
>
> Key: KAFKA-16371
> URL: https://issues.apache.org/jira/browse/KAFKA-16371
> Project: Kafka
> Issue Type: Bug
> Components: offset manager
>Affects Versions: 3.7.0
>Reporter: mlowicki
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.8.0, 3.7.1
>
>
> Issue is reproducible with simple CLI tool -
> [https://gist.github.com/mlowicki/c3b942f5545faced93dc414e01a2da70]
> {code:java}
> #!/usr/bin/env bash
> for i in {1..100}
> do
> kafka-committer --bootstrap "ADDR:9092" --topic "TOPIC" --group foo
> --metadata-min 6000 --metadata-max 1 --partitions 72 --fetch
> done{code}
> What it does it that initially it fetches committed offsets and then tries to
> commit for multiple partitions. If some of commits have metadata over the
> allowed limit then:
> 1. I see errors about too large commits (expected)
> 2. Another run the tool fails at the stage of fetching commits with (this is
> the problem):
> {code:java}
> config: ClientConfig { conf_map: { "group.id": "bar", "bootstrap.servers":
> "ADDR:9092", }, log_level: Error, }
> fetching committed offsets..
> Error: Meta data fetch error: OperationTimedOut (Local: Timed out) Caused by:
> OperationTimedOut (Local: Timed out){code}
> On the Kafka side I see _unstable_offset_commits_ errors reported by out
> internal metric which is derived from:
> {noformat}
>
> kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=X,error=Y{noformat}
> Increasing the timeout doesn't help and the only solution I've found is to
> trigger commits for all partitions with metadata below the limit or to use:
> {code:java}
> isolation.level=read_uncommitted{code}
>
> I don't know that code very well but
> [https://github.com/apache/kafka/blob/3.7/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L492-L496]
> seems fishy:
> {code:java}
> if (isTxnOffsetCommit) {
> addProducerGroup(producerId, group.groupId)
> group.prepareTxnOffsetCommit(producerId, offsetMetadata)
> } else {
> group.prepareOffsetCommit(offsetMetadata)
> }{code}
> as it's using _offsetMetadata_ and not _filteredOffsetMetadata_ and I see
> that while removing those pending commits we use filtered offset metadata
> around
> [https://github.com/apache/kafka/blob/3.7/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L397-L422]
>
> {code:java}
> val responseError = group.inLock {
> if (status.error == Errors.NONE) {
> if (!group.is(Dead)) {
> filteredOffsetMetadata.forKeyValue { (topicIdPartition,
> offsetAndMetadata) =>
> if (isTxnOffsetCommit)
> group.onTxnOffsetCommitAppend(producerId, topicIdPartition,
> CommitRecordMetadataAndOffset(Some(status.baseOffset), offsetAndMetadata))
> else
> group.onOffsetCommitAppend(topicIdPartition,
> CommitRecordMetadataAndOffset(Some(status.baseOffset), offsetAndMetadata))
> }
> }
> // Record the number of offsets committed to the log
> offsetCommitsSensor.record(records.size)
> Errors.NONE
> } else {
> if (!group.is(Dead)) {
> if (!group.hasPendingOffsetCommitsFromProducer(producerId))
> removeProducerGroup(producerId, group.groupId)
> filteredOffsetMetadata.forKeyValue { (topicIdPartition,
> offsetAndMetadata) =>
> if (isTxnOffsetCommit)
> group.failPendingTxnOffsetCommit(producerId, topicIdPartition)
> else
> group.failPendingOffsetWrite(topicIdPartition,
> offsetAndMetadata)
> }
> }
> {code}
> so the problem might be related to not cleaning up the data structure with
> pending commits properly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16846) Should TxnOffsetCommit API fail all the offsets if any fails the validation?
David Jacot created KAFKA-16846: --- Summary: Should TxnOffsetCommit API fail all the offsets if any fails the validation? Key: KAFKA-16846 URL: https://issues.apache.org/jira/browse/KAFKA-16846 Project: Kafka Issue Type: Improvement Reporter: David Jacot While working on KAFKA-16371, we realized that the handling of INVALID_COMMIT_OFFSET_SIZE errors while committer transaction offsets, is a bit inconsistent between the server and the client. On the server, the offsets are validated independently from each others. Hence if two offsets A and B are committed and A fails the validation, B is still written to the log as part of the transaction. On the client, when INVALID_COMMIT_OFFSET_SIZE is received, the transaction transitions to the fatal state. Hence the transaction will be eventually aborted. The client side API is quite limiting here because it does not return an error per committed offsets. It is all or nothing. From this point of view, the current behaviour is correct. It seems that we could either change the API and let the user decide what to do; or we could strengthen the validation on the server to fail all the offsets if any of them fails (all or nothing). We could also leave it as it is. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16625) Reverse Lookup Partition to Member in Assignors
[ https://issues.apache.org/jira/browse/KAFKA-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16625. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > Reverse Lookup Partition to Member in Assignors > --- > > Key: KAFKA-16625 > URL: https://issues.apache.org/jira/browse/KAFKA-16625 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Major > Fix For: 3.8.0 > > > Calculating unassigned partitions within the Uniform assignor is a costly > process, this can be improved by using a reverse lookup map between > topicIdPartition and the member -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16831) CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit
[ https://issues.apache.org/jira/browse/KAFKA-16831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16831. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size > write limit > - > > Key: KAFKA-16831 > URL: https://issues.apache.org/jira/browse/KAFKA-16831 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > Fix For: 3.8.0 > > > Otherwise, we default to the min buffer size of 16384 for the write limit. > This causes the coordinator to threw RecordTooLargeException even when it's > under the 1MB max batch size limit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16815) Handle FencedInstanceId on heartbeat for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16815. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > Handle FencedInstanceId on heartbeat for new consumer > - > > Key: KAFKA-16815 > URL: https://issues.apache.org/jira/browse/KAFKA-16815 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Major > Labels: kip-848-client-support > Fix For: 3.8.0 > > > With the new consumer group protocol, a member could receive a > FencedInstanceIdError in the heartbeat response. This could be the case when > an active member using a group instance id is removed from the group by an > admin client. If a second member joins with the same instance id, the first > member will receive a FencedInstanceId on the next heartbeat response. This > should be treated as a fatal error (consumer should not attempt to rejoin). > Currently, the FencedInstanceId is not explicitly handled by the client in > the HeartbeatRequestManager. It ends up being treated as a fatal error, see > [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L417] > (just because it lands on the "unexpected" error category). We should handle > it explicitly, just to make sure that we express that it's is an expected > error: log a proper message for it and fail (handleFatalFailure). We should > also that the error is included in the tests that cover the HB request error > handling > ([here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java#L798]) > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16626) Uuid to String for subscribed topic names in assignment spec
[ https://issues.apache.org/jira/browse/KAFKA-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16626. - Fix Version/s: 3.8.0 Resolution: Fixed > Uuid to String for subscribed topic names in assignment spec > > > Key: KAFKA-16626 > URL: https://issues.apache.org/jira/browse/KAFKA-16626 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Jeff Kim >Priority: Major > Fix For: 3.8.0 > > > In creating the assignment spec from the existing consumer subscription > metadata, quite some time is spent in converting the String to a Uuid > Change from Uuid to String for the subscribed topics in assignment spec and > convert on the fly -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16793) Heartbeat API for upgrading ConsumerGroup
[ https://issues.apache.org/jira/browse/KAFKA-16793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16793. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > Heartbeat API for upgrading ConsumerGroup > - > > Key: KAFKA-16793 > URL: https://issues.apache.org/jira/browse/KAFKA-16793 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16762) SyncGroup API for upgrading ConsumerGroup
[ https://issues.apache.org/jira/browse/KAFKA-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16762. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > SyncGroup API for upgrading ConsumerGroup > - > > Key: KAFKA-16762 > URL: https://issues.apache.org/jira/browse/KAFKA-16762 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16770) Coalesce records into bigger batches
David Jacot created KAFKA-16770: --- Summary: Coalesce records into bigger batches Key: KAFKA-16770 URL: https://issues.apache.org/jira/browse/KAFKA-16770 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot Fix For: 3.8.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16694) Remove rack aware code in assignors temporarily due to performance
[ https://issues.apache.org/jira/browse/KAFKA-16694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16694. - Fix Version/s: 3.8.0 Resolution: Fixed > Remove rack aware code in assignors temporarily due to performance > -- > > Key: KAFKA-16694 > URL: https://issues.apache.org/jira/browse/KAFKA-16694 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15578) Run System Tests for Old protocol in the New Coordinator
[ https://issues.apache.org/jira/browse/KAFKA-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15578. - Resolution: Fixed > Run System Tests for Old protocol in the New Coordinator > > > Key: KAFKA-15578 > URL: https://issues.apache.org/jira/browse/KAFKA-15578 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Major > Labels: kip-848-preview > Original Estimate: 672h > Remaining Estimate: 672h > > Change existing system tests related to the consumer group protocol and group > coordinator to test the old protocol running with the new coordinator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16117) Add Integration test for checking if the correct assignor is chosen
[
https://issues.apache.org/jira/browse/KAFKA-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16117.
-
Fix Version/s: 3.8.0
Resolution: Fixed
> Add Integration test for checking if the correct assignor is chosen
> ---
>
> Key: KAFKA-16117
> URL: https://issues.apache.org/jira/browse/KAFKA-16117
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Ritika Reddy
>Priority: Minor
> Fix For: 3.8.0
>
>
> h4. We are trying to test this section of the KIP-848
> h4. Assignor Selection
> The group coordinator has to determine which assignment strategy must be used
> for the group. The group's members may not have exactly the same assignors at
> any given point in time - e.g. they may migrate from an assignor to another
> one for instance. The group coordinator will chose the assignor as follow:
> * A client side assignor is used if possible. This means that a client side
> assignor must be supported by all the members. If multiple are, it will
> respect the precedence defined by the members when they advertise their
> supported client side assignors.
> * A server side assignor is used otherwise. If multiple server side
> assignors are specified in the group, the group coordinator uses the most
> common one. If a member does not provide an assignor, the group coordinator
> will default to the first one in {{{}group.consumer.assignors{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16735) Deprecate offsets.commit.required.acks in 3.8
[ https://issues.apache.org/jira/browse/KAFKA-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16735. - Resolution: Fixed > Deprecate offsets.commit.required.acks in 3.8 > - > > Key: KAFKA-16735 > URL: https://issues.apache.org/jira/browse/KAFKA-16735 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Blocker > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16736) Remove offsets.commit.required.acks in 4.0
David Jacot created KAFKA-16736: --- Summary: Remove offsets.commit.required.acks in 4.0 Key: KAFKA-16736 URL: https://issues.apache.org/jira/browse/KAFKA-16736 Project: Kafka Issue Type: Sub-task Affects Versions: 4.0.0 Reporter: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16735) Deprecate offsets.commit.required.acks in 3.8
David Jacot created KAFKA-16735: --- Summary: Deprecate offsets.commit.required.acks in 3.8 Key: KAFKA-16735 URL: https://issues.apache.org/jira/browse/KAFKA-16735 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16587) Store subscription model for consumer group in group state
[ https://issues.apache.org/jira/browse/KAFKA-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16587. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > Store subscription model for consumer group in group state > -- > > Key: KAFKA-16587 > URL: https://issues.apache.org/jira/browse/KAFKA-16587 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Major > Fix For: 3.8.0 > > > Currently we iterate through all the subscribed topics for each member in the > consumer group to determine whether all the members are subscribed to the > same set of topics aka it has a homogeneous subscription model. > Instead of iterating and comparing the topicIds on every rebalance, we want > to maintain this information in the group state -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16663) CoordinatorRuntime write timer tasks should be cancelled once HWM advances
[ https://issues.apache.org/jira/browse/KAFKA-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16663. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > CoordinatorRuntime write timer tasks should be cancelled once HWM advances > -- > > Key: KAFKA-16663 > URL: https://issues.apache.org/jira/browse/KAFKA-16663 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > Fix For: 3.8.0 > > > Otherwise, we pile up the number of timer tasks which are no-ops if > replication was successful. They stay in memory for 15 seconds and as the > rate of write increases, this may heavily impact memory usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16307) fix EventAccumulator thread idle ratio metric
[ https://issues.apache.org/jira/browse/KAFKA-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16307. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > fix EventAccumulator thread idle ratio metric > - > > Key: KAFKA-16307 > URL: https://issues.apache.org/jira/browse/KAFKA-16307 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > Fix For: 3.8.0 > > > The metric does not seem to be accurate, nor reporting metrics at every > interval. Requires investigation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16615) JoinGroup API for upgrading ConsumerGroup
[ https://issues.apache.org/jira/browse/KAFKA-16615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16615. - Fix Version/s: 3.8.0 Reviewer: David Jacot Assignee: Dongnuo Lyu Resolution: Fixed > JoinGroup API for upgrading ConsumerGroup > - > > Key: KAFKA-16615 > URL: https://issues.apache.org/jira/browse/KAFKA-16615 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16658) Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)
David Jacot created KAFKA-16658: --- Summary: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8) Key: KAFKA-16658 URL: https://issues.apache.org/jira/browse/KAFKA-16658 Project: Kafka Issue Type: New Feature Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16568) Add JMH Benchmarks for assignor performance testing
[ https://issues.apache.org/jira/browse/KAFKA-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16568. - Fix Version/s: 3.8.0 Resolution: Fixed > Add JMH Benchmarks for assignor performance testing > > > Key: KAFKA-16568 > URL: https://issues.apache.org/jira/browse/KAFKA-16568 > Project: Kafka > Issue Type: Sub-task >Reporter: Ritika Reddy >Assignee: Ritika Reddy >Priority: Major > Fix For: 3.8.0 > > > The 3 benchmarks that are being used to test the performance and efficiency > of the consumer group rebalance process. > * Client Assignors (assign method) > * Server Assignors (assign method) > * Target Assignment Builder (build method) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16294) Add group protocol migration enabling config
[ https://issues.apache.org/jira/browse/KAFKA-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16294. - Fix Version/s: 3.8.0 Resolution: Fixed > Add group protocol migration enabling config > > > Key: KAFKA-16294 > URL: https://issues.apache.org/jira/browse/KAFKA-16294 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > > The online upgrade is triggered when a consumer group heartbeat request is > received in a classic group. The downgrade is triggered when any old protocol > request is received in a consumer group. We only accept upgrade/downgrade if > the corresponding group migration config policy is enabled. > This is the first part of the implementation of online group protocol > migration, adding the kafka config group protocol migration. The config has > four valid values – both(both upgrade and downgrade are allowed), > upgrade(only upgrade is allowed), downgrade(only downgrade is allowed) and > none(neither is allowed.). > At present the default value is NONE. When we start enabling the migration, > we expect to set BOTH to default so that it's easier to roll back to the old > protocol as a quick fix for anything wrong in the new protocol; when using > consumer groups becomes default and the migration is near finished, we will > set the default policy to UPGRADE to prevent unwanted downgrade causing too > frequent migration. DOWNGRADE could be useful for revert or debug purposes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16503) getOrMaybeCreateClassicGroup should not thrown GroupIdNotFoundException
David Jacot created KAFKA-16503: --- Summary: getOrMaybeCreateClassicGroup should not thrown GroupIdNotFoundException Key: KAFKA-16503 URL: https://issues.apache.org/jira/browse/KAFKA-16503 Project: Kafka Issue Type: Sub-task Reporter: David Jacot It looks like `getOrMaybeCreateClassicGroup` method throws an `GroupIdNotFoundException` error when the group exists but with the wrong type. As `getOrMaybeCreateClassicGroup` is mainly used on the join-group/sync-group APIs, this seems incorrect. We need to double check and fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16470) kafka-dump-log --offsets-decoder should support new records
David Jacot created KAFKA-16470: --- Summary: kafka-dump-log --offsets-decoder should support new records Key: KAFKA-16470 URL: https://issues.apache.org/jira/browse/KAFKA-16470 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16148) Implement GroupMetadataManager#onUnloaded
[ https://issues.apache.org/jira/browse/KAFKA-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16148. - Fix Version/s: 3.8.0 Resolution: Fixed > Implement GroupMetadataManager#onUnloaded > - > > Key: KAFKA-16148 > URL: https://issues.apache.org/jira/browse/KAFKA-16148 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > Fix For: 3.8.0 > > > complete all awaiting futures with NOT_COORDINATOR (for classic group) > transition all groups to DEAD. > Cancel all timers related to the unloaded group metadata manager -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16353) Offline protocol migration integration tests
[ https://issues.apache.org/jira/browse/KAFKA-16353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16353. - Resolution: Fixed > Offline protocol migration integration tests > > > Key: KAFKA-16353 > URL: https://issues.apache.org/jira/browse/KAFKA-16353 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16374) High watermark updates should have a higher priority
[ https://issues.apache.org/jira/browse/KAFKA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16374. - Fix Version/s: 3.8.0 Resolution: Fixed > High watermark updates should have a higher priority > > > Key: KAFKA-16374 > URL: https://issues.apache.org/jira/browse/KAFKA-16374 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15989) Upgrade existing generic group to consumer group
[ https://issues.apache.org/jira/browse/KAFKA-15989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15989. - Fix Version/s: 3.8.0 Resolution: Fixed > Upgrade existing generic group to consumer group > > > Key: KAFKA-15989 > URL: https://issues.apache.org/jira/browse/KAFKA-15989 > Project: Kafka > Issue Type: Sub-task >Reporter: Emanuele Sabellico >Assignee: David Jacot >Priority: Minor > Fix For: 3.8.0 > > > It should be possible to upgrade an existing generic group to a new consumer > group, in case it was using either the previous generic protocol or manual > partition assignment and commit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15763) Group Coordinator should not deliver new assignment before previous one is acknowledged
[ https://issues.apache.org/jira/browse/KAFKA-15763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15763. - Resolution: Won't Fix We went with another approach. > Group Coordinator should not deliver new assignment before previous one is > acknowledged > --- > > Key: KAFKA-15763 > URL: https://issues.apache.org/jira/browse/KAFKA-15763 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > > In the initial implementation of the new consumer group protocol, the group > coordinators waits on received an acknowledgement from the consumer only when > there are partitions to be revoked. In the case of newly assigned partitions, > a new assignment can be delivered any time (e.g. in two subsequent > heartbeats). > While implementing the state machine on the client side, we found out that > this caused confusion because the protocol does not treat revocation and > assignment in the same way. We also found out that changing the assignment > before the previous one is fully processed by the member makes the client > side logic more complicated than it should be because the consumer can't > process any new assignment until it has completed the previous one. > In the end, it is better to change the server side to not deliver a new > assignment before the current one is acknowledged by the consumer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16313) Offline group protocol migration
[ https://issues.apache.org/jira/browse/KAFKA-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16313. - Fix Version/s: 3.8.0 Assignee: Dongnuo Lyu Resolution: Fixed > Offline group protocol migration > > > Key: KAFKA-16313 > URL: https://issues.apache.org/jira/browse/KAFKA-16313 > Project: Kafka > Issue Type: Sub-task >Reporter: Dongnuo Lyu >Assignee: Dongnuo Lyu >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16367) Full ConsumerGroupHeartbeat response must be sent when full request is received
[ https://issues.apache.org/jira/browse/KAFKA-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16367. - Fix Version/s: 3.8.0 Resolution: Fixed > Full ConsumerGroupHeartbeat response must be sent when full request is > received > --- > > Key: KAFKA-16367 > URL: https://issues.apache.org/jira/browse/KAFKA-16367 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16374) High watermark updates should have a higher priority
David Jacot created KAFKA-16374: --- Summary: High watermark updates should have a higher priority Key: KAFKA-16374 URL: https://issues.apache.org/jira/browse/KAFKA-16374 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15997) Ensure fairness in the uniform assignor
[
https://issues.apache.org/jira/browse/KAFKA-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-15997.
-
Resolution: Fixed
This issue got resolved by https://issues.apache.org/jira/browse/KAFKA-16249.
> Ensure fairness in the uniform assignor
> ---
>
> Key: KAFKA-15997
> URL: https://issues.apache.org/jira/browse/KAFKA-15997
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Emanuele Sabellico
>Assignee: David Jacot
>Priority: Minor
>
>
>
> Fairness has to be ensured in uniform assignor as it was in
> cooperative-sticky one.
> There's this test 0113 subtest u_multiple_subscription_changes in librdkafka
> where 8 consumers are subscribing to the same topic, and it's verifying that
> all of them are getting 2 partitions assigned. But with new protocol it seems
> two consumers get assigned 3 partitions and 1 has zero partitions. The test
> doesn't configure any client.rack.
> {code:java}
> [0113_cooperative_rebalance /478.183s] Consumer assignments
> (subscription_variation 0) (stabilized) (no rebalance cb):
> [0113_cooperative_rebalance /478.183s] Consumer C_0#consumer-3 assignment
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [5] (2000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [8] (4000msgs)
> [0113_cooperative_rebalance /478.183s] Consumer C_1#consumer-4 assignment
> (3): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [0] (1000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [3] (2000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [13] (1000msgs)
> [0113_cooperative_rebalance /478.184s] Consumer C_2#consumer-5 assignment
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [6] (1000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [10] (2000msgs)
> [0113_cooperative_rebalance /478.184s] Consumer C_3#consumer-6 assignment
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [7] (1000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [9] (2000msgs)
> [0113_cooperative_rebalance /478.184s] Consumer C_4#consumer-7 assignment
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [11] (1000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [14] (3000msgs)
> [0113_cooperative_rebalance /478.184s] Consumer C_5#consumer-8 assignment
> (3): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [1] (2000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [2] (2000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [4] (1000msgs)
> [0113_cooperative_rebalance /478.184s] Consumer C_6#consumer-9 assignment
> (0):
> [0113_cooperative_rebalance /478.184s] Consumer C_7#consumer-10 assignment
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [12] (1000msgs),
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [15] (1000msgs)
> [0113_cooperative_rebalance /478.184s] 16/32 partitions assigned
> [0113_cooperative_rebalance /478.184s] Consumer C_0#consumer-3 has 2
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_1#consumer-4 has 3
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_2#consumer-5 has 2
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_3#consumer-6 has 2
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_4#consumer-7 has 2
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_5#consumer-8 has 3
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_6#consumer-9 has 0
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance /478.184s] Consumer C_7#consumer-10 has 2
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [ /479.057s] 1 test(s) running:
> 0113_cooperative_rebalance
> [ /480.057s] 1 test(s) running:
> 0113_cooperative_rebalance
> [ /481.057s] 1 test(s) running:
> 0113_cooperative_rebalance
> [0113_cooperative_rebalance /482.498s] TEST FAILURE
> ### Test "0113_cooperative_rebalance (u_multiple_subscription_changes:2390:
> use_rebalance_cb: 0, subscription_variation: 0)" failed at
> test.c:1243:check_test_timeouts() at Thu Dec 7 15:52:15 2023: ###
> Test 0113_cooperative_rebalance (u_multiple_subscription_changes:2390:
> use_rebalance_cb: 0, subscription_variation: 0) timed out (timeout set to 480
> seconds)
> ./run-test.sh: line 62: 3512920 Killed $TEST $ARGS
> ###
> ### Test ./test-runner in bare mode FAILED! (return code 137) ###
> ###{code}
>
>
-
[jira] [Resolved] (KAFKA-16249) Improve reconciliation state machine
[ https://issues.apache.org/jira/browse/KAFKA-16249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16249. - Fix Version/s: 3.8.0 Resolution: Fixed > Improve reconciliation state machine > > > Key: KAFKA-16249 > URL: https://issues.apache.org/jira/browse/KAFKA-16249 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16367) Full ConsumerGroupHeartbeat response must be sent when full request is received
David Jacot created KAFKA-16367: --- Summary: Full ConsumerGroupHeartbeat response must be sent when full request is received Key: KAFKA-16367 URL: https://issues.apache.org/jira/browse/KAFKA-16367 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15462) Add group type filter to the admin client
[ https://issues.apache.org/jira/browse/KAFKA-15462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15462. - Fix Version/s: 3.8.0 Resolution: Fixed > Add group type filter to the admin client > - > > Key: KAFKA-15462 > URL: https://issues.apache.org/jira/browse/KAFKA-15462 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Ritika Reddy >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16306) GroupCoordinatorService logger is not configured
[
https://issues.apache.org/jira/browse/KAFKA-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16306.
-
Fix Version/s: 3.8.0
Resolution: Fixed
> GroupCoordinatorService logger is not configured
>
>
> Key: KAFKA-16306
> URL: https://issues.apache.org/jira/browse/KAFKA-16306
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Minor
> Fix For: 3.8.0
>
>
> The GroupCoordinatorService constructor initializes with the wrong logger
> class:
> ```
> GroupCoordinatorService(
> LogContext logContext,
> GroupCoordinatorConfig config,
> CoordinatorRuntime runtime,
> GroupCoordinatorMetrics groupCoordinatorMetrics
> ) {
> this.log = logContext.logger(CoordinatorLoader.class);
> this.config = config;
> this.runtime = runtime;
> this.groupCoordinatorMetrics = groupCoordinatorMetrics;
> }
> ```
> change this to GroupCoordinatorService.class
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16249) Improve reconciliation state machine
David Jacot created KAFKA-16249: --- Summary: Improve reconciliation state machine Key: KAFKA-16249 URL: https://issues.apache.org/jira/browse/KAFKA-16249 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16244) Move code style exceptions from suppressions.xml to the code
David Jacot created KAFKA-16244: --- Summary: Move code style exceptions from suppressions.xml to the code Key: KAFKA-16244 URL: https://issues.apache.org/jira/browse/KAFKA-16244 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16178) AsyncKafkaConsumer doesn't retry joining the group after rediscovering group coordinator
[
https://issues.apache.org/jira/browse/KAFKA-16178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16178.
-
Resolution: Fixed
> AsyncKafkaConsumer doesn't retry joining the group after rediscovering group
> coordinator
>
>
> Key: KAFKA-16178
> URL: https://issues.apache.org/jira/browse/KAFKA-16178
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Reporter: Dongnuo Lyu
>Assignee: Lianet Magrans
>Priority: Blocker
> Labels: client-transitions-issues, consumer-threading-refactor
> Fix For: 3.8.0
>
> Attachments: pkc-devc63jwnj_jan19_0_debug
>
>
> {code:java}
> [2024-01-17 21:34:59,500] INFO [Consumer
> clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0,
> groupId=consumer-groups-test-0] Discovered group coordinator
> Coordinator(key='consumer-groups-test-0', nodeId=3,
> host='b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud', port=9092,
> errorCode=0, errorMessage='')
> (org.apache.kafka.clients.consumer.internals.CoordinatorRequestManager:162)
> [2024-01-17 21:34:59,681] INFO [Consumer
> clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0,
> groupId=consumer-groups-test-0] GroupHeartbeatRequest failed because the
> group coordinator
> Optional[b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud:9092 (id:
> 2147483644 rack: null)] is incorrect. Will attempt to find the coordinator
> again and retry in 0ms: This is not the correct coordinator.
> (org.apache.kafka.clients.consumer.internals.HeartbeatRequestManager:407)
> [2024-01-17 21:34:59,681] INFO [Consumer
> clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0,
> groupId=consumer-groups-test-0] Group coordinator
> b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud:9092 (id: 2147483644 rack:
> null) is unavailable or invalid due to cause: This is not the correct
> coordinator.. Rediscovery will be attempted.
> (org.apache.kafka.clients.consumer.internals.CoordinatorRequestManager:136)
> [2024-01-17 21:34:59,882] INFO [Consumer
> clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0,
> groupId=consumer-groups-test-0] Discovered group coordinator
> Coordinator(key='consumer-groups-test-0', nodeId=3,
> host='b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud', port=9092,
> errorCode=0, errorMessage='')
> (org.apache.kafka.clients.consumer.internals.CoordinatorRequestManager:162){code}
> Some of the consumers don't consume any message. The logs show that after the
> consumer starts up and successfully logs in,
> # The consumer discovers the group coordinator.
> # The heartbeat to join group fails because "This is not the correct
> coordinator"
> # The consumer rediscover the group coordinator.
> Another heartbeat should follow the rediscovery of the group coordinator, but
> there's no logs showing sign of a heartbeat request.
> On the server side, there is completely no log about the group id. A
> suspicion is that the consumer doesn't send a heartbeat request after
> rediscover the group coordinator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16227) Console consumer fails with `IllegalStateException`
David Jacot created KAFKA-16227:
---
Summary: Console consumer fails with `IllegalStateException`
Key: KAFKA-16227
URL: https://issues.apache.org/jira/browse/KAFKA-16227
Project: Kafka
Issue Type: Sub-task
Components: clients
Reporter: David Jacot
Assignee: Kirk True
I have seen a few occurrences like the following one. There is a race between
the background thread and the foreground thread. I imagine the following steps:
* quickstart-events-2 is assigned by the background thread;
* the foreground thread starts the initialization of the partition (e.g. reset
offset);
* quickstart-events-2 is removed by the background thread;
* the initialization completes and quickstart-events-2 does not exist anymore.
{code:java}
[2024-02-06 16:21:57,375] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$)
java.lang.IllegalStateException: No current assignment for partition
quickstart-events-2
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367)
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.updateHighWatermark(SubscriptionState.java:579)
at
org.apache.kafka.clients.consumer.internals.FetchCollector.handleInitializeSuccess(FetchCollector.java:283)
at
org.apache.kafka.clients.consumer.internals.FetchCollector.initialize(FetchCollector.java:226)
at
org.apache.kafka.clients.consumer.internals.FetchCollector.collectFetch(FetchCollector.java:110)
at
org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.collectFetch(AsyncKafkaConsumer.java:1540)
at
org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.pollForFetches(AsyncKafkaConsumer.java:1525)
at
org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.poll(AsyncKafkaConsumer.java:711)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:874)
at
kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15460) Add group type filter to ListGroups API
[ https://issues.apache.org/jira/browse/KAFKA-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15460. - Fix Version/s: 3.8.0 Resolution: Fixed > Add group type filter to ListGroups API > --- > > Key: KAFKA-15460 > URL: https://issues.apache.org/jira/browse/KAFKA-15460 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: Ritika Reddy >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16189) Extend admin to support ConsumerGroupDescribe API
[ https://issues.apache.org/jira/browse/KAFKA-16189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16189. - Fix Version/s: 3.8.0 Resolution: Fixed > Extend admin to support ConsumerGroupDescribe API > - > > Key: KAFKA-16189 > URL: https://issues.apache.org/jira/browse/KAFKA-16189 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16168) Implement GroupCoordinator.onPartitionsDeleted
[ https://issues.apache.org/jira/browse/KAFKA-16168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16168. - Fix Version/s: 3.8.0 Resolution: Fixed > Implement GroupCoordinator.onPartitionsDeleted > -- > > Key: KAFKA-16168 > URL: https://issues.apache.org/jira/browse/KAFKA-16168 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16095) Update list group state type filter to include the states for the new consumer group type
[
https://issues.apache.org/jira/browse/KAFKA-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16095.
-
Fix Version/s: 3.8.0
Resolution: Fixed
> Update list group state type filter to include the states for the new
> consumer group type
> -
>
> Key: KAFKA-16095
> URL: https://issues.apache.org/jira/browse/KAFKA-16095
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Ritika Reddy
>Assignee: Lan Ding
>Priority: Minor
> Fix For: 3.8.0
>
>
> # While using *—list —state* the current accepted values correspond to the
> classic group type states. We need to include support for the new group type
> states.
> ## Consumer Group: Should list the state of the group. Accepted Values:
> ### _UNKNOWN(“unknown”)_
> ### {_}EMPTY{_}("empty"),
> ### *{_}ASSIGNING{_}("assigning"),*
> ### *{_}RECONCILING{_}("reconciling"),*
> ### {_}STABLE{_}("stable"),
> ### {_}DEAD{_}("dead");
> #
> ## Classic Group : Should list the state of the group. Accepted Values:
> ### {_}UNKNOWN{_}("Unknown"),
> ### {_}EMPTY{_}("Empty");
> ### *{_}PREPARING_REBALANCE{_}("PreparingRebalance"),*
> ### *{_}COMPLETING_REBALANCE{_}("CompletingRebalance"),*
> ### {_}STABLE{_}("Stable"),
> ### {_}DEAD{_}("Dead")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14505) Implement TnxOffsetCommit API
[ https://issues.apache.org/jira/browse/KAFKA-14505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14505. - Fix Version/s: 3.8.0 Resolution: Fixed > Implement TnxOffsetCommit API > - > > Key: KAFKA-14505 > URL: https://issues.apache.org/jira/browse/KAFKA-14505 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Labels: kip-848-preview > Fix For: 3.8.0 > > > Implement TnxOffsetCommit API in the new Group Coordinator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16194) KafkaConsumer.groupMetadata() should be correct when first records are returned
David Jacot created KAFKA-16194:
---
Summary: KafkaConsumer.groupMetadata() should be correct when
first records are returned
Key: KAFKA-16194
URL: https://issues.apache.org/jira/browse/KAFKA-16194
Project: Kafka
Issue Type: Sub-task
Reporter: David Jacot
The following code returns records before the group metadata is updated. This
fails the first transactions ever run by the Producer/Consumer.
{code:java}
Producer txnProducer = new KafkaProducer<>(txnProducerProps);
Consumer consumer = new KafkaConsumer<>(consumerProps);
txnProducer.initTransactions();
System.out.println("Init transactions called");
try {
txnProducer.beginTransaction();
System.out.println("Begin transactions called");
consumer.subscribe(Collections.singletonList("input"));
System.out.println("Consumer subscribed to topic -> KIP848-topic-2 ");
ConsumerRecords records =
consumer.poll(Duration.ofSeconds(10));
System.out.println("Returned " + records.count() + " records.");
// Process and send txn messages.
for (ConsumerRecord processedRecord : records) {
txnProducer.send(new ProducerRecord<>("output", processedRecord.key(),
"Processed: " + processedRecord.value()));
}
ConsumerGroupMetadata groupMetadata = consumer.groupMetadata();
System.out.println("Group metadata inside test" + groupMetadata);
Map offsetsToCommit = new HashMap<>();
for (ConsumerRecord record : records) {
offsetsToCommit.put(new TopicPartition(record.topic(),
record.partition()),
new OffsetAndMetadata(record.offset() + 1));
}
System.out.println("Offsets to commit" + offsetsToCommit);
// Send offsets to transaction with ConsumerGroupMetadata.
txnProducer.sendOffsetsToTransaction(offsetsToCommit, groupMetadata);
System.out.println("Send offsets to transaction done");
// Commit the transaction.
txnProducer.commitTransaction();
System.out.println("Commit transaction done");
} catch (ProducerFencedException | OutOfOrderSequenceException |
AuthorizationException e) {
e.printStackTrace();
txnProducer.close();
} catch (KafkaException e) {
e.printStackTrace();
txnProducer.abortTransaction();
} finally {
txnProducer.close();
consumer.close();
} {code}
The issue seems to be that while it waits in `poll`, the event to update the
group metadata is not processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16107) Ensure consumer does not start fetching from added partitions until onPartitionsAssigned completes
[ https://issues.apache.org/jira/browse/KAFKA-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16107. - Reviewer: David Jacot Resolution: Fixed > Ensure consumer does not start fetching from added partitions until > onPartitionsAssigned completes > -- > > Key: KAFKA-16107 > URL: https://issues.apache.org/jira/browse/KAFKA-16107 > Project: Kafka > Issue Type: Sub-task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Major > Labels: kip-848-client-support > Fix For: 3.8.0 > > > In the new consumer implementation, when new partitions are assigned, the > subscription state is updated and then the #onPartitionsAssigned triggered. > This sequence seems sensible but we need to ensure that no data is fetched > until the onPartitionsAssigned completes (where the user could be setting the > committed offsets it want to start fetching from). > We should pause the partitions newly added partitions until > onPartitionsAssigned completes, similar to how it's done on revocation to > avoid positions getting ahead of the committed offsets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16189) Extend admin to support ConsumerGroupDescribe API
David Jacot created KAFKA-16189: --- Summary: Extend admin to support ConsumerGroupDescribe API Key: KAFKA-16189 URL: https://issues.apache.org/jira/browse/KAFKA-16189 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16147) Partition is assigned to two members at the same time
[
https://issues.apache.org/jira/browse/KAFKA-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-16147.
-
Fix Version/s: 3.8.0
Resolution: Fixed
> Partition is assigned to two members at the same time
> -
>
> Key: KAFKA-16147
> URL: https://issues.apache.org/jira/browse/KAFKA-16147
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Emanuele Sabellico
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.8.0
>
> Attachments: broker1.log, broker2.log, broker3.log, librdkafka.log,
> server.properties, server1.properties, server2.properties
>
>
> While running [test 0113 of
> librdkafka|https://github.com/confluentinc/librdkafka/blob/8b6357f872efe2a5a3a2fd2828e4133f85e6b023/tests/0113-cooperative_rebalance.cpp#L2384],
> subtest _u_multiple_subscription_changes_ have received this error saying
> that a partition is assigned to two members at the same time.
> {code:java}
> Error: C_6#consumer-9 is assigned rdkafkatest_rnd550f20623daba04c_0113u_2 [0]
> which is already assigned to consumer C_5#consumer-8 {code}
> I've reconstructed this sequence:
> C_5 SUBSCRIBES TO T1
> {noformat}
> %7|1705403451.561|HEARTBEAT|C_5#consumer-8| [thrd:main]: GroupCoordinator/1:
> Heartbeat of member id "RaTCu6RXQH-FiSl95iZzdw", group id
> "rdkafkatest_rnd53b4eb0c2de343_0113u", generation id 6, group instance id
> "(null)", current assignment "", subscribe topics
> "rdkafkatest_rnd5a91902462d61c2e_0113u_1((null))[-1]"{noformat}
> C_5 ASSIGNMENT CHANGES TO T1-P7, T1-P8, T1-P12
> {noformat}
> [2024-01-16 12:10:51,562] INFO [GroupCoordinator id=1
> topic=__consumer_offsets partition=7] [GroupId
> rdkafkatest_rnd53b4eb0c2de343_0113u] Member RaTCu6RXQH-FiSl95iZzdw
> transitioned from CurrentAssignment(memberEpoch=6, previousMemberEpoch=0,
> targetMemberEpoch=6, state=assigning, assignedPartitions={},
> partitionsPendingRevocation={},
> partitionsPendingAssignment={IKXGrFR1Rv-Qes7Ummas6A=[3, 12]}) to
> CurrentAssignment(memberEpoch=14, previousMemberEpoch=6,
> targetMemberEpoch=14, state=stable,
> assignedPartitions={IKXGrFR1Rv-Qes7Ummas6A=[7, 8, 12]},
> partitionsPendingRevocation={}, partitionsPendingAssignment={}).
> (org.apache.kafka.coordinator.group.GroupMetadataManager){noformat}
>
> C_5 RECEIVES TARGET ASSIGNMENT
> {noformat}
> %7|1705403451.565|HEARTBEAT|C_5#consumer-8| [thrd:main]: GroupCoordinator/1:
> Heartbeat response received target assignment
> "(null)(IKXGrFR1Rv+Qes7Ummas6A)[7], (null)(IKXGrFR1Rv+Qes7Ummas6A)[8],
> (null)(IKXGrFR1Rv+Qes7Ummas6A)[12]"{noformat}
>
> C_5 ACKS TARGET ASSIGNMENT
> {noformat}
> %7|1705403451.566|HEARTBEAT|C_5#consumer-8| [thrd:main]: GroupCoordinator/1:
> Heartbeat of member id "RaTCu6RXQH-FiSl95iZzdw", group id
> "rdkafkatest_rnd53b4eb0c2de343_0113u", generation id 14, group instance id
> "NULL", current assignment
> "rdkafkatest_rnd5a91902462d61c2e_0113u_1(IKXGrFR1Rv+Qes7Ummas6A)[7],
> rdkafkatest_rnd5a91902462d61c2e_0113u_1(IKXGrFR1Rv+Qes7Ummas6A)[8],
> rdkafkatest_rnd5a91902462d61c2e_0113u_1(IKXGrFR1Rv+Qes7Ummas6A)[12]",
> subscribe topics "rdkafkatest_rnd5a91902462d61c2e_0113u_1((null))[-1]"
> %7|1705403451.567|HEARTBEAT|C_5#consumer-8| [thrd:main]: GroupCoordinator/1:
> Heartbeat response received target assignment
> "(null)(IKXGrFR1Rv+Qes7Ummas6A)[7], (null)(IKXGrFR1Rv+Qes7Ummas6A)[8],
> (null)(IKXGrFR1Rv+Qes7Ummas6A)[12]"{noformat}
>
> C_5 SUBSCRIBES TO T1,T2: T1 partitions are revoked, 5 T2 partitions are
> pending
> {noformat}
> %7|1705403452.612|HEARTBEAT|C_5#consumer-8| [thrd:main]: GroupCoordinator/1:
> Heartbeat of member id "RaTCu6RXQH-FiSl95iZzdw", group id
> "rdkafkatest_rnd53b4eb0c2de343_0113u", generation id 14, group instance id
> "NULL", current assignment "NULL", subscribe topics
> "rdkafkatest_rnd550f20623daba04c_0113u_2((null))[-1],
> rdkafkatest_rnd5a91902462d61c2e_0113u_1((null))[-1]"
> [2024-01-16 12:10:52,615] INFO [GroupCoordinator id=1
> topic=__consumer_offsets partition=7] [GroupId
> rdkafkatest_rnd53b4eb0c2de343_0113u] Member RaTCu6RXQH-FiSl95iZzdw updated
> its subscribed topics to: [rdkafkatest_rnd550f20623daba04c_0113u_2,
> rdkafkatest_rnd5a91902462d61c2e_0113u_1].
> (org.apache.kafka.coordinator.group.GroupMetadataManager)
> [2024-01-16 12:10:52,616] INFO [GroupCoordinator id=1
> topic=__consumer_offsets partition=7] [GroupId
> rdkafkatest_rnd53b4eb0c2de343_0113u] Member RaTCu6RXQH-FiSl95iZzdw
> transitioned from CurrentAssignment(memberEpoch=14, previousMemberEpoch=6,
> targetMemberEpoch=14, state=stable,
> assignedPartitions={IKXGrFR1Rv-Qes7Ummas6A=[7, 8, 12]},
> partitionsPendingRevocation={}, partitionsPendingAssignment={}) to
> CurrentAssignment(memberEpoch=14, previousMemberEpoch=6,
> targetMemberEpoch=16, state=revoking, assignedParti
[jira] [Created] (KAFKA-16168) Implement GroupCoordinator.onPartitionsDeleted
David Jacot created KAFKA-16168: --- Summary: Implement GroupCoordinator.onPartitionsDeleted Key: KAFKA-16168 URL: https://issues.apache.org/jira/browse/KAFKA-16168 Project: Kafka Issue Type: Sub-task Reporter: David Jacot Assignee: David Jacot -- This message was sent by Atlassian Jira (v8.20.10#820010)
