[jira] [Commented] (KAFKA-14955) Validate that all partitions are assigned in TargetAssignmentBuilder
[ https://issues.apache.org/jira/browse/KAFKA-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17887279#comment-17887279 ] PoAn Yang commented on KAFKA-14955: --- Hi [~dajac], if you're not working on this, may I take it? My idea is to add the validation and log warning message if there is a partition which is not assigned. Thank you. > Validate that all partitions are assigned in TargetAssignmentBuilder > > > Key: KAFKA-14955 > URL: https://issues.apache.org/jira/browse/KAFKA-14955 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Priority: Major > > We may want to ensure that all partitions are assigned when a new target > assignment is computed by the TargetAssignmentBuilder. On the server side, > there is no reason for a partition to not be assigned. On the client side, > this is debatable though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16460) New consumer times out consuming records in multiple consumer_test.py system tests
[
https://issues.apache.org/jira/browse/KAFKA-16460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17887252#comment-17887252
]
PoAn Yang commented on KAFKA-16460:
---
Hi [~kirktrue], if you're not working on this, may I take it? Thank you.
> New consumer times out consuming records in multiple consumer_test.py system
> tests
> --
>
> Key: KAFKA-16460
> URL: https://issues.apache.org/jira/browse/KAFKA-16460
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, system tests
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Priority: Critical
> Labels: kip-848-client-support, system-tests
> Fix For: 4.0.0
>
>
> The {{consumer_test.py}} system test fails with the following errors:
> {quote}
> * Timed out waiting for consumption
> {quote}
> Affected tests:
> * {{test_broker_failure}}
> * {{test_consumer_bounce}}
> * {{test_static_consumer_bounce}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17707) Remove zk from BaseConsumerTest
PoAn Yang created KAFKA-17707:
-
Summary: Remove zk from BaseConsumerTest
Key: KAFKA-17707
URL: https://issues.apache.org/jira/browse/KAFKA-17707
Project: Kafka
Issue Type: Sub-task
Reporter: PoAn Yang
Assignee: PoAn Yang
Remove {{{}Arguments.of("zk", "classic"){}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-17603) KRaftMigrationDriverTest testNoDualWriteBeforeMigration became flaky
[ https://issues.apache.org/jira/browse/KAFKA-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17603. --- Resolution: Not A Problem The test case is removed in trunk. We can reopen again if it's flaky in 3.x. > KRaftMigrationDriverTest testNoDualWriteBeforeMigration became flaky > > > Key: KAFKA-17603 > URL: https://issues.apache.org/jira/browse/KAFKA-17603 > Project: Kafka > Issue Type: Test > Components: kraft >Reporter: David Arthur >Assignee: PoAn Yang >Priority: Minor > Attachments: image-2024-09-24-13-25-22-534.png > > > This test suddenly became consistently flaky on trunk on Sep 16. > !image-2024-09-24-13-25-22-534.png|width=725,height=285! > [https://ge.apache.org/scans/tests?search.names=CI%20workflow&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.tags=github,trunk&search.tasks=test&search.timeZoneId=America%2FNew_York&search.values=CI&tests.container=org.apache.kafka.metadata.migration.KRaftMigrationDriverTest&tests.test=testNoDualWriteBeforeMigration()] > > There haven't been any changes to this test or the metadata module in that > time frame, so this is pretty weird. The commit when this became flaky was > [https://github.com/apache/kafka/commit/d0f4d691b592ed2d65c9df413f5d6660df7fc90e.|https://github.com/apache/kafka/commit/d0f4d691b592ed2d65c9df413f5d6660df7fc90e] > > Marking this a minor since this test will be deleted soon. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17648) AsyncKafkaConsumer#unsubscribe swallow TopicAuthorizationException
PoAn Yang created KAFKA-17648: - Summary: AsyncKafkaConsumer#unsubscribe swallow TopicAuthorizationException Key: KAFKA-17648 URL: https://issues.apache.org/jira/browse/KAFKA-17648 Project: Kafka Issue Type: Task Reporter: PoAn Yang Assignee: PoAn Yang Followup for [https://github.com/apache/kafka/pull/17244]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17647) Check PlaintextConsumerSubscriptionTest#testSubscribeInvalidTopicCanUnsubscribe really ignore ErrorEvent
PoAn Yang created KAFKA-17647: - Summary: Check PlaintextConsumerSubscriptionTest#testSubscribeInvalidTopicCanUnsubscribe really ignore ErrorEvent Key: KAFKA-17647 URL: https://issues.apache.org/jira/browse/KAFKA-17647 Project: Kafka Issue Type: Task Reporter: PoAn Yang Assignee: PoAn Yang Follow up for [https://github.com/apache/kafka/pull/17244]. We only check unsubscribe can succeed, but this call may run too fast, so there may not have further ErrorEvent. We may check new metric background-event-queue-size to make sure there is new ErrorEvent and unsubscribe can ignore it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17631) Convert SaslApiVersionsRequestTest to kraft
PoAn Yang created KAFKA-17631: - Summary: Convert SaslApiVersionsRequestTest to kraft Key: KAFKA-17631 URL: https://issues.apache.org/jira/browse/KAFKA-17631 Project: Kafka Issue Type: Task Reporter: PoAn Yang Assignee: PoAn Yang We would like to remove zk type from ClusterTest, but SaslApiVersionsRequestTest has no kraft test. We should convert the test to kraft. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17630) Convert ClientQuotasRequestTest#testClientQuotasForScramUsers to kraft
PoAn Yang created KAFKA-17630: - Summary: Convert ClientQuotasRequestTest#testClientQuotasForScramUsers to kraft Key: KAFKA-17630 URL: https://issues.apache.org/jira/browse/KAFKA-17630 Project: Kafka Issue Type: Task Reporter: PoAn Yang Assignee: PoAn Yang We would like to remove zk type from ClusterTest, but ClientQuotasRequestTest#testClientQuotasForScramUsers has no kraft test. We should convert the test to kraft. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17623) Flaky testSeekPositionAndPauseNewlyAssignedPartitionOnPartitionsAssignedCallback
[ https://issues.apache.org/jira/browse/KAFKA-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885047#comment-17885047 ] PoAn Yang commented on KAFKA-17623: --- Hi [~lianetm], may I take this? Thank you. > Flaky > testSeekPositionAndPauseNewlyAssignedPartitionOnPartitionsAssignedCallback > > > Key: KAFKA-17623 > URL: https://issues.apache.org/jira/browse/KAFKA-17623 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Lianet Magrans >Priority: Major > Labels: consumer-threading-refactor > > Flaky for the new consumer, failing with : > org.apache.kafka.common.KafkaException: User rebalance callback throws an > error at > app//org.apache.kafka.clients.consumer.internals.ConsumerUtils.maybeWrapAsKafkaException(ConsumerUtils.java:259) > at > app//org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.invokeRebalanceCallbacks(AsyncKafkaConsumer.java:1867) > at > app//org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer$BackgroundEventProcessor.process(AsyncKafkaConsumer.java:195) > at > app//org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer$BackgroundEventProcessor.process(AsyncKafkaConsumer.java:181) > at > app//org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.processBackgroundEvents(AsyncKafkaConsumer.java:1758) > at > app//org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.updateAssignmentMetadataIfNeeded(AsyncKafkaConsumer.java:1618) > ... > Caused by: java.lang.IllegalStateException: No current assignment for > partition topic-0 at > org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:378) > at > org.apache.kafka.clients.consumer.internals.SubscriptionState.seekUnvalidated(SubscriptionState.java:395) > at > org.apache.kafka.clients.consumer.internals.events.ApplicationEventProcessor.process(ApplicationEventProcessor.java:425) > at > org.apache.kafka.clients.consumer.internals.events.ApplicationEventProcessor.process(ApplicationEventProcessor.java:147) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.processApplicationEvents(ConsumerNetworkThread.java:171) > > Flaky behaviour: > > https://ge.apache.org/scans/tests?search.buildOutcome=failure&search.names=Git%20branch&search.rootProjectNames=kafka&search.startTimeMax=172740959&search.startTimeMin=172248480&search.timeZoneId=America%2FToronto&search.values=trunk&tests.container=integration.kafka.api.PlaintextConsumerCallbackTest&tests.test=testSeekPositionAndPauseNewlyAssignedPartitionOnPartitionsAssignedCallback(String%2C%20String)%5B3%5D -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17618) group consumer heartbeat interval should be less than session timeout
PoAn Yang created KAFKA-17618: - Summary: group consumer heartbeat interval should be less than session timeout Key: KAFKA-17618 URL: https://issues.apache.org/jira/browse/KAFKA-17618 Project: Kafka Issue Type: Task Reporter: PoAn Yang Assignee: PoAn Yang [KIP-848|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-Heartbeat&Session] mentions: bq. The member is expected to heartbeat every group.consumer.heartbeat.interval.ms in order to keep its session opened. If it does not heartbeat at least once within the group.consumer.session.timeout.ms, the group coordinator will kick the member out from the group. To avoid users configure _group.consumer.heartbeat.interval.ms_ bigger than _group.consumer.session.timeout.ms_, we can add validation for it. We can do similar validation for _group.share.heartbeat.interval.ms_ and _group.share.session.timeout.ms_ as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17618) group consumer heartbeat interval should be less than session timeout
[ https://issues.apache.org/jira/browse/KAFKA-17618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17618: -- Labels: kip-848 (was: ) > group consumer heartbeat interval should be less than session timeout > - > > Key: KAFKA-17618 > URL: https://issues.apache.org/jira/browse/KAFKA-17618 > Project: Kafka > Issue Type: Task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > Labels: kip-848 > > [KIP-848|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-Heartbeat&Session] > mentions: > bq. The member is expected to heartbeat every > group.consumer.heartbeat.interval.ms in order to keep its session opened. If > it does not heartbeat at least once within the > group.consumer.session.timeout.ms, the group coordinator will kick the member > out from the group. > To avoid users configure _group.consumer.heartbeat.interval.ms_ bigger than > _group.consumer.session.timeout.ms_, we can add validation for it. > We can do similar validation for _group.share.heartbeat.interval.ms_ and > _group.share.session.timeout.ms_ as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17614) Remove AclAuthorizer
[ https://issues.apache.org/jira/browse/KAFKA-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884810#comment-17884810 ] PoAn Yang commented on KAFKA-17614: --- Hi [~cmccabe], if you're not working on this, may I take this issue? Thank you. > Remove AclAuthorizer > > > Key: KAFKA-17614 > URL: https://issues.apache.org/jira/browse/KAFKA-17614 > Project: Kafka > Issue Type: Sub-task >Reporter: Colin McCabe >Priority: Major > > Remove AclAuthorizer as it has been superseded by StandardAuthorizer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17598) Command line validation tool for RE2J regex
[ https://issues.apache.org/jira/browse/KAFKA-17598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883917#comment-17883917 ] PoAn Yang commented on KAFKA-17598: --- Hi [~lianetm], I'm interested in this Jira. Could you elaborate more on it? Is it about to add a new command line tool to validate RE2J regex input? > Command line validation tool for RE2J regex > --- > > Key: KAFKA-17598 > URL: https://issues.apache.org/jira/browse/KAFKA-17598 > Project: Kafka > Issue Type: Sub-task >Reporter: Lianet Magrans >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17581) AsyncKafkaConsumer can't unsubscribe invalid topics
[
https://issues.apache.org/jira/browse/KAFKA-17581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883614#comment-17883614
]
PoAn Yang commented on KAFKA-17581:
---
Hi [~kirktrue], here is related Jira and PR:
* https://issues.apache.org/jira/browse/KAFKA-16764
* https://github.com/apache/kafka/pull/16043
> AsyncKafkaConsumer can't unsubscribe invalid topics
> ---
>
> Key: KAFKA-17581
> URL: https://issues.apache.org/jira/browse/KAFKA-17581
> Project: Kafka
> Issue Type: Bug
>Reporter: PoAn Yang
>Assignee: PoAn Yang
>Priority: Major
>
> When consumer subscribes an invalid topic name like " this is test", classic
> consumer can unsubscribe without error. However, async consumer can't. We can
> use following integration test to validate:
>
> {code:java}
> @ParameterizedTest(name =
> TestInfoUtils.TestWithParameterizedQuorumAndGroupProtocolNames)
> @MethodSource(Array("getTestQuorumAndGroupProtocolParametersAll"))
> def testSubscribeInvalidTopic(quorum: String, groupProtocol: String): Unit = {
> // Invalid topic name due to space
> val invalidTopicName = "topic abc"
> val consumer = createConsumer()
> consumer.subscribe(List(invalidTopicName).asJava)
> var exception : InvalidTopicException = null
> TestUtils.waitUntilTrue(() => {
> try consumer.poll(Duration.ofMillis(500)) catch {
> case e : InvalidTopicException => exception = e
> case e : Throwable => fail(s"An InvalidTopicException should be thrown.
> But ${e.getClass} is thrown")
> }
> exception != null
> }, waitTimeMs = 5000, msg = "An InvalidTopicException should be thrown.")
> assertEquals(s"Invalid topics: [${invalidTopicName}]", exception.getMessage)
> // AsyncKafkaConsumer sends request in background thread. Wait enough time
> to send next request.
> Thread.sleep(1000)
> assertDoesNotThrow(new Executable {
> override def execute(): Unit = consumer.unsubscribe()
> })
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17581) AsyncKafkaConsumer can't unsubscribe invalid topics
PoAn Yang created KAFKA-17581:
-
Summary: AsyncKafkaConsumer can't unsubscribe invalid topics
Key: KAFKA-17581
URL: https://issues.apache.org/jira/browse/KAFKA-17581
Project: Kafka
Issue Type: Bug
Reporter: PoAn Yang
Assignee: PoAn Yang
When consumer subscribes an invalid topic name like " this is test", classic
consumer can unsubscribe without error. However, async consumer can't. We can
use following integration test to validate:
{code:java}
@ParameterizedTest(name =
TestInfoUtils.TestWithParameterizedQuorumAndGroupProtocolNames)
@MethodSource(Array("getTestQuorumAndGroupProtocolParametersAll"))
def testSubscribeInvalidTopic(quorum: String, groupProtocol: String): Unit = {
// Invalid topic name due to space
val invalidTopicName = "topic abc"
val consumer = createConsumer()
consumer.subscribe(List(invalidTopicName).asJava)
var exception : InvalidTopicException = null
TestUtils.waitUntilTrue(() => {
try consumer.poll(Duration.ofMillis(500)) catch {
case e : InvalidTopicException => exception = e
case e : Throwable => fail(s"An InvalidTopicException should be thrown.
But ${e.getClass} is thrown")
}
exception != null
}, waitTimeMs = 5000, msg = "An InvalidTopicException should be thrown.")
assertEquals(s"Invalid topics: [${invalidTopicName}]", exception.getMessage)
// AsyncKafkaConsumer sends request in background thread. Wait enough time to
send next request.
Thread.sleep(1000)
assertDoesNotThrow(new Executable {
override def execute(): Unit = consumer.unsubscribe()
})
}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17578) Remove partitionRacks from TopicMetadata
[ https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883035#comment-17883035 ] PoAn Yang commented on KAFKA-17578: --- There are two failed cases `testGroupEpochBumpWhenNewStaticMemberJoins` and `testNewJoiningMemberTriggersNewTargetAssignment` in `GroupMetadataManagerTest`. I will take a look tomorrow. > Remove partitionRacks from TopicMetadata > > > Key: KAFKA-17578 > URL: https://issues.apache.org/jira/browse/KAFKA-17578 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: PoAn Yang >Priority: Major > Labels: kip-848 > > The ModernGroup#subscribedTopicMetadata takes too much memory due to > partitionRacks: > /** > Map of every partition Id to a set of its rack Ids, if they exist. > If rack information is unavailable for all partitions, this is an empty map. > */ > private final Map> partitionRacks; > > This is not being used at the moment as the consumer protocol does not > support rack aware assignments. > > A heap dump from a group with 500 members, 2K subscribed topic partitions > shows > 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object > holds 822,860 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17578) Remove partitionRacks from TopicMetadata
[ https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883008#comment-17883008 ] PoAn Yang commented on KAFKA-17578: --- PR: [https://github.com/apache/kafka/pull/17233] I will change to ready for review if CI result is good. I run test for all changed test files and there is no error. > Remove partitionRacks from TopicMetadata > > > Key: KAFKA-17578 > URL: https://issues.apache.org/jira/browse/KAFKA-17578 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: PoAn Yang >Priority: Major > Labels: kip-848 > > The ModernGroup#subscribedTopicMetadata takes too much memory due to > partitionRacks: > /** > Map of every partition Id to a set of its rack Ids, if they exist. > If rack information is unavailable for all partitions, this is an empty map. > */ > private final Map> partitionRacks; > > This is not being used at the moment as the consumer protocol does not > support rack aware assignments. > > A heap dump from a group with 500 members, 2K subscribed topic partitions > shows > 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object > holds 822,860 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17578) Remove partitionRacks from TopicMetadata
[ https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882992#comment-17882992 ] PoAn Yang commented on KAFKA-17578: --- Hi [~jeffkbkim] and [~dajac], there are 200+ references to TopicMetadata. Do we want to fully remove partitionRacks? Or we can create a new TopicMetadata constructor which always set empty to TopicMetadata. And we only need to change production code and related test cases? Thank you. > Remove partitionRacks from TopicMetadata > > > Key: KAFKA-17578 > URL: https://issues.apache.org/jira/browse/KAFKA-17578 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: PoAn Yang >Priority: Major > Labels: kip-848 > > The ModernGroup#subscribedTopicMetadata takes too much memory due to > partitionRacks: > /** > Map of every partition Id to a set of its rack Ids, if they exist. > If rack information is unavailable for all partitions, this is an empty map. > */ > private final Map> partitionRacks; > > This is not being used at the moment as the consumer protocol does not > support rack aware assignments. > > A heap dump from a group with 500 members, 2K subscribed topic partitions > shows > 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object > holds 822,860 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17578) Remove partitionRacks from TopicMetadata
[ https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882992#comment-17882992 ] PoAn Yang edited comment on KAFKA-17578 at 9/19/24 12:01 PM: - Hi [~jeffkbkim] and [~dajac], there are 200+ references to TopicMetadata. Do we want to fully remove partitionRacks? Or we can create a new TopicMetadata constructor which always set empty partitionRacks to TopicMetadata. And we only need to change production code and related test cases? Thank you. was (Author: JIRAUSER300229): Hi [~jeffkbkim] and [~dajac], there are 200+ references to TopicMetadata. Do we want to fully remove partitionRacks? Or we can create a new TopicMetadata constructor which always set empty to TopicMetadata. And we only need to change production code and related test cases? Thank you. > Remove partitionRacks from TopicMetadata > > > Key: KAFKA-17578 > URL: https://issues.apache.org/jira/browse/KAFKA-17578 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: PoAn Yang >Priority: Major > Labels: kip-848 > > The ModernGroup#subscribedTopicMetadata takes too much memory due to > partitionRacks: > /** > Map of every partition Id to a set of its rack Ids, if they exist. > If rack information is unavailable for all partitions, this is an empty map. > */ > private final Map> partitionRacks; > > This is not being used at the moment as the consumer protocol does not > support rack aware assignments. > > A heap dump from a group with 500 members, 2K subscribed topic partitions > shows > 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object > holds 822,860 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17578) Remove partitionRacks from TopicMetadata
[ https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17578: - Assignee: PoAn Yang > Remove partitionRacks from TopicMetadata > > > Key: KAFKA-17578 > URL: https://issues.apache.org/jira/browse/KAFKA-17578 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Assignee: PoAn Yang >Priority: Major > Labels: kip-848 > > The ModernGroup#subscribedTopicMetadata takes too much memory due to > partitionRacks: > /** > Map of every partition Id to a set of its rack Ids, if they exist. > If rack information is unavailable for all partitions, this is an empty map. > */ > private final Map> partitionRacks; > > This is not being used at the moment as the consumer protocol does not > support rack aware assignments. > > A heap dump from a group with 500 members, 2K subscribed topic partitions > shows > 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object > holds 822,860 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17578) Remove partitionRacks from TopicMetadata
[ https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882856#comment-17882856 ] PoAn Yang commented on KAFKA-17578: --- Hi [~jeffkbkim], if you're not working on this issue, may I take it? Thank you. > Remove partitionRacks from TopicMetadata > > > Key: KAFKA-17578 > URL: https://issues.apache.org/jira/browse/KAFKA-17578 > Project: Kafka > Issue Type: Sub-task >Reporter: Jeff Kim >Priority: Major > Labels: kip-848 > > The ModernGroup#subscribedTopicMetadata takes too much memory due to > partitionRacks: > /** > * Map of every partition Id to a set of its rack Ids, if they exist. > * If rack information is unavailable for all partitions, this is an empty map. > */ > private final Map> partitionRacks; > > This is not being used at the moment as the consumer protocol does not > support rack aware assignments. > > A heap dump from a group with 500 members, 2K subscribed topic partitions > shows > 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object > holds 822,860 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17024) add integration test for TransactionsCommand
[ https://issues.apache.org/jira/browse/KAFKA-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17024: - Assignee: Kuan Po Tseng (was: PoAn Yang) > add integration test for TransactionsCommand > > > Key: KAFKA-17024 > URL: https://issues.apache.org/jira/browse/KAFKA-17024 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > as title. currently we have only UT for TransactionsCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17573) Move KRaftMetadataCache to server module
PoAn Yang created KAFKA-17573: - Summary: Move KRaftMetadataCache to server module Key: KAFKA-17573 URL: https://issues.apache.org/jira/browse/KAFKA-17573 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang as title -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17220) Define new metrics for MirrorMaker2
[ https://issues.apache.org/jira/browse/KAFKA-17220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882382#comment-17882382 ] PoAn Yang commented on KAFKA-17220: --- KIP-1093: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1093%3A+Add+more+observability+for+MirrorMaker2] > Define new metrics for MirrorMaker2 > --- > > Key: KAFKA-17220 > URL: https://issues.apache.org/jira/browse/KAFKA-17220 > Project: Kafka > Issue Type: New Feature > Components: mirrormaker >Reporter: Greg Harris >Assignee: PoAn Yang >Priority: Major > Labels: needs-kip > > MirrorMaker2 provides some observability into it's operation, but lacks > observability in the following areas: > * Number of replicated topics/partitions > * Number of consumer groups & consumer offsets > * Background job success/failure/latency/duration > * Checkpoint task startup time > * Consumer group translation lag/expected redelivered data > * Error metrics for failed offset translations > These concepts should be more clearly defined and included in a general > observability KIP for MM2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17560) testCurrentLag flaky if enabled for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882216#comment-17882216 ] PoAn Yang commented on KAFKA-17560: --- Hi [~lianetm] and [~frankvicky], thanks for filing/taking the Jira. I would like to give this solution a try in original PR. If it works, probably we can close this one as well. Thank you. > testCurrentLag flaky if enabled for new consumer > > > Key: KAFKA-17560 > URL: https://issues.apache.org/jira/browse/KAFKA-17560 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: TengYao Chi >Priority: Minor > Labels: consumer, consumer-threading-refactor > > There was an initial attempt to enable testCurrentLag for the new consumer > with [https://github.com/apache/kafka/pull/16982.] Even though it included > several changes addressing flakiness, it seems not enough, given that the > test is still flaky when running for the new consumer. Fails with: > > org.opentest4j.AssertionFailedError: > Expected :OptionalLong[40] > Actual :OptionalLong.empty > > at > org.apache.kafka.clients.consumer.KafkaConsumerTest.testCurrentLag(KafkaConsumerTest.java:2529) > > I wonder if this flakiness is because after receiving a response to a list > offset request, maybe a single call to currentLag may not be enough (offsets > received in the response not updated yet in the background). Note that after > receiving a response to the ListOffsets request, the test moves right away > into a single call to currentLag that it expects to be 40, but in the > background we only update the subscription state when we get to: > [https://github.com/apache/kafka/blob/aaf3fc05f8e7cb6b8c79f8471f68faa3a994e1e5/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetsRequestManager.java#L565] > > So we could end up with that single call to currentLag not finding the > endOffsets we got in the response (it will find them "eventually" with the > new consumer I expect) > > We should review this test to be able to enable it for the new consumer > without flakiness. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17524) StreamThreadTest shouldReturnErrorIfProducerInstanceIdNotInitialized hanging
[
https://issues.apache.org/jira/browse/KAFKA-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882213#comment-17882213
]
PoAn Yang commented on KAFKA-17524:
---
Hi [~chia7712], if you're not start to working on the patch, may I take this
one? I'll follow your thought and try to give a PR. Thank you.
> StreamThreadTest shouldReturnErrorIfProducerInstanceIdNotInitialized hanging
>
>
> Key: KAFKA-17524
> URL: https://issues.apache.org/jira/browse/KAFKA-17524
> Project: Kafka
> Issue Type: Test
> Components: streams, unit tests
>Reporter: David Arthur
>Assignee: Chia-Ping Tsai
>Priority: Major
> Labels: flaky-test
>
> A trunk build had a timeout and it appears that this test was the cause.
> [https://github.com/apache/kafka/actions/runs/10798234851/job/29953919232]
> In the Gradle log, we see
> {code:java}
> 2024-09-10T20:31:26.6830206Z Gradle Test Run :streams:test > Gradle Test
> Executor 47 > StreamThreadTest >
> shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean, boolean) >
> "shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean,
> boolean).stateUpdaterEnabled=true, processingThreadsEnabled=true" STARTED
> {code}
> but no "FINISHED" or "FAILED" later in the log.
> It seems that this test was running for around 50 minutes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17552) Handle LIST_OFFSETS request for max_timestamp when remote storage is enabled
[ https://issues.apache.org/jira/browse/KAFKA-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881984#comment-17881984 ] PoAn Yang commented on KAFKA-17552: --- Hi [~chia7712], if you're not working on this, may I take it? Thank you. > Handle LIST_OFFSETS request for max_timestamp when remote storage is enabled > > > Key: KAFKA-17552 > URL: https://issues.apache.org/jira/browse/KAFKA-17552 > Project: Kafka > Issue Type: Task >Reporter: Kamal Chandraprakash >Assignee: Chia-Ping Tsai >Priority: Major > > context: https://github.com/apache/kafka/pull/16602#discussion_r1759135392 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17519) Define and validate correctness of Consumer.close() and its timeout when thread is interrupted
[
https://issues.apache.org/jira/browse/KAFKA-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
PoAn Yang reassigned KAFKA-17519:
-
Assignee: PoAn Yang
> Define and validate correctness of Consumer.close() and its timeout when
> thread is interrupted
> --
>
> Key: KAFKA-17519
> URL: https://issues.apache.org/jira/browse/KAFKA-17519
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.9.0
>Reporter: Kirk True
>Assignee: PoAn Yang
>Priority: Major
> Labels: consumer-threading-refactor, kip-848-client-support
>
> The repercussions of a thread's interrupt status on {{Consumer.close()}} and
> its timeout is not well defined. It _appears_ that the
> {{ClassicKafkaConsumer}} will continue to attempt to close all its resources
> even if an interrupt was triggered prior to—or during—the call to {{close()}}
> though it effectively ignores the user's supplied timeout since each call to
> {{NetworkClient.poll()}} will throw an {{InterruptException}} after first
> making an attempt to poll the socket.
> The task here is to review the existing code, verify the behavior with some
> unit/integration tests, and document it. Furthermore, once the intended
> behavior has been confirmed, the {{AsyncKafkaConsumer}} should be updated to
> behave likewise.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17535) Flaky testCloseNoWait in KafkaConsumerTest for Classic Consumer
[ https://issues.apache.org/jira/browse/KAFKA-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881358#comment-17881358 ] PoAn Yang commented on KAFKA-17535: --- Hi [~lianetm], if you're not working on this, may I take the issue? Thank you. > Flaky testCloseNoWait in KafkaConsumerTest for Classic Consumer > > > Key: KAFKA-17535 > URL: https://issues.apache.org/jira/browse/KAFKA-17535 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Lianet Magrans >Priority: Major > Labels: flaky-test > > Flaky failures due to TimeoutException: > java.util.concurrent.TimeoutException at > java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204) at > org.apache.kafka.clients.consumer.KafkaConsumerTest.consumerCloseTest(KafkaConsumerTest.java:2049) > Flaky in trunk for the classic consumer for a while: > [https://ge.apache.org/scans/tests?search.names=Git%20branch&search.rootProjectNames=kafka&search.startTimeMax=17261&search.startTimeMin=172248480&search.timeZoneId=America%2FToronto&search.values=trunk&tests.container=org.apache.kafka.clients.consumer.KafkaConsumerTest&tests.test=testCloseNoWait(GroupProtocol)%5B1%5D] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17154) New consumer subscribe may join group without a call to consumer.poll
[
https://issues.apache.org/jira/browse/KAFKA-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880824#comment-17880824
]
PoAn Yang commented on KAFKA-17154:
---
Yes, I'll work on it this week. Thank you.
> New consumer subscribe may join group without a call to consumer.poll
> -
>
> Key: KAFKA-17154
> URL: https://issues.apache.org/jira/browse/KAFKA-17154
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.8.0
>Reporter: Lianet Magrans
>Assignee: PoAn Yang
>Priority: Blocker
> Labels: kip-848-client-support
> Fix For: 4.0.0
>
>
> With the new consumer, a call to consumer.subscribe will update the client
> subscription state via a SubscriptionChangedEvent. This will be picked up by
> the background thread, and will send a Heartbeat RPC on the next poll of the
> background thread (without requiring a consumer.poll). This is a change in
> behaviour with the legacy consumer, that breaks the consumer#subscribe
> contract for "rebalances will only occur during an active call to \{@link
> #poll(Duration)}", so it should be fixed.
> In the legacy its a similar principle but different behaviour: subscribe
> changes the subscription state, and the coordinator picks it up to send the
> JoinRequest on the next poll of the coordinator (but this happens only as
> part of a consumer.poll)
> We should make the new consumer join (send HB to join) only on consumer.poll
> after a call to subscribed. We could consider having the
> SubscriptionChangedEvent only signal to the background that the subscription
> changed without doing the transition to JOINING that triggers the HB
> (membershipMgr#onSubscriptionUpdated could simply flip a flag, no
> transition), and then do the actual transition to JOINING when processing a
> PollEvent if the flag is on (flipping it back). (Just first simple approach
> that comes to mind, we should think about it a bit more and consider other
> interactions that could happen in between, like unsubscribe, close, etc).
> We should add test coverage to ensure the common behaviour with the legacy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17519) Define and validate correctness of Consumer.close() and its timeout when thread is interrupted
[
https://issues.apache.org/jira/browse/KAFKA-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880820#comment-17880820
]
PoAn Yang commented on KAFKA-17519:
---
Hi [~kirktrue], if you're not working on this, may I take it? Thank you.
> Define and validate correctness of Consumer.close() and its timeout when
> thread is interrupted
> --
>
> Key: KAFKA-17519
> URL: https://issues.apache.org/jira/browse/KAFKA-17519
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.9.0
>Reporter: Kirk True
>Priority: Major
> Labels: consumer-threading-refactor, kip-848-client-support
>
> The repercussions of a thread's interrupt status on {{Consumer.close()}} and
> its timeout is not well defined. It _appears_ that the
> {{ClassicKafkaConsumer}} will continue to attempt to close all its resources
> even if an interrupt was triggered prior to—or during—the call to {{close()}}
> though it effectively ignores the user's supplied timeout since each call to
> {{NetworkClient.poll()}} will throw an {{InterruptException}} after first
> making an attempt to poll the socket.
> The task here is to review the existing code, verify the behavior with some
> unit/integration tests, and document it. Furthermore, once the intended
> behavior has been confirmed, the {{AsyncKafkaConsumer}} should be updated to
> behave likewise.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-17511) Move ElectLeadersRequestOps to ElectLeadersRequest
[ https://issues.apache.org/jira/browse/KAFKA-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17511: - Assignee: Eric Chang (was: PoAn Yang) > Move ElectLeadersRequestOps to ElectLeadersRequest > -- > > Key: KAFKA-17511 > URL: https://issues.apache.org/jira/browse/KAFKA-17511 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Eric Chang >Priority: Major > > as title -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16143) New JMX metrics for AsyncKafkaConsumer
[ https://issues.apache.org/jira/browse/KAFKA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880635#comment-17880635 ] PoAn Yang commented on KAFKA-16143: --- Hi [~lianetm], I can help to finish this JIRA if [~brendendeluna] is not working on it. Thank you. > New JMX metrics for AsyncKafkaConsumer > -- > > Key: KAFKA-16143 > URL: https://issues.apache.org/jira/browse/KAFKA-16143 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer, metrics >Affects Versions: 3.7.0 >Reporter: Kirk True >Assignee: Brenden DeLuna >Priority: Minor > Labels: kip-848-client-support, metrics, needs-kip > Fix For: 4.0.0 > > > This task is to consider what _new_ metrics we need from the KIP-848 protocol > that aren't already exposed by the current set of metrics. This will require > a KIP to introduce the new metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[
https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878608#comment-17878608
]
PoAn Yang edited comment on KAFKA-16792 at 9/2/24 3:14 PM:
---
The {{ApplicationEventProcessor#process(PollEvent)}} only reset timer for some
request managers. Even if we use {{ApplicationEventHandler#addAndGet}}, we
still can't guarantee {{runOnce}} is executed. We may need
{{ConsumerNetworkThread}} to send a {{BackgroundEvent}} to make sure the
{{runOnce}} is happened. However, this approach may make the process more
complex.
was (Author: JIRAUSER300229):
The {{ApplicationEventProcessor#process(PollEvent)}} only reset timer for some
request managers. Even if we use {{ApplicationEventHandler#addAndGet}}, we
still can't guarantee {{runOnce}} is executed. We may need
{{ConsumerNetworkThread}} to send a {{BackgroundEvent}} to make sure the
{{runOnce}} is happened. However, this ㄇapproach may make the process more
complex.
> Enable consumer unit tests that fail to fetch offsets only for new consumer
> with poll(0)
>
>
> Key: KAFKA-16792
> URL: https://issues.apache.org/jira/browse/KAFKA-16792
> Project: Kafka
> Issue Type: Test
> Components: clients, consumer, unit tests
>Reporter: Lianet Magrans
>Assignee: PoAn Yang
>Priority: Blocker
> Labels: kip-848-client-support
> Fix For: 4.0.0
>
>
> Enable the following unit tests for the new async consumer in
> KafkaConsumerTest:
> - testCurrentLag
> - testFetchStableOffsetThrowInPoll
> - testListOffsetShouldUpdateSubscriptions
> - testPollReturnsRecords
> - testResetToCommittedOffset
> - testResetUsingAutoResetPolicy
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[
https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878608#comment-17878608
]
PoAn Yang commented on KAFKA-16792:
---
The {{ApplicationEventProcessor#process(PollEvent)}} only reset timer for some
request managers. Even if we use {{ApplicationEventHandler#addAndGet}}, we
still can't guarantee {{runOnce}} is executed. We may need
{{ConsumerNetworkThread}} to send a {{BackgroundEvent}} to make sure the
{{runOnce}} is happened. However, this ㄇapproach may make the process more
complex.
> Enable consumer unit tests that fail to fetch offsets only for new consumer
> with poll(0)
>
>
> Key: KAFKA-16792
> URL: https://issues.apache.org/jira/browse/KAFKA-16792
> Project: Kafka
> Issue Type: Test
> Components: clients, consumer, unit tests
>Reporter: Lianet Magrans
>Assignee: PoAn Yang
>Priority: Blocker
> Labels: kip-848-client-support
> Fix For: 4.0.0
>
>
> Enable the following unit tests for the new async consumer in
> KafkaConsumerTest:
> - testCurrentLag
> - testFetchStableOffsetThrowInPoll
> - testListOffsetShouldUpdateSubscriptions
> - testPollReturnsRecords
> - testResetToCommittedOffset
> - testResetUsingAutoResetPolicy
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17405) how to enforce the MV 3.5 when enabling tiered storage? maybe: enforce it once at the feature version level
[ https://issues.apache.org/jira/browse/KAFKA-17405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878461#comment-17878461 ] PoAn Yang commented on KAFKA-17405: --- Hi [~chia7712], I'm interested in this. If you're not working on it, may I take it? Thank you. > how to enforce the MV 3.5 when enabling tiered storage? maybe: enforce it > once at the feature version level > --- > > Key: KAFKA-17405 > URL: https://issues.apache.org/jira/browse/KAFKA-17405 > Project: Kafka > Issue Type: Wish >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > see discussion: > [https://github.com/apache/kafka/pull/16873#discussion_r1725621368] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17448) New consumer seek should update positions in background thread
[ https://issues.apache.org/jira/browse/KAFKA-17448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17448: - Assignee: PoAn Yang > New consumer seek should update positions in background thread > -- > > Key: KAFKA-17448 > URL: https://issues.apache.org/jira/browse/KAFKA-17448 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.7.0, 3.8.0, 3.7.1 >Reporter: Lianet Magrans >Assignee: PoAn Yang >Priority: Major > Labels: consumer-threading-refactor, kip-848-client-support > Fix For: 4.0.0 > > > In the new AsyncKafkaConsumer, a call to seek will update the positions in > subscription state for the assigned partitions in the app thread > ([https://github.com/apache/kafka/blob/c23b6b0365af5c58b76d8ad3fb628f766f95348f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L796]) > This could lead to race conditions like we've seen when subscription state > changes in the app thread (over a set of assigned partitions), that could > have been modified in the background thread, leading to errors on "No current > assignment for partition " > [https://github.com/apache/kafka/blob/c23b6b0365af5c58b76d8ad3fb628f766f95348f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java#L378] > > Also, positions update is moved the background with KAFKA-17066 for the same > reason, so even if the assignment does not change, we could have a race > between the background setting positions to the committed offsets for > instance, and the app thread setting them manually via seek. > To avoid all of the above, we should have seek generate an event, send it to > the background, and then update the subscription state when processing that > event (similar to other api calls, ex, assign with KAFKA-17064) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17448) New consumer seek should update positions in background thread
[ https://issues.apache.org/jira/browse/KAFKA-17448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878122#comment-17878122 ] PoAn Yang commented on KAFKA-17448: --- Hi [~lianetm], thank you. I can handle it. 👍 > New consumer seek should update positions in background thread > -- > > Key: KAFKA-17448 > URL: https://issues.apache.org/jira/browse/KAFKA-17448 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.7.0, 3.8.0, 3.7.1 >Reporter: Lianet Magrans >Priority: Major > Labels: consumer-threading-refactor, kip-848-client-support > Fix For: 4.0.0 > > > In the new AsyncKafkaConsumer, a call to seek will update the positions in > subscription state for the assigned partitions in the app thread > ([https://github.com/apache/kafka/blob/c23b6b0365af5c58b76d8ad3fb628f766f95348f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L796]) > This could lead to race conditions like we've seen when subscription state > changes in the app thread (over a set of assigned partitions), that could > have been modified in the background thread, leading to errors on "No current > assignment for partition " > [https://github.com/apache/kafka/blob/c23b6b0365af5c58b76d8ad3fb628f766f95348f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java#L378] > > Also, positions update is moved the background with KAFKA-17066 for the same > reason, so even if the assignment does not change, we could have a race > between the background setting positions to the committed offsets for > instance, and the app thread setting them manually via seek. > To avoid all of the above, we should have seek generate an event, send it to > the background, and then update the subscription state when processing that > event (similar to other api calls, ex, assign with KAFKA-17064) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17395) Flaky test testMissingOffsetNoResetPolicy for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-17395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17395: - Assignee: PoAn Yang > Flaky test testMissingOffsetNoResetPolicy for new consumer > -- > > Key: KAFKA-17395 > URL: https://issues.apache.org/jira/browse/KAFKA-17395 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: PoAn Yang >Priority: Major > Labels: consumer-threading-refactor, flaky-test > > KafkaConsumerTest.testMissingOffsetNoResetPolicy is flaky when running for > the new consumer (passing consistently for the classic consumer). > Fails with : > org.opentest4j.AssertionFailedError: Condition not met within timeout 15000. > Consumer was not able to update fetch positions on continuous calls with 0 > timeout ==> expected: but was: > It's been flaky since it was enabled for the new consumer with > [https://github.com/apache/kafka/pull/16587] > See last couple of month runs on trunk showing the flakiness: > [https://ge.apache.org/scans/tests?search.names=Git%20branch&search.rootProjectNames=kafka&search.startTimeMax=172429919&search.startTimeMin=171721440&search.timeZoneId=America%2FToronto&search.values=trunk&tests.container=org.apache.kafka.clients.consumer.KafkaConsumerTest&tests.test=testMissingOffsetNoResetPolicy(GroupProtocol)%5B2%5D] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17395) Flaky test testMissingOffsetNoResetPolicy for new consumer
[
https://issues.apache.org/jira/browse/KAFKA-17395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877989#comment-17877989
]
PoAn Yang commented on KAFKA-17395:
---
Hi [~taijuwu], yeah, like you said, if {{CoordinatorRequestManager}} sends
{{FindCoordinatorRequest}} before {{MockClient#prepareResponseFrom}}, the
response has no chance to match the request. However, refactor {{MockClient}}
is a big effort. I think we can just move {{MockClient#prepareResponseFrom}}
before {{newConsumer}}, so there will be response in list before the request. I
will file a PR later. Thanks.
> Flaky test testMissingOffsetNoResetPolicy for new consumer
> --
>
> Key: KAFKA-17395
> URL: https://issues.apache.org/jira/browse/KAFKA-17395
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Reporter: Lianet Magrans
>Priority: Major
> Labels: consumer-threading-refactor, flaky-test
>
> KafkaConsumerTest.testMissingOffsetNoResetPolicy is flaky when running for
> the new consumer (passing consistently for the classic consumer).
> Fails with :
> org.opentest4j.AssertionFailedError: Condition not met within timeout 15000.
> Consumer was not able to update fetch positions on continuous calls with 0
> timeout ==> expected: but was:
> It's been flaky since it was enabled for the new consumer with
> [https://github.com/apache/kafka/pull/16587]
> See last couple of month runs on trunk showing the flakiness:
> [https://ge.apache.org/scans/tests?search.names=Git%20branch&search.rootProjectNames=kafka&search.startTimeMax=172429919&search.startTimeMin=171721440&search.timeZoneId=America%2FToronto&search.values=trunk&tests.container=org.apache.kafka.clients.consumer.KafkaConsumerTest&tests.test=testMissingOffsetNoResetPolicy(GroupProtocol)%5B2%5D]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-17064) New consumer assign should update assignment in background thread
[ https://issues.apache.org/jira/browse/KAFKA-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17064. --- Resolution: Fixed > New consumer assign should update assignment in background thread > - > > Key: KAFKA-17064 > URL: https://issues.apache.org/jira/browse/KAFKA-17064 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.8.0 >Reporter: Lianet Magrans >Assignee: PoAn Yang >Priority: Blocker > Labels: consumer-threading-refactor, kip-848-client-support > Fix For: 4.0.0 > > > With the new async consumer, the subscriptionState object is shared between > the app thread and the background thread, but in principle all updates to the > assignment should happen in the background thread, to avoid race conditions. > Note that it's in the background where most updates to the assignment happen, > as result of app event processing like unsubscribe, reconciliations, etc.). > We've faced such races in places like unsubscribe and close, fixed by > ensuring that all assignment updates happen in the background, and this also > needs to be reviewed for the consumer.assign. The current implementation > triggers an AssignmentChange event that is processed in the background, but > that event is not really changing the assignment. It only commits offsets, > and the assignment is updated in the app thread by calling > subscriptionState.assignFromUser > We should consider moving the assignment update to the background thread, as > part of the AssignmentChangeEvent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17377) Consider using defaultApiTimeoutMs in AsyncKafkaConsumer#unsubscribe
[ https://issues.apache.org/jira/browse/KAFKA-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17377. --- Fix Version/s: 4.0.0 Resolution: Fixed > Consider using defaultApiTimeoutMs in AsyncKafkaConsumer#unsubscribe > > > Key: KAFKA-17377 > URL: https://issues.apache.org/jira/browse/KAFKA-17377 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Major > Labels: consumer-threading-refactor > Fix For: 4.0.0 > > > This is followup from > [https://github.com/apache/kafka/pull/16673#discussion_r1717009724] > AsyncKafkaConsumer#unsubscribe uses Long.MAX_VALUE to calculate deadline. > However, most of AsyncKafkaConsumer operations use > [default.api.timeout.ms|https://kafka.apache.org/documentation/#consumerconfigs_default.api.timeout.ms] > if users don't specify a timeout. In [design > document|https://cwiki.apache.org/confluence/display/KAFKA/Java+client+Consumer+timeouts#JavaclientConsumertimeouts-DefaultTimeouts], > it also mentions that using default.api.timeout.ms as default timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17384) Remove deprecated options in tools
[ https://issues.apache.org/jira/browse/KAFKA-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876664#comment-17876664 ] PoAn Yang commented on KAFKA-17384: --- Hi [~chia7712], I added related KIP to all subtasks. Thank you. > Remove deprecated options in tools > -- > > Key: KAFKA-17384 > URL: https://issues.apache.org/jira/browse/KAFKA-17384 > Project: Kafka > Issue Type: Improvement >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > There're deprecated options in following tools. We can consider to remove > them in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17392) Remove whitelist in ConsoleConsumerOptions
[ https://issues.apache.org/jira/browse/KAFKA-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17392: -- Description: The [whitelist|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L79-L83] is deprecated option in ConsoleConsumerOptions. We can consider to remove it in 4.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase was:The [whitelist|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L79-L83] is deprecated option in ConsoleConsumerOptions. We can consider to remove it in 4.0. > Remove whitelist in ConsoleConsumerOptions > -- > > Key: KAFKA-17392 > URL: https://issues.apache.org/jira/browse/KAFKA-17392 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: Ming-Yen Chung >Priority: Minor > > The > [whitelist|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L79-L83] > is deprecated option in ConsoleConsumerOptions. We can consider to remove it > in 4.0. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17388) Remove broker-list in VerifiableProducer
[ https://issues.apache.org/jira/browse/KAFKA-17388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17388: -- Description: The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java#L134-L140] is deprecated option in VerifiableProducer. We can consider to remove it in 4.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool was:The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java#L134-L140] is deprecated option in VerifiableProducer. We can consider to remove it in 4.0. > Remove broker-list in VerifiableProducer > > > Key: KAFKA-17388 > URL: https://issues.apache.org/jira/browse/KAFKA-17388 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: Logan Zhu >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java#L134-L140] > is deprecated option in VerifiableProducer. We can consider to remove it in > 4.0. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17387) Remove broker-list in VerifiableConsumer
[ https://issues.apache.org/jira/browse/KAFKA-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17387: -- Description: The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java#L522-L528] is deprecated option in VerifiableConsumer. We can consider to remove it in 4.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool was:The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java#L522-L528] is deprecated option in VerifiableConsumer. We can consider to remove it in 4.0. > Remove broker-list in VerifiableConsumer > > > Key: KAFKA-17387 > URL: https://issues.apache.org/jira/browse/KAFKA-17387 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: Eric Chang >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java#L522-L528] > is deprecated option in VerifiableConsumer. We can consider to remove it in > 4.0. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17386) Remove broker-list, threads, num-fetch-threads in ConsumerPerformance
[ https://issues.apache.org/jira/browse/KAFKA-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17386: -- Description: The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] are deprecated options in ConsumerPerformance. We can consider to remove them in 4.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool https://issues.apache.org/jira/browse/KAFKA-10126 was: The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] are deprecated options in ConsumerPerformance. We can consider to remove them in 4.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool > Remove broker-list, threads, num-fetch-threads in ConsumerPerformance > - > > Key: KAFKA-17386 > URL: https://issues.apache.org/jira/browse/KAFKA-17386 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], > > [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], > > [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] > are deprecated options in ConsumerPerformance. We can consider to remove > them in 4.0. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool > https://issues.apache.org/jira/browse/KAFKA-10126 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17386) Remove broker-list, threads, num-fetch-threads in ConsumerPerformance
[ https://issues.apache.org/jira/browse/KAFKA-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17386: -- Description: The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] are deprecated options in ConsumerPerformance. We can consider to remove them in 4.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool was:The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] are deprecated options in ConsumerPerformance. We can consider to remove them in 4.0. > Remove broker-list, threads, num-fetch-threads in ConsumerPerformance > - > > Key: KAFKA-17386 > URL: https://issues.apache.org/jira/browse/KAFKA-17386 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], > > [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], > > [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] > are deprecated options in ConsumerPerformance. We can consider to remove > them in 4.0. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17385) Remove authorizer, authorizer-properties, zk-tls-config-file in AclCommand
[ https://issues.apache.org/jira/browse/KAFKA-17385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17385: -- Summary: Remove authorizer, authorizer-properties, zk-tls-config-file in AclCommand (was: Remove authorizer in AclCommand) > Remove authorizer, authorizer-properties, zk-tls-config-file in AclCommand > -- > > Key: KAFKA-17385 > URL: https://issues.apache.org/jira/browse/KAFKA-17385 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > The > [authorizer|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L501-L506], > > [authorizer-properties|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L508-L514], > and > [zk-tls-config-file|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L610C63-L618] > are deprecated options. We can consider to remove it in 4.0. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-604%3A+Remove+ZooKeeper+Flags+from+the+Administrative+Tools] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17385) Remove authorizer in AclCommand
[ https://issues.apache.org/jira/browse/KAFKA-17385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17385: -- Description: The [authorizer|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L501-L506], [authorizer-properties|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L508-L514], and [zk-tls-config-file|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L610C63-L618] are deprecated options. We can consider to remove it in 4.0. [https://cwiki.apache.org/confluence/display/KAFKA/KIP-604%3A+Remove+ZooKeeper+Flags+from+the+Administrative+Tools] was: The authorizer is deprecated option. We can consider to remove it in 4.0. [https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L501-L506] > Remove authorizer in AclCommand > --- > > Key: KAFKA-17385 > URL: https://issues.apache.org/jira/browse/KAFKA-17385 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > The > [authorizer|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L501-L506], > > [authorizer-properties|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L508-L514], > and > [zk-tls-config-file|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L610C63-L618] > are deprecated options. We can consider to remove it in 4.0. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-604%3A+Remove+ZooKeeper+Flags+from+the+Administrative+Tools] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17417) Backport KAFKA-15751 and KAFKA-15752 to 3.8 and 3.7
[ https://issues.apache.org/jira/browse/KAFKA-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876465#comment-17876465 ] PoAn Yang commented on KAFKA-17417: --- Hi [~chia7712], if you're not work on this, may I assign to myself? Thank you. > Backport KAFKA-15751 and KAFKA-15752 to 3.8 and 3.7 > --- > > Key: KAFKA-17417 > URL: https://issues.apache.org/jira/browse/KAFKA-17417 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.7.2, 3.8.1 > > > KAFKA-15751 and KAFKA-15752 enable the kraft mode in > SaslSslAdminIntegrationTest, and that is useful in writing security-related > IT. Without the backport, the fixes to security could be obstructed from > backport due to IT (KAFKA-17315, for example) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17257) Migrate BrokerApiVersionsCommandTest to ClusterTestExtensions
[ https://issues.apache.org/jira/browse/KAFKA-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17257. --- Resolution: Done Close this Jira as it's already covered by [https://github.com/apache/kafka/pull/16850]. > Migrate BrokerApiVersionsCommandTest to ClusterTestExtensions > - > > Key: KAFKA-17257 > URL: https://issues.apache.org/jira/browse/KAFKA-17257 > Project: Kafka > Issue Type: Test >Reporter: PoAn Yang >Assignee: Chia Chuan Yu >Priority: Minor > > As title, we can migrate BrokerApiVersionsCommandTest to Java and use > ClusterTestExtensions framework. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16683) Extract security-related helpers from scala.TestUtils to java class
[ https://issues.apache.org/jira/browse/KAFKA-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876146#comment-17876146 ] PoAn Yang commented on KAFKA-16683: --- PR: [https://github.com/apache/kafka/pull/16912] > Extract security-related helpers from scala.TestUtils to java class > --- > > Key: KAFKA-16683 > URL: https://issues.apache.org/jira/browse/KAFKA-16683 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Minor > > We can merge them into `JaasTestUtils and then rename `JaasTestUtils` to > `SecurityTestUtils. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[ https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875883#comment-17875883 ] PoAn Yang commented on KAFKA-16792: --- Hi [~lianetm] and [~kirktrue], I did some investigation of testCurrentLag and testListOffsetShouldUpdateSubscriptions. Both these tests fail in ` client.respond(FindCoordinatorResponse.prepareResponse(...))`. In `ClassicKafkaConsumer#poll`, it always sends `FindCoordinatorRequest`. However, in `AsyncKafkaConsumer`, it sends `FindCoordinatorRequest` in `CoordinatorRequestManager` and uses `ConsumerNetworkThread` to trigger the request. The `ConsumerNetworkThread` is in different thread, so we don't guarantee `AsyncKafkaConsumer#poll` always sends `FindCoordinatorRequest`. My question is that can we just add `TestUtils#waitForCondition` to wait if there is `FindCoordinatorRequest` in `MockClient`? Or we want to align behavior between ClassicKafkaConsumer and AsyncKafkaConsumer? Aligning behavior may need some refactor. > Enable consumer unit tests that fail to fetch offsets only for new consumer > with poll(0) > > > Key: KAFKA-16792 > URL: https://issues.apache.org/jira/browse/KAFKA-16792 > Project: Kafka > Issue Type: Test > Components: clients, consumer, unit tests >Reporter: Lianet Magrans >Assignee: PoAn Yang >Priority: Blocker > Labels: kip-848-client-support > Fix For: 4.0.0 > > > Enable the following unit tests for the new async consumer in > KafkaConsumerTest: > - testCurrentLag > - testFetchStableOffsetThrowInPoll > - testListOffsetShouldUpdateSubscriptions > - testPollReturnsRecords > - testResetToCommittedOffset > - testResetUsingAutoResetPolicy > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17391) Remove bootstrap-servers in StreamsResetter
[ https://issues.apache.org/jira/browse/KAFKA-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17391. --- Resolution: Duplicate > Remove bootstrap-servers in StreamsResetter > --- > > Key: KAFKA-17391 > URL: https://issues.apache.org/jira/browse/KAFKA-17391 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: xuanzhang gong >Priority: Minor > > The > [bootstrap-servers|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L573C54-L576] > is a deprecated option in StreamsResetter. We can consider to remove it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17391) Remove bootstrap-servers in StreamsResetter
[ https://issues.apache.org/jira/browse/KAFKA-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875733#comment-17875733 ] PoAn Yang commented on KAFKA-17391: --- Hi [~mjsax], yeah, the content looks similar. Sorry for that, I will close this one. [~gongxuanzhang], if you're interested in this, probably you can leave message in https://issues.apache.org/jira/browse/KAFKA-16334. Thanks. > Remove bootstrap-servers in StreamsResetter > --- > > Key: KAFKA-17391 > URL: https://issues.apache.org/jira/browse/KAFKA-17391 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: xuanzhang gong >Priority: Minor > > The > [bootstrap-servers|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L573C54-L576] > is a deprecated option in StreamsResetter. We can consider to remove it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17388) Remove broker-list in VerifiableProducer
[ https://issues.apache.org/jira/browse/KAFKA-17388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17388: - Assignee: Logan Zhu (was: PoAn Yang) > Remove broker-list in VerifiableProducer > > > Key: KAFKA-17388 > URL: https://issues.apache.org/jira/browse/KAFKA-17388 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: Logan Zhu >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java#L134-L140] > is deprecated option in VerifiableProducer. We can consider to remove it in > 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17391) Remove bootstrap-servers in StreamsResetter
[ https://issues.apache.org/jira/browse/KAFKA-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17391: - Assignee: xuanzhang gong (was: PoAn Yang) > Remove bootstrap-servers in StreamsResetter > --- > > Key: KAFKA-17391 > URL: https://issues.apache.org/jira/browse/KAFKA-17391 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: xuanzhang gong >Priority: Minor > > The > [bootstrap-servers|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L573C54-L576] > is a deprecated option in StreamsResetter. We can consider to remove it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17390) Remove broker-list in GetOffsetShell
[ https://issues.apache.org/jira/browse/KAFKA-17390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17390: - Assignee: TaiJuWu (was: PoAn Yang) > Remove broker-list in GetOffsetShell > > > Key: KAFKA-17390 > URL: https://issues.apache.org/jira/browse/KAFKA-17390 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: TaiJuWu >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/GetOffsetShell.java#L110-L113] > is deprecated option in GetOffsetShell. We can consider to remove it in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17392) Remove whitelist in ConsoleConsumerOptions
[ https://issues.apache.org/jira/browse/KAFKA-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17392: - Assignee: Ming-Yen Chung (was: PoAn Yang) > Remove whitelist in ConsoleConsumerOptions > -- > > Key: KAFKA-17392 > URL: https://issues.apache.org/jira/browse/KAFKA-17392 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: Ming-Yen Chung >Priority: Minor > > The > [whitelist|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L79-L83] > is deprecated option in ConsoleConsumerOptions. We can consider to remove it > in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17387) Remove broker-list in VerifiableConsumer
[ https://issues.apache.org/jira/browse/KAFKA-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-17387: - Assignee: Eric Chang (was: PoAn Yang) > Remove broker-list in VerifiableConsumer > > > Key: KAFKA-17387 > URL: https://issues.apache.org/jira/browse/KAFKA-17387 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: Eric Chang >Priority: Minor > > The > [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java#L522-L528] > is deprecated option in VerifiableConsumer. We can consider to remove it in > 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17395) Flaky test testMissingOffsetNoResetPolicy for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-17395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875687#comment-17875687 ] PoAn Yang commented on KAFKA-17395: --- Thanks for raising the issue. I will check it as well. > Flaky test testMissingOffsetNoResetPolicy for new consumer > -- > > Key: KAFKA-17395 > URL: https://issues.apache.org/jira/browse/KAFKA-17395 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Reporter: Lianet Magrans >Priority: Major > Labels: consumer-threading-refactor > > KafkaConsumerTest.testMissingOffsetNoResetPolicy is flaky when running for > the new consumer (passing consistently for the classic consumer). > Fails with : > org.opentest4j.AssertionFailedError: Condition not met within timeout 15000. > Consumer was not able to update fetch positions on continuous calls with 0 > timeout ==> expected: but was: > It's been flaky since it was enabled for the new consumer with > [https://github.com/apache/kafka/pull/16587] > See last couple of month runs on trunk showing the flakiness: > [https://ge.apache.org/scans/tests?search.names=Git%20branch&search.rootProjectNames=kafka&search.startTimeMax=172429919&search.startTimeMin=171721440&search.timeZoneId=America%2FToronto&search.values=trunk&tests.container=org.apache.kafka.clients.consumer.KafkaConsumerTest&tests.test=testMissingOffsetNoResetPolicy(GroupProtocol)%5B2%5D] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17389) Remove ReplicaVerificationTool
[ https://issues.apache.org/jira/browse/KAFKA-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17389. --- Resolution: Duplicate > Remove ReplicaVerificationTool > -- > > Key: KAFKA-17389 > URL: https://issues.apache.org/jira/browse/KAFKA-17389 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > The ReplicaVerificationTool is deprecated. We can consider to remove it in > 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17389) Remove ReplicaVerificationTool
[ https://issues.apache.org/jira/browse/KAFKA-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875406#comment-17875406 ] PoAn Yang commented on KAFKA-17389: --- Sorry. I miss that. I will close this issue. Thanks for the reminder. > Remove ReplicaVerificationTool > -- > > Key: KAFKA-17389 > URL: https://issues.apache.org/jira/browse/KAFKA-17389 > Project: Kafka > Issue Type: Sub-task >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > The ReplicaVerificationTool is deprecated. We can consider to remove it in > 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17393) Remove message.format.version and message.timestamp.difference.max.ms in TopicConfig
PoAn Yang created KAFKA-17393: - Summary: Remove message.format.version and message.timestamp.difference.max.ms in TopicConfig Key: KAFKA-17393 URL: https://issues.apache.org/jira/browse/KAFKA-17393 Project: Kafka Issue Type: Improvement Reporter: PoAn Yang Assignee: PoAn Yang Both [message.format.version|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java#L199-L217] and [message.timestamp.difference.max.ms|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java#L223-L238] are deprecated config. We can consider to remove them in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17391) Remove bootstrap-servers in StreamsResetter
PoAn Yang created KAFKA-17391: - Summary: Remove bootstrap-servers in StreamsResetter Key: KAFKA-17391 URL: https://issues.apache.org/jira/browse/KAFKA-17391 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The [bootstrap-servers|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/StreamsResetter.java#L573C54-L576] is a deprecated option in StreamsResetter. We can consider to remove it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17392) Remove whitelist in ConsoleConsumerOptions
PoAn Yang created KAFKA-17392: - Summary: Remove whitelist in ConsoleConsumerOptions Key: KAFKA-17392 URL: https://issues.apache.org/jira/browse/KAFKA-17392 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The [whitelist|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L79-L83] is deprecated option in ConsoleConsumerOptions. We can consider to remove it in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17390) Remove broker-list in GetOffsetShell
PoAn Yang created KAFKA-17390: - Summary: Remove broker-list in GetOffsetShell Key: KAFKA-17390 URL: https://issues.apache.org/jira/browse/KAFKA-17390 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/GetOffsetShell.java#L110-L113] is deprecated option in GetOffsetShell. We can consider to remove it in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17389) Remove ReplicaVerificationTool
PoAn Yang created KAFKA-17389: - Summary: Remove ReplicaVerificationTool Key: KAFKA-17389 URL: https://issues.apache.org/jira/browse/KAFKA-17389 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The ReplicaVerificationTool is deprecated. We can consider to remove it in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17387) Remove broker-list in VerifiableConsumer
PoAn Yang created KAFKA-17387: - Summary: Remove broker-list in VerifiableConsumer Key: KAFKA-17387 URL: https://issues.apache.org/jira/browse/KAFKA-17387 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java#L522-L528] is deprecated option in VerifiableConsumer. We can consider to remove it in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17388) Remove broker-list in VerifiableProducer
PoAn Yang created KAFKA-17388: - Summary: Remove broker-list in VerifiableProducer Key: KAFKA-17388 URL: https://issues.apache.org/jira/browse/KAFKA-17388 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java#L134-L140] is deprecated option in VerifiableProducer. We can consider to remove it in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17386) Remove broker-list, threads, num-fetch-threads in ConsumerPerformance
PoAn Yang created KAFKA-17386: - Summary: Remove broker-list, threads, num-fetch-threads in ConsumerPerformance Key: KAFKA-17386 URL: https://issues.apache.org/jira/browse/KAFKA-17386 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The [broker-list|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L268C45-L271], [threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L298-L302], [num-fetch-threads|https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java#L303-L307] are deprecated options in ConsumerPerformance. We can consider to remove them in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17385) Remove authorizer in AclCommand
PoAn Yang created KAFKA-17385: - Summary: Remove authorizer in AclCommand Key: KAFKA-17385 URL: https://issues.apache.org/jira/browse/KAFKA-17385 Project: Kafka Issue Type: Sub-task Reporter: PoAn Yang Assignee: PoAn Yang The authorizer is deprecated option. We can consider to remove it in 4.0. [https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/core/src/main/scala/kafka/admin/AclCommand.scala#L501-L506] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17384) Remove deprecated options in tools
PoAn Yang created KAFKA-17384: - Summary: Remove deprecated options in tools Key: KAFKA-17384 URL: https://issues.apache.org/jira/browse/KAFKA-17384 Project: Kafka Issue Type: Improvement Reporter: PoAn Yang Assignee: PoAn Yang There're deprecated options in following tools. We can consider to remove them in 4.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17381) Reduce log output from checkstyle task
[
https://issues.apache.org/jira/browse/KAFKA-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875306#comment-17875306
]
PoAn Yang commented on KAFKA-17381:
---
Hi [~davidarthur], I'm interested in this issue. If you're not working on it,
may I take it? Thank you.
> Reduce log output from checkstyle task
> --
>
> Key: KAFKA-17381
> URL: https://issues.apache.org/jira/browse/KAFKA-17381
> Project: Kafka
> Issue Type: Task
> Components: build
>Reporter: David Arthur
>Priority: Minor
>
> When running the "check -x tests" Gradle task, I am noticing a lot of the
> output is some XSLT thing.
> {code:java}
> [ant:xslt] Processing
> /home/runner/work/kafka/kafka/streams/upgrade-system-tests-26/build/reports/checkstyle/test.xml
> to
> /home/runner/work/kafka/kafka/streams/upgrade-system-tests-26/build/reports/checkstyle/test.html
> 6374[ant:xslt] Loading stylesheet
> 6375
> 6389 xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
> xmlns:xl="http://www.w3.org/1999/XSL/Transform";>
> 6390
> 6391
> 6392
> 6393
> 6394
> 6395
> 6396Checkstyle Violations
> 6397
> 6398
> 6399body {
> 6400background-color: #fff;
> 6401color: #02303A;
> 6402}
> 6403a {
> 6404color: #1DA2BD;
> 6405}
> 6406a.link {
> 6407color: #02303A;
> 6408}
> 6409p {
> 6410font-size: 1rem;
> 6411}
> 6412h1 a[name] {
> 6413margin: 0;
> 6414padding: 0;
> 6415}
> 6416tr:nth-child(even) {
> 6417background: white;
> 6418}
> 6419th {
> 6420font-weight:bold;
> 6421}
> 6422tr {
> 6423background: #efefef;
> 6424}
> 6425table th, td, tr {
> 6426font-size:100%;
> 6427border: none;
> 6428text-align: left;
> 6429vertical-align: top;
> 6430}
> 6431
> 6432
> 6433
> 6434
> 6435Checkstyle Results
> 6436
> 6437
> 6438Summary
> 6439
> 6440
> 6441Total files checked
> 6442Total violations
> 6443Files with violations
> 6444
> 6445
> 6446
> 6447 value="count(descendant::file)"/>
> 6448
> 6449
> 6450 value="count(descendant::error)"/>
> 6451
> 6452
> 6453Error: value="count(descendant::file[error])"/>
> 6454
> 6455
> 6456
> 6457
> 6458
> 6459Violations
> 6460
> 6461
> 6462
> 6463
> 6464
> 6465File
> 6475Total violations
> 6476
> 6477Error:
> 6478
> 6479 order="descending" select="count(descendant::error)"/>
> 6480
> 6481 select="count(descendant::error)"/>
> 6482
> 6483 href="#{generate-id(@name)}">
> 6484 select="$errors"/>
> 6485
> 6486
> 6487
> 6488
> 6489
> 6490
> 6491
> 6492
> 6493
> 6494
> 6495
> 6496No violations were found.
> 6497
> 6498
[jira] [Commented] (KAFKA-15667) preCheck the invalid configuration for tiered storage replication factor
[ https://issues.apache.org/jira/browse/KAFKA-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875154#comment-17875154 ] PoAn Yang commented on KAFKA-15667: --- Hi [~showuon], is this issue still valid? If yes, may I assign to myself? Thank you. > preCheck the invalid configuration for tiered storage replication factor > > > Key: KAFKA-15667 > URL: https://issues.apache.org/jira/browse/KAFKA-15667 > Project: Kafka > Issue Type: Improvement > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Luke Chen >Priority: Major > > `remote.log.metadata.topic.replication.factor` is a config to set the > Replication factor of remote log metadata topic. For the > `min.insync.replicas`, we'll use the broker config. Today, if the > `remote.log.metadata.topic.replication.factor` < `min.insync.replicas` value, > everything still works until new remote log metadata records created. We > should be able to identify it when broker startup to notify users to fix the > invalid config. > ref: > https://kafka.apache.org/documentation/#remote_log_metadata_manager_remote.log.metadata.topic.replication.factor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17377) Consider using defaultApiTimeoutMs in AsyncKafkaConsumer#unsubscribe
PoAn Yang created KAFKA-17377: - Summary: Consider using defaultApiTimeoutMs in AsyncKafkaConsumer#unsubscribe Key: KAFKA-17377 URL: https://issues.apache.org/jira/browse/KAFKA-17377 Project: Kafka Issue Type: Improvement Reporter: PoAn Yang Assignee: PoAn Yang This is followup from [https://github.com/apache/kafka/pull/16673#discussion_r1717009724] AsyncKafkaConsumer#unsubscribe uses Long.MAX_VALUE to calculate deadline. However, most of AsyncKafkaConsumer operations use [default.api.timeout.ms|https://kafka.apache.org/documentation/#consumerconfigs_default.api.timeout.ms] if users don't specify a timeout. In [design document|https://cwiki.apache.org/confluence/display/KAFKA/Java+client+Consumer+timeouts#JavaclientConsumertimeouts-DefaultTimeouts], it also mentions that using default.api.timeout.ms as default timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17331) Throw/return unsupported version exception if the server/client does NOT support EarliestLocalSpec and LatestTieredSpec
[ https://issues.apache.org/jira/browse/KAFKA-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873194#comment-17873194 ] PoAn Yang commented on KAFKA-17331: --- Hi [~chia7712], I'm interested in this issue. If you're not working on it, may I take it? Thank you. > Throw/return unsupported version exception if the server/client does NOT > support EarliestLocalSpec and LatestTieredSpec > --- > > Key: KAFKA-17331 > URL: https://issues.apache.org/jira/browse/KAFKA-17331 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Blocker > Fix For: 3.9.0 > > > see comment: https://github.com/apache/kafka/pull/16841#discussion_r1714476438 > the version must be >=8 if timestamp=-4L and the version must be >=9 if > timestamp=-5 > *client-side* > it will throw exception in building request[0]. > *server-side* > it will return UNSUPPORTED_VERSION in processing ListOffsetsRequest [1] > [0] > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java#L142 > [1] > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1173 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15909) Throw error when consumer configured with empty/whitespace-only group.id for LegacyKafkaConsumer
[
https://issues.apache.org/jira/browse/KAFKA-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
PoAn Yang reassigned KAFKA-15909:
-
Assignee: PoAn Yang
> Throw error when consumer configured with empty/whitespace-only group.id for
> LegacyKafkaConsumer
>
>
> Key: KAFKA-15909
> URL: https://issues.apache.org/jira/browse/KAFKA-15909
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: PoAn Yang
>Priority: Major
> Fix For: 4.0.0
>
>
> Per
> [KIP-289|https://cwiki.apache.org/confluence/display/KAFKA/KIP-289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer],
> the use of an empty value for {{group.id}} configuration was deprecated back
> in 2.2.0.
> In 3.7, the {{AsyncKafkaConsumer}} implementation will throw an error (see
> KAFKA-14438).
> This task is to update the {{LegacyKafkaConsumer}} implementation to throw an
> error in 4.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[ https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang reassigned KAFKA-16792: - Assignee: PoAn Yang > Enable consumer unit tests that fail to fetch offsets only for new consumer > with poll(0) > > > Key: KAFKA-16792 > URL: https://issues.apache.org/jira/browse/KAFKA-16792 > Project: Kafka > Issue Type: Test > Components: clients, consumer, unit tests >Reporter: Lianet Magrans >Assignee: PoAn Yang >Priority: Blocker > Labels: kip-848-client-support > Fix For: 4.0.0 > > > Enable the following unit tests for the new async consumer in > KafkaConsumerTest: > - testCurrentLag > - testFetchStableOffsetThrowInPoll > - testListOffsetShouldUpdateSubscriptions > - testPollReturnsRecords > - testResetToCommittedOffset > - testResetUsingAutoResetPolicy > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17312) Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
[
https://issues.apache.org/jira/browse/KAFKA-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872702#comment-17872702
]
PoAn Yang commented on KAFKA-17312:
---
Sounds good to me. I create a PR for it. Thank you.
> Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods
> in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> ---
>
> Key: KAFKA-17312
> URL: https://issues.apache.org/jira/browse/KAFKA-17312
> Project: Kafka
> Issue Type: Sub-task
>Reporter: PoAn Yang
>Assignee: PoAn Yang
>Priority: Major
>
> CI:
> [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/3189/cloudbees-pipeline-explorer/]
>
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
> {code:java}
> org.opentest4j.AssertionFailedError: Should have called
> DBOptions.maxBackgroundCompactions() ==> expected: but was:
> at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:148)
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:131)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at java.util.ArrayList.forEach(ArrayList.java:1259) {code}
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldOverwriteAllOptionsMethods
> {code:java}
> java.lang.NoSuchMethodException:
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.maxBackgroundCompactions()
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldOverwriteAllOptionsMethods(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:120)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at java.util.ArrayList.forEach(ArrayList.java:1259) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-17312) Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
[
https://issues.apache.org/jira/browse/KAFKA-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
PoAn Yang updated KAFKA-17312:
--
Description:
CI:
[https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/3189/cloudbees-pipeline-explorer/]
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
{code:java}
org.opentest4j.AssertionFailedError: Should have called
DBOptions.maxBackgroundCompactions() ==> expected: but was:
at
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
at
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:148)
at
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:131)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259) {code}
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldOverwriteAllOptionsMethods
{code:java}
java.lang.NoSuchMethodException:
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.maxBackgroundCompactions()
at java.lang.Class.getDeclaredMethod(Class.java:2130)
at
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldOverwriteAllOptionsMethods(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:120)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259) {code}
was:
CI:
[https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/3189/cloudbees-pipeline-explorer/]
> Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods
> in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> ---
>
> Key: KAFKA-17312
> URL: https://issues.apache.org/jira/browse/KAFKA-17312
> Project: Kafka
> Issue Type: Test
>Reporter: PoAn Yang
>Assignee: PoAn Yang
>Priority: Major
>
> CI:
> [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/3189/cloudbees-pipeline-explorer/]
>
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
> {code:java}
> org.opentest4j.AssertionFailedError: Should have called
> DBOptions.maxBackgroundCompactions() ==> expected: but was:
> at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:148)
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:131)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at java.util.ArrayList.forEach(ArrayList.java:1259) {code}
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldOverwriteAllOptionsMethods
> {code:java}
> java.lang.NoSuchMethodException:
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.maxBackgroundCompactions()
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldOverwriteAllOptionsMethods(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:120)
> at java.lang.reflect.M
[jira] [Commented] (KAFKA-17312) Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
[ https://issues.apache.org/jira/browse/KAFKA-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872652#comment-17872652 ] PoAn Yang commented on KAFKA-17312: --- It looks like the root cause is https://issues.apache.org/jira/browse/KAFKA-12832 removing deprecated methods. Do we want to upgrade rocksdb version to avoid these functions as well? > Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods > in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest > --- > > Key: KAFKA-17312 > URL: https://issues.apache.org/jira/browse/KAFKA-17312 > Project: Kafka > Issue Type: Test >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Major > > CI: > [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/3189/cloudbees-pipeline-explorer/] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17312) Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
PoAn Yang created KAFKA-17312: - Summary: Fix tests shouldForwardAllDbOptionsCalls and shouldOverwriteAllOptionsMethods in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest Key: KAFKA-17312 URL: https://issues.apache.org/jira/browse/KAFKA-17312 Project: Kafka Issue Type: Test Reporter: PoAn Yang Assignee: PoAn Yang CI: [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/3189/cloudbees-pipeline-explorer/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15969) Align RemoteStorageThreadPool metrics name with KIP-405
[
https://issues.apache.org/jira/browse/KAFKA-15969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872540#comment-17872540
]
PoAn Yang commented on KAFKA-15969:
---
The document part also uses `org.apache.kafka.storage.internals.log`. Not sure
whether we want to fix this. If yes, I can create a KIP for it. Thanks.
[https://kafka.apache.org/documentation/#tiered_storage_monitoring]
> Align RemoteStorageThreadPool metrics name with KIP-405
> ---
>
> Key: KAFKA-15969
> URL: https://issues.apache.org/jira/browse/KAFKA-15969
> Project: Kafka
> Issue Type: Bug
> Components: metrics
>Reporter: Lixin Yao
>Priority: Minor
> Fix For: 4.0.0
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> In KIP-405, there are 2 metrics defined below:
> ^kafka.log.remote:type=RemoteStorageThreadPool,
> name=RemoteLogReaderTaskQueueSize^
> and
> ^kafka.log.remote:type=RemoteStorageThreadPool,
> name=RemoteLogReaderAvgIdlePercent^
> However, in Kafka 3.6 release, the actual metrics exposes are:
> ^org.apache.kafka.storage.internals.log:name=RemoteLogReaderAvgIdlePercent,type=RemoteStorageThreadPool^
> ^org.apache.kafka.storage.internals.log:name=RemoteLogReaderTaskQueueSize,type=RemoteStorageThreadPool^
> The problem is the bean domain name is changed from ^{{kafka.log.remote}}^ to
> {{{}^org.apache.kafka.storage.internals.log^{}}}. And the type name is also
> changed.
> We should either update the metrics path in KIP, or fix the path in the code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15909) Throw error when consumer configured with empty/whitespace-only group.id for LegacyKafkaConsumer
[
https://issues.apache.org/jira/browse/KAFKA-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872532#comment-17872532
]
PoAn Yang commented on KAFKA-15909:
---
Hi [~kirktrue], if you're not working on this issue, may I take it? Thank you.
> Throw error when consumer configured with empty/whitespace-only group.id for
> LegacyKafkaConsumer
>
>
> Key: KAFKA-15909
> URL: https://issues.apache.org/jira/browse/KAFKA-15909
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, consumer
>Reporter: Kirk True
>Priority: Major
> Fix For: 4.0.0
>
>
> Per
> [KIP-289|https://cwiki.apache.org/confluence/display/KAFKA/KIP-289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer],
> the use of an empty value for {{group.id}} configuration was deprecated back
> in 2.2.0.
> In 3.7, the {{AsyncKafkaConsumer}} implementation will throw an error (see
> KAFKA-14438).
> This task is to update the {{LegacyKafkaConsumer}} implementation to throw an
> error in 4.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-12989) MockClient should respect the request matcher passed to prepareUnsupportedVersionResponse
[
https://issues.apache.org/jira/browse/KAFKA-12989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
PoAn Yang reassigned KAFKA-12989:
-
Assignee: PoAn Yang
> MockClient should respect the request matcher passed to
> prepareUnsupportedVersionResponse
> -
>
> Key: KAFKA-12989
> URL: https://issues.apache.org/jira/browse/KAFKA-12989
> Project: Kafka
> Issue Type: Bug
>Reporter: David Jacot
>Assignee: PoAn Yang
>Priority: Major
>
> The {{MockClient}} offers the possibility to prepare an unsupported version
> response with {{MockClient#prepareUnsupportedVersionResponse}}. That method
> accepts a {{RequestMatcher}} but it is never applied. It should be or we
> should remove the matcher from the method.
> {code:java}
> UnsupportedVersionException unsupportedVersionException = null;
> if (futureResp.isUnsupportedRequest) {
> unsupportedVersionException = new UnsupportedVersionException(
> "Api " + request.apiKey() + " with version " +
> version);
> } else {
> AbstractRequest abstractRequest =
> request.requestBuilder().build(version);
> if (!futureResp.requestMatcher.matches(abstractRequest))
> throw new IllegalStateException("Request matcher did not
> match next-in-line request "
> + abstractRequest + " with prepared response " +
> futureResp.responseBody);
> }
> ClientResponse resp = new
> ClientResponse(request.makeHeader(version), request.callback(),
> request.destination(),
> request.createdTimeMs(), time.milliseconds(),
> futureResp.disconnected,
> unsupportedVersionException, null,
> futureResp.responseBody);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16818) Move event processing-related tests from ConsumerNetworkThreadTest to ApplicationEventProcessorTest
[
https://issues.apache.org/jira/browse/KAFKA-16818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872530#comment-17872530
]
PoAn Yang commented on KAFKA-16818:
---
Hi [~kirktrue], it looks like we need to move testApplicationEventIsProcessed,
testListOffsetsEventIsProcessed, and testResetPositionsProcessFailureIsIgnored
to ApplicationEventProcessorTest. If it's correct, may I take the issue? Thank
you.
> Move event processing-related tests from ConsumerNetworkThreadTest to
> ApplicationEventProcessorTest
> ---
>
> Key: KAFKA-16818
> URL: https://issues.apache.org/jira/browse/KAFKA-16818
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer, unit tests
>Affects Versions: 3.8.0
>Reporter: Kirk True
>Priority: Minor
> Labels: consumer-threading-refactor
> Fix For: 4.0.0
>
>
> The {{ConsumerNetworkThreadTest}} currently has a number of tests which do
> the following:
> # Add event of type _T_ to the event queue
> # Call {{ConsumerNetworkThread.runOnce()}} to dequeue the events and call
> {{ApplicationEventProcessor.process()}}
> # Verify that the appropriate {{ApplicationEventProcessor}} process method
> was invoked for the event
> Those types of tests should be moved to {{{}ApplicationEventProcessorTest{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[ https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872528#comment-17872528 ] PoAn Yang edited comment on KAFKA-16792 at 8/10/24 1:53 AM: https://issues.apache.org/jira/browse/KAFKA-17129 enable consumer group protocol for testResetToCommittedOffset and testResetUsingAutoResetPolicy, so remaining items are: - testCurrentLag - testFetchStableOffsetThrowInPoll - testListOffsetShouldUpdateSubscriptions - testPollReturnsRecords I can help to work on remaining items if there is no assignee. was (Author: JIRAUSER300229): https://issues.apache.org/jira/browse/KAFKA-17129 enable consumer group protocol for testResetToCommittedOffset and testResetUsingAutoResetPolicy, so remaining items are: - testCurrentLag - testFetchStableOffsetThrowInPoll - testListOffsetShouldUpdateSubscriptions - testPollReturnsRecords > Enable consumer unit tests that fail to fetch offsets only for new consumer > with poll(0) > > > Key: KAFKA-16792 > URL: https://issues.apache.org/jira/browse/KAFKA-16792 > Project: Kafka > Issue Type: Test > Components: clients, consumer, unit tests >Reporter: Lianet Magrans >Priority: Blocker > Labels: kip-848-client-support > Fix For: 4.0.0 > > > Enable the following unit tests for the new async consumer in > KafkaConsumerTest: > - testCurrentLag > - testFetchStableOffsetThrowInPoll > - testListOffsetShouldUpdateSubscriptions > - testPollReturnsRecords > - testResetToCommittedOffset > - testResetUsingAutoResetPolicy > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16792) Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0)
[ https://issues.apache.org/jira/browse/KAFKA-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872528#comment-17872528 ] PoAn Yang commented on KAFKA-16792: --- https://issues.apache.org/jira/browse/KAFKA-17129 enable consumer group protocol for testResetToCommittedOffset and testResetUsingAutoResetPolicy, so remaining items are: - testCurrentLag - testFetchStableOffsetThrowInPoll - testListOffsetShouldUpdateSubscriptions - testPollReturnsRecords > Enable consumer unit tests that fail to fetch offsets only for new consumer > with poll(0) > > > Key: KAFKA-16792 > URL: https://issues.apache.org/jira/browse/KAFKA-16792 > Project: Kafka > Issue Type: Test > Components: clients, consumer, unit tests >Reporter: Lianet Magrans >Priority: Blocker > Labels: kip-848-client-support > Fix For: 4.0.0 > > > Enable the following unit tests for the new async consumer in > KafkaConsumerTest: > - testCurrentLag > - testFetchStableOffsetThrowInPoll > - testListOffsetShouldUpdateSubscriptions > - testPollReturnsRecords > - testResetToCommittedOffset > - testResetUsingAutoResetPolicy > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17257) Migrate BrokerApiVersionsCommandTest to ClusterTestExtensions
[ https://issues.apache.org/jira/browse/KAFKA-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872224#comment-17872224 ] PoAn Yang commented on KAFKA-17257: --- Hi [~chiacyu], if you're not working on this, may I take it? I would like to do this with https://issues.apache.org/jira/browse/KAFKA-16324. It's production code migration. > Migrate BrokerApiVersionsCommandTest to ClusterTestExtensions > - > > Key: KAFKA-17257 > URL: https://issues.apache.org/jira/browse/KAFKA-17257 > Project: Kafka > Issue Type: Test >Reporter: PoAn Yang >Assignee: Chia Chuan Yu >Priority: Minor > > As title, we can migrate BrokerApiVersionsCommandTest to Java and use > ClusterTestExtensions framework. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16324) Move BrokerApiVersionsCommand to tools
[ https://issues.apache.org/jira/browse/KAFKA-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871646#comment-17871646 ] PoAn Yang commented on KAFKA-16324: --- Hi [~mimaison] and [~chia7712], we can start to implement 4.0 issue. I would like to check whether we want to remove `BrokerApiVersionsCommand` or move it to tools? IMHO, if we want to deprecate it, we may need to add warning message and remove it in 5.0. Thanks. > Move BrokerApiVersionsCommand to tools > -- > > Key: KAFKA-16324 > URL: https://issues.apache.org/jira/browse/KAFKA-16324 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Major > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/BrokerApiVersionsCommand.scala -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17273) Migrate BootstrapControllersIntegrationTest to use ClusterTestExtensions
[ https://issues.apache.org/jira/browse/KAFKA-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang updated KAFKA-17273: -- Summary: Migrate BootstrapControllersIntegrationTest to use ClusterTestExtensions (was: Migrate BootstrapControllersIntegrationTest to use ClusterTestExtention) > Migrate BootstrapControllersIntegrationTest to use ClusterTestExtensions > > > Key: KAFKA-17273 > URL: https://issues.apache.org/jira/browse/KAFKA-17273 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Minor > > The other sub tasks need to define custom configs for broker/controller, and > hence using `ClusterTestExtention` is more suitable to this test class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17271) Refactor the section "Reverting to ZooKeeper mode During the Migration"
[ https://issues.apache.org/jira/browse/KAFKA-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871369#comment-17871369 ] PoAn Yang commented on KAFKA-17271: --- Hi [~chia7712], I have an idea for this issue: * For "Final Migration Section Completed", put it in title. * For "Directions for Reverting", use list to put them under each title. * For "Notes", use a _*NOTE*_ section to list each point. Example: h4. Preparing for migration * The preparation section does not involve leaving ZooKeeper mode. So there is nothing to do in the case of a revert. h4. Provisioning the KRaft controller quorum * Deprovision the KRaft controller quorum. * Then you are done. h4. Enter Migration Mode on the brokers * Deprovision the KRaft controller quorum. * xxx. _*NOTE*_ * It is important to perform the xxx. > Refactor the section "Reverting to ZooKeeper mode During the Migration" > --- > > Key: KAFKA-17271 > URL: https://issues.apache.org/jira/browse/KAFKA-17271 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > see the discussion: > https://github.com/apache/kafka-site/pull/620/files#r1705624608 > The "table" format is not good choice to show a bunch of "words". Also, the > "same" content should have a specific hyperlink to be shared between > migration phases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17264) Audit validation of the new RPCs (AddVoter, RemoveVoter and UpdateVoter)
[ https://issues.apache.org/jira/browse/KAFKA-17264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871050#comment-17871050 ] PoAn Yang commented on KAFKA-17264: --- Hi [~jsancio], I'm interested in this issue. May you share which kind of inputs may throw uncaught exceptions? I can try to add validation and related test cases. Thank you. > Audit validation of the new RPCs (AddVoter, RemoveVoter and UpdateVoter) > > > Key: KAFKA-17264 > URL: https://issues.apache.org/jira/browse/KAFKA-17264 > Project: Kafka > Issue Type: Sub-task >Reporter: José Armando García Sancio >Priority: Major > > It is possible the the kafka raft client driver thread throws uncaught > exceptions for some inputs. KRaft should instead be catching this validation > errors and returning INVALID_REQUEST errors in the response. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-12830) Remove Deprecated constructor in TimeWindowedDeserializer and TimeWindowedSerde
[ https://issues.apache.org/jira/browse/KAFKA-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870929#comment-17870929 ] PoAn Yang commented on KAFKA-12830: --- Hi [~mjsax], I'm interested in this issue. May I take it? Thank you. > Remove Deprecated constructor in TimeWindowedDeserializer and > TimeWindowedSerde > --- > > Key: KAFKA-12830 > URL: https://issues.apache.org/jira/browse/KAFKA-12830 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Josep Prat >Priority: Blocker > Fix For: 4.0.0 > > > The single argument constructor and a factory method of the following classes > were deprecated in version 2.8: > * > org.apache.kafka.streams.kstream.TimeWindowedDeserializer#TimeWindowedDeserializer(org.apache.kafka.common.serialization.Deserializer) > * > org.apache.kafka.streams.kstream.WindowedSerdes.TimeWindowedSerde#TimeWindowedSerde(org.apache.kafka.common.serialization.Serde) > * > org.apache.kafka.streams.kstream.WindowedSerdes#timeWindowedSerdeFrom(java.lang.Class) > > > See KAFKA-10366 & KAFKA-9649 and KIP-659 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (KAFKA-12832) Remove Deprecated methods under RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter
[ https://issues.apache.org/jira/browse/KAFKA-12832 ] PoAn Yang deleted comment on KAFKA-12832: --- was (Author: JIRAUSER300229): Hi, I'm interested in this issue. May I take it and create a PR in 4.0.0 version? Thanks. > Remove Deprecated methods under > RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter > -- > > Key: KAFKA-12832 > URL: https://issues.apache.org/jira/browse/KAFKA-12832 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Josep Prat >Priority: Blocker > Fix For: 4.0.0 > > > The following methods under were deprecated in version 3.0.0 > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setBaseBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundFlushes > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundFlushes > > See KAFKA-8897 and KIP-471 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-12832) Remove Deprecated methods under RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter
[ https://issues.apache.org/jira/browse/KAFKA-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870927#comment-17870927 ] PoAn Yang commented on KAFKA-12832: --- Hi, I'm interested in this issue. May I take it and create a PR in 4.0.0 version? Thanks. > Remove Deprecated methods under > RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter > -- > > Key: KAFKA-12832 > URL: https://issues.apache.org/jira/browse/KAFKA-12832 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Josep Prat >Priority: Blocker > Fix For: 4.0.0 > > > The following methods under were deprecated in version 3.0.0 > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setBaseBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundFlushes > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundFlushes > > See KAFKA-8897 and KIP-471 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17255) Migrate ListOffsetsIntegrationTest to ClusterTestExtensions framework
[ https://issues.apache.org/jira/browse/KAFKA-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870817#comment-17870817 ] PoAn Yang commented on KAFKA-17255: --- Yes, I'm working on it and this may need ClusterTestExtensions to support overriding server properties first. https://issues.apache.org/jira/browse/KAFKA-17259 > Migrate ListOffsetsIntegrationTest to ClusterTestExtensions framework > - > > Key: KAFKA-17255 > URL: https://issues.apache.org/jira/browse/KAFKA-17255 > Project: Kafka > Issue Type: Test >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Minor > > As title, we can migrate ListOffsetsIntegrationTest to Java and use > ClusterTestExtensions framework. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17259) Support to override serverProperties and restart cluster in ClusterTestExtensions
PoAn Yang created KAFKA-17259: - Summary: Support to override serverProperties and restart cluster in ClusterTestExtensions Key: KAFKA-17259 URL: https://issues.apache.org/jira/browse/KAFKA-17259 Project: Kafka Issue Type: Test Reporter: PoAn Yang Assignee: PoAn Yang In some test cases like ListOffsetsIntegrationTest#testListVersion0, it stops brokers with old config, override config, and start brokers again. We should also support this in ClusterTestExtensions framework. -- This message was sent by Atlassian Jira (v8.20.10#820010)
