[jira] [Assigned] (KAFKA-15897) Flaky Test: testWrongIncarnationId() – kafka.server.ControllerRegistrationManagerTest
[
https://issues.apache.org/jira/browse/KAFKA-15897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai reassigned KAFKA-15897:
--
Assignee: Chia-Ping Tsai
> Flaky Test: testWrongIncarnationId() –
> kafka.server.ControllerRegistrationManagerTest
> -
>
> Key: KAFKA-15897
> URL: https://issues.apache.org/jira/browse/KAFKA-15897
> Project: Kafka
> Issue Type: Test
>Reporter: Apoorv Mittal
>Assignee: Chia-Ping Tsai
>Priority: Major
> Labels: flaky-test
>
> Build run:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/21/tests/
>
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <(false,1,0)> but was:
> <(true,0,0)>Stacktraceorg.opentest4j.AssertionFailedError: expected:
> <(false,1,0)> but was: <(true,0,0)>at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>at
> app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
> at
> app//kafka.server.ControllerRegistrationManagerTest.$anonfun$testWrongIncarnationId$3(ControllerRegistrationManagerTest.scala:228)
>at
> app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
> at
> app//kafka.server.ControllerRegistrationManagerTest.testWrongIncarnationId(ControllerRegistrationManagerTest.scala:226)
> at
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)at
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
> at
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> at
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
>at
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
>
[jira] [Commented] (KAFKA-15897) Flaky Test: testWrongIncarnationId() – kafka.server.ControllerRegistrationManagerTest
[
https://issues.apache.org/jira/browse/KAFKA-15897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841597#comment-17841597
]
Chia-Ping Tsai commented on KAFKA-15897:
It seems the root cause is that `context.mockChannelManager.poll()`
([https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/ControllerRegistrationManagerTest.scala#L221)]
is executed after event queue thread adds the `ControllerRegistrationRequest`
([https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ControllerRegistrationManager.scala#L233).]
The request is consumed without no response as we don't set response
([https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/ControllerRegistrationManagerTest.scala#L226)].
Hence, the following test
([https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/ControllerRegistrationManagerTest.scala#L230)]
gets failed as the request is gone.
I feel the first poll
([https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/ControllerRegistrationManagerTest.scala#L221)]
is unnecessary since the check is waiting for event queue thread to send
controller registration request. That does NOT need the poll, and removing the
poll can prevent above race condition.
> Flaky Test: testWrongIncarnationId() –
> kafka.server.ControllerRegistrationManagerTest
> -
>
> Key: KAFKA-15897
> URL: https://issues.apache.org/jira/browse/KAFKA-15897
> Project: Kafka
> Issue Type: Test
>Reporter: Apoorv Mittal
>Assignee: Chia-Ping Tsai
>Priority: Major
> Labels: flaky-test
>
> Build run:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/21/tests/
>
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <(false,1,0)> but was:
> <(true,0,0)>Stacktraceorg.opentest4j.AssertionFailedError: expected:
> <(false,1,0)> but was: <(true,0,0)>at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>at
> app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
> at
> app//kafka.server.ControllerRegistrationManagerTest.$anonfun$testWrongIncarnationId$3(ControllerRegistrationManagerTest.scala:228)
>at
> app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
> at
> app//kafka.server.ControllerRegistrationManagerTest.testWrongIncarnationId(ControllerRegistrationManagerTest.scala:226)
> at
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)at
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
> at
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> at
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>at
>
[jira] [Commented] (KAFKA-16643) Add ModifierOrder checkstyle rule
[ https://issues.apache.org/jira/browse/KAFKA-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842188#comment-17842188 ] Chia-Ping Tsai commented on KAFKA-16643: +10 to this jira! > Add ModifierOrder checkstyle rule > - > > Key: KAFKA-16643 > URL: https://issues.apache.org/jira/browse/KAFKA-16643 > Project: Kafka > Issue Type: Task > Components: build >Reporter: Greg Harris >Priority: Minor > > Checkstyle offers the ModifierOrder rule: > [https://checkstyle.sourceforge.io/checks/modifier/modifierorder.html] that > Kafka violates in a lot of places. We should decide if this is a checkstyle > rule we should be following or not, and potentially enable it moving forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16643) Add ModifierOrder checkstyle rule
[ https://issues.apache.org/jira/browse/KAFKA-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842189#comment-17842189 ] Chia-Ping Tsai commented on KAFKA-16643: I have saw a lot of conflicts caused by the inconsistent import order... > Add ModifierOrder checkstyle rule > - > > Key: KAFKA-16643 > URL: https://issues.apache.org/jira/browse/KAFKA-16643 > Project: Kafka > Issue Type: Task > Components: build >Reporter: Greg Harris >Priority: Minor > > Checkstyle offers the ModifierOrder rule: > [https://checkstyle.sourceforge.io/checks/modifier/modifierorder.html] that > Kafka violates in a lot of places. We should decide if this is a checkstyle > rule we should be following or not, and potentially enable it moving forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16639) AsyncKafkaConsumer#close does not send heartbeat to leave group
[
https://issues.apache.org/jira/browse/KAFKA-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842192#comment-17842192
]
Chia-Ping Tsai commented on KAFKA-16639:
{quote}
I took this because I wasn't sure if you wanted me to look into it. Please go
ahead assign it back. We would love to review it. Thanks. Chia-Ping Tsai
{quote}
Sorry I just make sure there is someone who can have a fix for it, and won't
have duplicate PR. Thanks for your response (and reviews in the future). We
will prepare the patch ASAP.
> AsyncKafkaConsumer#close does not send heartbeat to leave group
> ---
>
> Key: KAFKA-16639
> URL: https://issues.apache.org/jira/browse/KAFKA-16639
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Reporter: Chia-Ping Tsai
>Assignee: Philip Nee
>Priority: Major
> Labels: kip-848-client-support
>
> This bug can be reproduced by immediately closing a consumer which is just
> created.
> The root cause is that we skip the new heartbeat used to leave group when
> there is a in-flight heartbeat request
> ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L212).]
> It seems to me the simple solution is that we create a heartbeat request when
> meeting above situation and then send it by pollOnClose
> ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java#L62).]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15897) Flaky Test: testWrongIncarnationId() – kafka.server.ControllerRegistrationManagerTest
[
https://issues.apache.org/jira/browse/KAFKA-15897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-15897.
Fix Version/s: 3.8.0
Resolution: Fixed
> Flaky Test: testWrongIncarnationId() –
> kafka.server.ControllerRegistrationManagerTest
> -
>
> Key: KAFKA-15897
> URL: https://issues.apache.org/jira/browse/KAFKA-15897
> Project: Kafka
> Issue Type: Test
>Reporter: Apoorv Mittal
>Assignee: Chia-Ping Tsai
>Priority: Major
> Labels: flaky-test
> Fix For: 3.8.0
>
>
> Build run:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/21/tests/
>
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <(false,1,0)> but was:
> <(true,0,0)>Stacktraceorg.opentest4j.AssertionFailedError: expected:
> <(false,1,0)> but was: <(true,0,0)>at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>at
> app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
> at
> app//kafka.server.ControllerRegistrationManagerTest.$anonfun$testWrongIncarnationId$3(ControllerRegistrationManagerTest.scala:228)
>at
> app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
> at
> app//kafka.server.ControllerRegistrationManagerTest.testWrongIncarnationId(ControllerRegistrationManagerTest.scala:226)
> at
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)at
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
> at
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> at
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
>at
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
>
[jira] [Commented] (KAFKA-16643) Add ModifierOrder checkstyle rule
[ https://issues.apache.org/jira/browse/KAFKA-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842195#comment-17842195 ] Chia-Ping Tsai commented on KAFKA-16643: there is a jira for it https://issues.apache.org/jira/browse/KAFKA-10787 maybe we can discuss that in the jira > Add ModifierOrder checkstyle rule > - > > Key: KAFKA-16643 > URL: https://issues.apache.org/jira/browse/KAFKA-16643 > Project: Kafka > Issue Type: Task > Components: build >Reporter: Greg Harris >Priority: Minor > > Checkstyle offers the ModifierOrder rule: > [https://checkstyle.sourceforge.io/checks/modifier/modifierorder.html] that > Kafka violates in a lot of places. We should decide if this is a checkstyle > rule we should be following or not, and potentially enable it moving forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16639) AsyncKafkaConsumer#close does not send heartbeat to leave group
[ https://issues.apache.org/jira/browse/KAFKA-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842186#comment-17842186 ] Chia-Ping Tsai commented on KAFKA-16639: [~pnee] I just notice that you have took over this jira. Do you plan to offer a patch for it? If you have no free cycles, we can file a PR for it and then be waiting for your reviews :) > AsyncKafkaConsumer#close does not send heartbeat to leave group > --- > > Key: KAFKA-16639 > URL: https://issues.apache.org/jira/browse/KAFKA-16639 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Reporter: Chia-Ping Tsai >Assignee: Philip Nee >Priority: Major > Labels: kip-848-client-support > > This bug can be reproduced by immediately closing a consumer which is just > created. > The root cause is that we skip the new heartbeat used to leave group when > there is a in-flight heartbeat request > ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L212).] > It seems to me the simple solution is that we create a heartbeat request when > meeting above situation and then send it by pollOnClose > ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java#L62).] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16627) Remove ClusterConfig parameter in BeforeEach and AfterEach
[ https://issues.apache.org/jira/browse/KAFKA-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16627. Fix Version/s: 3.8.0 Resolution: Fixed > Remove ClusterConfig parameter in BeforeEach and AfterEach > -- > > Key: KAFKA-16627 > URL: https://issues.apache.org/jira/browse/KAFKA-16627 > Project: Kafka > Issue Type: Improvement >Reporter: Kuan Po Tseng >Assignee: Kuan Po Tseng >Priority: Minor > Fix For: 3.8.0 > > > In the past we modify configs like server broker properties by modifying the > ClusterConfig reference passed to BeforeEach and AfterEach based on the > requirements of the tests. > While after KAFKA-16560, the ClusterConfig become immutable, modify the > ClusterConfig reference no longer reflects any changes to the test cluster. > Then pass ClusterConfig to BeforeEach and AfterEach become redundant. We > should remove this behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16647) Remove setMetadataDirectory from BrokerNode/ControllerNode
[ https://issues.apache.org/jira/browse/KAFKA-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842298#comment-17842298 ] Chia-Ping Tsai commented on KAFKA-16647: btw, this jira can remove the duplicate assign about metadata folder (https://github.com/apache/kafka/blob/7c0a302c4da9d53a8fddc504a9fac8d8afecbec8/core/src/test/java/kafka/testkit/BrokerNode.java#L129) > Remove setMetadataDirectory from BrokerNode/ControllerNode > -- > > Key: KAFKA-16647 > URL: https://issues.apache.org/jira/browse/KAFKA-16647 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > `TestKitNodes` does not enable callers to define the location of "base > folder". That makes sense to me since callers should not care for it. That > means the location of metadata folder shoud be transparent to callers. Hence, > the setter of metadata folder is useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842328#comment-17842328 ] Chia-Ping Tsai commented on KAFKA-16223: It seems `KafkaConfigBackingStoreMockitoTest` offers many helpful captures to simplify the migration. For example, `testConsumerPropertiesNotInsertedByDefaultWithoutExactlyOnceSourceEnabled`, `testConsumerPropertiesOverrideUserSuppliedValuesWithExactlyOnceSourceEnabled`, and `testConsumerPropertiesInsertedByDefaultWithExactlyOnceSourceEnabled` can be migrated to `KafkaConfigBackingStoreMockitoTest` by using `capturedConsumerProps`. > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842328#comment-17842328 ] Chia-Ping Tsai edited comment on KAFKA-16223 at 4/30/24 10:29 AM: -- It seems `KafkaConfigBackingStoreMockitoTest` offers many helpful captures to simplify the migration. For example, `testConsumerPropertiesNotInsertedByDefaultWithoutExactlyOnceSourceEnabled`, `testConsumerPropertiesOverrideUserSuppliedValuesWithExactlyOnceSourceEnabled`, `testConsumerPropertiesInsertedByDefaultWithExactlyOnceSourceEnabled`, `testFencableProducerPropertiesInsertedByDefault, and `testExceptionOnStartWhenConfigTopicHasMultiplePartitions` can be migrated to `KafkaConfigBackingStoreMockitoTest` by using `capturedConsumerProps`. was (Author: chia7712): It seems `KafkaConfigBackingStoreMockitoTest` offers many helpful captures to simplify the migration. For example, `testConsumerPropertiesNotInsertedByDefaultWithoutExactlyOnceSourceEnabled`, `testConsumerPropertiesOverrideUserSuppliedValuesWithExactlyOnceSourceEnabled`, and `testConsumerPropertiesInsertedByDefaultWithExactlyOnceSourceEnabled` can be migrated to `KafkaConfigBackingStoreMockitoTest` by using `capturedConsumerProps`. > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16647) Remove setMetadataDirectory from BrokerNode/ControllerNode
Chia-Ping Tsai created KAFKA-16647: -- Summary: Remove setMetadataDirectory from BrokerNode/ControllerNode Key: KAFKA-16647 URL: https://issues.apache.org/jira/browse/KAFKA-16647 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Kuan Po Tseng `TestKitNodes` does not enable callers to define the location of "base folder". That makes sense to me since callers should not care for it. That means the location of metadata folder shoud be transparent to callers. Hence, the setter of metadata folder is useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10859) add @Test annotation to FileStreamSourceTaskTest.testInvalidFile and reduce the loop count to speedup the test
[ https://issues.apache.org/jira/browse/KAFKA-10859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10859. Resolution: Duplicate fixed by https://github.com/apache/kafka/commit/3dacdc5694da5db283524889d2270695defebbaa > add @Test annotation to FileStreamSourceTaskTest.testInvalidFile and reduce > the loop count to speedup the test > -- > > Key: KAFKA-10859 > URL: https://issues.apache.org/jira/browse/KAFKA-10859 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Tom Bentley >Priority: Major > Labels: newbie > > FileStreamSourceTaskTest.testInvalidFile miss a `@Test` annotation. Also, it > loops 100 times which spend about 2m to complete a unit test. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16506) add the scala version of tool-related classes back to core module to follow KIP-906
[ https://issues.apache.org/jira/browse/KAFKA-16506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842330#comment-17842330 ] Chia-Ping Tsai commented on KAFKA-16506: [~yangpoan] thanks for the check. please feel free to close this jira > add the scala version of tool-related classes back to core module to follow > KIP-906 > --- > > Key: KAFKA-16506 > URL: https://issues.apache.org/jira/browse/KAFKA-16506 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Major > > According to > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines > , we have to deprecate the scala version of tool-related classes instead of > deleting them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16639) AsyncKafkaConsumer#close does not send heartbeat to leave group
[ https://issues.apache.org/jira/browse/KAFKA-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841675#comment-17841675 ] Chia-Ping Tsai commented on KAFKA-16639: [~pnee] Could you please take a look if you have free cycle? thanks! > AsyncKafkaConsumer#close does not send heartbeat to leave group > --- > > Key: KAFKA-16639 > URL: https://issues.apache.org/jira/browse/KAFKA-16639 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Labels: kip-848-client-support > > This bug can be reproduced by immediately closing a consumer which is just > created. > The root cause is that we skip the new heartbeat used to leave group when > there is a in-flight heartbeat request > ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L212).] > It seems to me the simple solution is that we create a heartbeat request when > meeting above situation and then send it by pollOnClose > ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java#L62).] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16639) AsyncKafkaConsumer#close does not send heartbeat to leave group
Chia-Ping Tsai created KAFKA-16639: -- Summary: AsyncKafkaConsumer#close does not send heartbeat to leave group Key: KAFKA-16639 URL: https://issues.apache.org/jira/browse/KAFKA-16639 Project: Kafka Issue Type: Bug Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai This bug can be reproduced by immediately closing a consumer which is just created. The root cause is that we skip the new heartbeat used to leave group when there is a in-flight heartbeat request ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L212).] It seems to me the simple solution is that we create a heartbeat request when meeting above situation and then send it by pollOnClose ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java#L62).] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16640) Replace TestUtils#resource by scala.util.Using
Chia-Ping Tsai created KAFKA-16640: -- Summary: Replace TestUtils#resource by scala.util.Using Key: KAFKA-16640 URL: https://issues.apache.org/jira/browse/KAFKA-16640 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai `scala.util.Using` is in both scala 2.13 and scala-collection-compat so we don't need to have custom try-resource function. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16640) Replace TestUtils#resource by scala.util.Using
[ https://issues.apache.org/jira/browse/KAFKA-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16640: -- Assignee: TengYao Chi (was: Chia-Ping Tsai) > Replace TestUtils#resource by scala.util.Using > -- > > Key: KAFKA-16640 > URL: https://issues.apache.org/jira/browse/KAFKA-16640 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: TengYao Chi >Priority: Minor > > `scala.util.Using` is in both scala 2.13 and scala-collection-compat so we > don't need to have custom try-resource function. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16650) add integration test for Admin#abortTransaction
Chia-Ping Tsai created KAFKA-16650: -- Summary: add integration test for Admin#abortTransaction Key: KAFKA-16650 URL: https://issues.apache.org/jira/browse/KAFKA-16650 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai It seems there are only few unit tests. We should add IT includeing zk, kraft, and new group coordinator for it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16604) Deprecate ConfigDef.ConfigKey constructor from public APIs
[
https://issues.apache.org/jira/browse/KAFKA-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842031#comment-17842031
]
Chia-Ping Tsai commented on KAFKA-16604:
{quote}
If we want to prevent bugs like we saw in KAFKA-16592, one alternative is to
add a builder class for the ConfigKey class. This would allow us to add new
fields to the ConfigKey class without having to add new (public) constructors,
or alter existing ones.
{quote}
I love build pattern, but there are already a lot of helper methods - "define"
- which allows user to create ConfigKey object with fewer arguments. We have to
deprecate all "define" functions to encourage to use new builder, and that
means more breaking changes.
> Deprecate ConfigDef.ConfigKey constructor from public APIs
> --
>
> Key: KAFKA-16604
> URL: https://issues.apache.org/jira/browse/KAFKA-16604
> Project: Kafka
> Issue Type: Improvement
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>
> Currently, one can create ConfigKey by either invoking the public constructor
> directly and passing it to a ConfigDef object or by invoking the a bunch of
> define methods. The 2 ways can get confusing at times. Moreover, it could
> lead to errors as was noticed in KAFKA-16592
> We should ideally have only 1 way exposed to the users which IMO should be to
> create the objects only through the exposed define methods. This ticket is
> about marking the public constructor of ConfigKey as Deprecated first and
> then making it private eventually.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16553) log controller configs when startup
[
https://issues.apache.org/jira/browse/KAFKA-16553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842032#comment-17842032
]
Chia-Ping Tsai commented on KAFKA-16553:
{quote}
so that we can get this controller log first. wdyt?
{quote}
agree!
> log controller configs when startup
> ---
>
> Key: KAFKA-16553
> URL: https://issues.apache.org/jira/browse/KAFKA-16553
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
>
> We can't observe the controller configs from the log file. We can copy the
> solution used by broker
> (https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L492).
> Or this issue should be blocked by
> https://issues.apache.org/jira/browse/KAFKA-13105 to wait for more graceful
> solution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16553) log controller configs when startup
[ https://issues.apache.org/jira/browse/KAFKA-16553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16553: -- Assignee: Johnny Hsu (was: Chia-Ping Tsai) > log controller configs when startup > --- > > Key: KAFKA-16553 > URL: https://issues.apache.org/jira/browse/KAFKA-16553 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Johnny Hsu >Priority: Minor > > We can't observe the controller configs from the log file. We can copy the > solution used by broker > (https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L492). > Or this issue should be blocked by > https://issues.apache.org/jira/browse/KAFKA-13105 to wait for more graceful > solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16672) Fix flaky DedicatedMirrorIntegrationTest.testMultiNodeCluster
[ https://issues.apache.org/jira/browse/KAFKA-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-16672: --- Description: It is flaky on my jenkins, and sometimes it fails in Kafka CI[0] The error happens in virtue of race condition. `KafkaBasedLog` loads records from topic via thread, so `RebalanceNeededException` will be thrown if we check the task configs too soon. It seems to me `RebalanceNeededException` is a temporary exception so we should treat it as a retryable exception in waiting. In short, we should catch `RebalanceNeededException` in `awaitTaskConfigurations` [1] [0] https://ge.apache.org/scans/tests?search.buildOutcome=failure=gradle=P28D=kafka=Asia%2FTaipei=org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest=testMultiNodeCluster() [1] https://github.com/apache/kafka/blob/55a00be4e973f3f4c8869b6f70de1e285719e890/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/DedicatedMirrorIntegrationTest.java#L355 was: It is flaky on my jenkins, and sometimes it fails in Kafka [CI|https://ge.apache.org/scans/tests?search.buildOutcome=failure=gradle=P28D=kafka=Asia%2FTaipei=org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest=testMultiNodeCluster()] The error happens in virtue of race condition. `KafkaBasedLog` loads records from topic via thread, so `RebalanceNeededException` will be thrown if we check the task configs too soon. It seems to me `RebalanceNeededException` is a temporary exception so we should treat it as a retryable exception in waiting. > Fix flaky DedicatedMirrorIntegrationTest.testMultiNodeCluster > - > > Key: KAFKA-16672 > URL: https://issues.apache.org/jira/browse/KAFKA-16672 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > It is flaky on my jenkins, and sometimes it fails in Kafka CI[0] > The error happens in virtue of race condition. `KafkaBasedLog` loads records > from topic via thread, so `RebalanceNeededException` will be thrown if we > check the task configs too soon. It seems to me `RebalanceNeededException` is > a temporary exception so we should treat it as a retryable exception in > waiting. > In short, we should catch `RebalanceNeededException` in > `awaitTaskConfigurations` [1] > [0] > https://ge.apache.org/scans/tests?search.buildOutcome=failure=gradle=P28D=kafka=Asia%2FTaipei=org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest=testMultiNodeCluster() > [1] > https://github.com/apache/kafka/blob/55a00be4e973f3f4c8869b6f70de1e285719e890/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/DedicatedMirrorIntegrationTest.java#L355 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16672) Fix flaky DedicatedMirrorIntegrationTest.testMultiNodeCluster
Chia-Ping Tsai created KAFKA-16672: -- Summary: Fix flaky DedicatedMirrorIntegrationTest.testMultiNodeCluster Key: KAFKA-16672 URL: https://issues.apache.org/jira/browse/KAFKA-16672 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai It is flaky on my jenkins, and sometimes it fails in Kafka [CI|https://ge.apache.org/scans/tests?search.buildOutcome=failure=gradle=P28D=kafka=Asia%2FTaipei=org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest=testMultiNodeCluster()] The error happens in virtue of race condition. `KafkaBasedLog` loads records from topic via thread, so `RebalanceNeededException` will be thrown if we check the task configs too soon. It seems to me `RebalanceNeededException` is a temporary exception so we should treat it as a retryable exception in waiting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16671) Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured
Chia-Ping Tsai created KAFKA-16671: -- Summary: Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured Key: KAFKA-16671 URL: https://issues.apache.org/jira/browse/KAFKA-16671 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai loop 1000times on my local, and all pass. Let's enable the test to see what happens in our CI -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16393) SslTransportLayer doesn't implement write(ByteBuffer[], int, int) correctly
[
https://issues.apache.org/jira/browse/KAFKA-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-16393.
Fix Version/s: 3.8.0
Resolution: Fixed
> SslTransportLayer doesn't implement write(ByteBuffer[], int, int) correctly
> ---
>
> Key: KAFKA-16393
> URL: https://issues.apache.org/jira/browse/KAFKA-16393
> Project: Kafka
> Issue Type: Improvement
>Reporter: Haruki Okada
>Assignee: Haruki Okada
>Priority: Minor
> Fix For: 3.8.0
>
>
> As of Kafka 3.7.0, SslTransportLayer.write(ByteBuffer[], int, int) is
> implemented like below:
> {code:java}
> public long write(ByteBuffer[] srcs, int offset, int length) throws
> IOException {
> ...
> int i = offset;
> while (i < length) {
> if (srcs[i].hasRemaining() || hasPendingWrites()) {
>
> {code}
> The loop index starts at `offset` and ends with `length`.
> However this isn't correct because end-index should be `offset + length`.
> Let's say we have the array of ByteBuffer with length = 5 and try calling
> this method with offset = 3, length = 1.
> In current code, `write(srcs, 3, 1)` doesn't attempt any write because the
> loop condition is immediately false.
> For now, seems this method is only called with args offset = 0, length =
> srcs.length in Kafka code base so not causing any problem though, we should
> fix this because this could introduce subtle bug if use this method with
> different args in the future.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16672) Fix flaky DedicatedMirrorIntegrationTest.testMultiNodeCluster
[ https://issues.apache.org/jira/browse/KAFKA-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16672: -- Assignee: Johnny Hsu (was: Chia-Ping Tsai) > Fix flaky DedicatedMirrorIntegrationTest.testMultiNodeCluster > - > > Key: KAFKA-16672 > URL: https://issues.apache.org/jira/browse/KAFKA-16672 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Johnny Hsu >Priority: Minor > > It is flaky on my jenkins, and sometimes it fails in Kafka CI[0] > The error happens in virtue of race condition. `KafkaBasedLog` loads records > from topic via thread, so `RebalanceNeededException` will be thrown if we > check the task configs too soon. It seems to me `RebalanceNeededException` is > a temporary exception so we should treat it as a retryable exception in > waiting. > In short, we should catch `RebalanceNeededException` in > `awaitTaskConfigurations` [1] > [0] > https://ge.apache.org/scans/tests?search.buildOutcome=failure=gradle=P28D=kafka=Asia%2FTaipei=org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest=testMultiNodeCluster() > [1] > https://github.com/apache/kafka/blob/55a00be4e973f3f4c8869b6f70de1e285719e890/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/integration/DedicatedMirrorIntegrationTest.java#L355 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16660) reduce the check interval to speedup DelegationTokenRequestsTest
[ https://issues.apache.org/jira/browse/KAFKA-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16660. Fix Version/s: 3.8.0 Resolution: Fixed > reduce the check interval to speedup DelegationTokenRequestsTest > > > Key: KAFKA-16660 > URL: https://issues.apache.org/jira/browse/KAFKA-16660 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > Fix For: 3.8.0 > > > the check interval is 1 minute > (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/DelegationTokenRequestsTest.scala#L49), > and `DelegationTokenRequestsTest` waits 2 minutes before running the check > (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/DelegationTokenRequestsTest.scala#L159) > ... > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16484) Support to define per broker/controller property by ClusterConfigProperty
[ https://issues.apache.org/jira/browse/KAFKA-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16484. Fix Version/s: 3.8.0 Resolution: Fixed > Support to define per broker/controller property by ClusterConfigProperty > - > > Key: KAFKA-16484 > URL: https://issues.apache.org/jira/browse/KAFKA-16484 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Major > Fix For: 3.8.0 > > > the property set to `ClusterConfigProperty` gets applied to all brokers, and > hence we can't have individual props for each broker to test racks. > > It seems to me we can add new field "id" to `ClusterConfigProperty` to > declare the property should be applied to specific broker (or controller). > the default value is -1 and it should be applied to all nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12947) Replace EasyMock and PowerMock with Mockito for StreamsMetricsImplTest ...
[ https://issues.apache.org/jira/browse/KAFKA-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-12947. Resolution: Duplicate this is fixed by https://github.com/apache/kafka/pull/14623 > Replace EasyMock and PowerMock with Mockito for StreamsMetricsImplTest ... > -- > > Key: KAFKA-12947 > URL: https://issues.apache.org/jira/browse/KAFKA-12947 > Project: Kafka > Issue Type: Sub-task >Reporter: YI-CHEN WANG >Assignee: Dalibor Plavcic >Priority: Major > > For Kafka-7438 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14579) Move DumpLogSegments to tools
[ https://issues.apache.org/jira/browse/KAFKA-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843219#comment-17843219 ] Chia-Ping Tsai commented on KAFKA-14579: After https://github.com/apache/kafka/pull/15652, `DumpLogSegments` only depends on `TransactionState`, `TransactionLogKey`, `TransactionLogValue`, and `Decoder`. They can be moved to "transaction-coordinator" module. `TransactionState can be rewrite by java enum. Both `TransactionLogKey` and `TransactionLogValue` are generated code, so we need to enable "transaction-coordinator" to run generation. `Decoder` needs to be deprecated, so maybe we should file KIP for it first. > Move DumpLogSegments to tools > - > > Key: KAFKA-14579 > URL: https://issues.apache.org/jira/browse/KAFKA-14579 > Project: Kafka > Issue Type: Sub-task >Reporter: Mickael Maison >Assignee: Alexandre Dupriez >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[
https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842724#comment-17842724
]
Chia-Ping Tsai commented on KAFKA-16223:
{quote}
I have another set of tests already migrated and plan to open the second PR
soon.
{quote}
this is great! Let's review/merge your PR first. And we will take over
remaining if you have no enough time.
> Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
> ---
>
> Key: KAFKA-16223
> URL: https://issues.apache.org/jira/browse/KAFKA-16223
> Project: Kafka
> Issue Type: Sub-task
> Components: connect
>Reporter: Hector Geraldino
>Assignee: Hector Geraldino
>Priority: Minor
> Fix For: 3.8.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16654) Refactor kafka.test.annotation.Type and ClusterTestExtensions
[ https://issues.apache.org/jira/browse/KAFKA-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16654: -- Assignee: TaiJuWu (was: Chia-Ping Tsai) > Refactor kafka.test.annotation.Type and ClusterTestExtensions > - > > Key: KAFKA-16654 > URL: https://issues.apache.org/jira/browse/KAFKA-16654 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: TaiJuWu >Priority: Minor > > It seems to me the refactor could include following tasks. > 1. change `invocationContexts`, method invoked by `ClusterTemplate`, and > generate-related methods in `ClusterTestExtensions` to return a > java.util.Collection instead of accepting a `java.util.function.Consumer`. > That can brings two benefit. 1) more simple in production: we don't need to > create a List and then pass it to be a function to collect stuff. 2) more > easy to write unit test. > 2. separate `provideTestTemplateInvocationContexts` to multi methods to > handle each annotation. That can help us to write tests, and make core more > readable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16654) Refactor kafka.test.annotation.Type and ClusterTestExtensions
Chia-Ping Tsai created KAFKA-16654: -- Summary: Refactor kafka.test.annotation.Type and ClusterTestExtensions Key: KAFKA-16654 URL: https://issues.apache.org/jira/browse/KAFKA-16654 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai It seems to me the refactor could include following tasks. 1. change `invocationContexts`, method invoked by `ClusterTemplate`, and generate-related methods in `ClusterTestExtensions` to return a java.util.Collection instead of accepting a `java.util.function.Consumer`. That can brings two benefit. 1) more simple in production: we don't need to create a List and then pass it to be a function to collect stuff. 2) more easy to write unit test. 2. separate `provideTestTemplateInvocationContexts` to multi methods to handle each annotation. That can help us to write tests, and make core more readable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16614) Disallow `@ClusterTemplate("")`
[
https://issues.apache.org/jira/browse/KAFKA-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-16614.
Fix Version/s: 3.8.0
Resolution: Fixed
> Disallow `@ClusterTemplate("")`
> ---
>
> Key: KAFKA-16614
> URL: https://issues.apache.org/jira/browse/KAFKA-16614
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: TaiJuWu
>Priority: Minor
> Fix For: 3.8.0
>
>
> `@ClusterTemplate` enable us to create dynamic configs, and it expect to
> accept a method name which can create server configs at runtime. It throws
> error when we pass a nonexistent method name, but it works if we pass an
> empty name
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16647) Remove setMetadataDirectory from BrokerNode/ControllerNode
[ https://issues.apache.org/jira/browse/KAFKA-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16647. Fix Version/s: 3.8.0 Resolution: Fixed > Remove setMetadataDirectory from BrokerNode/ControllerNode > -- > > Key: KAFKA-16647 > URL: https://issues.apache.org/jira/browse/KAFKA-16647 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > Fix For: 3.8.0 > > > `TestKitNodes` does not enable callers to define the location of "base > folder". That makes sense to me since callers should not care for it. That > means the location of metadata folder shoud be transparent to callers. Hence, > the setter of metadata folder is useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10696) Replace ProduceResponse.PartitionResponse by auto-generated PartitionProduceResponse
[ https://issues.apache.org/jira/browse/KAFKA-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10696. Resolution: Won't Fix > Replace ProduceResponse.PartitionResponse by auto-generated > PartitionProduceResponse > > > Key: KAFKA-10696 > URL: https://issues.apache.org/jira/browse/KAFKA-10696 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > This is a follow-up of KAFKA-9628. > related discussion: > https://github.com/apache/kafka/pull/9401#discussion_r518976605 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-7014) Guarantee the byte buffer returned by Serializer is reusable
[ https://issues.apache.org/jira/browse/KAFKA-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-7014. --- Resolution: Won't Fix > Guarantee the byte buffer returned by Serializer is reusable > > > Key: KAFKA-7014 > URL: https://issues.apache.org/jira/browse/KAFKA-7014 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Priority: Minor > > The byte array returned by Serializer is copied to another ByteBuffer by > KafkaProducer so the byte array is reusable actually. If kafka can guarantee > this behavior (perhaps doc it on the Serializer), user can design the > size-fixed message and then reuse the byte array. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10290) fix flaky core/compatibility_test_new_broker_test.py
[
https://issues.apache.org/jira/browse/KAFKA-10290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-10290.
Resolution: Won't Fix
> fix flaky core/compatibility_test_new_broker_test.py
>
>
> Key: KAFKA-10290
> URL: https://issues.apache.org/jira/browse/KAFKA-10290
> Project: Kafka
> Issue Type: Sub-task
> Components: core, system tests
>Reporter: Chia-Ping Tsai
>Priority: Major
>
> {quote}
> Module: kafkatest.tests.core.compatibility_test_new_broker_test
> Class: ClientCompatibilityTestNewBroker
> Method: test_compatibility
> Arguments:
> {
> "compression_types": [
> "none"
> ],
> "consumer_version": "1.0.2",
> "producer_version": "1.0.2",
> "timestamp_type": "CreateTime"
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14874) Unable to create > 5000 topics for once when using Kraft
[ https://issues.apache.org/jira/browse/KAFKA-14874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-14874. Resolution: Won't Fix > Unable to create > 5000 topics for once when using Kraft > > > Key: KAFKA-14874 > URL: https://issues.apache.org/jira/browse/KAFKA-14874 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > the error happens due to > [https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L779] > I encountered this error when creating >5000 topics for mirroring the cluster > from zk to Kraft. The operation of creating a bunch of topics is allowed by > zk-based kafka. > It seems to me there are two improvements for this issue. > 1) add more precise error message for such case. > 2) make `maxRecordsPerBatch` configurable (there is already a setter > [https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L272]) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10809) Make all system tests stably run on either Travis CI or local
[ https://issues.apache.org/jira/browse/KAFKA-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10809. Resolution: Fixed > Make all system tests stably run on either Travis CI or local > - > > Key: KAFKA-10809 > URL: https://issues.apache.org/jira/browse/KAFKA-10809 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > It can encourage us to write more system tests if Travis CI can run a subset > of system tests automatically. > This initial work is https://github.com/apache/kafka/pull/9652 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10804) Tune travis system tests to avoid timeouts
[ https://issues.apache.org/jira/browse/KAFKA-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10804. Resolution: Won't Fix > Tune travis system tests to avoid timeouts > -- > > Key: KAFKA-10804 > URL: https://issues.apache.org/jira/browse/KAFKA-10804 > Project: Kafka > Issue Type: Sub-task >Reporter: Jason Gustafson >Assignee: Chia-Ping Tsai >Priority: Major > > Thanks to https://github.com/apache/kafka/pull/9652, we are now running > system tests for PRs. However, it looks like we need some tuning because many > of the subsets are timing out. For example: > https://travis-ci.com/github/apache/kafka/jobs/453241933. This might just be > a matter of adding more subsets or changing the timeout, but we should > probably also consider whether we want to run all system tests or if there is > a more useful subset of them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-6992) Configuring the batch.size to zero won't disable the batching entirely
[
https://issues.apache.org/jira/browse/KAFKA-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-6992.
---
Resolution: Won't Fix
> Configuring the batch.size to zero won't disable the batching entirely
> --
>
> Key: KAFKA-6992
> URL: https://issues.apache.org/jira/browse/KAFKA-6992
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Chia-Ping Tsai
>Priority: Minor
>
> In MAGIC_VALUE_V2, the varint is used and the estimated size won't be equal
> with the actual size. So it has chance to accept more records in a single
> produce batch.
> Perhaps we should revise the docs of batch.size.
> {code:java}
> public static final String BATCH_SIZE_CONFIG = "batch.size";
> private static final String BATCH_SIZE_DOC = "The producer will attempt to
> batch records together into fewer requests whenever multiple records are
> being sent"
> + " to the same partition. This
> helps performance on both the client and the server. This configuration
> controls the "
> + "default batch size in bytes. "
> + ""
> + "No attempt will be made to
> batch records larger than this size. "
> + ""
> + "Requests sent to brokers will
> contain multiple batches, one for each partition with data available to be
> sent. "
> + ""
> + "A small batch size will make
> batching less common and may reduce throughput (a batch size of zero will
> disable "
> + "batching entirely). A very
> large batch size may use memory a bit more wastefully as we will always
> allocate a "
> + "buffer of the specified batch
> size in anticipation of additional records.";{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-10822) Force some stdout from system tests for Travis
[ https://issues.apache.org/jira/browse/KAFKA-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10822. Resolution: Won't Fix > Force some stdout from system tests for Travis > -- > > Key: KAFKA-10822 > URL: https://issues.apache.org/jira/browse/KAFKA-10822 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > downgrade_test.py/upgrade_test.py does upgrade/downgrade for each tests. the > upgrade/downgrade tasks take 10+ mins in Travis env so we ought to print > something in order to avoid timeout caused by Travis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10805) More useful reporting from travis system tests
[ https://issues.apache.org/jira/browse/KAFKA-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10805. Resolution: Won't Fix > More useful reporting from travis system tests > -- > > Key: KAFKA-10805 > URL: https://issues.apache.org/jira/browse/KAFKA-10805 > Project: Kafka > Issue Type: Sub-task >Reporter: Jason Gustafson >Priority: Major > > Inspecting system test output from travis is a bit painful at the moment > because you have to check the build logs to find the tests that failed. > Additionally, there is no logging available from the workers which is often > essential to debug a failure. We should look into how we can improve the > build so that the output is more convenient and useful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-10288) fix flaky client/client_compatibility_features_test.py
[
https://issues.apache.org/jira/browse/KAFKA-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-10288.
Resolution: Won't Fix
> fix flaky client/client_compatibility_features_test.py
> --
>
> Key: KAFKA-10288
> URL: https://issues.apache.org/jira/browse/KAFKA-10288
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, system tests
>Reporter: Chia-Ping Tsai
>Priority: Major
>
> {quote}
> Module: kafkatest.tests.client.client_compatibility_features_test
> Class: ClientCompatibilityFeaturesTest
> Method: run_compatibility_test
> Arguments:
> {
> "broker_version": "0.10.0.1"
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-10291) fix flaky tools/log4j_appender_test.py
[
https://issues.apache.org/jira/browse/KAFKA-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-10291.
Resolution: Won't Fix
> fix flaky tools/log4j_appender_test.py
> --
>
> Key: KAFKA-10291
> URL: https://issues.apache.org/jira/browse/KAFKA-10291
> Project: Kafka
> Issue Type: Sub-task
> Components: system tests, tools
>Reporter: Chia-Ping Tsai
>Priority: Major
>
> {quote}
> Module: kafkatest.tests.tools.log4j_appender_test
> Class: Log4jAppenderTest
> Method: test_log4j_appender
> Arguments:
> {
> "security_protocol": "SSL"
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16668) Enable to set tags by `ClusterTest`
Chia-Ping Tsai created KAFKA-16668: -- Summary: Enable to set tags by `ClusterTest` Key: KAFKA-16668 URL: https://issues.apache.org/jira/browse/KAFKA-16668 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai Currently, the display name can be customized by only `name` (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L42). However, the "key" is hard-code to "name=xxx". Also, it is impossible to set more "tags" for display name. https://github.com/apache/kafka/pull/15766 is a example that we want to add "xxx=bbb" to display name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16668) Enable to set tags by `ClusterTest`
[ https://issues.apache.org/jira/browse/KAFKA-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843481#comment-17843481 ] Chia-Ping Tsai commented on KAFKA-16668: Personally, introducing a new annotation "Tag" is a solution. "Tag[] tags()" > Enable to set tags by `ClusterTest` > > > Key: KAFKA-16668 > URL: https://issues.apache.org/jira/browse/KAFKA-16668 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > Currently, the display name can be customized by only `name` > (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L42). > However, the "key" is hard-code to "name=xxx". Also, it is impossible to set > more "tags" for display name. > https://github.com/apache/kafka/pull/15766 is a example that we want to add > "xxx=bbb" to display name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16666) Remove unused class `GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`
Chia-Ping Tsai created KAFKA-1: -- Summary: Remove unused class `GroupMetadataMessageFormatter` and `OffsetsMessageFormatter` Key: KAFKA-1 URL: https://issues.apache.org/jira/browse/KAFKA-1 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai After https://github.com/apache/kafka/pull/15652, both `GroupMetadataMessageFormatter`[0] and `OffsetsMessageFormatter`[1] get unused. We should remove them. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16666) Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`to tools module
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-1: --- Summary: Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`to tools module (was: Remove unused class `GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`) > Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and > `OffsetsMessageFormatter`to tools module > - > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > After https://github.com/apache/kafka/pull/15652, both > `GroupMetadataMessageFormatter`[0] and `OffsetsMessageFormatter`[1] get > unused. We should remove them. > [0] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 > [1] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16666) Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`to tools module
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-1: --- Description: After https://github.com/apache/kafka/pull/15652, both `GroupMetadataMessageFormatter`[0] and `OffsetsMessageFormatter`[1] get unused. We should remove them. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 [2] https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 was: After https://github.com/apache/kafka/pull/15652, both `GroupMetadataMessageFormatter`[0] and `OffsetsMessageFormatter`[1] get unused. We should remove them. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 > Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and > `OffsetsMessageFormatter`to tools module > - > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > After https://github.com/apache/kafka/pull/15652, both > `GroupMetadataMessageFormatter`[0] and `OffsetsMessageFormatter`[1] get > unused. We should remove them. > [0] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 > [1] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 > [2] > https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16666) Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`to tools module
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-1: --- Description: `GroupMetadataMessageFormatter`[0], `OffsetsMessageFormatter`[1], and `TransactionLogMessageFormatter`[2] are used by ConsoleConsumer to parse data of internal topics. Following the migration plan, we should move them to tools module. Also, we need to keep the compatibility of command line. That is to say, `ConsoleConsumer` can accept the previous package name and then use the (java) implementation to parse and make same output. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 [2] https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 was: After https://github.com/apache/kafka/pull/15652, both `GroupMetadataMessageFormatter`[0] and `OffsetsMessageFormatter`[1] get unused. We should remove them. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 [2] https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 > Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and > `OffsetsMessageFormatter`to tools module > - > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > `GroupMetadataMessageFormatter`[0], `OffsetsMessageFormatter`[1], and > `TransactionLogMessageFormatter`[2] are used by ConsoleConsumer to parse data > of internal topics. Following the migration plan, we should move them to > tools module. Also, we need to keep the compatibility of command line. That > is to say, `ConsoleConsumer` can accept the previous package name and then > use the (java) implementation to parse and make same output. > [0] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 > [1] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 > [2] > https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14579) Move DumpLogSegments to tools
[
https://issues.apache.org/jira/browse/KAFKA-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843422#comment-17843422
]
Chia-Ping Tsai commented on KAFKA-14579:
{quote}
currently only DumpLogSegments is using Decoder, if it's removed then Decoder
should be safe to be deprecated since no one will be using that anymore
{quote}
According to Kafka compatibility rule, command line tools belong to public
interfaces. Hence, we need a replacement for `kafka.serializer.Decoder` and
then deprecate `kafka.serializer.Decoder`. Also, the replacement should be
written by Java and put into tools-api module.
> Move DumpLogSegments to tools
> -
>
> Key: KAFKA-14579
> URL: https://issues.apache.org/jira/browse/KAFKA-14579
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Alexandre Dupriez
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16668) Enable to set tags by `ClusterTest`
[ https://issues.apache.org/jira/browse/KAFKA-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16668: -- Assignee: Johnny Hsu (was: Chia-Ping Tsai) > Enable to set tags by `ClusterTest` > > > Key: KAFKA-16668 > URL: https://issues.apache.org/jira/browse/KAFKA-16668 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Johnny Hsu >Priority: Minor > > Currently, the display name can be customized by only `name` > (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L42). > However, the "key" is hard-code to "name=xxx". Also, it is impossible to set > more "tags" for display name. > https://github.com/apache/kafka/pull/15766 is a example that we want to add > "xxx=bbb" to display name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16698) Fix flaky kafka.network.DynamicConnectionQuotaTest#testDynamicIpConnectionRateQuota
Chia-Ping Tsai created KAFKA-16698:
--
Summary: Fix flaky
kafka.network.DynamicConnectionQuotaTest#testDynamicIpConnectionRateQuota
Key: KAFKA-16698
URL: https://issues.apache.org/jira/browse/KAFKA-16698
Project: Kafka
Issue Type: Test
Reporter: Chia-Ping Tsai
{code:java}
org.opentest4j.AssertionFailedError: Timed out waiting for connection rate
update to propagate at
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38) at
app//org.junit.jupiter.api.Assertions.fail(Assertions.java:138) at
app//kafka.network.DynamicConnectionQuotaTest.updateIpConnectionRate(DynamicConnectionQuotaTest.scala:279)
at
app//kafka.network.DynamicConnectionQuotaTest.testDynamicIpConnectionRateQuota(DynamicConnectionQuotaTest.scala:255)
at
java.base@21.0.2/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base@21.0.2/java.lang.reflect.Method.invoke(Method.java:580)at
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
at
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16697) Fix flaky kafka.api.SaslGssapiSslEndToEndAuthorizationTest#testProduceConsumeWithWildcardAcls
Chia-Ping Tsai created KAFKA-16697:
--
Summary: Fix flaky
kafka.api.SaslGssapiSslEndToEndAuthorizationTest#testProduceConsumeWithWildcardAcls
Key: KAFKA-16697
URL: https://issues.apache.org/jira/browse/KAFKA-16697
Project: Kafka
Issue Type: Test
Reporter: Chia-Ping Tsai
{code:java}
org.opentest4j.AssertionFailedError: Should have been zero expired connections
killed: 1(total=0.0) ==> expected: <0> but was: <1> at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197) at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166) at
app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:664) at
app//kafka.api.EndToEndAuthorizationTest.$anonfun$confirmReauthenticationMetrics$1(EndToEndAuthorizationTest.scala:202)
at
app//kafka.api.EndToEndAuthorizationTest.$anonfun$confirmReauthenticationMetrics$1$adapted(EndToEndAuthorizationTest.scala:200)
at app//scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
at app//scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
at app//scala.collection.AbstractIterable.foreach(Iterable.scala:933)
at
app//kafka.api.EndToEndAuthorizationTest.confirmReauthenticationMetrics(EndToEndAuthorizationTest.scala:200)
at
app//kafka.api.EndToEndAuthorizationTest.testProduceConsumeWithWildcardAcls(EndToEndAuthorizationTest.scala:236)
at
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)at
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
at
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChai
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16704) Fix flaky kafka.log.remote.RemoteIndexCacheTest#testIndexFileAlreadyExistOnDiskButNotInCache
Chia-Ping Tsai created KAFKA-16704: -- Summary: Fix flaky kafka.log.remote.RemoteIndexCacheTest#testIndexFileAlreadyExistOnDiskButNotInCache Key: KAFKA-16704 URL: https://issues.apache.org/jira/browse/KAFKA-16704 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /tmp/kafka-RemoteIndexCacheTest3690189103734187552/R68MBnutRfmqJY66XXFoOA:foo-0/remote-log-index-cache/2147584984_Ma8JCqucS7mqKIHfSSDeow.txnindex.deleted at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87) at java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at kafka.log.remote.RemoteIndexCacheTest.renameRemoteCacheIndexFileFromDisk$1(RemoteIndexCacheTest.scala:832) at kafka.log.remote.RemoteIndexCacheTest.testIndexFileAlreadyExistOnDiskButNotInCache(RemoteIndexCacheTest.scala:851) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16705) the flag "started" of RaftClusterInstance is false even though the cluster is started
Chia-Ping Tsai created KAFKA-16705: -- Summary: the flag "started" of RaftClusterInstance is false even though the cluster is started Key: KAFKA-16705 URL: https://issues.apache.org/jira/browse/KAFKA-16705 Project: Kafka Issue Type: Bug Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai we should set `started` to true after https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java#L113 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-9401) High lock contention for kafka.server.FetchManager.newContext
[
https://issues.apache.org/jira/browse/KAFKA-9401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845625#comment-17845625
]
Chia-Ping Tsai commented on KAFKA-9401:
---
The PR is pushed to trunk
> High lock contention for kafka.server.FetchManager.newContext
> -
>
> Key: KAFKA-9401
> URL: https://issues.apache.org/jira/browse/KAFKA-9401
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Reporter: Lucas Bradstreet
>Assignee: Gaurav Narula
>Priority: Major
> Fix For: 3.8.0, 3.7.1
>
>
> kafka.server.FetchManager.newContext takes out what is essentially a global
> fetch lock on kafka.server.FetchSessionCache, for updates to not only the
> FetchSessionCache but the also update the fetch sessions stored with in it.
> This causes a high amount of lock contention for fetches, as every fetch
> request must go through this lock.
> I have taken an async-profiler lock profile on a high throughput cluster, and
> I see around 25s of waiting on this lock for a sixty second profile.
> {noformat}
> *— 25818577497 ns (20.84%), 5805 samples
> [ 0] kafka.server.FetchSessionCache
> [ 1] kafka.server.FetchManager.newContext
> [ 2] kafka.server.KafkaApis.handleFetchRequest
> [ 3] kafka.server.KafkaApis.handle
> [ 4] kafka.server.KafkaRequestHandler.run
> [ 5] java.lang.Thread.run
> {noformat}
> FetchSession.scala:
> {code:java}
> cache.synchronized {
> cache.get(reqMetadata.sessionId) match {
> case None => {
> debug(s"Session error for ${reqMetadata.sessionId}: no such session
> ID found.")
> new SessionErrorContext(Errors.FETCH_SESSION_ID_NOT_FOUND,
> reqMetadata)
> }
> case Some(session) => session.synchronized {
> if (session.epoch != reqMetadata.epoch) {
> debug(s"Session error for ${reqMetadata.sessionId}: expected epoch
> " +
> s"${session.epoch}, but got ${reqMetadata.epoch} instead.");
> new SessionErrorContext(Errors.INVALID_FETCH_SESSION_EPOCH,
> reqMetadata)
> } else {
> val (added, updated, removed) = session.update(fetchData, toForget,
> reqMetadata)
> if (session.isEmpty) {
> debug(s"Created a new sessionless FetchContext and closing
> session id ${session.id}, " +
> s"epoch ${session.epoch}: after removing
> ${partitionsToLogString(removed)}, " +
> s"there are no more partitions left.")
> cache.remove(session)
> new SessionlessFetchContext(fetchData)
> } else {
> cache.touch(session, time.milliseconds())
> session.epoch = JFetchMetadata.nextEpoch(session.epoch)
> debug(s"Created a new incremental FetchContext for session id
> ${session.id}, " +
> s"epoch ${session.epoch}: added
> ${partitionsToLogString(added)}, " +
> s"updated ${partitionsToLogString(updated)}, " +
> s"removed ${partitionsToLogString(removed)}")
> new IncrementalFetchContext(time, reqMetadata, session)
> }
> }
> }
> }
> }
> {code}
> Contention has been made worse by the solution for "KAFKA-9137: Fix incorrect
> FetchSessionCache eviction logic"
> ([https://github.com/apache/kafka/pull/7640),] as the cache is correctly
> touched now, whereas previously the touch was being skipped.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16679) Merge unit test down to the class of integration test
[ https://issues.apache.org/jira/browse/KAFKA-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16679. Fix Version/s: 3.8.0 Resolution: Fixed > Merge unit test down to the class of integration test > - > > Key: KAFKA-16679 > URL: https://issues.apache.org/jira/browse/KAFKA-16679 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Cheng-Kai, Zhang >Priority: Minor > Fix For: 3.8.0 > > > Normally, we don't put multi test classes into single file. Those test > classes can be extracted into a new class file. Or we can merge them into > single class by using "@Test" annotation. That can make those test cases run > without embedded cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16671) Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured
[ https://issues.apache.org/jira/browse/KAFKA-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16671: -- Assignee: PoAn Yang (was: Chia-Ping Tsai) > Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured > > > Key: KAFKA-16671 > URL: https://issues.apache.org/jira/browse/KAFKA-16671 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Minor > > loop 1000times on my local, and all pass. Let's enable the test to see what > happens in our CI -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16705) the flag "started" of RaftClusterInstance is false even though the cluster is started
[ https://issues.apache.org/jira/browse/KAFKA-16705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16705: -- Assignee: xuanzhang gong (was: Chia-Ping Tsai) > the flag "started" of RaftClusterInstance is false even though the cluster is > started > - > > Key: KAFKA-16705 > URL: https://issues.apache.org/jira/browse/KAFKA-16705 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: xuanzhang gong >Priority: Minor > > we should set `started` to true after > https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java#L113 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array
[
https://issues.apache.org/jira/browse/KAFKA-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-16677.
Fix Version/s: 3.8.0
Resolution: Fixed
> Replace ClusterType#ALL and ClusterType#DEFAULT by Array
>
>
> Key: KAFKA-16677
> URL: https://issues.apache.org/jira/browse/KAFKA-16677
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
> Fix For: 3.8.0
>
>
> Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of
> true "type". It seems to me they can be removed by using Array. For example:
> ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}
> ClusterType#DEFAULT -> {}
> There are two benefits
> 1. That is more readable for "ALL type".
> 2. We don't throw the awkward "exception" when seeing "DEFAULT".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16666) Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`to tools module
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-1: --- Labels: need-kip (was: ) > Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and > `OffsetsMessageFormatter`to tools module > - > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: need-kip > > `GroupMetadataMessageFormatter`[0], `OffsetsMessageFormatter`[1], and > `TransactionLogMessageFormatter`[2] are used by ConsoleConsumer to parse data > of internal topics. Following the migration plan, we should move them to > tools module. Also, we need to keep the compatibility of command line. That > is to say, `ConsoleConsumer` can accept the previous package name and then > use the (java) implementation to parse and make same output. > [0] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 > [1] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 > [2] > https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16470) kafka-dump-log --offsets-decoder should support new records
[ https://issues.apache.org/jira/browse/KAFKA-16470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16470. Fix Version/s: 3.8.0 Resolution: Fixed > kafka-dump-log --offsets-decoder should support new records > --- > > Key: KAFKA-16470 > URL: https://issues.apache.org/jira/browse/KAFKA-16470 > Project: Kafka > Issue Type: Sub-task >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16608) AsyncKafkaConsumer doesn't honor interrupted thread status on KafkaConsumer.poll(Duration)
[ https://issues.apache.org/jira/browse/KAFKA-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16608. Fix Version/s: 3.8.0 Resolution: Fixed > AsyncKafkaConsumer doesn't honor interrupted thread status on > KafkaConsumer.poll(Duration) > -- > > Key: KAFKA-16608 > URL: https://issues.apache.org/jira/browse/KAFKA-16608 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.8.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Minor > Fix For: 3.8.0 > > > The behaviour for KafkaConsumer.poll(Duration) when the calling thread is in > interrupted state is to throw InterruptException. The AsyncKafkaConsumer > doesn't do this. It only throws that exception if the interruption occurs > while it is waiting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16539) Can't update specific broker configs in pre-migration mode
[ https://issues.apache.org/jira/browse/KAFKA-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844071#comment-17844071 ] Chia-Ping Tsai commented on KAFKA-16539: [~davidarthur] I have pushed it to trunk. Cherry-pick causes some conflicts on branch 3.7, so maybe we need to file a PR for backporting. Also, not sure whether we need to backport to branch 3.6, because it seems we don't release x.x.3 normally. > Can't update specific broker configs in pre-migration mode > -- > > Key: KAFKA-16539 > URL: https://issues.apache.org/jira/browse/KAFKA-16539 > Project: Kafka > Issue Type: Bug > Components: config, kraft >Affects Versions: 3.6.0, 3.7.0, 3.6.1, 3.6.2 >Reporter: David Arthur >Assignee: David Arthur >Priority: Major > Fix For: 3.8.0, 3.7.1, 3.6.3 > > > In migration mode, ZK brokers will have a forwarding manager configured. This > is used to forward requests to the KRaft controller once we get to that part > of the migration. However, prior to KRaft taking over as the controller > (known as pre-migration mode), the ZK brokers are still attempting to forward > IncrementalAlterConfigs to the controller. > This works fine for cluster level configs (e.g., "-entity-type broker > --entity-default"), but this fails for specific broker configs (e.g., > "-entity-type broker --entity-id 1"). > This affects BROKER and BROKER_LOGGER config types. > To workaround this bug, you can either disable migrations on the brokers > (assuming no migration has taken place), or proceed with the migration and > get to the point where KRaft is the controller. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16681) Rewrite MiniKDC by Java
[ https://issues.apache.org/jira/browse/KAFKA-16681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16681: -- Assignee: PoAn Yang (was: Chia-Ping Tsai) > Rewrite MiniKDC by Java > --- > > Key: KAFKA-16681 > URL: https://issues.apache.org/jira/browse/KAFKA-16681 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Major > > Noted: > # we need to move it from scala folder to java folder > # don't change the package name since system tests requires it -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16689) Move LogValidatorTest to storage module
[
https://issues.apache.org/jira/browse/KAFKA-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844513#comment-17844513
]
Chia-Ping Tsai commented on KAFKA-16689:
hi [~ijuma] thanks for your reminder.
{quote}
we can create subtasks for specific tests under that if we want to break it
down.
{quote}
https://issues.apache.org/jira/browse/KAFKA-14488 is a subtask already, so I
add the link to it.
> Move LogValidatorTest to storage module
> ---
>
> Key: KAFKA-16689
> URL: https://issues.apache.org/jira/browse/KAFKA-16689
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
>
> `LogValidator` is moved to storage module already but its unit test is still
> in core module. That is a bit weird. We ought to rewrite it by java and then
> move it to storage module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16684) FetchResponse#responseData could return incorrect data
[ https://issues.apache.org/jira/browse/KAFKA-16684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16684: -- Assignee: Johnny Hsu (was: Chia-Ping Tsai) > FetchResponse#responseData could return incorrect data > -- > > Key: KAFKA-16684 > URL: https://issues.apache.org/jira/browse/KAFKA-16684 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Johnny Hsu >Priority: Minor > > [https://github.com/apache/kafka/commit/2b8aff58b575c199ee8372e5689420c9d77357a5] > make it accept input to return "partial" data. The content of output is > based on the input but we cache the output ... It will return same output > even though we pass different input. That is a potential bug. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16679) Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into FeatureCommandTest
[ https://issues.apache.org/jira/browse/KAFKA-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-16679: --- Summary: Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into FeatureCommandTest (was: Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into ) > Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, > `FeatureCommandUnitTest` into FeatureCommandTest > -- > > Key: KAFKA-16679 > URL: https://issues.apache.org/jira/browse/KAFKA-16679 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Cheng-Kai, Zhang >Priority: Minor > > Normally, we don't put multi test classes into single file. Those test > classes can be extracted into a new class file. Or we can merge them into > single class by using "@Test" annotation. That can make those test cases run > without embedded cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16689) Move LogValidatorTest to storage module
[ https://issues.apache.org/jira/browse/KAFKA-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16689: -- Assignee: TaiJuWu (was: Chia-Ping Tsai) > Move LogValidatorTest to storage module > --- > > Key: KAFKA-16689 > URL: https://issues.apache.org/jira/browse/KAFKA-16689 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: TaiJuWu >Priority: Minor > > `LogValidator` is moved to storage module already but its unit test is still > in core module. That is a bit weird. We ought to rewrite it by java and then > move it to storage module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16679) Merge unit test down to the class of integration test
[ https://issues.apache.org/jira/browse/KAFKA-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-16679: --- Summary: Merge unit test down to the class of integration test (was: Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into FeatureCommandTest) > Merge unit test down to the class of integration test > - > > Key: KAFKA-16679 > URL: https://issues.apache.org/jira/browse/KAFKA-16679 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Cheng-Kai, Zhang >Priority: Minor > > Normally, we don't put multi test classes into single file. Those test > classes can be extracted into a new class file. Or we can merge them into > single class by using "@Test" annotation. That can make those test cases run > without embedded cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest
[ https://issues.apache.org/jira/browse/KAFKA-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16678. Fix Version/s: 3.8.0 Resolution: Fixed > Remove unimplementedquorum from EndToEndAuthorizationTest > - > > Key: KAFKA-16678 > URL: https://issues.apache.org/jira/browse/KAFKA-16678 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: TengYao Chi >Priority: Minor > Fix For: 3.8.0 > > > `unimplementedquorum`[0] is used to skip test cases if they don't support to > run by kraft. However, KAFKA-15219 , KAFKA-14765 and KAFKA-14776 make related > tests support to run by kraft. > In short, it is time to remove the unused variable :) > [0] > [https://github.com/apache/kafka/blob/d76352e2151178521dc447e3406dabb8fcd4c57c/core/src/test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala#L146] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16640) Replace TestUtils#resource by scala.util.Using
[ https://issues.apache.org/jira/browse/KAFKA-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16640. Fix Version/s: 3.8.0 Resolution: Fixed > Replace TestUtils#resource by scala.util.Using > -- > > Key: KAFKA-16640 > URL: https://issues.apache.org/jira/browse/KAFKA-16640 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: TengYao Chi >Priority: Minor > Fix For: 3.8.0 > > > `scala.util.Using` is in both scala 2.13 and scala-collection-compat so we > don't need to have custom try-resource function. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16679) Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into
Chia-Ping Tsai created KAFKA-16679: -- Summary: Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into Key: KAFKA-16679 URL: https://issues.apache.org/jira/browse/KAFKA-16679 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai Normally, we don't put multi test classes into single file. Those test classes can be extracted into a new class file. Or we can merge them into single class by using "@Test" annotation. That can make those test cases run without embedded cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array
[
https://issues.apache.org/jira/browse/KAFKA-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai reassigned KAFKA-16677:
--
Assignee: PoAn Yang (was: Chia-Ping Tsai)
> Replace ClusterType#ALL and ClusterType#DEFAULT by Array
>
>
> Key: KAFKA-16677
> URL: https://issues.apache.org/jira/browse/KAFKA-16677
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
>
> Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of
> true "type". It seems to me they can be removed by using Array. For example:
> ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}
> ClusterType#DEFAULT -> {}
> There are two benefits
> 1. That is more readable for "ALL type".
> 2. We don't throw the awkward "exception" when seeing "DEFAULT".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16680) Make ClusterTestExtensions support SASL
[ https://issues.apache.org/jira/browse/KAFKA-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-16680: --- Description: This is a umbrella issue. In order to migrate more tests to new test infra, we ought to make it support SASL at least. *phase1: reuse/rewrite existent SASL utils by Java* # MiniKdc # JaasTestUtils # Move security-related helpers from scala.TestUtils # extract/rewrite non-zk code from SaslSetup to new java class *phase2: make `ClusterTest#securityProtocol` works. It does not work for kraft mode :(* # add client-related helper to generate consumer/producer/admin class with security configs # configure kraft server with security settings # migrate tests of tools to use new test infra with security was: This is a umbrella issue. In order to migrate more tests to new test infra, we ought to make it support SASL at least. *phase1: reuse/rewrite existent SASL utils by Java* # MiniKdc # JaasTestUtils # Move security-related helpers from scala.TestUtils to java.TestUtils # extract/rewrite non-zk code from SaslSetup to new java class *phase2: make `ClusterTest#securityProtocol` works. It does not work for kraft mode :(* # add client-related helper to generate consumer/producer/admin class with security configs # configure kraft server with security settings # migrate tests of tools to use new test infra with security > Make ClusterTestExtensions support SASL > --- > > Key: KAFKA-16680 > URL: https://issues.apache.org/jira/browse/KAFKA-16680 > Project: Kafka > Issue Type: New Feature >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > This is a umbrella issue. > In order to migrate more tests to new test infra, we ought to make it support > SASL at least. > *phase1: reuse/rewrite existent SASL utils by Java* > # MiniKdc > # JaasTestUtils > # Move security-related helpers from scala.TestUtils > # extract/rewrite non-zk code from SaslSetup to new java class > *phase2: make `ClusterTest#securityProtocol` works. It does not work for > kraft mode :(* > # add client-related helper to generate consumer/producer/admin class with > security configs > # configure kraft server with security settings > # migrate tests of tools to use new test infra with security > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16683) Extract security-related helpers from scala.TestUtils to java class
Chia-Ping Tsai created KAFKA-16683: -- Summary: Extract security-related helpers from scala.TestUtils to java class Key: KAFKA-16683 URL: https://issues.apache.org/jira/browse/KAFKA-16683 Project: Kafka Issue Type: Sub-task Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai We can merge them into `JaasTestUtils and then rename `JaasTestUtils` to `SecurityTestUtils. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16680) Make ClusterTestExtensions support SASL
Chia-Ping Tsai created KAFKA-16680: -- Summary: Make ClusterTestExtensions support SASL Key: KAFKA-16680 URL: https://issues.apache.org/jira/browse/KAFKA-16680 Project: Kafka Issue Type: New Feature Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai This is a umbrella issue. In order to migrate more tests to new test infra, we ought to make it support SASL at least. *phase1: reuse/rewrite existent SASL utils by Java* # MiniKdc # JaasTestUtils # Move security-related helpers from scala.TestUtils to java.TestUtils # extract/rewrite non-zk code from SaslSetup to new java class *phase2: make `ClusterTest#securityProtocol` works. It does not work for kraft mode :(* # add client-related helper to generate consumer/producer/admin class with security configs # configure kraft server with security settings # migrate tests of tools to use new test infra with security -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16589) Consider removing `ClusterInstance#createAdminClient` since callers are not sure whether they need to call close
[
https://issues.apache.org/jira/browse/KAFKA-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated KAFKA-16589:
---
Parent: KAFKA-16680
Issue Type: Sub-task (was: Improvement)
> Consider removing `ClusterInstance#createAdminClient` since callers are not
> sure whether they need to call close
>
>
> Key: KAFKA-16589
> URL: https://issues.apache.org/jira/browse/KAFKA-16589
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
>
> Sometimes we close the admin created by `createAdminClient`, and sometimes we
> don't. That is not a true problem since the `ClusterInstance` will call
> `close` when stopping.
> However, that cause a lot of inconsistent code, and in fact it does not save
> much time since creating a Admin is not a hard work. We can get
> `bootstrapServers` and `bootstrapControllers` from `ClusterInstance` easily.
>
> {code:java}
> // before
> try (Admin admin = cluster.createAdminClient()) { }
> // after v0
> try (Admin admin = Admin.create(Collections.singletonMap(
> CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
> cluster.bootstrapServers( {}
> {code}
> Personally, the `after` version is not verbose, but we can have alternatives:
> `Map clientConfigs`.
>
> {code:java}
> // after v1
> try (Admin admin = Admin.create(cluster.clientConfigs())) {}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16681) Rewrite MiniKDC by Java
Chia-Ping Tsai created KAFKA-16681: -- Summary: Rewrite MiniKDC by Java Key: KAFKA-16681 URL: https://issues.apache.org/jira/browse/KAFKA-16681 Project: Kafka Issue Type: Sub-task Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai Noted: # we need to move it from scala folder to java folder # don't change the package name since system tests requires it -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16682) Rewrite JassTestUtils by Java
Chia-Ping Tsai created KAFKA-16682: -- Summary: Rewrite JassTestUtils by Java Key: KAFKA-16682 URL: https://issues.apache.org/jira/browse/KAFKA-16682 Project: Kafka Issue Type: Sub-task Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai as title one more thing is that we should change the package name from kafka.utils to kafka.security -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16593) Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions
[ https://issues.apache.org/jira/browse/KAFKA-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16593. Fix Version/s: 3.8.0 Resolution: Fixed > Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions > -- > > Key: KAFKA-16593 > URL: https://issues.apache.org/jira/browse/KAFKA-16593 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: TengYao Chi >Priority: Minor > Fix For: 3.8.0 > > > as title. the test is in tools module and it does not need to depend on test > code of core module directly -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest
Chia-Ping Tsai created KAFKA-16678: -- Summary: Remove unimplementedquorum from EndToEndAuthorizationTest Key: KAFKA-16678 URL: https://issues.apache.org/jira/browse/KAFKA-16678 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai `unimplementedquorum`[0] is used to skip test cases if they don't support to run by kraft. However, KAFKA-15219 , KAFKA-14765 and KAFKA-14776 make related tests support to run by kraft. In short, it is time to remove the unused variable :) [0] [https://github.com/apache/kafka/blob/d76352e2151178521dc447e3406dabb8fcd4c57c/core/src/test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala#L146] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest
[ https://issues.apache.org/jira/browse/KAFKA-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16678: -- Assignee: TengYao Chi (was: Chia-Ping Tsai) > Remove unimplementedquorum from EndToEndAuthorizationTest > - > > Key: KAFKA-16678 > URL: https://issues.apache.org/jira/browse/KAFKA-16678 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: TengYao Chi >Priority: Minor > > `unimplementedquorum`[0] is used to skip test cases if they don't support to > run by kraft. However, KAFKA-15219 , KAFKA-14765 and KAFKA-14776 make related > tests support to run by kraft. > In short, it is time to remove the unused variable :) > [0] > [https://github.com/apache/kafka/blob/d76352e2151178521dc447e3406dabb8fcd4c57c/core/src/test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala#L146] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16689) Move LogValidatorTest to storage module
Chia-Ping Tsai created KAFKA-16689: -- Summary: Move LogValidatorTest to storage module Key: KAFKA-16689 URL: https://issues.apache.org/jira/browse/KAFKA-16689 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai `LogValidator` is moved to storage module already but its unit test is still in core module. That is a bit weird. We ought to rewrite it by java and then move it to storage module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array
Chia-Ping Tsai created KAFKA-16677:
--
Summary: Replace ClusterType#ALL and ClusterType#DEFAULT by Array
Key: KAFKA-16677
URL: https://issues.apache.org/jira/browse/KAFKA-16677
Project: Kafka
Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of
true "type". It seems to me they can be removed by using Array. For example:
ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}
ClusterType#DEFAULT -> {}
There are two benefits
1. That is more readable for "ALL type".
2. We don't throw the awkward "exception" when seeing "DEFAULT".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16666) Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`to tools module
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated KAFKA-1: --- Description: `GroupMetadataMessageFormatter`[0], `OffsetsMessageFormatter`[1], and `TransactionLogMessageFormatter`[2] are used by ConsoleConsumer to parse data of internal topics. Following the migration plan, we should move them to tools-api module. Also, we need to keep the compatibility of command line. That is to say, `ConsoleConsumer` can accept the previous package name and then use the (java) implementation to parse and make same output. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 [2] https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 was: `GroupMetadataMessageFormatter`[0], `OffsetsMessageFormatter`[1], and `TransactionLogMessageFormatter`[2] are used by ConsoleConsumer to parse data of internal topics. Following the migration plan, we should move them to tools module. Also, we need to keep the compatibility of command line. That is to say, `ConsoleConsumer` can accept the previous package name and then use the (java) implementation to parse and make same output. [0] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 [1] https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 [2] https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 > Migrate `TransactionLogMessageFormatter`, GroupMetadataMessageFormatter` and > `OffsetsMessageFormatter`to tools module > - > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: need-kip > > `GroupMetadataMessageFormatter`[0], `OffsetsMessageFormatter`[1], and > `TransactionLogMessageFormatter`[2] are used by ConsoleConsumer to parse data > of internal topics. Following the migration plan, we should move them to > tools-api module. Also, we need to keep the compatibility of command line. > That is to say, `ConsoleConsumer` can accept the previous package name and > then use the (java) implementation to parse and make same output. > [0] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1269 > [1] > https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1248 > [2] > https://github.com/apache/kafka/blob/9b8aac22ec7ce927a2ceb2bfe7afd57419ee946c/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala#L145 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array
[
https://issues.apache.org/jira/browse/KAFKA-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844148#comment-17844148
]
Chia-Ping Tsai commented on KAFKA-16677:
For another, we can simplify the code
(https://github.com/apache/kafka/blob/trunk/tools/src/test/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommandTestUtils.java#L77)
{code:java}
// Some comments here
ClusterConfig raftWithNewGroupCoordinator =
ClusterConfig.defaultBuilder()
.setTypes(Arrays.asList(KRAFT, CO_KRAFT))
.setName("newGroupCoordinator")
.setServerProperties(serverProperties)
.build();
{code}
> Replace ClusterType#ALL and ClusterType#DEFAULT by Array
>
>
> Key: KAFKA-16677
> URL: https://issues.apache.org/jira/browse/KAFKA-16677
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
>
> Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of
> true "type". It seems to me they can be removed by using Array. For example:
> ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}
> ClusterType#DEFAULT -> {}
> There are two benefits
> 1. That is more readable for "ALL type".
> 2. We don't throw the awkward "exception" when seeing "DEFAULT".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16761) ZkClusterInstance#start does not create ClusterConfigurableIntegrationHarness
Chia-Ping Tsai created KAFKA-16761: -- Summary: ZkClusterInstance#start does not create ClusterConfigurableIntegrationHarness Key: KAFKA-16761 URL: https://issues.apache.org/jira/browse/KAFKA-16761 Project: Kafka Issue Type: Bug Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai [https://github.com/apache/kafka/blob/440f5f6c09720bb9414524781342bbf35973c281/core/src/test/java/kafka/test/junit/ZkClusterInvocationContext.java#L103] `ClusterConfigurableIntegrationHarness` is created only in "beforeEach" phase, and that makes `ZkClusterInstance#start` does not work as it could cause NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16763) Upgrade to scala 2.12.19 and scala 2.13.14
Chia-Ping Tsai created KAFKA-16763: -- Summary: Upgrade to scala 2.12.19 and scala 2.13.14 Key: KAFKA-16763 URL: https://issues.apache.org/jira/browse/KAFKA-16763 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai scala 2.12.19 (https://github.com/scala/scala/releases/tag/v2.12.19) scala 2.13.14 (https://github.com/scala/scala/releases/tag/v2.13.14) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16763) Upgrade to scala 2.12.19 and scala 2.13.14
[ https://issues.apache.org/jira/browse/KAFKA-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned KAFKA-16763: -- Assignee: 黃竣陽 (was: Chia-Ping Tsai) > Upgrade to scala 2.12.19 and scala 2.13.14 > -- > > Key: KAFKA-16763 > URL: https://issues.apache.org/jira/browse/KAFKA-16763 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: 黃竣陽 >Priority: Minor > > scala 2.12.19 (https://github.com/scala/scala/releases/tag/v2.12.19) > > scala 2.13.14 (https://github.com/scala/scala/releases/tag/v2.13.14) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16671) Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured
[ https://issues.apache.org/jira/browse/KAFKA-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16671. Fix Version/s: 3.8.0 Resolution: Fixed > Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured > > > Key: KAFKA-16671 > URL: https://issues.apache.org/jira/browse/KAFKA-16671 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: PoAn Yang >Priority: Minor > Fix For: 3.8.0 > > > loop 1000times on my local, and all pass. Let's enable the test to see what > happens in our CI -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16773) Fix flaky QuorumControllerTest#testDelayedConfigurationOperations
Chia-Ping Tsai created KAFKA-16773: -- Summary: Fix flaky QuorumControllerTest#testDelayedConfigurationOperations Key: KAFKA-16773 URL: https://issues.apache.org/jira/browse/KAFKA-16773 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: exception while renouncing leadership: Attempt to resign from epoch 1 which is larger than the current epoch 0 at org.apache.kafka.metalog.LocalLogManager.resign(LocalLogManager.java:788) at org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1267) at org.apache.kafka.controller.QuorumController.handleEventException(QuorumController.java:546) at org.apache.kafka.controller.QuorumController.access$800(QuorumController.java:179) at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:878) at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:868) at org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:149) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:138) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:211) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:182) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.IllegalArgumentException: Attempt to resign from epoch 1 which is larger than the current epoch 0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
Chia-Ping Tsai created KAFKA-16774: -- Summary: fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled Key: KAFKA-16774 URL: https://issues.apache.org/jira/browse/KAFKA-16774 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai java.util.ConcurrentModificationException at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1720) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1686) at org.apache.kafka.streams.processor.internals.TaskManager.releaseLockedUnassignedTaskDirectories(TaskManager.java:1364) at org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceComplete(TaskManager.java:208) at org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:79) at org.apache.kafka.streams.processor.internals.StreamThreadTest.shouldCloseAllTaskProducersOnCloseIfEosEnabled(StreamThreadTest.java:1408) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16775) Fix flaky PlaintextAdminIntegrationTest#testCreateExistingTopicsThrowTopicExistsException
Chia-Ping Tsai created KAFKA-16775: -- Summary: Fix flaky PlaintextAdminIntegrationTest#testCreateExistingTopicsThrowTopicExistsException Key: KAFKA-16775 URL: https://issues.apache.org/jira/browse/KAFKA-16775 Project: Kafka Issue Type: Test Reporter: Chia-Ping Tsai org.opentest4j.AssertionFailedError: timed out waiting for topics at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38) at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:138) at app//kafka.api.BaseAdminIntegrationTest.waitForTopics(BaseAdminIntegrationTest.scala:236) at app//kafka.api.PlaintextAdminIntegrationTest.testCreateExistingTopicsThrowTopicExistsException(PlaintextAdminIntegrationTest.scala:140) at java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16668) Enable to set tags by `ClusterTest`
[ https://issues.apache.org/jira/browse/KAFKA-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16668. Fix Version/s: 3.8.0 Resolution: Fixed > Enable to set tags by `ClusterTest` > > > Key: KAFKA-16668 > URL: https://issues.apache.org/jira/browse/KAFKA-16668 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Johnny Hsu >Priority: Minor > Fix For: 3.8.0 > > > Currently, the display name can be customized by only `name` > (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L42). > However, the "key" is hard-code to "name=xxx". Also, it is impossible to set > more "tags" for display name. > https://github.com/apache/kafka/pull/15766 is a example that we want to add > "xxx=bbb" to display name. -- This message was sent by Atlassian Jira (v8.20.10#820010)
