Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 merged PR #15916: URL: https://github.com/apache/kafka/pull/15916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
TaiJuWu commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1606897632 ## core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala: ## @@ -64,7 +64,9 @@ import scala.jdk.CollectionConverters._ object ZkMigrationIntegrationTest { - def zkClustersForAllMigrationVersions(clusterGenerator: ClusterGenerator): Unit = { + // FIXME: test times from 8 to 25? + def zkClustersForAllMigrationVersions(): java.util.List[ClusterConfig] = { +val ret : java.util.List[ClusterConfig] = new java.util.ArrayList[ClusterConfig]() Review Comment: Done. Thanks for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1606846442 ## core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala: ## @@ -64,7 +64,9 @@ import scala.jdk.CollectionConverters._ object ZkMigrationIntegrationTest { - def zkClustersForAllMigrationVersions(clusterGenerator: ClusterGenerator): Unit = { + // FIXME: test times from 8 to 25? + def zkClustersForAllMigrationVersions(): java.util.List[ClusterConfig] = { +val ret : java.util.List[ClusterConfig] = new java.util.ArrayList[ClusterConfig]() Review Comment: ```scala Seq( MetadataVersion.IBP_3_4_IV0, MetadataVersion.IBP_3_5_IV2, MetadataVersion.IBP_3_6_IV2, MetadataVersion.IBP_3_7_IV0, MetadataVersion.IBP_3_7_IV1, MetadataVersion.IBP_3_7_IV2, MetadataVersion.IBP_3_7_IV4, MetadataVersion.IBP_3_8_IV0 ).map { mv => val serverProperties = new util.HashMap[String, String]() serverProperties.put("inter.broker.listener.name", "EXTERNAL") serverProperties.put("listeners", "PLAINTEXT://localhost:0,EXTERNAL://localhost:0") serverProperties.put("advertised.listeners", "PLAINTEXT://localhost:0,EXTERNAL://localhost:0") serverProperties.put("listener.security.protocol.map", "EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT") ClusterConfig.defaultBuilder() .setMetadataVersion(mv) .setBrokers(3) .setServerProperties(serverProperties) .setTypes(Set(Type.ZK).asJava) .build() }.asJava ``` How about using above style? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on PR #15916: URL: https://github.com/apache/kafka/pull/15916#issuecomment-2119259591 @TaiJuWu Please update all tests using `ClusterTemplate`. For example: zkClustersForAllMigrationVersions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
TaiJuWu commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1605682673 ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -122,31 +115,57 @@ public Stream provideTestTemplateInvocationContex return generatedContexts.stream(); } -void processClusterTemplate(ExtensionContext context, ClusterTemplate annot, - Consumer testInvocations) { + + +List processClusterTemplate(ExtensionContext context, ClusterTemplate annot) { // If specified, call cluster config generated method (must be static) List generatedClusterConfigs = new ArrayList<>(); Review Comment: Sorry for this mistake... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1605371498 ## core/src/test/java/kafka/test/ClusterTestExtensionsTest.java: ## @@ -62,10 +63,10 @@ public class ClusterTestExtensionsTest { } // Static methods can generate cluster configurations -static void generate1(ClusterGenerator clusterGenerator) { +static List generate1() { Map serverProperties = new HashMap<>(); serverProperties.put("foo", "bar"); -clusterGenerator.accept(ClusterConfig.defaultBuilder() +return Arrays.asList(ClusterConfig.defaultBuilder() Review Comment: `Collections.singletonList` ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -122,31 +115,57 @@ public Stream provideTestTemplateInvocationContex return generatedContexts.stream(); } -void processClusterTemplate(ExtensionContext context, ClusterTemplate annot, - Consumer testInvocations) { + + +List processClusterTemplate(ExtensionContext context, ClusterTemplate annot) { // If specified, call cluster config generated method (must be static) List generatedClusterConfigs = new ArrayList<>(); Review Comment: Please remove this unused variable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1603229776 ## tools/src/test/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommandTestUtils.java: ## @@ -71,7 +72,8 @@ static void generator(ClusterGenerator clusterGenerator) { .setServerProperties(serverProperties) .setTags(Collections.singletonList("newGroupCoordinator")) .build(); -clusterGenerator.accept(consumerGroupCoordinator); +ret.add(consumerGroupCoordinator); +return ret; Review Comment: return Arrays.asList(classicGroupCoordinator, consumerGroupCoordinator); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1603194691 ## core/src/test/java/kafka/test/junit/ClusterTestExtensionsUnitTest.java: ## @@ -33,16 +31,16 @@ public class ClusterTestExtensionsUnitTest { void testProcessClusterTemplate() { Review Comment: `@SuppressWarnings("unchecked")` is unused now ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -94,59 +93,80 @@ public Stream provideTestTemplateInvocationContex // Process the @ClusterTemplate annotation ClusterTemplate clusterTemplateAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTemplate.class); if (clusterTemplateAnnot != null) { -processClusterTemplate(context, clusterTemplateAnnot, generatedContexts::add); -if (generatedContexts.isEmpty()) { -throw new IllegalStateException("ClusterConfig generator method should provide at least one config"); -} +generatedContexts.addAll(processClusterTemplate(context, clusterTemplateAnnot)); } // Process single @ClusterTest annotation ClusterTest clusterTestAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTest.class); if (clusterTestAnnot != null) { -processClusterTest(context, clusterTestAnnot, defaults, generatedContexts::add); +generatedContexts.addAll(processClusterTest(context, clusterTestAnnot, defaults)); } // Process multiple @ClusterTest annotation within @ClusterTests ClusterTests clusterTestsAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTests.class); if (clusterTestsAnnot != null) { -for (ClusterTest annot : clusterTestsAnnot.value()) { -processClusterTest(context, annot, defaults, generatedContexts::add); -} -} - -if (generatedContexts.isEmpty()) { -throw new IllegalStateException("Please annotate test methods with @ClusterTemplate, @ClusterTest, or " + -"@ClusterTests when using the ClusterTestExtensions provider"); +generatedContexts.addAll(processClusterTests(context, clusterTestsAnnot, defaults)); } return generatedContexts.stream(); } -void processClusterTemplate(ExtensionContext context, ClusterTemplate annot, Review Comment: We can simplify the code by lambda. For example: ```java List processClusterTemplate(ExtensionContext context, ClusterTemplate annot) { if (annot.value().trim().isEmpty()) throw new IllegalStateException("ClusterTemplate value can't be empty string."); String baseDisplayName = context.getRequiredTestMethod().getName(); List contexts = generateClusterConfigurations(context, annot.value()) .stream().flatMap(config -> config.clusterTypes().stream() .map(type -> type.invocationContexts(baseDisplayName, config))).collect(Collectors.toList()); if (contexts.isEmpty()) throw new IllegalStateException("ClusterConfig generator method should provide at least one config"); return contexts; } ``` ## core/src/test/java/kafka/test/ClusterTestExtensionsTest.java: ## @@ -62,14 +64,16 @@ public class ClusterTestExtensionsTest { } // Static methods can generate cluster configurations -static void generate1(ClusterGenerator clusterGenerator) { +static List generate1() { +List ret = new ArrayList<>(); Map serverProperties = new HashMap<>(); serverProperties.put("foo", "bar"); -clusterGenerator.accept(ClusterConfig.defaultBuilder() +ret.add(ClusterConfig.defaultBuilder() .setTypes(Collections.singleton(Type.ZK)) .setServerProperties(serverProperties) .setTags(Collections.singletonList("Generated Test")) .build()); +return ret; Review Comment: `return Arrays.asList(classicGroupCoordinator, consumerGroupCoordinator);` ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -94,59 +93,80 @@ public Stream provideTestTemplateInvocationContex // Process the @ClusterTemplate annotation ClusterTemplate clusterTemplateAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTemplate.class); if (clusterTemplateAnnot != null) { -processClusterTemplate(context, clusterTemplateAnnot, generatedContexts::add); -if (generatedContexts.isEmpty()) { -throw new IllegalStateException("ClusterConfig generator method should provide at least one config"); -} +generatedContexts.addAll(processClusterTemplate(context, clusterTemplateAnnot)); }
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1600041904 ## core/src/test/java/kafka/test/annotation/Type.java: ## @@ -22,30 +22,31 @@ import kafka.test.junit.ZkClusterInvocationContext; import org.junit.jupiter.api.extension.TestTemplateInvocationContext; -import java.util.function.Consumer; +import java.util.List; +import java.util.Collections; /** * The type of cluster config being requested. Used by {@link kafka.test.ClusterConfig} and the test annotations. */ public enum Type { KRAFT { @Override -public void invocationContexts(String baseDisplayName, ClusterConfig config, Consumer invocationConsumer) { -invocationConsumer.accept(new RaftClusterInvocationContext(baseDisplayName, config, false)); +public List invocationContexts(String baseDisplayName, ClusterConfig config) { Review Comment: We don't need `List` now, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on PR #15916: URL: https://github.com/apache/kafka/pull/15916#issuecomment-2106746824 @TaiJuWu please fix the conflicts :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1597801203 ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -140,8 +141,38 @@ private void generateClusterConfigurations(ExtensionContext context, String gene ReflectionUtils.invokeMethod(method, testInstance, generator); } -private void processClusterTest(ExtensionContext context, ClusterTest annot, ClusterTestDefaults defaults, -Consumer testInvocations) { +private List processClusterTests(ExtensionContext context, + ClusterTestDefaults defaults) { + +ClusterTests clusterTestsAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTests.class); +List ret = new ArrayList<>(); + +if (clusterTestsAnnot != null) { +for (ClusterTest annot : clusterTestsAnnot.value()) { +ret.addAll(processClusterTestInternal(context, annot, defaults)); +} +} + +if (ret.isEmpty()) { +throw new IllegalStateException("processClusterTests method should provide at least one config"); +} + +return ret; +} + +private List processClusterTest(ExtensionContext context, ClusterTest annot, + ClusterTestDefaults defaults) { +List ret = processClusterTestInternal(context, annot, defaults); + +if (ret.isEmpty()) { +throw new IllegalStateException("processClusterTest method should provide at least one config"); +} + +return ret; +} + +private List processClusterTestInternal(ExtensionContext context, ClusterTest annot, Review Comment: Sorry for my unclear comments. My point was they can be merged into single method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
TaiJuWu commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1597748792 ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -93,24 +92,19 @@ public Stream provideTestTemplateInvocationContex // Process the @ClusterTemplate annotation ClusterTemplate clusterTemplateAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTemplate.class); if (clusterTemplateAnnot != null) { -processClusterTemplate(context, clusterTemplateAnnot, generatedContexts::add); -if (generatedContexts.isEmpty()) { -throw new IllegalStateException("ClusterConfig generator method should provide at least one config"); -} +generatedContexts.addAll(processClusterTemplate(context, clusterTemplateAnnot)); } // Process single @ClusterTest annotation ClusterTest clusterTestAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTest.class); if (clusterTestAnnot != null) { -processClusterTest(context, clusterTestAnnot, defaults, generatedContexts::add); +generatedContexts.addAll(processClusterTest(context, clusterTestAnnot, defaults)); } // Process multiple @ClusterTest annotation within @ClusterTests ClusterTests clusterTestsAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTests.class); if (clusterTestsAnnot != null) { -for (ClusterTest annot : clusterTestsAnnot.value()) { -processClusterTest(context, annot, defaults, generatedContexts::add); -} +generatedContexts.addAll(processClusterTests(context, defaults)); Review Comment: There is some redunant code, I need to rework it. Thanks for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
TaiJuWu commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1597748550 ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -140,8 +141,38 @@ private void generateClusterConfigurations(ExtensionContext context, String gene ReflectionUtils.invokeMethod(method, testInstance, generator); } -private void processClusterTest(ExtensionContext context, ClusterTest annot, ClusterTestDefaults defaults, -Consumer testInvocations) { +private List processClusterTests(ExtensionContext context, + ClusterTestDefaults defaults) { + +ClusterTests clusterTestsAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTests.class); +List ret = new ArrayList<>(); + +if (clusterTestsAnnot != null) { +for (ClusterTest annot : clusterTestsAnnot.value()) { +ret.addAll(processClusterTestInternal(context, annot, defaults)); +} +} + +if (ret.isEmpty()) { +throw new IllegalStateException("processClusterTests method should provide at least one config"); +} + +return ret; +} + +private List processClusterTest(ExtensionContext context, ClusterTest annot, + ClusterTestDefaults defaults) { +List ret = processClusterTestInternal(context, annot, defaults); + +if (ret.isEmpty()) { +throw new IllegalStateException("processClusterTest method should provide at least one config"); +} + +return ret; +} + +private List processClusterTestInternal(ExtensionContext context, ClusterTest annot, Review Comment: `processClusterTestInternal ` is used by `processClusterTest` and `processClusterTests` so we need to keep it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16654:Refactor kafka.test.annotation.Type and ClusterTestExtensions [kafka]
chia7712 commented on code in PR #15916: URL: https://github.com/apache/kafka/pull/15916#discussion_r1597475237 ## core/src/test/java/kafka/test/annotation/Type.java: ## @@ -22,44 +22,53 @@ import kafka.test.junit.ZkClusterInvocationContext; import org.junit.jupiter.api.extension.TestTemplateInvocationContext; -import java.util.function.Consumer; +import java.util.List; +import java.util.ArrayList; /** * The type of cluster config being requested. Used by {@link kafka.test.ClusterConfig} and the test annotations. */ public enum Type { KRAFT { @Override -public void invocationContexts(String baseDisplayName, ClusterConfig config, Consumer invocationConsumer) { -invocationConsumer.accept(new RaftClusterInvocationContext(baseDisplayName, config, false)); +public List invocationContexts(String baseDisplayName, ClusterConfig config) { +List ret = new ArrayList<>(); +ret.add(new RaftClusterInvocationContext(baseDisplayName, config, false)); +return ret; } }, CO_KRAFT { @Override -public void invocationContexts(String baseDisplayName, ClusterConfig config, Consumer invocationConsumer) { -invocationConsumer.accept(new RaftClusterInvocationContext(baseDisplayName, config, true)); +public List invocationContexts(String baseDisplayName, ClusterConfig config) { +List ret = new ArrayList<>(); +ret.add(new RaftClusterInvocationContext(baseDisplayName, config, true)); +return ret; Review Comment: ditto ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -140,8 +141,38 @@ private void generateClusterConfigurations(ExtensionContext context, String gene ReflectionUtils.invokeMethod(method, testInstance, generator); } -private void processClusterTest(ExtensionContext context, ClusterTest annot, ClusterTestDefaults defaults, -Consumer testInvocations) { +private List processClusterTests(ExtensionContext context, + ClusterTestDefaults defaults) { + +ClusterTests clusterTestsAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTests.class); +List ret = new ArrayList<>(); + +if (clusterTestsAnnot != null) { +for (ClusterTest annot : clusterTestsAnnot.value()) { +ret.addAll(processClusterTestInternal(context, annot, defaults)); +} +} + +if (ret.isEmpty()) { +throw new IllegalStateException("processClusterTests method should provide at least one config"); +} + +return ret; +} + +private List processClusterTest(ExtensionContext context, ClusterTest annot, + ClusterTestDefaults defaults) { +List ret = processClusterTestInternal(context, annot, defaults); + +if (ret.isEmpty()) { +throw new IllegalStateException("processClusterTest method should provide at least one config"); +} + +return ret; +} + +private List processClusterTestInternal(ExtensionContext context, ClusterTest annot, Review Comment: It seems we don't need `processClusterTestInternal` as it just has a check. We can move the check to `processClusterTest` ## core/src/test/java/kafka/test/annotation/Type.java: ## @@ -22,44 +22,53 @@ import kafka.test.junit.ZkClusterInvocationContext; import org.junit.jupiter.api.extension.TestTemplateInvocationContext; -import java.util.function.Consumer; +import java.util.List; +import java.util.ArrayList; /** * The type of cluster config being requested. Used by {@link kafka.test.ClusterConfig} and the test annotations. */ public enum Type { KRAFT { @Override -public void invocationContexts(String baseDisplayName, ClusterConfig config, Consumer invocationConsumer) { -invocationConsumer.accept(new RaftClusterInvocationContext(baseDisplayName, config, false)); +public List invocationContexts(String baseDisplayName, ClusterConfig config) { +List ret = new ArrayList<>(); +ret.add(new RaftClusterInvocationContext(baseDisplayName, config, false)); +return ret; Review Comment: How about `return Collections.singletonList(new RaftClusterInvocationContext(baseDisplayName, config, false));`? ## core/src/test/java/kafka/test/junit/ClusterTestExtensions.java: ## @@ -93,24 +92,19 @@ public Stream provideTestTemplateInvocationContex // Process the @ClusterTemplate annotation ClusterTemplate clusterTemplateAnnot = context.getRequiredTestMethod().getDeclaredAnnotation(ClusterTemplate.class); if (clusterTemplateAnnot != null) { -processCluster