Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri closed pull request #14847: KAFKA-14585: Move StorageTool to tools URL: https://github.com/apache/kafka/pull/14847 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on code in PR #14847: URL: https://github.com/apache/kafka/pull/14847#discussion_r1580757743 ## metadata/src/main/java/org/apache/kafka/metadata/properties/MetaProperties.java: ## @@ -47,7 +47,7 @@ public final class MetaProperties { /** * The property that specifies the node id. Replaces broker.id in V1. */ -static final String NODE_ID_PROP = "node.id"; +public static final String NODE_ID_PROP = "node.id"; Review Comment: Ok, done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on code in PR #14847: URL: https://github.com/apache/kafka/pull/14847#discussion_r1580759331 ## tools/src/main/java/org/apache/kafka/tools/StorageTool.java: ## @@ -0,0 +1,555 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.kafka.tools; + +import net.sourceforge.argparse4j.ArgumentParsers; +import net.sourceforge.argparse4j.inf.ArgumentParser; +import net.sourceforge.argparse4j.inf.Namespace; +import net.sourceforge.argparse4j.inf.Subparser; +import net.sourceforge.argparse4j.inf.Subparsers; +import org.apache.kafka.common.Uuid; +import org.apache.kafka.common.config.ConfigException; +import org.apache.kafka.common.metadata.FeatureLevelRecord; +import org.apache.kafka.common.metadata.UserScramCredentialRecord; +import org.apache.kafka.common.protocol.ApiMessage; +import org.apache.kafka.common.security.scram.internals.ScramFormatter; +import org.apache.kafka.common.security.scram.internals.ScramMechanism; +import org.apache.kafka.common.utils.Exit; +import org.apache.kafka.common.utils.Utils; +import org.apache.kafka.metadata.bootstrap.BootstrapDirectory; +import org.apache.kafka.metadata.bootstrap.BootstrapMetadata; +import org.apache.kafka.metadata.properties.MetaProperties; +import org.apache.kafka.metadata.properties.MetaPropertiesEnsemble; +import org.apache.kafka.metadata.properties.MetaPropertiesVersion; +import org.apache.kafka.server.common.ApiMessageAndVersion; +import org.apache.kafka.server.common.MetadataVersion; +import org.apache.kafka.server.common.ProcessRole; +import org.apache.kafka.server.config.ReplicationConfigs; +import org.apache.kafka.server.config.ServerLogConfigs; +import org.apache.kafka.storage.internals.log.LogConfig; + +import java.io.IOException; +import java.io.PrintStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.InvalidKeyException; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Properties; +import java.util.Set; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; +import java.util.stream.Collectors; + +import static net.sourceforge.argparse4j.impl.Arguments.append; +import static net.sourceforge.argparse4j.impl.Arguments.store; +import static net.sourceforge.argparse4j.impl.Arguments.storeTrue; + +public class StorageTool { +// TODO delete these constants when they are migrated from KafkaConfig +public static final String METADATA_LOG_DIR_CONFIG = "metadata.log.dir"; +public static final String UNSTABLE_METADATA_VERSIONS_ENABLE_CONFIG = "unstable.metadata.versions.enable"; +public static final String PROCESS_ROLES_CONFIG = "process.roles"; + +public static void main(String... args) { +try { + +Namespace namespace = parseArguments(args); +String command = namespace.getString("command"); +Optional logConfig = Optional.ofNullable(namespace.getString("config")).map(p -> { +try { +return new LogConfig(Utils.loadProps(p)); +} catch (IOException e) { +throw new RuntimeException(e); +} +}); + +executeCommand(namespace, command, logConfig); Review Comment: Ok. At least this allows to test the migration without generating too many conflicts with the work about migrating KafkaConfig. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on code in PR #14847: URL: https://github.com/apache/kafka/pull/14847#discussion_r1580757743 ## metadata/src/main/java/org/apache/kafka/metadata/properties/MetaProperties.java: ## @@ -47,7 +47,7 @@ public final class MetaProperties { /** * The property that specifies the node id. Replaces broker.id in V1. */ -static final String NODE_ID_PROP = "node.id"; +public static final String NODE_ID_PROP = "node.id"; Review Comment: Ok, agreed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on code in PR #14847: URL: https://github.com/apache/kafka/pull/14847#discussion_r1580741403 ## tools/src/test/java/org/apache/kafka/tools/StorageToolTest.java: ## @@ -0,0 +1,517 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.kafka.tools; + +import net.sourceforge.argparse4j.inf.Namespace; +import org.apache.kafka.common.DirectoryId; +import org.apache.kafka.common.KafkaException; +import org.apache.kafka.common.metadata.UserScramCredentialRecord; +import org.apache.kafka.common.utils.Exit; +import org.apache.kafka.common.utils.Utils; +import org.apache.kafka.metadata.bootstrap.BootstrapMetadata; +import org.apache.kafka.metadata.properties.MetaProperties; +import org.apache.kafka.metadata.properties.MetaPropertiesEnsemble; +import org.apache.kafka.metadata.properties.MetaPropertiesVersion; +import org.apache.kafka.metadata.properties.PropertiesUtils; +import org.apache.kafka.raft.RaftConfig; +import org.apache.kafka.server.common.MetadataVersion; +import org.apache.kafka.server.config.ServerLogConfigs; +import org.apache.kafka.storage.internals.log.LogConfig; +import org.apache.kafka.test.TestUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.security.InvalidKeyException; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.Properties; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static org.apache.kafka.test.TestUtils.tempFile; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@ExtendWith(MockitoExtension.class) +@Timeout(value = 40) +public class StorageToolTest { +private Properties newSelfManagedProperties() { +Properties properties = new Properties(); +properties.setProperty(ServerLogConfigs.LOG_DIRS_CONFIG, "/tmp/foo,/tmp/bar"); +properties.setProperty(StorageTool.PROCESS_ROLES_CONFIG, "controller"); +properties.setProperty(MetaProperties.NODE_ID_PROP, "2"); +properties.setProperty(RaftConfig.QUORUM_VOTERS_CONFIG, "2@localhost:9092"); +return properties; +} + +@Test +public void testConfigToLogDirectories() { +LogConfig config = new LogConfig(newSelfManagedProperties()); +assertEquals(new ArrayList<>(Arrays.asList("/tmp/bar", "/tmp/foo")), StorageTool.configToLogDirectories(config)); +} + +@Test +public void testConfigToLogDirectoriesWithMetaLogDir() { +Properties properties = newSelfManagedProperties(); +properties.setProperty(StorageTool.METADATA_LOG_DIR_CONFIG, "/tmp/baz"); +LogConfig config = new LogConfig(properties); +assertEquals(new ArrayList<>(Arrays.asList("/tmp/bar", "/tmp/baz", "/tmp/foo")), StorageTool.configToLogDirectories(config)); +} + +@Test +public void testInfoCommandOnEmptyDirectory() throws IOException { +ByteArrayOutputStream stream = new ByteArrayOutputStream(); +File tempDir = tempDir(); +try { +assertEquals(1, StorageTool.infoCommand(new PrintStream(stream), true, new ArrayList<>(Collections.singletonList(tempDir.toString(); +assertEquals("Found log directory:\n " + tempDir + "\n\nFound problem:\n " + tempDir + " is not formatted.\n\n", stream.toString()); +} finally { +Utils.delete(tempDir); +} +
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
mimaison commented on code in PR #14847: URL: https://github.com/apache/kafka/pull/14847#discussion_r1579785296 ## metadata/src/main/java/org/apache/kafka/metadata/properties/MetaProperties.java: ## @@ -47,7 +47,7 @@ public final class MetaProperties { /** * The property that specifies the node id. Replaces broker.id in V1. */ -static final String NODE_ID_PROP = "node.id"; +public static final String NODE_ID_PROP = "node.id"; Review Comment: This is really not nice that we have to do that. I wonder if we need to wait for the `NodeIdProp` from KafkaConfig to move instead of doing that. ## tools/src/test/java/org/apache/kafka/tools/StorageToolTest.java: ## @@ -0,0 +1,517 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.kafka.tools; + +import net.sourceforge.argparse4j.inf.Namespace; +import org.apache.kafka.common.DirectoryId; +import org.apache.kafka.common.KafkaException; +import org.apache.kafka.common.metadata.UserScramCredentialRecord; +import org.apache.kafka.common.utils.Exit; +import org.apache.kafka.common.utils.Utils; +import org.apache.kafka.metadata.bootstrap.BootstrapMetadata; +import org.apache.kafka.metadata.properties.MetaProperties; +import org.apache.kafka.metadata.properties.MetaPropertiesEnsemble; +import org.apache.kafka.metadata.properties.MetaPropertiesVersion; +import org.apache.kafka.metadata.properties.PropertiesUtils; +import org.apache.kafka.raft.RaftConfig; +import org.apache.kafka.server.common.MetadataVersion; +import org.apache.kafka.server.config.ServerLogConfigs; +import org.apache.kafka.storage.internals.log.LogConfig; +import org.apache.kafka.test.TestUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.security.InvalidKeyException; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.Properties; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static org.apache.kafka.test.TestUtils.tempFile; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@ExtendWith(MockitoExtension.class) +@Timeout(value = 40) +public class StorageToolTest { +private Properties newSelfManagedProperties() { +Properties properties = new Properties(); +properties.setProperty(ServerLogConfigs.LOG_DIRS_CONFIG, "/tmp/foo,/tmp/bar"); +properties.setProperty(StorageTool.PROCESS_ROLES_CONFIG, "controller"); +properties.setProperty(MetaProperties.NODE_ID_PROP, "2"); +properties.setProperty(RaftConfig.QUORUM_VOTERS_CONFIG, "2@localhost:9092"); +return properties; +} + +@Test +public void testConfigToLogDirectories() { +LogConfig config = new LogConfig(newSelfManagedProperties()); +assertEquals(new ArrayList<>(Arrays.asList("/tmp/bar", "/tmp/foo")), StorageTool.configToLogDirectories(config)); +} + +@Test +public void testConfigToLogDirectoriesWithMetaLogDir() { +Properties properties = newSelfManagedProperties(); +properties.setProperty(StorageTool.METADATA_LOG_DIR_CONFIG, "/tmp/baz"); +LogConfig config = new LogConfig(properties); +assertEquals(new ArrayList<>(Arrays.asList("/tmp/bar", "/tmp/baz", "/tmp/foo")), StorageTool.configToLogDirectories(config)); +} + +@Test +public void testInfoCommandOn
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-2068086501 @showuon @mimaison I think this is now ready for review. I think now changes are well isolated. There is no code refactoring or Kafka configuration changes, so comparison with the original code should be straightforward. I'm using `LogConfig` to get all required configurations. Most keys are taken from the configuration objects migrated from `KafkaConfig`. I only miss 3 of them, that I'm defining as constants in `StorageTool`, and will be removed as soon as they are migrated. I tried all commands and options, comparing the output with the old implementation. I also ran a local 3-nodes Kafka cluster in KRaft mode. Finally, I ran all unit and integration tests. Please have a look. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
nizhikov commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1893726661 Can we extract changes regarding `LogConfig`, `RĐ°ftConfig` into separate PR to simplify review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
showuon commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1893320028 @fvaleri , I think you could extract the refactor part of code into another PR, and ping me there when ready for review. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1872986366 Rebased. Waiting for review. Thanks. @showuon @mimaison @cmccabe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
showuon commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1853758861 OK, let's wait until https://issues.apache.org/jira/browse/KAFKA-15853 is merged, or we've got response from Colin. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1842545770 @showuon it is called when you instantiate the KafkaConfig object, so it's the constructor: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaConfig.scala#L2266 I also want to mention that there is some work going on about migrating the KafkaConfig class here: https://issues.apache.org/jira/browse/KAFKA-15853 That said, I guess the question is still opened whether we want full configuration validation on format operation or not. I have no preference, but keep in mind that this configuration may change between the format operation and first broker startup. I also see that there were some changes to the original scala code since I opened this PR. I will make sure to add them once we decide on this matter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
showuon commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1842216032 > The tool calls KafkaConfig.validateValues which runs the full set of configuration validations. @fvaleri , sorry, I didn't see where we invoke `KafkaConfig.validateValues` in StorageTool. Could you guide me where it is? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1831441198 > The tool calls KafkaConfig.validateValues which runs the full set of configuration validations. Here we only have the ones that makes sense with the tool commands, the rest will be performed at startup. If this is not fine, then we would need to migrate the whole KafkaConfig first. @cmccabe hi, may I ask your opinion on this point? Do you see it as a pre-requisite for completing this tool's migration? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-14585: Move StorageTool to tools [kafka]
fvaleri commented on PR #14847: URL: https://github.com/apache/kafka/pull/14847#issuecomment-1828371630 @mimaison @showuon this is big one, but changes are fairly isolated. It works, but there are some open points which I mention in the PR description. Let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org