Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
mjsax commented on PR #586: URL: https://github.com/apache/kafka-site/pull/586#issuecomment-2173743835 @stanislavkozlovski -- what is the status of this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on PR #586: URL: https://github.com/apache/kafka-site/pull/586#issuecomment-1960469437 This is still WIP, I should have marked it as such - sorry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1500034979 ## 36/documentation.html: ## @@ -54,12 +54,10 @@ Kafka 3.6 Documentation 2.6.X, 2.7.X, 2.8.X, -3.0.X, -3.1.X, -3.2.X, -3.3.X, -3.4.X, -3.5.X. +3.0.X. +3.1.X. +3.2.X. +3.3.X. Review Comment: yes, I don't intend to - I captured it in https://github.com/apache/kafka-site/pull/586/files#r1499488695 - that commit is missing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1500034521 ## 36/upgrade.html: ## @@ -214,10 +214,8 @@ Upgrading KRaft-based cl ./bin/kafka-features.sh upgrade --metadata 3.5 -Note that cluster metadata downgrade is not supported in this version since it has metadata changes. -Every https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java;>MetadataVersion -after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). -Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between. +Note that the cluster metadata version cannot be downgraded to a pre-production 3.0.x, 3.1.x, or 3.2.x version once it has been upgraded. Review Comment: ok so this was fixed in 3.7+ onwards for the old versions with https://github.com/apache/kafka/commit/aec07f76d763068feb6c1d19e4fc326cffd9c620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
mimaison commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499499148 ## 36/documentation.html: ## @@ -33,7 +33,7 @@ Documentation -Kafka 3.6 Documentation +Kafka 3.4 Documentation Review Comment: This does not seem right. This is the 3.6 documentation so it should be `Kafka 3.6`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
mimaison commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499499955 ## 36/documentation.html: ## @@ -54,12 +54,10 @@ Kafka 3.6 Documentation 2.6.X, 2.7.X, 2.8.X, -3.0.X, -3.1.X, -3.2.X, -3.3.X, -3.4.X, -3.5.X. +3.0.X. +3.1.X. +3.2.X. +3.3.X. Review Comment: This is the 3.6 docs, so it should point to all previous releases including 3.4 and 3.5. Why are we removing them? ## 36/generated/connect_rest.yaml: ## @@ -8,7 +8,7 @@ info: name: Apache 2.0 url: https://www.apache.org/licenses/LICENSE-2.0.html title: Kafka Connect REST API - version: 3.6.1 + version: 3.6.2-SNAPSHOT Review Comment: Again we don't want this change. The docs should cover the last released version for 3.6, hence 3.6.1. ## 36/ops.html: ## @@ -3984,95 +3984,27 @@ Quick Start Example - -Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage - feature, the https://github.com/apache/kafka/blob/trunk/storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java;>LocalTieredStorage - implemented for integration test can be used, which will create a temporary directory in local storage to simulate the remote storage. - - -To adopt the `LocalTieredStorage`, the test library needs to be built locally -# please checkout to the specific version tag you're using before building it -# ex: `git checkout 3.6.1` -./gradlew clean :storage:testJar -After build successfully, there should be a `kafka-storage-x.x.x-test.jar` file under `storage/build/libs`. -Next, setting configurations in the broker side to enable tiered storage feature. +Configurations Example +Here is a sample configuration to enable tiered storage feature in broker side: # Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092 remote.log.storage.system.enable=true - -# Setting the listener for the clients in RemoteLogMetadataManager to talk to the brokers. +# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage. +# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager +# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers. remote.log.metadata.manager.listener.name=PLAINTEXT - -# Please provide the implementation info for remoteStorageManager. -# This is the mandatory configuration for tiered storage. -# Here, we use the `LocalTieredStorage` built above. -remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage -remote.log.storage.manager.class.path=/PATH/TO/kafka-storage-x.x.x-test.jar - -# These 2 prefix are default values, but customizable -remote.log.storage.manager.impl.prefix=rsm.config. -remote.log.metadata.manager.impl.prefix=rlmm.config. - -# Configure the directory used for `LocalTieredStorage` -# Note, please make sure the brokers need to have access to this directory -rsm.config.dir=/tmp/kafka-remote-storage - -# This needs to be changed if number of brokers in the cluster is more than 1 -rlmm.config.remote.log.metadata.topic.replication.factor=1 - -# Try to speed up the log retention check interval for testing -log.retention.check.interval.ms=1000 -Following quick start guide to start up the kafka environment. - Then, create a topic with tiered storage enabled with configs: - - -# remote.storage.enable=true -> enables tiered storage on the topic -# local.retention.ms=1000 -> The number of milliseconds to keep the local log segment before it gets deleted. - Note that a local log segment is eligible for deletion only after it gets uploaded to remote. -# retention.ms=360 -> when segments exceed this time, the segments in remote storage will be deleted -# segment.bytes=1048576 -> for test only, to speed up the log segment rolling interval -# file.delete.delay.ms=1 -> for test only, to speed up the local-log segment file delete delay - -bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 \ ---config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=360 \ ---config segment.bytes=1048576 --config file.delete.delay.ms=1000 +After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature: +bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000 -Try to send messages to the `tieredTopic` topic to roll the
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499490542 ## 36/ops.html: ## @@ -3984,95 +3984,27 @@ Quick Start Example - -Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage - feature, the https://github.com/apache/kafka/blob/trunk/storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java;>LocalTieredStorage - implemented for integration test can be used, which will create a temporary directory in local storage to simulate the remote storage. - - -To adopt the `LocalTieredStorage`, the test library needs to be built locally -# please checkout to the specific version tag you're using before building it -# ex: `git checkout 3.6.1` -./gradlew clean :storage:testJar -After build successfully, there should be a `kafka-storage-x.x.x-test.jar` file under `storage/build/libs`. -Next, setting configurations in the broker side to enable tiered storage feature. +Configurations Example Review Comment: https://github.com/apache/kafka-site/commit/81773f6d1afe2ce8e6305299a085f8dd559d8140 was not done in AK 3.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499488695 ## 36/documentation.html: ## @@ -33,7 +33,7 @@ Documentation -Kafka 3.6 Documentation +Kafka 3.4 Documentation Review Comment: https://github.com/apache/kafka/commit/4302653d9efa0906f4b9a2f59798dd55b8510ef9 is missing in AK/3.6.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499482011 ## 36/upgrade.html: ## @@ -214,10 +214,8 @@ Upgrading KRaft-based cl ./bin/kafka-features.sh upgrade --metadata 3.5 -Note that cluster metadata downgrade is not supported in this version since it has metadata changes. -Every https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java;>MetadataVersion -after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). -Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between. +Note that the cluster metadata version cannot be downgraded to a pre-production 3.0.x, 3.1.x, or 3.2.x version once it has been upgraded. Review Comment: not sure about the source of truth here either -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499478729 ## 36/ops.html: ## @@ -3984,95 +3984,27 @@ Quick Start Example - -Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage - feature, the https://github.com/apache/kafka/blob/trunk/storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java;>LocalTieredStorage - implemented for integration test can be used, which will create a temporary directory in local storage to simulate the remote storage. - - -To adopt the `LocalTieredStorage`, the test library needs to be built locally -# please checkout to the specific version tag you're using before building it -# ex: `git checkout 3.6.1` -./gradlew clean :storage:testJar -After build successfully, there should be a `kafka-storage-x.x.x-test.jar` file under `storage/build/libs`. -Next, setting configurations in the broker side to enable tiered storage feature. +Configurations Example +Here is a sample configuration to enable tiered storage feature in broker side: # Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092 remote.log.storage.system.enable=true - -# Setting the listener for the clients in RemoteLogMetadataManager to talk to the brokers. +# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage. +# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager +# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers. remote.log.metadata.manager.listener.name=PLAINTEXT - -# Please provide the implementation info for remoteStorageManager. -# This is the mandatory configuration for tiered storage. -# Here, we use the `LocalTieredStorage` built above. -remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage -remote.log.storage.manager.class.path=/PATH/TO/kafka-storage-x.x.x-test.jar - -# These 2 prefix are default values, but customizable -remote.log.storage.manager.impl.prefix=rsm.config. -remote.log.metadata.manager.impl.prefix=rlmm.config. - -# Configure the directory used for `LocalTieredStorage` -# Note, please make sure the brokers need to have access to this directory -rsm.config.dir=/tmp/kafka-remote-storage - -# This needs to be changed if number of brokers in the cluster is more than 1 -rlmm.config.remote.log.metadata.topic.replication.factor=1 - -# Try to speed up the log retention check interval for testing -log.retention.check.interval.ms=1000 -Following quick start guide to start up the kafka environment. - Then, create a topic with tiered storage enabled with configs: - - -# remote.storage.enable=true -> enables tiered storage on the topic -# local.retention.ms=1000 -> The number of milliseconds to keep the local log segment before it gets deleted. - Note that a local log segment is eligible for deletion only after it gets uploaded to remote. -# retention.ms=360 -> when segments exceed this time, the segments in remote storage will be deleted -# segment.bytes=1048576 -> for test only, to speed up the log segment rolling interval -# file.delete.delay.ms=1 -> for test only, to speed up the local-log segment file delete delay - -bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 \ ---config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=360 \ ---config segment.bytes=1048576 --config file.delete.delay.ms=1000 +After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature: +bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000 -Try to send messages to the `tieredTopic` topic to roll the log segment: Review Comment: Need to find the commit that introduced this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499478265 ## 36/ops.html: ## @@ -3984,95 +3984,27 @@ Quick Start Example - -Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage - feature, the https://github.com/apache/kafka/blob/trunk/storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java;>LocalTieredStorage - implemented for integration test can be used, which will create a temporary directory in local storage to simulate the remote storage. - - -To adopt the `LocalTieredStorage`, the test library needs to be built locally -# please checkout to the specific version tag you're using before building it -# ex: `git checkout 3.6.1` -./gradlew clean :storage:testJar -After build successfully, there should be a `kafka-storage-x.x.x-test.jar` file under `storage/build/libs`. -Next, setting configurations in the broker side to enable tiered storage feature. +Configurations Example +Here is a sample configuration to enable tiered storage feature in broker side: # Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092 remote.log.storage.system.enable=true - -# Setting the listener for the clients in RemoteLogMetadataManager to talk to the brokers. +# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage. +# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager +# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers. remote.log.metadata.manager.listener.name=PLAINTEXT - -# Please provide the implementation info for remoteStorageManager. -# This is the mandatory configuration for tiered storage. -# Here, we use the `LocalTieredStorage` built above. -remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage -remote.log.storage.manager.class.path=/PATH/TO/kafka-storage-x.x.x-test.jar - -# These 2 prefix are default values, but customizable -remote.log.storage.manager.impl.prefix=rsm.config. -remote.log.metadata.manager.impl.prefix=rlmm.config. - -# Configure the directory used for `LocalTieredStorage` -# Note, please make sure the brokers need to have access to this directory -rsm.config.dir=/tmp/kafka-remote-storage - -# This needs to be changed if number of brokers in the cluster is more than 1 -rlmm.config.remote.log.metadata.topic.replication.factor=1 - -# Try to speed up the log retention check interval for testing -log.retention.check.interval.ms=1000 -Following quick start guide to start up the kafka environment. - Then, create a topic with tiered storage enabled with configs: - - -# remote.storage.enable=true -> enables tiered storage on the topic -# local.retention.ms=1000 -> The number of milliseconds to keep the local log segment before it gets deleted. - Note that a local log segment is eligible for deletion only after it gets uploaded to remote. -# retention.ms=360 -> when segments exceed this time, the segments in remote storage will be deleted -# segment.bytes=1048576 -> for test only, to speed up the log segment rolling interval -# file.delete.delay.ms=1 -> for test only, to speed up the local-log segment file delete delay - -bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 \ ---config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=360 \ ---config segment.bytes=1048576 --config file.delete.delay.ms=1000 +After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature: +bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000 -Try to send messages to the `tieredTopic` topic to roll the log segment: Review Comment: Seems like this should be kept? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586:
URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499477330
##
36/js/templateData.js:
##
@@ -19,6 +19,6 @@ limitations under the License.
var context={
"version": "36",
"dotVersion": "3.6",
-"fullDotVersion": "3.6.1",
+"fullDotVersion": "3.6.2-SNAPSHOT",
Review Comment:
shouldn't be snapshot, I should check out 3.6.1 and build from there I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINOR: Copy over apache/kafka/3.6 docs into here [kafka-site]
stanislavkozlovski commented on code in PR #586: URL: https://github.com/apache/kafka-site/pull/586#discussion_r1499476790 ## 36/documentation.html: ## @@ -33,7 +33,7 @@ Documentation -Kafka 3.6 Documentation +Kafka 3.4 Documentation Review Comment: this is stale ## 36/generated/connect_rest.yaml: ## @@ -8,7 +8,7 @@ info: name: Apache 2.0 url: https://www.apache.org/licenses/LICENSE-2.0.html title: Kafka Connect REST API - version: 3.6.1 + version: 3.6.2-SNAPSHOT Review Comment: shouldn't be snapshot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
