Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
junrao commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1645128202
##
tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java:
##
@@ -146,7 +146,7 @@ public void
testDowngradeMetadataVersionWithKRaft(ClusterInstance cluster) {
);
// Change expected message to reflect possible MetadataVersion range
1-N (N increases when adding a new version)
assertEquals("Could not disable metadata.version. Invalid update
version 0 for feature " +
-"metadata.version. Local controller 3000 only supports
versions 1-21", commandOutput);
+"metadata.version. Local controller 3000 only supports
versions 1-23", commandOutput);
Review Comment:
Should we change IBP_3_7_IV4 and 3.7.-IV4 in the following code to IBP_3_8_0?
```
@ClusterTest(types = {Type.KRAFT}, metadataVersion =
MetadataVersion.IBP_3_7_IV4)
"SupportedMaxVersion: 4.0-IV0\tFinalizedVersionLevel:
3.7-IV4\t", outputWithoutEpoch(features.get(1)));
```
##
core/src/test/scala/unit/kafka/server/ApiVersionsRequestTest.scala:
##
@@ -47,7 +47,7 @@ object ApiVersionsRequestTest {
List(ClusterConfig.defaultBuilder()
.setTypes(java.util.Collections.singleton(Type.ZK))
.setServerProperties(serverProperties)
- .setMetadataVersion(MetadataVersion.IBP_4_0_IV0)
+ .setMetadataVersion(MetadataVersion.latestTesting())
Review Comment:
Should we change the MV in the following to the new production MV?
```
def testApiVersionsRequestValidationV0Template():
java.util.List[ClusterConfig] = {
val serverProperties: java.util.HashMap[String, String] =
controlPlaneListenerProperties()
serverProperties.put("unstable.api.versions.enable", "false")
serverProperties.put("unstable.feature.versions.enable", "false")
List(ClusterConfig.defaultBuilder()
.setTypes(java.util.Collections.singleton(Type.ZK))
.setMetadataVersion(MetadataVersion.IBP_3_7_IV4)
.build()).asJava
}
```
```
@ClusterTest(types = Array(Type.KRAFT, Type.CO_KRAFT), metadataVersion =
MetadataVersion.IBP_3_7_IV4, serverProperties = Array(
new ClusterConfigProperty(key = "unstable.api.versions.enable", value
= "false"),
new ClusterConfigProperty(key = "unstable.feature.versions.enable",
value = "false"),
))
```
##
metadata/src/test/java/org/apache/kafka/controller/PartitionChangeBuilderTest.java:
##
@@ -125,7 +125,7 @@ private static MetadataVersion
metadataVersionForPartitionChangeRecordVersion(sh
case (short) 1:
return MetadataVersion.IBP_3_7_IV2;
case (short) 2:
-return MetadataVersion.IBP_3_8_IV0;
+return MetadataVersion.IBP_3_9_IV1;
Review Comment:
Should we add 3.8-IV0 to the following? The intention seems to be testing
the latest production MV for each minor release. Then, I don't understand why
we include 3.6-IV0 instead of 3.6-IV2. Is there a way to avoid manually add the
latest production MV in the future?
```
@ValueSource(strings = {"3.6-IV0", "3.7-IV4"})
public void testNoLeaderEpochBumpOnIsrShrink(String
metadataVersionString) {
```
```
@ValueSource(strings = {"3.6-IV0", "3.7-IV4"})
public void testLeaderEpochBumpOnIsrShrinkWithZkMigration(String
metadataVersionString) {
```
```
@ValueSource(strings = {"3.4-IV0", "3.5-IV2", "3.6-IV0", "3.7-IV4"})
public void testNoLeaderEpochBumpOnIsrExpansion(String
metadataVersionString) {
```
```
@ValueSource(strings = {"3.4-IV0", "3.5-IV2", "3.6-IV0", "3.7-IV4"})
public void testNoLeaderEpochBumpOnIsrExpansionDuringMigration(String
metadataVersionString) {
```
```
@ValueSource(strings = {"3.4-IV0", "3.5-IV2", "3.6-IV0", "3.7-IV4"})
public void testLeaderEpochBumpOnNewReplicaSetDisjoint(String
metadataVersionString) {
```
```
@ValueSource(strings = {"3.4-IV0", "3.5-IV2", "3.6-IV0", "3.7-IV4"})
public void testNoLeaderEpochBumpOnEmptyTargetIsr(String
metadataVersionString) {
```
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -331,7 +343,7 @@ public boolean isDirectoryAssignmentSupported() {
}
public boolean isElrSupported() {
-return this.isAtLeast(IBP_3_8_IV0);
+return this.isAtLeast(IBP_3_9_IV0);
Review Comment:
This needs to be IBP_3_9_IV1 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644941547
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,20 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+
+//
+// NOTE: MetadataVersions after this point are unstable and may be changed.
+// If users attempt to use an unstable MetadataVersion, they will get an
error.
+// Please move this comment when updating the LATEST_PRODUCTION constant.
+//
+
// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+IBP_3_9_IV0(21, "3.9", "IV0", true),
Review Comment:
OK. I will put that in 3.9-IV0, and we can put ELR in 3.9-IV1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644940428
##
server-common/src/test/java/org/apache/kafka/server/common/FeaturesTest.java:
##
@@ -110,7 +110,7 @@ public void
testLatestProductionMapsToLatestMetadataVersion(Features features) {
@EnumSource(MetadataVersion.class)
public void testDefaultTestVersion(MetadataVersion metadataVersion) {
short expectedVersion;
-if (!metadataVersion.isLessThan(MetadataVersion.IBP_3_8_IV0)) {
+if (!metadataVersion.isLessThan(MetadataVersion.IBP_3_9_IV0)) {
Review Comment:
Is the one you're talking about
`testDescribeWithKRaftAndBootstrapControllers` in `FeatureCommandTest.java`? I
think it would be OK to leave that test case on 3.7-IV4, just to provide some
variety.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644937626
##
server-common/src/test/java/org/apache/kafka/server/common/MetadataVersionTest.java:
##
@@ -184,8 +184,11 @@ public void testFromVersionString() {
assertEquals(IBP_3_7_IV3,
MetadataVersion.fromVersionString("3.7-IV3"));
assertEquals(IBP_3_7_IV4,
MetadataVersion.fromVersionString("3.7-IV4"));
+assertEquals(IBP_3_8_IV0, MetadataVersion.fromVersionString("3.8"));
Review Comment:
The 3.7 comment is still true but I will add an a similar comment for 3.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644936406
##
metadata/src/test/java/org/apache/kafka/metadata/PartitionRegistrationTest.java:
##
@@ -371,7 +371,7 @@ public void
testPartitionRegistrationToRecord_ElrShouldBeNullIfEmpty() {
setPartitionEpoch(0);
List exceptions = new ArrayList<>();
ImageWriterOptions options = new ImageWriterOptions.Builder().
-setMetadataVersion(MetadataVersion.IBP_3_8_IV0).
+setMetadataVersion(MetadataVersion.IBP_3_9_IV0).
Review Comment:
good find, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644935988
##
core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala:
##
@@ -73,7 +73,7 @@ object ZkMigrationIntegrationTest {
MetadataVersion.IBP_3_7_IV2,
MetadataVersion.IBP_3_7_IV4,
MetadataVersion.IBP_3_8_IV0,
- MetadataVersion.IBP_4_0_IV0
+ MetadataVersion.IBP_3_9_IV0
Review Comment:
1. added
2. changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644934445
##
tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java:
##
@@ -321,4 +321,4 @@ public void testHandleDisableDryRun() {
"Can not disable metadata.version. Can't downgrade below 4%n" +
"quux can be disabled."), disableOutput);
}
-}
\ No newline at end of file
+}
Review Comment:
That was unintentional, will remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644933858
##
core/src/test/scala/unit/kafka/server/ApiVersionsRequestTest.scala:
##
@@ -47,7 +47,7 @@ object ApiVersionsRequestTest {
List(ClusterConfig.defaultBuilder()
.setTypes(java.util.Collections.singleton(Type.ZK))
.setServerProperties(serverProperties)
- .setMetadataVersion(MetadataVersion.IBP_4_0_IV0)
+ .setMetadataVersion(MetadataVersion.IBP_3_9_IV0)
Review Comment:
Good point. I'll just set `latestTesting` so that we don't have to keep
updating it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644933588
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -331,7 +340,7 @@ public boolean isDirectoryAssignmentSupported() {
}
public boolean isElrSupported() {
-return this.isAtLeast(IBP_3_8_IV0);
+return this.isAtLeast(IBP_3_9_IV0);
Review Comment:
Maybe. We'll move it out again if it's not done next month
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
junrao commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644931539
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,20 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+
+//
+// NOTE: MetadataVersions after this point are unstable and may be changed.
+// If users attempt to use an unstable MetadataVersion, they will get an
error.
+// Please move this comment when updating the LATEST_PRODUCTION constant.
+//
+
// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+IBP_3_9_IV0(21, "3.9", "IV0", true),
Review Comment:
Could we add a separate MV for ListOffset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347: URL: https://github.com/apache/kafka/pull/16347#discussion_r1644931292 ## clients/src/main/resources/common/message/ListOffsetsRequest.json: ## @@ -34,9 +34,12 @@ // Version 7 enables listing offsets by max timestamp (KIP-734). // // Version 8 enables listing offsets by local log start offset (KIP-405). - "validVersions": "0-8", + // + // Version 9 enables listing offsets by last tiered offset (KIP-1005). + "validVersions": "0-9", "deprecatedVersions": "0", "flexibleVersions": "6+", + "latestVersionUnstable": true, Review Comment: @junrao : right, I'll have a separate PR for 3.8 that will be a bit different (and include the revert) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
junrao commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644855151
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -331,7 +340,7 @@ public boolean isDirectoryAssignmentSupported() {
}
public boolean isElrSupported() {
-return this.isAtLeast(IBP_3_8_IV0);
+return this.isAtLeast(IBP_3_9_IV0);
Review Comment:
Since 3.9 is a short release, will ELR be supported in 3.9?
##
core/src/test/scala/unit/kafka/server/ApiVersionsRequestTest.scala:
##
@@ -47,7 +47,7 @@ object ApiVersionsRequestTest {
List(ClusterConfig.defaultBuilder()
.setTypes(java.util.Collections.singleton(Type.ZK))
.setServerProperties(serverProperties)
- .setMetadataVersion(MetadataVersion.IBP_4_0_IV0)
+ .setMetadataVersion(MetadataVersion.IBP_3_9_IV0)
Review Comment:
Could we add some comments on which MV this test should set? Is it the first
unstable MV? Could we add sth like MetadataVersion.earliestTesting() to avoid
having to keep changing it?
##
clients/src/main/resources/common/message/ListOffsetsRequest.json:
##
@@ -34,9 +34,12 @@
// Version 7 enables listing offsets by max timestamp (KIP-734).
//
// Version 8 enables listing offsets by local log start offset (KIP-405).
- "validVersions": "0-8",
+ //
+ // Version 9 enables listing offsets by last tiered offset (KIP-1005).
+ "validVersions": "0-9",
"deprecatedVersions": "0",
"flexibleVersions": "6+",
+ "latestVersionUnstable": true,
Review Comment:
@clolov : Is this change ok with you for 3.9?
@cmccabe : I guess this part won't be included in 3.8?
##
server-common/src/test/java/org/apache/kafka/server/common/MetadataVersionTest.java:
##
@@ -184,8 +184,11 @@ public void testFromVersionString() {
assertEquals(IBP_3_7_IV3,
MetadataVersion.fromVersionString("3.7-IV3"));
assertEquals(IBP_3_7_IV4,
MetadataVersion.fromVersionString("3.7-IV4"));
+assertEquals(IBP_3_8_IV0, MetadataVersion.fromVersionString("3.8"));
Review Comment:
Should we change the following comment accordingly?
`// 3.7-IV4 is the latest production version in the 3.7 line`
##
metadata/src/test/java/org/apache/kafka/metadata/PartitionRegistrationTest.java:
##
@@ -371,7 +371,7 @@ public void
testPartitionRegistrationToRecord_ElrShouldBeNullIfEmpty() {
setPartitionEpoch(0);
List exceptions = new ArrayList<>();
ImageWriterOptions options = new ImageWriterOptions.Builder().
-setMetadataVersion(MetadataVersion.IBP_3_8_IV0).
+setMetadataVersion(MetadataVersion.IBP_3_9_IV0).
Review Comment:
We have the following code in this class. Should we change IBP_3_8_IV0 to
IBP_3_9_IV0?
```
private static Stream
metadataVersionsForTestPartitionRegistration() {
return Stream.of(
MetadataVersion.IBP_3_7_IV1,
MetadataVersion.IBP_3_7_IV2,
MetadataVersion.IBP_3_8_IV0
).map(mv -> Arguments.of(mv));
}
```
##
core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala:
##
@@ -73,7 +73,7 @@ object ZkMigrationIntegrationTest {
MetadataVersion.IBP_3_7_IV2,
MetadataVersion.IBP_3_7_IV4,
MetadataVersion.IBP_3_8_IV0,
- MetadataVersion.IBP_4_0_IV0
+ MetadataVersion.IBP_3_9_IV0
Review Comment:
(1) I guess we don't support ZK migration from 4.0. Could we add a comment?
(2) We have the following code in this class. Should we bump IBP_3_8_IV0 to
IBP_3_9_IV0?
`@ClusterTest(types = Array(Type.ZK), brokers = 3, metadataVersion =
MetadataVersion.IBP_3_8_IV0, serverProperties = Array(
`
##
tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java:
##
@@ -321,4 +321,4 @@ public void testHandleDisableDryRun() {
"Can not disable metadata.version. Can't downgrade below 4%n" +
"quux can be disabled."), disableOutput);
}
-}
\ No newline at end of file
+}
Review Comment:
Is this change needed?
##
server-common/src/test/java/org/apache/kafka/server/common/FeaturesTest.java:
##
@@ -110,7 +110,7 @@ public void
testLatestProductionMapsToLatestMetadataVersion(Features features) {
@EnumSource(MetadataVersion.class)
public void testDefaultTestVersion(MetadataVersion metadataVersion) {
short expectedVersion;
-if (!metadataVersion.isLessThan(MetadataVersion.IBP_3_8_IV0)) {
+if (!metadataVersion.isLessThan(MetadataVersion.IBP_3_9_IV0)) {
Review Comment:
Should we change 3.7.-IV4 in the following code to IBP_3_8_0?
`@ClusterTest(types = {Type.KRAFT}, metadataVersion =
MetadataVersion.IBP_3_7_IV4)`
` "SupportedMaxVersion: 4.0-IV0\tFinalizedVersionLevel:
3.7-IV4\t", outputWithoutEpoch(featur
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2176593301 @junrao @clolov @dajac @jlprat : It seems like the client side of the KIP-1005 changes is not implemented. So I would propose we revert #15213 from 3.8 since the KIP is only half-complete, and feature freeze has come and gone. This will unblock the 3.8.0 release. We can then retarget KIP-1005 at 3.9.0 as usual. As part of that, we can bump the ListOffsets version and mark the latest version of ListOffsets as unstable. This will all be in 3.9-IV0. I'll update this PR in a moment with those changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
junrao commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2176493383 > Maybe someone can explain more about what https://github.com/apache/kafka/pull/15673 does exactly. @cmccabe : The main issue that https://github.com/apache/kafka/pull/15673 fixes is described in https://issues.apache.org/jira/browse/KAFKA-16480. ``` https://issues.apache.org/jira/browse/KAFKA-16154 introduced the changes to the ListOffsets API to accept latest-tiered-timestamp and return the corresponding offset. Those changes should have a) increased the version of the ListOffsets API b) increased the inter-broker protocol version c) hidden the latest version of the ListOffsets behind the latestVersionUnstable flag ``` So, let's hold off on merging this PR until we understand how to consolidate it with https://github.com/apache/kafka/pull/15673. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2176365238 @dajac Just pushed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
dajac commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2176354721 @cmccabe I don't see the last changes. Did you push? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644427304
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,20 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+
+//
+// NOTE: MetadataVersions after this point are unstable and may be changed.
+// If users attempt to use an unstable MetadataVersion, they will get an
error.
+// Please move this comment when updating the LATEST_PRODUCTION constant.
+//
+
// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+IBP_3_9_IV0(21, "3.9", "IV0", true),
// Introduce version 1 of the GroupVersion feature (KIP-848).
-IBP_4_0_IV0(21, "4.0", "IV0", false);
+IBP_4_0_IV0(22, "4.0", "IV0", true);
Review Comment:
I don't think this needs to be `true` (at the moment, at least: someone may
add metadata changes in `IBP_4_0_IV0` later). I will change it to `false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644425919
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
@dajac : Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2176042112 > @cmccabe : https://github.com/apache/kafka/pull/15673 is fixing a mistake that shouldn't be in 3.8.0. We should have bumped up the API version for ListOffset, but we didn't. To me, that seems a blocker for 3.8.0, right? The discussion thread [here](https://github.com/apache/kafka/pull/15673/files#r1624734035) seems to suggest that #15673 is adding a new feature.: > @junrao : To let the AdminClient use this, we need to add a new type of OffsetSpec and a way to set oldestAllowedVersion in ListOffsetsRequest.Build to version 9, right? >@clolov: Yes, correct, but I want to get this PR in before I move to that. I do not want to bunch all of these changes in the same PR Additionally, the PR title is "KAFKA-16480: Bump ListOffsets version, IBP version and mark last version of ListOffsets as unstable #15673". But when I look at [clients/src/main/resources/common/message/ListOffsetsRequest.json](https://github.com/apache/kafka/pull/15673/files#diff-fac7080d67da905a80126d58fc1745c9a1409de7ef7d093c2ac66a888b134633) in the PR, I do not see `"latestVersionUnstable": true`. Maybe someone can explain more about what #15673 does exactly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
dajac commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1644109817
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
We need to revert this change.
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,20 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+
+//
+// NOTE: MetadataVersions after this point are unstable and may be changed.
+// If users attempt to use an unstable MetadataVersion, they will get an
error.
+// Please move this comment when updating the LATEST_PRODUCTION constant.
+//
+
// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+IBP_3_9_IV0(21, "3.9", "IV0", true),
// Introduce version 1 of the GroupVersion feature (KIP-848).
-IBP_4_0_IV0(21, "4.0", "IV0", false);
+IBP_4_0_IV0(22, "4.0", "IV0", true);
Review Comment:
For my understanding, what's the reason for using `true` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
jlprat commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2175231482 hi @junrao I marked https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-16480 as a blocker for 3.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
junrao commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2174646474 @cmccabe : https://github.com/apache/kafka/pull/15673 is fixing a mistake that shouldn't be in 3.8.0. We should have bumped up the API version for ListOffset, but we didn't. To me, that seems a blocker for 3.8.0, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1643463801
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,12 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
-// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+// Add ELR related supports (KIP-966).
Review Comment:
I will add a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1643441501
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
I can leave it attached to IBP_4_0_IV0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2174438309 > @cmccabe @AndrewJSchofield @dajac , there is another PR adding a new metadata version of 3.8.IV1(it will be the next production ready MV), it also tries to move KIP-966 from 3.8IV0 to 4.0IV0. https://github.com/apache/kafka/pull/15673/files The PR has been modified for quite some time, If we want to make KIP-966 to 3.9IV0, I can do it in a separate PR. Thanks for pointing out this existing PR. As discussed on the mailnig list, 3.8 is done at this point. No new features are going into it. The changes from #15673 can be done in 3.9-IV0 once this PR creates it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
cmccabe commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1643436916
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -232,7 +233,7 @@ public enum MetadataVersion {
* Think carefully before you update this value. ONCE A METADATA
VERSION IS PRODUCTION,
* IT CANNOT BE CHANGED.
*/
-public static final MetadataVersion LATEST_PRODUCTION = IBP_3_7_IV4;
+public static final MetadataVersion LATEST_PRODUCTION = IBP_3_8_IV0;
Review Comment:
The whole PR needs to be backported to 3.8.
But yes, this line as well :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
CalvinConfluent commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2173868554 @cmccabe @AndrewJSchofield @dajac , there is another PR adding a new metadata version of 3.8.IV1(it will be the next production ready MV), it also tries to move KIP-966 from 3.8IV0 to 4.0IV0. https://github.com/apache/kafka/pull/15673/files The PR has been modified for quite some time, If we want to make KIP-966 to 3.9IV0, I can do it in a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
AndrewJSchofield commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1642456703
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -232,7 +233,7 @@ public enum MetadataVersion {
* Think carefully before you update this value. ONCE A METADATA
VERSION IS PRODUCTION,
* IT CANNOT BE CHANGED.
*/
-public static final MetadataVersion LATEST_PRODUCTION = IBP_3_7_IV4;
+public static final MetadataVersion LATEST_PRODUCTION = IBP_3_8_IV0;
Review Comment:
I guess this single line needs to be backported to 3.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
AndrewJSchofield commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1642454880
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
I personally would leave it attached to `IBP_4_0_IV0`. Even if there's a
3.10, we still know GV_1 is in 4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
jlprat commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2172726481 Thanks @dajac! Let's get this to 3.8 once merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
dajac commented on PR #16347: URL: https://github.com/apache/kafka/pull/16347#issuecomment-2172721012 cc @jlprat We need this on in 3.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16968: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 [kafka]
dajac commented on code in PR #16347:
URL: https://github.com/apache/kafka/pull/16347#discussion_r1642442783
##
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java:
##
@@ -202,11 +202,12 @@ public enum MetadataVersion {
// Add new fetch request version for KIP-951
IBP_3_7_IV4(19, "3.7", "IV4", false),
-// Add ELR related supports (KIP-966).
-IBP_3_8_IV0(20, "3.8", "IV0", true),
+// New version for the Kafka 3.8.0 release.
+IBP_3_8_IV0(20, "3.8", "IV0", false),
+// Add ELR related supports (KIP-966).
Review Comment:
I wonder if we should put a comment here explaining that the MV below this
point are not production ready. We could also add a few words explaining that
features must be reviewed and they must moved to the next one if they are not
ready when the next MV is made production ready.
##
server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java:
##
@@ -22,7 +22,7 @@
public enum GroupVersion implements FeatureVersion {
// Version 1 enables the consumer rebalance protocol (KIP-848).
-GV_1(1, MetadataVersion.IBP_4_0_IV0, Collections.emptyMap());
+GV_1(1, MetadataVersion.IBP_3_9_IV0, Collections.emptyMap());
Review Comment:
Should we keep it attached to `IBP_4_0_IV0` as we already know that it will
be release in 4.0? If not, we should file a blocker for 3.9 to ensure that we
change it before we release 3.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
