[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1658538656 The failures in JDK 8/Scala 2.12 have to do with formatting on code which hasn't been touched as part of this pull request, so I believe it is safe to go ahead. There are 2 failures in JDK 20/Scala 2.13 - one concerned with tiered storage (RemoteIndexCacheTest - cleaner did not shutdown) and one with simple production (PlaintextProducerSendTest - not enough brokers for specified RF). I believe these are transient failures as they do not manifest when running locally, but if people are unconvinced we can rerun the tests. Otherwise, I think we are ready to merge this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1658242751 It appears that in Zookeeper 3.8.0 the Zookeeper community moved away from log4j to logback (source: https://issues.apache.org/jira/browse/ZOOKEEPER-4427) In the failing test cases we attach an appender to the root logger (Kafka uses a log4j logger) and assert that we see certain log messages. Because of the mismatch in expectation (log4j) and reality (logback) these tests have started failing as the test-appender has no messages. The difference in Zookeeper dependencies between current trunk (31 July 2023) and this pull request is: ``` trunk org.apache.zookeeper:zookeeper:3.6.4 > io.netty:netty-handler:4.1.94.Final > io.netty:netty-transport-native-epoll:4.1.94.Final > org.apache.yetus:audience-annotations:0.13.0 > org.apache.zookeeper:zookeeper-jute:3.6.4 > org.slf4j:slf4j-api:1.7.36 pull request org.apache.zookeeper:zookeeper:3.8.2 > ch.qos.logback:logback-classic:1.2.10 <- NEW > ch.qos.logback:logback-core:1.2.10<- NEW > commons-io:commons-io:2.11.0 <- NEW > io.netty:netty-handler:4.1.94.Final > io.netty:netty-transport-native-epoll:4.1.94.Final > org.apache.yetus:audience-annotations:0.12.0 > org.apache.zookeeper:zookeeper-jute:3.8.2 > org.slf4j:slf4j-api:1.7.36 ``` I have removed the `ch.qos.logback:*` dependencies in the latest commit and tests have started passing locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655850239 Zoning in on ControllerIntegrationTest#testControllerMoveOnTopicCreation: ``` ... val event = appender.getMessages.find(e => e.getLevel == Level.INFO && e.getThrowableInformation != null && e.getThrowableInformation.getThrowable.getClass.getName.equals(classOf[ControllerMovedException].getName)) assertTrue(event.isDefined) <- THIS IS THE ASSERTION WHICH FAILS ... ``` However from the test logs: ``` 16:03:02.258 [controller-event-thread] INFO kafka.controller.KafkaController - [Controller id=0] Controller moved to another broker when processing TopicChange. org.apache.kafka.common.errors.ControllerMovedException: Controller epoch zkVersion check fails. Expected zkVersion = 1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655821383
At least on a first look it appears something changed/is wrong with the
loggers:
```
[2023-02-22T17:35:34.655Z] Gradle Test Run :core:unitTest > Gradle Test
Executor 136 > ConfigAdminManagerTest >
testPreprocessIncrementalWithLoggerChanges() FAILED
[2023-02-22T17:35:34.655Z] org.opentest4j.AssertionFailedError:
expected: <{AlterConfigsResource(resourceType=8, resourceName='1',
configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest',
configOperation=0, value='INFO')])=ApiError(error=NONE, message=null)}> but
was: <{AlterConfigsResource(resourceType=8, resourceName='1',
configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest',
configOperation=0, value='INFO')])=ApiError(error=INVALID_CONFIG,
message=Logger kafka.server.ConfigAdminManagerTest does not exist!)}>
[2023-02-22T17:35:34.655Z] at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:35:34.655Z] at
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:35:34.655Z] at
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:35:34.655Z] at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:35:34.655Z] at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:35:34.655Z] at
app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:35:34.656Z] at
app//kafka.server.ConfigAdminManagerTest.testPreprocessIncrementalWithLoggerChanges(ConfigAdminManagerTest.scala:352)
...
[2023-02-22T17:35:34.656Z] Gradle Test Run :core:unitTest > Gradle Test
Executor 136 > ConfigAdminManagerTest > testValidateLogLevelConfigs() FAILED
[2023-02-22T17:35:34.656Z]
org.apache.kafka.common.errors.InvalidConfigurationException: Logger
kafka.server.ConfigAdminManagerTest does not exist!
...
[2023-02-22T17:36:10.045Z] Gradle Test Run :core:unitTest > Gradle Test
Executor 136 > LoggingTest > testLoggerLevelIsResolved() FAILED
[2023-02-22T17:36:10.045Z] org.opentest4j.AssertionFailedError:
expected: but was:
[2023-02-22T17:36:10.045Z] at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:36:10.045Z] at
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:36:10.045Z] at
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:36:10.045Z] at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:36:10.045Z] at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:36:10.045Z] at
app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:36:10.045Z] at
app//kafka.utils.LoggingTest.testLoggerLevelIsResolved(LoggingTest.scala:81)
...
[2023-02-22T17:37:30.427Z] Gradle Test Run :core:unitTest > Gradle Test
Executor 173 > ConfigAdminManagerTest >
testPreprocessIncrementalWithLoggerChanges() FAILED
[2023-02-22T17:37:30.427Z] org.opentest4j.AssertionFailedError:
expected: <{AlterConfigsResource(resourceType=8, resourceName='1',
configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest',
configOperation=0, value='INFO')])=ApiError(error=NONE, message=null)}> but
was: <{AlterConfigsResource(resourceType=8, resourceName='1',
configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest',
configOperation=0, value='INFO')])=ApiError(error=INVALID_CONFIG,
message=Logger kafka.server.ConfigAdminManagerTest does not exist!)}>
[2023-02-22T17:37:30.427Z] at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:37:30.427Z] at
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:37:30.427Z] at
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:37:30.427Z] at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:37:30.427Z] at
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:37:30.427Z] at
app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:37:30.427Z] at
app//kafka.server.ConfigAdminManagerTest.testPreprocessIncrementalWithLoggerChanges(ConfigAdminManagerTest.scala:352)
...
[2023-02-22T17:37:30.427Z] Gradle Test Run :core:unitTest > Gradle Test
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655753598 At least the below tests have started to continuously fail. I am still looking as to why: ``` Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnPreferredReplicaElection() FAILED org.opentest4j.AssertionFailedError: expected: but was: at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31) at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180) at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPreferredReplicaElection(ControllerIntegrationTest.scala:1885) Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnTopicCreation() FAILED org.opentest4j.AssertionFailedError: expected: but was: at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31) at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180) at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicCreation(ControllerIntegrationTest.scala:1885) Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnPartitionReassignment() FAILED org.opentest4j.AssertionFailedError: expected: but was: at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31) at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180) at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPartitionReassignment(ControllerIntegrationTest.scala:1885) Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnTopicDeletion() FAILED org.opentest4j.AssertionFailedError: expected: but was: at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31) at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180) at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicDeletion(ControllerIntegrationTest.scala:1885) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1641957263 Hey @forlack! Thank you for the suggestion, I have upgraded the dependency so that tests can run. Heya @mimaison, I have addressed your comment about adding the latest version to the LICENSE-binary file and I would be grateful for any last review as this has been open for sometime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1628903388 > I'm not familiar with these tests so please correct me if I'm not understanding something correctly. > > First, the KIP states (Compatibility, Deprecation, and Migration Plan section): > > ``` > Users of Kafka clusters with Zookeeper clients older than 3.5.x won't be able to communicate with a 3.8.1 Zookeeper cluster. > ``` > > Now looking at the tests in `upgrade_test.py`, `test_upgrade()` tries deploying a Kafka cluster and ZooKeeper from all previous releases and upgrade them to the latest version. > > So for example, it deploys a 0.10.0.0 Kafka cluster with ZooKeeper 3.4.6. Then proceeds to upgrade ZooKeeper to the version from the latest Kafka which in this case should be 3.8.1. At this stage we should be in the scenario described in the KIP, Kafka brokers (still running 0.10.0.0) have ZooKeeper clients older than 3.5.x and have to communicate with ZooKeeper 3.8.1 which should not work. > > Can you provide the output you got when running these tests? Heya @mimaison! Yes, as mentioned in a previous comment of mine, I have confirmed that we start with an older version of Zookeeper and as part of the tests we upgrade to the latest. The KIP was written based on information from Zookeeper's official documentation as I was unaware of these tests at the time. I suspect the tests do not fail because the Zookeeper APIs Kafka uses are compatible across all versions despite what Zookeeper's documentation says. This being said, these upgrade/downgrade tests only test production + consumption so I would rather be safe than sorry and go forward with recommending to do a double-rolling-upgrade as documented in the KIP. The current server.log output of the tests is ``` ubuntu@ip-172-31-94-245:~/kafka/results$ du -sh -- */ | sort -rh 50G 2023-06-23--003/ 50G 2023-06-22--002/ ``` so I suspect it will be difficult to share via GitHub. Where would you like me to put the results? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1604416528 I ran https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/upgrade_test.py#L33 and all tests passed. I have confirmed that as part of the tests we start with an older version of Zookeeper ``` jinfo output ... /opt/kafka-3.3.1/bin/../libs/zookeeper-3.6.3.jar ``` and we end up with a new one ``` jinfo output ... /opt/kafka-dev/bin/../core/build/dependant-libs-2.13.10/zookeeper-3.8.1.jar ``` I was expecting upgrade tests to fail as detailed in the KIP, but they haven't done this. I will run https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/downgrade_test.py#L25 as well and report back my findings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1581120917 I have not, but I will aim to carry out the check in the upcoming days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov commented on PR #13260: URL: https://github.com/apache/kafka/pull/13260#issuecomment-1516273036 Hello @mimaison, @ijuma and @cmccabe! Since [KIP-902](https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1) has been accepted I have rebased this pull request and added to the documentation for 3.6. I would be grateful for all suggestions for improvements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
