ConfX created CASSANDRA-19732: --------------------------------- Summary: Schema Change Timeout when Cassandra all cassandra nodes restart Key: CASSANDRA-19732 URL: https://issues.apache.org/jira/browse/CASSANDRA-19732 Project: Cassandra Issue Type: Bug Components: Cluster/Schema Reporter: ConfX
h3. What happened The Cassandra nodes cannot change schema successfully after a restart. h3. How to reproduce Put the following code under `cassandra/test/distributed/org/apache/cassandra/distributed/upgrade`. {code:java} package org.apache.cassandra.distributed.upgrade; public class DemoSchemaChangeTest extends UpgradeTestBase { @Test public void demoTest() throws Throwable { new TestCase() .nodes(2) .nodesToUpgrade(1) .upgradesToCurrentFrom(v3X) .setup((cluster) -> { cluster.schemaChangeIgnoringStoppedInstances("CREATE TABLE "+KEYSPACE+".tbl1 (id int primary key, i int)"); }) .runAfterNodeUpgrade((cluster, node) -> { cluster.get(2).shutdown(true).get(1, TimeUnit.MINUTES); cluster.schemaChangeIgnoringStoppedInstances("CREATE TABLE "+KEYSPACE+".tbl2 (id int primary key, i int)"); cluster.get(1).shutdown(true).get(1, TimeUnit.MINUTES); cluster.get(1).startup(); cluster.get(2).startup(); assertFalse(cluster.get(1).isShutdown()); assertFalse(cluster.get(2).isShutdown()); cluster.schemaChangeIgnoringStoppedInstances("CREATE TABLE "+KEYSPACE+".tbl3 (id int primary key, i int)"); assertRows(cluster.get(1).executeInternal("SELECT table_name FROM system_schema.tables WHERE keyspace_name = ?", KEYSPACE), row("tbl1"), row("tbl2"), row("tbl3")); assertRows(cluster.get(2).executeInternal("SELECT table_name FROM system_schema.tables WHERE keyspace_name = ?", KEYSPACE), row("tbl1"), row("tbl2"), row("tbl3")); }).run(); } } {code} Build and run the above tests with dtest version jars >= 4.1-alpha1. In my case, I'm using dtest-4.1-alpha1 and dtest-4.1-beta1.jar. Run it with the following command: {code:java} $ ant test-jvm-dtest-some -Duse.jdk11=true -Dtest.name=org.apache.cassandra.distributed.upgrade.DemoSchemaChangeTest {code} You will see the following error message: {code:java} [junit-timeout] Testcase: demoTest(org.apache.cassandra.distributed.upgrade.DemoSchemaChangeTest)-_jdk11: FAILED [junit-timeout] Error in test '4.1-beta1 -> [4.1-beta1]' while upgrading to '4.1-beta1'; successful upgrades [] [junit-timeout] junit.framework.AssertionFailedError: Error in test '4.1-beta1 -> [4.1-beta1]' while upgrading to '4.1-beta1'; successful upgrades [] [junit-timeout] at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:442) [junit-timeout] at org.apache.cassandra.distributed.upgrade.JVMDTestUpgradeTest.demoTest(JVMDTestUpgradeTest.java:236) [junit-timeout] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit-timeout] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit-timeout] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit-timeout] Caused by: java.lang.IllegalStateException: Schema agreement not reached. Schema versions of the instances: [7ce42f2a-d2a7-31aa-94d5-daba2e4ef24d, f55fc01d-53eb-325a-98d6-dda22d5208d0] [junit-timeout] at org.apache.cassandra.distributed.impl.AbstractCluster$ChangeMonitor.waitForCompletion(AbstractCluster.java:939) [junit-timeout] at org.apache.cassandra.distributed.impl.AbstractCluster.lambda$schemaChange$9(AbstractCluster.java:867) [junit-timeout] at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81) [junit-timeout] at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47) [junit-timeout] at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57) [junit-timeout] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [junit-timeout] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [junit-timeout] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [junit-timeout] at java.base/java.lang.Thread.run(Thread.java:829) {code} The java.lang.IllegalStateException is thrown by getMonitorTimeoutMessage() in AbstractCluster.java due to timeout. The expected behavior should be that the schema change can be successfully executed after all nodes restart. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org