[ https://issues.apache.org/jira/browse/CASSANDRA-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281118#comment-17281118 ]
Jacek Lewandowski commented on CASSANDRA-16415: ----------------------------------------------- I encountered a weird problem - see my PR for 3.11: https://github.com/apache/cassandra/pull/891 Basically there are no changes in the production code. I modified a little bit the mixed read test mostly by adding gossip and networking features to the cluster configuration (as gossiper is required for this test run). Apparently I did something wrong because the cluster does not run properly and it complains with the following exception (repeated many, many times): {noformat} [junit-timeout] ERROR 14:35:29 Unable to merge schema from /127.0.0.2 [junit-timeout] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down [junit-timeout] at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58) ~[dtest-3.11.11.jar:na] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) [na:1.8.0_212] [junit-timeout] at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:904) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:885) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:954) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:310) ~[dtest-3.11.11.jar:na] [junit-timeout] at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_212] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:310) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1398) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:367) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:404) [dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:393) [dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) [dtest-3.11.11.jar:na] [junit-timeout] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] [junit-timeout] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [dtest-3.11.11.jar:na] [junit-timeout] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212] [junit-timeout] ERROR [node1_InternalResponseStage:1] node1 2021-02-08 15:35:29,093 MigrationCoordinator.java:408 - Unable to merge schema from /127.0.0.2 [junit-timeout] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down [junit-timeout] at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58) ~[dtest-3.11.11.jar:na] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) [na:1.8.0_212] [junit-timeout] at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:904) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:885) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:954) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:310) ~[dtest-3.11.11.jar:na] [junit-timeout] at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_212] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:310) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1398) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:367) ~[dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:404) [dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:393) [dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [dtest-3.11.11.jar:na] [junit-timeout] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) [dtest-3.11.11.jar:na] [junit-timeout] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] [junit-timeout] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] [junit-timeout] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [dtest-3.11.11.jar:na] [junit-timeout] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212] {noformat} Can someone help me a bit with this? (cc [~jwest], [~blerer]) > Broken test mixedModeReadColumnSubsetDigestCheck - > org.apache.cassandra.distributed.upgrade.MixedModeReadTest > ------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-16415 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16415 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Coordination, Test/dtest/java > Reporter: Yifan Cai > Assignee: Jacek Lewandowski > Priority: Normal > Fix For: 3.11.x, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > The test has been failing and can always be reproduced in the recent CI. > Stack trace: > {code:java} > junit.framework.AssertionFailedError: Found Digest Mismatch > at > org.apache.cassandra.distributed.upgrade.MixedModeReadTest.checkTraceForDigestMismatch(MixedModeReadTest.java:89) > at > org.apache.cassandra.distributed.upgrade.MixedModeReadTest.lambda$mixedModeReadColumnSubsetDigestCheck$0(MixedModeReadTest.java:63) > at > org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:171) > at > org.apache.cassandra.distributed.upgrade.MixedModeReadTest.mixedModeReadColumnSubsetDigestCheck(MixedModeReadTest.java:76) > {code} > The initial investigation shows that > * The test only fails in the setup phase of > mixedModeReadColumnSubsetDigestCheck. The cluster version is > *Versions.Major.v3X* > * The test failure is likely a consequence of CASSANDRA-15962. After > dropping the commit in branch cassandra-3.11 and rebuild the dtest jar, the > upgrade test can pass. Meanwhile, dropping the other commits does not help. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org