[ https://issues.apache.org/jira/browse/CASSANDRA-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17777150#comment-17777150 ]
Jacek Lewandowski commented on CASSANDRA-18747: ----------------------------------------------- Few problems with testing... On 4.1, one junit test failed but I don't believe that is related: https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/977/workflows/f6f81939-5b1e-4e17-b756-bfb8791dc23d/jobs/38838/tests On 5.0, it turned out that {{ShortPaxosSimulationTest}} is failing. I've retried both locally and on CI with {{cassandra-5.0}} and it fails there as well: https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1013/workflows/132ec030-5eae-4622-ad92-ffea76cbb5a4/jobs/39718 Additionally, {{LeaveAndBootstrapTest}} turned out to be flaky. I did some investigation and discovered that this particular test is very sensitive to the execution speed - when the test is executed a bit slower, it fails consistently in the same way (both {{cassandra-5.0}} and this feature branch). The key temporal option in this case is either Gossiper delay/interval or quarantine delay. The failed runs were slower and Gossiper check state task was executed wiping some endpoint from the state map due to elapsed quarantine time. Thas is clearly visible in the logs. I'm hoping to fix this issue by increasing the quarantine time so that it is less likely to elapse during the test. On trunk, all tests passed. > Test failure: Fix assertion error AssertionError: Unknown keyspace > system_auth\n\tat > org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)\n\tat > org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162) > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-18747 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18747 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema, Test/dtest/python > Reporter: Ekaterina Dimitrova > Assignee: Jacek Lewandowski > Priority: Normal > Fix For: 4.1.x, 5.0.x, 5.x > > Time Spent: 7.5h > Remaining Estimate: 0h > > I've been seeing this assertion error in different tests lately. > Full error message: > {code:java} > failed on teardown with "Unexpected error found in node logs (see stdout for > full details). Errors: [[node2] 'ERROR [PendingRangeCalculator:1] 2023-08-11 > 16:35:14,445 JVMStabilityInspector.java:70 - Exception in thread > Thread[PendingRangeCalculator:1,5,PendingRangeCalculator]\njava.lang.AssertionError: > Unknown keyspace system_auth\n\tat > org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)\n\tat > org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162)\n\tat > org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)\n\tat > > org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)\n\tat > org.apache.cassandra.db.Keyspace.open(Keyspace.java:162)\n\tat > org.apache.cassandra.db.Keyspace.open(Keyspace.java:151)\n\tat > org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)\n\tat > > org.apache.cassandra.concurrent.SingleThreadExecutorPlus$AtLeastOnce.run(SingleThreadExecutorPlus.java:60)\n\tat > > org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)\n\tat > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat > java.base/java.lang.Thread.run(Thread.java:829)']" Unexpected error found in > node logs (see stdout for full details). Errors: [[node2] 'ERROR > [PendingRangeCalculator:1] 2023-08-11 16:35:14,445 > JVMStabilityInspector.java:70 - Exception in thread > Thread[PendingRangeCalculator:1,5,PendingRangeCalculator]\njava.lang.AssertionError: > Unknown keyspace system_auth\n\tat > org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)\n\tat > org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162)\n\tat > org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)\n\tat > > org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)\n\tat > org.apache.cassandra.db.Keyspace.open(Keyspace.java:162)\n\tat > org.apache.cassandra.db.Keyspace.open(Keyspace.java:151)\n\tat > org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)\n\tat > > org.apache.cassandra.concurrent.SingleThreadExecutorPlus$AtLeastOnce.run(SingleThreadExecutorPlus.java:60)\n\tat > > org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)\n\tat > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat > java.base/java.lang.Thread.run(Thread.java:829)']{code} > Example failures: > test_failed_snitch_update_property_file_snitch - > [https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2475/workflows/2086619e-0f21-464b-a866-84aca516b5e5/jobs/36716/tests] > test_gcgs_validation - > [https://ci-cassandra.apache.org/job/Cassandra-trunk/1666/testReport/junit/dtest.materialized_views_test/TestMaterializedViews/test_gcgs_validation/] -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org