[
https://issues.apache.org/jira/browse/CASSANDRA-21664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130469#comment-18130469
]
Patrick McFadin commented on CASSANDRA-21664:
---------------------------------------------
Fix for the jvm-dtest hangs reported above: the coalesced flush task ran a
blocking system_schema flush on the single-threaded
ScheduledExecutors.nonPeriodicTasks. That thread is shared with sstable
deletions, the LifecycleTransaction.waitForDeletions() barrier that dropping a
table waits on, and TCM commit retries. The task now only starts the flushes
and does not wait for them. New regression test:
SchemaFlushCoalesceNonBlockingTest. Branch heads: 6.0 ef2862358e, trunk
52bf89dcfe.
A/B on trunk (jvm-dtest, JDK 11/17/21), counting hangs other than
IdenticalTopologyTest, which times out in every build:
- Before the fix: 6 hangs across three runs (6.0 #96, trunk #53, trunk #55), at
least one per run.
- Control with schema_flush_coalescing_window defaulted to 0ms (trunk #54): 0.
- After the fix: 0 in trunk #56 (jvm-dtest only) and 0 in trunk #57 (full
pre-commit).
Trunk pre-commit on 52bf89dcfe: https://astro-cass.ci/job/cassandra/57/
(UNSTABLE: 32 failures, 115,920 passed). ci_summary_trunk_build57.html and
results_details_trunk_build57.tar.xz are attached. The trunk #53 artifacts
attached earlier are from before the fix.
None of the remaining failures looks related to this patch:
- Same as on 6.0 and on unrelated branches: LocalRangesAllSettledBench
(UTFDataFormatException), Accord testMultipleShards (Full/Interop/MixedRead),
the simulator tests (AccordHarrySimulationTest, EpochStressTest,
HarrySimulatorTest, ShortAccordSimulationTest, ShortPaxosSimulationTest,
SingleNodeSingleTableASTTest, TrivialSimulationTest) and IdenticalTopologyTest.
- Known flaky per Butler: AccordJournalBurnTest.testOne, and
accord.NewSchemaTest.test (39 failed runs on trunk last month).
- ReadRepairTest.readRepairTimeoutTest (JDK 17, trunk #56 only): its assertion
that the read timeout fires within 100 ms of the configured value failed. That
is a timing check on a loaded CI host, not a hang, and the test passed in #57.
The 6.0 pre-commit on ef2862358e is running (cassandra-6.0 #98). Its results
will follow.
> Make schema changes cost O(1) in the number of tables and fail loudly at
> schema limits
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21664
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21664
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Cluster/Schema
> Reporter: Patrick McFadin
> Assignee: Patrick McFadin
> Priority: Normal
> Fix For: 6.0.x, 7.x
>
> Attachments: ci_summary_cassandra-6.0_build96.html,
> ci_summary_trunk_build53.html, ci_summary_trunk_build57.html,
> results_details_cassandra-6.0_build96.tar.xz,
> results_details_trunk_build53.tar.xz, results_details_trunk_build57.tar.xz
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h2. What
> Creating 10,000 tables takes 28 minutes (ManyTablesScalingTest, single node
> in-JVM). Per CREATE TABLE at small N: ~121 ms, of which 92 ms is a
> synchronous flush of every system_schema table on the log follower
> (SchemaKeyspace.applyChanges -> flush) and 22ms a blocking TCM log flush. On
> top of that, every schema change diffs the whole schema and rebuilds
> whole-keyspace maps, so cost grows with the number of existing tables: 112 ->
> 234 ms per statement from 0 to 10,000 tables.
> Past ~31,000 minimal tables the serialised cluster metadata exceeds
> max_mutation_size and the TCM snapshot silently stops being written (WARN
> only). Table-count guardrails ship disabled.
> h2. Change
> * Schema diffs and keyspace/table map updates touch only what changed:
> Keyspaces.diff, Tables (persistent map), DistributedSchema table map,
> Keyspaces.withAddedOrUpdated, DistributedSchema.validate, ThreadLocalMeter
> array growth.
> * system_schema flush is coalesced behind a new yaml setting
> schema_flush_coalescing_window (default 1000ms; 0ms restores the synchronous
> flush). system_schema has durable_writes and is rebuilt from the cluster
> metadata log on startup, so delaying the flush cannot lose schema. Drain
> still flushes synchronously.
> * Snapshot store failure is logged at ERROR with a metric
> (TCM.SnapshotStoreFailures), and a WARN fires once serialised metadata passes
> 75% of max_mutation_size.
> * tables_warn_threshold defaults to 1000; tested envelope and per-table cost
> documented.
> * Build: the in-JVM dtest targets' -Xmx8G was overridden by junit's maxmemory
> default of 1024m.
> h2. Result
> Same test, same machine; before = synchronous flush, after = this patch.
> ||N||before||after||per statement, first -> last bucket (after)||
> |1,000|120.9 s|29.9 s|32 -> 31 ms|
> |5,000|675.1 s|204.5 s|32 -> 55 ms|
> |10,000|28.0 min|9.7 min|32 -> 95 ms|
> The fixed per-statement floor is gone. A residual ~6 ms per 1,000 existing
> tables remains (TablesDiff full scans); follow-up.
> h2. Tests
> Scaling assertions at N vs 8N on every touched path; SchemaFlushCoalesceTest;
> SchemaFlushRestartTest (30 tables, stop without drain, restart, all present);
> MetadataSnapshotSizeWarningTest; schema, tcm, guardrail and config suites;
> checkstyle clean.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
