[
https://issues.apache.org/jira/browse/CASSANDRA-21664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129868#comment-18129868
]
Patrick McFadin commented on CASSANDRA-21664:
---------------------------------------------
Summary: the per-statement cost of a schema change no longer depends on the
table count. Creating 10k tables goes from 28.0 to 9.7 min, and 1k from 121 s
to 30 s. Snapshot-store failures are now logged at ERROR.
Why 6.0: 6.0 is the first release where TCM serialises every DDL through the
log. Operators running multi-tenant clusters with thousands of tables hit this
as minutes-long schema rollouts and silent snapshot failures. Both new
operator-visible behaviours (schema_flush_coalescing_window,
tables_warn_threshold=1000) can be reverted with one yaml line. The flush
default is under discussion on dev@ ("[DISCUSS] Making the synchronous
system_schema flush on every DDL async").
6.0 PR: https://github.com/apache/cassandra/pull/5128 (a40069ba77)
Trunk branch:
https://github.com/pmcfadin/cassandra/tree/pmcfadin/schema-at-scale-trunk
(8de4c18b68)
The CI artifacts for both will be attached.
> Make schema changes cost O(1) in the number of tables and fail loudly at
> schema limits
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21664
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21664
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Cluster/Schema
> Reporter: Patrick McFadin
> Assignee: Patrick McFadin
> Priority: Normal
> Fix For: 6.0.x, 7.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h2. What
> Creating 10,000 tables takes 28 minutes (ManyTablesScalingTest, single node
> in-JVM). Per CREATE TABLE at small N: ~121 ms, of which 92 ms is a
> synchronous flush of every system_schema table on the log follower
> (SchemaKeyspace.applyChanges -> flush) and 22ms a blocking TCM log flush. On
> top of that, every schema change diffs the whole schema and rebuilds
> whole-keyspace maps, so cost grows with the number of existing tables: 112 ->
> 234 ms per statement from 0 to 10,000 tables.
> Past ~31,000 minimal tables the serialised cluster metadata exceeds
> max_mutation_size and the TCM snapshot silently stops being written (WARN
> only). Table-count guardrails ship disabled.
> h2. Change
> * Schema diffs and keyspace/table map updates touch only what changed:
> Keyspaces.diff, Tables (persistent map), DistributedSchema table map,
> Keyspaces.withAddedOrUpdated, DistributedSchema.validate, ThreadLocalMeter
> array growth.
> * system_schema flush is coalesced behind a new yaml setting
> schema_flush_coalescing_window (default 1000ms; 0ms restores the synchronous
> flush). system_schema has durable_writes and is rebuilt from the cluster
> metadata log on startup, so delaying the flush cannot lose schema. Drain
> still flushes synchronously.
> * Snapshot store failure is logged at ERROR with a metric
> (TCM.SnapshotStoreFailures), and a WARN fires once serialised metadata passes
> 75% of max_mutation_size.
> * tables_warn_threshold defaults to 1000; tested envelope and per-table cost
> documented.
> * Build: the in-JVM dtest targets' -Xmx8G was overridden by junit's maxmemory
> default of 1024m.
> h2. Result
> Same test, same machine; before = synchronous flush, after = this patch.
> ||N||before||after||per statement, first -> last bucket (after)||
> |1,000|120.9 s|29.9 s|32 -> 31 ms|
> |5,000|675.1 s|204.5 s|32 -> 55 ms|
> |10,000|28.0 min|9.7 min|32 -> 95 ms|
> The fixed per-statement floor is gone. A residual ~6 ms per 1,000 existing
> tables remains (TablesDiff full scans); follow-up.
> h2. Tests
> Scaling assertions at N vs 8N on every touched path; SchemaFlushCoalesceTest;
> SchemaFlushRestartTest (30 tables, stop without drain, restart, all present);
> MetadataSnapshotSizeWarningTest; schema, tcm, guardrail and config suites;
> checkstyle clean.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
