[
https://issues.apache.org/jira/browse/CASSANDRA-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Haddad updated CASSANDRA-21667:
-----------------------------------
Change Category: Performance
Complexity: Low Hanging Fruit
Component/s: Observability/Metrics
Status: Open (was: Triage Needed)
> Reduce allocation from metrics collection
> ------------------------------------------
>
> Key: CASSANDRA-21667
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21667
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Observability/Metrics
> Reporter: Jon Haddad
> Assignee: Jon Haddad
> Priority: Normal
>
> The {{EstimatedPartitionCount}} table gauge recomputes its value from scratch
> every time it is read. Each read opens and deserialises the
> {{Statistics.db}} component of every live SSTable in the table, builds a
> HyperLogLog++ estimator from each one, and merges them. Nothing is cached.
> A metrics agent scrapes this gauge on every collection interval. {{nodetool
> tablestats}} reads it as well. The work is therefore repeated at the scrape
> rate, per table, for the life of the node.
> The inputs are immutable. An SSTable's cardinality estimator never changes
> after the file is written. The merged value can only change when the SSTable
> set changes.
> {{TableMetrics}} registers the gauge:
> {code:java}
> estimatedPartitionCount = createTableGauge("EstimatedPartitionCount",
> "EstimatedRowCount", new Gauge<Long>()
> {
> public Long getValue()
> {
> long memtablePartitions = 0;
> for (Memtable memtable : cfs.getTracker().getView().getAllMemtables())
> memtablePartitions += memtable.partitionCount();
> try(ColumnFamilyStore.RefViewFragment refViewFragment =
> cfs.selectAndReference(View.selectFunction(SSTableSet.CANONICAL)))
> {
> return
> SSTableReader.getApproximateKeyCount(refViewFragment.sstables) +
> memtablePartitions;
> }
> }
> }, null);
> {code}
> {{SSTableReader.getApproximateKeyCount}} then does this per SSTable:
> {code:java}
> CompactionMetadata metadata =
> StatsComponent.load(sstable.descriptor).compactionMetadata();
> ...
> if (cardinality == null)
> cardinality = metadata.cardinalityEstimator;
> else
> cardinality = cardinality.merge(metadata.cardinalityEstimator);
> {code}
> Three costs come out of those two lines.
> * {{StatsComponent.load(descriptor)}} passes {{MetadataType.values()}}. It
> deserialises {{VALIDATION}}, {{COMPACTION}}, {{STATS}} and {{HEADER}}. Only
> {{COMPACTION}} is used here. The other three are decoded and discarded.
> * Every call reopens the file. {{RandomAccessReader.open}} is called per
> SSTable, per scrape.
> * {{ICardinality.merge}} allocates a whole new estimator per SSTable and
> copies the running one into it.
> The per-table overhead was measured with JMH, {{AverageTime}} mode plus the
> {{gc}} profiler, on a table with 1000 partitions per SSTable and
> auto-compaction disabled. The benchmark reads the gauge and reports the cost
> of one call.
> ||SSTables in the table||Time per read||Heap allocated per read||
> |1|~49 us|~79 KiB|
> |10|~0.49 ms|~0.79 MiB|
> |100|4.9 ms|7.9 MiB|
> The cost is linear in the SSTable count. It does not depend on the partition
> count, because the estimator is a fixed-width structure once it leaves sparse
> mode.
> Per SSTable, per scrape: *~49 us of CPU and ~79 KiB of garbage*.
> That ~79 KiB divides into two parts.
> *Cardinality estimator: ~29 KiB per SSTable.* Measured separately with
> {{ThreadMXBean.getThreadAllocatedBytes}} over a deserialise-and-merge loop,
> at the precision Cassandra uses ({{new HyperLogLogPlus(13, 25)}}). A dense
> estimator at that precision serialises to 5473 bytes. Three copies of it
> exist per SSTable during the merge:
> * the {{byte[]}} read out of {{Statistics.db}}, ~5.5 KiB;
> * the register set of the deserialised estimator, {{int[1366]}}, ~5.5 KiB;
> * the fresh estimator that {{merge()}} allocates to hold the result, ~5.5 KiB.
> The rest is temporaries inside the deserialise and the merge.
> *The remaining metadata: ~50 KiB per SSTable.* This is the {{VALIDATION}},
> {{STATS}} and {{HEADER}} components that the caller never reads.
> {{StatsMetadata}} alone carries two {{EstimatedHistogram}} instances, a
> tombstone histogram, an {{IntervalSet}} of commit log positions, the covered
> clustering slice, the first and last keys and the encoding stats.
> {{SerializationHeader.Component}} carries the column maps. All of it is
> built and dropped.
> At the node level the arithmetic is {{tables x SSTables per table x 79 KiB /
> scrape interval}}.
> ||Node||Garbage per scrape||At a 10 s scrape interval||
> |100 tables, 20 SSTables each|~154 MiB|~15 MiB/s|
> |200 tables, 50 SSTables each|~790 MiB|~79 MiB/s|
> |500 tables, 50 SSTables each|~1.9 GiB|~193 MiB/s|
> This is short-lived garbage, so it lands in the young generation. It still
> sets a floor on the young collection rate that no workload asked for. On a
> node with many tables the scrape alone can dominate the allocation rate at
> idle.
> The CPU cost lands on the JMX or agent thread, not on a read path. The
> allocation cost lands on every thread, because it drives the collector.
> Two more per-scrape costs sit next to this one. Both read data that changes
> far less often than the scrape rate.
> * {{SecondaryIndexManager.getAllIndexColumnFamilyStores()}} builds a new
> {{HashSet}} and walks every index on every call. The backing tables only
> change when an index is registered or dropped. {{TableMetrics}} reads it per
> scrape, and {{ColumnFamilyStore.concatWithIndexes()}} reads it on the flush
> and compaction paths.
> * {{CompactionStrategyManager.getEstimatedRemainingTasks()}} calls
> {{getAllStrategies()}}, which builds a concat and a transform chain per call,
> to sum a handful of integers. It backs the {{PendingCompactions}} gauge.
> A per-table cache of the merged partition count should remove most of this.
> The value is a pure function of the SSTable set, so the cache holds until
> that set changes, and it must be invalidated whenever the SSTables change for
> any reason: flush, compaction, streaming, bulk load, index build, cleanup,
> secondary index rebuild, and any other lifecycle transaction. A cache that
> only watches compaction is wrong.
> With that in place a scrape reads a cached long instead of touching the file
> system, and the ~79 KiB per SSTable per scrape becomes ~79 KiB per SSTable
> set change.
> A JMH benchmark that measures both the cached and the uncached path is on the
> {{metrics-gauge-caching}} branch as {{EstimatedPartitionCountBench}}. It
> parameterises on the SSTable count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
