[
https://issues.apache.org/jira/browse/CASSANDRA-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17925434#comment-17925434
]
Dmitry Konstantinov edited comment on CASSANDRA-20250 at 2/9/25 10:29 PM:
--------------------------------------------------------------------------
I re-run e2e write tests using the current trunk (69dc5d05 revision) as VS same
revision + all metrics updates are disabled in write path
h2. Test 1: max throughput
{code:java}
./tools/bin/cassandra-stress "write n=10m" -rate threads=100 -node myhost{code}
h3. With metrics
{code:java}
Results:
Op rate : 161,099 op/s [WRITE: 161,099 op/s]
Partition rate : 161,099 pk/s [WRITE: 161,099 pk/s]
Row rate : 161,099 row/s [WRITE: 161,099 row/s]
Latency mean : 0.6 ms [WRITE: 0.6 ms]
Latency median : 0.5 ms [WRITE: 0.5 ms]
Latency 95th percentile : 0.9 ms [WRITE: 0.9 ms]
Latency 99th percentile : 1.3 ms [WRITE: 1.3 ms]
Latency 99.9th percentile : 7.8 ms [WRITE: 7.8 ms]
Latency max : 96.5 ms [WRITE: 96.5 ms]
Total partitions : 10,000,000 [WRITE: 10,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0 B
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:01:02
{code}
CPU flamegraph: [^5.1_profile_cpu.html]
"codahale" code: 8.65%
h3. Without metrics
{code:java}
Results:
Op rate : 170,530 op/s [WRITE: 170,530 op/s]
Partition rate : 170,530 pk/s [WRITE: 170,530 pk/s]
Row rate : 170,530 row/s [WRITE: 170,530 row/s]
Latency mean : 0.6 ms [WRITE: 0.6 ms]
Latency median : 0.5 ms [WRITE: 0.5 ms]
Latency 95th percentile : 0.8 ms [WRITE: 0.8 ms]
Latency 99th percentile : 1.2 ms [WRITE: 1.2 ms]
Latency 99.9th percentile : 8.9 ms [WRITE: 8.9 ms]
Latency max : 99.2 ms [WRITE: 99.2 ms]
Total partitions : 10,000,000 [WRITE: 10,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0 B
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:00:58
{code}
CPU flamegraph: [^5.1_profile_cpu_without_metrics.html]
h2. Test 2: fixed rate and CPU usage comparison
{code:java}
./tools/bin/cassandra-stress "write n=10m" -rate threads=100 fixed=120000/s
-node myhost{code}
CPU is captured using:
{code:java}
vmstat -w 1
{code}
as 100% - idle
h3. With metrics
avg CPU usage: 71%
[^vmstat.log]
h3. Without metrics
avg CPU usage: 66%
[^vmstat_without_metrics.log]
Absolute delta: 5%, relative delta 5%/71% = 7% - it is not equal but close
enough to 8.65% reported in framegraph for metrics code.
So, I more or less convinced myself that it is not a skid in framegraph and %
of metrics overhead is more or less real.
Continuing with delta + Int2IntHashMap implementations..
was (Author: dnk):
I re-run e2e write tests using the current trunk (69dc5d05 revision) as VS same
revision + all metrics updates are disabled in write path
h2. Test 1: max throughput
{code:java}
./tools/bin/cassandra-stress "write n=10m" -rate threads=100 -node myhost{code}
h3. With metrics
{code:java}
Results:
Op rate : 161,099 op/s [WRITE: 161,099 op/s]
Partition rate : 161,099 pk/s [WRITE: 161,099 pk/s]
Row rate : 161,099 row/s [WRITE: 161,099 row/s]
Latency mean : 0.6 ms [WRITE: 0.6 ms]
Latency median : 0.5 ms [WRITE: 0.5 ms]
Latency 95th percentile : 0.9 ms [WRITE: 0.9 ms]
Latency 99th percentile : 1.3 ms [WRITE: 1.3 ms]
Latency 99.9th percentile : 7.8 ms [WRITE: 7.8 ms]
Latency max : 96.5 ms [WRITE: 96.5 ms]
Total partitions : 10,000,000 [WRITE: 10,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0 B
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:01:02
{code}
CPU flamegraph: [^5.1_profile_cpu.html]
"codahale" code: 8.65%
h3. Without metrics
{code:java}
Results:
Op rate : 170,530 op/s [WRITE: 170,530 op/s]
Partition rate : 170,530 pk/s [WRITE: 170,530 pk/s]
Row rate : 170,530 row/s [WRITE: 170,530 row/s]
Latency mean : 0.6 ms [WRITE: 0.6 ms]
Latency median : 0.5 ms [WRITE: 0.5 ms]
Latency 95th percentile : 0.8 ms [WRITE: 0.8 ms]
Latency 99th percentile : 1.2 ms [WRITE: 1.2 ms]
Latency 99.9th percentile : 8.9 ms [WRITE: 8.9 ms]
Latency max : 99.2 ms [WRITE: 99.2 ms]
Total partitions : 10,000,000 [WRITE: 10,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0 B
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:00:58
{code}
CPU flamegraph: [^5.1_profile_cpu_without_metrics.html]
h2. Test 2: fixed rate and CPU usage comparison
{code:java}
./tools/bin/cassandra-stress "write n=10m" -rate threads=100 fixed=120000/s
-node myhost{code}
CPU is captured using:
{code:java}
vmstat -w 1
{code}
as 100% - idle
h3. With metrics
avg CPU usage: 71%
h3. Without metrics
avg CPU usage: 66%
Absolute delta: 5%, relative delta 5%/71% = 7% - it is not equal but close
enough to 8.65% reported in framegraph for metrics code.
So, I more or less convinced myself that it is not a skid in framegraph and %
of metrics overhead is more or less real.
Continuing with delta + Int2IntHashMap implementations..
> Provide the ability to disable specific metrics collection
> ----------------------------------------------------------
>
> Key: CASSANDRA-20250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20250
> Project: Apache Cassandra
> Issue Type: New Feature
> Components: Observability/Metrics
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Attachments: 5.1_profile_cpu.html,
> 5.1_profile_cpu_without_metrics.html, async_profiler_cpu_profiles.zip,
> cpu_profile_insert.html, jmh-result.json, vmstat.log,
> vmstat_without_metrics.log
>
>
> Cassandra has a lot of metrics collected, many of them are collected per
> table, so their instance number is multiplied by number of tables. From one
> side it gives a better observability, from another side metrics are not for
> free, there is an overhead associated with them:
> 1) CPU overhead: in case of simple CPU bound load: I already see like 5.5% of
> total CPU spent for metrics in cpu framegraphs for read load and 11% for
> write load.
> Example: [^cpu_profile_insert.html] (search by "codahale" pattern). The
> framegraph is captured using Async profiler build:
> async-profiler-3.0-29ee888-linux-x64
> 2) memory overhead: we spend memory for entities used to aggregate metrics
> such as LongAdders and reservoirs + for MBeans (String concatenation within
> object names is a major cause of it, for each table+metric name combination a
> new String is created)
>
> The idea of this ticket is to allow an operator to configure a list of
> disabled metrics in cassandra.yaml, like:
> {code:java}
> disabled_metrics:
> - metric_a
> - metric_b
> {code}
> From implementation point of view I see two possible approaches (which can be
> combined):
> # Generic: when a metric is registering if it is listed in disabled_metrics
> we do not publish it via JMX and provide a noop implementation of metric
> object (such as histogram) for it.
> Logging analogy: log level check within log method
> # Specialized: for some metrics the process of value calculation is not for
> free and introduces an overhead as well, in such cases it would be useful to
> check within specific logic using an API (like: isMetricEnabled) do we need
> to do it. Example of such metric:
> ClientRequestSizeMetrics.recordRowAndColumnCountMetrics
> Logging analogy: an explicit 'if (isDebugEnabled())' condition used when a
> message parameter is expensive.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
