[
https://issues.apache.org/jira/browse/CASSANDRA-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15024539#comment-15024539
]
Jonathan Shook commented on CASSANDRA-10742:
--------------------------------------------
[~krummas],
Some notes on test setup, and some observations from data models we've seen. We
can try to get some additional details from willing users if this doesn't get
us close enough.
The baseline test I use is high-ingest, read-most-recent, with some read-cold
mixed-in. The idea is to simulate the typical access patterns of time-series
telemetry with roll-up processing, with the occasional historic query or
reprocessing of old data. I use 90/10/1 ratio for write/recent-read/cold-read
as a starting point. I usually back off the ingest rate from a saturating load
in order to find a stable steady-state reference point. This still is much
higher load per-node than you would often have in a production scenario. It
does provide for good contrast with trade-offs, like compaction load. Often,
you will be accumulating data over a longer period of time, so ingest rates
that approach the reasonable saturating load are closer to stress tests than
real-world. As such, they are still good tests. If you can run a node at 10x to
1000x the data rates that you would expect in production, then 1) you can
complete the test in a reasonable amount of time and 2) you're not too worried
about the margin of error.
The data model I use is essentially ((datasource, timebucket), parametername,
timestamp) -> value, although future testing will likely drop the timebucket
component, relying instead on the time-based layout of sstables as a
simplification. (Still needs supporting data from tests). parametername is just
a variable name that is associated with a type of measurement. This is selected
from a fixed set, as is often the case in the wild. The value can vary in type
and size according to the type of data logging. I use a range from 1k to 5k,
depending on the type of test. In the simplest cases, a value is an int or
float, but it can also be a log line from a stack trace.
The model of writes/read-most-recent/read-cold can cover lots of ground in
terms of time-series. The ratios can be varied. Also, the number of partitions
per node in conjunction with the number of parameters should vary. In some
cases in the wild, time-series partitions are single-series. In other cases,
they can have hundreds of related series by name (by cluster). In some cases,
the parameters associated with a data source are distributed by partition to
support async loading the cluster for responsive reads of significant data. To
cover this, simply move the parenthesis right by one term above.
If you cover some of the permutations above for op ratios, clustering
structure, grain of partition, and payload size, then you'll be covering lots
of the space we see in practice.
> Real world DateTieredCompaction tests
> -------------------------------------
>
> Key: CASSANDRA-10742
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10742
> Project: Cassandra
> Issue Type: Test
> Reporter: Marcus Eriksson
>
> So, to be able to actually evaluate DTCS (or TWCS) we need stress profiles
> that are similar to something that could be found in real production systems.
> We should then run these profiles for _weeks_, and do regular operational
> tasks on the cluster - like bootstrap, decom, repair etc.
> [~jjirsa] [~jshook] (or anyone): could you describe any write/read patterns
> you have seen people use with DTCS in production?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
