[
https://issues.apache.org/jira/browse/CASSANDRA-9161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177651#comment-15177651
]
Sylvain Lebresne commented on CASSANDRA-9161:
---------------------------------------------
bq. the problems we faced with upgrade tests due to the randomness of how the
data is distributed
Yeah, but that's a bad example: that randomness was not properly controlled, it
was making things hard to reproduce and that was the problem. But this is not
what I want to do here, this will _not_ make tests non reproducible at all.
In testing, sometimes (always really, but that's a different subject) the state
space is just too big to be systematically explored on every run. Because of
that, you do your best at exploring the most meaningful part of the space (and
I'm not saying we can't improve on that part btw, we can and we should), but
there is still space you can't explore. Hoping you'll be so good at finding
the meaningful subset of the states to test so that no bug will lurk in the
remaining space is just wishful thinking. So this is just about getting
incrementally better coverage of the full space by using some new random state
on every run _for the parts we can't reasonably explore systematically_ (and
I'm happy to discuss which parts can reasonably be explored systematically and
which aren't btw).
bq. For flushing, the main problem that I have seen is that only the read or
write path for memtables was tested not the one for SSTables
It's really more complex. Unless your test has a single insert, there isn't
_just_ one path for memtables and one for sstables. There is the cases where
some data is in memtable and some in sstables, where there is more than one
sstable involved (and we can flush for every insert, or only in some places),
whether we compact before reading etc... I have seen bugs in pretty much all of
those cases (I'm genuinely not kidding: there has been cases with range
tombstone in particular where things got only messes up when data was flushed
at specific point and compaction was run before reading).
And here again, don't get me wrong: for some tests, there may be a clear place
where we want to systematically test both with and without flush because we
want that to be tested every time and that's fine, we can do it. But we just
can't systematically test all combinations.
> Add random interleaving for flush/compaction when running CQL unit tests
> ------------------------------------------------------------------------
>
> Key: CASSANDRA-9161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9161
> Project: Cassandra
> Issue Type: Test
> Reporter: Sylvain Lebresne
> Labels: retrospective_generated
>
> Most CQL tests don't bother flushing, which means that they overwhelmingly
> test the memtable path and not the sstables one. A simple way to improve on
> that would be to make {{CQLTester}} issue flushes and compactions randomly
> between statements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
