Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Benedict]

Why do we want to move to JUnit 5?  I’m generally opposed to churn unless well justified, which it may be - just not immediately obvious to me. On 11 Dec 2023, at 08:33, Jacek Lewandowski <lewandowski.ja...@gmail.com> wrote:  Nobody referred so far to the idea of moving to JUnit 5, what are the opinions? niedz., 10 gru 2023 o 11:03 Benedict <bened...@apache.org> napisał(a): Alex’s suggestion was that we meta randomise, ie we randomise the config parameters to gain better rather than lesser coverage overall. This means we cover these specific configs and more - just not necessarily on any single commit. I strongly endorse this approach over the status quo. On 8 Dec 2023, at 13:26, Mick Semb Wever <m...@apache.org> wrote:        I think everyone agrees here, but…. these variations are still catching failures, and until we have an improvement or replacement we do rely on them.   I'm not in favour of removing them until we have proof /confidence that any replacement is catching the same failures.  Especially oa, tries, vnodes. (Not tries and offheap is being replaced with "latest", which will be valuable simplification.)   What kind of proof do you expect? I cannot imagine how we could prove that because the ability of detecting failures results from the randomness of those tests. That's why when such a test fail you usually cannot reproduce that easily. Unit tests that fail consistently but only on one configuration, should not be removed/replaced until the replacement also catches the failure.   We could extrapolate that to - why we only have those configurations? why don't test trie / oa + compression, or CDC, or system memtable?  Because, along the way, people have decided a certain configuration deserves additional testing and it has been done this way in lieu of any other more efficient approach.