Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Benedict]

A burn test is a randomised test targeting broad coverage of a single system, subsystem or utility, that may be long-running and that could be run indefinitely, each run providing incrementally more assurance of quality of the system. A long test is a unit test that sometimes takes a long time to run, no more no less. I’m not sure any of these offer all that much value anymore, and perhaps we could look to deprecate them. On 30 Nov 2023, at 17:20, Josh McKenzie <jmcken...@apache.org> wrote:  Strongly agree. I started working on a declarative refactor out of our CI configuration so circle, ASFCI, and other systems could inherit from it (for instance, see pre-commit pipeline declaration here); I had to set that down while I finished up implementing an internal CI system since the code in neither the ASF CI structure nor circle structure (.sh embedded in .yml /cry) was re-usable in their current form. Having a jvm.options and cassandra.yaml file per suite and referencing them from a declarative job definition would make things a lot easier to wrap our heads around and maintain I think. As for what qualifies as burn vs. long... /shrug couldn't tell you. Would have to go down the git blame + dev ML + JIRA rabbit hole. :) Maybe someone else on-list knows. On Thu, Nov 30, 2023, at 4:25 AM, Jacek Lewandowski wrote: Hi, I'm getting a bit lost - what are the exact differences between those test scenarios? What are the criteria for qualifying a test to be part of a certain scenario? I'm working a little bit with tests and build scripts and the number of different configurations for which we have a separate target in the build starts to be problematic, I cannot imagine how problematic it is for a new contributor. It is not urgent, but we should at least have a plan on how to simplify and unify things. I'm in favour of reducing the number of test targets to the minimum - for different configurations I think we should provide a parameter pointing to jvm options file and maybe to cassandra.yaml. I know that we currently do some super hacky things with cassandra yaml for different configs - like concatenting parts of it. I presume it is not necessary - we can have a default test config yaml and a directory with overriding yamls; while building we could have a tool which is able to load the default configuration, apply the override and save the resulting yaml somewhere in the build/test/configs for example. That would allows us to easily use those yamls in IDE as well - currently it is impossible. What do you think? Thank you and my apologize for bothering about lower priority stuff while we have a 5.0 release headache... Jacek