I don’t know - I’m not sure what fuzz test means in this context. It’s a newer concept that I didn’t introduce. On 30 Nov 2023, at 20:06, Jacek Lewandowski <lewandowski.ja...@gmail.com> wrote:
How those burn tests then compare to the fuzz tests? (the new ones) By “could run indefinitely” I don’t mean by default they run forever. There will be parameters that change how much work is done for a given run, but just running repeatedly (each time with a different generated seeds) is the expected usage. Until you run out of compute or patience.
I agree they are only of value pre-commit to check they haven’t been broken in some way by changes.
that may be long-running and that could be run indefinitely
Perfect. That was the distinction I wasn't aware of. Also means having the burn target as part of regular CI runs is probably a mistake, yes? i.e. if someone adds a burn tests that runs indefinitely, are there any guardrails or built-in checks or timeouts to keep it from running right up to job timeout and then failing?
On Thu, Nov 30, 2023, at 1:11 PM, Benedict wrote:
A burn test is a randomised test targeting broad coverage of a single system, subsystem or utility, that may be long-running and that could be run indefinitely, each run providing incrementally more assurance of quality of the system.
A long test is a unit test that sometimes takes a long time to run, no more no less. I’m not sure any of these offer all that much value anymore, and perhaps we could look to deprecate them.
Strongly agree. I started working on a declarative refactor out of our CI configuration so circle, ASFCI, and other systems could inherit from it (for instance, see pre-commit pipeline declaration here); I had to set that down while I finished up implementing an internal CI system since the code in neither the ASF CI structure nor circle structure (.sh embedded in .yml /cry) was re-usable in their current form.
Having a jvm.options and cassandra.yaml file per suite and referencing them from a declarative job definition would make things a lot easier to wrap our heads around and maintain I think.
As for what qualifies as burn vs. long... /shrug couldn't tell you. Would have to go down the git blame + dev ML + JIRA rabbit hole. :) Maybe someone else on-list knows.
On Thu, Nov 30, 2023, at 4:25 AM, Jacek Lewandowski wrote:
Hi,
I'm getting a bit lost - what are the exact differences between those test scenarios? What are the criteria for qualifying a test to be part of a certain scenario?
I'm working a little bit with tests and build scripts and the number of different configurations for which we have a separate target in the build starts to be problematic, I cannot imagine how problematic it is for a new contributor.
It is not urgent, but we should at least have a plan on how to simplify and unify things.
I'm in favour of reducing the number of test targets to the minimum - for different configurations I think we should provide a parameter pointing to jvm options file and maybe to cassandra.yaml. I know that we currently do some super hacky things with cassandra yaml for different configs - like concatenting parts of it. I presume it is not necessary - we can have a default test config yaml and a directory with overriding yamls; while building we could have a tool which is able to load the default configuration, apply the override and save the resulting yaml somewhere in the build/test/configs for example. That would allows us to easily use those yamls in IDE as well - currently it is impossible.
What do you think?
Thank you and my apologize for bothering about lower priority stuff while we have a 5.0 release headache...
Jacek
|
