[DISCUSS] Formalizing requirements for pre-commit patches on new CI

Context: we're looking to get away from having split CircleCI and ASF CI as well as getting ASF CI to a stable state. There's a variety of reasons why it's flaky (orchestration, heterogenous hardware, hardware failures, flaky tests, non-deterministic runs, noisy neighbors, etc), many of which Mick

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Thanks Josh, this looks great! I think the constraints you've outlined are reasonable for an initial attempt. We can always evolve if we run into issues. Cheers, Derek On Fri, Jun 30, 2023 at 11:19 AM Josh McKenzie wrote: > Context: we're looking to get away from having split CircleCI and ASF

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Thank you, Josh and Mick Immediate questions on my mind: - Currently we run at most two parallel CI runs in Jenkins-dev, I guess you will try to improve that limitation? - There are hw constraints, is there any approximation on how long it will take to run all tests? Or is there a stated goal that

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

All great questions I don't have answers to Ekaterina. :) Thoughts though: > - Currently we run at most two parallel CI runs in Jenkins-dev, I guess you > will try to improve that limitation? If we get to using cloud-based resources for CI instead of our donated hardware w/a budget, we could the

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> > - There are hw constraints, is there any approximation on how long it will > take to run all tests? Or is there a stated goal that we will strive to > reach as a project? > > Have to defer to Mick on this; I don't think the changes outlined here > will materially change the runtime on our curre

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> Not everyone will have access to such resources, if all you have is 1 such > pod you'll be waiting a long time (in theory one month, and you actually need > a few bigger pods for some of the more extensive tests, e.g. large upgrade > tests)…. One thing worth calling out: I believe we have *

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

For me, the biggest benefit of keeping the build scripts and CI configurations as well in the same project is that these files are versioned in the same way as the main sources do. This ensures that we can build past releases without having any annoying errors in the scripts, so I would say that th

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> Instead of running all the tests through available CI agents every time we > can have presets of tests: Back when I joined the project in 2014, unit tests took ~ 5 minutes to run on a local machine. We had pre-commit and post-commit tests as a distinction as well, but also had flakes in the pr

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Ultimately I think we have to invest in two directions: first, choose a consistent, representative subset of stable tests that we feel give us a reasonable level of confidence in return for a reasonable amount of runtime. Second, we need to invest in figuring out why certain tests fail. I strongly

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Currently a dtest is being ran in j8 w/wo vnodes , j8/j11 w/wo vnodes and j11 w/wo vnodes. That is 6 times total. I wonder about that ROI. On dtest cluster reusage yes, I stopped that as at the time we had lots of CI changes, an upcoming release and priorities. But when the CI starts flexing i

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Perhaps pre-commit checks should include mostly the typical configuration of Cassandra rather than some subset of possible combinations. Like it was said somewhere above - test with the default number of vnodes, test with the default compression settings, and test with the default heap/off-heap buf

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> choose a consistent, representative subset of stable tests that we feel give > us a reasonable level of confidence in return for a reasonable amount of > runtime > > ... > Currently a dtest is being ran in j8 w/wo vnodes , j8/j11 w/wo vnodes and j11 > w/wo vnodes. That is 6 times total. I won

Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

- From: Josh McKenzie Date: Wed, 5 Jul 2023 at 8:25 Subject: Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI To: dev choose a consistent, representative subset of stable tests that we feel give us a reasonable level of confidence in return for a reasonab

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

deterministic, I believe we will learn a thing or two, and those > types of things will happen less in time. > > Best regards, > Ekaterina > > -- Forwarded message - > From: *Josh McKenzie* > Date: Wed, 5 Jul 2023 at 8:25 > Subject: Re: [DISCUSS] For

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

On Fri, Jul 7, 2023 at 10:09 AM Josh McKenzie wrote: > 3. Multiplexed tests (changed, added) run against all JDK's and a broader > range of configs (no-vnode, vnode default, compression, etc) I think this is going to be too heavy...we're taking 500 iterations and multiplying that by like 4 or 5?

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Maybe. Kind of depends on how long we write our tests to run doesn't it? :) But point taken. Any non-trivial test would start to be something of a beast under this approach. On Fri, Jul 7, 2023, at 11:12 AM, Brandon Williams wrote: > On Fri, Jul 7, 2023 at 10:09 AM Josh McKenzie wrote: > > 3. M

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

I think 500 runs combining all configs could be reasonable, since it's unlikely to have config-specific flaky tests. As in five configs with 100 repetitions each. On Fri, 7 Jul 2023 at 16:14, Josh McKenzie wrote: > Maybe. Kind of depends on how long we write our tests to run doesn't it? :) > > B

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

+1 to Josh which is exactly my line of thought as well. But that is only valid if we have a solid Jenkins that will eventually run all test configs. So I think I lost track a bit here. Are you proposing: 1- CircleCI: Run pre-commit a single (the most common/meaningful, TBD) config of tests 2

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

I'm personally not thinking about CircleCI at all; I'm envisioning a world where all of us have 1 CI *software* system (i.e. reproducible on any env) that we use for pre-commit validation, and then post-commit happens on reference ASF hardware. So: 1: Pre-commit subset of tests (suites + matric

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Add a 'devBranch' jenkins job to that imo: The possibility to run the full suite + multiplex new tests before commit when you're about to release a Kraken into the codebase: Accord, TCM, TTL, SAI, Vector, JDK... So: 1: Pre-commit subset of tests (suites + matrices + env) runs. On green, merge

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

A strong +1 to getting to a single CI system. CircleCI definitely has some niceties and I understand why it's currently used, but right now we get 2 CI systems for twice the price. +1 on the proposed subsets. Derek On Mon, Jul 10, 2023 at 9:37 AM Josh McKenzie wrote: > I'm personally not thinki

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> 2: Pre-commit 'devBranch' full suite for high risk/disruptive merges: at > reviewer's discretion In general, maybe offering a dev the option of choosing either "pre-commit smoke" or "post-commit full" at their discretion for any work would be the right play. A follow-on thought: even with som

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

On our 4.0 release I remember a number of such failures but not recently. What is more common though is packaging errors, cdc/compression/system_ks_directory targeted fixes, CI w/wo upgrade tests, being less responsive post-commit as you already moved on,... Either the smoke pre-commit has appr

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Isn't novnodes a special case of vnodes with n=1 ? We should rather select a subset of tests for which it makes sense to run with different configurations. The set of configurations against which we run the tests currently is still only the subset of all possible cases. I could ask - why don't ru

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

“ On our 4.0 release I remember a number of such failures but not recently. ” Based on all the 5.0 work I’d say we need as a minimum to build and start a node with all JDK versions pre-commit. On Wed, 12 Jul 2023 at 7:29, Jacek Lewandowski wrote: > Isn't novnodes a special case of vnodes with

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

(This response ended up being a bit longer than intended; sorry about that) > What is more common though is packaging errors, > cdc/compression/system_ks_directory targeted fixes, CI w/wo > upgrade tests, being less responsive post-commit as you already > moved on *Two that ***should ***be resolve

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

jenkins_jira_integration script updating the JIRA ticket with test results if you cause a regression + us building a muscle around reverting your commit if they break tests.“ I am not sure

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Would it be re-opening the ticket or creating a new ticket with "revert of fix" ? śr., 12 lip 2023 o 14:51 Ekaterina Dimitrova napisał(a): > jenkins_jira_integration > > script > updat

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

I believe some tools can determine which tests make sense to multiplex, given that some exact lines of code were changed using code coverage analysis. After the initial run, we should have data from the coverage analysis, which would tell us which test classes are tainted - that is, they cover the

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> Would it be re-opening the ticket or creating a new ticket with "revert of > fix" ? I have a weak preference for re-opening the original ticket and tracking the revert + fix there. Keeps the workflow in one place. "Downside" is having multiple commits with "CASSANDRA-XX" in the message but

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Revert for only trunk patches right? I’d say we need to completely stabilize the environment, no noise before we go into that direction. On Wed, 12 Jul 2023 at 8:55, Jacek Lewandowski wrote: > Would it be re-opening the ticket or creating a new ticket with "revert of > fix" ? > > > > śr., 12 lip

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> Revert for only trunk patches right? > I’d say we need to completely stabilize the environment, no noise before we > go into that direction. Hm. Is the concern multi-branch reverts w/merge commits being awful? Because I hear that. Starting trunk-only would be reasonable enough I think, especia

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> A strong +1 to getting to a single CI system. CircleCI definitely has some > niceties and I understand why it's currently used, but right now we get 2 > CI systems for twice the price. +1 on the proposed subsets. > That's not entirely true, it provides value in "double accounting" and that has

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

On Wed, 12 Jul 2023 at 15:05, Jacek Lewandowski wrote: > I believe some tools can determine which tests make sense to multiplex, > given that some exact lines of code were changed using code coverage > analysis. After the initial run, we should have data from the coverage > analysis, which would