Thanks Andres! Let's sit tight and wait for the release vote for beta-1 to settle before we ratify these changes.
On Wed, Oct 5, 2022, at 6:20 AM, Andrés de la Peña wrote: > The proposal looks good to me. I have created tickets for: > - Increasing the default number of repeated test iterations to 500 > (CASSANDRA-17937 <https://issues.apache.org/jira/browse/CASSANDRA-17937>, > ready to commit) > - Automatically detecting and repeating new or modified JUnit tests > (CASSANDRA-17939 <https://issues.apache.org/jira/browse/CASSANDRA-17939>, > patch available) > - Allowing to specify multiple tests in the test multiplexer > (CASSANDRA-17938 <https://issues.apache.org/jira/browse/CASSANDRA-17938>, in > progress) > > On Mon, 3 Oct 2022 at 15:23, Josh McKenzie <jmcken...@apache.org> wrote: >> __ >> Any further revisions or objections to this or are we good to take it to a >> vote? >> >> On Wed, Sep 28, 2022, at 10:54 AM, Josh McKenzie wrote: >>> So revised proposal: >>> >>> On Release Lifecycle cwiki page: >>> - Ensure we have parity on jobs run between circle and asf-ci >>> - Allow usage of circleci as gatekeeper for releases. 1 green run -> beta, >>> 3 green runs consecutive -> ga >>> - No new consistent regressions on CI for asf compared to prior branches >>> - Explicitly do not consider ci-cassandra asf flaky tests as release >>> blockers >>> >>> Changes to codify into documentation: >>> - On patch before commit, multiplex @500 all new tests, changed tests, or >>> expected to be impacted tests ("expected to be impacted" piece pending >>> multi-class multiplexing support): >>> - Add support for multi-class specification in multiplexer and document >>> >>> Add informal project commitment during next major release lifecycle to >>> continue working on bringing asf ci-cassandra up to where it can be formal >>> gatekeeper for release. >>> >>> On Wed, Sep 28, 2022, at 10:13 AM, Ekaterina Dimitrova wrote: >>>> If we talk blockers nothing more than ensuring we see all tests we want >>>> pre-release, IMHO. >>>> The other points sound to me like future important improvements that will >>>> help us significantly in the flaky test fight. >>>> >>>> On Wed, 28 Sep 2022 at 10:08, Josh McKenzie <jmcken...@apache.org> wrote: >>>>> __ >>>>> I'm receptive to that but I wouldn't gate our ability to get 4.1 out the >>>>> door based on circle on that. Honestly probably only need to have the >>>>> parity of coverage be the blocker for its use in retrospect. >>>>> >>>>> On Wed, Sep 28, 2022, at 1:32 AM, Berenguer Blasi wrote: >>>>>> I would add an option for generate.sh to detect all changed *Test.java >>>>>> files, that would be handy imo. >>>>>> >>>>>> On 28/9/22 4:29, Josh McKenzie wrote: >>>>>>> So: >>>>>>> 1. 500 iterations on multiplexer >>>>>>> 2. Augmenting generate.sh to allow providing multiple class names and >>>>>>> generating a single config that'll multiplex all the tests provided >>>>>>> 3. Test parity / pre-release config added on circleci (see >>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-17930), specifically >>>>>>> dtest-large, dtest-offheap, test-large-novnode >>>>>>> If we get the above 3, are we at a place where we're good to consider >>>>>>> vetting releases on circleci for beta / rc / ga? >>>>>>> >>>>>>> On Tue, Sep 27, 2022, at 11:28 AM, Ekaterina Dimitrova wrote: >>>>>>>>> “I have plans on modifying the multiplexer to allow specifying a list >>>>>>>>> of classes per test target, so we don't have to needlessly suffer >>>>>>>>> with this” >>>>>>>>> >>>>>>>>> >>>>>>>>> That would be great, I was thinking of that the other day too. With >>>>>>>>> that said I’ll be happy to support you in that effort too :-) >>>>>>>> >>>>>>>> On Tue, 27 Sep 2022 at 11:18, Josh McKenzie <jmcken...@apache.org> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I have plans on modifying the multiplexer to allow specifying a list >>>>>>>>>> of classes per test target, so we don't have to needlessly suffer >>>>>>>>>> with this >>>>>>>>> This sounds integral to us multiplexing tests on large diffs whether >>>>>>>>> we go with circle for releases or not and would be a great addition! >>>>>>>>> >>>>>>>>> On Tue, Sep 27, 2022, at 6:19 AM, Andrés de la Peña wrote: >>>>>>>>>>> 250 iterations isn't enough; I use 500 as a low water mark. >>>>>>>>>> >>>>>>>>>> I agree that 500 iterations would be a reasonable minimum. We have >>>>>>>>>> seen flaky unit tests requiring far more iterations, but that's not >>>>>>>>>> very common. We could use to 500 iterations as default, and >>>>>>>>>> discretionary use a higher limit in tests that are quick and might >>>>>>>>>> be prone to concurrency issues. I can change the defaults on CirceCI >>>>>>>>>> config file if we agree to a new limit, the current default of 100 >>>>>>>>>> iterations is quite arbitrary. >>>>>>>>>> >>>>>>>>>> The test multiplexer allows to either run test individual test >>>>>>>>>> methods or entire classes. It is quite frequent to see tests methods >>>>>>>>>> that pass individually but fail when they are run together with the >>>>>>>>>> other tests in the same class. Because of this, I think that we >>>>>>>>>> should always run entire classes when repeating new or modified >>>>>>>>>> tests. The only exception to this would be Python dtests, which >>>>>>>>>> usually are more resource intensive and not so prone to that type of >>>>>>>>>> issues. >>>>>>>>>> >>>>>>>>>>> For CI on a patch, run the pre-commit suite and also run >>>>>>>>>>> multiplexer with 250 runs on new, changed, or related tests to >>>>>>>>>>> ensure not flaky >>>>>>>>>> >>>>>>>>>> The multiplexer only allows to run a single test class per push. >>>>>>>>>> This is ok for fixing existing flakies (its original purpose), and >>>>>>>>>> for most minor changes, but it can be quite inconvenient for testing >>>>>>>>>> large patches that add or modify many tests. For example, the patch >>>>>>>>>> for CEP-19 directly modifies 31 test classes, which means 31 >>>>>>>>>> CircleCI config pushes. This number can be somewhat reduced with >>>>>>>>>> some wildcards on the class names, but the process is still quite >>>>>>>>>> inconvenient. I guess that other large patches will find the same >>>>>>>>>> problem. I have plans on modifying the multiplexer to allow >>>>>>>>>> specifying a list of classes per test target, so we don't have to >>>>>>>>>> needlessly suffer with this. >>>>>>>>>> >>>>>>>>>> On Mon, 26 Sept 2022 at 22:44, Brandon Williams <dri...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>> On Mon, Sep 26, 2022 at 1:31 PM Josh McKenzie >>>>>>>>>>> <jmcken...@apache.org> wrote: >>>>>>>>>>> > >>>>>>>>>>> > 250 iterations isn't enough; I use 500 as a low water mark. >>>>>>>>>>> > >>>>>>>>>>> > Say more here. I originally had it at 500 but neither Mick nor I >>>>>>>>>>> > knew why and figured we could suss this out on this thread. >>>>>>>>>>> >>>>>>>>>>> I've seen flakies that passed with less later exhibit at that point. >>>>>>>>>>> >>>>>>>>>>> > This is also assuming that circle and ASF CI run the same tests, >>>>>>>>>>> > which >>>>>>>>>>> > is not entirely true. >>>>>>>>>>> > >>>>>>>>>>> > +1: we need to fix this. My intuition is the path to getting >>>>>>>>>>> > circle-ci in parity on coverage is a shorter path than getting >>>>>>>>>>> > ASF CI to 3 green runs for GA. That consistent w/your perception >>>>>>>>>>> > as well or do you disagree? >>>>>>>>>>> >>>>>>>>>>> I agree that bringing parity to the coverage will be the shorter >>>>>>>>>>> path. >>>>>>>>> >>>>>>> >>>>> >>> >>
