Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

> Thanks Andres! Let's sit tight and wait for the release vote for beta-1 to > settle before we ratify these changes. > Reading this, I believe we have a lazy consensus in place and the 4.1-beta1 thread has settled, so we can finalise the release, while this thread will go to ratification separat

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

Thanks Andres! Let's sit tight and wait for the release vote for beta-1 to settle before we ratify these changes. On Wed, Oct 5, 2022, at 6:20 AM, Andrés de la Peña wrote: > The proposal looks good to me. I have created tickets for: > - Increasing the default number of repeated test iterations t

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

The proposal looks good to me. I have created tickets for: - Increasing the default number of repeated test iterations to 500 ( CASSANDRA-17937 , ready to commit) - Automatically detecting and repeating new or modified JUnit tests ( CASSANDRA

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

Any further revisions or objections to this or are we good to take it to a vote? On Wed, Sep 28, 2022, at 10:54 AM, Josh McKenzie wrote: > So revised proposal: > > On Release Lifecycle cwiki page: > - Ensure we have parity on jobs run between circle and asf-ci > - Allow usage of circleci as gat

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

So revised proposal: On Release Lifecycle cwiki page: - Ensure we have parity on jobs run between circle and asf-ci - Allow usage of circleci as gatekeeper for releases. 1 green run -> beta, 3 green runs consecutive -> ga - No new consistent regressions on CI for asf compared to prior branches

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

If we talk blockers nothing more than ensuring we see all tests we want pre-release, IMHO. The other points sound to me like future important improvements that will help us significantly in the flaky test fight. On Wed, 28 Sep 2022 at 10:08, Josh McKenzie wrote: > I'm receptive to that but I wou

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

I'm receptive to that but I wouldn't gate our ability to get 4.1 out the door based on circle on that. Honestly probably only need to have the parity of coverage be the blocker for its use in retrospect. On Wed, Sep 28, 2022, at 1:32 AM, Berenguer Blasi wrote: > I would add an option for generat

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

I would add an option for generate.sh to detect all changed *Test.java files, that would be handy imo. On 28/9/22 4:29, Josh McKenzie wrote: So: 1. 500 iterations on multiplexer 2. Augmenting generate.sh to allow providing multiple class names and generating a single config that'll multi

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

So: 1. 500 iterations on multiplexer 2. Augmenting generate.sh to allow providing multiple class names and generating a single config that'll multiplex all the tests provided 3. Test parity / pre-release config added on circleci (see https://issues.apache.org/jira/browse/CASSANDRA-17930), spec

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

“I have plans on modifying the multiplexer to allow specifying a list of classes per test target, so we don't have to needlessly suffer with this” That would be great, I was thinking of that the other day too. With that said I’ll be happy to support you in that effort too :-) On Tue, 27 Sep 202

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

> I have plans on modifying the multiplexer to allow specifying a list of > classes per test target, so we don't have to needlessly suffer with this This sounds integral to us multiplexing tests on large diffs whether we go with circle for releases or not and would be a great addition! On Tue, S

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

> > 250 iterations isn't enough; I use 500 as a low water mark. I agree that 500 iterations would be a reasonable minimum. We have seen flaky unit tests requiring far more iterations, but that's not very common. We could use to 500 iterations as default, and discretionary use a higher limit in te

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

On Mon, Sep 26, 2022 at 1:31 PM Josh McKenzie wrote: > > 250 iterations isn't enough; I use 500 as a low water mark. > > Say more here. I originally had it at 500 but neither Mick nor I knew why and > figured we could suss this out on this thread. I've seen flakies that passed with less later ex

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

Happy to help! Certainly getting Circle CI and ASF CI at parity seems like a good goal. I'm going to probably have some extra cycles next week while I'm at ApacheCon, so I'm hoping to make some meaningful progress :) Cheers, Derek On Mon, Sep 26, 2022 at 12:44 PM Ekaterina Dimitrova wrote: > T

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

Thanks Josh for bringing this up. I agree with Brandon on both points he made. I have also other examples of testing some things only in Jenkins or only in CircleCI. If people are fine with that, I can open a ticket and I just talked n Slack to Derek who expressed interest to help with that activi

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

> 250 iterations isn't enough; I use 500 as a low water mark. Say more here. I originally had it at 500 but neither Mick nor I knew why and figured we could suss this out on this thread. > This is also assuming that circle and ASF CI run the same tests, which > is not entirely true. +1: we need

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

On Mon, Sep 26, 2022 at 12:20 PM Josh McKenzie wrote: > For CI on a patch, run the pre-commit suite and also run multiplexer with 250 > runs on new, changed, or related tests to ensure not flaky 250 iterations isn't enough; I use 500 as a low water mark. > I think it's worth calling out: relyin

[DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

Mick and I have been chatting back and forth a bit about 4.1 beta, ASF CI, circleci, and the current challenges we face as a project with our CI environment(s). Given the consistent difficulty we continue to have with the ASF CI env that's been blocking our beta + ga for months now, I think we s