Hi Joan, I’ve seen the Elixir suite implicated more frequently as well. I haven’t done the analysis to see if the failures are concentrated in one or two flakes or if they’re more evenly distributed. If it’s a small number of flaky tests I think we have time to fix/disable them rather than knocking out the whole suite.
I agree that we need `make check` to be trustworthy in 3.0 release candidates. I would like to keep running the elixir tests in the CI regardless of whether they’re in the `check` suite. Cheers, Adam > On Dec 12, 2019, at 4:39 PM, Joan Touzet <woh...@apache.org> wrote: > > Hi again, > > As I've been looking more closely at the CI suite for the Jenkins transition, > I've noticed that our Elixir test cases are actually the most likely to fail. > In 6 consecutive CI runs, 5 runs failed due to failures in the Elixir suite. > (The 6th failed due to a JS test failure.) > > We started the Elixir effort to retire the JS suite. We reached a decision > some months ago to put it into `make check` so that people would pay > attention to its output, and work to fix those tests, accelerating our > chances to get rid of the JS suite. > > Unfortunately, that's not materialised. Our Elixir test porters seem to have > stopped their work for a while now, and no one is systematically addressing > the failures in that suite. I've also heard other developers mention (via > IRC) that some of the test cases hold invalid assumptions about how CouchDB > works, especially with the Erlang-based clustering code. It sounds to me like > the effort needs a full code review. > > With 3.0 around the corner, I want people to be able to trust the output of > `make check` when downloading the tarball. If there is no objection, when I > merge the Erlang version / CI changes on Monday, I will also comment out the > call to `make elixir` as part of make check. > > When the Elixir porting team is more confident in the reliability and > completeness of their work, and we can successfully retire the JS suite, we > can reconsider. > > -Joan "really wanting to see green, but only seeing red" Touzet
