Thank you Jarek for your work! +1 for the idea of quarantine tests. Just one question: are we marking single tests or whole classes? This question is mostly related to tests that requires some side effects from previous tests.
Tomek On Mon, Apr 20, 2020 at 2:38 AM Jarek Potiuk <[email protected]> wrote: > > Hello everyone, > > I have a proposal - very much COVID-19-inspired on how to fix our CI tests... > > After the recent problems with CI together with Daniel and Tomek we > decided to make an emergency migration to Github Actions. So we did. > > I think overall it was a good move, but we had some problems with it. > It turns out that while we were blaming Travis for everything wrong > that happened in our builds, it was not always Travis' fault. We have > some tests that are also failing in Github Actions and I think it's > the highest time we fix them. > > I spend a better part of the weekend bring trying different things and > implementing numerous optimizations back to our CI configuration (a > lot of those were lost during the emergency move). > > While running it I had many issues and I think I found a good way to > handle our flaky tests. I would love that others think about it. > > Those interested - please take a look at the PR "Bring back CI > optimisations" https://github.com/apache/airflow/pull/8393 > Corresponding GituhbActions here: > https://github.com/apache/airflow/actions/runs/82410109 > > I implemented a lot of optimizations in this PR (some of them will > only take effect after we merge to master) but most of all I wanted to > introduce a concept of "quarantined tests" (good name isn't it :) ) > > Here is the idea: > > - tests that are marked as @pytest.mark.quarantined are skipped in > regular runs (I identified 58 potential candidates - not all of them > are flaky but I wanted to be safe) > - there is one dedicated "Quarantine" job that runs only quarantined > tests (it's Postgres 9.6 with Python 3.6 for now) > - those "quarantined" tests are run with 90 s. timeout each and rerun > up to 3 times if they fail > - failure of any of the Quarantine tests does not fail the whole CI > - I plan to create GithUb issues for groups of those tests > (MoveOutOfQuarantine NNNN) > - I think it's best if we split them between committers > - The job of the committers will be to observe the stability of those tests > - once we fix and observe that the tests are "stable" we move them > out of Quarantine back to regular tests (by removing > @pytest.mark.quarantined) > - the goal is to move all our tests out of Quarantine > - in the future we can move any flaky test to Quarantine (by adding > @pytest.mark.quarantined) and it will give us time to observe it and > fix any flakiness. > > Let me know what you think of it? > > J. > > -- > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 -- Tomasz Urbaszek Polidea | Software Engineer M: +48 505 628 493 E: [email protected] Unique Tech Check out our projects!
