TL;DR: Improving a flaky test no longer requires it being healed from all its flakiness. Add commit footer: Flaky-test-function: tst_Class::function
Dear Qt Community, <https://codereview.qt-project.org/c/qt/qtbase/+/701839> and follow-ups introduced detection of newly added or modified test functions. Such functions must pass 10 consecutive times in CI before they are considered trustworthy, instead of relying on the usual retry logic. This is generally a good idea. New tests should demonstrate that they are stable, before intoxicating CI with new flakiness. However, reality has shown an unfortunate edge case: someone improves a flaky test function, but does not manage to eliminate all flakiness. Perhaps because the remaining failures are caused by timing, virtualization, infrastructure, scheduler noise, cosmic radiation, phase of the moon, or other stakeholders. In that situation, a test can become significantly better - and impossible to merge. To address this, <https://codereview.qt-project.org/c/qt/qtbase/+/785771> adds an escape hatch - a commit message footer that exempts specific modified test functions from the "new test" treatment: Flaky-test-function: tst_Class::function One footer per function. This is not intended as a "my test is special" marker, nor as a substitute for fixing flaky tests. It merely acknowledges that moving from bad to better is still progress, when perfect remains out of reach. Or, phrased differently, we must encourage reducing flakiness, not require contributors to eradicate it before they are allowed to submit the patch. Cheers, Axel Confidential
-- Development mailing list [email protected] https://lists.qt-project.org/listinfo/development
