lhotari commented on pull request #10148: URL: https://github.com/apache/pulsar/pull/10148#issuecomment-814106963
> The flaky test does not mean it is a useless test, it can fail with A problem and B problem, A and B introduced the `flaky`. But the new change might cause C problem. If we always ignore the flaky test, the new break change might be merged into the master branch. I agree with you. There's just a special urgency now all PRs are blocked because of the very high flakiness of a few tests. > And if the test is very flaky, my point is we need to fix it. If it has not failed frequently before, but frequently now, this is most likely caused by some concurrent merge. A similar situation happened before, most of them are caused by the concurrent merge. Yes, the tests have to be fixed as top priority. However we need a way to unblock the CI. That is by moving the most flaky tests to a quarantine test group. The tests have been very flaky for a longer period of time. One fairly recent change in GitHub Actions has been the change in `ubuntu-latest` image. It now points to `ubuntu-20.04` instead of `ubuntu-18.04`. Another change that has increased the flakiness is simply the regrouping of tests. Since some tests are very fragile and flaky, a small change in the environment, such as regrouping the tests, can increase the flakiness. These 2 major changes seems to be the reason for the sudden increase in flakiness. However, the tests have been flaky even before the changes, so it's the tests that should be fixed instead of rolling back to the old environmental setup (test grouping and ubuntu-18.04 image). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
