Failure flaky tests reduce the passion of contributors. I suggest making stable tests as required. I have 3 points. 1. Find the existing failure flaky tests and fix them. I believe that there is already a project to record them. 2. Move the failure flaky tests out of the flaky test group. 3. Add them into a new group and fix. And make it required if they are stable.
PengHui Li <peng...@apache.org> 于2022年8月22日周一 17:27写道： > I agree with moving the tests out of the flaky test group. > I just checking some new PRs > > https://github.com/apache/pulsar/pull/17195 > https://github.com/apache/pulsar/pull/17201 > https://github.com/apache/pulsar/pull/17193 > https://github.com/apache/pulsar/pull/17204 > > The `Pulsar CI / CI - Unit - Brokers - Flaky` looks more stable than other > groups such as > > `Pulsar CI / CI - System - Pulsar Connectors - Thread` > `Pulsar CI / CI - Unit - Brokers - Broker Group 2` > `Pulsar CI / CI - Unit - Brokers - Broker Group 1` > > It looks like we can change the test group for now and make it required. > To a new test group, or move them to `Pulsar CI / CI - Unit - Brokers - * ` > > Thanks, > Penghui > > On Mon, Aug 22, 2022 at 4:33 PM tison <wander4...@gmail.com> wrote: > > > While agree to require more tests to pass, here are my two coins: > > > > 1. This group is named "flaky tests" so I regard it as flaky tests > > literally. NOT require these tests to pass could be by design. Besides, > > IIRC some developers keep investigating tests in the flaky test group, > try > > to make them stable, and move out of the flaky test group. This seems the > > desired approach to resolve flaky tests. > > > > 2. Instead of barely "require broker-flaky" test group, do you have a > list > > of tests that fail frequently? Otherwise, we just go back to the > situation > > where we want to exclude it from the required status - it's quite > unstable. > > > > Best, > > tison. > > > > > > mattison chao <mattisonc...@apache.org> 于2022年8月22日周一 15:56写道： > > > > > Hi all > > > > > > Recently, some tests in the `broker-flaky` test group always failed, > but > > > since it doesn't block CI, no one cared for a long time. > > > > > > This behaviour causes some test scenarios to go unchecked and risk > > > introducing some regressions, and I think we need to make this test > group > > > set required. > > > > > > e.g. https://github.com/apache/pulsar/pull/17163 > > > > > > WDYT? > > > > > > Best, > > > Mattison > > > > > > -- BR, Qiang Huang
