On Mon, Nov 30, 2020 at 10:31:09AM +0000, Daniel P. Berrangé wrote: > On Fri, Nov 27, 2020 at 07:29:31PM +0100, Thomas Huth wrote: > > On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote: > > > On 11/27/20 6:47 PM, Thomas Huth wrote: > > >> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote: > > >>> We lately realized that the Avocado framework was not designed > > >>> to be regularly run on CI environments. Therefore, as of 5.2 > > >>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt > > >>> current users, it is possible to keep the current behavior by > > >>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable > > >>> (see [*]). > > >>> From now on, using these jobs (or adding new tests to them) > > >>> is strongly discouraged. > > >>> > > >>> Tests based on Avocado will be ported to new job schemes during > > >>> the next releases, with better documentation and templates. > > >> > > >> Why should we disable the jobs by default as long as there is no > > >> replacement > > >> available yet? > > > > > > Why keep it enabled if it is failing randomly > > > > We can still disable single jobs if they are not stable, but that's no > > reason to disable all of them by default, is it? > > Agreed, the jobs which are known to be broken or unreliable should > be unconditonally disabled in QEMU as a whole. This isn't specific > to gitlab config - the qemu build makefiles/mesonfiles should disable > the problem tests entirely, as we don't want developers wasting time > running them locally either if they're known to be broken/unreliable. >
The problem is identifying when a test is broken/unreliable. No complex test is 100% reliable: change the environment, (configuration, build options, platform, etc) and any test complex enough will start to fail. I've worked in projects orders of magnitude simpler than QEMU and that was a golden rule. Testing QEMU is *HARD*. Which is why I defend test-automation separated from CI: * Have a stable CI with tests cherry-picked by whoever is maintaining a particular CI runner (we shouldn't have orphan runners). * Have as many tests as possible in the git repo: maintained, reviewed and run (outside of a CI) by people who care about them. I absolutely agree with you that maintainers and developers should care and our goal should be a gating CI. The question is how to create a strategy and a plan to get there. Forcing people to care rarely works. Thanks. - Ademar -- Ademar Reis Jr Red Hat ^[:wq!