Hi Phil,
On 11/27/20 2:41 PM, Philippe Mathieu-Daudé wrote:
We lately realized that the Avocado framework was not designed
to be regularly run on CI environments. Therefore, as of 5.2
we deprecate the gitlab-ci jobs using Avocado. To not disrupt
current users, it is possible to keep the current behavior by
setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
(see [*]).
Could you please detail where have you seen the avocado based tests
failing that much to justify their removal of CI?
See below some data that I generated from the pipelines for
https://gitlab.com/qemu-project/qemu/. It seems that jobs which failed
due timeout are marked as 'skip' and that explain why some pipelines
don't have a list of failed jobs.
---
Failed 21 out of 99
Pipeline -- 221617704 (staging)
check-patch
Pipeline -- 219276598 (master)
pages
Pipeline -- 218121424 (master)
Pipeline -- 217995873 (staging)
acceptance-system-fedora
Pipeline -- 217995872 (master)
build-tci
Pipeline -- 217551771 (v5.2.0-rc2)
Pipeline -- 217503505 (master)
Pipeline -- 217362429 (master)
Pipeline -- 217328707 (master)
check-crypto-old-nettle
Pipeline -- 216770735 (staging)
acceptance-system-fedora
check-patch
Pipeline -- 215772908 (master)
Pipeline -- 215715025 (staging)
check-crypto-old-gcrypt
Pipeline -- 215715024 (master)
check-crypto-old-nettle
Pipeline -- 214944520 (master)
acceptance-system-debian
Pipeline -- 214460230 (v5.2.0-rc1)
Pipeline -- 214438601 (master)
check-crypto-only-gnutls
Pipeline -- 214273938 (master)
Pipeline -- 214183970 (master)
Pipeline -- 214140305 (master)
Pipeline -- 213892224 (master)
Pipeline -- 213871132 (master)
From now on, using these jobs (or adding new tests to them)
is strongly discouraged.
Tests based on Avocado will be ported to new job schemes during
the next releases, with better documentation and templates.
[*]
https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui
Signed-off-by: Philippe Mathieu-Daudé <phi...@redhat.com>
---
.gitlab-ci.yml | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d0173e82b16..2674407cd13 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -66,6 +66,15 @@ include:
- cd build
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in
r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
- du -chs ${CI_PROJECT_DIR}/avocado-cache
+ rules:
+ # As of QEMU 5.2, Avocado is not yet ready to run in CI environments,
therefore
+ # the jobs based on this template are not run automatically (except if the
user
+ # explicitly sets the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE environment
+ # variable). Adding new jobs on top of this template is strongly discouraged.
+ - if: $QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE == null
+ when: manual
+ allow_failure: true
+ - when: always
build-system-ubuntu:
<<: *native_build_job_definition
The script I used to generate the above metrics:
$ pip install --user python-gitlab
$ cat gitlab-stats
#!/bin/env python3
import gitlab
gl = gitlab.Gitlab('http://gitlab.com')
qemu_project = gl.projects.get(11167699)
pipelines = qemu_project.pipelines.list()
total=0
failed=[]
for pipeline in qemu_project.pipelines.list(page=1, per_page=300):
if pipeline.status == 'running':
continue
elif pipeline.status == 'failed':
failed.append(pipeline)
total += 1
print("Failed %d out of %d" % (len(failed), total))
for pipeline in failed:
print('Pipeline -- %d (%s)' % (pipeline.id, pipeline.ref))
jobs_failed=[]
for job in pipeline.jobs.list():
if job.status == 'failed':
print(job.name)
