Peter Maydell <[email protected]> writes:
> On Wed, 21 Oct 2026 at 03:41, Brian Cain <[email protected]> wrote:
>>
>>
>> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
>> > When merging PR on the project, it's possible to observe failures on CI
>> > that can be related to sporadic test failures or infrastructure issues
>> > (pod eviction with Kubernetes). As a result, it adds latency on the
>> > process and workload for our top maintainers.
>> >
>> > This patch sets an automatic retry (supported by gitlab out of the box),
>> > so failing jobs are automatically retriggered.
>> >
>> > It's important to note that in case there is a real test or build
>> > failure, concerned jobs will run a second time, and fail again. That
>> > said, it happens seldomly since sub maintainers are supposed to test for
>> > this before sending it. Also, the retry is limited to the job that
>> > failed, so it should be cheap in terms of CI minutes.
>>
>> Will this effectively mask those flaky failures so that no one hears
>> about them? Is there a way to find out about them with something like a
>> weekly/monthly digest of failures?
>
> Broadly speaking, it's already the case that flaky failures are masked.
> Random infra-type failures / random timeouts /etc are common enough
> that when I'm doing a merge CI run I will just hit the retry button
> without much deep investigation.
>
> In an ideal world we'd pull down the gitlab job history and look
> through it for cases of "job X failed running against commit hash Y,
> but we also saw job X succeed on commit hash Y" (i.e. the retry
> worked second time) and then check the failure logs for what
> kinds of things are intermittent like that. But that requires
> somebody to put in the work to do that analysis. Gitlab's own
> CI job history web UI is very bad and doesn't let you do even
> basic things like "show me just the history for job X".
The gitlab-failure-analysis script was an attempt to start doing this,
so the last 10 failing staging pipelines where:
$ ./scripts/ci/gitlab-failure-analysis --count 10
failed pipeline 2949515618, total jobs 120, skipped 5, failed 8, 45059
tests, 1112 skipped tests, 8 failed tests
Failed job 17194356901, block: [luks],
https://gitlab.com/qemu-project/qemu/-/jobs/17194356901
Failed job 17192659533, block: [vmdk],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659533
Failed job 17192659440, block: [vhdx],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659440
Failed job 17192659398, block: [vdi],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659398
Failed job 17192659309, block: [qed],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659309
Failed job 17192659287, block: [qcow2],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659287
Failed job 17192659245, block: [parallels],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659245
Failed job 17192659237, block: [nbd],
https://gitlab.com/qemu-project/qemu/-/jobs/17192659237
Failed test qemu.block-nbd-optional+block-thorough -
qemu:io-nbd-nbd-stale-connection, block: [nbd], 1 raw nbd-stale-connection
Failed test qemu.block-parallels-optional+block-thorough -
qemu:io-parallels-nbd-stale-connection, block: [parallels], 1 parallels
nbd-stale-connection
Failed test qemu.block-qcow2-optional+block-slow -
qemu:io-qcow2-nbd-stale-connection, block: [qcow2], 1 qcow2 nbd-stale-connection
Failed test qemu.block-qed-optional+block-thorough -
qemu:io-qed-nbd-stale-connection, block: [qed], 1 qed nbd-stale-connection
Failed test qemu.block-vdi-optional+block-thorough -
qemu:io-vdi-nbd-stale-connection, block: [vdi], 1 vdi nbd-stale-connection
Failed test qemu.block-vhdx-optional+block-thorough -
qemu:io-vhdx-nbd-stale-connection, block: [vhdx], 1 vhdx nbd-stale-connection
Failed test qemu.block-vmdk-optional+block-thorough -
qemu:io-vmdk-nbd-stale-connection, block: [vmdk], 1 vmdk nbd-stale-connection
Failed test qemu.block-luks-optional+block-thorough -
qemu:io-luks-nbd-stale-connection, block: [luks], 1 luks nbd-stale-connection
failed pipeline 2945631627, total jobs 120, skipped 5, failed 13, 0 tests, 0
skipped tests, 0 failed tests
Failed job 17162982778, check-python-minreqs,
https://gitlab.com/qemu-project/qemu/-/jobs/17162982778
Failed job 17162982652, block: [vpc],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982652
Failed job 17162982622, block: [vmdk],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982622
Failed job 17162982605, block: [vhdx],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982605
Failed job 17162982484, block: [vdi],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982484
Failed job 17162982417, block: [raw],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982417
Failed job 17162982413, block: [qed],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982413
Failed job 17162982411, block: [qcow2],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982411
Failed job 17162982380, block: [parallels],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982380
Failed job 17162982361, block: [nbd],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982361
Failed job 17162982339, block: [luks],
https://gitlab.com/qemu-project/qemu/-/jobs/17162982339
Failed job 17162261239, check-cfi-x86_64,
https://gitlab.com/qemu-project/qemu/-/jobs/17162261239
Failed job 17162261192, ubuntu-24.04-s390x-all-system,
https://gitlab.com/qemu-project/qemu/-/jobs/17162261192
failed pipeline 2929887766, total jobs 120, skipped 7, failed 18, 0 tests, 0
skipped tests, 0 failed tests
Failed job 17055360127, check-python-minreqs,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360127
Failed job 17055360119, functional-system-opensuse,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360119
Failed job 17055360117, functional-system-centos,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360117
Failed job 17055360116, check-system-centos,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360116
Failed job 17055360113, block: [vpc],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360113
Failed job 17055360112, block: [vmdk],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360112
Failed job 17055360111, block: [vhdx],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360111
Failed job 17055360110, block: [vdi],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360110
Failed job 17055360109, block: [raw],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360109
Failed job 17055360108, block: [qed],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360108
Failed job 17055360107, block: [qcow2],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360107
Failed job 17055360106, block: [parallels],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360106
Failed job 17055360105, block: [nbd],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360105
Failed job 17055360104, block: [luks],
https://gitlab.com/qemu-project/qemu/-/jobs/17055360104
Failed job 17055360078, ubuntu-24.04-s390x-all-system,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360078
Failed job 17055360066, tsan-build,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360066
Failed job 17055360065, build-cfi-x86_64,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360065
Failed job 17055360061, clang-system,
https://gitlab.com/qemu-project/qemu/-/jobs/17055360061
failed pipeline 2908947622, total jobs 119, skipped 5, failed 5, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16911683051, functional-system-debian,
https://gitlab.com/qemu-project/qemu/-/jobs/16911683051
Failed job 16911681150, functional-cfi-x86_64,
https://gitlab.com/qemu-project/qemu/-/jobs/16911681150
Failed job 16911679766, functional-system-opensuse,
https://gitlab.com/qemu-project/qemu/-/jobs/16911679766
Failed job 16911678710, functional-system-centos,
https://gitlab.com/qemu-project/qemu/-/jobs/16911678710
Failed job 16911433872, ubuntu-24.04-s390x-all-system,
https://gitlab.com/qemu-project/qemu/-/jobs/16911433872
failed pipeline 2906503036, total jobs 119, skipped 5, failed 4, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16896095104, functional-system-centos,
https://gitlab.com/qemu-project/qemu/-/jobs/16896095104
Failed job 16896090260, functional-system-opensuse,
https://gitlab.com/qemu-project/qemu/-/jobs/16896090260
Failed job 16896083592, functional-cfi-x86_64,
https://gitlab.com/qemu-project/qemu/-/jobs/16896083592
Failed job 16895432496, functional-system-debian,
https://gitlab.com/qemu-project/qemu/-/jobs/16895432496
failed pipeline 2876539219, total jobs 119, skipped 5, failed 6, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16692899473, functional-system-fedora,
https://gitlab.com/qemu-project/qemu/-/jobs/16692899473
Failed job 16692899464, aarch64-macos-26-build,
https://gitlab.com/qemu-project/qemu/-/jobs/16692899464
Failed job 16692899463, aarch64-macos-15-build,
https://gitlab.com/qemu-project/qemu/-/jobs/16692899463
Failed job 16692899432, clang-system,
https://gitlab.com/qemu-project/qemu/-/jobs/16692899432
Failed job 16692899429, build-some-softmmu,
https://gitlab.com/qemu-project/qemu/-/jobs/16692899429
Failed job 16692899413, cross-win64-system,
https://gitlab.com/qemu-project/qemu/-/jobs/16692899413
failed pipeline 2840522620, total jobs 119, skipped 5, failed 7, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16454091776, functional-system-opensuse,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091776
Failed job 16454091774, functional-cfi-x86_64,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091774
Failed job 16454091761, check-cfi-x86_64,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091761
Failed job 16454091760, check-system-opensuse,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091760
Failed job 16454091758, functional-system-centos,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091758
Failed job 16454091740, check-system-centos,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091740
Failed job 16454091730, ubuntu-24.04-s390x-all-system,
https://gitlab.com/qemu-project/qemu/-/jobs/16454091730
failed pipeline 2839257817, total jobs 119, skipped 5, failed 2, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16436167533, ubuntu-24.04-s390x-all-system,
https://gitlab.com/qemu-project/qemu/-/jobs/16436167533
Failed job 16436167532, ubuntu-24.04-s390x-all-linux,
https://gitlab.com/qemu-project/qemu/-/jobs/16436167532
failed pipeline 2830538304, total jobs 119, skipped 5, failed 3, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16377263801, check-rust-tools-nightly,
https://gitlab.com/qemu-project/qemu/-/jobs/16377263801
Failed job 16377263686, ubuntu-24.04-s390x-all-system,
https://gitlab.com/qemu-project/qemu/-/jobs/16377263686
Failed job 16377263684, ubuntu-24.04-s390x-all-linux,
https://gitlab.com/qemu-project/qemu/-/jobs/16377263684
failed pipeline 2820341086, total jobs 119, skipped 5, failed 2, 0 tests, 0
skipped tests, 0 failed tests
Failed job 16307778927, check-rust-tools-nightly,
https://gitlab.com/qemu-project/qemu/-/jobs/16307778927
Failed job 16307778897, functional-system-debian,
https://gitlab.com/qemu-project/qemu/-/jobs/16307778897
It has rudimentary access to the tests but I think the deep dive would
require better retention of the job artefacts. The second pipeline in
the above list lists the "job artefacts" as removed 3hrs ago which is I
guess why we don't get the test breakdown for the earlier jobs.
Anyway perhaps we can expand this to be more useful?
>
> thanks
> -- PMM
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
