On 10/21/2026 3:56 AM, Alex Bennée wrote: > Peter Maydell <[email protected]> writes: > >> On Wed, 21 Oct 2026 at 03:41, Brian Cain <[email protected]> wrote: >>> >>> >>> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote: >>>> When merging PR on the project, it's possible to observe failures on CI >>>> that can be related to sporadic test failures or infrastructure issues >>>> (pod eviction with Kubernetes). As a result, it adds latency on the >>>> process and workload for our top maintainers. >>>> >>>> This patch sets an automatic retry (supported by gitlab out of the box), >>>> so failing jobs are automatically retriggered. >>>> >>>> It's important to note that in case there is a real test or build >>>> failure, concerned jobs will run a second time, and fail again. That >>>> said, it happens seldomly since sub maintainers are supposed to test for >>>> this before sending it. Also, the retry is limited to the job that >>>> failed, so it should be cheap in terms of CI minutes. >>> >>> Will this effectively mask those flaky failures so that no one hears >>> about them? Is there a way to find out about them with something like a >>> weekly/monthly digest of failures? >> >> Broadly speaking, it's already the case that flaky failures are masked. >> Random infra-type failures / random timeouts /etc are common enough >> that when I'm doing a merge CI run I will just hit the retry button >> without much deep investigation. >> >> In an ideal world we'd pull down the gitlab job history and look >> through it for cases of "job X failed running against commit hash Y, >> but we also saw job X succeed on commit hash Y" (i.e. the retry >> worked second time) and then check the failure logs for what >> kinds of things are intermittent like that. But that requires >> somebody to put in the work to do that analysis. Gitlab's own >> CI job history web UI is very bad and doesn't let you do even >> basic things like "show me just the history for job X". > > The gitlab-failure-analysis script was an attempt to start doing this, > so the last 10 failing staging pipelines where: > > $ ./scripts/ci/gitlab-failure-analysis --count 10 > failed pipeline 2949515618, total jobs 120, skipped 5, failed 8, 45059 > tests, 1112 skipped tests, 8 failed tests > Failed job 17194356901, block: [luks], > https://gitlab.com/qemu-project/qemu/-/jobs/17194356901 > Failed job 17192659533, block: [vmdk], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659533 > Failed job 17192659440, block: [vhdx], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659440 > Failed job 17192659398, block: [vdi], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659398 > Failed job 17192659309, block: [qed], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659309 > Failed job 17192659287, block: [qcow2], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659287 > Failed job 17192659245, block: [parallels], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659245 > Failed job 17192659237, block: [nbd], > https://gitlab.com/qemu-project/qemu/-/jobs/17192659237 > Failed test qemu.block-nbd-optional+block-thorough - > qemu:io-nbd-nbd-stale-connection, block: [nbd], 1 raw nbd-stale-connection > Failed test qemu.block-parallels-optional+block-thorough - > qemu:io-parallels-nbd-stale-connection, block: [parallels], 1 parallels > nbd-stale-connection > Failed test qemu.block-qcow2-optional+block-slow - > qemu:io-qcow2-nbd-stale-connection, block: [qcow2], 1 qcow2 > nbd-stale-connection > Failed test qemu.block-qed-optional+block-thorough - > qemu:io-qed-nbd-stale-connection, block: [qed], 1 qed nbd-stale-connection > Failed test qemu.block-vdi-optional+block-thorough - > qemu:io-vdi-nbd-stale-connection, block: [vdi], 1 vdi nbd-stale-connection > Failed test qemu.block-vhdx-optional+block-thorough - > qemu:io-vhdx-nbd-stale-connection, block: [vhdx], 1 vhdx nbd-stale-connection > Failed test qemu.block-vmdk-optional+block-thorough - > qemu:io-vmdk-nbd-stale-connection, block: [vmdk], 1 vmdk nbd-stale-connection > Failed test qemu.block-luks-optional+block-thorough - > qemu:io-luks-nbd-stale-connection, block: [luks], 1 luks nbd-stale-connection > failed pipeline 2945631627, total jobs 120, skipped 5, failed 13, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 17162982778, check-python-minreqs, > https://gitlab.com/qemu-project/qemu/-/jobs/17162982778 > Failed job 17162982652, block: [vpc], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982652 > Failed job 17162982622, block: [vmdk], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982622 > Failed job 17162982605, block: [vhdx], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982605 > Failed job 17162982484, block: [vdi], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982484 > Failed job 17162982417, block: [raw], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982417 > Failed job 17162982413, block: [qed], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982413 > Failed job 17162982411, block: [qcow2], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982411 > Failed job 17162982380, block: [parallels], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982380 > Failed job 17162982361, block: [nbd], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982361 > Failed job 17162982339, block: [luks], > https://gitlab.com/qemu-project/qemu/-/jobs/17162982339 > Failed job 17162261239, check-cfi-x86_64, > https://gitlab.com/qemu-project/qemu/-/jobs/17162261239 > Failed job 17162261192, ubuntu-24.04-s390x-all-system, > https://gitlab.com/qemu-project/qemu/-/jobs/17162261192 > failed pipeline 2929887766, total jobs 120, skipped 7, failed 18, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 17055360127, check-python-minreqs, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360127 > Failed job 17055360119, functional-system-opensuse, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360119 > Failed job 17055360117, functional-system-centos, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360117 > Failed job 17055360116, check-system-centos, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360116 > Failed job 17055360113, block: [vpc], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360113 > Failed job 17055360112, block: [vmdk], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360112 > Failed job 17055360111, block: [vhdx], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360111 > Failed job 17055360110, block: [vdi], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360110 > Failed job 17055360109, block: [raw], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360109 > Failed job 17055360108, block: [qed], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360108 > Failed job 17055360107, block: [qcow2], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360107 > Failed job 17055360106, block: [parallels], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360106 > Failed job 17055360105, block: [nbd], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360105 > Failed job 17055360104, block: [luks], > https://gitlab.com/qemu-project/qemu/-/jobs/17055360104 > Failed job 17055360078, ubuntu-24.04-s390x-all-system, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360078 > Failed job 17055360066, tsan-build, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360066 > Failed job 17055360065, build-cfi-x86_64, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360065 > Failed job 17055360061, clang-system, > https://gitlab.com/qemu-project/qemu/-/jobs/17055360061 > failed pipeline 2908947622, total jobs 119, skipped 5, failed 5, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16911683051, functional-system-debian, > https://gitlab.com/qemu-project/qemu/-/jobs/16911683051 > Failed job 16911681150, functional-cfi-x86_64, > https://gitlab.com/qemu-project/qemu/-/jobs/16911681150 > Failed job 16911679766, functional-system-opensuse, > https://gitlab.com/qemu-project/qemu/-/jobs/16911679766 > Failed job 16911678710, functional-system-centos, > https://gitlab.com/qemu-project/qemu/-/jobs/16911678710 > Failed job 16911433872, ubuntu-24.04-s390x-all-system, > https://gitlab.com/qemu-project/qemu/-/jobs/16911433872 > failed pipeline 2906503036, total jobs 119, skipped 5, failed 4, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16896095104, functional-system-centos, > https://gitlab.com/qemu-project/qemu/-/jobs/16896095104 > Failed job 16896090260, functional-system-opensuse, > https://gitlab.com/qemu-project/qemu/-/jobs/16896090260 > Failed job 16896083592, functional-cfi-x86_64, > https://gitlab.com/qemu-project/qemu/-/jobs/16896083592 > Failed job 16895432496, functional-system-debian, > https://gitlab.com/qemu-project/qemu/-/jobs/16895432496 > failed pipeline 2876539219, total jobs 119, skipped 5, failed 6, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16692899473, functional-system-fedora, > https://gitlab.com/qemu-project/qemu/-/jobs/16692899473 > Failed job 16692899464, aarch64-macos-26-build, > https://gitlab.com/qemu-project/qemu/-/jobs/16692899464 > Failed job 16692899463, aarch64-macos-15-build, > https://gitlab.com/qemu-project/qemu/-/jobs/16692899463 > Failed job 16692899432, clang-system, > https://gitlab.com/qemu-project/qemu/-/jobs/16692899432 > Failed job 16692899429, build-some-softmmu, > https://gitlab.com/qemu-project/qemu/-/jobs/16692899429 > Failed job 16692899413, cross-win64-system, > https://gitlab.com/qemu-project/qemu/-/jobs/16692899413 > failed pipeline 2840522620, total jobs 119, skipped 5, failed 7, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16454091776, functional-system-opensuse, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091776 > Failed job 16454091774, functional-cfi-x86_64, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091774 > Failed job 16454091761, check-cfi-x86_64, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091761 > Failed job 16454091760, check-system-opensuse, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091760 > Failed job 16454091758, functional-system-centos, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091758 > Failed job 16454091740, check-system-centos, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091740 > Failed job 16454091730, ubuntu-24.04-s390x-all-system, > https://gitlab.com/qemu-project/qemu/-/jobs/16454091730 > failed pipeline 2839257817, total jobs 119, skipped 5, failed 2, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16436167533, ubuntu-24.04-s390x-all-system, > https://gitlab.com/qemu-project/qemu/-/jobs/16436167533 > Failed job 16436167532, ubuntu-24.04-s390x-all-linux, > https://gitlab.com/qemu-project/qemu/-/jobs/16436167532 > failed pipeline 2830538304, total jobs 119, skipped 5, failed 3, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16377263801, check-rust-tools-nightly, > https://gitlab.com/qemu-project/qemu/-/jobs/16377263801 > Failed job 16377263686, ubuntu-24.04-s390x-all-system, > https://gitlab.com/qemu-project/qemu/-/jobs/16377263686 > Failed job 16377263684, ubuntu-24.04-s390x-all-linux, > https://gitlab.com/qemu-project/qemu/-/jobs/16377263684 > failed pipeline 2820341086, total jobs 119, skipped 5, failed 2, 0 tests, > 0 skipped tests, 0 failed tests > Failed job 16307778927, check-rust-tools-nightly, > https://gitlab.com/qemu-project/qemu/-/jobs/16307778927 > Failed job 16307778897, functional-system-debian, > https://gitlab.com/qemu-project/qemu/-/jobs/16307778897 > > It has rudimentary access to the tests but I think the deep dive would > require better retention of the job artefacts. The second pipeline in > the above list lists the "job artefacts" as removed 3hrs ago which is I > guess why we don't get the test breakdown for the earlier jobs. > > Anyway perhaps we can expand this to be more useful? >
I guess take the latest 1000 runs, report failing jobs, see if there is a pattern. Then for each run of job failing, see if they have a pattern inside also. Running CI on GitHub side, I found there are some recurrent flaky (that I simply deactivated for now) but some real sporadic that are rarely triggered. Since adding an automatic retry on individual tests, it's almost green 100% of the time, which convinced me to propose this here. >> >> thanks >> -- PMM >
