Original (very broken) v1 posting: https://lists.nongnu.org/archive/html/qemu-devel/2025-10/msg01650.html
The recent set of regressions identified in the LUKS block driver re-inforced that despite having a hugely useful set of I/O tests, our CI coverage is still letting through too many bugs. The core goals of this series were/are: * Add LUKS and NBD to the tested formats/protocols integrated into "make check-block SPEED=thorough" * Ensure that all qcow2 tests can be run with "SPEED=slow", not staying limited to only the 'auto' group used by the 'make check-block' target in its default 'quick' mode * Add 'make check-block-$FORMAT' to expose a standalone target for running all tests for a given format (or equivalently a protocol) * Add GitLab CI jobs for exercising tests for all formats A sample pipeline for this is https://gitlab.com/berrange/qemu/-/pipelines/2258731804/ Showing the 'block-FORMAT-centos' jobs which test the 10 block formats/protocols currently enabled. This series fixes a couple of bugs * sendmsg generates a deprecation warning on centos * Test 128 check for device mapper usage doesn't work sufficiently well to detect gitlab's containe env can't do dynamic devfs * Test 185 is reliably failing on gitlab I'm unclear if there are other non-deterministic failures that would only appear if we start running the tests regularly, or against QEMU's private runners instead of gitlab shared runners. The ones I've fixed have been consistent across quite a few test pipelines so far with the shared runners. At this point it is a chicken & egg problem though. If we don't bite the bullet and enable block I/O tests in GitLab we'll never find out if they're stable enough to rely on. As a mitigation against instability a patch in this series adds a QEMU_TEST_IO_SKIP env variable that we are able to set as a GitLab CI env variable. This lets us skip broken tests without waiting for a git commit to disable them. Changed in v2: * TL;DR: this time is actally works as described, rather unlike v1 which was majorly broken in its understanding of how mtest2make.py was working * Expand mtest2make.py to support a concept of an "optional" test suite, to create new standalone make check-xxx targets that don't get run by default * Fix sorting of Makefile.mtest to be stable * Fix or disable some bugs in iotests exposed by gitlab * Fix the 'make check-block-$FORMAT' targets so they really don't get triggered by 'make chck' by default. * Avoid duplicating the same qcow2 tests in the 'quick' suite and 'slow' suite. The latter must only have the tests not in the 'auto' group * Add RST documentation of the new make targets * Add env variable to let us dynamically skip I/O tests without git commit changes Daniel P. Berrangé (13): Revert "python/aqmp: fix send_fd_scm for python 3.6.x" tests: print reason when I/O test is skipped in TAP mode tests: remove redundant meson suite for iotests tests: ensure all qcow2 I/O tests are able to be run via make scripts/mtest2make: ensure output has stable sorting scripts/mtest2make: support optional tests grouping tests: add a meson suite / make target per block I/O tests format docs/devel/testing: expand documentation for 'make check-block' tests: add nbd and luks to the I/O test suites tests: skip I/O test 128 on GitLab due to broken dmsetup tests: fix check for sudo access in LUKS I/O test tests: add QEMU_TEST_IO_SKIP for skipping I/O tests gitlab: add jobs for thorough block tests .gitlab-ci.d/buildtest.yml | 92 ++++++++++++++++++++++++++++++++ docs/devel/testing/main.rst | 34 ++++++++++-- python/qemu/qmp/qmp_client.py | 9 ++-- scripts/mtest2make.py | 30 +++++++---- tests/Makefile.include | 3 +- tests/qemu-iotests/128 | 2 + tests/qemu-iotests/149 | 13 +++-- tests/qemu-iotests/meson.build | 55 ++++++++++++++++--- tests/qemu-iotests/testrunner.py | 18 ++++++- 9 files changed, 224 insertions(+), 32 deletions(-) -- 2.52.0
