On Wed, May 13, 2026 at 11:11:47AM -0300, Fabiano Rosas wrote:
> In the case of an environment variable, it's possible to set it without
> any code change (even to .yaml files), by setting "-o ci.variable=<val>"
> at git push time and also by setting directly in GitLab's web interface
> under "New Pipeline". So it allows anyone wanting to "just get work
> done" to skip broken tests for the duration of a pipeline.
>
> It also works when running locally and skipping some tests for the
> single local run.
>
> All temporary changes, of course.
>
> For the case of a flaky test that will need to be disabled for longer
> than a single run, could we standardize on a list committed to
> meson.build, so it works for both CI and local? I think so, and I think
> it's a good idea. Having recently taking up maintainership of QTests I
> sometimes stumble into disabled tests that have been there for years and
> everyone forgot about them. Same with the migration subsystem. Having a
> single place to check from time to time would be helpful.
That was one of the reasons why when wwe introduced the "QEMU_TEST_FLAKY_TESTS"
env in tests/functional, we mandated a gitlab issue URL is provided. This
gives us a record of why it was turned off both in the code and issue tracker.
If we can keep all permanent test disablement behind QEMU_TEST_FLAKY_TESTS
we avoid needing to add logic to meson.build too.
> Migration, by the way, has been using the env variable approach in qtest
> like this:
>
> /*
> * Our CI system has problems with shared memory.
> * Don't run this test until we find a workaround.
> */
> if (getenv("QEMU_TEST_FLAKY_TESTS")) {
>
> Which is a case in which a single list committed in meson.build wouldn't
> help. We actually do want to skip this only for CI.
> ("problems with shared memory" here means that CI environments simply
> don't have enough shared memory space available to run all the tests
> that make use of it)
>
> A further complication is that in the migration case above, that skip
> does not apply to the entire migration-test, it applies to a sub-test
> (./migration-test -p /x86/migration/mode/reboot). Constructs in
> meson.build would not be able to skip this, they can only skip the
> entire migration-test. There is an -x option that gtester accepts, so
> we'd need to write some meson-fu to invoke "./migration-test -x
> /x86/migration/mode/reboot -x ..." if any subtest is to be skipped (note
> that this is not migration-specific, all qtests that invoke
> g_test_add_data* more than once are like this).
>
> Maybe we could work on a proposal to improve this in a generic way for
> all test frameworks.
I have dropped this patch in my v6 posting, as it would indeed be
much more useful if it worked across all the test subsystems.
My v6 has meanwhile introduced the QEMU_TEST_FLAKY_TESTS concept
to the I/O tests (as suggested by Thomas), to skip the dodgy I/O
test universally in all scenarios which satisfies Pierrick's
request.
> PS: I'd even say there's a lot we could make common between
> frameworks. A bunch of what happens around running a test is generic
> programming, not tied to a specific programming language or accelerator
> (in case of qtest).
Yeah, getting everything being represented as meson tests was one
of the things we did to make the frameworks more common.
I think there's scope for having more common env variable names
too. eg we have different env variable names to specify the name
of the QEMU emulator binary which is a bit silly.
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
