The nature of block I/O tests is such that there can be unexpected false positive failures in certain scenarios that have not been encountered before, and sometimes non-deterministic failures that are hard to reproduce.
Before enabling the I/O tests as gating jobs in CI, there needs to be a mechanism to dynamically mark tests as skipped, without having to commit code changes. This introduces the QEMU_TEST_IO_SKIP environment variable that is set to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this variable can be set as a GitLab CI pipeline variable to temporarily disable a test while problems are being debugged. Signed-off-by: Daniel P. Berrangé <[email protected]> --- docs/devel/testing/main.rst | 7 +++++++ tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst index 910ec11ab4..bb90ea5f15 100644 --- a/docs/devel/testing/main.rst +++ b/docs/devel/testing/main.rst @@ -285,6 +285,13 @@ that are specific to certain cache mode. More options are supported by the ``./check`` script, run ``./check -h`` for help. +If a test program is known to be broken, it can be disabled by setting +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the +leading component can be omitted to skip the test for all formats and +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding`` +will skip for LUKS only, and ``185`` and ``iov-padding`` for all. + Writing a new test case ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py index dbe2dddc32..ecb5d4529f 100644 --- a/tests/qemu-iotests/testrunner.py +++ b/tests/qemu-iotests/testrunner.py @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False, self._stack: contextlib.ExitStack + self.skip = {} + for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "): + rule = rule.strip() + if rule == "": + continue + if ":" in rule: + fmt, name = rule.split(":") + if fmt in ("", env.imgfmt, env.imgproto): + self.skip[name] = True + else: + self.skip[rule] = True + def __enter__(self) -> 'TestRunner': self._stack = contextlib.ExitStack() self._stack.enter_context(self.env) @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult: description='No qualified output ' f'(expected {f_reference})') + if f_test.name in self.skip: + return TestResult(status='not run', + description='Listed in QEMU_TEST_IO_SKIP') + args = [str(f_test.resolve())] env = self.env.prepare_subprocess(args) -- 2.52.0
