make check is currently by default running tests tagged
"slow". Running make check with no SPEED variable set should be
equivalent to SPEED=quick and run the meson 'quick' test setup, which
explicitly excludes slow tests:
add_test_setup('quick', exclude_suites: ['slow', 'thorough'] ...
The problem is that the mtest2make.py script is effectively overriding
the "meson test --setup" setting by using --suite which takes
precedence over --setup according to meson documentation.
Fix by excluding suites directly with the "--no-suite" option while
generating Makefile.mtest.
Although the 'exclude_suites' in meson.build are useless for make
check, they still help when running 'meson test' from the
command-line, so keep them around.
For the configure line I'm using:
make check goes from 1100 to 996 tests
make SPEED=slow check stays at 2158 tests
make SPEED=thorough check stays at 2279 tests
(the rest of the test targets look sane, but maintainers please
double-check)
Signed-off-by: Fabiano Rosas <[email protected]>
---
- With this change we obviously lose coverage in make check. I suggest
we use this opportunity to review what gets put in the quick
suite.
For migration, I want to keep only the smoke tests in make
check. I'll send a patch adding them back if this change is
accepted.
- Some of the suffixes from functional tests could probably be removed
and rely on having the tests in the proper suite instead. I.e. use
'slow' instead of 'foobar_slow'.
---
scripts/mtest2make.py | 11 +++++++++--
tests/qemu-iotests/meson.build | 2 +-
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 383ea68b16..a61a83f775 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -26,6 +26,11 @@ def names(self, base):
.speed.slow = $(sort $(filter-out %-thorough, $1))
.speed.thorough = $(sort $1)
+.speed_exclude_suites.quick = --no-suite slow --no-suite thorough \
+ --no-suite optional
+.speed_exclude_suites.slow = --no-suite thorough
+.speed_exclude_suites.thorough =
+
TIMEOUT_MULTIPLIER ?= 1
.mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER)
ifneq ($(SPEED), quick)
@@ -34,7 +39,8 @@ def names(self, base):
.mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword -j1
$(filter -j%, $(MAKEFLAGS)))))
.check.mtestargs = $(MTESTARGS) $(.mtestargs) $(if
$(V),--verbose,--print-errorlogs) \
- $(foreach s, $(sort $(.check.mtest-suites)), --suite $s)
+ $(foreach s, $(sort $(.check.mtest-suites)), --suite $s) \
+ $(.speed_exclude_suites.$(SPEED))
.bench.mtestargs = $(MTESTARGS) $(.mtestargs) --benchmark --verbose \
$(foreach s, $(sort $(.bench.mtest-suites)), --suite $s)''')
@@ -102,7 +108,8 @@ def emit_suite(name, suite, prefix):
targets += f' {prefix} {prefix}-report.junit.xml'
print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
# for the "base" suite possibly add FOO-slow and FOO-thorough
- print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names})")
+ print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names}) "
+ ".speed_exclude_suites.$(SPEED)")
print(f'endif')
targets = {t['id']: [os.path.relpath(f) for f in t['filename']]
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 8857f25f29..34cfcdc53b 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -68,7 +68,7 @@ foreach driver, speed: qemu_iotests_drivers
endif
# Every driver gets put in the driver specific suite
- suites = ['block-' + driver + '-optional']
+ suites = ['block-' + driver + '-optional', 'optional']
# Any driver tagged quick or slow also gets added to slow
# otherwise its tagged thorough
if speed != 'thorough'
--
2.53.0
