On 12/07/2024 12.26, Daniel P. Berrangé wrote:
On Fri, Jul 12, 2024 at 12:14:45PM +0200, Thomas Huth wrote:
On 12/07/2024 11.01, Daniel P. Berrangé wrote:
On Thu, Jul 11, 2024 at 01:55:42PM +0200, Thomas Huth wrote:
From: Ani Sinha <a...@anisinha.ca>
Integrate the pytest framework with the meson build system. This
will make meson run all the pytests under the pytest directory.
Lets add a note about the compelling benefit of this new approach
With this change, each functional test becomes subject
to an individual execution timeout, defaulting to 60
seconds, but overridable per-test.
The avocado runner uses timeouts, too, so it's not really an additional
benefit that we get here.
At the meson level though, we can't put an overall cap on
the execution time, as there's only 1 huge test visible,
and thus the meson timeout multiplier also won't get
honoured IIUC.
For CI purposes we'll need to add 'python3-pytest' to
tests/lcitool/projects/qemu.yml, and re-generate the
the dockerfiles. Some of the other non-gitlab CI
integrations probably need manual additions of pytest
packages.
I'm currently rather looking into getting rid of pytest and to use pycotap
instead: Using the TAP protocol for running the tests, you get a much nicer
output from the meson test runner, which can then count the subtests and
properly report SKIPs for tests that have not been run.
diff --git a/tests/Makefile.include b/tests/Makefile.include
index d39d5dd6a4..68151717d7 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -3,12 +3,14 @@
.PHONY: check-help
check-help:
@echo "Regression testing targets:"
- @echo " $(MAKE) check Run block, qapi-schema, unit,
softfloat, qtest and decodetree tests"
+ @echo " $(MAKE) check Run block, qapi-schema, unit,
softfloat, qtest, pytest and decodetree tests"
@echo " $(MAKE) bench Run speed tests"
@echo
@echo "Individual test suites:"
@echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target"
@echo " $(MAKE) check-qtest Run qtest tests"
+ @echo " $(MAKE) check-pytest Run pytest tests"
+ @echo " $(MAKE) check-pytest-TARGET Run pytest for a given target"
Or name it after the type of test rather than harness ?
eg check-functional / check-functional-TARGET
For that matter perhaps also for the dir name ?
tests/functional/*.py
I almost expected that discussion again ... (see
https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html ) ...
last time we couldn't really agree on such a name and decided to go with the
name of the framework...
I agree that "pytest" is likely not the best name here, especially if
switching to the pycotap test runner instead of using the "pytest" program,
but "functional" might trigger the same discussion again as last time ...
should it rather be "functional" or "validation" or "integration" etc.?
IMHO you can just make an executive decision and pick one of those
three. None of them are terrible, any would be a valid choice.
Ok, I think I'll go with "functional" since that still seems to be the best
fit to me.
Thomas
