Il mar 16 lug 2024, 13:26 Thomas Huth <th...@redhat.com> ha scritto:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>
I think we should take another look at that. Avocado 92 should work, though
I am not sure if it's also depending on "imp", and if I recall correctly
the problem with more recent version was more that it conflicted with
distros that didn't have it packaged. That should also not be a problem
anymore with the pythondeps.toml mechanism.
Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>
This is true.
Paolo
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>
>
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
>
> make check-functional
>
> You can also run single targets e.g. with:
>
> make check-functional-ppc
>
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>
> export PYTHONPATH=$HOME/qemu/python
> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> export PYTHONPATH=$HOME/qemu/build
> ~/qemu/tests/functional/test_virtio_version.py
>
> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.
>
> Still to be done: Update the documentation for this new test framework.
>
> RFC -> v1:
> - Now using pycotap for running the tests instead of "pytest"
> - Change the name from "tests/pytest" to "tests/functional" accordingly
> - Make it possible to run the tests directly
> - Use Python's urllib instead of wget for downloading
> - Lots of makefile / meson integration improvements
> - Converted more tests
> - Update MAINTAINERS file accordingly
> - Added a patch to run check-functional in the gitlab-CI
> - ... lots of other changes I forgot about ... in fact, I changed so
> many things that I also did not dare to pick up the Reviewed-bys
> from the RFC
>
> Thomas Huth (11):
> tests/functional: Add base classes for the upcoming pytest-based tests
> tests/functional: Convert simple avocado tests into standalone python
> tests
> tests/functional: Convert avocado tests that just need a small
> adjustment
> tests/functional: Add python-based tests to the meson build system
> tests/functional: Implement fetch_asset() method for downloading
> assets
> tests/functional: Convert some tests that download files via
> fetch_asset()
> tests/functional: Add a function for extracting files from an archive
> tests/functional: Convert some avocado tests that needed
> avocado.utils.archive
> tests/functional: Set up logging
> tests/functional: Convert the s390x avocado tests into standalone
> tests
> gitlab-ci: Add "check-functional" to the build tests
>
> MAINTAINERS | 22 +-
> .gitlab-ci.d/buildtest-template.yml | 3 +-
> .gitlab-ci.d/buildtest.yml | 60 +--
> pythondeps.toml | 3 +-
> tests/Makefile.include | 18 +-
> tests/functional/meson.build | 112 +++++
> tests/functional/qemu_test/__init__.py | 384 ++++++++++++++++++
> tests/functional/qemu_test/utils.py | 28 ++
> .../test_arm_canona1100.py} | 21 +-
> .../test_arm_n8x0.py} | 25 +-
> .../test_avr_mega2560.py} | 11 +-
> .../test_cpu_queries.py} | 7 +-
> .../test_empty_cpu_model.py} | 7 +-
> .../test_info_usernet.py} | 11 +-
> .../test_loongarch64_virt.py} | 16 +-
> .../test_mem_addr_space.py} | 53 +--
> .../test_mips64el_loongson3v.py} | 26 +-
> .../test_netdev_ethtool.py} | 32 +-
> .../test_pc_cpu_hotplug_props.py} | 11 +-
> .../ppc_405.py => functional/test_ppc_405.py} | 19 +-
> .../test_ppc_74xx.py} | 74 ++--
> .../test_ppc_bamboo.py} | 23 +-
> .../test_s390x_ccw_virtio.py} | 32 +-
> .../test_s390x_topology.py} | 70 ++--
> .../test_sparc64_sun4u.py} | 25 +-
> .../version.py => functional/test_version.py} | 13 +-
> .../test_virtio_version.py} | 8 +-
> tests/meson.build | 1 +
> 28 files changed, 795 insertions(+), 320 deletions(-)
> create mode 100644 tests/functional/meson.build
> create mode 100644 tests/functional/qemu_test/__init__.py
> create mode 100644 tests/functional/qemu_test/utils.py
> rename tests/{avocado/machine_arm_canona1100.py =>
> functional/test_arm_canona1100.py} (71%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py}
> (71%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py}
> (90%)
> mode change 100644 => 100755
> rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py}
> (89%)
> mode change 100644 => 100755
> rename tests/{avocado/empty_cpu_model.py =>
> functional/test_empty_cpu_model.py} (84%)
> mode change 100644 => 100755
> rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py}
> (87%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_loongarch.py =>
> functional/test_loongarch64_virt.py} (89%)
> mode change 100644 => 100755
> rename tests/{avocado/mem-addr-space-check.py =>
> functional/test_mem_addr_space.py} (92%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_mips_loongson3v.py =>
> functional/test_mips64el_loongson3v.py} (55%)
> mode change 100644 => 100755
> rename tests/{avocado/netdev-ethtool.py =>
> functional/test_netdev_ethtool.py} (81%)
> mode change 100644 => 100755
> rename tests/{avocado/pc_cpu_hotplug_props.py =>
> functional/test_pc_cpu_hotplug_props.py} (90%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py}
> (75%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_s390_ccw_virtio.py =>
> functional/test_s390x_ccw_virtio.py} (95%)
> mode change 100644 => 100755
> rename tests/{avocado/s390_topology.py =>
> functional/test_s390x_topology.py} (90%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_sparc64_sun4u.py =>
> functional/test_sparc64_sun4u.py} (60%)
> mode change 100644 => 100755
> rename tests/{avocado/version.py => functional/test_version.py} (78%)
> mode change 100644 => 100755
> rename tests/{avocado/virtio_version.py =>
> functional/test_virtio_version.py} (98%)
> mode change 100644 => 100755
>
> --
> 2.45.2
>
>
