On Thu, Jul 11, 2024, 7:55 AM Thomas Huth <th...@redhat.com> wrote:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>
> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel.
>
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset since
> it is just an RFC so far, but as you can see, many tests only need
> small modifications to work without Avocado.
>
> If you want to try it: Apply the patches, make sure that you have the
> "pytest" program installed, then recompile and then run:
>
> make check-pytest
>
> Things that need further attention though:
>
> - All tests that use the LinuxTest / LinuxDistro classes (e.g. based
> on cloud-init images) really depend on the Avocado framework,
> thus we'd need a solution for those if we want to continue with
> this approach
>
> - Same for all tests that require the LinuxSSHMixIn class - we'd
> need to provide a solution for ssh-based tests, too.
>
> - We lose the way of running tests via the avocado tags this way...
> single targets can still be tested by running "make check-pytest-arm"
> for example, but running selected tests by other tags does not
> work anymore.
>
> - I haven't looked into logging yet ... this still needs some work
> so that you could e.g. inspect the console output of the guests
> somewhere
>
This has spilled the most developer blood of any other problem with the
Python-based tests. Be very careful here.
I still have a prototype for replacing QMPMachine with an asyncio variant
that should have more robust logging features, but I put it on the
back-burner.
Avocado tests are the primary user of the QMP Machine interface I hate the
very most, a multi-threaded buffer-reader that works only by the grace of
god. If you do go down this path, I may want to take the opportunity to
abolish that interface once and for all.
I think simplifying the console buffering will help ease debuggability.
(Note, this isn't an avocado exclusive problem so much as it is the
emergent evolution of both qmp machine and avocado developing their own
solutions to console logging problems, resulting in two layers that are
trying to do similar things.)
> - I did not work on documentation updates yet (will do that if we
> agree to continue with this patch series)
>
> What's your thoughts? Is it worth to continue with this approach?
> Or shall I rather forget about it and wait for the Avocado version
> update?
>
I'm personally ambivalent on avocado; I use it for the python self-tests as
dogfooding but I can likely switch back over to plain pytest if that's the
direction we head. I don't think I use any crazy features except some
asyncio helpers i advocated for. I'm not sure what pytest's asyncio support
looks like, but I have to imagine as the premier testing framework that it
has *something* for me to use.
My only ask is that we keep the tests running in the custom venv
environment we set up at build time. We have some funky post-hoc
initialization of avocado that allows us to use internet packages
post-config for testing purposes. If we move to pytest, it's possible we
can eliminate that funkiness, which would be a win.
I'm also not so sure about recreating all of the framework that pulls vm
images on demand, that sounds like it'd be a lot of work, but maybe I'm
wrong about that.
Tacit ACK from me on this project in general, provided we are still using
the configure venv.
> Thomas
>
>
> Ani Sinha (1):
> tests/pytest: add pytest to the meson build system
>
> Thomas Huth (7):
> tests/pytest: Add base classes for the upcoming pytest-based tests
> tests/pytest: Convert some simple avocado tests into pytests
> tests/pytest: Convert info_usernet and version test with small
> adjustments
> tests_pytest: Implement fetch_asset() method for downloading assets
> tests/pytest: Convert some tests that download files via fetch_asset()
> tests/pytest: Add a function for extracting files from an archive
> tests/pytest: Convert avocado test that needed avocado.utils.archive
>
> tests/Makefile.include | 4 +-
> tests/meson.build | 1 +
> tests/pytest/meson.build | 74 ++++
> tests/pytest/qemu_pytest/__init__.py | 362 ++++++++++++++++++
> tests/pytest/qemu_pytest/utils.py | 21 +
> .../test_arm_canona1100.py} | 16 +-
> .../test_cpu_queries.py} | 2 +-
> .../test_empty_cpu_model.py} | 2 +-
> .../test_info_usernet.py} | 6 +-
> .../test_machine_arm_n8x0.py} | 20 +-
> .../test_machine_avr6.py} | 7 +-
> .../test_machine_loongarch.py} | 11 +-
> .../test_machine_mips_loongson3v.py} | 19 +-
> .../test_mem_addr_space.py} | 3 +-
> .../test_ppc_bamboo.py} | 18 +-
> .../version.py => pytest/test_version.py} | 8 +-
> .../test_virtio_version.py} | 2 +-
> 17 files changed, 502 insertions(+), 74 deletions(-)
> create mode 100644 tests/pytest/meson.build
> create mode 100644 tests/pytest/qemu_pytest/__init__.py
> create mode 100644 tests/pytest/qemu_pytest/utils.py
> rename tests/{avocado/machine_arm_canona1100.py =>
> pytest/test_arm_canona1100.py} (74%)
> rename tests/{avocado/cpu_queries.py => pytest/test_cpu_queries.py} (96%)
> rename tests/{avocado/empty_cpu_model.py =>
> pytest/test_empty_cpu_model.py} (94%)
> rename tests/{avocado/info_usernet.py => pytest/test_info_usernet.py}
> (91%)
> rename tests/{avocado/machine_arm_n8x0.py =>
> pytest/test_machine_arm_n8x0.py} (71%)
> rename tests/{avocado/machine_avr6.py => pytest/test_machine_avr6.py}
> (91%)
> rename tests/{avocado/machine_loongarch.py =>
> pytest/test_machine_loongarch.py} (89%)
> rename tests/{avocado/machine_mips_loongson3v.py =>
> pytest/test_machine_mips_loongson3v.py} (59%)
> rename tests/{avocado/mem-addr-space-check.py =>
> pytest/test_mem_addr_space.py} (99%)
> rename tests/{avocado/ppc_bamboo.py => pytest/test_ppc_bamboo.py} (75%)
> rename tests/{avocado/version.py => pytest/test_version.py} (82%)
> rename tests/{avocado/virtio_version.py => pytest/test_virtio_version.py}
> (99%)
>
> --
> 2.45.2
>
>
