On Fri, Jul 26, 2024 at 6:07 AM Thomas Huth <th...@redhat.com> wrote: > > On 25/07/2024 16.21, Cleber Rosa wrote: > > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <th...@redhat.com> wrote: > ... > >> There have been several attempts to update the test suite in QEMU > >> to a newer version of Avocado, but so far no attempt has successfully > >> been merged yet. > > > > So, we've seen in the past an attempt to update Avocado from 88.1 to a > > regular release, and the troubles it caused, including a revert. My > > take was that a LTS version should be used, but during this time, > > Avocado experienced a rewrite and having it replacing the old > > implementation in a production level project such as QEMU was tricky. > > Then 103.0 LTS was released, and there was extensive work to test the > > QEMU tests before that release was cut. Additionally, there was > > further work, but unfortunately not posted yet, to make use of 103.0 > > features in the existing tests[2]. I've tested on GitLab with tests > > running in parallel, cutting job times in 1/3[2]. A side node is > > that, because 103.0 is an LTS release, it will receive the needed bug > > fixes and updates that we deem necessary, including things we find in > > QEMU tests. In fact, 103.1[3] is in the works. > > Hi Cleber, > > thanks for the explanation, but we really need to replace v88 rather *now* > since v88 does not work with the latest versions of Python anymore (there is > a work-around on Fedora fortunately, but it's completely broken on Ubuntu > 24.04 as far as I know). So even a single-threaded execution with v103 would > have been better than waiting forever for your update to land. The problem > with v88 being broken has been raised a couple of times already, but it's > incredibly hard to get a response from you Avocado folks, so with hardly any > help from the Avocado side, and nobody on the QEMU side being really > familiar with the Avocado stuff, and with the meson test runner being used > by all other subsystems in QEMU already, I think it's best if we continue > with this series here. >
Hi Thomas, I agree with the urgency. I've posted the updates here: https://lists.gnu.org/archive/html/qemu-devel/2024-07/msg06236.html Sorry for the delay, and I hope this gives more time to make the evolution towards a solution that better suits QEMU. Regards, - Cleber. > >> Additionally, the whole "make check" test suite in QEMU is using the > >> meson test runner nowadays, so running the python-based tests via the > >> Avocodo test runner looks and feels quite like an oddball, requiring > >> the users to deal with the knowledge of multiple test runners in > >> parallel (e.g. the timeout settings work completely differently). > >> > > > > Now I believe we can be very much in sync here. I've thought for a > > while that there's no reason for Avocado to cooperate or be compatible > > with Meson. There's no reason why users can't simply pick how the > > test gets run. In fact, with the new Avocado architecture, you don't > > even need to run "avocado" to run an "avocado-instrumented" test. You > > could pretty much run "avocado-runner-avocado-instrumented" with the > > right parameters through Meson. > > Ok, good to know, we could maybe use that eventually for the tests that > really require the Avocado framework (i.e. the cloud-init based tests). > > For the others, as Daniel said in an earlier mail, it's much more convenient > if you can also run the tests directly instead, without such a layer in > between - that makes debugging way easier. > > >> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > >> classes (e.g. based on cloud-init images or using SSH) really depend > >> on the Avocado framework, so we'd need a solution for those if we > >> want to continue using them. One solution might be to simply use the > >> required functions from avocado.utils for these tests, and still run > >> them via the meson test runner instead, but that needs some further > >> investigation that will be done later. > > > > So, I believe this type of higher level testing is something that > > needs to remain, and even grow. Speaking for Red Hat, I see the > > movement of QE contributing more Avocado-VT style tests into QEMU > > itself. > > I didn't really see such a movement in recent times yet ... Could you point > to an example? > Indeed, there is no public evidence of that, yet. > > This means way more libraries and features that go into a > > common set of utilities and features (more on that later) than it > > currently exists in avocado.utils. > > > > This brings the autils[4] initiative into the picture. We're about > > 80% done with the project structure, and after that, it will be a > > common utility project (such as the cloudinit and ssh) which can be > > released automatically when the maintainer votes (through GitHub) that > > a new release is needed. > > autils sounds promising, but I just had a look at the repository, and there > does not seem to be that much code available there yet, so I guess it will > take still quite a long time 'til that's ready? > Like I said previously, we were working on the project infrastructure. We haven't ported much code over there yet. There are very few things missing, and all of them have been assigned to be worked on. It's been tracked here: https://github.com/avocado-framework/autils/issues/1 My estimation is that 2-4 weeks for the project infrastructure to be complete. Regards, - Cleber. > Thomas >
