On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote: > Hi Cleber, > > Here are my notes from talking about Avocado with various people during the > KVM forum in Lyon last month. > > All comments are QEMU oriented. > > > 1) Working offline > > Various people complained Avocado requires online access, and they would > like to use it offline. >
Just to be extra clear, Avocado itself doesn't require online access, but: * "make check-venv" will download packages from PyPI if the same Avocado version is not previously installed in the system * "make check-acceptance" depends on "make check-venv" * many tests depend on downlodable assets > Maintainer workflow example is: > > - run avocado > - hack QEMU, build > - git pull > - build > - hack QEMU > (go offline) > - hack QEMU > - build > - run avocado <- FAILS > > Failure is because mainstream added new tests, which got pulled in, and the > user only notice when running avocado again, but offline. > Test example is boot_linux_console.py, which added various tests from other > subsystems, so the maintainer has to disable the new tests manually to be > able to run his previous tests. > > Expected solution: skip tests when artifact is not available, eventually > when the --offline option is used > Understood and agreed. I've already adopted this approach in the "boot_linux.py" series I'm working (about to send v8). If the download/preparation of images fail, we cancel the tests. I'll look into making that the default across all tests or in the base avocado_qemu.Test class. We could also have a "make check-acceptance-prepare" kind of target, that won't run the tests, but will download all needed assets. > > 2) Add artifacts manually to the cache > > Not all artifacts can be easily downloadable, some are public but require > the user to accept an End User License Agreement. > Users would like to share their tests with the documentation about where/how > to download the requisite files (accepting the EULA) to run the tests. > > OK, RFE taken. > 2b) Add reference to artifact to the cache > > Group of users might share group of files (like NFS storage) and would like > to use directly their remote read-only files, instead of copying it to their > home directory. > > The underlying "asset fetcher" utility code supports multiple locations and multiple cache dirs (and one could be a read-only NFS-like storage): https://avocado-framework.readthedocs.io/en/73.0/api/utils/avocado.utils.html#avocado.utils.asset.Asset But we need to make that easily accessible/configurable to users of the fetch_asset() Test method. RFE taken. > 3) Provide qemu/avocado-qemu Python packages > > The mainstream project uses Avocado to test QEMU. Others projects use QEMU > to test their code, and would like to automatize that using Avocado. They > usually not rebuild QEMU but use a stable binary from distributions. The > python classes are not available, so they have to clone QEMU to use Avocado > (I guess they only need 5 python files). > When running on Continuous Integration, this is overkill, because when you > clone QEMU you also clone various other submodules. > Agreed, and I already have a prototype. I'll send the RFC/patches to the list ASAP. > > 4) Warn the user when Avocado is too old for the tests > > Some users tried Avocado following the examples on the mailing list and the > one in some commit's descriptions where we simply show "avocado run ...". > They installed the distribution Avocado package and tried and it fails for > few of them with no obvious reason (the .log file is hard to read when you > are not custom to). IIUC their distribution provides a older Avocado (69?) > while we use recent features (72). > > We never noticed it because we use 'make check-venv' and do not test the > distrib Avocado. While we can not test all distribs, we could add a version > test if the Avocado version is too old, display a friendly message to the > console (not the logfile). > OK, agreed. RFE taken. > > That it for my notes. > > Eduardo/Wainer, are there other topics I forgot? > > > Regards, > > Phil. > Thanks *so much* for this feedback! I'll provide individual feedback as each of those items progresses. - Cleber.