> On Fri, Feb 3, 2017 at 11:05 PM, Kamil Paral < kpa...@redhat.com > wrote:
> > I spent a bit of time fixing minor issues in our test suite and makefiles > > and > > would like to do the following further changes across all our taskotron > > projects: > > > 1. run the test suite while inside virtualenv with simple `pytest` command > > > 2. run the test suite outside of virtualenv with `make test` or `doit test` > > and recommend this approach in readme > > > 3. use a separate virtualenv when running under `make test`, without > > --system-site-packages if possible, and ensure up-to-date deps are always > > installed, to eliminate any differences that can occur on different setups > > The only problem I see here, is that some of the packages that you'd need to > install into the test-virtualenv need some C-compilation when installing > from PyPi, and that (if used without the --system-site-packages) would mean > having to install not only gcc but also plenty of -dev packages. > Not a _huge_ ussue but an issue nevertheless. That's a good point. But do we have a good alternative here? If we depend on packages like that, I see only two options: a) ask the person to install pyfoo as an RPM (in readme) b) ask the person to install gcc and libfoo-devel as an RPM (in readme) and pyfoo will be then compiled and installed from pypi Approach a) is somewhat easier and does not require compilation stack and devel libraries. OTOH it requires using virtualenv with --system-site-packages, which means people get different results on different setups. That's exactly what I'm trying to eliminate (or at least reduce). E.g. https://phab.qa.fedoraproject.org/D1111 where I can run the test suite from makefile and you can't, and it's quite difficult to figure out why. With b) approach, you need compilation stack on the system. I don't think it's such a huge problem, because you're a developer after all. The advantage is that virtualenv can be created without --system-site-packages, which means locally installed libraries do not affect the execution/test suite results. Also, pyfoo is installed with exactly the right version, further reducing differences between setups. The only thing that can differ is the version of libfoo-devel, which can affect the behavior. But the likeliness of that happening is much smaller than having pyfoo of a different version or pulling any deps from the system site packages. The reason why I want to recommend `make test` for running the test suite (at least in readme), is because in the makefile we can ensure that a clean virtualenv with correct properties is created, and only and exactly the right versions of deps from requirements.txt are installed. We can perform further necessary steps, like installing the project . That further increases reliability. Compare this to manually running `pytest`- a custom virtualenv must be active; it can be configured differently than recommended in readme, it can be out of date, or it can have more packages installed than needed; you might forget some necessary steps. Of course nothing prevents you from simply running the test suite using `pytest`. It's the same approach that Phab will do when submitting a patch. However, when some issues arises, I'd like all parties to be able to run `make test` and it should return the same result. That should be the most reliable method, and if it doesn't return the same thing, it means we have an important problem somewhere, and it's not just "a wrongly configured project on one dev machine". So, I see these main use cases for `make test` and b) approach: * good a reliable default for newcomers, an approach that's the least likely to go wrong * determining the reason for failures that only one party sees and the other doesn't * `make test-ci` target, that will hopefully be used one day to perform daily/per-commit CI testing of our codebases. Again, using the most reliable method available. For some codebases this is not viable anyway, e.g. libtaskotron, because they depend on packages not available in pypi (koji) and thus need --system-site-packages. But e.g. resultsdb projects seem that they could go without it.
_______________________________________________ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
