On 18/09/17 09:48, Stuart Prescott wrote: > Hi Diane, > > On Sunday, 17 September 2017 22:14:18 AEST Diane Trout wrote: >> I just did it that way because it was the least disruptive change I >> could make that would let me build and test the package. > > Sure, that's entirely sensible. > >> In my experience I'm much more likely to to notice a build time test >> failure than one from the CI system. > > Absolutely agreed. I'm thinking that this is a rather exceptional situation > because of the circular dependency and the fall-out that we regularly see > from > that. > >> What do other people think? If there are autopkgtests, should you still >> let dh_auto_test run tests? > > (I know I'm not the 'other people' from whom you solicit replies, I was just > perhaps unclear in what I was musing on before so I want to be more clear now) > > Like you, I would *normally* run all tests in both places: buildd tests catch > severe problems right now; ci.d.n reruns the tests each time new versions of > dependencies are uploaded too and later breakage is caught. > > Perhaps this is not a normal situation, however. Either one of "circular > dependencies" or "packages that often FTBFS on one or more architectures" > would be unusual enough to justify doing something different. All I am > wondering (from my position of ignorance!) if in this case, perhaps the tests > that cause the circular dependency can be disabled or xfailed, with the > remaining tests run as normal.
I now prefer to use autopkgtest in place of build-time tests; the former has the advantages: * testing the installed package rather than the source tree, which ensures that the binary package has sufficient dependencies and the correct (or at least sufficient) contents * avoids dependency loops, as noted * avoids the need for workarounds required because of the location of the files being tested at build time (see nbconvert issues with entry_points not being found at build time) * can detect external breakage between uploads via ci.d.n The downsides are that: * boilerplate is required (copying test files out of the source tree to $AUTOPKGTEST_TMP to avoid testing against the source tree, instead of the installed package) * extended build times if using autopkgtest as a post-build hook (installing a chroot multiple times, if using sbuild/similar), and other devs aren't guaranteed to run such tests at build/upload time I wonder if it is possible (or a worthwhile use of time) to try and provide an extended autodep8 python mode which runs the equivalent of dh_auto_test as an autopkgtest, for packages which would opt-in with, eg "Testsuite: autopkgtest-pkg-python-pytest|nose", rather than the basic namespace-can-be-imported test. Gordon