On 18.09.2017 19:24, Gordon Ball wrote: > On 18/09/17 09:48, Stuart Prescott wrote: >> Hi Diane, >> >> On Sunday, 17 September 2017 22:14:18 AEST Diane Trout wrote: >>> I just did it that way because it was the least disruptive change I >>> could make that would let me build and test the package. >> >> Sure, that's entirely sensible. >> >>> In my experience I'm much more likely to to notice a build time test >>> failure than one from the CI system. >> >> Absolutely agreed. I'm thinking that this is a rather exceptional situation >> because of the circular dependency and the fall-out that we regularly see >> from >> that. >> >>> What do other people think? If there are autopkgtests, should you still >>> let dh_auto_test run tests? >> >> (I know I'm not the 'other people' from whom you solicit replies, I was just >> perhaps unclear in what I was musing on before so I want to be more clear >> now) >> >> Like you, I would *normally* run all tests in both places: buildd tests >> catch >> severe problems right now; ci.d.n reruns the tests each time new versions of >> dependencies are uploaded too and later breakage is caught. >> >> Perhaps this is not a normal situation, however. Either one of "circular >> dependencies" or "packages that often FTBFS on one or more architectures" >> would be unusual enough to justify doing something different. All I am >> wondering (from my position of ignorance!) if in this case, perhaps the >> tests >> that cause the circular dependency can be disabled or xfailed, with the >> remaining tests run as normal. > > I now prefer to use autopkgtest in place of build-time tests; the former > has the advantages: > > * testing the installed package rather than the source tree, which > ensures that the binary package has sufficient dependencies and the > correct (or at least sufficient) contents > * avoids dependency loops, as noted > * avoids the need for workarounds required because of the location of > the files being tested at build time (see nbconvert issues with > entry_points not being found at build time) > * can detect external breakage between uploads via ci.d.n > > The downsides are that: > > * boilerplate is required (copying test files out of the source tree to > $AUTOPKGTEST_TMP to avoid testing against the source tree, instead of > the installed package) > * extended build times if using autopkgtest as a post-build hook > (installing a chroot multiple times, if using sbuild/similar), and other > devs aren't guaranteed to run such tests at build/upload time > > I wonder if it is possible (or a worthwhile use of time) to try and > provide an extended autodep8 python mode which runs the equivalent of > dh_auto_test as an autopkgtest, for packages which would opt-in with, eg > "Testsuite: autopkgtest-pkg-python-pytest|nose", rather than the basic > namespace-can-be-imported test.
the biggest downside with this approach is that you *completely* skip any testing on other architectures than amd64. Is that what you really want? Dear porters, have fun where to search for bugs in packages without testsuites! autopkg testing is not a replacement for runnning tests at build time from my point of view, but to run a different set of tests. It's a waste of build time on the autopkg testers if you need to build the package again and then run against the tests in the build directory. Many testsuites are not made for autpkg testing from the installed place from what I can see. Matthias
