Bug#926409: lintian: autopkgtest takes very long to finish
Felix asked me to report the performance data for the lintian testsuite I'm getting on my new desktop computer. Let me first say I'm a very casual contributor to Lintian and I'm certainly not in a position to judge if the testsuite takes too much time to run or is too extensive by default. Take this as additional data to make informed decisions :) On my previous desktop computer (AMD FX-8530, 8GB DDR3 RAM, SATA SSD), running the testsuite used to take up to 15mins. Now with the new PC (AMD Ryzen 5900x, 64GB DDR4 RAM, M.2 NVME SSD), it takes less than 3 minutes. This is the result of running `private/runtests` from a newly created git clone of the current master branch: --- All tests successful. Files=1306, Tests=59485, 178 wallclock secs ( 6.85 usr 2.34 sys + 3356.34 cusr 422.52 csys = 3788.05 CPU) Result: PASS The test suite ran for 2 minutes and 59 seconds. --- Then again, that's a brand new machine running top of the line consumer-grade hardware, with a beefy cooler :) Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau ⢿⡄⠘⠷⠚⠋ po...@debian.org / veronneau.org ⠈⠳⣄ OpenPGP_signature Description: OpenPGP digital signature
Bug#926409: lintian: autopkgtest takes very long to finish
Felix Lechner wrote: > > We do similar in some pkg-gnome packages, for example glib2.0 ships a > > -tests package that contains "installed tests" which are compiled as > > part of the package build and then executed during the autopkgtests. > > Should we ship all built test packages as part of our releases? I > can't think of a better way to close this bug. I'd be -1 on that for difficult-to-elucidate reasons and there are likely other approaches to explore outside of this bug. I think we have sped up things enough to close this particular issue, at least for now. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#926409: lintian: autopkgtest takes very long to finish
On Wed, Jun 03, 2020 at 02:50:04AM +0200, Mattia Rizzolo wrote: > On Wed, 3 Jun 2020, 1:09 am Felix Lechner, > ... > > On Fri, Apr 5, 2019 at 4:33 AM Iain Lane wrote: > > > > > > We do similar in some pkg-gnome packages, for example glib2.0 ships a > > > -tests package that contains "installed tests" which are compiled as > > > part of the package build and then executed during the autopkgtests. > > > > Should we ship all built test packages as part of our releases? I > > can't think of a better way to close this bug. > > > Now lintian autopkgtests take approximately 1 hour everywhere I checked. > Honestly, I believe 1 hour to be acceptable. It is broadly acceptable, but if you can reduce the time by assembling artifacts in advance, I think that it is still worth doing to save time and not repeat computation that doesn't need to be repeated. As a bonus you're then not also testing the package build toolchain with each Lintian CI run - you are (mostly) only testing Lintian itself. Cheers, -- Iain Lane [ i...@orangesquash.org.uk ] Debian Developer [ la...@debian.org ] Ubuntu Developer [ la...@ubuntu.com ] signature.asc Description: PGP signature
Bug#926409: lintian: autopkgtest takes very long to finish
On Wed, 3 Jun 2020, 1:09 am Felix Lechner, wrote: > Hi Chris, > > On Fri, Apr 5, 2019 at 4:33 AM Iain Lane wrote: > > > > We do similar in some pkg-gnome packages, for example glib2.0 ships a > > -tests package that contains "installed tests" which are compiled as > > part of the package build and then executed during the autopkgtests. > > Should we ship all built test packages as part of our releases? I > can't think of a better way to close this bug. Now lintian autopkgtests take approximately 1 hour everywhere I checked. Honestly, I believe 1 hour to be acceptable. >
Bug#926409: lintian: autopkgtest takes very long to finish
Hi Chris, On Fri, Apr 5, 2019 at 4:33 AM Iain Lane wrote: > > We do similar in some pkg-gnome packages, for example glib2.0 ships a > -tests package that contains "installed tests" which are compiled as > part of the package build and then executed during the autopkgtests. Should we ship all built test packages as part of our releases? I can't think of a better way to close this bug. Kind regards Felix Lechner
Bug#926409: lintian: autopkgtest takes very long to finish
On Thu, Apr 04, 2019 at 12:10:31PM -0700, Felix Lechner wrote: > On Thu, Apr 4, 2019 at 10:42 AM Chris Lamb wrote: > > > > * I'm not sure *how* we can speed up the tests. I mean, they all > >essentially involve building Debian packages with all the usual > >debhelper calls, etc. Speeding *this* up is somewhat out-of-scope > >of this Lintian wishlist issue, alas. > > > > However, perhaps Felix has some input here as he has been doing a lot > > of work on the test suite recently? > > About 95% of the time is spent building packages, even though they > almost never change. The tests would run much faster if we shipped > pre-built packages. One way to accomplish that would be to package the > tests separately. Yep, that'd be the way to go IMO. You aren't trying to test dpkg-buildpackage or parts of the package-building toolchain - you're trying to test Lintian, which operates on the results of that. Shunting this part to a one-time operation would be eminently sensible. We do similar in some pkg-gnome packages, for example glib2.0 ships a -tests package that contains "installed tests" which are compiled as part of the package build and then executed during the autopkgtests. Cheers, -- Iain Lane [ i...@orangesquash.org.uk ] Debian Developer [ la...@debian.org ] Ubuntu Developer [ la...@ubuntu.com ] signature.asc Description: PGP signature
Bug#926409: lintian: autopkgtest takes very long to finish
On Thu, Apr 4, 2019 at 12:19 PM Chris Lamb wrote: > > Felix Lechner wrote: > > > About 95% of the time is spent building packages, even though they > > almost never change. The tests would run much faster if we shipped > > pre-built packages. > > Another way to accomplish this could be that we could cache/store them > across autopkgtest runs? IIRC (at least) Gitlab supports caching stuff > like this. > Upon reflection, each test should be packaged separately. That way I no longer have to worry about using chroot to build tests with potentially conflicting build dependencies.
Bug#926409: lintian: autopkgtest takes very long to finish
On Thu, Apr 4, 2019 at 11:24 AM Balint Reczey wrote: > > One criterion that came to my mind is filtering by severity, including > errors for sure, but not pedantic ones. The pedantic setting may become the default for tests, but very little time is spent running Lintian. Things may speed up a little if we run only the checks being tested (using the '-C' option), but that won't make much of a difference in the overall run time of the tests, which is primarily spent building packages.
Bug#926409: lintian: autopkgtest takes very long to finish
Balint Reczey wrote: > One criterion that came to my mind is filtering by severity, including > errors for sure, but not pedantic ones. Mmm, unfortunately I think we would still want to know if, for example, the runtime toolchain changes such that a pedantic tag changes behaviour. :( Felix Lechner wrote: > About 95% of the time is spent building packages, even though they > almost never change. The tests would run much faster if we shipped > pre-built packages. Another way to accomplish this could be that we could cache/store them across autopkgtest runs? IIRC (at least) Gitlab supports caching stuff like this. Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#926409: lintian: autopkgtest takes very long to finish
On Thu, Apr 4, 2019 at 10:42 AM Chris Lamb wrote: > > * I'm not sure *how* we can speed up the tests. I mean, they all >essentially involve building Debian packages with all the usual >debhelper calls, etc. Speeding *this* up is somewhat out-of-scope >of this Lintian wishlist issue, alas. > > However, perhaps Felix has some input here as he has been doing a lot > of work on the test suite recently? About 95% of the time is spent building packages, even though they almost never change. The tests would run much faster if we shipped pre-built packages. One way to accomplish that would be to package the tests separately.
Bug#926409: lintian: autopkgtest takes very long to finish
Hi Chris, On Thu, Apr 4, 2019 at 7:30 PM Chris Lamb wrote: > > tags 926409 + moreinfo > thanks > > Hi Balint, > > > Lintian has a lot of tests which is great for coverage, but maybe some > > of them could be skipped in autopkgtest runs. > > Interesting. I guess I would have three follow-up questions here: > > * On what criterion or criteria could we include or exclude tests >from the autopkgtest runs? Whilst we could skip the unit tests (as >these are "just" Perl that is unlikely to vary) the most >interesting ones to run in terms of detecting regressions in an >real-world environment (the entire point of autopkgtests from my >point of view) would be the tests of the checks themselves and >these likely constitute the vast majority of the total time. One criterion that came to my mind is filtering by severity, including errors for sure, but not pedantic ones. The full suite can run during the build thus we don't loose a lot of coverage. I hoped to rely on Lintian maintainer's judgement about what to omit. > * I'm not sure *how* we can speed up the tests. I mean, they all >essentially involve building Debian packages with all the usual >debhelper calls, etc. Speeding *this* up is somewhat out-of-scope >of this Lintian wishlist issue, alas. A profiling round with perf would point out a few things IMO, but as a start I did a timestamped run here to find slowest tests: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-rbalint-scratch/disco/arm64/l/lintian/20190404_002129_96f0c@/log.gz If there many package builds in the tests just changing the faster compression helps, a lot with little effort (until zstd becomes the default ;-)). > * Why not simply increase Ubuntu's timeout? I would concede this is >not the best use of CI resources, but the trade-off with "human" >time would appear to be worth it here. I agree, and we may increase the timeout, but the running the tests seems to take longer in general than seems reasonable. Cheers, Balint > However, perhaps Felix has some input here as he has been doing a lot > of work on the test suite recently? > > > Best wishes, > > -- > ,''`. > : :' : Chris Lamb > `. `'` la...@debian.org chris-lamb.co.uk >`- -- Balint Reczey Ubuntu & Debian Developer
Bug#926409: lintian: autopkgtest takes very long to finish
tags 926409 + moreinfo thanks Hi Balint, > Lintian has a lot of tests which is great for coverage, but maybe some > of them could be skipped in autopkgtest runs. Interesting. I guess I would have three follow-up questions here: * On what criterion or criteria could we include or exclude tests from the autopkgtest runs? Whilst we could skip the unit tests (as these are "just" Perl that is unlikely to vary) the most interesting ones to run in terms of detecting regressions in an real-world environment (the entire point of autopkgtests from my point of view) would be the tests of the checks themselves and these likely constitute the vast majority of the total time. * I'm not sure *how* we can speed up the tests. I mean, they all essentially involve building Debian packages with all the usual debhelper calls, etc. Speeding *this* up is somewhat out-of-scope of this Lintian wishlist issue, alas. * Why not simply increase Ubuntu's timeout? I would concede this is not the best use of CI resources, but the trade-off with "human" time would appear to be worth it here. However, perhaps Felix has some input here as he has been doing a lot of work on the test suite recently? Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#926409: lintian: autopkgtest takes very long to finish
Package: lintian Version: 2.11.0 Severity: wishlist Hi, Lintian has a lot of tests which is great for coverage, but maybe some of them could be skipped in autopkgtest runs. Currently autopkgtest takes ~1 wall clock hours on Debian's amd64 CI [1], but Ubuntu runs autopkgtest on all architectures and arm64 runs for example take more than 3 hours [2]. Setting the tests to run in parallel helps with the wall clock time, but still this may not be the best use of CI CPU resources or if all tests are needed, maybe lintian itself could be sped up a bit. Cheers, Balint [1] https://ci.debian.net/packages/l/lintian/unstable/amd64/ [2] http://autopkgtest.ubuntu.com/packages/l/lintian/disco/arm64 -- Balint Reczey Ubuntu & Debian Developer
