Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi again, On 19-07-30 21:39:00, Chris Lamb wrote: > I'm not going to implement this bit just yet until I solve the more > deeper problem of it, well, not actually caching anything at the > moment. See previous messages for more background. As far as I can tell, the cache problem was caused by a breaking upstream change. This got fixed [1]. Probably, didn't test, this "should work" again. Cheers, Georg [1] https://salsa.debian.org/salsa/salsa-ansible/commit/c6404e6b31267a8f9c46a2eb7838b829e08a451d
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Felix Lechner wrote: > > caching the result of the former if the tests and some other key files > > have not changed. > > Will it also force an update of the test packages when the build chain > has changed? It may expose Lintian bugs like #932339. Whilst my *current* implementation does not rebuild if a build-depends version changes, we would only need to include these package version in the hash key. It would then (correctly) cache miss in that instance and rebuild the test packages. It's a shame we would not have a .buildinfo file at this point... I'm not going to implement this bit just yet until I solve the more deeper problem of it, well, not actually caching anything at the moment. See previous messages for more background. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi Chris, On Sat, Jul 20, 2019 at 1:33 PM Chris Lamb wrote: > > I've been hacking on this on a Salsa-local fork of Lintian that splits > the generation of the test packages and the testing itself, crucially > caching the result of the former if the tests and some other key files > have not changed. Will it also force an update of the test packages when the build chain has changed? It may expose Lintian bugs like #932339. Kind regards, Felix
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi Georg, > An alternative might be to use artifacts [1], which work as expected on > Salsa. Thanks. Unfortunately, I did investigate using artifacts at first and whilst they do indeed work on Salsa they are for a different use-case that does not fit our "re-use from a previous build" requirement (they re-use files between *parts* of a build). Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi, I'm not sure if the following is of help, as I didn't saw the involved code, but: On 19-07-20 17:28:56, Chris Lamb wrote: > I have something working except that I am running into a blocker > whereby the GitLab cache is not seen in subsequent runs. > > For example, the cache is being stored at the end of the test package > build stage: > > Creating cache default-2... > .cache/: found 2 matching files > No URL provided, cache will be not uploaded to shared cache server. Cache > will be stored only locally. > Created cache > > … but it is not seen when you re-run that same pipeline: > > Checking cache for default-2... > No URL provided, cache will not be downloaded from shared cache server. > Instead a local version of cache will be extracted. > Successfully extracted cache > > $ ls -l .cache || true > ls: cannot access '.cache': No such file or directory > > I will try and hunt down a Salsa admin during DebConf. This might be > due to the use of shared runners and us not using a centralised cache. An alternative might be to use artifacts [1], which work as expected on Salsa. Cheers, Georg [1] https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html signature.asc Description: PGP signature
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi On Sun, Jul 21, 2019 at 6:28 AM Chris Lamb wrote: > > I am very much disposed towards simple and braindead obvious in things > like this, especially when they hit 90% of the use-case. And then we > can get back to other stuff... I realized we are talking about related but different things. Please look out for a merge request on the topics that go beyond the scope of this bug.
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi, On Sun, Jul 21, 2019 at 6:28 AM Chris Lamb wrote: > > Felix Lechner wrote: > > $ find t/ debian/ -type f | sort | xargs sha1sum | sha1sum | cut -d' ' -f1 That's the same approach I suggested, except I would save the result of find t/ debian/ -type f | sort | xargs sha1sum and do so on a per-test basis. Then t/bin/runtest can rebuild the test packages adaptively. Please remember, contributors currently cannot run tests with '--onlyrun=check:XXX' when they add or change a tag, except after the time-consuming 'rm -rf debian/test-out; t/bin/build-test-packages'. (It was much faster until I separated the building of test packages.) The adaptive rebuilding would be super helpful even without caching on Salsa. > .. throwing the entire set of build packages away in the case of cache > miss. I did not suggest by-package caching on Salsa. I would write the test packages in to a different directory tree in d/test-out and only cache that. For example, I would write the filled templates to d/test-out/src. Only this tree needs to be hashed. During building, I would then write the test packages to d/test-out/built. Only that tree needs to be cached. > Anyway, without the Salsa caching working, all of this is moot. > One small step for a man; one big step for mankind. :)
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Felix Lechner wrote: > > I've been hacking on this on a Salsa-local fork of Lintian that splits > > the generation of the test packages and the testing itself, crucially > > caching the result of the former if the tests and some other key files > > have not changed. > > How do you tell when files have changed? I would like to use an MD5 > manifest of all files generated after templating. Computing the checksum of the output not only seems unnecessarily complicated but prone to error in the details. I am hashing the entire "input" roughly as follows: $ find t/ debian/ -type f | sort | xargs sha1sum | sha1sum | cut -d' ' -f1 .. throwing the entire set of build packages away in the case of cache miss. Of course there are more refined approaches but, in lieu of that, if you skim over the pipeline job history this will save the testsuite package builds almost all of the time. This also has the advantage of being obvious how it works and captures any type of change, including the missing files that you specifically highlight, but possibly even others we have not yet thought of. I am very much disposed towards simple and braindead obvious in things like this, especially when they hit 90% of the use-case. And then we can get back to other stuff... Anyway, without the Salsa caching working, all of this is moot. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
On Sat, Jul 20, 2019 at 2:02 PM Felix Lechner wrote: > > This will require a splitting of all 'desc' files > in the test directories. Actually, since the values from 'desc' are incorporated in the completed templates, all 'desc' files can be excluded from the manifest as well. No splitting is necessary. Kind regards, Felix
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi Chris, On Sat, Jul 20, 2019 at 1:33 PM Chris Lamb wrote: > > I've been hacking on this on a Salsa-local fork of Lintian that splits > the generation of the test packages and the testing itself, crucially > caching the result of the former if the tests and some other key files > have not changed. How do you tell when files have changed? I would like to use an MD5 manifest of all files generated after templating. That will also catch changes in defaults and templates---even missing files. Eventually, the expected tags (or output, if applicable) should be excluded from that manifest, as should those portions of each 'desc' file that apply only to lintian invocation (i.e. Profile, Output-Format) or runner behavior (i.e. Match-Strategy, Test-Architectures). This will require a splitting of all 'desc' files in the test directories. Also, the specifications for test packages should then be moved into dedicated directories in each test, named '$test/test-object' or similar. Kind regards, Felix
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Hi all, > lintian: use GitLab caching of test packages to speed up test suite CI I've been hacking on this on a Salsa-local fork of Lintian that splits the generation of the test packages and the testing itself, crucially caching the result of the former if the tests and some other key files have not changed. I have something working except that I am running into a blocker whereby the GitLab cache is not seen in subsequent runs. For example, the cache is being stored at the end of the test package build stage: Creating cache default-2... .cache/: found 2 matching files No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. Created cache … but it is not seen when you re-run that same pipeline: Checking cache for default-2... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache $ ls -l .cache || true ls: cannot access '.cache': No such file or directory I will try and hunt down a Salsa admin during DebConf. This might be due to the use of shared runners and us not using a centralised cache. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
[changing subject to match updated bug title] Hi Felix, > On Mon, Jun 17, 2019 at 4:03 PM Chris Lamb wrote: > We used filesystem timestamps for a while, but the standard resolution > (1 sec) was not granular enough. AFAIR, we now generate everything > every time. By "filesystem timestamps" here are you referring to comparing the timestamp of each generated test package and its source? If so, I am unclear why one second was not enough to determine whether a *developer* had changed something under "t/". > We could probably skip the generation of test packages if they are > already present and nothing in t/ has changed. Indeed, but that is (roughly) what this entire bug report is about. > > But are we barking up the wrong tree here and what we need to do is > > use different GitLab CI stage altogether and pass "artifacts" around > > instead? > > > > https://docs.gitlab.com/ee/ci/caching/index.html#cache-vs-artifacts > > Artifacts may work, but uploading them separately without a dependency > scheme seems to invite other problems. […] Hm, I think you may have been accidentally misled about how the Gitlab CI build stages — there would be no upload whatsoever. I am therefore unsure how to effectively and productively respond to your remarks, alas. :( Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Maybe prove parallel execution can help here (--jobs N)
Bug#930487: lintian: use GitLab caching of test packages to speed up test suite CI
Le 13/06/2019 à 18:34, Xavier a écrit : > Maybe prove parallel execution can help here (--jobs N) Sorry, parallel execution is already set
