Re: Package testing with Salsa CI for new packages
On Saturday, 19 August 2023 20:32:59 CEST Carles Pina i Estany wrote: > > Quick answer for now... And a very quick reply from me... :-) [...] > I don't know if a .asc files are allowed. Actually, from my > .bash_history: > --- > wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg > --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --- > > (I was following instructions, I didn't try leaving the .asc file there) They are allowed in recent versions of apt, as confirmed by the man page. [...] > You could try using: > --- > variables: > SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh' > --- > (and write and ship the ci/setup-the-repo.sh in the repo, do whatever > you need there) This could be very useful. > Or, probably even better but less flexible: > --- > variables: > SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE > COMPONENT' --- > (I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS. > I haven't used add-apt-source, I don't know what happens with the gpg > keys... but you could use [trusted=yes] :-| ) > > For the MIRROR you have salsa variables that might help if you need to > specify the pipeline. So could this. > > So, what I now need to do is to find out how I can make the new packages > > available to autopkgtest specifically. > > We will get there! (one way or another) Well, I have engineered something very inelegant, revealed in full here: https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml It defines a package list file and uses wget to obtain each repository's public key. Fortunately, wget works at the "script" level of the jobs involved, whereas it didn't work at what must have been the global "before_script" level (suggested in the documentation) and needed to be installed first. To make autopkgtest and piuparts find the packages, the repository details are then copied into the appropriate location in each case. However, since such customisation is not readily incorporated into the existing job descriptions, I have had to duplicate the existing job description code and insert additional statements in the script sections. As a consequence of these changes, the extra packages are found and installed within the environments concerned. This is enough to make the piuparts job succeed, but autopkgtest still fails. Fortunately, that may be due only to problems with the actual tests: I think the upstream maintainers may have made some errors. [...] > hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in > salsa-ci/pipeline last year because I was trying to do a similar thing > to what you are doing (I had to pin a package from backports). > > (I'm just a user of salsa-ci/pipeline, not a member of the team neither > I can speak for them!) If this can simplify what I've done, which is really quite horrible, then I will adopt it instead. The way that the artefacts of the dependencies are bound up in specifically numbered jobs is also particularly unfortunate. Of course, if the packages enter the Debian archives, all of this extra effort will be made largely redundant, anyway, even if coordinated testing of new package versions would still benefit from such effort. [...] > I don't know if an FAQ, conventions, best practises or something might > help... I think that the documentation could be reworked, but that is largely true for a lot of the materials available for Debian. And since some of that documentation resides on the Debian Wiki, that is another reason why I am trying to move this particular packaging exercise forwards. Thanks once again! Paul
Re: Package testing with Salsa CI for new packages
Hi, Quick answer for now... On 19 Aug 2023 at 19:04:09, Paul Boddie wrote: > On Friday, 18 August 2023 19:54:55 CEST Carles Pina i Estany wrote: > > > > Ha! I wasn't aware of the aptly option > > (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-> > > apt-repository and SALSA_CI_DISABLE_APTLY=0). > > I think that I might have re-invented the wheel in a tiny part of > > Debusine CI/CD. > > It is certainly a way of propagating packages to those that might need > them. However, the instructions indicating how a package might access > these dependencies appear to be deficient. I haven't used aptly on salsa but on Monday I want to give it a try, to see if I can replace my solution... > It does not appear to be sufficient to merely specify the dependency > package repositories and mark them as trusted. Doing that will just > cause the repositories to be regarded as ignored and the GPG keys > signalled as unrecognised. > > So, the GPG keys need to be obtained. This is a hassle because something like > wget is needed to do that, and then apt has to be persuaded not to fail in an > opaque way. So the modified recipe is something like this: > > before_script: > - apt-get update > - NON_INTERACTIVE=1 apt-get install -y wget > - echo "deb https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/ > artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list > ... > - wget -q -O /etc/apt/trusted.gpg.d/emeraldtree.asc https:// > salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly/ > public-key.asc I'm ensure that this is correct. I have: - deb [arch=amd64 signed-by=/usr/share/keyrings/oracle-virtualbox-2016.gpg] https://download.virtualbox.org/virtualbox/debian bookworm contrib - So I thought that the "signed-by" was needed; and that the files that I have there are binary files (not .asc) of this type: carles@pinux:~$ file /etc/apt/trusted.gpg.d/skype.gpg /etc/apt/trusted.gpg.d/skype.gpg: OpenPGP Public Key Version 4, Created Wed Jun 22 09:36:35 2016, RSA (Encrypt or Sign, 2048 bits); User ID; Signature; OpenPGP Certificate carles@pinux:~$ I don't know if a .asc files are allowed. Actually, from my .bash_history: --- wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --- (I was following instructions, I didn't try leaving the .asc file there) > ... > - apt-get update > > This seems to make the various jobs happy, but the one that I was most > concerned with remains unhappy! I don't actually need the dependencies > for anything other than autopkgtest, but that job employs its own > environment where the above recipe has no effect. You could try using: --- variables: SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh' --- (and write and ship the ci/setup-the-repo.sh in the repo, do whatever you need there) Or, probably even better but less flexible: --- variables: SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE COMPONENT' --- (I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS. I haven't used add-apt-source, I don't know what happens with the gpg keys... but you could use [trusted=yes] :-| ) For the MIRROR you have salsa variables that might help if you need to specify the pipeline. > So, what I now need to do is to find out how I can make the new packages > available to autopkgtest specifically. We will get there! (one way or another) > > I will point out at some things that might safe some time to you > > (great) or not (ignore! :-) ): > > > > debusine's .gitlab-ci.yml: > > https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml > > This is definitely useful for examples of the CI definition syntax and how > hidden jobs can be used to provide additional script fragments. > > [...] > > > The job autopkgtest, via debci command, runs autopkgtest: > > https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65 > > The difficult thing appears to be the configuration of this testing > environment. One approach might involve overriding the existing job > definitions to incorporate the injection of new packages. hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in salsa-ci/pipeline last year because I was trying to do a similar thing to what you are doing (I had to pin a package from backports). (I'm just a user of salsa-ci/pipeline, not a member of the team neither I can speak for them!) > [...] > > > It's possible for sure. Other people in this list might come up with a > > different idea. I don't have almost any experience with Debian > > packaging, but I have some experience on the salsa CI. So I might be > > giving you solutions that might be sub-optimal! :-) > > I think that you are giving me some good ideas at least. I do feel that I am > encountering problems that must have been
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 19:54:55 CEST Carles Pina i Estany wrote: > > Ha! I wasn't aware of the aptly option > (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-> > apt-repository and SALSA_CI_DISABLE_APTLY=0). > I think that I might have re-invented the wheel in a tiny part of > Debusine CI/CD. It is certainly a way of propagating packages to those that might need them. However, the instructions indicating how a package might access these dependencies appear to be deficient. It does not appear to be sufficient to merely specify the dependency package repositories and mark them as trusted. Doing that will just cause the repositories to be regarded as ignored and the GPG keys signalled as unrecognised. So, the GPG keys need to be obtained. This is a hassle because something like wget is needed to do that, and then apt has to be persuaded not to fail in an opaque way. So the modified recipe is something like this: before_script: - apt-get update - NON_INTERACTIVE=1 apt-get install -y wget - echo "deb https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/ artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list ... - wget -q -O /etc/apt/trusted.gpg.d/emeraldtree.asc https:// salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly/ public-key.asc ... - apt-get update This seems to make the various jobs happy, but the one that I was most concerned with remains unhappy! I don't actually need the dependencies for anything other than autopkgtest, but that job employs its own environment where the above recipe has no effect. So, what I now need to do is to find out how I can make the new packages available to autopkgtest specifically. > I will point out at some things that might safe some time to you > (great) or not (ignore! :-) ): > > debusine's .gitlab-ci.yml: > https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml This is definitely useful for examples of the CI definition syntax and how hidden jobs can be used to provide additional script fragments. [...] > The job autopkgtest, via debci command, runs autopkgtest: > https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65 The difficult thing appears to be the configuration of this testing environment. One approach might involve overriding the existing job definitions to incorporate the injection of new packages. [...] > It's possible for sure. Other people in this list might come up with a > different idea. I don't have almost any experience with Debian > packaging, but I have some experience on the salsa CI. So I might be > giving you solutions that might be sub-optimal! :-) I think that you are giving me some good ideas at least. I do feel that I am encountering problems that must have been encountered many times before, but I could imagine that people do not use these tools in this way or have different expectations of what they are able to accomplish. As for Debian's tooling, I consider it rather chaotic. When I previously did packaging, things like pbuilder were still in use, being slowly pushed aside by sbuild and probably other things that are now forgotten or obsolete. There always seems to be new things coming along, new mixtures of technologies, and the potential for more confusion and uncertainty. Paul
Bug#1050081: ITP: python-parsl -- Parallel Scripting Library
Package: wnpp Severity: wishlist Owner: Étienne Mollier X-Debbugs-Cc: debian-de...@lists.debian.org X-Debbugs-Cc: debian-python@lists.debian.org * Package name: python-parsl Version : 2023.08.14 Upstream Contact: Yadu Nand Babuji , et al. * URL : https://github.com/Parsl/parsl * License : Apache 2.0 Programming Lang: Python Description : Parallel Scripting Library Parsl extends parallelism in Python beyond a single computer. . One can use Parsl just like Python's parallel executors but across multiple cores and nodes. However, the real power of Parsl is in expressing multi-step workflows of functions. Parsl lets one chain functions together and will launch each function as inputs and computing resources are available. This package is a new dependency for upgrading the qiime2 distribution on Debian Med radar. Since the module parsl itself is a bit more generic than medical field, I intend to put the package under the Debian Python team umbrella. A stub of packaging is available on Salsa[1]. [1]: https://salsa.debian.org/python-team/packages/python-parsl/ Have a nice day, :) -- .''`. Étienne Mollier : :' : gpg: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da `. `' sent from /dev/pts/2, please excuse my verbosity `- signature.asc Description: PGP signature
