Re: On (semi-)automated testing and improved workflow of LTS uploads
Hi, On 11/07/2019 15:20, Jonas Meurer wrote: >> Many packages are packaged in Git already (probably on Salsa) and have a >> repo location of their own. With applying GitLab based CI to the >> workflow, the LTS team would add an extra Git repo, just for the LTS >> uploads done by the paid contributors. > Yeah, I agree that this is a downside. An ideal workflow would probably > look like this: > > 1a: for packages on salsa, fork the repo to lts-team/packages/ > 1a.1: if repo doesn't provide a 'jessie' branch, 'gbp import-dsc' the > jessie sources into a new branch > 1b: for packages not on salsa, push the latest package version there > 2: apply the package updating workflow, as discussed above > 3a: file a merge request against the official package repo that asks > to merge back the changes > > 99: whenever support for a LTS release ends, cleanup our team workspace > and remove packages that no longer exist in the next LTS release(s). I have my doubts about enforcing Git. Large packages such as Firefox can take several GB once extracted and are time-consuming enough to handle as-is, without an extra layer of storage, you see what I mean? Beware that packages stored in Git may not use git-build-package. TBH I don't understand much how Salsa-CI is to be used, are will it (re)build massive packages from source, or will we commit pre-built packages? > We could use a dedicated private subgroup and move the working repos > there whenever we need to handle embargoed issues. Yes, embargoed issues are very rare and special-casing them sounds better than making everything private (especially from a transparency PoV). > One advantage of Gitlab/Salsa is that reviewing changes is very > convenient in the Gitlab UI, especially if we used merge requests for > new security uploads and require a second developer to merge them. Note that some issues are purely compilation-related (e.g. building the source package with a more recent Debian release) and won't appear in a Salsa source diff view. debdiff is my authoritative source :) Cheers! Sylvain
[SECURITY] [DLA 1852-1] python3.4 security update
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
Package: python3.4
Version: 3.4.2-1+deb8u5
CVE ID : CVE-2019-9948
The urllib library in Python ships support for a second, not well known
URL scheme for accessing local files ("local_file://"). This scheme can
be used to circumvent protections that try to block local file access
and only block the well-known "file://" schema. This update addresses
the vulnerability by disallowing the "local_file://" URL scheme.
This update also fixes another regresssion introduced in the update
issued as DLA-1835-1 that broke installation of libpython3.4-testsuite.
For Debian 8 "Jessie", this problem has been fixed in version
3.4.2-1+deb8u5.
We recommend that you upgrade your python3.4 packages.
For the detailed security status of python3.4 please refer to
its security tracker page at:
https://security-tracker.debian.org/tracker/python3.4
Further information about Debian LTS security advisories, how to apply
these updates to your system and frequently asked questions can be
found at: https://wiki.debian.org/LTS
- --
Jonas Meurer
-BEGIN PGP SIGNATURE-
iQIyBAEBCgAdFiEELIzSg9Pv30M4kOeDUmLn/0kQSf4FAl0nbTkACgkQUmLn/0kQ
Sf7OrQ/3U0DeYrV0ACOnQFd0I4dVSyJjV8c4WfVvvqMpy413mXVLNBtaZOD+PSRE
0t3jm+MsZBf20zk1wuiqku/Qg5MRLa4ijSu6iFpB2TM81UQ69rhg0b8COAxhorMe
G6eIZlxkLju3Rmx08+n2xLmMYZLw4/z7H23XQg8IhyICeWL8FzjnjBOLQ2xtXMMh
r0Lnmtr1ClfXoqcboeNmdIi8XHs2hHV6sUlJ27oldBjHKB2Mu8hbBSL5nOdrtqPW
6mW3D+0Kx4XJ1dgqkJN2XT2QktperE1uOinssRTjXrVhP2ZFQx9Y1cQkELsljmrK
0Zgd7ME09dX5mGd6Weu/EVBntbG0vFagXmBtTIY8eHcymyvPpb5npuy8PzCdWo7P
/VyPyqFMI7rB4cg7jBX1FEsbPzhO9kIs8mJBemF3xSrF6vF5SNu2FnN6C7OiRUMj
vkq+EWzLTb0WQaqRtVQkZrcYIfdTayVe3GjuxMBahFguZ8OeEcZQKQIUFDltu5Pf
9GPSiakbW8itowsdN2mEIpnsLRcX9cNMDkW0vDolETqyPTenVrmlyeibMo9uQ9MT
IHVDz5Y5rrTl6i7fADaIaG/ewuwHNen+iGxUjB9DXjGpXdOX+6lAMdKmPpIRD/2p
pCOvwjC9jMJ6tczyzYrStTy+9nwb8ncfSE0NAKiVngJsv4GiZg==
=i1aZ
-END PGP SIGNATURE-
Accepted python3.4 3.4.2-1+deb8u5 (source all amd64) into oldoldstable
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Format: 1.8 Date: Thu, 11 Jul 2019 12:58:03 +0200 Source: python3.4 Binary: python3.4 python3.4-venv libpython3.4-stdlib python3.4-minimal libpython3.4-minimal libpython3.4 python3.4-examples python3.4-dev libpython3.4-dev libpython3.4-testsuite idle-python3.4 python3.4-doc python3.4-dbg libpython3.4-dbg Architecture: source all amd64 Version: 3.4.2-1+deb8u5 Distribution: jessie-security Urgency: high Maintainer: Matthias Klose Changed-By: Jonas Meurer Description: idle-python3.4 - IDE for Python (v3.4) using Tkinter libpython3.4 - Shared Python runtime library (version 3.4) libpython3.4-dbg - Debug Build of the Python Interpreter (version 3.4) libpython3.4-dev - Header files and a static library for Python (v3.4) libpython3.4-minimal - Minimal subset of the Python language (version 3.4) libpython3.4-stdlib - Interactive high-level object-oriented language (standard library libpython3.4-testsuite - Testsuite for the Python standard library (v3.4) python3.4 - Interactive high-level object-oriented language (version 3.4) python3.4-dbg - Debug Build of the Python Interpreter (version 3.4) python3.4-dev - Header files and a static library for Python (v3.4) python3.4-doc - Documentation for the high-level object-oriented language Python python3.4-examples - Examples for the Python language (v3.4) python3.4-minimal - Minimal subset of the Python language (version 3.4) python3.4-venv - Interactive high-level object-oriented language (pyvenv binary, v Changes: python3.4 (3.4.2-1+deb8u5) jessie-security; urgency=high . * Non-maintainer upload by the LTS Security Team. * CVE-2019-9948: Make urllib reject local_file:// scheme * Fix more regressions in the patch for CVE-2019-9740/CVE-2019-9947 that broke installation of libpython3.4-testsuite and the test_urllib pytests. Checksums-Sha1: 6e30d780e8eb2fc432a7c26e66f884955b015825 3246 python3.4_3.4.2-1+deb8u5.dsc 12654fd42b7c9d506052e1482720bf2f0005e49b 365377 python3.4_3.4.2-1+deb8u5.diff.gz f25e4f980e2ee54de4c6196fb06860159af0b09e 392362 python3.4-examples_3.4.2-1+deb8u5_all.deb 146d0263ea2b010a8c855629ed622bd9a6f15405 2412720 libpython3.4-testsuite_3.4.2-1+deb8u5_all.deb ea35fa6943e851a17fa7d92fc258932201260727 82898 idle-python3.4_3.4.2-1+deb8u5_all.deb 744cd7d833a2468dd9fe676cd8378a91e557aa99 5163094 python3.4-doc_3.4.2-1+deb8u5_all.deb ce48e992686f04a98fc716cce7f0ddd024ef1fb4 205078 python3.4_3.4.2-1+deb8u5_amd64.deb afc67a158859bdca73320f31d0f810d69d2a7540 1437750 python3.4-venv_3.4.2-1+deb8u5_amd64.deb e66cd3d07f50cb80c625c7695517b844f5c21974 2089292 libpython3.4-stdlib_3.4.2-1+deb8u5_amd64.deb fd33a2d39404a4754bfdd929bdacfd1748cd1737 1645090 python3.4-minimal_3.4.2-1+deb8u5_amd64.deb f12c45775e2408f60769a288ff69afebba3857dc 493698 libpython3.4-minimal_3.4.2-1+deb8u5_amd64.deb 2cfca3e922fc48278bb961b7ddf5ddf747be73fe 1313626 libpython3.4_3.4.2-1+deb8u5_amd64.deb cba243484f46559d1961ebc4359fb8abf2c61db4 412920 python3.4-dev_3.4.2-1+deb8u5_amd64.deb 1e667b1fe64df43492d61dac93938c43af5cda16 39853008 libpython3.4-dev_3.4.2-1+deb8u5_amd64.deb a1afa70ab3d6cb2cc12aa157ff74670814c924c6 7877748 python3.4-dbg_3.4.2-1+deb8u5_amd64.deb 378298939ee312e2423b0bb0d1b52d5e021d4aaa 5268702 libpython3.4-dbg_3.4.2-1+deb8u5_amd64.deb Checksums-Sha256: 5dd96c210f1740f7c0dc582e5f37fc6eaa94dedd75a35da88f5e644c645decbf 3246 python3.4_3.4.2-1+deb8u5.dsc a12f38d5bfe28e96db57362710b01587ac28e15b02048db37cdc8010d275577e 365377 python3.4_3.4.2-1+deb8u5.diff.gz 57021dfed9dbf58507472499e335bfe1013f2322aab4e00a469b04f0bfaf4fe7 392362 python3.4-examples_3.4.2-1+deb8u5_all.deb 92342ca564bee958636000e367fe993255fd4cf86cc2addf9e6f9af9c9979f72 2412720 libpython3.4-testsuite_3.4.2-1+deb8u5_all.deb 80d9868011a6c4bf554fe71424bf7f848d62360194014a6a685c11d709ff44e4 82898 idle-python3.4_3.4.2-1+deb8u5_all.deb 576892682eb36c8370160b2c21bad81f1e22cf44180a1913522b1cedebe6838d 5163094 python3.4-doc_3.4.2-1+deb8u5_all.deb ed6e9bf48096d6ca78b87b432b7669c9901c4b163721642467c37f86b546cdab 205078 python3.4_3.4.2-1+deb8u5_amd64.deb 89aebdfbb3be8421bc310102deb881698c6e0bac7becf32e214b3522c33ac7f8 1437750 python3.4-venv_3.4.2-1+deb8u5_amd64.deb 79575c63bd4eb86d2a4ee4c1ef15db4c31a0e3313ecebe293d62b9b51bcef7cd 2089292 libpython3.4-stdlib_3.4.2-1+deb8u5_amd64.deb f3cd11218518e5883b6d8af09e27ff26be848c835026131c52188edb592eed9f 1645090 python3.4-minimal_3.4.2-1+deb8u5_amd64.deb ddfb69166ab135a41c63645165185544f2eaa205c577ddefbe6aa1bb1c1c3deb 493698 libpython3.4-minimal_3.4.2-1+deb8u5_amd64.deb 5d00c24279724ef5ec18cb1647170e9277ada87df05a803eb844dc7f5375a7ed 1313626 libpython3.4_3.4.2-1+deb8u5_amd64.deb 09d5eda5ea2126ac9240b18a36a6e32becd0ed5ddaeeb0760a46baf820cf2391 412920 python3.4-dev_3.4.2-1+deb8u5_amd64.deb ee73dc8c642c62add13b37145bfc6ccd68fa23ef0cc23e269f656cad5b2430ee 39853008 libpython3.4-dev_3.4.2-1+deb8u5_amd64.deb
Re: On (semi-)automated testing and improved workflow of LTS uploads
Hello, Mike Gabriel: >> In the internal discussions, the following vision for an improved upload >> workflow arose: >> >> 1. Upload packages targeted at LTS suites to some dedicated place for >> automated testing > >> 2. Run automatic tests (piuparts, autopkgtests, lintian?, ...) > > Maybe, probably not lintian. As the package maintainer, at the time > (old)oldstable (meaning the current Debian LTS) turned from testing to > stable, might not have had their packages in shape lintian-wise. Also > only using an old lintian would be appropriate. A recent lintian on old > packages just creates too much noise (which we won't fix anyway). The Salsa-CI runs lintian from the target suite against the packages. While I agree that we should not put any effort into fixing longstanding lintian warnings, running lintian per default and providing the output for review seems sensible to me. >> 4. (Optionally?) demand acknowledgement by a second (different) LTS >> developer > > Although demanding a second ACK adds an extra delay to our workflow, I > sense that such a second pair of eyes peering at security patches might > greatly improve the quality of the LTS work. Even if we don't come up > with some auto-test engine, we should consider "peer-"reviewed uploads. I agree. >> So far, two implementation approaches have been discussed: >> >> */ Build an own service that provides a dedicated upload queue (e.g. >> 'lts-proposed-updates') which accepts uploads targeted at LTS suites, >> and processes the uploaded packages according to the workflow >> described above. >> >> */ Use Salsa-CI and their pipeline[2] for as much of the above proposal >> as possible. >> >> What's your thoughts on this? Do you think that we could implement >> most/all of the desired workflow using Salsa-CI/Gitlab-CI? Or would it >> be better to build it entirely independently of Salsa - e.g. implement >> it in dak? > > Personally, I think that using Salsa for this, adds an extra layer of > complexity to the uploading workflow, because we have to pump all > packages that we want to fix in LTS through GitLab. We need a place to upload them to (for tests and reviews), anyway. So what's the advantage of having a dedicated service (and use resources there) compared to using some of Salsa's resources for it. To be honest, I don't consider the expected extra CPU cylces for Salsa that high. The amount of LTS security uploads by us (and if the Security Team would use a similar solution: even the amount of security uploads by us and them) isn't that high compared to the number of Debian packages hosted on Salsa already. > Many packages are packaged in Git already (probably on Salsa) and have a > repo location of their own. With applying GitLab based CI to the > workflow, the LTS team would add an extra Git repo, just for the LTS > uploads done by the paid contributors. Yeah, I agree that this is a downside. An ideal workflow would probably look like this: 1a: for packages on salsa, fork the repo to lts-team/packages/ 1a.1: if repo doesn't provide a 'jessie' branch, 'gbp import-dsc' the jessie sources into a new branch 1b: for packages not on salsa, push the latest package version there 2: apply the package updating workflow, as discussed above 3a: file a merge request against the official package repo that asks to merge back the changes 99: whenever support for a LTS release ends, cleanup our team workspace and remove packages that no longer exist in the next LTS release(s). > Some package uploads may even be > embargoed, so generically, the LTS-team namespace on Salsa needs to be > private (which excludes other contributors, also the usual package > maintainers/uploaders, by default). We could use a dedicated private subgroup and move the working repos there whenever we need to handle embargoed issues. > As our intention is to operate on packages (not on upstream code in > Git), so I'd suggest deploying/extending some sort of > setup/infrastructure that utilizes Debian means for auto-testing LTS > package upload candidates. And I really love the idea of a review > workflow for package uploads. One advantage of Gitlab/Salsa is that reviewing changes is very convenient in the Gitlab UI, especially if we used merge requests for new security uploads and require a second developer to merge them. Cheers jonas signature.asc Description: OpenPGP digital signature
Re: On (semi-)automated testing and improved workflow of LTS uploads
Hi, On Thu, Jul 11, 2019 at 11:15:34AM +, Mike Gabriel wrote: [..snip..] > Personally, I think that using Salsa for this, adds an extra layer of > complexity to the uploading workflow, because we have to pump all packages > that we want to fix in LTS through GitLab. On the plus side of salsa/gitlab is: - already existing account management (including groups) - being able to leverage the work of the gitlab ci project - already existing commenting system so all information would be in one place It would basically allow to start right away, worst case an additional worker would need to be added. Cheers, -- Guido
Re: On (semi-)automated testing and improved workflow of LTS uploads
Hi Jonas, hi all,
thanks for summarizing the discussion we had on the non-public paid
LTS contributors' "mailing list".
On Di 09 Jul 2019 16:21:47 CEST, Jonas Meurer wrote:
Hello,
Some LTS members recently started discussing options for better
(semi-)automated testing of LTS uploads and an improved upload workflow.
I'll try to summarize the discussion in order to bring it to this public
mailinglist. [1]
The motivation for an improved package upload workflow basically is to
lower the risk of (simple) regressions and improve the overall quality
of LTS security uploads. The way to get there is to run (semi-)automated
tests against packages before uploading them to ${LTS}-security and
(optionally) enforce a second review+acknowledgement by another LTS
developer.
In the internal discussions, the following vision for an improved upload
workflow arose:
1. Upload packages targeted at LTS suites to some dedicated place for
automated testing
Yep.
2. Run automatic tests (piuparts, autopkgtests, lintian?, ...)
Maybe, probably not lintian. As the package maintainer, at the time
(old)oldstable (meaning the current Debian LTS) turned from testing to
stable, might not have had their packages in shape lintian-wise. Also
only using an old lintian would be appropriate. A recent lintian on
old packages just creates too much noise (which we won't fix anyway).
3. If tests passed, publish the packages somewhere to do manual
testing (and reviews)
If either step (2. or 3.) fails, we go back to 1.
4. (Optionally?) demand acknowledgement by a second (different) LTS
developer
Although demanding a second ACK adds an extra delay to our workflow, I
sense that such a second pair of eyes peering at security patches
might greatly improve the quality of the LTS work. Even if we don't
come up with some auto-test engine, we should consider "peer-"reviewed
uploads.
5. Automatically upload packages that got uploaded, passed tests and got
second acknowledgement to the targeted LTS upload queue
yep
While that would be very nice to have, it's probably a long way to go
until we have such infrastructure.
There seems to be some agreement that the first step would be to run
(semi-)automated tests (e.g. piuparts and autopkgtests) against the
packages before uploading them to ${LTS}-security, i.e. point 2 of the
list above.
So far, two implementation approaches have been discussed:
*/ Build an own service that provides a dedicated upload queue (e.g.
'lts-proposed-updates') which accepts uploads targeted at LTS suites,
and processes the uploaded packages according to the workflow
described above.
*/ Use Salsa-CI and their pipeline[2] for as much of the above proposal
as possible.
What's your thoughts on this? Do you think that we could implement
most/all of the desired workflow using Salsa-CI/Gitlab-CI? Or would it
be better to build it entirely independently of Salsa - e.g. implement
it in dak?
Personally, I think that using Salsa for this, adds an extra layer of
complexity to the uploading workflow, because we have to pump all
packages that we want to fix in LTS through GitLab.
Many packages are packaged in Git already (probably on Salsa) and have
a repo location of their own. With applying GitLab based CI to the
workflow, the LTS team would add an extra Git repo, just for the LTS
uploads done by the paid contributors. Some package uploads may even
be embargoed, so generically, the LTS-team namespace on Salsa needs to
be private (which excludes other contributors, also the usual package
maintainers/uploaders, by default).
As our intention is to operate on packages (not on upstream code in
Git), so I'd suggest deploying/extending some sort of
setup/infrastructure that utilizes Debian means for auto-testing LTS
package upload candidates. And I really love the idea of a review
workflow for package uploads.
And, open question: Would such a workflow be an option for the
security team's workflow, too?
Greets,
Mike
--
DAS-NETZWERKTEAM
c\o Technik- und Ökologiezentrum Eckernförde
Mike Gabriel, Marienthaler str. 17, 24340 Eckernförde
mobile: +49 (1520) 1976 148
landline: +49 (4351) 486 14 27
GnuPG Fingerprint: 9BFB AEE8 6C0A A5FF BF22 0782 9AF4 6B30 2577 1B31
mail: mike.gabr...@das-netzwerkteam.de, http://das-netzwerkteam.de
pgpL6ZV17YY4O.pgp
Description: Digitale PGP-Signatur
