Re: https://tracker.debian.org/pkg/dballe
On Sat, 2020-01-18 at 08:25 -0500, Sam Hartman wrote: > I don't care about the mechanism. > What I care is that we not go through a period where invoking the > mechanism involves adding a round trip with ftpmaster, with waiting for > an upload to be accepted, or with the release team, or otherwise delay > the process. The proposed mechanism does not involve anyone other than the maintainer and doesn't substantially change existing workflows. The only difference is that for bootstrap builds instead of doing a source-and-binary upload, they do a source-only upload like they do for non-bootstrap builds and then afterwards a binary-only upload (which would normally be done by the buildds for non-bootstrap builds). Personally I don't think that is a significant burden, but I assume a .changes file option could be added for folks who think it is an issue. I won't be doing any work on this and this will be my last email on it. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: https://tracker.debian.org/pkg/dballe
On Sat, Jan 18, 2020 at 08:40:29AM +0800, Paul Wise wrote: > On Fri, 2020-01-17 at 03:45 -0500, Sam Hartman wrote: > > Bootstrap uploads of compilers etc are actually more common than I > > thought before I started following debian-release. > The important part of my statement is that they are special, rather > than that they are rare. this. and on top of this, these uploads only need to be done by a handful of people, not all >1000 uploaders we have in the keyring. > > They are common enough that requiring interactions from ftpmaster and/or > > release team would make being a developer of such packages suck > > significantly more. > Hmm, OK. I can't recall seeing any recently on debian-release. in the last half year I do recall rustc... -- cheers, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: https://tracker.debian.org/pkg/dballe
> "Paul" == Paul Wise writes: >> If we throw away binaries by default, I do believe we need a mechanism >> for maintainers to signal that this is a bootstrap upload. Paul> My proposed mechanism is that when the buildds cannot do the job due to Paul> need for a bootstrap process, the maintainer should do a binary-only Paul> build (using whatever ugly hacks are needed), dak should import that Paul> into a special part of the archive for bootstrapping packages that need Paul> that and then tell the buildds to do a new non-bootstrap build but Paul> using the bootstrap archive. I don't care about the mechanism. What I care is that we not go through a period where invoking the mechanism involves adding a round trip with ftpmaster, with waiting for an upload to be accepted, or with the release team, or otherwise delay the process. We've heard again and again from maintainers that the more they can accomplish within a single unit of attention span, the better. Every time you have to come back later after some process has done its thing, it makes maintaining stuff more frustrating, less rewarding, and more difficult. Many have argued that the current dance with new--where you have to upload binaries for new, but then later upload without binaries for testing--is one step too far in that direction. If those people are right, we should have waited until we could throw away binaries from new before requiring built-on-buildd for testing. I understand the politics are complex and that the release team's decision may have given motivation to push for patches to throw away binaries from new. Based on the discussion of maintainer motivations, I think going through a similar period where bootstrap uploads got harder--even if it eventually got fixed--would be a mistake and would not value the time of our maintainers. So, in order to value the time of our maintainers, I think a change to throw away binaries from uploads that don't hit new needs to block on a way of handling bootstrap builds. (Ideally that would be part of a patch to throw away binaries from uploads to new, but the trade offs are more complex for that situation.)
Re: https://tracker.debian.org/pkg/dballe
On Fri, Jan 17, 2020 at 6:18 PM Richard Laager wrote: > If I'm following correctly: The packager would use rustcc >= y+3 (in > practice, likely rustcc y+5) to locally build rustcc y+5 and then do a > binary upload. But if dak (or whatever, I'm not so familiar with the > server side here) throws away the binary, then we're sunk. So there > needs to be a way to note that the binary needs to be kept. > > Assuming I'm on the right track, here are a couple of questions: Right. > In such a case, would the source package have a Build-Depends of >= y+3? > It seems like it would. Presumably, but it is definitely possible the maintainer wouldn't know or wouldn't record which version was needed. > Could that be the way to indicate a bootstrap upload? In other words, if > the package has a Build-Depends on one of the binary packages it > produces that is _not_ satisfied in the suite it's being uploaded to but > is satisfied by this upload, then it's a bootstrap upload, and the > binaries should be kept. This would avoiding adding a new field for this > and would ensure the Build-Depends are set correctly in bootstrapping > scenarios. At this time the workflow is like this: ftp-master.d.o accepts or rejects the source/binaries, then it passes the source to buildd.d.o, which checks for build-dep installability and passes the source to the buildds. I think for your proposal to work the build-dep installability checks would also have to move to ftp-master.d.o. > Regardless of how the "keep the binaries" flag is implemented... should > the uploaded binary be published, or should the package be rebuilt? I > think I'd argue for the latter. The uploaded binary package should be > installed on the buildd and then the package should be rebuilt from > source, with _that_ result being put into the archive. This doesn't > protect against the trojaned-compiler problem, but it does at least > ensure that the package builds from source. I definitely agree that all bootstrap binaries, whether built automatically by the buildds (as proposed by Josch) or hacky maintainer-built binaries, should get rebuilt on the buildds. We already did that in the past when importing new architecture packages from ftp.ports.d.o to ftp-master.d.o. -- bye, pabs https://wiki.debian.org/PaulWise
Re: https://tracker.debian.org/pkg/dballe
On Fri, 2020-01-17 at 03:45 -0500, Sam Hartman wrote: > Bootstrap uploads of compilers etc are actually more common than I > thought before I started following debian-release. The important part of my statement is that they are special, rather than that they are rare. > They are common enough that requiring interactions from ftpmaster and/or > release team would make being a developer of such packages suck > significantly more. Hmm, OK. I can't recall seeing any recently on debian-release. > If we throw away binaries by default, I do believe we need a mechanism > for maintainers to signal that this is a bootstrap upload. My proposed mechanism is that when the buildds cannot do the job due to need for a bootstrap process, the maintainer should do a binary-only build (using whatever ugly hacks are needed), dak should import that into a special part of the archive for bootstrapping packages that need that and then tell the buildds to do a new non-bootstrap build but using the bootstrap archive. If bikesheds become a thing, we could even have one bootstrap archive per instance of a bootstrap being needed. Like Josch says, where possible, we also want the buildds to be capable of doing automated bootstrap builds that are enabled by automatic or maintainer-selected addition of bootstrap related build profiles. Until the special bootstrap archive and automated bootstrap builds are implemented, we could of course just import the maintainer bootstrap binaries into the main archive with a flag saying they need to be rebuilt before they can enter testing, and then do binNMUs. ISTR we have done similar things when importing whole architectures to ftp-master so I think some parts or flags for some of these things might already be in place. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: https://tracker.debian.org/pkg/dballe
On 1/17/20 6:55 AM, Sam Hartman wrote: >> "Johannes" == Johannes Schauer writes: > > Johannes> or have a mechanism that allows maintainers to tell buildds > "please build this > Johannes> source package with build profiles X and Y enabled". That would > then build the > Johannes> binary packages necessary to do a full build in a second upload. > > That doesn't help. > > If I need version y+3 of rustcc to build y+5 of rustcc and only y is in > the archive, no build profile is going to help me. If I'm following correctly: The packager would use rustcc >= y+3 (in practice, likely rustcc y+5) to locally build rustcc y+5 and then do a binary upload. But if dak (or whatever, I'm not so familiar with the server side here) throws away the binary, then we're sunk. So there needs to be a way to note that the binary needs to be kept. Assuming I'm on the right track, here are a couple of questions: In such a case, would the source package have a Build-Depends of >= y+3? It seems like it would. Could that be the way to indicate a bootstrap upload? In other words, if the package has a Build-Depends on one of the binary packages it produces that is _not_ satisfied in the suite it's being uploaded to but is satisfied by this upload, then it's a bootstrap upload, and the binaries should be kept. This would avoiding adding a new field for this and would ensure the Build-Depends are set correctly in bootstrapping scenarios. Regardless of how the "keep the binaries" flag is implemented... should the uploaded binary be published, or should the package be rebuilt? I think I'd argue for the latter. The uploaded binary package should be installed on the buildd and then the package should be rebuilt from source, with _that_ result being put into the archive. This doesn't protect against the trojaned-compiler problem, but it does at least ensure that the package builds from source. -- Richard signature.asc Description: OpenPGP digital signature
Re: https://tracker.debian.org/pkg/dballe
> "Johannes" == Johannes Schauer writes: Johannes> or have a mechanism that allows maintainers to tell buildds "please build this Johannes> source package with build profiles X and Y enabled". That would then build the Johannes> binary packages necessary to do a full build in a second upload. That doesn't help. If I need version y+3 of rustcc to build y+5 of rustcc and only y is in the archive, no build profile is going to help me. There are a lot of packages that need themselves to build themselves in Debian. It's not just a new arch issue.
Re: https://tracker.debian.org/pkg/dballe
Quoting Sam Hartman (2020-01-17 09:45:43) > > "Paul" == Paul Wise writes: > > Paul> On Mon, Jan 13, 2020 at 4:11 PM Wouter Verhelst wrote: > >> You'll make it unnecessarily harder to bootstrap environments that need > >> themselves to build if you do that. > > Paul> The idea here is that bootstrap builds are special and so they > should > Paul> be very explicit rather than happen as a side effect of regular > upload > Paul> workflows. Since they are relatively rare, making them explicit via > Paul> this mechanism shouldn't be too much of an imposition and on balance > Paul> might be the right way to go. > > Bootstrap uploads of compilers etc are actually more common than I > thought before I started following debian-release. > They are common enough that requiring interactions from ftpmaster and/or > release team would make being a developer of such packages suck > significantly more. > > If we throw away binaries by default, I do believe we need a mechanism for > maintainers to signal that this is a bootstrap upload. or have a mechanism that allows maintainers to tell buildds "please build this source package with build profiles X and Y enabled". That would then build the binary packages necessary to do a full build in a second upload. In an even better world, dak would figure out itself, that a source package first has to be built with build profiles X and Y and then schedule a full build afterwards. Thanks! cheers, josch signature.asc Description: signature
Re: https://tracker.debian.org/pkg/dballe
> "Paul" == Paul Wise writes: Paul> On Mon, Jan 13, 2020 at 4:11 PM Wouter Verhelst wrote: >> You'll make it unnecessarily harder to bootstrap environments that need >> themselves to build if you do that. Paul> The idea here is that bootstrap builds are special and so they should Paul> be very explicit rather than happen as a side effect of regular upload Paul> workflows. Since they are relatively rare, making them explicit via Paul> this mechanism shouldn't be too much of an imposition and on balance Paul> might be the right way to go. Bootstrap uploads of compilers etc are actually more common than I thought before I started following debian-release. They are common enough that requiring interactions from ftpmaster and/or release team would make being a developer of such packages suck significantly more. If we throw away binaries by default, I do believe we need a mechanism for maintainers to signal that this is a bootstrap upload. --Sam
Re: https://tracker.debian.org/pkg/dballe
On Mon, Jan 13, 2020 at 4:11 PM Wouter Verhelst wrote: > You'll make it unnecessarily harder to bootstrap environments that need > themselves to build if you do that. The idea here is that bootstrap builds are special and so they should be very explicit rather than happen as a side effect of regular upload workflows. Since they are relatively rare, making them explicit via this mechanism shouldn't be too much of an imposition and on balance might be the right way to go. -- bye, pabs https://wiki.debian.org/PaulWise
Re: https://tracker.debian.org/pkg/dballe
On Mon, Dec 30, 2019 at 02:47:45AM +, Paul Wise wrote: > On Sun, Dec 29, 2019 at 11:32 AM Paul Gevers wrote: > > > [1] Source packages that build binaries unknown to the archive currently > > need these binaries to be uploaded by the maintainers for reviewing by > > ftp-master in NEW. IIRC there have been multiple proposals to avoid > > these binaries from either being needed or being uploaded to the Debian > > archive, but so far the current tooling requires this. > > There has been some recent work on this part of the problem, I > focussed on throwing away binaries for all sourceful uploads and ivodd > focussed on doing that for NEW uploads only. Since ivodd's patch is > further along than mine, I hope he will extend it to all sourceful > uploads. You'll make it unnecessarily harder to bootstrap environments that need themselves to build if you do that. Throwing it away for NEW makes sense, but not for regular uploads, IMO. -- To the thief who stole my anti-depressants: I hope you're happy -- seen somewhere on the Internet on a photo of a billboard
Re: https://tracker.debian.org/pkg/dballe
On Mon, Dec 30, 2019 at 01:39:14PM +0100, Mattia Rizzolo wrote: > On Mon, Dec 30, 2019 at 11:29:52AM +0100, Kurt Roeckx wrote: > > Note that the name of the .changes file by the maintainer and the > > buildd will be the same, and dak will reject it if that .changes > > file already exists. > > That's true only in case of policy queues nowadays. What is a policy queue? All the recent rejects I get seems to be stable/security uploads. Kurt
Re: https://tracker.debian.org/pkg/dballe
On Mon, 2019-12-30 at 11:29 +0100, Kurt Roeckx wrote: > Is it .deb and .changes file that you would move? That would be the idea yeah. > Note that the name of the .changes file by the maintainer and the > buildd will be the same, and dak will reject it if that .changes > file already exists. I expect that could be dealt with easily in the dak patches, for e.g. by renaming the .changes to include the fingerprint of the signing key or by naming the morgue directory after the signing key fingerprint. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: https://tracker.debian.org/pkg/dballe
On Mon, Dec 30, 2019 at 11:29:52AM +0100, Kurt Roeckx wrote: > Note that the name of the .changes file by the maintainer and the > buildd will be the same, and dak will reject it if that .changes > file already exists. That's true only in case of policy queues nowadays. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. More about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Re: https://tracker.debian.org/pkg/dballe
On Mon, Dec 30, 2019 at 02:52:54AM +, Paul Wise wrote: > On Sun, Dec 29, 2019 at 1:29 PM Roberto C. Sánchez wrote: > > > Would it not be possible to eliminate the need for the second > > unnecessary upload by requiring two signed .changes files to go into > > NEW? A signed binary changes which would form the basis of the FTP > > master review and a signed source changes to enter the archive if the > > package is approved? > > Another approach is to simply have dak discard binaries from all > sourceful uploads (in dak parlance that means .changes files that have > a .dsc) (and save them to an audit directory). The buildds currently > only produce non-sourceful uploads so all their binaries get accepted > fine. For bootstrap scenarios, maintainers can just do an additional > binary-only upload. See the patches myself/ivodd posted recently for a > work in progress on this. Is it .deb and .changes file that you would move? Note that the name of the .changes file by the maintainer and the buildd will be the same, and dak will reject it if that .changes file already exists. Kurt
Re: https://tracker.debian.org/pkg/dballe
On Sun, Dec 29, 2019 at 1:29 PM Roberto C. Sánchez wrote: > Would it not be possible to eliminate the need for the second > unnecessary upload by requiring two signed .changes files to go into > NEW? A signed binary changes which would form the basis of the FTP > master review and a signed source changes to enter the archive if the > package is approved? Another approach is to simply have dak discard binaries from all sourceful uploads (in dak parlance that means .changes files that have a .dsc) (and save them to an audit directory). The buildds currently only produce non-sourceful uploads so all their binaries get accepted fine. For bootstrap scenarios, maintainers can just do an additional binary-only upload. See the patches myself/ivodd posted recently for a work in progress on this. -- bye, pabs https://wiki.debian.org/PaulWise
Re: https://tracker.debian.org/pkg/dballe
On Sun, Dec 29, 2019 at 11:32 AM Paul Gevers wrote: > [1] Source packages that build binaries unknown to the archive currently > need these binaries to be uploaded by the maintainers for reviewing by > ftp-master in NEW. IIRC there have been multiple proposals to avoid > these binaries from either being needed or being uploaded to the Debian > archive, but so far the current tooling requires this. There has been some recent work on this part of the problem, I focussed on throwing away binaries for all sourceful uploads and ivodd focussed on doing that for NEW uploads only. Since ivodd's patch is further along than mine, I hope he will extend it to all sourceful uploads. https://lists.debian.org/msgid-search/5ec2e979cd7d7ec9bf386fbf77e3399c7aeeb473.ca...@debian.org https://salsa.debian.org/pabs/dak/commits/discard-binaries https://lists.debian.org/msgid-search/87sgm2lhwc@marvin.43-1.org https://lists.debian.org/msgid-search/de353950121612a86ca706cbdf36153b25b9fa36.ca...@debian.org https://lists.debian.org/msgid-search/27641434-e45a-404f-254f-daea89916...@debian.org https://salsa.debian.org/ivodd/dak/tree/new-throwaway-binary-v10 -- bye, pabs https://wiki.debian.org/PaulWise
Re: https://tracker.debian.org/pkg/dballe
On Sun, Dec 29, 2019 at 12:32:24PM +0100, Paul Gevers wrote: > > Now I see that things got stuck for the transition to testing, and I'm > > asking for help figuring out what to do. > > I'm happy to help. Thanks! <3 > The python3.8 transition is not a classical transition, so this normally > helpful comment from tracker.d.o doesn't really apply. Please ignore it. Ack, wonderful. > As explained above, you'll not be interfering, so go ahead. Perfect, I have gone ahead with a new source-only upload. Enrico -- GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini signature.asc Description: PGP signature
Re: https://tracker.debian.org/pkg/dballe
On Sun, Dec 29, 2019 at 09:52:44AM +0100, Enrico Zini wrote: > Hello, > > some time ago I uploaded a new version of dballe, which went through NEW > because of a change in binary package names (SONAME bump, IIRC). It took > two weeks to go through NEW and I turned my energy towards other things > since then. Wow, two weeks? I uploaded a new version of f2fs-tools back in July, with the same issue (SONAME bump), and it's still not gotten through NEW. I had assumed everyone was waiting 5-6+ months to get through NEW - Ted
Re: https://tracker.debian.org/pkg/dballe
On Sun, Dec 29, 2019 at 12:32:24PM +0100, Paul Gevers wrote: > > [1] Source packages that build binaries unknown to the archive currently > need these binaries to be uploaded by the maintainers for reviewing by > ftp-master in NEW. IIRC there have been multiple proposals to avoid > these binaries from either being needed or being uploaded to the Debian > archive, but so far the current tooling requires this. On the other > hand, the release team has decided that we want all binaries in our > releases to be build on buildd. For that we have enhanced our migration > software to check for non-buildd uploads and add a block for them. > Unfortunately, once uploaded we can't fix the situation for arch:all > packages as binNMU'ing arch:all is currently broken. There is slow > progress to improve that situation, see e.g. bug #916601. > I've encountered this issue myself in the past with a SONAME bump in a library package. It was rather surprising. Would it not be possible to eliminate the need for the second unnecessary upload by requiring two signed .changes files to go into NEW? A signed binary changes which would form the basis of the FTP master review and a signed source changes to enter the archive if the package is approved? When I build packages I always end up with both changes files, so requiring both for NEW processing would be a triviality (for me, at least) and eliminate this peculiarity as well. Regards, -Roberto -- Roberto C. Sánchez
Re: https://tracker.debian.org/pkg/dballe
Hi Enrico, On 29-12-2019 09:52, Enrico Zini wrote: > some time ago I uploaded a new version of dballe, which went through NEW > because of a change in binary package names (SONAME bump, IIRC). It took > two weeks to go through NEW and I turned my energy towards other things > since then. > > Now I see that things got stuck for the transition to testing, and I'm > asking for help figuring out what to do. I'm happy to help. > I have this: > >> This package is part of the ongoing testing transition known as >> python3.8. Please avoid uploads unrelated to this transition, they >> would likely delay it and require supplementary work from the release >> managers. On the other hand, if your package has problems preventing >> it to migrate to testing, please fix them as soon as possible. You can >> probably find supplementary information in the debian-release archives >> or in the corresponding release.debian.org bug. The python3.8 transition is not a classical transition, so this normally helpful comment from tracker.d.o doesn't really apply. Please ignore it. If you would follow the link to the transition page, you would find a link to a transition bug (#942106) and it will explain (already in the title) that it is different from most transitions. (While writing this response I realize it may be possible for me to "hide" this info on the tracker, but I haven't done that before). > And I have this: > >> Not built on buildd: arch all binaries uploaded by enrico, a new >> source-only upload is needed to allow migration > > I was asked to do a binary upload to go through NEW, and now I'm asked > to do a source only upload to go to testing. There are surely good > reasons for that, and I wish there weren't. I agree with this. And from your phrasing your not really interested in the explanation, so find that at the bottom [1]. > I don't really have a new version to upload, but I suppose I need to at > least bump the debian version with no other changes in the package for > it to be accepted. Unfortunately, indeed. > Then, if I did that, would I be helping the python3.8 transition by > enabling a migration to testing, or getting in the way by delaying the > transition and requiring supplementary work from the release managers? As explained above, you'll not be interfering, so go ahead. > I feel a bit caught in a bureaucratic corner. Please help me with some > directions to get out of that. Hope this helps. Paul [1] Source packages that build binaries unknown to the archive currently need these binaries to be uploaded by the maintainers for reviewing by ftp-master in NEW. IIRC there have been multiple proposals to avoid these binaries from either being needed or being uploaded to the Debian archive, but so far the current tooling requires this. On the other hand, the release team has decided that we want all binaries in our releases to be build on buildd. For that we have enhanced our migration software to check for non-buildd uploads and add a block for them. Unfortunately, once uploaded we can't fix the situation for arch:all packages as binNMU'ing arch:all is currently broken. There is slow progress to improve that situation, see e.g. bug #916601. signature.asc Description: OpenPGP digital signature
