Re: Do not make gratuitous source uploads just to provoke the buildds!
Mike Fedyk [EMAIL PROTECTED] writes: Goswin von Brederlow wrote: I was thinking of having support in the buildd to fetch source, check a local patch archive for fixes, patch source, build package, add patch to each debs /usr/share/doc/package/. Would that satisfy the GPL or other DFSG licenses? As long as the patches are made available to the same people who can get the binaries, you should be set. Though, just for ease of access, it should be made available in a similar way to what the tier1 arches do for their sources. Mike As I said, the patch relative to the debian source package would be in the deb. I believe the changes file and version can be made to make DAK keep the right source version in the archive. So people that have the deb do have the patch and can get the source to apply it to. The idea for this comes from having some 5 line patch to add amd64 support to a package and maintainers that don't add the patch for several uploads. I guess for tier1/2 archs porters can just NMU them in. But for tier3, totaly new ports or experimental archives like the gcc-4.0 compiled amd64 one where the patches might not be ready for unstable this could be usefull. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 21, 2005 at 12:35:28AM +0100, Wouter Verhelst wrote: On Fri, Mar 18, 2005 at 11:43:48PM -0800, Steve Langasek wrote: the more or less aspect of the urgency is relevant here. We obviously have a system for classifying the severity of bugs in packages, and it's possible to relate these bug severities to the urgency field in uploads; even assuming it does get abused by maintainers, I don't think the possibility of something like that being abused is as strange as you seem to imply. As proof of that statement, I faintly remember someone doing a gratuitous source upload just to provoke the buildds... On the contrary, I do expect there would be a certain amount of abuse of such a system -- I just can't imagine such abuse reaching a level where it constitutes a worse failure scenario than the one we currently have. *Particularly* if we apply appropriate social pressures against abusers. how would considering urgency for package build ordering be worse than what we have now given that it should only be an issue in either case when the buildds are not working the way they should? It would be worse in that it would increase the impact of a re-upload. Not only would it trigger a rebuild on all architectures, it would now suddenly also throw the build ordering around, possibly worsening the problem that prompted the gratuitous upload in the first place by not building urgent (in build-dependency order) packages first. But, uploads impact the ordering of Needs-Build all the time; I don't see why that would generally be any worse than the status quo. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Wouter Verhelst ([EMAIL PROTECTED]) [050321 00:25]: On Thu, Mar 17, 2005 at 10:40:43AM +0100, Andreas Barth wrote: If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Is that statement based on experience of current behaviour, or rather on expected behaviour? It's based on experience. (Currently, architectures only get ignored if they're lagging behind anyway, so then the above statement is obviously true. When architectures are being ignored for other reasons, the above isn't as obvious anymore) It's probably true that an arch that is always ignored doesn't get out of sync as bad as an ignored arch now. However, every arch is sometimes a bit out-of-sync on one or another package, and I doubt that it will really work. But well, of course this could be tried out if it seems to be the best solution. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth wrote: * Wouter Verhelst ([EMAIL PROTECTED]) [050321 00:25]: On Thu, Mar 17, 2005 at 10:40:43AM +0100, Andreas Barth wrote: If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Why not include diffs of the source for the arches that required porting before the packages would compile and work properly for that arch to comply with this legal requirement? This would allow for Tier2 arches to be released with a later point release or some other schedule. Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Mike Fedyk [EMAIL PROTECTED] writes: Andreas Barth wrote: * Wouter Verhelst ([EMAIL PROTECTED]) [050321 00:25]: On Thu, Mar 17, 2005 at 10:40:43AM +0100, Andreas Barth wrote: If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Why not include diffs of the source for the arches that required porting before the packages would compile and work properly for that arch to comply with this legal requirement? This would allow for Tier2 arches to be released with a later point release or some other schedule. Mike I was thinking of having support in the buildd to fetch source, check a local patch archive for fixes, patch source, build package, add patch to each debs /usr/share/doc/package/. Would that satisfy the GPL or other DFSG licenses? MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Goswin von Brederlow wrote: Mike Fedyk [EMAIL PROTECTED] writes: Andreas Barth wrote: * Wouter Verhelst ([EMAIL PROTECTED]) [050321 00:25]: On Thu, Mar 17, 2005 at 10:40:43AM +0100, Andreas Barth wrote: If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Why not include diffs of the source for the arches that required porting before the packages would compile and work properly for that arch to comply with this legal requirement? This would allow for Tier2 arches to be released with a later point release or some other schedule. Mike I was thinking of having support in the buildd to fetch source, check a local patch archive for fixes, patch source, build package, add patch to each debs /usr/share/doc/package/. Would that satisfy the GPL or other DFSG licenses? As long as the patches are made available to the same people who can get the binaries, you should be set. Though, just for ease of access, it should be made available in a similar way to what the tier1 arches do for their sources. Mike
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Thu, Mar 17, 2005 at 10:40:43AM +0100, Andreas Barth wrote: If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Is that statement based on experience of current behaviour, or rather on expected behaviour? (Currently, architectures only get ignored if they're lagging behind anyway, so then the above statement is obviously true. When architectures are being ignored for other reasons, the above isn't as obvious anymore) -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Fri, Mar 18, 2005 at 11:43:48PM -0800, Steve Langasek wrote: the more or less aspect of the urgency is relevant here. We obviously have a system for classifying the severity of bugs in packages, and it's possible to relate these bug severities to the urgency field in uploads; even assuming it does get abused by maintainers, I don't think the possibility of something like that being abused is as strange as you seem to imply. As proof of that statement, I faintly remember someone doing a gratuitous source upload just to provoke the buildds... how would considering urgency for package build ordering be worse than what we have now given that it should only be an issue in either case when the buildds are not working the way they should? It would be worse in that it would increase the impact of a re-upload. Not only would it trigger a rebuild on all architectures, it would now suddenly also throw the build ordering around, possibly worsening the problem that prompted the gratuitous upload in the first place by not building urgent (in build-dependency order) packages first. [1] this is technically possible, but only in a kindof hackish way, by manually adding the package to a buildd's REDO file and (also manually) setting it to 'building' by that host. Yes, I don't think it scales very well to either have the release team asking for this, or for the buildd maintainers to be fielding such manual requests. If anything, the current workaround options (ignoring select out-of-date binaries for an arch) seem more efficient. Whatever you say; I don't mind either way. The offer still stands :) -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst [EMAIL PROTECTED] writes: I don't think the possibility of something like that being abused is as strange as you seem to imply. As proof of that statement, I faintly remember someone doing a gratuitous source upload just to provoke the buildds... Of course, there was no abuse here about priority; the package was incorrectly in extra. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Tue, Mar 15, 2005 at 09:56:10AM +0100, Goswin von Brederlow wrote: I would like to see some stats showing on how many days in the last year an arch reached 0 needs-build. I highly doubt that any arch managed to do it every day troughout the last year. You know why goals are important? 0 needs-build is definitly a goal we should work to. I disagree. 0 needs-build once a day is a bad line to draw. Saying packages must be build a day before they become testing candidates would be a better line. But that would require a non starving queue to mean anything but 0 needs-build. I don't see any great harm with packages getting build 5 days late if they have to wait 10 days for testing. As long as they do get build on time. Ah, but packages that are being uploaded with urgency=low are usually not a concern for the release team at all; it's the *high*-urgency uploads that normally demand the release team's attention, and these are precisely the ones that slower build architectures are going to have a harder time building within the purgatory interval (2 days, not 10, for high-urgency packages). To say nothing of the fact that urgency is not currently considered for package build ordering, which means that it's very possible for a high-urgency upload to go for 2 days without being tried at all. But what do I know, I'm not an RM. So lets thing about the criterion: Strictly requiring 0 needs-builds every day means the buildd must have enough power to cope even with huge upload peeks and if one of the buildds fail at a peak time no arch will cope with that. Obviously some leaway will have to be given for arch to temporarily not meat the criterion, say 0 needs-build on 75% of all days and no more than 3 consecuitve failures wihtout special circumstances or something of that sort. Right? Or do you realy want to remove i386 from the release if it fails 0 needs-build 10 times before etch release? Andreas never said anything about this being the criterion for RC architectures. He said it should be a *goal*. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
Except the possibility to profit from the release team's efforts, and to create an actually supported release. It is not reasonable to believe a small porter team can do security updates for a unstable snapshot when a task of similiar size already overloads the stable security team. No. As neither the release team, not the security team will take non-release archs into account. Cheers, Peter (p2). signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
Porters who have worked on getting an arch to REGUALR status are in a much better position (demonstrated commitment, technical aptness and experiencewise) to solve those problems than random-joe-developer. I have no idea what you're trying to say here. Always remember that the main reason that it is easier for a porters team to release within the (current) Debian framework than outside is that _others_ do work for them. That has been the case up to now, but won't be the case after sarge. Cheers, Peter (p2). signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
Peter 'p2' De Schrijver [EMAIL PROTECTED] writes: Except the possibility to profit from the release team's efforts, and to create an actually supported release. It is not reasonable to believe a small porter team can do security updates for a unstable snapshot when a task of similiar size already overloads the stable security team. No. As neither the release team, not the security team will take non-release archs into account. Cheers, Peter (p2). You can still benefit from their work. The stable sources are hopefully in sync with each other, the software is known to work. Most importantly there are source CD/DVD sets out there for it. That saves a lot of working and server space. I don't think Debian should support having a complete fork, every port can do that on their own. As for security stuff probably all fixes can be reused as is by the porters if they have the sources in sync with stable. They just have to setup a wanna-build and buildd and they get the fixes quickly after Debian releases them. The more sources are out of sync the more likely it is a vulnerable package has a different version which means extra work. But I would hope there could be some working together like one porter joining the security team and including non release archs in the DSAs _if_ they compile their packages on time. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Friday 18 March 2005 11:35, Peter 'p2' De Schrijver wrote: Porters who have worked on getting an arch to REGUALR status are in a much better position (demonstrated commitment, technical aptness and experiencewise) to solve those problems than random-joe-developer. I have no idea what you're trying to say here. Always remember that the main reason that it is easier for a porters team to release within the (current) Debian framework than outside is that _others_ do work for them. That has been the case up to now, but won't be the case after sarge. Indeed that is quite the point I wanted to make with the first sentence. Please excuse my inability to express that clearly. I will try again: Up until and including the sarge release, central figures in release management did most of the work that was needed between a arch-fixed upload to unstable and a stable release. Reading the Vancouver-proposal, I come to the conclusion that they recognise that they are not able to make a timely release (sarge demonstrated that quite clearly). Therefore they propose certain criteria a arch should fulfil that they are willing to support a release for the arch. If you try to visualise the effects of that I hope that this will happen: 1) people realize that $arch won't be REGULAR for etch because the people working on a release don't want to handhold it through testing and autobuilding is too slow to properly keep up. 2) people realize that whining and bitching on debian-devel won't convince the people working on a release that $arch is worth the work 3a) people create $arch-specific-release-mechanism on ports.d.o and learn much about the pain of keeping $arch in sync with REGULAR development. 3b) people create security-$arch and take care of packages which are affected by DSAs but are not yet fixed in $arch 4) by going through 3a and 3b people demonstrate commitment and technical aptness as well as gather experience regarding the release cycle. This makes them perfect release and security assistants for $arch. As far as I can tell, Debian on the whole is shortly before ending phase 2 and I have already seen people work on 3a. And if anyone dares to object to point 4 that the current release team should get their act together, because they could just improve $releae-mechanism and release etch with all 15+ arches within the next two years really hasn't understood how Debian works[1]. If this pace keeps up I am convinced that etch will have more than four REGULAR arches. Regards, David [1] ... and should send in some of the stuff he smokes. -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir ber ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 12:19:15PM +0100, Wouter Verhelst wrote: Op ma, 14-03-2005 te 00:10 -0800, schreef Steve Langasek: Well, my objection is basically the same as Thomas's here -- all package builds are *not* equally urgent, Of course not, that is exactly my point. But from the POV of a package's maintainer, all fixes are more or less urgent. If some fixes weren't necessary, the upload wouldn't have been there in the first place. I find this line of reasoning fairly incomprehensible; the more or less aspect of the urgency is relevant here. We obviously have a system for classifying the severity of bugs in packages, and it's possible to relate these bug severities to the urgency field in uploads; even assuming it does get abused by maintainers, how would considering urgency for package build ordering be worse than what we have now given that it should only be an issue in either case when the buildds are not working the way they should? and in fact, we have an urgency field in uploads that expresses this fact quite clearly. Certainly there's some danger of abuse by uploaders, but there are dozens of other things that maintainers *could* abuse right now and are only stopped from doing because they *shouldn't* do them. I wouldn't be bothered by porters choosing how to order builds, if the ordering they chose more closely corresponded to what the release team (and britney) want it to be. :) I from my side wouldn't mind if someone from the release team would ask me to prioritize a build[1] if necessary; but I feel irky at the thought of allowing other people to prioritize their packages' builds over others, because that *will* make sure that eventually, those people that do what is actually the right thing will have to wait for all eternity for their packages to be built. [1] this is technically possible, but only in a kindof hackish way, by manually adding the package to a buildd's REDO file and (also manually) setting it to 'building' by that host. Yes, I don't think it scales very well to either have the release team asking for this, or for the buildd maintainers to be fielding such manual requests. If anything, the current workaround options (ignoring select out-of-date binaries for an arch) seem more efficient. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Mike Fedyk ([EMAIL PROTECTED]) [050316 20:55]: Andreas Barth wrote: If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Wed, 16 Mar 2005 14:44:34 +0100, Wouter Verhelst [EMAIL PROTECTED] wrote: Op di, 15-03-2005 te 16:19 -0500, schreef Anthony DeRobertis: Wouter Verhelst wrote: | You misunderstood. I don't fight generic changes to the order; I just | don't think it would be a good thing that any random developer could | prioritize his pet package. | Any random developer already has root on X thousand debian systems. We can trust him with that, but not with helping determine build order? I'm afraid it won't actually be 'helping'. I've seen enough of My package isn't built and it's urgent and the buildd admins suck! to expect what would happen. Aah, so the trick should be to implement this on the quiet. Tweak the values according to have it's progressing. Eventually you should see a reduction in complaints. It wouldn't be hard to be able to guarentee an upper bound for building, say two months. Once people stop complaining you can tell them. Maybe they won't feel so inclined to fiddle when they can see it works. That's not to say that a request to prioritize a package is to be ignored; however, the power of deciding which packages get built first should be with those that actually build the packages, rather than with those who want their packages to be built. The former are expected to be following the larger picture; the latter are not. As far as I can tell, buildd admins don't spend all day prioritising builds manually anyway. We're dealing with computers here, they don't care how complicated the calculations are. Decide where your priorities are and implement it. If you don't like to hear people complain, add a rule saying if package is older then one month, build it now. It cost you nothing and saves heaps of grief. As a bonus, repeatedly uploading new versions would keep resetting the counter, so they'd be encouraged to upload less. The current process of indefinitly starving extra packages is just silly. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth wrote: * Mike Fedyk ([EMAIL PROTECTED]) [050316 20:55]: Andreas Barth wrote: If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Removing only the affected packages of that arch instead of the whole thing looks more sensible. Of course this assumes the core packages are kept in shape. Thiemo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth wrote: * Mike Fedyk ([EMAIL PROTECTED]) [050316 20:55]: Andreas Barth wrote: If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) That would be understandable with the intention to release all arches at the same time. In this case the release should be at a different time *if* that arch is in a releasable state. Otherwise, it is not released. The point is that propagating to testing is very useful, and it still leaves the status of that arch to the porters. With testing there for SCC arches, it everyone will just point to the porters for that arch if there is a propagation problem. Mike
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth [EMAIL PROTECTED] writes: * Mike Fedyk ([EMAIL PROTECTED]) [050316 20:55]: Andreas Barth wrote: If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Not if each arch has it's own source tracking. And you already need that for snapshot fixes. Non release archs should be released by the porters alone (no burden to RMs) with a minimum of arch specific versions or patches. There should be a strong encouragement to remove software instead of patching it when it comes close to the actual release so when the port does release (after the main release) it is based on stable source for everything but last minute flaws in essential packages. Maintaining those patches in case of security updates or for the point releases again should lie with the porters. The reason why I favour this is that I have in mind that some archs will be too slow, they won't be able to keep up every day. But given an extra month they can compile the backlog or kick out out-of-sync software and release seperately with a nearly identical source tree. The remaining source changes can (and basically must for legal reasons) be contained on the binary CD/DVD set and in a branch of the scc.d.o tree. Take for example m68k. It will always be at risk of lagging a few days behind because some packages do build a few days. It is always out-of-sync longer than the other archs but it is not getting worse, it is just a step behind. That is totaly different than arm or s390 taking a deep dive getting some 300+ package backlog and having packages stuck for a month. If an arch has enough developers on it to keep stuff building, and that means supplying patches to unstable fast and early enough to get them into testing and ultimately stable I see no reason why the arch shouldn't release. Make some rule to limit out-of-sync, say no more than 1% sources differences to stable for the release. Any problems with that? Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Thursday 17 March 2005 20:22, Goswin von Brederlow wrote: [very sensible suggestions removed] Any problems with that? Not with the procedure in itself. I just want to chip in, that it is (not only) my opinion, that a REGULAR Debian release cannot allow delaying security updates and there might be issues caused by delaying builds across library transitions. Perhaps there will be a need for defining ALMOSTREGULAR. Regards, David -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Thu, Mar 17, 2005 at 08:22:04PM +0100, Goswin von Brederlow wrote: Andreas Barth [EMAIL PROTECTED] writes: * Mike Fedyk ([EMAIL PROTECTED]) [050316 20:55]: Andreas Barth wrote: If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Not if each arch has it's own source tracking. And you already need that for snapshot fixes. Non release archs should be released by the porters alone (no burden to RMs) with a minimum of arch specific versions or patches. There should be a strong encouragement to remove software instead of patching it when it comes close to the actual release so when the port does release (after the main release) it is based on stable source for Why would a port release after the main release ? Why, if debian doesn't care about the non-release archs, would the porters even bother to follow the release arch sources and not just release whenever they like ? They don't gain anything by following the main release. everything but last minute flaws in essential packages. Maintaining those patches in case of security updates or for the point releases again should lie with the porters. The reason why I favour this is that I have in mind that some archs will be too slow, they won't be able to keep up every day. But given an extra month they can compile the backlog or kick out out-of-sync software and release seperately with a nearly identical source tree. The remaining source changes can (and basically must for legal reasons) be contained on the binary CD/DVD set and in a branch of the scc.d.o tree. Take for example m68k. It will always be at risk of lagging a few days behind because some packages do build a few days. It is always out-of-sync longer than the other archs but it is not getting worse, it is just a step behind. That is totaly different than arm or s390 taking a deep dive getting some 300+ package backlog and having packages stuck for a month. If an arch has enough developers on it to keep stuff building, and that means supplying patches to unstable fast and early enough to get them into testing and ultimately stable I see no reason why the arch shouldn't release. Make some rule to limit out-of-sync, say no more than 1% sources differences to stable for the release. Any problems with that? Yes. It doesn't make sense. Either debian releases with all archs, or every arch releases on its own. The latter is favoured by the current proposal and will diminish debian's value. The former is the way to go. Scalability problems need to be solved by improving infrastructure or procedures as appropriate. A middle ground between both release approaches is not sensible as it will still make the ports dependent on a possibly suboptimal upstream tree without having the benefit of debian security updates. Ie. ports get all the disadvantages of a synchronized release without getting any of the advantages. That's just plain unfair. Cheers, Peter (p2). signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Mike Fedyk ([EMAIL PROTECTED]) [050317 19:30]: Andreas Barth wrote: If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) That would be understandable with the intention to release all arches at the same time. I was speaking about release archs (i.e. the archs where the release team has to keep the strings together). For others archs, things may be different. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Peter 'p2' De Schrijver wrote: [snip] Why would a port release after the main release ? Probably to fix up a few remaining arch-specific bugs. Why, if debian doesn't care about the non-release archs, would the porters even bother to follow the release arch sources and not just release whenever they like ? They don't gain anything by following the main release. Except the possibility to profit from the release team's efforts, and to create an actually supported release. It is not reasonable to believe a small porter team can do security updates for a unstable snapshot when a task of similiar size already overloads the stable security team. [snip] If an arch has enough developers on it to keep stuff building, and that means supplying patches to unstable fast and early enough to get them into testing and ultimately stable I see no reason why the arch shouldn't release. Make some rule to limit out-of-sync, say no more than 1% sources differences to stable for the release. 1% is huge when it is glibc. :-) I think the yes / no decision about a ports release should be finally done by the security team, because they are the ones to get the work thrown at. A no would mean it can be still distributed along with the released ports, but it would signal the port's users that they have to ask the porter team for security updates (which would then lag a bit). Any problems with that? Yes. It doesn't make sense. Either debian releases with all archs, or every arch releases on its own. The latter is favoured by the current proposal and will diminish debian's value. The former is the way to go. Scalability problems need to be solved by improving infrastructure or procedures as appropriate. A middle ground between both release approaches is not sensible as it will still make the ports dependent on a possibly suboptimal upstream tree without having the benefit of debian security updates. Ie. ports get all the disadvantages of a synchronized release without getting any of the advantages. That's just plain unfair. I believe there _is_ some space between those extremes, if the port's source is synced enough with the main release. Thiemo signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Thursday 17 March 2005 23:44, Peter 'p2' De Schrijver wrote: On Thu, Mar 17, 2005 at 08:22:04PM +0100, Goswin von Brederlow wrote: Andreas Barth [EMAIL PROTECTED] writes: * Mike Fedyk ([EMAIL PROTECTED]) [050316 20:55]: Andreas Barth wrote: If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. If we don't wait for an arch, it gets out-of-sync quite soon, and due to e.g. legal requirements, we can't release that arch. (In other words, if an arch is too long ignored for testing, we should remove it, as we can't release it in any case.) Not if each arch has it's own source tracking. And you already need that for snapshot fixes. Non release archs should be released by the porters alone (no burden to RMs) with a minimum of arch specific versions or patches. There should be a strong encouragement to remove software instead of patching it when it comes close to the actual release so when the port does release (after the main release) it is based on stable source for Why would a port release after the main release ? Why, if debian doesn't care about the non-release archs, would the porters even bother to follow the release arch sources and not just release whenever they like ? They don't gain anything by following the main release. everything but last minute flaws in essential packages. Maintaining those patches in case of security updates or for the point releases again should lie with the porters. The reason why I favour this is that I have in mind that some archs will be too slow, they won't be able to keep up every day. But given an extra month they can compile the backlog or kick out out-of-sync software and release seperately with a nearly identical source tree. The remaining source changes can (and basically must for legal reasons) be contained on the binary CD/DVD set and in a branch of the scc.d.o tree. Take for example m68k. It will always be at risk of lagging a few days behind because some packages do build a few days. It is always out-of-sync longer than the other archs but it is not getting worse, it is just a step behind. That is totaly different than arm or s390 taking a deep dive getting some 300+ package backlog and having packages stuck for a month. If an arch has enough developers on it to keep stuff building, and that means supplying patches to unstable fast and early enough to get them into testing and ultimately stable I see no reason why the arch shouldn't release. Make some rule to limit out-of-sync, say no more than 1% sources differences to stable for the release. Any problems with that? Yes. It doesn't make sense. Either debian releases with all archs, or every arch releases on its own. The latter is favoured by the current proposal and will diminish debian's value. The former is the way to go. Scalability problems need to be solved by improving infrastructure or procedures as appropriate. Porters who have worked on getting an arch to REGUALR status are in a much better position (demonstrated commitment, technical aptness and experiencewise) to solve those problems than random-joe-developer. Always remember that the main reason that it is easier for a porters team to release within the (current) Debian framework than outside is that _others_ do work for them. Regards, David -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir ber ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op di, 15-03-2005 te 16:19 -0500, schreef Anthony DeRobertis: Wouter Verhelst wrote: | You misunderstood. I don't fight generic changes to the order; I just | don't think it would be a good thing that any random developer could | prioritize his pet package. | Any random developer already has root on X thousand debian systems. We can trust him with that, but not with helping determine build order? I'm afraid it won't actually be 'helping'. I've seen enough of My package isn't built and it's urgent and the buildd admins suck! to expect what would happen. That's not to say that a request to prioritize a package is to be ignored; however, the power of deciding which packages get built first should be with those that actually build the packages, rather than with those who want their packages to be built. The former are expected to be following the larger picture; the latter are not. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Hello Wouter On 2005-03-16 Wouter Verhelst wrote: That's not to say that a request to prioritize a package is to be ignored; however, the power of deciding which packages get built first should be with those that actually build the packages, rather than with those who want their packages to be built. The former are expected to be following the larger picture; the latter are not. Given that the only really relevant thing is when the package enters testing, which already can be changed by the maintainers, the only thing a maintainer can wish is to have his package build within the 2 or 5 days, or? As only 2 days might be a problem, wouldn't just prefering high+security uploads be enough to make everybody happy without disrupting the buildd order too much? friendly, -christian- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst wrote: That's not to say that a request to prioritize a package is to be ignored; however, the power of deciding which packages get built first should be with those that actually build the packages, rather than with those who want their packages to be built. The former are expected to be following the larger picture; the latter are not. The latter, however, also know a good deal more about the particular details of the upload. It'd seem to at least make sense to use critical/medium/low as part of the sort criteria, for example before alphabetical. Maybe even ahead of standard/optional/extra. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Hi, Matthew Palmer wrote:
I think the queue
needs to be as FIFO as possible for fairness and principle of least
surprise sake.
See my patch on d-d (also mailed to the ftpmasters), which inserts age in
queue (actually, timestamp of last status change, but that's
more-or-less equivalent) right before name in the sort order.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth wrote: * Matthew Palmer ([EMAIL PROTECTED]) [050313 01:05]: On Sat, Mar 12, 2005 at 03:12:12PM -0800, Steve Langasek wrote: Er, packages *do* eventually get built; they just don't get built in any kind of FIFO order. Er, no. Unless there's some sort of aging process (not yet described in the threads here) which will result in an extra package called zappa in section x11 from eventually being promoted above a package aardvark in section admin, it is entirely possible that package will never be built. All it requires is for the rate of new packages entering the queue before zappa to be equal to or greater than the rate of packages leaving the queue due to having been built or removed. If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. I think it makes sense to shorten the list of arches we wait upon for testing migration, but I fail to see the usefulness of removing an arch from testing. You may not see the usefulness of testing, but I run production servers on it because there are just too many packages that are too old and buggy or not available in the old stable release. Yes, I've only used i386 and PPC, but that should not exclude that functionality from the users of those architectures that are not as popular (even if you need a log scale to see their relative popularity). Mike
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth [EMAIL PROTECTED] writes: * Goswin von Brederlow ([EMAIL PROTECTED]) [050314 15:35]: Andreas Barth [EMAIL PROTECTED] writes: * Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:45]: On Sun, Mar 13, 2005 at 11:16:56PM +0100, Andreas Barth wrote: Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. How often should the queue be emptied, or when will an architecture be declarared not-keeping-up? In light of http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html the release architecture must have N+1 buildds where N is the number required to keep up with the volume of uploaded packages at least once per day for etch. That means no more m68k. Given that some packages compile up to 12 days there will be plenty of times the queue doesn't empty once per day. needs-build can be empty even if packages _are_ currently building. Not with N may not be 2. Say 3 mozilla clones get uploaded. Each of the N+1 buildds grabs one and ist busy for 5 days. If any other package gets uploaded in that time the queue will not be empty. Unless you take packages even while building. But I'm sure a long building queue on the buildds would be cheating and not count. I would like to see some stats showing on how many days in the last year an arch reached 0 needs-build. I highly doubt that any arch managed to do it every day troughout the last year. You know why goals are important? 0 needs-build is definitly a goal we should work to. I disagree. 0 needs-build once a day is a bad line to draw. Saying packages must be build a day before they become testing candidates would be a better line. But that would require a non starving queue to mean anything but 0 needs-build. I don't see any great harm with packages getting build 5 days late if they have to wait 10 days for testing. As long as they do get build on time. But what do I know, I'm not an RM. So lets thing about the criterion: Strictly requiring 0 needs-builds every day means the buildd must have enough power to cope even with huge upload peeks and if one of the buildds fail at a peak time no arch will cope with that. Obviously some leaway will have to be given for arch to temporarily not meat the criterion, say 0 needs-build on 75% of all days and no more than 3 consecuitve failures wihtout special circumstances or something of that sort. Right? Or do you realy want to remove i386 from the release if it fails 0 needs-build 10 times before etch release? Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst [EMAIL PROTECTED] writes: Op ma, 14-03-2005 te 17:59 +0100, schreef Goswin von Brederlow: Wouter Verhelst [EMAIL PROTECTED] writes: Op vr, 11-03-2005 te 19:14 -0800, schreef Steve Langasek: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - previous compilation state (already built packages are prioritized above packages never built for the target architecture) - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I agree with the w-b maintainers. The queue order is only interesting in the case where there is a backlog; in other cases, packages are usually built rather fast. In the case where there is a backlog, those trying to fix the architecture (usually those that are working on it) should be in charge of deciding what package gets built first, not the maintainer of a random package -- /all/ package builds are urgent if there's a backlog. Since you think an empty queue is normal why then fight changes to the queue order? You misunderstood. I don't fight generic changes to the order; I just don't think it would be a good thing that any random developer could prioritize his pet package. My suggestion is to use an aging algorithm with time * alpha + beta. Aspects of a source would then affect alpha and beta. Example: - Essential: yes adds 100 to beta and 5 to alpha. - Urgency: critical adds 10 to beta and 1 to alpha The package with the highest score gets build first. Alpha makes a package advance the queue faster overtaking other packages while beta makes packages start further up in the queue. By adjusting the changes to alpha and beta each aspect of a source can be finely tuned to have some affect on the queue. So random developer can influence the priority by setting the urgency but not so much as to abuse the power and deadlock other packages. Instead of the urgency of uploads the number of bugs (weighted by priority) could be used instead or on top of that. Or the number of depends, revers depends, dep-waits, A lot of aspects could be added up to set alpha and beta for each package. Even alpha=1, beta=-current static queue position and time in minutes would be an improvement. That would mean that after 6 days any package would be build before a fresh upload of the most essential package. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Wouter Verhelst wrote: | You misunderstood. I don't fight generic changes to the order; I just | don't think it would be a good thing that any random developer could | prioritize his pet package. | Any random developer already has root on X thousand debian systems. We can trust him with that, but not with helping determine build order? -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCN1FE+z+IwlXqWf4RArSmAJwLA1aiNaTQtVzgXWYcua2061CsbACfUTnG Ye6KWZ+xsscs+7VRHRTL8Dw= =kHlW -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sun, Mar 13, 2005 at 11:36:39PM +0100, Wouter Verhelst wrote: Op vr, 11-03-2005 te 19:14 -0800, schreef Steve Langasek: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - previous compilation state (already built packages are prioritized above packages never built for the target architecture) Yep, remembered that one after I sent the message. - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I agree with the w-b maintainers. The queue order is only interesting in the case where there is a backlog; in other cases, packages are usually built rather fast. In the case where there is a backlog, those trying to fix the architecture (usually those that are working on it) should be in charge of deciding what package gets built first, not the maintainer of a random package -- /all/ package builds are urgent if there's a backlog. Well, my objection is basically the same as Thomas's here -- all package builds are *not* equally urgent, and in fact, we have an urgency field in uploads that expresses this fact quite clearly. Certainly there's some danger of abuse by uploaders, but there are dozens of other things that maintainers *could* abuse right now and are only stopped from doing because they *shouldn't* do them. I wouldn't be bothered by porters choosing how to order builds, if the ordering they chose more closely corresponded to what the release team (and britney) want it to be. :) -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 11:43:54AM +1100, Hamish Moffatt wrote: s390 is also rising steeply. gcc problems and non-responding ftp-masters for the new buildd. Bastian -- Those who hate and fight must stop themselves -- otherwise it is not stopped. -- Spock, Day of the Dove, stardate unknown signature.asc Description: Digital signature
Re: buildd queue starvation (Was: Re: Do not make gratuitous source uploads just to provoke the buildds!)
On Sun, Mar 13, 2005 at 11:07:29PM -0800, Thomas Bushnell BSG wrote: Wouter Verhelst [EMAIL PROTECTED] writes: None of the documentation calls it a 'queue', in fact; only people not really involved in buildd stuff do. Does that include you? In two recent messages, you referred to it as a queue. Yes, that's correct; I shouldn't have. Sorry :-) -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sun, Mar 13, 2005 at 08:15:13PM -0800, Thomas Bushnell BSG wrote: Hamish Moffatt [EMAIL PROTECTED] writes: Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. I don't think those sections are causing the problem. There are also not so many uploads for them. The problem from my perspective is that True, but that isn't making my electronics packages build any faster. optional packages all have to be built before any extra packages get considered. That's what's hosing gnucash. hamradio is only one notch above extra in the pecking order anyway. I'm very pleased to hear this will be less important for etch anyway. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 07:37:56AM +0100, Ingo Juergensmann wrote: On Mon, Mar 14, 2005 at 11:49:34AM +1100, Hamish Moffatt wrote: Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. Frankly there are not likely to be any users for hamradio on s390, mips* arm, or m68k. Nor electronics. Instead those architectures just prevent the migration to testing for those packages, for no good reason. How can archs prevent the migration when the software is already uptodate, f.e. ax25-tools? http://unstable.buildd.net/cgi/package_status?unstable_all_pkg=ax25-toolssearchtype=go Yes. So I haven't uploaded any of those lately. I have one ready but don't want to confuse the issue for sarge. How about geda-gschem? Waiting on arm for a couple of weeks now. Holding up migration of all of geda* on all architectures. I couldn't work out where wanna-build CVS is hosted so I couldn't actually check the order to see where electronics fits in. Instead of considering dropping archs or excluding archs from building, you should consider improving the buildd process. The current wanna-build is known to have many drawbacks. It's an ancient program that doesn't fit any longer on todays needs. Patching it to death doesn't help much, imho. But right now, arm (for example) just cannot build all the packages that need building. Changing the order won't help. Only adding machines or removing packages will help. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:45]: On Sun, Mar 13, 2005 at 11:16:56PM +0100, Andreas Barth wrote: Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. How often should the queue be emptied, or when will an architecture be declarared not-keeping-up? In light of http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html the release architecture must have N+1 buildds where N is the number required to keep up with the volume of uploaded packages at least once per day for etch. According to http://people.debian.org/~igloo/needs-build-graph/index.php?days=30 arm hasn't been empty in over 3 weeks, and it's getting worse still. s390 is also rising steeply. It is not only a question of days, but also, what expectations we have for the architecture. If we e.g. know that in the next days a buildd will be switched on again (as the buildd crashed, and the local admin is away), we are of course a bit more tolerant. Also, vacations are accepted. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:55]: On Mon, Mar 14, 2005 at 12:01:59AM +0100, Andreas Barth wrote: It is a highly ordered list, more or less libs+base first, than devel, shells, perl, python. After that graphics, admin, utils. Just to look at the other side of the sorting order, at the end of the list is hamradio, non-US and embedded. The large ones like gnome and kde are in the middle of the list. Please see wanna-builds source for the full list. Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. We won't omit some sections on some architectures, but all sections on some architectures :) Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 08:50:15PM +1100, Hamish Moffatt wrote: How about geda-gschem? Waiting on arm for a couple of weeks now. Holding up migration of all of geda* on all architectures. I couldn't work out where wanna-build CVS is hosted so I couldn't actually check the order to see where electronics fits in. Yes, it's not easy to find and sometimes it migrates from host to host even... Maybe it's at newraff or at db.d.o? Instead of considering dropping archs or excluding archs from building, you should consider improving the buildd process. The current wanna-build is known to have many drawbacks. It's an ancient program that doesn't fit any longer on todays needs. Patching it to death doesn't help much, imho. But right now, arm (for example) just cannot build all the packages that need building. Changing the order won't help. Only adding machines or removing packages will help. Yes, adding machines will help, but this becomes a problem, when the in-person union of wanna-build and buildd admin for that arch don't want you to play in his Sandkasten... That's a human problem, not a hardware or arch specific problem. -- Ciao... // Ingo \X/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op ma, 14-03-2005 te 00:10 -0800, schreef Steve Langasek: Well, my objection is basically the same as Thomas's here -- all package builds are *not* equally urgent, Of course not, that is exactly my point. But from the POV of a package's maintainer, all fixes are more or less urgent. If some fixes weren't necessary, the upload wouldn't have been there in the first place. and in fact, we have an urgency field in uploads that expresses this fact quite clearly. Certainly there's some danger of abuse by uploaders, but there are dozens of other things that maintainers *could* abuse right now and are only stopped from doing because they *shouldn't* do them. I wouldn't be bothered by porters choosing how to order builds, if the ordering they chose more closely corresponded to what the release team (and britney) want it to be. :) I from my side wouldn't mind if someone from the release team would ask me to prioritize a build[1] if necessary; but I feel irky at the thought of allowing other people to prioritize their packages' builds over others, because that *will* make sure that eventually, those people that do what is actually the right thing will have to wait for all eternity for their packages to be built. [1] this is technically possible, but only in a kindof hackish way, by manually adding the package to a buildd's REDO file and (also manually) setting it to 'building' by that host. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 11:00:52AM +0100, Andreas Barth wrote: * Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:55]: Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. We won't omit some sections on some architectures, but all sections on some architectures :) That's a solution too! Well done to the release and ftpmaster teams for the plan for etch. You made some tough and possibly unpopular decisions, but I think they were the right decisions for the project. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst [EMAIL PROTECTED] writes: Op za, 12-03-2005 te 15:01 -0800, schreef Thomas Bushnell BSG: Goswin von Brederlow [EMAIL PROTECTED] writes: Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. This is news to me. It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. This paragraph assumes nobody ever looks the length of the needs-build queue of his architecture, and nobody feels bad when the queue is longer than normal. That idea would be hilarious if it wasn't sad. In reality, the upper bound is determined by motivated porters who try hard to avoid the queue ever to grow too long, day after day. The queue length is bound and easily detected if it grows too long. But do you notice when the same packages keep showing up at the end of the queue for weeks? The queue can be as small as 1 package inbetween and that 1 package could still never get build. Eventualy someone notices, usualy the maintainer, and a new why isn't my package build yet? flame starts. People should stop repeating the fiction then that just wait means your package will eventually get built. Why, if it is true? It usualy is. It might not be. And it can be an awfully long wait. The last one is the problem. The first two not. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth [EMAIL PROTECTED] writes: * Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:45]: On Sun, Mar 13, 2005 at 11:16:56PM +0100, Andreas Barth wrote: Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. How often should the queue be emptied, or when will an architecture be declarared not-keeping-up? In light of http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html the release architecture must have N+1 buildds where N is the number required to keep up with the volume of uploaded packages at least once per day for etch. That means no more m68k. Given that some packages compile up to 12 days there will be plenty of times the queue doesn't empty once per day. According to http://people.debian.org/~igloo/needs-build-graph/index.php?days=30 arm hasn't been empty in over 3 weeks, and it's getting worse still. s390 is also rising steeply. It is not only a question of days, but also, what expectations we have for the architecture. If we e.g. know that in the next days a buildd will be switched on again (as the buildd crashed, and the local admin is away), we are of course a bit more tolerant. Also, vacations are accepted. I would like to see some stats showing on how many days in the last year an arch reached 0 needs-build. I highly doubt that any arch managed to do it every day troughout the last year. If w-b had some aging algorithm instead of a fixed order we could say an arch may not have more than 1 week lag for more than 1 week or something. Cheers, Andi MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, March 14, 2005 15:09, Goswin von Brederlow said: People should stop repeating the fiction then that just wait means your package will eventually get built. It usualy is. It might not be. And it can be an awfully long wait. The last one is the problem. The first two not. This could easily be prevented: the priority of a package could be increased if it is longer than T in the queue (or if it's really long even give it top priority over any other package). You would get the situation that higher priority packages get built first, but from time to time a lower-but-long-waiting package gets built to prevent starvation. Seems like a fair situation to me. Thijs -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
[debian-release dropped] Goswin von Brederlow [EMAIL PROTECTED] writes: But do you notice when the same packages keep showing up at the end of the queue for weeks? The queue can be as small as 1 package inbetween and that 1 package could still never get build. Just out of curiosity, what is the record time for any package on any arch to have been on the list (or, what is the longest ever time any arch has had a non-empty list)? -- * Sufficiently advanced magic is indistinguishable from technology (T.P) * * PGP public key available @ http://www.iki.fi/killer * -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Goswin von Brederlow ([EMAIL PROTECTED]) [050314 15:35]: Andreas Barth [EMAIL PROTECTED] writes: * Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:45]: On Sun, Mar 13, 2005 at 11:16:56PM +0100, Andreas Barth wrote: Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. How often should the queue be emptied, or when will an architecture be declarared not-keeping-up? In light of http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html the release architecture must have N+1 buildds where N is the number required to keep up with the volume of uploaded packages at least once per day for etch. That means no more m68k. Given that some packages compile up to 12 days there will be plenty of times the queue doesn't empty once per day. needs-build can be empty even if packages _are_ currently building. I would like to see some stats showing on how many days in the last year an arch reached 0 needs-build. I highly doubt that any arch managed to do it every day troughout the last year. You know why goals are important? 0 needs-build is definitly a goal we should work to. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Monday 14 March 2005 15:31, Goswin von Brederlow wrote: Andreas Barth [EMAIL PROTECTED] writes: * Hamish Moffatt ([EMAIL PROTECTED]) [050314 01:45]: On Sun, Mar 13, 2005 at 11:16:56PM +0100, Andreas Barth wrote: Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. How often should the queue be emptied, or when will an architecture be declarared not-keeping-up? In light of http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html the release architecture must have N+1 buildds where N is the number required to keep up with the volume of uploaded packages at least once per day for etch. That means no more m68k. Given that some packages compile up to 12 days there will be plenty of times the queue doesn't empty once per day. Perhaps that was a slight misunderstanding: the Nybbles only require the release architecture must have N+1 buildds where N is the number required to keep up with the volume of uploaded packages with N = 2. The part about emptying once per day was only added by Andreas. Considering the effects of a twelve-day build of something big like KDE, GNOME or X: delays in security updates, shlib-deps, build-depends and testing migration, I can see the roots of the requirements on buildds. Regards, David -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst [EMAIL PROTECTED] writes: Op vr, 11-03-2005 te 19:14 -0800, schreef Steve Langasek: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - previous compilation state (already built packages are prioritized above packages never built for the target architecture) - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I agree with the w-b maintainers. The queue order is only interesting in the case where there is a backlog; in other cases, packages are usually built rather fast. In the case where there is a backlog, those trying to fix the architecture (usually those that are working on it) should be in charge of deciding what package gets built first, not the maintainer of a random package -- /all/ package builds are urgent if there's a backlog. Since you think an empty queue is normal why then fight changes to the queue order? If it is empty most of the time as you say the specific order hardly matters. Packages will be build within 15 minutes of their upload no matter what order as they will be the only packae in the queue. As you say, the oder is only relevant when there is a backlog and that is currently a badly starving algorithm. The problem is that a backlog is more and more the normal day to day business while an empty queue is rare. Obviously not for every arch but always some archs. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op ma, 14-03-2005 te 17:59 +0100, schreef Goswin von Brederlow: Wouter Verhelst [EMAIL PROTECTED] writes: Op vr, 11-03-2005 te 19:14 -0800, schreef Steve Langasek: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - previous compilation state (already built packages are prioritized above packages never built for the target architecture) - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I agree with the w-b maintainers. The queue order is only interesting in the case where there is a backlog; in other cases, packages are usually built rather fast. In the case where there is a backlog, those trying to fix the architecture (usually those that are working on it) should be in charge of deciding what package gets built first, not the maintainer of a random package -- /all/ package builds are urgent if there's a backlog. Since you think an empty queue is normal why then fight changes to the queue order? You misunderstood. I don't fight generic changes to the order; I just don't think it would be a good thing that any random developer could prioritize his pet package. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, 12 Mar 2005 14:42:33 +0100, Ingo Juergensmann [EMAIL PROTECTED] wrote: Been there, done that. The short answer: We don't want anyone else to play in our playground! The longer answer: More machines mean more work for the the buildd admin. Additional buildd admins for those archs are not wanted, because that would need communication between the buildd admins. Because w-b admins are the same persons as the buildd admin for certain archs which give regular problems, you can guess, where the real problem lies. Given that there is all this discussion about putting the responsibility for keeping up, security updates and releasing back onto the porters, maybe they should also be given the right to manage and setup as many autobuilders as they see fit. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
[remove -release, nothing they can do about it] Steve Langasek [EMAIL PROTECTED] writes: On Sat, Mar 12, 2005 at 03:01:28PM -0800, Thomas Bushnell BSG wrote: Goswin von Brederlow [EMAIL PROTECTED] writes: Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. This is news to me. It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. People should stop repeating the fiction then that just wait means your package will eventually get built. Er, packages *do* eventually get built; they just don't get built in any The only way to get the last package in the queue (zvbi or something) build is when the queue is empty. Unfortunately some archs are on the border of being too slow which means the time between empty queues is long. The effect of being (too) slow is that some few package will not be build for weeks/month instead of all packages being build a few days late as a simple FIFO would do. kind of FIFO order. I don't particularly care for this arrangement myself (it means there are plenty of times that a high-priority bug in a low-priority package stays on the release team's watchlist for far too long), but I don't have any proof that a different queue ordering would actually work better for the project, and the buildd admins *are* committed to keeping up with the queue even though hardware circumstances sometimes prevent it from time to time. -- Steve Langasek postmodern programmer Check http://unstable.buildd.net/buildd/Needs-Build_stats.png Queue not empty since: arm Feb 18th mipsel Feb 26th s390Mar 6th mipsMar 10th (possibly) That means (just an example) that an upload of zsh on Feb. 18th didn't get build at all on arm while the 5 uploads of ash all got build. Does that sound fair? Just because ash starts with an 'a' it gets prefered treatment? I see the point of sorting by priority to get the essentials build with priority in case of backlogs. I don't see a point in sorting by section as base is already done through priority and an automatic Dep-Wait would get libs build first. And last I feel the sorting by name is actualy harmfull. That should be exchanged with the time of upload, i.e. FIFO if the rest matches. We all know FIFO isn't the best but it is simple, fair, predictable and does not starve. I think it would avoid a lot of those Why am I stuck in the buildd queue? questions. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sun, Mar 13, 2005 at 05:19:25PM +0100, Goswin von Brederlow wrote: And last I feel the sorting by name is actualy harmfull. That should be exchanged with the time of upload, i.e. FIFO if the rest matches. We all know FIFO isn't the best but it is simple, fair, predictable and does not starve. I think it would avoid a lot of those Why am I stuck in the buildd queue? questions. I second that. Greetings Torsten signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Thomas Bushnell BSG ([EMAIL PROTECTED]) [050312 02:05]: Since the all but one of the other arch buildd's have empty needs-build queues, it is harmless to force them to execute a recompile and costs no scarce resources. I did check this before uploading. Even in the case the buildds otherwise idleing, it costs time of the buildd admin to check the log and sign the upload. If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. The ordering applied is (in this order, first difference wins) - = standard - not uncompiled - priority of the package - priority of the section - name That all is BTW visible from the code (or you could just ask). Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Thomas Bushnell BSG ([EMAIL PROTECTED]) [050312 04:25]: Steve Langasek [EMAIL PROTECTED] writes: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I see, the problem is clearly that gnucash is in the Extra priority instead of the Optional section where it belongs. I'll request the ftpmasters to change it. Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Matthew Palmer ([EMAIL PROTECTED]) [050312 05:25]: On Fri, Mar 11, 2005 at 07:14:35PM -0800, Steve Langasek wrote: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I'm trying to work out why package *section* matters at all. Package name is a bit odd, too, but including the section in there is just totally whack. Because we want packages in base to be preferred, as well as packages in libs. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Ingo Juergensmann ([EMAIL PROTECTED]) [050312 14:15]: On Sat, Mar 12, 2005 at 02:06:24PM +0100, Martin Zobel-Helas wrote: As it has been said earlier, the machines are back, but just need to catch up. So just waiting might be the easiest thing to do. More machines can catch up faster than few can do. When one machine out of a dozen machines is unavailable, it has less impact than one machine failure out of two machines, although the chances will raise that a machine will fail somewhen with more machines. But the impact is less critical... If you take a look at http://buildd.debian.org/stats/graph2-week-big.png, you can e.g. see for mipsel exactly at which date the slow and at which the fast machine became available again. The fast machine alone is able to keep up with the usual upload rates. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sun, Mar 13, 2005 at 11:25:33PM +0100, Andreas Barth wrote: More machines can catch up faster than few can do. When one machine out of a dozen machines is unavailable, it has less impact than one machine failure out of two machines, although the chances will raise that a machine will fail somewhen with more machines. But the impact is less critical... If you take a look at http://buildd.debian.org/stats/graph2-week-big.png, you can e.g. see for mipsel exactly at which date the slow and at which the fast machine became available again. The fast machine alone is able to keep up with the usual upload rates. And? Where's the conflicting part of what I've said? If you go back just long enough in time, you would see that there was a time when kullervo had no problems with keeping up at all. And then you see that arrakis helped kullervo in doing that job. And coming back to today, you'll notice that m68k has no problems with keeping up at all. Spot the differences to other archs yourself. BTW: don't forget to apply the passwords to your buildds for status updates. -- Ciao... // Ingo \X/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Matthew Palmer ([EMAIL PROTECTED]) [050313 01:05]: On Sat, Mar 12, 2005 at 03:12:12PM -0800, Steve Langasek wrote: Er, packages *do* eventually get built; they just don't get built in any kind of FIFO order. Er, no. Unless there's some sort of aging process (not yet described in the threads here) which will result in an extra package called zappa in section x11 from eventually being promoted above a package aardvark in section admin, it is entirely possible that package will never be built. All it requires is for the rate of new packages entering the queue before zappa to be equal to or greater than the rate of packages leaving the queue due to having been built or removed. If that happens for a too long period, we might consider such an architecture to be too slow to keep up, and will eventually discuss about kicking it out of the architectures we wait for testing migration at all, or even kicking it out of testing at all. Not waiting for such an arch has happened and might happen again. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sun, Mar 13, 2005 at 11:17:52PM +0100, Andreas Barth wrote: * Matthew Palmer ([EMAIL PROTECTED]) [050312 05:25]: On Fri, Mar 11, 2005 at 07:14:35PM -0800, Steve Langasek wrote: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I'm trying to work out why package *section* matters at all. Package name is a bit odd, too, but including the section in there is just totally whack. Because we want packages in base to be preferred, as well as packages in libs. I think I slightly misunderstood the ordering by section bit -- I was assuming an alphabetical ordering by section. So once base and libs have had their priority building, what's the ordering after that? Alphabetical by section, or does it go straight into a alphabetical by package name? - Matt signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 09:44:33AM +1100, Matthew Palmer wrote: On Sun, Mar 13, 2005 at 11:17:52PM +0100, Andreas Barth wrote: Because we want packages in base to be preferred, as well as packages in libs. I think I slightly misunderstood the ordering by section bit -- I was assuming an alphabetical ordering by section. So once base and libs have had their priority building, what's the ordering after that? Alphabetical by section, or does it go straight into a alphabetical by package name? There is a list in wanna-build for each section. The start of it looks like: libs debian-installer base devel shells perl python If you want to full list, look at the source. Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Andreas Barth [EMAIL PROTECTED] writes: Because we want packages in base to be preferred, as well as packages in libs. I think that is a given, but it's not uploads to base and libs that are hosing the recompilation of gnucash at present. I think it's worth looking at the perverse incentives that the current system offers. (A perverse incentive is one that rewards people for doing the wrong thing.) My top release priority is getting my own packages in shape, which means closing all release critical bugs and fixing all the important bugs which can be. Gnucash in stable and testing currently has a very serious RC bug which can cause massive data loss, *in an accounting application* where such data loss is all the more serious. My second release priority is doing what I can to fix RC bugs in other packages, and clean up and monitor the QA packages as best as I can. But I will not be fixing any RC bugs in other packages or making any QA uploads, because nearly every such package comes ahead of gnucash in the list, and not because they are base and libs, but because they are optional and gnucash has the wrong priority (extra). Any work I do on my second release priority will delay my top release priority. I believe that taking care of my own packages' bugs should be my top priority--as it should be for every DD--and if I do any uploads of other packages, it will delay that first priority. So the current system is creating a perverse incentive for me to sit on my hands, and only fix bugs in other packages once gnucash has *finally* gotten rebuilt, which may well be three months from the date the bug was fixed. The bug was reported January 21; I confirmed the bug, implemented the fix, and uploaded the fixed package the same day. This, it seems to me, is what should happen for such a dangerous bug. It is now nearly two months later, and the fix still isn't in testing. And I will do nothing to further hose gnucash users by delaying more the bug's entry into sarge. In the past two days, gnucash has slipped from being 90th on s390 to being 189th, and has moved essentially not at all on arm and mipsel. It may well be another month before it actually gets rebuilt. Any upload I do for any other package, any bug fix I suggest which some other maintainer uploads, further hoses my primary responsibility. I want a system where I can fix bugs in other packages while I'm waiting, and upload them, *without* it hosing my primary responsibility. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Andreas Barth ([EMAIL PROTECTED]) [050313 23:15]: * Thomas Bushnell BSG ([EMAIL PROTECTED]) [050312 02:05]: If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. The ordering applied is (in this order, first difference wins) - = standard - not uncompiled - priority of the package - priority of the section - name Which goes in front of that ordering is that all buildds check wanna-build in the order of the priority of the suites, i.e. security uploads first, than stable, ..., but only take packages not in noauto, and, if all queues are empty, also take up packages in noauto_weak in the same order (both are individual settings at each buildd). Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
* Matthew Palmer ([EMAIL PROTECTED]) [050313 23:50]: I think I slightly misunderstood the ordering by section bit -- I was assuming an alphabetical ordering by section. So once base and libs have had their priority building, what's the ordering after that? Alphabetical by section, or does it go straight into a alphabetical by package name? It is a highly ordered list, more or less libs+base first, than devel, shells, perl, python. After that graphics, admin, utils. Just to look at the other side of the sorting order, at the end of the list is hamradio, non-US and embedded. The large ones like gnome and kde are in the middle of the list. Please see wanna-builds source for the full list. Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sun, Mar 13, 2005 at 11:16:56PM +0100, Andreas Barth wrote: Our goal is that the queue gets empty from time to time, and so, priority shouldn't prevent a package from being built. How often should the queue be emptied, or when will an architecture be declarared not-keeping-up? According to http://people.debian.org/~igloo/needs-build-graph/index.php?days=30 arm hasn't been empty in over 3 weeks, and it's getting worse still. s390 is also rising steeply. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 12:01:59AM +0100, Andreas Barth wrote: It is a highly ordered list, more or less libs+base first, than devel, shells, perl, python. After that graphics, admin, utils. Just to look at the other side of the sorting order, at the end of the list is hamradio, non-US and embedded. The large ones like gnome and kde are in the middle of the list. Please see wanna-builds source for the full list. Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. Frankly there are not likely to be any users for hamradio on s390, mips* arm, or m68k. Nor electronics. Instead those architectures just prevent the migration to testing for those packages, for no good reason. I'm in favour of building all packages on all architectures (when time permits), and FTBFS bugs should be release-critical. But do we need to actually make those packages critical for release? For zero users? Having half your packages prioritised last in the build queue is rather insulting to be honest.. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Hamish Moffatt [EMAIL PROTECTED] writes: Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. I don't think those sections are causing the problem. There are also not so many uploads for them. The problem from my perspective is that optional packages all have to be built before any extra packages get considered. That's what's hosing gnucash. Removing a few sections also wouldn't help the perverse incentives that the current system offers. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Mon, Mar 14, 2005 at 11:49:34AM +1100, Hamish Moffatt wrote: Given how low hamradio (and the like) are prioritised, I suggest that we get smarter about 'tesing' and omit some sections on some architectures. Frankly there are not likely to be any users for hamradio on s390, mips* arm, or m68k. Nor electronics. Instead those architectures just prevent the migration to testing for those packages, for no good reason. How can archs prevent the migration when the software is already uptodate, f.e. ax25-tools? http://unstable.buildd.net/cgi/package_status?unstable_all_pkg=ax25-toolssearchtype=go I'm in favour of building all packages on all architectures (when time permits), and FTBFS bugs should be release-critical. But do we need to actually make those packages critical for release? For zero users? Let the user decide. But a user can only use a package, when it's there. Hence the name user... Having half your packages prioritised last in the build queue is rather insulting to be honest.. Instead of considering dropping archs or excluding archs from building, you should consider improving the buildd process. The current wanna-build is known to have many drawbacks. It's an ancient program that doesn't fit any longer on todays needs. Patching it to death doesn't help much, imho. http://www.buildd.net/files/Multibuild-Draft.pdf -- Ciao... // Ingo \X/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op za, 12-03-2005 te 16:24 -0800, schreef Thomas Bushnell BSG: Matthew Palmer [EMAIL PROTECTED] writes: Practically, buildd admins can notice a longer-than-usual queue and throw hardware at the problem, and that seems to work well enough, and we could reduce the rate of package inflow through various means, but the problem still remains -- the queue prioritisation *can* lead to starvation. I'm not advocating that it be on the top of anyone's todo list to fix it, because we have relatively effective workarounds, but it's not healthy to say the problem does not exist, either. What are these relatively effective workarounds? Ask people on port-specific mailinglists to manually build a package for you (not every Debian Developer on those are buildd admins). Build the package on the port machine available to all Debian Developers (no, that indeed doesn't work for all packages). Find someone with a machine of the target architecture and build it on there yourself. Etcetera. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op za, 12-03-2005 te 15:01 -0800, schreef Thomas Bushnell BSG: Goswin von Brederlow [EMAIL PROTECTED] writes: Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. This is news to me. It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. This paragraph assumes nobody ever looks the length of the needs-build queue of his architecture, and nobody feels bad when the queue is longer than normal. That idea would be hilarious if it wasn't sad. In reality, the upper bound is determined by motivated porters who try hard to avoid the queue ever to grow too long, day after day. People should stop repeating the fiction then that just wait means your package will eventually get built. Why, if it is true? -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op za, 12-03-2005 te 15:19 +1100, schreef Matthew Palmer: I'm trying to work out why package *section* matters at all. This is simply an attempt to avoid as much needs-build-building-dep-wait cycles as possible; packages that are usually build-dependencies are built before packages that are usually build-depending. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op vr, 11-03-2005 te 17:03 -0800, schreef Thomas Bushnell BSG: Steve Langasek [EMAIL PROTECTED] writes: Re-uploading a package to provoke a buildd response is counterproductive, *particularly* when the package is already in Needs-Build on the missing architectures. Re-uploading doesn't change its position in the queue, but it *does* force buildds for all the other archs to needlessly rebuild the package. This is why the answer to your previous email was please be patient. Unfortunately, the queue ordering policy is unclear. I was guessing that the priority of the upload would have something to do with queueing policy. As is explained on http://www.debian.org/devel/buildd/wanna-build-states, this is not the case. Since the all but one of the other arch buildd's have empty needs-build queues, it is harmless to force them to execute a recompile and costs no scarce resources. If everyone would reason like that, then it will cost scarce resources. This is a bogus argument. I did check this before uploading. I made an upload because a related package (grisbi) just seemed to get compiled by all the buildd's in a nifty two-day round trip time. You were lucky that time around, then. It was uploaded March 10, compiled by most buildds on the 10th, and by arm and mipsel on the 11th. I concluded that the queue must take note of priority or something like that. It does not. [...] If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. That indication is there, and it mainly boils down to 'buildd builds packages in a more or less predefined order which a maintainer has no direct influence on'. Of course, we can massage the ordering if required, but that is only done in exceptional cases. If your problem is 'my package will not migrate to testing!', then you are wrong, too. There are precedents for release managers forcibly moving packages to testing, even if the architectures are not in sync; there are precedents for an architecture with a huge backlog being temporarily ignored for the testing migration. In any case, re-uploading to force a package rebuild is /always/ the bad thing to do. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Op vr, 11-03-2005 te 19:14 -0800, schreef Steve Langasek: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - previous compilation state (already built packages are prioritized above packages never built for the target architecture) - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I agree with the w-b maintainers. The queue order is only interesting in the case where there is a backlog; in other cases, packages are usually built rather fast. In the case where there is a backlog, those trying to fix the architecture (usually those that are working on it) should be in charge of deciding what package gets built first, not the maintainer of a random package -- /all/ package builds are urgent if there's a backlog. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: buildd queue starvation (Was: Re: Do not make gratuitous source uploads just to provoke the buildds!)
Op za, 12-03-2005 te 21:12 -0800, schreef Thomas Bushnell BSG: Jeroen van Wolffelaar [EMAIL PROTECTED] writes: If the queue is non-zero for a longer time, there is a problem in buildd machine power, and the wanna-build admin has choosen to in this case allocate the buildd power that remains to the building of packages that are of higher priority, regardless of their age in the queue. The allocation of a scarce resource is almost by definition a trade-off, and this is the decision that has been made. First off, I think much confusion has been caused by using the word queue here. A queue is a FIFO list; if there isn't even the least bit FIFO in its management, which seems to be the case, then it shouldn't be called a queue. If it were not called a queue, I would not have made many wrong assumptions, and I think others too, to assume that of course some kind of FIFO processing was happening. So PLEASE change the name; stop calling it a queue. None of the documentation calls it a 'queue', in fact; only people not really involved in buildd stuff do. I can see excellent reasons why age in the list shouldn't matter. But package priority and section are extremely poor bases to decide what the actual importance of a package is. Why would that be the case? You're telling me you think gnome-games is way more important than gcc for us to build? -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst [EMAIL PROTECTED] writes: It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. This paragraph assumes nobody ever looks the length of the needs-build queue of his architecture, and nobody feels bad when the queue is longer than normal. That idea would be hilarious if it wasn't sad. There is no need-build queue; please don't call it that because it causes people to misunderstand it. Queue means FIFO, at least to some degree, where needs-build isn't FIFO in any degree. I have plenty of evidence that the buildd maintainers look at the length and feel bad when it's long. But looking at it and feeling bad aren't sufficient to get a package built. In reality, the upper bound is determined by motivated porters who try hard to avoid the queue ever to grow too long, day after day. I have no doubt about their good intentions and trying hard. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Wouter Verhelst [EMAIL PROTECTED] writes: I agree with the w-b maintainers. The queue order is only interesting in the case where there is a backlog; in other cases, packages are usually built rather fast. In the case where there is a backlog, those trying to fix the architecture (usually those that are working on it) should be in charge of deciding what package gets built first, not the maintainer of a random package -- /all/ package builds are urgent if there's a backlog. First, there is no queue. It's a list, but not a queue. I agree that we need policy here, but a serious problem is that currently when there is a backlog, maintainers of penalized packages have a perverse incentive not to fix any bugs in any *other* package because it will cause the penalty on their package to continue. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: buildd queue starvation (Was: Re: Do not make gratuitous source uploads just to provoke the buildds!)
Wouter Verhelst [EMAIL PROTECTED] writes: None of the documentation calls it a 'queue', in fact; only people not really involved in buildd stuff do. Does that include you? In two recent messages, you referred to it as a queue. I can see excellent reasons why age in the list shouldn't matter. But package priority and section are extremely poor bases to decide what the actual importance of a package is. Why would that be the case? You're telling me you think gnome-games is way more important than gcc for us to build? No. I'm saying that when priorities are marked badly, the results are disastrous. I'm also saying that a static ordering produces a perverse incentive, especially when a priority is marked badly, not to upload fixes for any other package. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
[please don't cc: me on this thread, one copy is plenty, thanks; and please don't cc: debian-release unless there's a specific reason it's on-topic there, which explaining wanna-build is not. ;)] On Sun, Mar 13, 2005 at 11:30:45PM +0100, Wouter Verhelst wrote: Op vr, 11-03-2005 te 17:03 -0800, schreef Thomas Bushnell BSG: [...] If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. That indication is there, and it mainly boils down to 'buildd builds packages in a more or less predefined order which a maintainer has no direct influence on'. Of course, we can massage the ordering if required, but that is only done in exceptional cases. If your problem is 'my package will not migrate to testing!', then you are wrong, too. There are precedents for release managers forcibly moving packages to testing, even if the architectures are not in sync; there are precedents for an architecture with a huge backlog being temporarily ignored for the testing migration. Yes, though we're generally avoiding pushing packages into testing right now because it's not clear that the t-p-u queue is picking up out-of-date packages for building, particularly for archs that are currently hardware strapped; so waiting on the package to build for all archs in unstable may be the *quickest* way to get the package in sync in testing. It's an ugly trade-off, but I think we've erred on the side of caution for long enough and will probably be more aggressive with buildd-stalled RC fixes going forward. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Fri, Mar 11, 2005 at 09:03:16PM -0800, Steve Langasek wrote: On Sat, Mar 12, 2005 at 03:19:23PM +1100, Matthew Palmer wrote: I'm trying to work out why package *section* matters at all. Package name is a bit odd, too, but including the section in there is just totally whack. Consider that libraries have their own section. Certain package sections are (on the whole) more central to the dependency graph than others, so it's to our advantage to order those first to reduce the need for give-backs or dep-waits. libs, devel and some others makes sense, but the others all have a defined order too according to recent discussion on debian-release. Of course this is never apparently when things are running smoothly with all the buildds. Is there anything that can be done to help with arm/mipsel? I think I read that new machines/owners aren't welcome currently anyway. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Hi Hamish, On Saturday, 12 Mar 2005, you wrote: Is there anything that can be done to help with arm/mipsel? ironic Not uploading any new packages *g* /ironic As it has been said earlier, the machines are back, but just need to catch up. So just waiting might be the easiest thing to do. Greetings Martin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, Mar 12, 2005 at 02:06:24PM +0100, Martin Zobel-Helas wrote: As it has been said earlier, the machines are back, but just need to catch up. So just waiting might be the easiest thing to do. More machines can catch up faster than few can do. When one machine out of a dozen machines is unavailable, it has less impact than one machine failure out of two machines, although the chances will raise that a machine will fail somewhen with more machines. But the impact is less critical... I can't understand why no additional machines are being accepted. -- Ciao... // Ingo \X/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Hi Ingo, On Saturday, 12 Mar 2005, you wrote: On Sat, Mar 12, 2005 at 02:06:24PM +0100, Martin Zobel-Helas wrote: As it has been said earlier, the machines are back, but just need to catch up. So just waiting might be the easiest thing to do. More machines can catch up faster than few can do. When one machine out of a dozen machines is unavailable, it has less impact than one machine failure out of two machines, although the chances will raise that a machine will fail somewhen with more machines. But the impact is less critical... I can't understand why no additional machines are being accepted. Please clarify this with the wanna-build admin(s). Greetings Martin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, Mar 12, 2005 at 02:26:43PM +0100, Martin Zobel-Helas wrote: As it has been said earlier, the machines are back, but just need to catch up. So just waiting might be the easiest thing to do. More machines can catch up faster than few can do. When one machine out of a dozen machines is unavailable, it has less impact than one machine failure out of two machines, although the chances will raise that a machine will fail somewhen with more machines. But the impact is less critical... I can't understand why no additional machines are being accepted. Please clarify this with the wanna-build admin(s). Been there, done that. The short answer: We don't want anyone else to play in our playground! The longer answer: More machines mean more work for the the buildd admin. Additional buildd admins for those archs are not wanted, because that would need communication between the buildd admins. Because w-b admins are the same persons as the buildd admin for certain archs which give regular problems, you can guess, where the real problem lies. -- Ciao... // Ingo \X/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Steve Langasek [EMAIL PROTECTED] writes: On Sat, Mar 12, 2005 at 03:19:23PM +1100, Matthew Palmer wrote: [Probably going a bit off track for -release; MFT to -devel] On Fri, Mar 11, 2005 at 07:14:35PM -0800, Steve Langasek wrote: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I'm trying to work out why package *section* matters at all. Package name is a bit odd, too, but including the section in there is just totally whack. Consider that libraries have their own section. Certain package sections are (on the whole) more central to the dependency graph than others, so it's to our advantage to order those first to reduce the need for give-backs or dep-waits. Build-Depends should be set automaticaly as Dep-Wait for every source upload. That would reduce a lot of needless work for both machines and admins. Upload priority would be nice to sort by, but I think the queue needs to be as FIFO as possible for fairness and principle of least surprise sake. I think package urgency isn't considered because people would abuse it to get their packages build faster, or so someone nameless fears. Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. Now I have this urge to go and make surgery on w-b. Yes please. Packages should Dep-Wait automatically, should be ordered by '(age * alpha) + beta' with alpha / beta set by at least the package priority and upload urgency. Optionaly also the number of Build-Depends and revers Build-Depends on the package. Given that the w-b maintainers disagree, changing the code is not the hard part. The system is designed such that it only really works properly if the buildds drain the Needs-Build queue on a regular basis. This doesn't seem terribly robust to me, but it's not my call. If you can convince the w-b admins to allow changes I could send you patches. Having a queue that will hapily starve packages is quite unfair to maintainers. Next you know x* will be renamed to a* just to get it build faster. -- Steve Langasek postmodern programmer MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Goswin von Brederlow [EMAIL PROTECTED] writes: Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. This is news to me. It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. People should stop repeating the fiction then that just wait means your package will eventually get built. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, Mar 12, 2005 at 03:01:28PM -0800, Thomas Bushnell BSG wrote: Goswin von Brederlow [EMAIL PROTECTED] writes: Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. This is news to me. It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. People should stop repeating the fiction then that just wait means your package will eventually get built. Er, packages *do* eventually get built; they just don't get built in any kind of FIFO order. I don't particularly care for this arrangement myself (it means there are plenty of times that a high-priority bug in a low-priority package stays on the release team's watchlist for far too long), but I don't have any proof that a different queue ordering would actually work better for the project, and the buildd admins *are* committed to keeping up with the queue even though hardware circumstances sometimes prevent it from time to time. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
Steve Langasek [EMAIL PROTECTED] writes: Er, packages *do* eventually get built; they just don't get built in any kind of FIFO order. This is not true. The current system has an unbounded wait time. For example, the effect of the Bug Squashing Party, which causes a bunch of uploads to be queued, forces extra priority packages to make no progress towards building, because all those optional packages shove right in ahead. This is true even when the extra priority package is fixing a severity critical bug, and the optional packages are fixing only, say, important bugs. As evidence, I note that gnucash is moving *backwards* in the queue, and as long as more uploads happen, it will continue to. Therefore, just wait doesn't actually mean that anything will happen. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, Mar 12, 2005 at 03:12:12PM -0800, Steve Langasek wrote: On Sat, Mar 12, 2005 at 03:01:28PM -0800, Thomas Bushnell BSG wrote: Goswin von Brederlow [EMAIL PROTECTED] writes: Remember that the buildd queue is not FIFO at all. The queue has a completly static order. Any changes to the queue are just packages hiding because they are not needs-build. I consider that the biggest flaw of all in wanna-build. This is news to me. It means that when one is told just wait, your package will get rebuilt; it is not necessarily true at all. There is no upper bound at all on time to wait for building, and that's a disaster. People should stop repeating the fiction then that just wait means your package will eventually get built. Er, packages *do* eventually get built; they just don't get built in any kind of FIFO order. Er, no. Unless there's some sort of aging process (not yet described in the threads here) which will result in an extra package called zappa in section x11 from eventually being promoted above a package aardvark in section admin, it is entirely possible that package will never be built. All it requires is for the rate of new packages entering the queue before zappa to be equal to or greater than the rate of packages leaving the queue due to having been built or removed. Practically, buildd admins can notice a longer-than-usual queue and throw hardware at the problem, and that seems to work well enough, and we could reduce the rate of package inflow through various means, but the problem still remains -- the queue prioritisation *can* lead to starvation. I'm not advocating that it be on the top of anyone's todo list to fix it, because we have relatively effective workarounds, but it's not healthy to say the problem does not exist, either. - Matt signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, Mar 12, 2005 at 02:06:24PM +0100, Martin Zobel-Helas wrote: Hi Hamish, On Saturday, 12 Mar 2005, you wrote: Is there anything that can be done to help with arm/mipsel? As it has been said earlier, the machines are back, but just need to catch up. So just waiting might be the easiest thing to do. arm doesn't appear to be catching up. New packages are being uploaded at least as quickly as they're being built. My package geda-gschem has not really changed its placement in the last week; actually I think it's slipped further down the list. Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Matthew Palmer [EMAIL PROTECTED] writes: Practically, buildd admins can notice a longer-than-usual queue and throw hardware at the problem, and that seems to work well enough, and we could reduce the rate of package inflow through various means, but the problem still remains -- the queue prioritisation *can* lead to starvation. I'm not advocating that it be on the top of anyone's todo list to fix it, because we have relatively effective workarounds, but it's not healthy to say the problem does not exist, either. What are these relatively effective workarounds? I recall recently hearing that the buildd admins don't want extra machines. So what then? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Sat, Mar 12, 2005 at 04:24:35PM -0800, Thomas Bushnell BSG wrote: Matthew Palmer [EMAIL PROTECTED] writes: Practically, buildd admins can notice a longer-than-usual queue and throw hardware at the problem, and that seems to work well enough, and we could reduce the rate of package inflow through various means, but the problem still remains -- the queue prioritisation *can* lead to starvation. I'm not advocating that it be on the top of anyone's todo list to fix it, because we have relatively effective workarounds, but it's not healthy to say the problem does not exist, either. What are these relatively effective workarounds? Not being a buildd admin, I have no idea as to the specifics, but I infer the existence of these workarounds due to the fact that, on the whole, the build queues don't appear to suffer from hideous starvation problems. I recall recently hearing that the buildd admins don't want extra machines. So what then? Presumably that was we don't want extra buildds administered by random people who we have no control over, a policy which has been discussed to death before. Nowhere have I seen a buildd admin say we're happy not having enough machines to keep up with demand. - Matt signature.asc Description: Digital signature
buildd queue starvation (Was: Re: Do not make gratuitous source uploads just to provoke the buildds!)
On Sun, Mar 13, 2005 at 11:43:41AM +1100, Matthew Palmer wrote: On Sat, Mar 12, 2005 at 04:24:35PM -0800, Thomas Bushnell BSG wrote: What are these relatively effective workarounds? Not being a buildd admin, I have no idea as to the specifics, but I infer the existence of these workarounds due to the fact that, on the whole, the build queues don't appear to suffer from hideous starvation problems. Apparantly an explanation is needed here: - normally, buildd power is more than new packages getting uploaded (otherwise, queue would only be growing, and never reduce) - as a consequence, normally the Needs-Build queue size will become zero frequently, only not if there are temporary buildd power issues The graph at [1] illustrates this nicely, you can see that for example somewhere around 15 feb 2005 all architecture's Needs-Build queue was zero, and that only s390, mipsel and arm have not reached zero for the last few days - if Needs-Build is zero, all packages that are in this state are build, including package z from x11-extra. As long as the queue does reach zero from time to time, there is no starvation. If the queue is non-zero for a longer time, there is a problem in buildd machine power, and the wanna-build admin has choosen to in this case allocate the buildd power that remains to the building of packages that are of higher priority, regardless of their age in the queue. The allocation of a scarce resource is almost by definition a trade-off, and this is the decision that has been made. --Jeroen [1] http://people.debian.org/~igloo/needs-build-graph/index.php?days=30 -- Jeroen van Wolffelaar [EMAIL PROTECTED] (also for Jabber MSN; ICQ: 33944357) http://Jeroen.A-Eskwadraat.nl
Re: buildd queue starvation (Was: Re: Do not make gratuitous source uploads just to provoke the buildds!)
Jeroen van Wolffelaar [EMAIL PROTECTED] writes: If the queue is non-zero for a longer time, there is a problem in buildd machine power, and the wanna-build admin has choosen to in this case allocate the buildd power that remains to the building of packages that are of higher priority, regardless of their age in the queue. The allocation of a scarce resource is almost by definition a trade-off, and this is the decision that has been made. First off, I think much confusion has been caused by using the word queue here. A queue is a FIFO list; if there isn't even the least bit FIFO in its management, which seems to be the case, then it shouldn't be called a queue. If it were not called a queue, I would not have made many wrong assumptions, and I think others too, to assume that of course some kind of FIFO processing was happening. So PLEASE change the name; stop calling it a queue. I can see excellent reasons why age in the list shouldn't matter. But package priority and section are extremely poor bases to decide what the actual importance of a package is. I think the three most critical factors are whether the package closes bugs, and the priority of the bugs it closes (counting all the bugs closed between the current unstable version for that arch and the upload being considered); the stated priority of the upload itself, whether low, medium, or high; and *particular* cases of section and priority. It makes sense to have Required and Standard packages go first; it makes sense to have libraries go first. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Steve Langasek [EMAIL PROTECTED] writes: Re-uploading a package to provoke a buildd response is counterproductive, *particularly* when the package is already in Needs-Build on the missing architectures. Re-uploading doesn't change its position in the queue, but it *does* force buildds for all the other archs to needlessly rebuild the package. This is why the answer to your previous email was please be patient. Unfortunately, the queue ordering policy is unclear. I was guessing that the priority of the upload would have something to do with queueing policy. Since the all but one of the other arch buildd's have empty needs-build queues, it is harmless to force them to execute a recompile and costs no scarce resources. I did check this before uploading. I made an upload because a related package (grisbi) just seemed to get compiled by all the buildd's in a nifty two-day round trip time. It was uploaded March 10, compiled by most buildds on the 10th, and by arm and mipsel on the 11th. I concluded that the queue must take note of priority or something like that. Perhaps it got through the queue because the grisbi upload fixed an serious RC bug. Well, the gnucash one fixes a critical RC bug, but that isn't indicated in the changelog. Maybe I should add that? If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Fri, Mar 11, 2005 at 05:03:55PM -0800, Thomas Bushnell BSG wrote: If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. You were *told*[1] to wait. Do not fall back to guesswork when someone knowingly like Steve Langasek tells you what to do. Ignoring what gets told to you by a release manager, and instead doing some uninformed guesswork is not helping the release. --Jeroen [1] http://lists.debian.org/debian-release/2005/03/msg00047.html -- Jeroen van Wolffelaar [EMAIL PROTECTED] (also for Jabber MSN; ICQ: 33944357) http://Jeroen.A-Eskwadraat.nl -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
Jeroen van Wolffelaar [EMAIL PROTECTED] writes: On Fri, Mar 11, 2005 at 05:03:55PM -0800, Thomas Bushnell BSG wrote: If, perhaps, there was a clear indication of the buildd ordering policy, then it could be properly used. Until then, I go on the basis of guesswork. You were *told*[1] to wait. Do not fall back to guesswork when someone knowingly like Steve Langasek tells you what to do. Ignoring what gets told to you by a release manager, and instead doing some uninformed guesswork is not helping the release. [1] http://lists.debian.org/debian-release/2005/03/msg00047.html You are misreading that message. In that message, Steve is advising me to wait for the buildd's to clean up the xfree86-common chroot mess, and saying not that a reupload does nothing, but instead is saying that further email to the buildd maintainers is unnecessary. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
On Fri, Mar 11, 2005 at 05:03:55PM -0800, Thomas Bushnell BSG wrote: Steve Langasek [EMAIL PROTECTED] writes: Re-uploading a package to provoke a buildd response is counterproductive, *particularly* when the package is already in Needs-Build on the missing architectures. Re-uploading doesn't change its position in the queue, but it *does* force buildds for all the other archs to needlessly rebuild the package. This is why the answer to your previous email was please be patient. Unfortunately, the queue ordering policy is unclear. I was guessing that the priority of the upload would have something to do with queueing policy. Yes, it is unclear. The reality is that upload priority does not contribute to queue ordering. There's been sufficient confusion on this point that I had to check the wanna-build code for myself to be sure of it, and I'm aware that confusion persists for many. Since the all but one of the other arch buildd's have empty needs-build queues, it is harmless to force them to execute a recompile and costs no scarce resources. I did check this before uploading. It is not harmless; it costs buildd admin time to review/process the build logs, and if the libraries available in unstable change on one or more architectures between the time the package was previously built and the next time it's built, there can be additional delay resulting from waiting on those other libraries to transition to testing. I made an upload because a related package (grisbi) just seemed to get compiled by all the buildd's in a nifty two-day round trip time. It was uploaded March 10, compiled by most buildds on the 10th, and by arm and mipsel on the 11th. I concluded that the queue must take note of priority or something like that. Perhaps it got through the queue because the grisbi upload fixed an serious RC bug. Well, the gnucash one fixes a critical RC bug, but that isn't indicated in the changelog. Maybe I should add that? The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: Do not make gratuitous source uploads just to provoke the buildds!
Steve Langasek [EMAIL PROTECTED] writes: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I see, the problem is clearly that gnucash is in the Extra priority instead of the Optional section where it belongs. I'll request the ftpmasters to change it. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Do not make gratuitous source uploads just to provoke the buildds!
[Probably going a bit off track for -release; MFT to -devel] On Fri, Mar 11, 2005 at 07:14:35PM -0800, Steve Langasek wrote: The queue ordering is entirely automatic, and AIUI the queue(s) is (are) sorted by: - target suite - package priority - package section - package name I personally believe it would be beneficial to prioritize by upload urgency as well (probably as a sort criterion between package priority and package section), but the w-b maintainers disagree. I'm trying to work out why package *section* matters at all. Package name is a bit odd, too, but including the section in there is just totally whack. Upload priority would be nice to sort by, but I think the queue needs to be as FIFO as possible for fairness and principle of least surprise sake. Now I have this urge to go and make surgery on w-b. - Matt signature.asc Description: Digital signature
