Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 03:01 PM, Nils Philippsen wrote: > On Tue, 2011-09-20 at 13:53 +0200, Johannes Lips wrote: >> What's wrong with all that broken deps? Is this just a missing rebuild >> against opencv and other libs or what's the reason for all this >> "mess". I mean the release of F16 is not that far away and the amount >> of broken deps is quite big imho. >> I would be glad helping out if this is due to some orphaned packages. > > Some of these seem to be a case of "new library version was built and > pushed as an update without the rebuilds of dependent components". It > seems to be unclear who is responsible for the dependents to be rebuilt > and included in the same update as the library being updated. When you have a closer look, you'll notice that such "mass rebuilts" were being delayed by QA's "delay queue" and now are stuck. > Currently > I only see mails of maintainers who plan updating the library, but the > rest of it pretty much depends on the maintainers of the depending > components rebuilding them quickly enough, and the original maintainer > to include them in the F-16 branched update. > > I'd like to see a discussion about how we can ensure -- within > reasonable limits -- that e.g. bumping a library's SONAME is followed by > dependent components being rebuilt and included with the providing > component in one update. I'd like to see a discussion on the proceedures currently being applied by QA, esp. "during freezes". IMO, they are unsuitable and harmful. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 03:19:17PM +0200, Ralf Corsepius wrote: > When you have a closer look, you'll notice that such "mass rebuilts" > were being delayed by QA's "delay queue" and now are stuck. Yeah. I rebuilt maatkit on the 1st of September and it still hasn't made it to the -stable repository. It's a mix of "it needs to stay in -testing for a week" and "no update pushes during Alpha/Beta preparation". Didn't we have the time an update had to stay in -testing changed to three days during the F15 stabilization phase? Could we implement this again for F16 to mitigate the issue? -- sven === jabber/xmpp: s...@lankes.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 15:01:06 +0200, Nils Philippsen wrote: > > I'd like to see a discussion about how we can ensure -- within > reasonable limits -- that e.g. bumping a library's SONAME is followed by > dependent components being rebuilt and included with the providing > component in one update. This should be easier to do now with the (relatively) new way to set up build overrides. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 9/20/11 9:19 AM, Ralf Corsepius wrote: >> Currently >> I only see mails of maintainers who plan updating the library, but the >> rest of it pretty much depends on the maintainers of the depending >> components rebuilding them quickly enough, and the original maintainer >> to include them in the F-16 branched update. >> >> I'd like to see a discussion about how we can ensure -- within >> reasonable limits -- that e.g. bumping a library's SONAME is followed by >> dependent components being rebuilt and included with the providing >> component in one update. > > I'd like to see a discussion on the proceedures currently being applied > by QA, esp. "during freezes". IMO, they are unsuitable and harmful. I'd like to see a rationale for jamming a soname-changing update into the OS so close to a release. In the absence of a very good motivation, that's not good engineering practice, and it's not consistent with the feature process. Perhaps you're not clear on what the word "freeze" means. - ajax -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
2011/9/20 Nils Philippsen : > On Tue, 2011-09-20 at 13:53 +0200, Johannes Lips wrote: >> What's wrong with all that broken deps? Is this just a missing rebuild >> against opencv and other libs or what's the reason for all this >> "mess". I mean the release of F16 is not that far away and the amount >> of broken deps is quite big imho. >> I would be glad helping out if this is due to some orphaned packages. > > Some of these seem to be a case of "new library version was built and > pushed as an update without the rebuilds of dependent components". It > seems to be unclear who is responsible for the dependents to be rebuilt > and included in the same update as the library being updated. Currently > I only see mails of maintainers who plan updating the library, but the > rest of it pretty much depends on the maintainers of the depending > components rebuilding them quickly enough, and the original maintainer > to include them in the F-16 branched update. I'm the maintainer of opencv here. quick answear: I have no right to submit a bodhi update for packages I do not own. Given that I'm no in the provenpackager group. So as I cannot expect every single maintainers to respond in time, the consequence is that I depend on a provenpackager to do the whole task of "administrative rebuilt of dependent packages". Unfortunately it became a way more complicated task with the collapse of two bodhi tickets and others unexpected behaviour. Nicolas (kwizart) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 03:47 PM, Bruno Wolff III wrote: > On Tue, Sep 20, 2011 at 15:01:06 +0200, >Nils Philippsen wrote: >> >> I'd like to see a discussion about how we can ensure -- within >> reasonable limits -- that e.g. bumping a library's SONAME is followed by >> dependent components being rebuilt and included with the providing >> component in one update. > > This should be easier to do now with the (relatively) new way to set up > build overrides. These are hardly applicable, if several maintainers are involved or if non-trivial technical difficulties arise. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 04:03 PM, Adam Jackson wrote: > On 9/20/11 9:19 AM, Ralf Corsepius wrote: > >>> Currently >>> I only see mails of maintainers who plan updating the library, but the >>> rest of it pretty much depends on the maintainers of the depending >>> components rebuilding them quickly enough, and the original maintainer >>> to include them in the F-16 branched update. >>> >>> I'd like to see a discussion about how we can ensure -- within >>> reasonable limits -- that e.g. bumping a library's SONAME is followed by >>> dependent components being rebuilt and included with the providing >>> component in one update. >> >> I'd like to see a discussion on the proceedures currently being applied >> by QA, esp. "during freezes". IMO, they are unsuitable and harmful. > > I'd like to see a rationale for jamming a soname-changing update into > the OS so close to a release. Maintainers on vacation, non-trivial changes? In my case, a major change was introduced into rawhide many weeks ago, which had caused breakage in rawhide. One maintainer being involved was in vacation, another one was non-responsive. Ca. 4 weeks later the issues were resolved in rawhide and we started to propagate these changes to f16 and where caught by the delay queues. > In the absence of a very good motivation, > that's not good engineering practice, and it's not consistent with the > feature process. > > Perhaps you're not clear on what the word "freeze" means. Perhaps you're not clear on what the word defective procedures means? This socalled QA now is testing non-installable rsp. obsolete packages. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 04:13:27PM +0200, Ralf Corsepius wrote: > On 09/20/2011 04:03 PM, Adam Jackson wrote: > > I'd like to see a rationale for jamming a soname-changing update into > > the OS so close to a release. > Maintainers on vacation, non-trivial changes? > > In my case, a major change was introduced into rawhide many weeks ago, > which had caused breakage in rawhide. One maintainer being involved was > in vacation, another one was non-responsive. > > Ca. 4 weeks later the issues were resolved in rawhide and we started to > propagate these changes to f16 and where caught by the delay queues. We're in the freeze for beta. It's not reasonable to push new sonames into the distribution at this point. -- Matthew Garrett | mj...@srcf.ucam.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 10:03 -0400, Adam Jackson wrote: > I'd like to see a rationale for jamming a soname-changing update into > the OS so close to a release. In the absence of a very good motivation, > that's not good engineering practice, and it's not consistent with the > feature process. > > Perhaps you're not clear on what the word "freeze" means. It may be worth pointing out that our release cycle has undergone significant changes with the early branching that we've introduced in F15. We align our releases closes to things like the GNOME release. That used to be fine in the old days, but now we have added early branching and made it considerably harder to push large sets of updates post-alpha, which makes it quite onerous to get e.g. GNOME updates into a branched Fedora release. We've set our freezes as if we expect all major development to be done at that point, but we've aligned our schedules in a way that guarantees that (at least for GNOME) major development is still happening at the time of branching. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 9/20/11 10:13 AM, Ralf Corsepius wrote: > On 09/20/2011 04:03 PM, Adam Jackson wrote: >> I'd like to see a rationale for jamming a soname-changing update into >> the OS so close to a release. > Maintainers on vacation, non-trivial changes? > > In my case, a major change was introduced into rawhide many weeks ago, > which had caused breakage in rawhide. One maintainer being involved was > in vacation, another one was non-responsive. That sounds like the kind of upheaval I'd want to keep out of a release that's in its stabilization phase. > Ca. 4 weeks later the issues were resolved in rawhide and we started to > propagate these changes to f16 and where caught by the delay queues. Of course, you had the option of not pulling the new OpenSceneGraph back to F16, or simply not doing so yet. Particularly during a phase where people are trying to keep change to a minimum so we know what we're working with. I'm sorry that you don't understand release engineering, but since you insist on not understanding it I don't really have any sympathy for your packages being so visibly at fault for breakages. - ajax -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 04:16 PM, Matthew Garrett wrote: > On Tue, Sep 20, 2011 at 04:13:27PM +0200, Ralf Corsepius wrote: >> On 09/20/2011 04:03 PM, Adam Jackson wrote: >>> I'd like to see a rationale for jamming a soname-changing update into >>> the OS so close to a release. >> Maintainers on vacation, non-trivial changes? >> >> In my case, a major change was introduced into rawhide many weeks ago, >> which had caused breakage in rawhide. One maintainer being involved was >> in vacation, another one was non-responsive. >> >> Ca. 4 weeks later the issues were resolved in rawhide and we started to >> propagate these changes to f16 and where caught by the delay queues. > > We're in the freeze for beta. It's not reasonable to push new sonames > into the distribution at this point. I disagree. A reasonable QA would strive to resolve the issues and apply the solution candidates others already have contributed. That said, a reasonable QA would cherry-pick such "solution candidates" from *-testing and integrate them. Simply flooding maintainers "with complaint mails" about broken deps, maintainers believe to already have fixed doesn't help anybody. Neither the testers (who can't test because of these broken deps), nor the maintainers (who believe to have done everything possible), nor the users (who will end up with low-quality distros). Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 10:21:52AM -0400, Matthias Clasen wrote: > We've set our freezes as if we expect all major development to be done > at that point, but we've aligned our schedules in a way that guarantees > that (at least for GNOME) major development is still happening at the > time of branching. That does seem like pretty fair criticism. We should probably discuss this for the F17 timeframe. -- Matthew Garrett | mj...@srcf.ucam.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 04:35:16PM +0200, Ralf Corsepius wrote: > That said, a reasonable QA would cherry-pick such "solution > candidates" from *-testing and integrate them. Simply flooding > maintainers "with complaint mails" about broken deps, maintainers > believe to already have fixed doesn't help anybody. Neither the > testers (who can't test because of these broken deps), nor the > maintainers (who believe to have done everything possible), nor the > users (who will end up with low-quality distros). What the maintainers could have done is not upload a package that breaks binary compatibility into a distribution that's attempting to stabalise for release. Really. Don't do that. -- Matthew Garrett | mj...@srcf.ucam.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 04:37 PM, Matthew Garrett wrote: > On Tue, Sep 20, 2011 at 04:35:16PM +0200, Ralf Corsepius wrote: > >> That said, a reasonable QA would cherry-pick such "solution >> candidates" from *-testing and integrate them. Simply flooding >> maintainers "with complaint mails" about broken deps, maintainers >> believe to already have fixed doesn't help anybody. Neither the >> testers (who can't test because of these broken deps), nor the >> maintainers (who believe to have done everything possible), nor the >> users (who will end up with low-quality distros). > > What the maintainers could have done is not upload a package that breaks > binary compatibility into a distribution that's attempting to stabalise > for release. That's a way too simplistic view - It's simply that other processes (upstream release cycles, upstream release processes, package maintainer's time slots, etc.) are not in sync with Fedora's cycles and that Fedora's wanna-be QA's delay slots are severely adding to the already existing problems. > Really. Don't do that. Really, your vision is impractical and non-applicable. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 04:52:28PM +0200, Ralf Corsepius wrote: > On 09/20/2011 04:37 PM, Matthew Garrett wrote: > >What the maintainers could have done is not upload a package that breaks > >binary compatibility into a distribution that's attempting to stabalise > >for release. > > That's a way too simplistic view - It's simply that other processes > (upstream release cycles, upstream release processes, package > maintainer's time slots, etc.) are not in sync with Fedora's cycles > and that Fedora's wanna-be QA's delay slots are severely adding to > the already existing problems. You're not obliged to upload the latest upstream. It's very practical to simply not do so. -- Matthew Garrett | mj...@srcf.ucam.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 4:57 PM, Matthew Garrett wrote: > On Tue, Sep 20, 2011 at 04:52:28PM +0200, Ralf Corsepius wrote: >> On 09/20/2011 04:37 PM, Matthew Garrett wrote: >> >What the maintainers could have done is not upload a package that breaks >> >binary compatibility into a distribution that's attempting to stabalise >> >for release. >> >> That's a way too simplistic view - It's simply that other processes >> (upstream release cycles, upstream release processes, package >> maintainer's time slots, etc.) are not in sync with Fedora's cycles >> and that Fedora's wanna-be QA's delay slots are severely adding to >> the already existing problems. > > You're not obliged to upload the latest upstream. It's very practical to > simply not do so. So when _is_ a good time to do binary-incompatible changes to libraries? * It's not after beta freeze, because they are unwanted at that time * It's not 14 days before beta freeze, because they won't get out of updates-testing in time * It's not 14 days + 3 (4?) weeks before beta freeze - even if the library gets out of updates-testing in time, its users may not be rebuilt because the maintainer is on vacation. * What if there are two layers of users that need to be rebuilt? The delays just pile one upon another... Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 16:06 +0200, Nicolas Chauvet wrote: > 2011/9/20 Nils Philippsen : > > On Tue, 2011-09-20 at 13:53 +0200, Johannes Lips wrote: > >> What's wrong with all that broken deps? Is this just a missing rebuild > >> against opencv and other libs or what's the reason for all this > >> "mess". I mean the release of F16 is not that far away and the amount > >> of broken deps is quite big imho. > >> I would be glad helping out if this is due to some orphaned packages. > > > > Some of these seem to be a case of "new library version was built and > > pushed as an update without the rebuilds of dependent components". It > > seems to be unclear who is responsible for the dependents to be rebuilt > > and included in the same update as the library being updated. Currently > > I only see mails of maintainers who plan updating the library, but the > > rest of it pretty much depends on the maintainers of the depending > > components rebuilding them quickly enough, and the original maintainer > > to include them in the F-16 branched update. > I'm the maintainer of opencv here. > > quick answear: I have no right to submit a bodhi update for packages I > do not own. Given that I'm no in the provenpackager group. > So as I cannot expect every single maintainers to respond in time, the > consequence is that I depend on a provenpackager to do the whole task > of "administrative rebuilt of dependent packages". > Unfortunately it became a way more complicated task with the collapse > of two bodhi tickets and others unexpected behaviour. I wasn't picking on you, or anyone for what it's worth. It's just a situation I've seen often enough so that I think it deserves some kind of a solution, at least for the majority of cases where this shouldn't be too much of a hassle. With "responsible" I didn't mean that this person needs to do all of the work either. Ordinary (non-proven-)packagers need to be part of the picture. Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
2011/9/20 Miloslav Trmač : > On Tue, Sep 20, 2011 at 4:57 PM, Matthew Garrett wrote: >> On Tue, Sep 20, 2011 at 04:52:28PM +0200, Ralf Corsepius wrote: >>> On 09/20/2011 04:37 PM, Matthew Garrett wrote: >>> >What the maintainers could have done is not upload a package that breaks >>> >binary compatibility into a distribution that's attempting to stabalise >>> >for release. >>> >>> That's a way too simplistic view - It's simply that other processes >>> (upstream release cycles, upstream release processes, package >>> maintainer's time slots, etc.) are not in sync with Fedora's cycles >>> and that Fedora's wanna-be QA's delay slots are severely adding to >>> the already existing problems. >> >> You're not obliged to upload the latest upstream. It's very practical to >> simply not do so. > > So when _is_ a good time to do binary-incompatible changes to libraries? > > * It's not after beta freeze, because they are unwanted at that time > > * It's not 14 days before beta freeze, because they won't get out of > updates-testing in time > > * It's not 14 days + 3 (4?) weeks before beta freeze - even if the > library gets out of updates-testing in time, its users may not be > rebuilt because the maintainer is on vacation. > > * What if there are two layers of users that need to be rebuilt? > > The delays just pile one upon another... You can update rawhide at any time and accomplish that work without delays. Then it shows up in the next Fedora version. josh -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > I'd like to see a rationale for jamming a soname-changing update into > the OS so close to a release. In the absence of a very good > motivation, > that's not good engineering practice, and it's not consistent with > the > feature process. > > Perhaps you're not clear on what the word "freeze" means. One rationale is that if we don't get it *before* the release when everyone is actively testing, then it ends up going in post release, likely with far less testing, and risks destabilizing the already released product. Unless we want to change the Fedora update policy that updates such as this are not allowed after the product goes GA, then there is an argument to be made that before GA when people are testing is better than after GA when testing isn't so widespread (the counter argument being that we need to stabilize what we have, and then deal with destabilizing changes in later updates because if we don't pick some line in the sand to call a stabilization point, then destabilizing changes will never end). However, if a group were to take this approach, then the entire CRITPATH and normal update process for an early branched release is totally backwards from what it should be. If we were to say that we want the stuff in early instead of post-GA, then on an early branched process things should go direct to stable without hitting testing at all on the theory that getting it in the most hands as quickly as possible maximizes testing prior to the product going GA. Yes, it might destabilize the early branched release momentarily, but without anything blocking a fix from being pushed right on out, the iterative "push, test, report breakage, fix, push, test" cycle goes much faster. Note: I'm not putting my $.02 in towards either side, just playing devil's advocate. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 9/20/11 11:10 AM, Miloslav Trmač wrote: > * It's not 14 days + 3 (4?) weeks before beta freeze - even if the > library gets out of updates-testing in time, its users may not be > rebuilt because the maintainer is on vacation. You could have an earthquake, too. If you're having problems rebuilding packages downstream from you, get a provenpackager's help. I've done a few dozen such rebuilds for soname changes over the course of F16 alone. Of course, the accounts system _still_ doesn't have groups, five years later, so provenpackager is the big hammer we have. We could get groups any day now, that'd be just fine. - ajax -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 16:07 +0200, Ralf Corsepius wrote: > On 09/20/2011 03:47 PM, Bruno Wolff III wrote: > > On Tue, Sep 20, 2011 at 15:01:06 +0200, > >Nils Philippsen wrote: > >> > >> I'd like to see a discussion about how we can ensure -- within > >> reasonable limits -- that e.g. bumping a library's SONAME is followed by > >> dependent components being rebuilt and included with the providing > >> component in one update. > > > > This should be easier to do now with the (relatively) new way to set up > > build overrides. > > These are hardly applicable, if several maintainers are involved or if > non-trivial technical difficulties arise. Well, easy build overrides are helping getting the technical side done without involving rel-eng. Several maintainers being involved and non-trivial technical difficulties are exactly the topics that need to be sorted out, but they don't have a thing to do with who sets up the build overrides. Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 11:33 -0400, Adam Jackson wrote: > Of course, the accounts system _still_ doesn't have groups, five years > later, so provenpackager is the big hammer we have. We could get groups > any day now, that'd be just fine. Do you mean "groups of groups", like in "provenpackager-kde", "provenpackager-gnome", and "provenpackager" means all of these? Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > So when _is_ a good time to do binary-incompatible changes to > libraries? > > * It's not after beta freeze, because they are unwanted at that time > > * It's not 14 days before beta freeze, because they won't get out of > updates-testing in time > > * It's not 14 days + 3 (4?) weeks before beta freeze - even if the > library gets out of updates-testing in time, its users may not be > rebuilt because the maintainer is on vacation. > > * What if there are two layers of users that need to be rebuilt? > > The delays just pile one upon another... >Mirek I'd like to expand on my previous email (the one where I played devil's advocate) and pick up where Mirek left off here. I'm fine with our new early branched methodology, to a point. I think it's a good idea that we do an early branch and separate our upcoming release from rawhide. However, I think the process we use to get packages from updates-testing into the actual GA candidate tree is the problem. I think we are gating package updates on the wrong thing: testing. I say this because the *real* testing happens with the alpha/beta/rc candidate install images, not with individual testers pulling packages from updates-testing. And when trying to stabilize a product for GA, you want *everyone* testing the updates, not just a few. Instead, I think we ought to revamp the process like this: Maintainer A builds new package B Maintainer A files a bodhi ticket for package B In that ticket, the maintainer is responsible for list each item of change from the previous package already in the compose tree. For example, did the upstream source get bumped, did any new patches get applied, did any old patches stop being applied, are the changes verified bug fixes as tested in rawhide, are the changes isolated or are there feature additions as well, will this update create dependency problems from things such as an soname bump, will other packages need to be rebuilt. Finally, the bodhi update should be reviewed by people from release engineering, and if the ticket meets the requirements of a reasonable change at this late stage of the game, the ticket should be approved and the package pushed to stable. The point of this process is that when stabilizing the product for GA, we want to minimize unnecessary or risky churn, and that doesn't need testing, that needs a human to make a judgement call. Then, once the judgement call is made, we need as much testing as possible to make the release as good as possible. Holding things up in updates-testing and then releasing them later throws away untold instances of testing, wasting those resources on a package that may never be on the final product. We need to make that judgement call, put the package in if we are going to, test the hell out of it, and fix any breakage we find. We need this iterative "test, report breakage, fix, retest" process to be as fast as possible, and our current process moves at the speed of a salt coated slug. That's my proposed process for our early branched release. Thoughts? -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 15:19 +0200, Ralf Corsepius wrote: > On 09/20/2011 03:01 PM, Nils Philippsen wrote: > > On Tue, 2011-09-20 at 13:53 +0200, Johannes Lips wrote: > >> What's wrong with all that broken deps? Is this just a missing rebuild > >> against opencv and other libs or what's the reason for all this > >> "mess". I mean the release of F16 is not that far away and the amount > >> of broken deps is quite big imho. > >> I would be glad helping out if this is due to some orphaned packages. > > > > Some of these seem to be a case of "new library version was built and > > pushed as an update without the rebuilds of dependent components". It > > seems to be unclear who is responsible for the dependents to be rebuilt > > and included in the same update as the library being updated. > > When you have a closer look, you'll notice that such "mass rebuilts" > were being delayed by QA's "delay queue" and now are stuck. I didn't want to (re)start that particular discussion ;-). We need some guidelines who should drive rebuilds in such a situation, regardless of whether it happens in Rawhide or Branched or wherever. Otherwise we'll end up with nobody doing the driving. Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 11:45 -0400, Doug Ledford wrote: > - Original Message - > > So when _is_ a good time to do binary-incompatible changes to > > libraries? > > > > * It's not after beta freeze, because they are unwanted at that time > > > > * It's not 14 days before beta freeze, because they won't get out of > > updates-testing in time > > > > * It's not 14 days + 3 (4?) weeks before beta freeze - even if the > > library gets out of updates-testing in time, its users may not be > > rebuilt because the maintainer is on vacation. > > > > * What if there are two layers of users that need to be rebuilt? > > > > The delays just pile one upon another... > >Mirek > > I'd like to expand on my previous email (the one where I played > devil's advocate) and pick up where Mirek left off here. > > I'm fine with our new early branched methodology, to a point. I think > it's a good idea that we do an early branch and separate our upcoming > release from rawhide. However, I think the process we use to get > packages from updates-testing into the actual GA candidate tree is the > problem. > > I think we are gating package updates on the wrong thing: testing. I > say this because the *real* testing happens with the alpha/beta/rc > candidate install images, not with individual testers pulling packages > from updates-testing. And when trying to stabilize a product for GA, > you want *everyone* testing the updates, not just a few. > > Instead, I think we ought to revamp the process like this: > > Maintainer A builds new package B > Maintainer A files a bodhi ticket for package B > In that ticket, the maintainer is responsible for list each item of > change from the previous package already in the compose tree. For > example, did the upstream source get bumped, did any new patches get I'd like to mention that an upstream source getting bumped doesn't mean anything per se, so we should rather have criteria agnostic of arbitrary parameters like this. For instance, it shouldn't make a shred of difference whether I apply a patch in the spec file, or upstream, all other things being the same (i.e. if tarball-1.0 + patch == tarball-1.1 + changes we want to have anyway like updated translations). > applied, did any old patches stop being applied, are the changes > verified bug fixes as tested in rawhide, are the changes isolated or > are there feature additions as well, will this update create > dependency problems from things such as an soname bump, will other > packages need to be rebuilt. ^^ This. > Finally, the bodhi update should be reviewed by people from release > engineering, and if the ticket meets the requirements of a reasonable > change at this late stage of the game, the ticket should be approved > and the package pushed to stable. > > The point of this process is that when stabilizing the product for GA, > we want to minimize unnecessary or risky churn, and that doesn't need > testing, that needs a human to make a judgement call. Then, once the > judgement call is made, we need as much testing as possible to make > the release as good as possible. Holding things up in updates-testing > and then releasing them later throws away untold instances of testing, > wasting those resources on a package that may never be on the final > product. We need to make that judgement call, put the package in if > we are going to, test the hell out of it, and fix any breakage we > find. We need this iterative "test, report breakage, fix, retest" > process to be as fast as possible, and our current process moves at > the speed of a salt coated slug. > > That's my proposed process for our early branched release. Thoughts? Looks like it would get things done. Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 9/20/11 11:43 AM, Nils Philippsen wrote: > On Tue, 2011-09-20 at 11:33 -0400, Adam Jackson wrote: >> Of course, the accounts system _still_ doesn't have groups, five years >> later, so provenpackager is the big hammer we have. We could get groups >> any day now, that'd be just fine. > > Do you mean "groups of groups", like in "provenpackager-kde", > "provenpackager-gnome", and "provenpackager" means all of these? I don't see any real reason to do that instead of just unix-style groups, but either one would be an improvement. - ajax -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > I'd like to mention that an upstream source getting bumped doesn't > mean > anything per se, so we should rather have criteria agnostic of > arbitrary > parameters like this. For instance, it shouldn't make a shred of > difference whether I apply a patch in the spec file, or upstream, all > other things being the same (i.e. if tarball-1.0 + patch == > tarball-1.1 > + changes we want to have anyway like updated translations). I agree. Each source update would have to be justified. I know I've done a source tarball update when it was just a roll up bug fix release. A source update that implements new features is another issue. The maintainer is in the best position to know this and can note the distinction in the bodhi ticket. > Looks like it would get things done. That's what I thought ;-) -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
Le mardi 20 septembre 2011 à 17:10 +0200, Miloslav Trmač a écrit : > So when _is_ a good time to do binary-incompatible changes to libraries? The answer is obvious - in rawhide, before branching point. Anything after branching will interact with various groups schedules and crash into the barriers put in place to try to bring some order to the release. Now of course that supposes rawhide is kept in dogfoodable state and people don't let problems fester there because 'it eats babies, so who cares' -- Nicolas Mailhot -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
>> * What if there are two layers of users that need to be rebuilt? >> >> The delays just pile one upon another... > > You can update rawhide at any time and accomplish that work without > delays. Then it shows up in the next Fedora version. > Yes, but then we have align the schedules, so have a new gnome release in good time before a new fedora release tree is forked and when a new Fedora GA is released, it will be close to the next Gnome release and Fedora will not be the latest and greatest. As an example i was hit by the latest gnome 3.2 pre-release in Rawhide and F16 in awn-extras-applets, there contains a lot of applets to awn written in C, Python & Vala with a lot of different requirements to all kind of different gnome stuff (fx. gnome-menu 2.0) this was bumped to 3.0 and some python binding disappeared. It will take me 2-3 weeks before figure out how to work around the issues, remove stuff there can't be fixed and get into updates-testing and to stable. Lucky for me, nothing depends on awn-extras-applets, but users will have problems updating to the new gnome packages without removing the awn-extras-applets package. This is not ideal between alpha and beta, but it the way things goes, if we want to have the latest gnome close to the fedora release. Tim -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
Am Dienstag, den 20.09.2011, 15:39 +0200 schrieb Sven Lankes: > Didn't we have the time an update had to stay in -testing changed to > three days during the F15 stabilization phase? Could we implement this > again for F16 to mitigate the issue? I think we should. Please file a bug against bodhi because it should be at 3 days for F16 already. Regards, Christoph -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
Am Dienstag, den 20.09.2011, 16:06 +0200 schrieb Nicolas Chauvet: > I'm the maintainer of opencv here. > > quick answear: I have no right to submit a bodhi update for packages I > do not own. Given that I'm no in the provenpackager group. > So as I cannot expect every single maintainers to respond in time, the > consequence is that I depend on a provenpackager to do the whole task > of "administrative rebuilt of dependent packages". > Unfortunately it became a way more complicated task with the collapse > of two bodhi tickets and others unexpected behaviour. IHMO the proper way to deal with this is: 1. Mail fedora-devel and owners of dependent packages (at least) one week in advance. This is a requirement and written down in our wiki [1]. repoquery and the foo-owner aliases should help here. 2. Ask maintainers if they are ok with the update and willing/able to do a rebuild in time. Offer to rebuild packages if people are not able to do it. Request the necessary commit access in packagedb. 3. Once you have sufficient feedback, update opencv. 4. Submit a buildroot overwrite for opencv but do not push it to stable or testing. 5. Mail owners again and tell them they can now rebuild their packages 6. Wait for feedback before you create an update. If you have commit access, you can include dependent packages in the update. Proven packager will not work. 7. Mail owners again when you push the update from one tag to another. Regards, Christoph [1] http://fedoraproject.org/wiki/Package_maintainer_responsibilities#Notify_others_of_changes_that_may_affect_their_packages -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 08:19 PM, Nicolas Mailhot wrote: > Le mardi 20 septembre 2011 à 17:10 +0200, Miloslav Trmač a écrit : > >> So when _is_ a good time to do binary-incompatible changes to libraries? > > The answer is obvious - in rawhide, before branching point. Anything > after branching will interact with various groups schedules and crash > into the barriers put in place to try to bring some order to the > release. > > Now of course that supposes rawhide is kept in dogfoodable state and > people don't let problems fester there because 'it eats babies, so who > cares' Also related to rawhide dogfoodability is this recent trend where after a branch point, "all" work appears switch to the branched distro, resulting in branches having newer packages than rawhide and such. This was very visible at least after F15 branching, this time around I haven't paid enough attention to comment whether its the same now. The effect of this is that the "active rawhide window" *seems* awfully short these days, because rawhide is relatively quiet for large number of packages during branced state. That's not how the branching procedure intends this to work, but that's how it seems to go in practise to a large extent. My personal pet-peeve with the current branching policy is that the mass-branching happens way way too early for packages where there are no significant new development to be introduced in rawhide during branched state. So for every single tiny fix that needs to go in, it needs to be built into rawhide and also branched. Minor annoyance maybe but annoying things tend to get negletted - perhaps this is one of the reasons for rawhide lagging behind branched. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Sep 20, 2011, at 8:45 AM, Doug Ledford wrote: > > Instead, I think we ought to revamp the process like this: > > Maintainer A builds new package B > Maintainer A files a bodhi ticket for package B > In that ticket, the maintainer is responsible for list each item of change > from the previous package already in the compose tree. For example, did the > upstream source get bumped, did any new patches get applied, did any old > patches stop being applied, are the changes verified bug fixes as tested in > rawhide, are the changes isolated or are there feature additions as well, > will this update create dependency problems from things such as an soname > bump, will other packages need to be rebuilt. > Finally, the bodhi update should be reviewed by people from release > engineering, and if the ticket meets the requirements of a reasonable change > at this late stage of the game, the ticket should be approved and the package > pushed to stable. > > The point of this process is that when stabilizing the product for GA, we > want to minimize unnecessary or risky churn, and that doesn't need testing, > that needs a human to make a judgement call. Then, once the judgement call > is made, we need as much testing as possible to make the release as good as > possible. Holding things up in updates-testing and then releasing them later > throws away untold instances of testing, wasting those resources on a package > that may never be on the final product. We need to make that judgement call, > put the package in if we are going to, test the hell out of it, and fix any > breakage we find. We need this iterative "test, report breakage, fix, > retest" process to be as fast as possible, and our current process moves at > the speed of a salt coated slug. > > That's my proposed process for our early branched release. Thoughts? This is essentially what we had a while ago, only with trac tickets instead of bodhi requests. There were a couple of problems with this. 1) Nowhere near enough releng folks to properly review all the tickets. 2) 9 times out of 10 there was very little data put into the ticket. 3) releng folks were often not the best people to decide whether a change was "too risky" 4) There was no easy way to get at the package and assess its stability. I think there were more issues, but those come to mind first. We decided it was best instead to make a repository out of proposed changes, and let your packaging peers decide whether or not the update was safe enough to go into the release, thus we have bodhi and updates-testing as a gateway to get into the release. - jlk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Sep 20, 2011, at 11:11 AM, Panu Matilainen wrote: > > My personal pet-peeve with the current branching policy is that the > mass-branching happens way way too early for packages where there are no > significant new development to be introduced in rawhide during branched > state. So for every single tiny fix that needs to go in, it needs to be > built into rawhide and also branched. Minor annoyance maybe but annoying > things tend to get negletted - perhaps this is one of the reasons for > rawhide lagging behind branched. This isn't quite correct. If you haven't built anything explicitly for Rawhide since the branch point, the stable packages from the branched repo will be inherited into rawhide. You still should merge your changes from the branch up to master (for rawhide) but there is no reason to do a build. Let the build system inheritance take care of that. One change to make this better might be to move the inheritance point to updates-testing so that things built from the fresh branch are immediately inherited into rawhide. - jlk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 09:18 PM, Jesse Keating wrote: > On Sep 20, 2011, at 11:11 AM, Panu Matilainen wrote: >> >> My personal pet-peeve with the current branching policy is that the >> mass-branching happens way way too early for packages where there are no >> significant new development to be introduced in rawhide during branched >> state. So for every single tiny fix that needs to go in, it needs to be >> built into rawhide and also branched. Minor annoyance maybe but annoying >> things tend to get negletted - perhaps this is one of the reasons for >> rawhide lagging behind branched. > > This isn't quite correct. If you haven't built anything explicitly > forRawhide since the branch point, the stable packages from the branched > repo will be inherited into rawhide. > > You still should merge your changes from the branch up to master > (forrawhide) but there is no reason to do a build. Let the build system > inheritance take care of that. Oh, I certainly wasn't aware of that. And I would've expected this to work the other way around (because doing new work in a branch instead of rawhide just feels wrong to me, but there's obviously a point to doing it this way), but good to know, thanks. > One change to make this better might be to move the inheritance > pointto updates-testing so that things built from the fresh branch are > immediately inherited into rawhide. As long as it's rawhide inheriting from branched, that sounds like a winner to me. Who knows, might even get those test-updates a bit of additional testing. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > This is essentially what we had a while ago, only with trac tickets > instead of bodhi requests. Bodhi is definitely a better place to track this stuff, regardless of how decisions are made. > There were a couple of problems with > this. > > 1) Nowhere near enough releng folks to properly review all the > tickets. Fair enough. > 2) 9 times out of 10 there was very little data put into the ticket. Multiple options here. Kick back incomplete tickets, or the better option IMNSHO, run rpmdiff runs between the package currently in the compose and the one in testing to check for various failures and require the developer to justify failures. > 3) releng folks were often not the best people to decide whether a > change was "too risky" The rpmdiff option above would help with this. > 4) There was no easy way to get at the package and assess its > stability. Using bodhi instead of trac solves this, no? > I think there were more issues, but those come to mind first. We > decided it was best instead to make a repository out of proposed > changes, But in practice that's not really what updates-testing on the early branched release really is. It's a repo all right, but not of proposed changes, it's a repo of packages, and getting to the actual changes versus the final package would require installing the current source rpm, the new source rpm, then doing a manual inspection for changes. An automated rpmdiff run would be a *far* superior means of presenting the proposed changes to the community. > and let your packaging peers decide whether or not the > update was safe enough to go into the release, But they can't, not really. For instance, a proventester may install my mdadm package, but if they don't have a raid array, they can't decide anything. Where as if they had access to the code diffs and could tell that all I did was change a typo in the udev rules file, and verify for themselves via code inspection that the code as it stands in the repo can't work and the fix should work, then they could make an educated contribution. Because the testing repo doesn't really present changes, but only a final product, there is no possibility for my peers to look at my changes and say "Yeah, that's should be a safe change, let it in, and if it breaks then we'll fix it". So the judgement call that I mentioned in my previous email is taken entirely out of the loop, and there is no ability for my peers to make any contribution to whether or not a package should be allowed in *except* unit testing, there is no ability for them to easily do an actual analysis of the changes and ma ke an engineering decision versus a QA decision. > thus we have bodhi > and updates-testing as a gateway to get into the release. It's a gateway, I just don't think it serves as useful a purpose as it was intended to. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
2011/9/20 Christoph Wickert : > Am Dienstag, den 20.09.2011, 16:06 +0200 schrieb Nicolas Chauvet: >> I'm the maintainer of opencv here. >> >> quick answear: I have no right to submit a bodhi update for packages I >> do not own. Given that I'm no in the provenpackager group. >> So as I cannot expect every single maintainers to respond in time, the >> consequence is that I depend on a provenpackager to do the whole task >> of "administrative rebuilt of dependent packages". >> Unfortunately it became a way more complicated task with the collapse >> of two bodhi tickets and others unexpected behaviour. > > IHMO the proper way to deal with this is: > 1. Mail fedora-devel and owners of dependent packages (at least) > one week in advance. This is a requirement and written down in > our wiki [1]. repoquery and the foo-owner aliases should help > here. > 2. Ask maintainers if they are ok with the update and willing/able > to do a rebuild in time. Offer to rebuild packages if people are > not able to do it. Request the necessary commit access in > packagedb. > 3. Once you have sufficient feedback, update opencv. > 4. Submit a buildroot overwrite for opencv but do not push it to > stable or testing. > 5. Mail owners again and tell them they can now rebuild their > packages > 6. Wait for feedback before you create an update. If you have > commit access, you can include dependent packages in the update. > Proven packager will not work. > 7. Mail owners again when you push the update from one tag to > another. You missed the point, this process was started 3 weeks away from now with the help of rdieter since then. The problem is about this legitimate update that was truncated by some bodhi ACL weirdness I didn't expected. https://admin.fedoraproject.org/updates/FEDORA-2011-12320 In others words, I've pushed this ticket to stable but only packages I actually own was pushed to dist-16, others packages in this tickets went back to limbo. This unexpected behavior evidently broke dependencies. While submitting this update to stable I was even informed that packages I didn't own was tagged as dist-16. This didn't take into. Nicolas (kwizart) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 05:52 PM, Nils Philippsen wrote: > On Tue, 2011-09-20 at 15:19 +0200, Ralf Corsepius wrote: >> When you have a closer look, you'll notice that such "mass rebuilts" >> were being delayed by QA's "delay queue" and now are stuck. > > I didn't want to (re)start that particular discussion ;-). > > We need some guidelines who should drive rebuilds in such a situation, > regardless of whether it happens in Rawhide or Branched or wherever. a) These situation can only happen in release branches. b) To me, Fedora is like coping with "German Tax Laws". We don't need more regulations/guidelines, we need a fundamental change of the tax system. > Otherwise we'll end up with nobody doing the driving. Well, packagers are driving ... it's the QA process which is causing their measures to show effect. In a nutshell: Fedora's QA process is cause of many of these "broken deps" complaints. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 05:30 PM, Doug Ledford wrote: > - Original Message - >> I'd like to see a rationale for jamming a soname-changing update into >> the OS so close to a release. In the absence of a very good >> motivation, >> that's not good engineering practice, and it's not consistent with >> the >> feature process. >> >> Perhaps you're not clear on what the word "freeze" means. > > One rationale is that if we don't get it *before* the release when everyone > is actively testing, then it ends up going in post release, Agreed, but what currently is happening, is packagers not being able to submit package chains _in time_ because of the delays. Reality is, when the root of a dependency chain changes incompatibly, there are situations, it takes weeks until the whole chain has been rebuilt. And when a freeze "closes down" update submissions, the repos end up in inconsistent and broken state. > likely with far less testing, and risks destabilizing the already released > product. The way things currently are, these packages will land as part of "day one" mass updates. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/20/2011 04:33 PM, Adam Jackson wrote: > Of course, you had the option of not pulling the new OpenSceneGraph back > to F16, or simply not doing so yet. Correct. I could have opted to ship the "distro which embraces novelty" with outdated, upstream unmaintained and upstream dead packages, no user is interested in using in anymore. > Particularly during a phase where > people are trying to keep change to a minimum so we know what we're > working with. The change had affected 5 packages ... You are right insofar as I underestimated the harmful impact of Fedora's wanna-be QA bureaucracy. > I'm sorry that you don't understand release engineering, but since you > insist on not understanding it I don't really have any sympathy for your > packages being so visibly at fault for breakages. Please understand, I can not take you seriously anymore. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 20 Sep 2011 15:19:29 -0400 (EDT) Doug Ledford wrote: ...snip... > > > 2) 9 times out of 10 there was very little data put into the ticket. > > Multiple options here. Kick back incomplete tickets, or the better > option IMNSHO, run rpmdiff runs between the package currently in the > compose and the one in testing to check for various failures and > require the developer to justify failures. Which rpmdiff are we talking about here? The free/included in fedora one is not that great... it gives you files and deps that changed, but that doesn't help you see what changed in them... > > > 3) releng folks were often not the best people to decide whether a > > change was "too risky" > > The rpmdiff option above would help with this. So, I run it on xfwm4 updates: rpmdiff xfwm4-4.8.1-2.fc15.x86_64.rpm xfwm4-4.8.1-3.fc16.x86_64.rpm removed REQUIRES libpng12.so.0()(64bit) removed PROVIDES xfwm4(x86-64) = 4.8.1-2.fc15 added PROVIDES xfwm4(x86-64) = 4.8.1-3.fc16 S.5...T /usr/bin/xfwm4 S.5...T /usr/bin/xfwm4-settings S.5...T /usr/bin/xfwm4-tweaks-settings S.5...T /usr/bin/xfwm4-workspace-settings ..T /usr/lib64/xfce4/xfwm4 S.5...T /usr/lib64/xfce4/xfwm4/helper-dialog ...all the doc files have different timestamp... What does that help me with? ;) > > 4) There was no easy way to get at the package and assess its > > stability. > > Using bodhi instead of trac solves this, no? well, not bodhi, but a repo like updates-testing, yeah. > > I think there were more issues, but those come to mind first. We > > decided it was best instead to make a repository out of proposed > > changes, > > But in practice that's not really what updates-testing on the early > branched release really is. It's a repo all right, but not of > proposed changes, it's a repo of packages, and getting to the actual > changes versus the final package would require installing the current > source rpm, the new source rpm, then doing a manual inspection for > changes. An automated rpmdiff run would be a *far* superior means of > presenting the proposed changes to the community. I'd love to see something more detailed from rpmdiff. ;) kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Sep 20, 2011, at 12:19 PM, Doug Ledford wrote: > - Original Message - >> This is essentially what we had a while ago, only with trac tickets >> instead of bodhi requests. > > Bodhi is definitely a better place to track this stuff, regardless of how > decisions are made. > >> There were a couple of problems with >> this. >> >> 1) Nowhere near enough releng folks to properly review all the >> tickets. > > Fair enough. > >> 2) 9 times out of 10 there was very little data put into the ticket. > > Multiple options here. Kick back incomplete tickets, or the better option > IMNSHO, run rpmdiff runs between the package currently in the compose and the > one in testing to check for various failures and require the developer to > justify failures. There was supposed to be AutoQA filling this role here, catching obvious issues, sanity checking, and letting things through that looked OK. We're still waiting for autoQA. There certainly wasn't enough releng man power/volunteers to manually look through this for every potential update. > >> 3) releng folks were often not the best people to decide whether a >> change was "too risky" > > The rpmdiff option above would help with this. > >> 4) There was no easy way to get at the package and assess its >> stability. > > Using bodhi instead of trac solves this, no? No, what was missing was a repository with yum metadata to get at the package and any sibling packages that needed to come with it for a comprehensive picture of what was going on. > >> I think there were more issues, but those come to mind first. We >> decided it was best instead to make a repository out of proposed >> changes, > > But in practice that's not really what updates-testing on the early branched > release really is. It's a repo all right, but not of proposed changes, it's > a repo of packages, and getting to the actual changes versus the final > package would require installing the current source rpm, the new source rpm, > then doing a manual inspection for changes. An automated rpmdiff run would > be a *far* superior means of presenting the proposed changes to the community. It depends on what you're after. If you just want a list of changes, sure, but if you want some idea as to whether or not those changes introduce instability then you need to look more comprehensively. It is rather difficult to look at a small list of changes and gauge how well/ill it will effect the stability of the distribution going forward. Either you're too liberal and we have rampant instability, or you're too draconian and we have very strict barriers to entry, which are suddenly removed once a product goes GA. Using the same criteria prior to and post GA to assess the validity of an update makes sense to me. > >> and let your packaging peers decide whether or not the >> update was safe enough to go into the release, > > But they can't, not really. For instance, a proventester may install my > mdadm package, but if they don't have a raid array, they can't decide > anything. Where as if they had access to the code diffs and could tell that > all I did was change a typo in the udev rules file, and verify for themselves > via code inspection that the code as it stands in the repo can't work and the > fix should work, then they could make an educated contribution. Sadly we have far far too few real developers who could do that comparison. > Because the testing repo doesn't really present changes, but only a final > product, there is no possibility for my peers to look at my changes and say > "Yeah, that's should be a safe change, let it in, and if it breaks then we'll > fix it". So the judgement call that I mentioned in my previous email is > taken entirely out of the loop, and there is no ability for my peers to make > any contribution to whether or not a package should be allowed in *except* > unit testing, there is no ability for them to easily do an actual analysis of > the changes and ma > ke an engineering decision versus a QA decision. I think that's largely because we don't have a community of engineers. We have a community of /packagers/ who are able to cause packages to be built, and are able to do some measure of QA to see if those builds work, but do not have the skill set to look at a code diff and give a honest opinion as to whether or not the change is "safe". If the makeup of our community were different, then you would have a case for your proposal. I just don't believe we have the community it would take to accomplish it on a large scale. > >> thus we have bodhi >> and updates-testing as a gateway to get into the release. > > It's a gateway, I just don't think it serves as useful a purpose as it was > intended to. The question though really is whether or not it is more useful than a few (like 4) release engineers looking at trac tickets and making best guesses depending on whatever the ticket reporter put in the ti
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Sep 20, 2011, at 1:35 PM, Ralf Corsepius wrote: > On 09/20/2011 05:30 PM, Doug Ledford wrote: >> - Original Message - >>> I'd like to see a rationale for jamming a soname-changing update into >>> the OS so close to a release. In the absence of a very good >>> motivation, >>> that's not good engineering practice, and it's not consistent with >>> the >>> feature process. >>> >>> Perhaps you're not clear on what the word "freeze" means. >> >> One rationale is that if we don't get it *before* the release when everyone >> is actively testing, then it ends up going in post release, > Agreed, but what currently is happening, is packagers not being able to > submit package chains _in time_ because of the delays. > > Reality is, when the root of a dependency chain changes incompatibly, > there are situations, it takes weeks until the whole chain has been > rebuilt. And when a freeze "closes down" update submissions, the repos > end up in inconsistent and broken state. > >> likely with far less testing, and risks destabilizing the already released >> product. > > The way things currently are, these packages will land as part of "day > one" mass updates. > > Ralf A few releases ago, I believe a decision was made, or at least proposed, that broken dep resolution rebuilds should be automatically considered "Nice to Have", that is they are allowed to break the freeze, but the release would not wait for them. Maybe the missing part here is getting visibility on these broken deps into the blocker meetings so that QA/releng can do the necessary work to let those updates through. - jlk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, Sep 20, 2011 at 11:18:18 -0700, Jesse Keating wrote: > One change to make this better might be to move the inheritance point to > updates-testing so that things built from the fresh branch are immediately > inherited into rawhide. I think this would be a change for the better. I've noticed f17 is not picking up fixes from f16 for a long time as stuff sits in updates-testing. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
Am Dienstag, den 20.09.2011, 22:25 +0200 schrieb Ralf Corsepius: > In a nutshell: Fedora's QA process is cause of many of these "broken > deps" complaints. Please make a proposal to improve the situation and submit it to FESCo. TIA, Christoph -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
2011/9/20 Jesse Keating : ... >>> thus we have bodhi >>> and updates-testing as a gateway to get into the release. >> >> It's a gateway, I just don't think it serves as useful a purpose as it was >> intended to. > > > The question though really is whether or not it is more useful than a few > (like 4) release engineers looking at trac tickets and making best guesses > depending on whatever the ticket reporter put in the ticket about the change. > We were trying to improve the workflow, not perfect it. Has it been > improved? I think the situation improved with bodhi buildroot overrides over trac tickets. But I've hit several issues with the opencv case: - Buildroot overrides disappear on bodhi ticket submission: When an override is scheduled for a period of time, it is discarded when a bodhi updates is submitted. In my case, some dependencies wasn't rebuilt yet, but was also FTBFS for other reason. Pushing an update didn't invalidate the need to rebuilt this update with the new ABI. That could have been done later and would still allows to test the actual override. - Buildroot overrides should be included in the dependency checks report. That cannot replace an announcement to know if such ABI change is legitimate. But that would help maintainers to keep in mind that their action is needed and for which branch. Specially if they have missed the announcement for any reason. - About Bodhi and ACLs, Tickets integrity shouldn't be affected by the ACL mismatch of each single package by the submitter. As I'm submitting an override, then I should inherit "some rights" to push an update with consistency from testing to stable. Probably bodhi should be aware that buildroot overrides are a premise to an update and suggest to add the packages rebuilt with the new version, and allow an ACL on it for the ticket. I can understand that there is too much rights involved with the bodhi buildroot override for non provenpackager. So it become a provenpackager duty to drive an ABI change. That wasn't the case with trac tickets because that has involved also some ACLs override to rebuild a list of packages. And that was conduced in a single process. Nicolas (kwizart) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Sep 20, 2011, at 3:56 PM, Nicolas Chauvet wrote: > > I think the situation improved with bodhi buildroot overrides over trac > tickets. > But I've hit several issues with the opencv case: You make some valid points, however I was more concerned with the freeze break requests in trac, not necessarily the build root override requests. I was not active in the fedora releng team when the build root override change happened. - jlk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 12:21 -0400, Adam Jackson wrote: > On 9/20/11 11:43 AM, Nils Philippsen wrote: > > On Tue, 2011-09-20 at 11:33 -0400, Adam Jackson wrote: > >> Of course, the accounts system _still_ doesn't have groups, five years > >> later, so provenpackager is the big hammer we have. We could get groups > >> any day now, that'd be just fine. > > > > Do you mean "groups of groups", like in "provenpackager-kde", > > "provenpackager-gnome", and "provenpackager" means all of these? > > I don't see any real reason to do that instead of just unix-style > groups, but either one would be an improvement. Hmm, it seems I don't quite get what you mean with "the accounts system _still_ doesn't have groups" -- while provenpackager among others is a group. Would you please elaborate? TIA, Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-09-20 at 22:25 +0200, Ralf Corsepius wrote: > On 09/20/2011 05:52 PM, Nils Philippsen wrote: > > On Tue, 2011-09-20 at 15:19 +0200, Ralf Corsepius wrote: > > >> When you have a closer look, you'll notice that such "mass rebuilts" > >> were being delayed by QA's "delay queue" and now are stuck. > > > > I didn't want to (re)start that particular discussion ;-). > > > > We need some guidelines who should drive rebuilds in such a situation, > > regardless of whether it happens in Rawhide or Branched or wherever. > a) These situation can only happen in release branches. Uhm, no. A library SONAME bump can happen in Rawhide as well as in branches and if dependent packages don't get built with the new version, things are broken. Waiting for dependent components to be built at their maintainers leisure or whenever a mass rebuild comes around isn't a solution, not if we want to have a "more stable Rawhide". Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/21/2011 01:25 PM, Nils Philippsen wrote: > On Tue, 2011-09-20 at 22:25 +0200, Ralf Corsepius wrote: >> On 09/20/2011 05:52 PM, Nils Philippsen wrote: >>> On Tue, 2011-09-20 at 15:19 +0200, Ralf Corsepius wrote: >> When you have a closer look, you'll notice that such "mass rebuilts" were being delayed by QA's "delay queue" and now are stuck. >>> >>> I didn't want to (re)start that particular discussion ;-). >> > >>> We need some guidelines who should drive rebuilds in such a situation, >>> regardless of whether it happens in Rawhide or Branched or wherever. >> a) These situation can only happen in release branches. > > Uhm, no. A library SONAME bump can happen in Rawhide as well as in > branches and if dependent packages don't get built with the new version, > things are broken. Right. They break in rawhide, where issues are inevitable and can be addressed within short terms because maintainers are not being forced to play "10 days per package update" "you wait for me/I wait for you" delay ping-pong. Or am I misunderstanding? Are you referring to points in time when "stable fedoras" are being sync'ed and inherit "broken deps" during this fork? If this happens, something would be very defective with Fedora's rel-eng's procedures. The situation currently to be found in f16 is "longer dep chains" being in inconsistent build-state (showing as broken deps), because fixed packages in *-testing did not make it into "stable" in time because of these QA delays and because Fedora policies _forbit_ fixing these at this point in time. > Waiting for dependent components to be built at their > maintainers leisure or whenever a mass rebuild comes around isn't a > solution, not if we want to have a "more stable Rawhide". To we want this? I don't. To me, rawhide is the "big package dumping ground" for the bleading edge". Certainly, it should be as little broken as possible, but "its brokenness" is part of its working principle and inevitable to me. That said, I find a stable "rawhide" to be nonrealisting and inapplicable. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Wed, 2011-09-21 at 15:51 +0200, Ralf Corsepius wrote: > On 09/21/2011 01:25 PM, Nils Philippsen wrote: > > On Tue, 2011-09-20 at 22:25 +0200, Ralf Corsepius wrote: > >> On 09/20/2011 05:52 PM, Nils Philippsen wrote: > >>> On Tue, 2011-09-20 at 15:19 +0200, Ralf Corsepius wrote: > >> > When you have a closer look, you'll notice that such "mass rebuilts" > were being delayed by QA's "delay queue" and now are stuck. > >>> > >>> I didn't want to (re)start that particular discussion ;-). > >> > > >>> We need some guidelines who should drive rebuilds in such a situation, > >>> regardless of whether it happens in Rawhide or Branched or wherever. > >> a) These situation can only happen in release branches. > > > > Uhm, no. A library SONAME bump can happen in Rawhide as well as in > > branches and if dependent packages don't get built with the new version, > > things are broken. > Right. They break in rawhide, where issues are inevitable and can be > addressed within short terms because maintainers are not being forced to > play "10 days per package update" "you wait for me/I wait for you" delay > ping-pong. And that's always fine and dandy if these issues are resolved in a reasonable amount of time. Right now Rawhide has packages with dependencies broken since pre-F15. This isn't acceptable. > Or am I misunderstanding? Are you referring to points in time when > "stable fedoras" are being sync'ed and inherit "broken deps" during this > fork? If this happens, something would be very defective with Fedora's > rel-eng's procedures. I'm not talking about branched Fedora vs. Rawhide at all. I thought I made myself clear on that. > > Waiting for dependent components to be built at their > > maintainers leisure or whenever a mass rebuild comes around isn't a > > solution, not if we want to have a "more stable Rawhide". > > To we want this? I don't. To me, rawhide is the "big package dumping > ground" for the bleading edge". Certainly, it should be as little broken > as possible, but "its brokenness" is part of its working principle and > inevitable to me. That said, I find a stable "rawhide" to be > nonrealisting and inapplicable. You're twisting my words a bit. I wasn't opting for a stable, but a more stable release. If somebody wants to test something in Rawhide they shouldn't have to debug other parts of the distribution. This means that changes should be done with enough circumspection that breakage is reduced to a minimum. People who actually run Rawhide (and I know their number is low) would thank us for that. Right now this is not the case: a substantial number of components is broken due to unsatisfied dependencies alone, meaning that everybody who wants to test Rawhide in conjunction with anything on that list is simply out of luck right now. I admit that the impact of being broken is not the same for every component in the distribution: some stuff more on the fringe is sufficiently isolated that it will only affect few testers if it doesn't work (ideally the ones fixing the breakage), so it won't hurt many if these are broken for a longer time, but other components are central enough that they can't afford that luxury. It would certainly help here if buildroots could be created for Rawhide so that breakage can be resolved in isolation there. In this case they'd need to be created before the first package is built however, so it doesn't break unrelated builds. This is because we don't file Bodhi updates for Rawhide packages (nobody in their right mind would want that). Nils -- Nils Philippsen "Those who would give up Essential Liberty to purchase Red Hat a little Temporary Safety, deserve neither Liberty n...@redhat.com nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Wed, Sep 21, 2011 at 04:43:38PM +0200, Nils Philippsen wrote: > And that's always fine and dandy if these issues are resolved in a > reasonable amount of time. Right now Rawhide has packages with > dependencies broken since pre-F15. This isn't acceptable. If you notice this, ask FESCo to ask FES to perform a rebuild to fix the dependency: https://fedoraproject.org/wiki/Fedora_Engineering_Services Probably any member of the provenpackager group can help you here. Regards Till pgpze8fJ76uSR.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/21/2011 04:43 PM, Nils Philippsen wrote: > On Wed, 2011-09-21 at 15:51 +0200, Ralf Corsepius wrote: >> On 09/21/2011 01:25 PM, Nils Philippsen wrote: >>> On Tue, 2011-09-20 at 22:25 +0200, Ralf Corsepius wrote: On 09/20/2011 05:52 PM, Nils Philippsen wrote: > On Tue, 2011-09-20 at 15:19 +0200, Ralf Corsepius wrote: >> When you have a closer look, you'll notice that such "mass rebuilts" >> were being delayed by QA's "delay queue" and now are stuck. > > I didn't want to (re)start that particular discussion ;-). > > We need some guidelines who should drive rebuilds in such a situation, > regardless of whether it happens in Rawhide or Branched or wherever. a) These situation can only happen in release branches. >>> >>> Uhm, no. A library SONAME bump can happen in Rawhide as well as in >>> branches and if dependent packages don't get built with the new version, >>> things are broken. >> Right. They break in rawhide, where issues are inevitable and can be >> addressed within short terms because maintainers are not being forced to >> play "10 days per package update" "you wait for me/I wait for you" delay >> ping-pong. > > And that's always fine and dandy if these issues are resolved in a > reasonable amount of time. Right now Rawhide has packages with > dependencies broken since pre-F15. This isn't acceptable. Agreed - If so, these need to be addressed. IMO, if packagers and/or proven packagers are not able to fix these in "reasonable time", I'd consider it to be QA's job to take care about these. As experience as a proven packager, who did try to address such cases in the past tells, such cases typically originate from a) packagers not being sufficently skilled to fix such breakages and them not asking for assitance. b) packagers having gone AWOL or being unreachable c) packages being sufficiently incompatible to an upgrade that they better should be removed/retired. Wrt. a) experience tells, some packagers are hesitant to ask for assitance and prefer to "sit out the issue", until some proven packager or upstream takes action. Wrt. the packages I had stepped in, case b) was fairly common. IMO the cause is Fedora lacking a spirit of "teaming up". Wrt. c), here the issue seems to be packagers being hesitant to "draw a cut" and to retire a package. Surprisingly, when directly contacting maintainers of such packages, they often agree to such retirement. >>> Waiting for dependent components to be built at their >>> maintainers leisure or whenever a mass rebuild comes around isn't a >>> solution, not if we want to have a "more stable Rawhide". >> >> To we want this? I don't. To me, rawhide is the "big package dumping >> ground" for the bleading edge". Certainly, it should be as little broken >> as possible, but "its brokenness" is part of its working principle and >> inevitable to me. That said, I find a stable "rawhide" to be >> nonrealisting and inapplicable. > > You're twisting my words a bit. I wasn't opting for a stable, but a more > stable release. If somebody wants to test something in Rawhide they > shouldn't have to debug other parts of the distribution. This means that > changes should be done with enough circumspection that breakage is > reduced to a minimum. People who actually run Rawhide (and I know their > number is low) would thank us for that. Well, what am I supposed to say? Anybody would be grateful for less bugs and breakage, but ... on rawhide breakage is simply inevitable. > Right now this is not the case: a substantial number of components is > broken due to unsatisfied dependencies alone, meaning that everybody who > wants to test Rawhide in conjunction with anything on that list is > simply out of luck right now. That's one view. From a packager's view, whenever something changes incompatibly, no matter how careful and speedy the packager tries to be, there always be situations when something breaks. Typical case are: The packager who is pushing an incompatible upgrade misses a "to be rebuild package", the packager doesn't have sufficient privileges to rebuild a package in need of a rebuilt or the upgrade renders a package unbuildable ... IMO, the last on this list is the critical case, which is causing rawhide users to complain. > I admit that the impact of being broken is > not the same for every component in the distribution: some stuff more on > the fringe is sufficiently isolated that it will only affect few testers > if it doesn't work (ideally the ones fixing the breakage), so it won't > hurt many if these are broken for a longer time, but other components > are central enough that they can't afford that luxury. No disagreement. > It would certainly help here if buildroots could be created for Rawhide > so that breakage can be resolved in isolation there. I am using local mock environments which inherit from "upstream" (== Fedora), as a band-aid to identify potential breakage and to keep impact o
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/21/2011 05:33 PM, Till Maas wrote: > On Wed, Sep 21, 2011 at 04:43:38PM +0200, Nils Philippsen wrote: > >> And that's always fine and dandy if these issues are resolved in a >> reasonable amount of time. Right now Rawhide has packages with >> dependencies broken since pre-F15. This isn't acceptable. > > If you notice this, ask FESCo to ask FES to perform a rebuild to fix the > dependency: > https://fedoraproject.org/wiki/Fedora_Engineering_Services > > Probably any member of the provenpackager group can help you here. > > Regards > Till > > > I hope you don't suggest for every rebuild of few dependent packages one FESCo ticket. -- Marcela Mašláňová BaseOS team Brno -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 22/09/11 09:15, Marcela Mašláňová wrote: > On 09/21/2011 05:33 PM, Till Maas wrote: >> On Wed, Sep 21, 2011 at 04:43:38PM +0200, Nils Philippsen wrote: >> >>> And that's always fine and dandy if these issues are resolved >>> in a reasonable amount of time. Right now Rawhide has packages >>> with dependencies broken since pre-F15. This isn't acceptable. >> >> If you notice this, ask FESCo to ask FES to perform a rebuild to >> fix the dependency: >> https://fedoraproject.org/wiki/Fedora_Engineering_Services >> >> Probably any member of the provenpackager group can help you >> here. >> >> Regards Till >> >> >> > I hope you don't suggest for every rebuild of few dependent > packages one FESCo ticket. > Thinking some time about this: Would it help to have a person (per branch), responsible for rebuilding packages with broken deps? The person may rebuild himself or try to force package maintainers to rebuild, retire packages, when they stay in broken state since .. days/releases/...? Cheers, Matthias - -- Matthias Runge -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOewLRAAoJEOnz8qQwcaIW/IkH/A4BNpTcqwH8+x9FXdHGWeNv pw1FoeT1SZx+UtLdVbSmXiDKAsIvZ9InGD8+9hXF1Af6QRz0+ECJXhJ6Vn16FEO4 zmLtZ7SX/gZGtcafxw4ua7W0QjW/96EY+E5sAzIJ34DqSaDmklt/rOTuqf37oRuQ pf1wim9KYcbeLTJhV2iJ3OWEAXW2lmyH2JQSg+sfZv7QQTSjGl6VmD+asV7Ktn/3 aiYmquIRtwIdxLkJtfEvVq4yUUqVvsAg3GWgH2HQXG3QHvzFTT0hpWgQ00h5M0n+ a0YVLpBqleiVRP9x1/evh5qdywxtdWZ2uucQCE6rQu/zZXEPXhkhZg1BSXF7uv4= =1iuC -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Thu, Sep 22, 2011 at 09:15:38AM +0200, Marcela Mašláňová wrote: > I hope you don't suggest for every rebuild of few dependent packages one > FESCo ticket. This is what is currently required to ask FES for help. It is certainly a lot better and more efficient to open one FESCo and one FES ticket to get dependent packages rebuild than to open several Bugzilla tickets for each dependent package. If the respective maintainer can perform the rebuilds all by himself, then there is off course no need to ask for help from FES. Regards Till -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On 09/22/2011 05:58 PM, Till Maas wrote: > On Thu, Sep 22, 2011 at 09:15:38AM +0200, Marcela Mašláňová wrote: > >> I hope you don't suggest for every rebuild of few dependent packages one >> FESCo ticket. > > This is what is currently required to ask FES for help. It is certainly > a lot better and more efficient to open one FESCo and one FES ticket to > get dependent packages rebuild than to open several Bugzilla tickets for > each dependent package. If the respective maintainer can perform the > rebuilds all by himself, then there is off course no need to ask for > help from FES. Does the word "bureaucracy" ring a bell to you? You to me sound like a the proverbial German "Finanzbeamter" (tax officer), who wonders why people are sick of filling out forms. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > On Tue, 20 Sep 2011 15:19:29 -0400 (EDT) > Doug Ledford wrote: > > ...snip... > > > Which rpmdiff are we talking about here? > The free/included in fedora one is not that great... it gives you > files > and deps that changed, but that doesn't help you see what changed in > them... I was referring to the setup we have inside Red Hat. I'm not sure how hard it would be to implement in Fedora, but we get much more useful information and it's broken out into specific areas that you need to review. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
> I think that's largely because we don't have a community of > engineers. We have a community of /packagers/ who are able to cause > packages to be built, and are able to do some measure of QA to see > if those builds work, but do not have the skill set to look at a > code diff and give a honest opinion as to whether or not the change > is "safe". > > If the makeup of our community were different, then you would have a > case for your proposal. I just don't believe we have the community > it would take to accomplish it on a large scale. You may be right. Which, unfortunately, just makes me feel like an outcast that doesn't belong. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Sep 22, 2011, at 11:27 AM, Doug Ledford wrote: > - Original Message - >> On Tue, 20 Sep 2011 15:19:29 -0400 (EDT) >> Doug Ledford wrote: >> >> ...snip... >> >> >> Which rpmdiff are we talking about here? >> The free/included in fedora one is not that great... it gives you >> files >> and deps that changed, but that doesn't help you see what changed in >> them... > > I was referring to the setup we have inside Red Hat. I'm not sure how hard > it would be to implement in Fedora, but we get much more useful information > and it's broken out into specific areas that you need to review. The setup inside Red Hat cannot be (directly) copied outside at this time. Instead the autoQA project was started to re-create it as an open source project. That's where effort should continue. - jlk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > The setup inside Red Hat cannot be (directly) copied outside at this > time. Instead the autoQA project was started to re-create it as an > open source project. That's where effort should continue. Am I right in saying that AutoQA is basically mired in the muck and going nowhere at the moment? -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Fri, Sep 23, 2011 at 14:31, Doug Ledford wrote: > - Original Message - >> The setup inside Red Hat cannot be (directly) copied outside at this >> time. Instead the autoQA project was started to re-create it as an >> open source project. That's where effort should continue. > > Am I right in saying that AutoQA is basically mired in the muck and going > nowhere at the moment? > Doug, = If Autoqa is currently slow it is mainly because what developers who are working on it are also tasked with doing other things. How long did it take for autoqa to show up inside of RH? I know it was started in part by Wanger in 97-98 and reimplemented multiple times never getting very far because who ever was writing it would be told that qa'ing what is out there now was the high priority task. If you have extra time because kernel.org is down, here are some places to look at what is going on, where you can help. There is a list here which shows what autoqa is doing per day: https://fedorahosted.org/pipermail/autoqa-results/ Development is here https://fedorahosted.org/mailman/listinfo/autoqa-devel Here is the main webpage on autoqa http://fedoraproject.org/wiki/AutoQA -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
> Am I right in saying that AutoQA is basically mired in the muck and > going nowhere at the moment? > > -- > Doug Ledford Our progress is very slow at the moment, correct. We will happily welcome some help. We don't have many tasks that you could do in a free afternoon, however. A free week or month could work well. AutoQA development should speed up again once Fedora 16 is gold. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
- Original Message - > Doug, > > = If Autoqa is currently slow it is mainly because what developers > who > are working on it are also tasked with doing other things. I made no reference or allusion to it being slow because people were slacking. I myself have more to do than I have time in the day to accomplish it, so I know how that goes. > How long > did it take for autoqa to show up inside of RH? I know it was started > in part by Wanger in 97-98 and reimplemented multiple times never > getting very far because who ever was writing it would be told that > qa'ing what is out there now was the high priority task. One would hope that today we get the benefit of all those years of aborted attempts and can at least build upon what is already done internally. > If you have > extra time because kernel.org is down, kernel.org up or down doesn't affect my free time unfortunately. > here are some places to look > at > what is going on, where you can help. > > There is a list here which shows what autoqa is doing per day: > https://fedorahosted.org/pipermail/autoqa-results/ > > Development is here > https://fedorahosted.org/mailman/listinfo/autoqa-devel > > Here is the main webpage on autoqa > http://fedoraproject.org/wiki/AutoQA I've already volunteered for one project that's at least tangentially related to AutoQA and I'm having a hard time finding the time to do it justice, more would simply make everything I do worse. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Fri, 2011-09-23 at 16:31 -0400, Doug Ledford wrote: > - Original Message - > > The setup inside Red Hat cannot be (directly) copied outside at this > > time. Instead the autoQA project was started to re-create it as an > > open source project. That's where effort should continue. > > Am I right in saying that AutoQA is basically mired in the muck and going > nowhere at the moment? "at the moment", yes, but that's with a very narrow scope of "at the moment" - i.e. for the last few weeks. I don't want Kamil's email to give the impression that AutoQA has been going nowhere for months, because that certainly isn't the case. Work has slowed down in the last few weeks because Fedora 16 Beta testing was extremely problematic and sucked up most of the AutoQA manpower, and QA is anyway undermanned (from a Red Hat paid staff viewpoint) ATM. It should speed up again to a lesser degree now, and to a greater degree when Final is done and we have a couple more bodies in place. I think adding some more rpmdiff-type tests to current AutoQA wouldn't actually be a huge stretch, but I'd bow to Tim or Kamil on the detail front there. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes
On Tue, 2011-10-04 at 09:07 -0700, Adam Williamson wrote: > On Fri, 2011-09-23 at 16:31 -0400, Doug Ledford wrote: > > - Original Message - > > > The setup inside Red Hat cannot be (directly) copied outside at this > > > time. Instead the autoQA project was started to re-create it as an > > > open source project. That's where effort should continue. > > > > Am I right in saying that AutoQA is basically mired in the muck and going > > nowhere at the moment? > > "at the moment", yes, but that's with a very narrow scope of "at the > moment" - i.e. for the last few weeks. I don't want Kamil's email to > give the impression that AutoQA has been going nowhere for months, > because that certainly isn't the case. Work has slowed down in the last > few weeks because Fedora 16 Beta testing was extremely problematic and > sucked up most of the AutoQA manpower, and QA is anyway undermanned > (from a Red Hat paid staff viewpoint) ATM. It should speed up again to a > lesser degree now, and to a greater degree when Final is done and we > have a couple more bodies in place. > > I think adding some more rpmdiff-type tests to current AutoQA wouldn't > actually be a huge stretch, but I'd bow to Tim or Kamil on the detail > front there. Not to bash the thing to death, but Kamil did a presentation on AutoQA at FUDCon Milan, the slides are well worth reading: https://fedoraproject.org/w/uploads/2/27/AutoQA-FUDCon-Milan-2011.odp they provide a pretty good potted explanation of what the big priorities in AutoQA development are right now, why they're important, why AutoQA can't really accept outside test contributions right now (and what needs fixing before it can), and in general why AutoQA is kind of in 'swan mode' at the moment - it looks like nothing much is going on above the waterline, but below the surface the legs are pumping away madly :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
