Re: wheezy postmortem re rc bugfixing
On Wed, May 08, 2013 at 04:51:14PM +0100, Ian Jackson wrote: * Consider other ways in which our RC-bug-fixing efforts can be improved, especially during the latter part of the freeze. I think one way to improve hard to reproduce bugs or bugs in uncommon package would be to get more users involved. I think the way to go there is to get users better informed about the problem. This could come in two parts: 1) A developer (maintainer or random person doing triage) tags the bug RFUH (request for user help) in some way. Helpfull would be a short summary of the problem and what help is needed. E.g. needs hardware XYZ, look for XYZ to happen and then run FOO and send the output. 2) A tool for users to run that checks the installed packages against RFUH tags and points out matches. Maybe requirements could be checked automatically too. So only users with hardware XYZ see the RFUH that requires XYZ. Or RFUH could be for combinarions of packages: users of foo + bar + baz. The goal would be to limit message to only those users that can help. This (2) could also be helpfull before removing packages. The tool could warn stable users about their favourite package being in danger of removal and get them to test out the alternatives or scream bloody murder before it is too late. Is there already something for that? Then it needs to be advertised more. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130513104528.GE8366@frosties
Re: wheezy postmortem re rc bugfixing
Yea? When are you filing a patch that corrects it? complaining does nothing. we all know what would be better and move toward it and your forgetting the by-law: you don't fix what you think is better by breaking software that works unless you can really prevent all breakage after all. what you think is better, compared to software filed by, ie, ATT experts long ago? Your probably wrong an not studied enough has to be my first assumption. Neil Williams wrote: On Wed, 8 May 2013 16:51:14 +0100 Ian Jackson ijack...@chiark.greenend.org.uk wrote: So I would like to suggest that we should have a thread where we: I suspect a wiki page will be needed at some point... * Try to identify the main ways in which bugs can be hard (which might be technical, political, or a mixture) These are the issues which discouraged me from working on particular RC bugs during this release and the one before it. * Hard to reproduce - uncommon packages, specialised setups, specific hardware or hardware configurations, intermittent issues. * Un- / under maintained packages not yet orphaned. IMHO we should be much more aggressive with such packages - strict time limits for all RC-buggy leaf packages, regardless of the uncertainties of popcon, leading to at least automatic removal from testing. Warning bugs against reverse dependencies of non-leaf packages warning of problems. (We have this now in the PTS for WNPP issues, an extension to RC bugs in dependencies could also be really useful.) * Disagreements between submitter maintainer / fixer * Architecture-specific bugs for less common ports - maybe be more aggressive here also on which architectures are deemed fit for release on the basis of the occurrence and likely delays caused by such bugs. * Non-standard packaging or build system or packages using methods known to make NMU's difficult. * Languages other than C, C++, perl, shell or python, reducing the possible number of people who can get a full overview of the package. * Lapsed release goals (like building sanely twice in a row, not just in a clean chroot but during a typical NMU process too, so that test changes can be easily and quickly reverted and the package rebuilt.) Particularly important for packages which take a long time to build or have esoteric / uncommon build-dependencies. * Poor quality documentation within the package, especially for build-system idiosyncrasies and internal API/ABI requirements. Simple things like a comment in each source code file saying what the code in that file is meant to achieve. foo.c - wraps the bar interface in the foo class etc. Other steps to take as preventative measures: * More QA runs through the archive prior to freeze to catch things like embedded libraries or binaries without sources or licence irregularities, FTBFS and piuparts errors. The actual decision of the freeze date could be made to be reliant on a % clean report from such runs. * Make it a *MUST* that all transitions, no matter how small, are checked with the release team starting from as soon as the freeze is announced (not just after it starts) such that uploads which start a transition could be pushed into DELAYED or REJECTed automatically. (Not easy to implement this one, I know.) * Try to think of workflows which might overcome these problems debian/rules must be a makefile, only specific packages can be allowed to not use debhelper, toolchain packages to be frozen earlier than the rest of the archive... I think the Debian PPA approach could also ease the process substantially - we could, for example, require that all uploads during a freeze which do not fix RC bugs must go into a Debian PPA. This reduces the need for t-p-u builds and should help the slow isolation of desired changes from packaging fluff. * In particular, try to think of workflows and/or support tools which might be able to connect the people with the skills and/or authority to solve the problem, with the bug - and help motivate those people to do the work needed to unblock the bug. That sounds like a requirement for tags and a search interface allied with rc-alert or similar. BSP's could also make use of such tags, possibly via an interface akin to the reverse of dd-list. Maybe the debtags information could be used to feed data into UDD relating to the likely experience of the maintainers based on the implemented-in and works-with tags of the packages which they maintain? Then a BSP can just feed the names into the search and get a list of likely bug numbers. The data would also help identify tags which have poor coverage and high proportions of bugs. * Consider other ways in which our RC-bug-fixing efforts can be improved, especially during the latter part of the freeze. I have deliberately avoided suggesting any answers to these questions myself in this article. But I may do so later. Also we should try to treat this discussion as a kind of semi-brainstorm: if someone makes
Re: wheezy postmortem re rc bugfixing
On Mon, 2013-05-13 at 21:41 -0400, John D. Hendrickson and Sara Darnell wrote: Yea? When are you filing a patch that corrects it? complaining does nothing. we all know what would be better and move toward it and your forgetting the by-law: you don't fix what you think is better by breaking software that works unless you can really prevent all breakage after all. what you think is better, compared to software filed by, ie, ATT experts long ago? Your probably wrong an not studied enough has to be my first assumption. debian-devel is not the YouTube comment box. Please only post while sober, and using complete English sentences. Ben. -- Ben Hutchings Make three consecutive correct guesses and you will be considered an expert. signature.asc Description: This is a digitally signed message part
Re: wheezy postmortem re rc bugfixing
On 2013-05-10 14:57:46 +0200, Piotr Ożarowski wrote: [Charles Plessy, 2013-05-09] For a large number of packages if not all, we should allow the package maintainers to manually migrate their packages to Testing during the Freeze, within boundaries set on debian-devel-announce by the release team. +1 +1 The boundaries shouldn't be very strict, or there could be discussions between the maintainers and the release team in particular cases where there could be a big benefit for the user. Note also that not all new upstream versions are equal (e.g. some are bug-fix releases), not packages are equal (e.g. some have regression tests...), and not all upstreams are equal. or a soft freeze (as described above) a month (or two) before hard freeze I doubt that it would change much, unless the hard freeze is time limited. -- Vincent Lefèvre vinc...@vinc17.net - Web: http://www.vinc17.net/ 100% accessible validated (X)HTML - Blog: http://www.vinc17.net/blog/ Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130511235456.ge26...@ioooi.vinc17.net
Re: wheezy postmortem re rc bugfixing
Hi, On Wed, May 08, 2013 at 05:28:58PM +0100, Neil Williams wrote: Other steps to take as preventative measures: * Make it a *MUST* that all transitions, no matter how small, are checked with the release team starting from as soon as the freeze is announced (not just after it starts) such that uploads which start a transition could be pushed into DELAYED or REJECTed automatically. (Not easy to implement this one, I know.) This could be implemented by building uploads to unstable against testing instead of unstable. Currently problematic uploads to unstable don't affect users of testing because they will not migrate, but they do affect development of testing (which is done in unstable), because they will prevent reverse build-dependencies from migrating. If uploads to unstable were built against testing, uncoordinated uploads of build-dependencies would not affect development, because they would not be used by the autobuilders until they were allowed to migrate to testing. They would still be used (and tested) by developers running unstable on their systems. To allow developers to adapt their packages to newer versions of build-dependencies, they should be able to selectively choose build-dependencies from unstable. A similar setup is already implemented for experimental (which builds against unstable by default, unless the build-dependencies explicitly specify packages from experimental). Newer versions of build-dependencies could also be specified for binNMU's, to allow rebuilds for transitions. The implementation of PPA's might allow individual developers to build their packages against testing and move these to unstable. Cheers, Ivo -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130510092334.ga...@ugent.be
Re: wheezy postmortem re rc bugfixing
On Fri, 10 May 2013 11:24:30 +0200 Ivo De Decker ivo.dedec...@ugent.be wrote: Hi, On Wed, May 08, 2013 at 05:28:58PM +0100, Neil Williams wrote: Other steps to take as preventative measures: * Make it a *MUST* that all transitions, no matter how small, are checked with the release team starting from as soon as the freeze is announced (not just after it starts) such that uploads which start a transition could be pushed into DELAYED or REJECTed automatically. (Not easy to implement this one, I know.) This could be implemented by building uploads to unstable against testing instead of unstable. The whole point with a transition is that a library is updated in unstable and, for the updated library to migrate, it's reverse dependencies in unstable need to migrate to that updated library by being built against the version of that library in unstable. Otherwise, migration is blocked because reverse dependencies already in testing become uninstallable. There may be source code changes required in those reverse dependencies. There needs to be time for this to happen and then for all affected packages to migrate. There's no point building packages against testing as, at the critical stage, the version of the library to which the packages are meant to transition is *not* in testing. If those reverse dependencies are to be rebuilt at all, the rebuild needs to happen against the version which is trying to migrate - the one in unstable. Packages don't get rebuilt in testing - rebuilds go back through unstable. Problems arise from the number of reverse dependencies and when transitions get blocked by other transitions because of some packages which depend on two or more libraries already involved. Rebuilding any of those against testing is pointless - the packages need to be rebuilt against the new versions in unstable and, where necessary, migrated to the updated API. Prior to those rebuilds, the affected packages in testing and unstable were likely last built against the version of the library already in testing - I see no point in building them again for the same version. We have that build, it's that build which needs to be replaced by a build against the version of the transitioning library in unstable. Currently problematic uploads to unstable don't affect users of testing because they will not migrate, but they do affect development of testing (which is done in unstable), because they will prevent reverse build-dependencies from migrating. The point is to keep testing installable, not to allow updated applications into testing without having a transition at all. The applications must transition and that must happen in unstable if testing is to remain installable. There is no point allowing applications using libfoo0 already in testing to be rebuilt against libfoo0 (which no longer exists in unstable) when the whole point is that it is the lack of a build of the applications against libfoo1 which is blocking the migration of those applications *and* libfoo1 into testing. Any update, any new version of those applications, needs to be built against libfoo1, the version in unstable. There would be no libfoo1 in testing and no libfoo0 in unstable. The exceptions to this are libraries which use the libfoo0-dev mechanism, generally because there are so many reverse dependencies that both versions need to exist not only in unstable but in testing and stable too - think libgtk2.0-0 and libgtk3.0. That requires new source packages for every SONAME change - something which is only worth doing for packages like GTK. If uploads to unstable were built against testing, uncoordinated uploads of build-dependencies would not affect development, because they would not be used by the autobuilders until they were allowed to migrate to testing. They would still be used (and tested) by developers running unstable on their systems. What builds the packages in unstable? I think you're only considering a single architecture. Even just thinking of x86, there are just as many i386 uploads as amd64 - something has to build amd64 uploads for i386 in unstable and vice versa. Those builds need to be against the versions of dependencies in unstable - that is the job of the autobuilders. Developers can't run unstable if there are no builds other than uploads and unstable isn't just about x86 anyway. Even if the intent is to rebuild uncoordinated uploads of applications which would inadvertently tie two different transitions together, I see no gain over simply waiting until the transitions are complete before making the upload. If you build against the version in testing, the application will then have to be rebuilt against the version in unstable before the application can migrate anyway. Where is the advantage? If you allow the application to migrate, it will still have to transition to the new library before the library itself can migrate. So instead of one rebuild, you now have three. How does
Re: wheezy postmortem re rc bugfixing
On Fri, 10 May 2013 11:24:30 +0200 Ivo De Decker ivo.dedec...@ugent.be wrote: Hi, On Wed, May 08, 2013 at 05:28:58PM +0100, Neil Williams wrote: Other steps to take as preventative measures: * Make it a *MUST* that all transitions, no matter how small, are checked with the release team starting from as soon as the freeze is announced (not just after it starts) such that uploads which start a transition could be pushed into DELAYED or REJECTed automatically. (Not easy to implement this one, I know.) This could be implemented by building uploads to unstable against testing instead of unstable. The whole point with a transition is that a library is updated in unstable and, for the updated library to migrate, it's reverse dependencies in unstable need to migrate to that updated library by being built against the version of that library in unstable. Otherwise, migration is blocked because reverse dependencies already in testing become uninstallable. There may be source code changes required in those reverse dependencies. There needs to be time for this to happen and then for all affected packages to migrate. There's no point building packages against testing as, at the critical stage, the version of the library to which the packages are meant to transition is *not* in testing. If those reverse dependencies are to be rebuilt at all, the rebuild needs to happen against the version which is trying to migrate - the one in unstable. Packages don't get rebuilt in testing - rebuilds go back through unstable. Problems arise from the number of reverse dependencies and when transitions get blocked by other transitions because of some packages which depend on two or more libraries already involved. Rebuilding any of those against testing is pointless - the packages need to be rebuilt against the new versions in unstable and, where necessary, migrated to the updated API. Prior to those rebuilds, the affected packages in testing and unstable were likely last built against the version of the library already in testing - I see no point in building them again for the same version. We have that build, it's that build which needs to be replaced by a build against the version of the transitioning library in unstable. Currently problematic uploads to unstable don't affect users of testing because they will not migrate, but they do affect development of testing (which is done in unstable), because they will prevent reverse build-dependencies from migrating. The point is to keep testing installable, not to allow updated applications into testing without having a transition at all. The applications must transition and that must happen in unstable if testing is to remain installable. There is no point allowing applications using libfoo0 already in testing to be rebuilt against libfoo0 (which no longer exists in unstable) when the whole point is that it is the lack of a build of the applications against libfoo1 which is blocking the migration of those applications *and* libfoo1 into testing. Any update, any new version of those applications, needs to be built against libfoo1, the version in unstable. There would be no libfoo1 in testing and no libfoo0 in unstable. The exceptions to this are libraries which use the libfoo0-dev mechanism, generally because there are so many reverse dependencies that both versions need to exist not only in unstable but in testing and stable too - think libgtk2.0-0 and libgtk3.0. That requires new source packages for every SONAME change - something which is only worth doing for packages like GTK. If uploads to unstable were built against testing, uncoordinated uploads of build-dependencies would not affect development, because they would not be used by the autobuilders until they were allowed to migrate to testing. They would still be used (and tested) by developers running unstable on their systems. What builds the packages in unstable? I think you're only considering a single architecture. Even just thinking of x86, there are just as many i386 uploads as amd64 - something has to build amd64 uploads for i386 in unstable and vice versa. Those builds need to be against the versions of dependencies in unstable - that is the job of the autobuilders. Developers can't run unstable if there are no builds other than uploads and unstable isn't just about x86 anyway. Even if the intent is to rebuild uncoordinated uploads of applications which would inadvertently tie two different transitions together, I see no gain over simply waiting until the transitions are complete before making the upload. If you build against the version in testing, the application will then have to be rebuilt against the version in unstable before the application can migrate anyway. Where is the advantage? If you allow the application to migrate, it will still have to transition to the new library before the library itself can migrate. So instead of one rebuild, you now have three. How does
Re: wheezy postmortem re rc bugfixing
Le mercredi 08 mai 2013 à 16:51 +0100, Ian Jackson a écrit : * Try to identify the main ways in which bugs can be hard (which might be technical, political, or a mixture) One of the general problems I have been running into include several (sometimes all) of the following patterns. * Few users affected by the bug, and they are not necessarily skilled enough to test updated packages. * Maintainers able to fix the affected packages not able (or willing to setup something complex) to test the fix. * Maintainers with not enough time to work on the release and more focused on experimental. * Consequences of poor upstream design choices. * Upstream insisting (sometimes rightfully) that the only reasonable fix is a major update incompatible with the freeze policy, sometimes with an impact on dozens of other packages. -- .''`. Josselin Mouette : :' : `. `' `- -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1368187202.3585.1341.camel@pi0307572
Re: wheezy postmortem re rc bugfixing
[Charles Plessy, 2013-05-09] For a large number of packages if not all, we should allow the package maintainers to manually migrate their packages to Testing during the Freeze, within boundaries set on debian-devel-announce by the release team. +1 or a soft freeze (as described above) a month (or two) before hard freeze -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130510125746.ga24...@p1otr.com
Re: wheezy postmortem re rc bugfixing
On Wednesday, May 08, 2013 04:51:14 PM Ian Jackson wrote: So I would like to suggest that we should have a thread where we: * Try to identify the main ways in which bugs can be hard (which might be technical, political, or a mixture) * Try to think of workflows which might overcome these problems Currently there are formally two classes of RC bugs relative to a release, ignored and not ignored. Informally not all RC bugs without the $RELEASE- ignore tag are not the same and I think this ambiguity has been a source of uncertainty about where people should focus time (this applies both potential RC bug fixers and the release team, as I see it). Stated non-rigorously, active (not ignored) RC bugs can fall into several bins: 1. Things that definitely must be fixed for release 2. Things that must either be fixed or the package removed 3. Not evaluated 4. Will probably ignore, but not ready to make that call yet If we had additional tags for #1 and #2 to communicate that the release team has evaluated a bug and concluded it's in the must fix category, then that would help people trying to fix RC bugs focus on the things that are known release issues. I have seen release team discussions on bugs along the lines of I will probably ignore that one, but I'm not ready to decide for sure yet. I think release team members are rationally cautious about applying ignore tags because once that's done, it's pretty certain no one will look at it. If there were a tag for will probably ignore then that would both be a clear indicator for RC bug fixers to perhaps focus elsewhere if it's not easy and perhaps help the release team avoid having to rediscuss bugs as much. I've stayed away from suggesting any tag names. I think we can adequately bikeshed that if there's some agreement the added granularity would be worth the added complexity. Scott K -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2783904.tkUOjtZ9EW@scott-latitude-e6320
Re: wheezy postmortem re rc bugfixing
On Fri, 10 May 2013 10:03:46 -0400 Scott Kitterman deb...@kitterman.com wrote: On Wednesday, May 08, 2013 04:51:14 PM Ian Jackson wrote: So I would like to suggest that we should have a thread where we: * Try to identify the main ways in which bugs can be hard (which might be technical, political, or a mixture) * Try to think of workflows which might overcome these problems Currently there are formally two classes of RC bugs relative to a release, ignored and not ignored. Informally not all RC bugs without the $RELEASE- ignore tag are not the same and I think this ambiguity has been a source of uncertainty about where people should focus time (this applies both potential RC bug fixers and the release team, as I see it). Stated non-rigorously, active (not ignored) RC bugs can fall into several bins: 1. Things that definitely must be fixed for release A lot of these determinations could be assisted by some simple metrics. From the sidelines of two releases, I've observed that release team decisions on this kind of tag would likely involve things like the Priority of the affected package, it's involvement in existing transitions, the implications of `dak rm -Rn $package` on ries and the history of the package (e.g. if this is going to lead to the fourth NMU on this package since the freeze started). All of that data can be automatically generated as part of the summary of the RC bug or the package itself, then fed into the information visible to developers reviewing the RC bug list. The more of this triage process that can be automated, the more developers can see a standard process being applied and the less work the release team needs to do for every individual RC bug. The release team need the ability to override the calculations but that isn't a problem. The aim, IMHO, would be to reduce the workload by only putting in an override where required or on explicit request by a developer to investigate a particular bug/package. I'd like it to be that only the release team can set the overrides, unlike BTS severity or tags which requests people avoid ping-pong but cannot prevent it. 2. Things that must either be fixed or the package removed Quite possibly falls out of the triage data for 1. I'd also like this to be tied to some automated removal process, just like the one which was used towards the end of the wheezy freeze. Maintainers ping'd about individual problems with a clear warning that the package is at risk of removal if nothing is done, followed by removal from testing if the bug isn't downgraded or fixed. Again, obvious time limits, clear decisions and processes. I've stayed away from suggesting any tag names. I think we can adequately bikeshed that if there's some agreement the added granularity would be worth the added complexity. I think it would be a useful addition and something which does not need to be restricted to solely within a freeze. We need more of this kind of stuff running constantly (or at least starting a few weeks after the previous stable release), with the time limits being adjusted as we get closer and closer to freeze and then release. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpa3Wz6mK2Pv.pgp Description: PGP signature
Re: wheezy postmortem re rc bugfixing
On 2013-05-09 00:48, anarcat wrote: [...] In fact, I am of the opinion that we should relax the requirements that the release team systematically review every diff posted during the freeze, especially if the freeze is going to last almost a year... That always seemed to me to be an insane amount of work. [...] A. FTR, I believe we (i.e. the RT) never wanted or aimed for a freeze taking a year. I can see how your idea might seem attractive for maintainers, you can get that fix in you just missed or upstream has a fix and don't have to cherry-pick from a lot of other changes. That said, for the former, the freeze date was announced a year ahead of time. Yet, there were quite a few maintainers that seemed to be caught by surprised by the freeze. If you add that slush period, I fear maintainers would just be even more relaxed (because I can always fix it during the slush). To be honest, I am not convinced that people are vigilant enough to avoid doing ABI breakages, so a slush upload might end up starting a transition[1]. What I would like to see is a way to reduce the need for changes post freeze. It was my understanding that time-based freeze was intended to do that - by letting you know ahead of time so you could have your things ready (before last minute). The execution of the time-based freeze might have failed. Also, testing did not serving its purpose of always being in (a near-)releasable state[2] with its 500+ RC bugs at the start of the freeze was not ideal (either?). ~Niels [1] During the Wheezy development cycle we did have quite a few uncoordinated library transitions. Combine that with some people's acceptance of huge diffs post-freeze... [2] At least it is my understanding that this is why we (i.e. the RT) wants testing around. Admittedly a lot of people seem to expect it to be a rolling distribution instead. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/518b430e.2030...@thykier.net
Re: wheezy postmortem re rc bugfixing
On Wed, May 08, 2013 at 04:51:14PM +0100, Ian Jackson wrote:
It is good to have it released now, but I think we are all (mostly?)
agreed that wheezy took longer to release than we would have liked.
In particular, the RC bug count didn't drop quickly enough.
Thanks for bringing this up!
I would like to take this opportunity to post an experience report and
give some conclusions I'd make from that.
During the wheezy cycle I participated in the 2011 Hildesheim BSP. In my
experience BSPs are among the best places to get difficult issues sorted
out. Political issues tend to play less of a role there and it
participants tend to motivate each other not to give up on technical
challenges. I would like to thank all the BSP organizers for their
valuable contributions to the project.
During said BSP I settled on fixing #88010 (yes five digits, policy
violation, {lenny,squeeze}-ignore). It clearly meets the criterion of
hard and fixing it took more than a year. Here are a few observations
on the process:
* The largest amount of time used for fixing went into communication.
Sometimes I would wait for multiple months to receive an essential
answer from involved parties. This had a multitude of reasons and I
would like to avoid a blame game here, but this ultimately resulted
in missing the freeze and later resulted in last-minute changes.
* There were a great many people who helped me with technical aspects,
that were sufficiently isolated and did not require a broad view on
the issue. This applies especially to the changes in packaging, the
involved perl code, the usage of dpkg triggers and the transition
tool set.
* Even though I had a variety of people review the changes introduced,
the first attempts resulted in a variety of new failure modes.
Remark: The PPA thingy might be part of the solution here. As it
would make testing transitions easier.
* Try to think of workflows which might overcome these problems
Given the experience above and the experience with other RC bugs, I
would like to suggest to form semi-spontaneous teams around some RC
bugs. The rationale here being is that some hard RC bugs tend to be
quite complex and complexity made me give up on other issues. We already
have the notion of owner in the BTS, but its usage appears to be
limited (and I didn't use it either for RC bugs so far). By forming a
team around a particular issue, we can contain the complexity and
motivate each other. This is not to say that such a team is to do the
full fix, but to give a direction and coordinate the involved parties.
The team would be tasked with avoiding stalls in the progress, pinging
and possibly timing out involved parties. Possibly writing regular
progress summaries would also be helpful to evaluate the performance of
the team. Such summaries would also make switching the owner later
easier. This model should also work with single person teams, but I'd
fear that a single person could more easily run into political
acceptance issues.
This is just a rough sketch so far, and I cannot tell whether it
actually works. Rereading the above paragraph, it sounds a bit like
mini release goal.
Helmut
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130509090056.ga28...@alf.mars
Re: wheezy postmortem re rc bugfixing
On 09/05/13 at 08:32 +0200, Niels Thykier wrote: The execution of the time-based freeze might have failed. Also, testing did not serving its purpose of always being in (a near-)releasable state[2] with its 500+ RC bugs at the start of the freeze was not ideal (either?). I think that one problem is that our current workflows are based on the illusion that all RC bugs are equal. Our tools (e.g. http://udd.debian.org/bugs.cgi) should get better at differenciating different kinds of RC bugs. For example: - old RC bugs vs new RC bugs: experienced bug squashers should focus on the old RC bugs, not on the maybe-trivial-to-fix new RC bugs. - RC bugs affecting popular/important packages vs RC bugs affecting minor packages (that we could remove without). We should include such distinctions in the various graphs we use to track progress. Also, we should be more agressive at getting down the number of RC bugs by automatically removing RC-buggy not-so-important packages. For example, if we keep the current time-based freeze policy for jessie, we could announce that all not-so-important RC-buggy packages will be removed from testing on freeze date. Lucas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130509105503.ga21...@xanadu.blop.info
Re: wheezy postmortem re rc bugfixing
Hi! Lucas Nussbaum lu...@debian.org writes: Also, we should be more agressive at getting down the number of RC bugs by automatically removing RC-buggy not-so-important packages. For example, if we keep the current time-based freeze policy for jessie, we could announce that all not-so-important RC-buggy packages will be removed from testing on freeze date. I'm not so persuaded this will actually improve anything. During lenny freeze I've fixed a couple of these easy RC bugs on unmaintained leave packages when I had some 30 minutes of time and no good Idea of what to do. I have had similar timeslots too small to actually get into more difficult RC bugs but couldn't find anything to do during squeeze and especially wheezy freeze so I ended up doing *nothing*. Removing RC-buggy (and potentially not-taken-care-of) packages early may increase the average quality of software in the release (because these fixes mostly do exactly as much as is needed to fix the RC bug) but I'm far from persuaded it will increase release speed. Different thing for will-remove and dropping such packages late of course. Christoph -- 9FED 5C6C E206 B70A 5857 70CA 9655 22B9 D49A E731 Debian Developer | Lisp Hacker | CaCert Assurer -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/8761ysbe09@hepworth.siccegge.de
Re: wheezy postmortem re rc bugfixing
On Thu, May 9, 2013 at 12:55:03 +0200, Lucas Nussbaum wrote: Also, we should be more agressive at getting down the number of RC bugs by automatically removing RC-buggy not-so-important packages. For example, if we keep the current time-based freeze policy for jessie, we could announce that all not-so-important RC-buggy packages will be removed from testing on freeze date. define:not-so-important. I'm sure that'll be fun. Cheers, Julien signature.asc Description: Digital signature
Re: wheezy postmortem re rc bugfixing
Lucas Nussbaum lu...@debian.org writes: Also, we should be more agressive at getting down the number of RC bugs by automatically removing RC-buggy not-so-important packages. This sounds like a good idea. If somebody is interested in the package they can easily reintroduce it after they have fixed the bug. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/84fvxwpez8@sauna.l.org
Re: wheezy postmortem re rc bugfixing
On 09/05/13 at 13:20 +0200, Julien Cristau wrote: On Thu, May 9, 2013 at 12:55:03 +0200, Lucas Nussbaum wrote: Also, we should be more agressive at getting down the number of RC bugs by automatically removing RC-buggy not-so-important packages. For example, if we keep the current time-based freeze policy for jessie, we could announce that all not-so-important RC-buggy packages will be removed from testing on freeze date. define:not-so-important. I'm sure that'll be fun. Given that: * the not-so-important status would only be used to decide that some packages can be safely removed from testing when they are RC-buggy, * the release team has authority on deciding what's inside testing I think that the release team can decide on such criterias. Actually, that's something the team already did during the wheezy freeze. I'm just proposing to do that doing that earlier, with more advertised criterias. For example, we could use the following rules: * source packages with binary packages of priority 'standard', 'important', 'required' are important * source packages building udebs are important * source packages building binary packages installed on more than 5% of popcon-reporting packages (that's popcon 7714 currently) are important * build-dependencies of important packages are important A quick hack resulted in: http://udd.debian.org/cgi-bin/important_packages.cgi http://anonscm.debian.org/gitweb/?p=collab-qa/udd.git;a=blob;f=web/cgi-bin/important_packages.cgi Which gives 2567 such important packages. Of the 963 RC bugs affecting sid currently, only 185 affect sid's important packages. Lucas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130509124450.ga22...@xanadu.blop.info
wheezy postmortem re rc bugfixing
It is good to have it released now, but I think we are all (mostly?) agreed that wheezy took longer to release than we would have liked. In particular, the RC bug count didn't drop quickly enough. Firstly, I want to say that I don't think this was anyone's fault. So I don't want to lay any blame. I think we should consider, though, how we can try to do better next time. For me the key problem is this: we are lacking a useful workflow for fixing RC bugs (well, many of them, anyway). Easy RC bugs tend to get fixed early in the freeze or even beforehand. But as the freeze continues, what are left are hard bugs. I don't think we necessarily have a lack of effort. I know that many people were _trying_ to improve the situation with RC bugs. But it's difficult. One ends up picking bugs more or less at random and then reading them. One will then naturally discover that the bug is difficult somehow - after all, if were easy it would probably have been fixed already. Unless one is already surrounded by enthusiastic and authoritative experts with a lot of spare time and (where needed) the right authority, one can end up blocked and discouraged. So I would like to suggest that we should have a thread where we: * Try to identify the main ways in which bugs can be hard (which might be technical, political, or a mixture) * Try to think of workflows which might overcome these problems * In particular, try to think of workflows and/or support tools which might be able to connect the people with the skills and/or authority to solve the problem, with the bug - and help motivate those people to do the work needed to unblock the bug. * Consider other ways in which our RC-bug-fixing efforts can be improved, especially during the latter part of the freeze. I have deliberately avoided suggesting any answers to these questions myself in this article. But I may do so later. Also we should try to treat this discussion as a kind of semi-brainstorm: if someone makes what seems like a poor suggestion, try to improve on it or post a better one yourself, rather than merely criticising theirs. That will help keep the discussion open and avoid it getting hung up on specific disagreements (which is of course a think that mailing list conversations have a tendency to to). Thanks, Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20874.29810.191974.559...@chiark.greenend.org.uk
Re: wheezy postmortem re rc bugfixing
On Wed, 8 May 2013 16:51:14 +0100 Ian Jackson ijack...@chiark.greenend.org.uk wrote: So I would like to suggest that we should have a thread where we: I suspect a wiki page will be needed at some point... * Try to identify the main ways in which bugs can be hard (which might be technical, political, or a mixture) These are the issues which discouraged me from working on particular RC bugs during this release and the one before it. * Hard to reproduce - uncommon packages, specialised setups, specific hardware or hardware configurations, intermittent issues. * Un- / under maintained packages not yet orphaned. IMHO we should be much more aggressive with such packages - strict time limits for all RC-buggy leaf packages, regardless of the uncertainties of popcon, leading to at least automatic removal from testing. Warning bugs against reverse dependencies of non-leaf packages warning of problems. (We have this now in the PTS for WNPP issues, an extension to RC bugs in dependencies could also be really useful.) * Disagreements between submitter maintainer / fixer * Architecture-specific bugs for less common ports - maybe be more aggressive here also on which architectures are deemed fit for release on the basis of the occurrence and likely delays caused by such bugs. * Non-standard packaging or build system or packages using methods known to make NMU's difficult. * Languages other than C, C++, perl, shell or python, reducing the possible number of people who can get a full overview of the package. * Lapsed release goals (like building sanely twice in a row, not just in a clean chroot but during a typical NMU process too, so that test changes can be easily and quickly reverted and the package rebuilt.) Particularly important for packages which take a long time to build or have esoteric / uncommon build-dependencies. * Poor quality documentation within the package, especially for build-system idiosyncrasies and internal API/ABI requirements. Simple things like a comment in each source code file saying what the code in that file is meant to achieve. foo.c - wraps the bar interface in the foo class etc. Other steps to take as preventative measures: * More QA runs through the archive prior to freeze to catch things like embedded libraries or binaries without sources or licence irregularities, FTBFS and piuparts errors. The actual decision of the freeze date could be made to be reliant on a % clean report from such runs. * Make it a *MUST* that all transitions, no matter how small, are checked with the release team starting from as soon as the freeze is announced (not just after it starts) such that uploads which start a transition could be pushed into DELAYED or REJECTed automatically. (Not easy to implement this one, I know.) * Try to think of workflows which might overcome these problems debian/rules must be a makefile, only specific packages can be allowed to not use debhelper, toolchain packages to be frozen earlier than the rest of the archive... I think the Debian PPA approach could also ease the process substantially - we could, for example, require that all uploads during a freeze which do not fix RC bugs must go into a Debian PPA. This reduces the need for t-p-u builds and should help the slow isolation of desired changes from packaging fluff. * In particular, try to think of workflows and/or support tools which might be able to connect the people with the skills and/or authority to solve the problem, with the bug - and help motivate those people to do the work needed to unblock the bug. That sounds like a requirement for tags and a search interface allied with rc-alert or similar. BSP's could also make use of such tags, possibly via an interface akin to the reverse of dd-list. Maybe the debtags information could be used to feed data into UDD relating to the likely experience of the maintainers based on the implemented-in and works-with tags of the packages which they maintain? Then a BSP can just feed the names into the search and get a list of likely bug numbers. The data would also help identify tags which have poor coverage and high proportions of bugs. * Consider other ways in which our RC-bug-fixing efforts can be improved, especially during the latter part of the freeze. I have deliberately avoided suggesting any answers to these questions myself in this article. But I may do so later. Also we should try to treat this discussion as a kind of semi-brainstorm: if someone makes what seems like a poor suggestion, try to improve on it or post a better one yourself, rather than merely criticising theirs. That will help keep the discussion open and avoid it getting hung up on specific disagreements (which is of course a think that mailing list conversations have a tendency to to). -- Neil Williams = http://www.linux.codehelp.co.uk/ pgp678phNWTJP.pgp Description: PGP signature
Re: wheezy postmortem re rc bugfixing
On Thu, May 9, 2013 at 12:28 AM, Neil Williams wrote: (We have this now in the PTS for WNPP issues, an extension to RC bugs in dependencies could also be really useful.) Thanks for the idea, I'll pursue implementing this with QA infrastructure folks. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAKTje6EyEAm6+WNNyVez=w4blh640+-hbb5pfa0l9m36wdy...@mail.gmail.com
Re: wheezy postmortem re rc bugfixing
How about a slush? A few projects have this period where changes are not completely forbidden, but slightly restricted. For example, we could have a period where new upstream releases (yes, with huge diffstats) would be accepted if they fix a RC bug. In fact, I am of the opinion that we should relax the requirements that the release team systematically review every diff posted during the freeze, especially if the freeze is going to last almost a year... That always seemed to me to be an insane amount of work. And yes, I know that we have a progression of exceptions for the freeze already, I just feel that we could add an extra window... But maybe that's just me. :) A. -- We will create a civilization of the Mind in Cyberspace. May it be more humane and fair than the world your governments have made before. - John Perry Barlow pgpxS6_0I2gW7.pgp Description: PGP signature
Re: wheezy postmortem re rc bugfixing
Le Wed, May 08, 2013 at 06:48:01PM -0400, anarcat a écrit : In fact, I am of the opinion that we should relax the requirements that the release team systematically review every diff posted during the freeze, especially if the freeze is going to last almost a year... That always seemed to me to be an insane amount of work. I agree. For a large number of packages if not all, we should allow the package maintainers to manually migrate their packages to Testing during the Freeze, within boundaries set on debian-devel-announce by the release team. It looks like DAK is growing a set of nice commands, and that developers will be familiar with them by using PPAs, so let's use them for that purpose as well ! Like any other service (BTS, uploads to Unstable, ...), repeated abuse can be solved by blocking the access, and in the worst case scenario, an expulsion procedure. The goal is nevertheless to save time to everybody, and to make the released stable version more exciting by including more upstream fixes and improvements. Looking at http://bugs.debian.org/release-critical, it takes only a few monthes to find hundreds of new RC bugs in stable releases. I think that we should focus on regressions rather than RC bugs. New upstream releases in simple packages tend to solve problems in the core parts of the packages, and may introduce new parts that are not fully tested. It is to our benefit and the benefit of our users to incorporate in Stable new upstream releases that fix bugs in existing tools, even if they introduce new tools that are not as well tested, because on the other hand these new tools do not have a user base as large as the older ones. Cheers, -- Charles Plessy Debian Med packaging team, http://www.debian.org/devel/debian-med Tsurumi, Kanagawa, Japan -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130508232658.ga30...@falafel.plessy.net
