Re: Update testing policy: how to use Bodhi
On Wed, Mar 31, 2010 at 01:21, Adam Williamson awill...@redhat.com wrote: On Sat, 2010-03-27 at 16:33 +0100, Till Maas wrote: 8. The package updated sucessfully, but was not used intentionally. No breakage noticed. This shows, that at least on the test machine, there are no broken deps, conflicts or broken scriptlets. In my head I sort of had this wrapped up with 'no regressions', but you might be right that it's best to split it out as a choice. I'm not convinced by the value of this one. IMHO, testing is supposed to be a conscious action. You don't test unintentionally, and if you installed an update unintentionally, it means (to me) that you didn't test it. I guess it's related to the issue of do we want to only provide updates that add value or do we want to push any updates that don't break anything? In any case, like I said I've started playing with the idea in the Bodhi tg2 branch (no ETA yet, I do that on my spare time and I have very few, and lots of other stuff have to be written in the tg2 branch anyway). The way I did it, several karma types can be defined (and I'm taking suggestions on those types from these emails), the only thing that matters is: are they global to the update or are they tied to a specific bug? That's what will be most important when displaying the comment form. Until now, the only proposed thing that wouldn't really fit is number 7: fixes bug X, but does not claim to fix it. It could probably be a special case of the karma type tied to a specific bug, except the bug is not known in advance but rather specified by the tester. Gotta think about it. :) -- Mathieu Bridon -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Wed, Mar 31, 2010 at 11:37:33AM +0200, Mathieu Bridon wrote: On Wed, Mar 31, 2010 at 01:21, Adam Williamson awill...@redhat.com wrote: On Sat, 2010-03-27 at 16:33 +0100, Till Maas wrote: 8. The package updated sucessfully, but was not used intentionally. No breakage noticed. This shows, that at least on the test machine, there are no broken deps, conflicts or broken scriptlets. In my head I sort of had this wrapped up with 'no regressions', but you might be right that it's best to split it out as a choice. I'm not convinced by the value of this one. IMHO, testing is supposed to be a conscious action. You don't test unintentionally, and if you installed an update unintentionally, it means (to me) that you didn't test it. I guess it's related to the issue of do we want to only provide updates that add value or do we want to push any updates that don't break anything? IMHO the question what updates to provide is not related to this. Even if an update is there to fix something, it does not mean that one can or will test it completely (special hardware might be required). In this case it is still interesting to know, whether it installs cleanly or not. And testing whether it updates cleanly can still be done intentionally, even when the package is not used. Just taking a look at the output of yum --enablerepo=*-testing update is enough for this. Also there are packages like libraries, that will probably never be intentionally manually tested, because it is not that obvious when a library is used by another program and which code paths are used. Regards Till pgp31HcPnssH1.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Wed, Mar 31, 2010 at 01:13:18PM +0200, Björn Persson wrote: Till Maas wrote: Even if an update is there to fix something, it does not mean that one can or will test it completely (special hardware might be required). In this case it is still interesting to know, whether it installs cleanly or not. And testing whether it updates cleanly can still be done intentionally, even when the package is not used. Just taking a look at the output of yum --enablerepo=*-testing update is enough for this. I thought the plan was to have AutoQA verify that packages install cleanly. Isn't it better to automate such simple checks and save human resources for the things that can't be automated? Or do you think even just installing updates is too complex to be tested automatically? I agree, that automation is the way to do this, but still it might not catch everything and there is not much effort required to do this testing (just look at the yum output) and report it. But there are also updates that might interfer with RPMFusion, like it happened with gstreamer. Here AutoQA might not be able to test this, because of legal issues. Regards Till pgp4Yo7RaAYwu.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On 27/03/10 04:12, Adam Williamson wrote: On Sat, 2010-03-27 at 03:50 +0100, Kevin Kofler wrote: So I don't think blocking an update outright for having received type 2 feedback is sane at all. Sigh. That was why I said not to sidetrack the discussion because it was the least important bit of the post. It was just an example of how easily the policy can be adapted. I'm really not interested in thrashing out the tiny details of *that* in *this* thread, that is not what it's for. I had a whole paragraph about the possibility of an override mechanism for maintainers which I left out precisely in order to avoid this kind of discussion, but apparently that wasn't enough... I'm not sure if your usage policy covers changes to Bodhi, but how about the system emailing the upstream developers (direct and/or email lists) when a release is made available for testing/release and also on any problems found ? Terry -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Sat, 2010-03-27 at 08:19 +, Terry Barnaby wrote: I'm not sure if your usage policy covers changes to Bodhi, but how about the system emailing the upstream developers (direct and/or email lists) when a release is made available for testing/release and also on any problems found ? That's not really part of this proposal. It's a reasonable feature request for Bodhi, though. You could file it with the Bodhi developers (I think infrastructure trac - https://fedorahosted.org/fedora-infrastructure/ - is the appropriate venue). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
Terry Barnaby ter...@beam.ltd.uk writes: I'm not sure if your usage policy covers changes to Bodhi, but how about the system emailing the upstream developers (direct and/or email lists) when a release is made available for testing/release and also on any problems found ? As an upstream developer, I can hardly think of a quicker way to piss me off than if every distro were to start sending me such nagmail. I would not want such a thing turned on on *any* of my packages. regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Sat, 2010-03-27 at 08:19 +, Terry Barnaby wrote: I'm not sure if your usage policy covers changes to Bodhi, but how about the system emailing the upstream developers (direct and/or email lists) when a release is made available for testing/release and also on any problems found ? I would be _very_ careful with sending autogenerated spam to random people that have not signed up for it. I would expect many people to react negatively to it. Sending a small note along the lines of 'Hey, I'm packaging your software for Fedora. How about you mention on the website that it is not only available in Ubuntu but also in Fedora' is a quite different thing, and should be done. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Fri, Mar 26, 2010 at 03:49:28PM -0700, Adam Williamson wrote: 1. I have tried this update in my regular day-to-day use and seen no regressions. 2. I have tried this update in my regular day-to-day use and seen a regression: bug #XX. 3. (Where the update claims to fix bug #XX) I have tried this update and found that it does fix bug #XX. 4. (Where the update claims to fix bug #XX) I have tried this update and found that it does not fix bug #XX. 5. I have performed the following planned testing on the update: (link to test case / test plan) and it passes. 6. I have performed the following planned testing on the update: (link to test case / test plan) and it fails: bug #XX. I have some additions: 7. fixes bug X, but does not claim to fix it This can often happen with hardware related bugs, e.g. with the kernel where something starts to work again 8. The package updated sucessfully, but was not used intentionally. No breakage noticed. This shows, that at least on the test machine, there are no broken deps, conflicts or broken scriptlets. Also it would be nice to provide hardware testing feedback, e.g. for Xorg updates to say Works with nouveau, Geforce XY, using VGA out and XV, which then shows that e.g. 3D support, DVI out or multi screen support was not tested. This is kind of related to testing with a test plan, but having this data available in a format that can be easily parsed, would be nice, too. Maybe this could be done with adding smolt information in the feedback and the tested features (XV, VGA, DVI, 3D, ...) and the update needs to have some meta data, which kind of devices are supported (e.g. only Geforce devices for the nouveau driver package). Regards Till pgpHcRSV60m0K.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Update testing policy: how to use Bodhi
Hi, folks. At the last QA meeting, I volunteered (dumb of me!) to draft a policy for testing updates - basically, a policy for what kind of feedback should be posted in Bodhi for candidate updates. This turns out to be pretty hard. =) Thinking about it from an high-level perspective like this, I think it becomes pretty clear that the current system is just broken. The major problem is it attempts to balance things that don't really balance. It lets you say 'works for me' or 'doesn't work' and then sums the two and subtracts the second from the first to give you a 'rating' for the update. This doesn't really mean anything. As has been rehashed many times, there are situations where an update with a positive rating shouldn't go out, and situations where an update with a negative rating should. So the current system isn't really that great. I can't think of a way to draft a policy to guide the use of the current system in such a way that it will be really reliable. I think it'd be much more productive to revise the Bodhi feedback system alongside producing a policy. So, here's a summary of what the new system should aim for. At the high level, what is this system for? It's there for three purposes: 1) to provide maintainers with information they can use in deciding whether to push updates. 2) to provide a mechanism for mandating a certain minimum level of manual testing for 'important' packages, under Bill Nottingham's current update acceptance criteria proposal. 3) to provide an 'audit trail' we can use to look back on how the release of a particular update was handled, in the case where there are problems. Given the above, we need to capture the following types of feedback, as far as I can tell. I don't think there is any sensible way to assign numeric values to any of this feedback. I think we have to trust people to make sensible decisions as long as it's provided, in accordance with any policy we decide to implement on what character updates should have. 1. I have tried this update in my regular day-to-day use and seen no regressions. 2. I have tried this update in my regular day-to-day use and seen a regression: bug #XX. 3. (Where the update claims to fix bug #XX) I have tried this update and found that it does fix bug #XX. 4. (Where the update claims to fix bug #XX) I have tried this update and found that it does not fix bug #XX. 5. I have performed the following planned testing on the update: (link to test case / test plan) and it passes. 6. I have performed the following planned testing on the update: (link to test case / test plan) and it fails: bug #XX. Testers should be able to file multiple types of feedback in one operation - for instance, 4+1 (the update didn't fix the bug it claimed to, but doesn't seem to cause any regressions either). Ideally, the input of feedback should be 'guided' with a freeform element, so there's a space to enter bug numbers, for instance. There is one type of feedback we don't really want or need to capture: I have tried this update and it doesn't fix bug #XX, where the update doesn't claim to fix that bug. This is a quite common '-1' in the current system, and one we should eliminate. I think Bill's proposed policy can be modified quite easily to fit this. All it would need to say is that for 'important' updates to be accepted, they would need to have one 'type 1' feedback from a proven tester, and no 'type 2' feedback from anyone (or something along those lines; this isn't the main thrust of my post, please don't sidetrack it too much :). The system could do a count of how many of each type of feedback any given update has received, but I don't think there's any way we can sensibly do some kind of mathematical operation on those numbers and have a 'rating' for the update. Such a system would always give odd / undesirable results in some cases, I think (just as the current one does). I believe the above system would be sufficiently clear that there would be no need for such a number, and we would be able to evaluate updates properly based just on the information listed. What are everyone's thoughts on this? Thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
Hi, On Fri, Mar 26, 2010 at 23:49, Adam Williamson awill...@redhat.com wrote: [... snip ...] 1. I have tried this update in my regular day-to-day use and seen no regressions. 2. I have tried this update in my regular day-to-day use and seen a regression: bug #XX. 3. (Where the update claims to fix bug #XX) I have tried this update and found that it does fix bug #XX. 4. (Where the update claims to fix bug #XX) I have tried this update and found that it does not fix bug #XX. 5. I have performed the following planned testing on the update: (link to test case / test plan) and it passes. 6. I have performed the following planned testing on the update: (link to test case / test plan) and it fails: bug #XX. This is basically what Doug had proposed, except that you added 5. and 6. I know Luke also likes the idea, and I've been toying with implementing Doug's idea in the Bodhi2 branch. Adding those 2 more karma types shouldn't be too hard. Testers should be able to file multiple types of feedback in one operation - for instance, 4+1 (the update didn't fix the bug it claimed to, but doesn't seem to cause any regressions either). Ideally, the input of feedback should be 'guided' with a freeform element, so there's a space to enter bug numbers, for instance. That's what I had in mind, but the Bodhi2 branch currently doesn't have any controllers/template, so it will be some time before I have something to show. I think Bill's proposed policy can be modified quite easily to fit this. All it would need to say is that for 'important' updates to be accepted, they would need to have one 'type 1' feedback from a proven tester, and no 'type 2' feedback from anyone (or something along those lines; this isn't the main thrust of my post, please don't sidetrack it too much :). That's actually one of the points I was missing : rules for (auto-)push / (auto-)unpush. But yeah, it can be agreed on later. The system could do a count of how many of each type of feedback any given update has received, but I don't think there's any way we can sensibly do some kind of mathematical operation on those numbers and have a 'rating' for the update. Such a system would always give odd / undesirable results in some cases, I think (just as the current one does). I believe the above system would be sufficiently clear that there would be no need for such a number, and we would be able to evaluate updates properly based just on the information listed. What are everyone's thoughts on this? Thanks! I totally agree that this would be far more desirable than the current +1/-1 system. -- Mathieu Bridon -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote: Hi, On Fri, Mar 26, 2010 at 23:49, Adam Williamson awill...@redhat.com wrote: [... snip ...] 1. I have tried this update in my regular day-to-day use and seen no regressions. 2. I have tried this update in my regular day-to-day use and seen a regression: bug #XX. 3. (Where the update claims to fix bug #XX) I have tried this update and found that it does fix bug #XX. 4. (Where the update claims to fix bug #XX) I have tried this update and found that it does not fix bug #XX. 5. I have performed the following planned testing on the update: (link to test case / test plan) and it passes. 6. I have performed the following planned testing on the update: (link to test case / test plan) and it fails: bug #XX. This is basically what Doug had proposed, except that you added 5. and 6. Great, glad to hear we're thinking in the same direction from different angles :) Do you have a link to his proposal? I don't recall reading it. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Update testing policy: how to use Bodhi
On Sat, 2010-03-27 at 03:50 +0100, Kevin Kofler wrote: So I don't think blocking an update outright for having received type 2 feedback is sane at all. Sigh. That was why I said not to sidetrack the discussion because it was the least important bit of the post. It was just an example of how easily the policy can be adapted. I'm really not interested in thrashing out the tiny details of *that* in *this* thread, that is not what it's for. I had a whole paragraph about the possibility of an override mechanism for maintainers which I left out precisely in order to avoid this kind of discussion, but apparently that wasn't enough... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
