Re: Silly bodhi karma games
On Fri, 20 Mar 2015 21:07:19 +0800, Christopher Meng wrote: On 3/20/15, Kamil Paral wrote: We need to let those hundreds or thousands of people running with updates-testing to install it as well, and give it the invisible thumbs up, which is not present in bodhi, but which is expressed by the lack of critical issues reported in bodhi, bugzilla or on mailing lists. And the best way to achieve that at the moment is to make the updates sit it updates-testing for at least a certain time. I doubt if these hundreds or thousands of people are already registered yet in FAS. But I agree with you. I think you've misunderstood the invisible thumbs up in above paragraph. It refers to the unknown number of users of the updates-testing repo, who are _potentially_ affected by a test-update because it's a package that's installed on their machine as a strictly needed system component. It's something they don't know whether/when/how it is used, but it's likely that it's used somehow. They don't know either how to test the changes introduced by the test-update. So, these people would not rush into bodhi with a +1 that claims the test-update is fine for everyone else, but they are prepared to report any oddities they find = which can be much more important when it helps with withdrawing a test-update that's bad _actually_. Reporting crashes via ABRT or manually in bugzilla does not need an account in FAS. Better, of course, is negative feedback in bodhi (especially for those maintainers who don't pay much attention to bugzilla). Lots of test-updates become stable updates either with or without positive feedback from any testers. That turns many Fedora users into consumers of test-updates, just that they get them only when they appear in the updates repo. A fresh example of how the testing process has failed completely for F20 and EPEL 7 is this exaile ticket: https://bugzilla.redhat.com/1203573#c3 No positive feedback from any testers. No bugzilla tickets linked from within the bodhi ticket, so the bug reporters have not been notified about this version upgrade and have not been given a chance to evaluate it. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On Fri, 20 Mar 2015 04:55:34 -0400 (EDT), Kamil Paral wrote: Just one note - A great one, IMO! the number of people giving karma is not the same as the number of people actually using that proposed package build. For example, some of system-critical libraries don't usually receive too much karma, because people are not sure how to test them, and whether running app X or Y is sufficient to test them. So they don't report any feedback for them. But if they have updates-testing enabled, they still test them at least unknowingly. If those libraries got broken, they would know it very fast, and bugs would get discovered. So, in this case, we have almost no positive feedback things work, but the important thing here is the absence of the negative feedback. And that's the purpose of the timeout. Exactly. Just because a package is installed does not mean it is in use. That the very latest kernel works for me does not imply it works for everyone else with different hardware, different filesystems, and so on. I consider it dangerous to file +1 in such a case. I find it more important to actually run with updates-testing enabled and watch out for breakage to stop from entering the stable repo. And for lots of core components, for each package, a tiny guide on how to test basic funtionality would be good. From time to time, a few packagers add such testing instructions to their updates. For example, they request a specific test to be performed which they consider essential/fundamental. Still, with the steady flood of updates (and possibly a lot of flawless ones in there), it would be a tedious task to perform individual tests manually every other day and submit positive karma points manually, too. What's much more important is _the chance_ to give test-updates a try before it will be declared stable officially. That may involve - waiting for the stuff to appear on your nearby mirror, - installing the updates (possibly on another different machine, too), - rebooting (perhaps more than once to be certain everything still works), - daily usage specially of the changed software/packages. Meanwhile, if expert testers are much faster than the timeout and vote on the update to make it reach its karma threshold, fine. They are responsible for doing so. In some cases it may be more clever, more helpful, to vote +0 and point out that the test-update may affect other users differently than the tester. That is also covered by http://fedoraproject.org/wiki/QA:Update_feedback_guidelines#Neutral_feedback_or_no_feedback.3F as the case where the update seems to work for you but you don't have the insight to claim that it will be troubleless also for other users. This also applies to many leaf packages. We have many more people running with updates-testing than people regularly giving feedback to everything they have installed (my personal guess would be by several orders of magnitude). Even if the leaf package doesn't receive any karma, that timeout interval is very useful, because it gives people time to report issues, if they spot any. From my QA point of view, that timeout might be even more important than the feedback provided. +1 ;-) Except for negative feedback, which is most important when it is not a false positive. And I have been very angry about some critical path packages pushing to stable with just 10 karma in _one or two days_. There are millions of Fedora users, with various hardware and software combinations - we can't afford to push something like kernel, mesa or X to stable if only 10 users give it a thumbs up. We need to let those hundreds or thousands of people running with updates-testing to install it as well, and give it the invisible thumbs up, which is not present in bodhi, but which is expressed by the lack of critical issues reported in bodhi, bugzilla or on mailing lists. And the best way to achieve that at the moment is to make the updates sit it updates-testing for at least a certain time. I sit here stunned, not believing what I just read. Thank you very much for that post.Great! Not everything is lost it seems. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On 3/20/15, Kamil Paral kpa...@redhat.com wrote: We need to let those hundreds or thousands of people running with updates-testing to install it as well, and give it the invisible thumbs up, which is not present in bodhi, but which is expressed by the lack of critical issues reported in bodhi, bugzilla or on mailing lists. And the best way to achieve that at the moment is to make the updates sit it updates-testing for at least a certain time. I doubt if these hundreds or thousands of people are already registered yet in FAS. But I agree with you. -- Yours sincerely, Christopher Meng http://cicku.me -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
RE: Silly bodhi karma games
+1 I suggest giving us karma testers more in the way of test cases. If it is possible to automate tests by scripting them, perhaps you could even add some elements of integration testing here. I generally skip past updates that there is either not a test case for, something I've never heard of, or something I have no idea how to test. I don’t' think it's fair to anyone to give positive karma for some library I can't even run a script or something against. JC -Original Message- From: test-boun...@lists.fedoraproject.org [mailto:test-boun...@lists.fedoraproject.org] On Behalf Of Kamil Paral Sent: Friday, March 20, 2015 4:56 AM To: For testing and quality assurance of Fedora releases Subject: Re: Silly bodhi karma games I think it can be a judgement call on certain packages. For example, I maintain the Review Board packages which almost never get karma from more than one person (and that usually only for whichiver Fedora or EPEL branch that person is currently deploying to). Even at karma 1, most Review Board packages sit in updates-testing until the timeout passes. Just one note - the number of people giving karma is not the same as the number of people actually using that proposed package build. For example, some of system-critical libraries don't usually receive too much karma, because people are not sure how to test them, and whether running app X or Y is sufficient to test them. So they don't report any feedback for them. But if they have updates-testing enabled, they still test them at least unknowingly. If those libraries got broken, they would know it very fast, and bugs would get discovered. So, in this case, we have almost no positive feedback things work, but the important thing here is the absence of the negative feedback. And that's the purpose of the timeout. This also applies to many leaf packages. We have many more people running with updates-testing than people regularly giving feedback to everything they have installed (my personal guess would be by several orders of magnitude). Even if the leaf package doesn't receive any karma, that timeout interval is very useful, because it gives people time to report issues, if they spot any. From my QA point of view, that timeout might be even more important than the feedback provided. And I have been very angry about some critical path packages pushing to stable with just 10 karma in _one or two days_. There are millions of Fedora users, with various hardware and software combinations - we can't afford to push something like kernel, mesa or X to stable if only 10 users give it a thumbs up. We need to let those hundreds or thousands of people running with updates-testing to install it as well, and give it the invisible thumbs up, which is not present in bodhi, but which is expressed by the lack of critical issues reported in bodhi, bugzilla or on mailing lists. And the best way to achieve that at the moment is to make the updates sit it updates-testing for at least a certain time. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On Fri, 2015-03-20 at 10:57 -0400, Jonathan Calloway wrote: +1 I suggest giving us karma testers more in the way of test cases. If it is possible to automate tests by scripting them, perhaps you could even add some elements of integration testing here. I generally skip past updates that there is either not a test case for, something I've never heard of, or something I have no idea how to test. I don’t' think it's fair to anyone to give positive karma for some library I can't even run a script or something against. That's certainly something that would help out a lot, but it's an awful lot of work and it can feel like trying to empty the Pacific with a leaky bucket, which I think is why people don't do more work on it :/ -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
I think it can be a judgement call on certain packages. For example, I maintain the Review Board packages which almost never get karma from more than one person (and that usually only for whichiver Fedora or EPEL branch that person is currently deploying to). Even at karma 1, most Review Board packages sit in updates-testing until the timeout passes. Just one note - the number of people giving karma is not the same as the number of people actually using that proposed package build. For example, some of system-critical libraries don't usually receive too much karma, because people are not sure how to test them, and whether running app X or Y is sufficient to test them. So they don't report any feedback for them. But if they have updates-testing enabled, they still test them at least unknowingly. If those libraries got broken, they would know it very fast, and bugs would get discovered. So, in this case, we have almost no positive feedback things work, but the important thing here is the absence of the negative feedback. And that's the purpose of the timeout. This also applies to many leaf packages. We have many more people running with updates-testing than people regularly giving feedback to everything they have installed (my personal guess would be by several orders of magnitude). Even if the leaf package doesn't receive any karma, that timeout interval is very useful, because it gives people time to report issues, if they spot any. From my QA point of view, that timeout might be even more important than the feedback provided. And I have been very angry about some critical path packages pushing to stable with just 10 karma in _one or two days_. There are millions of Fedora users, with various hardware and software combinations - we can't afford to push something like kernel, mesa or X to stable if only 10 users give it a thumbs up. We need to let those hundreds or thousands of people running with updates-testing to install it as well, and give it the invisible thumbs up, which is not present in bodhi, but which is expressed by the lack of critical issues reported in bodhi, bugzilla or on mailing lists. And the best way to achieve that at the moment is to make the updates sit it updates-testing for at least a certain time. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On Sun, 2015-03-15 at 14:47 +0100, Michael Schwendt wrote: Can we please fix those people, who hate bodhi so much that they feel the need to submit updates with a low karma: 1 threshold? It just doesn't work and causes breakage too often. Broken dependencies and/or breakage at runtime. Keep in mind that it takes some time for packages to be picked up by the world-wide mirroring system. By the time such updates appear in the repositories, in bodhi they are marked stable already or are even on their way into the updates repo. That makes it impossible to test them in time and leave feedback. It's too late. Yeah, you want to rush out builds because you don't care. That sucks. I think it can be a judgement call on certain packages. For example, I maintain the Review Board packages which almost never get karma from more than one person (and that usually only for whichiver Fedora or EPEL branch that person is currently deploying to). Even at karma 1, most Review Board packages sit in updates-testing until the timeout passes. Now, this makes sense for Review Board because it's a leaf package and an application with a fairly limited audience. For packages in wider use or those that are dependencies for other projects, I think having a higher threshold makes sense. Hopefully, some of the new changes in Bodhi 2 will improve upon this situation. I hear that's coming Real Soon Now. signature.asc Description: This is a digitally signed message part -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On Thu, 2015-03-19 at 09:44 -0400, Stephen Gallagher wrote: On Sun, 2015-03-15 at 14:47 +0100, Michael Schwendt wrote: Can we please fix those people, who hate bodhi so much that they feel the need to submit updates with a low karma: 1 threshold? It just doesn't work and causes breakage too often. Broken dependencies and/or breakage at runtime. Keep in mind that it takes some time for packages to be picked up by the world-wide mirroring system. By the time such updates appear in the repositories, in bodhi they are marked stable already or are even on their way into the updates repo. That makes it impossible to test them in time and leave feedback. It's too late. Yeah, you want to rush out builds because you don't care. That sucks. I think it can be a judgement call on certain packages. For example, I maintain the Review Board packages which almost never get karma from more than one person (and that usually only for whichiver Fedora or EPEL branch that person is currently deploying to). Even at karma 1, most Review Board packages sit in updates-testing until the timeout passes. Now, this makes sense for Review Board because it's a leaf package and an application with a fairly limited audience. For packages in wider use or those that are dependencies for other projects, I think having a higher threshold makes sense. I agree that not all packages are the same; FWIW I wrote up some ideas about this in a blog post: http://dmalcolm.livejournal.com/5013.html (What variability exists within proposed updates to the Fedora package collection?) [that blog post is 5 years old now, where did the time go?] Hopefully, some of the new changes in Bodhi 2 will improve upon this situation. I hear that's coming Real Soon Now. Dave -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On Thu, 19 Mar 2015 09:44:27 -0400, Stephen Gallagher wrote: I think it can be a judgement call on certain packages. For example, I maintain the Review Board packages which almost never get karma from more than one person (and that usually only for whichiver Fedora or EPEL branch that person is currently deploying to). Always there is at least one example of an unhappy/impatient packager, who considers Bodhi unnecessary bureaucracy or who finds examples of updates that don't need any testing and could be unleashed much more quickly. [The repo metadata would change a hundred times a day, if packagers could push changes to the repo without any delay.] I only have a problem with those packagers who try to outsmart Bodhi (and its defaults (e.g. karma threshold 3) and shoot themselves into their feet with runtime breakage or dependency breakage. * It's just a rebuild of packages because of a SONAME bump in some lib! * I only cleaned up the spec file! * It's just a minor version update! * It's just a rename of package(s)! It has happened before. Broken rebuilds because of changes in the tool-chain (previous build several months old!), in the other build dependencies, because of regression (or even a brown paperbag bug) in the upgraded lib, or even because of expired buildroot overrides. Some packages sleep in the distribution for months, survive the development cycle and alpha/beta test releases of the distribution, but are replaced too quickly with updates/upgrades _without_ any adequate testing. Even at karma 1, most Review Board packages sit in updates-testing until the timeout passes. Which is a good thing. Waiting for the timeout is a good habit. At least it gives users a chance to test the update. But nobody gives feedback on my test updates! And obviously, for those packagers who don't mind the timeout, Bodhi could improve and offer an option to push an update to stable automatically after the timeout. Or after a customizable timeout, so e.g. the maintainer could request 21 days instead of 7 days. Meanwhile, a sufficient number of karma changes could still speed up the release or veto it, too. Hopefully, some of the new changes in Bodhi 2 will improve upon this situation. I hear that's coming Real Soon Now. It's the packagers, who should act more responsibly and accept any aid offered by Bodhi instead of trying to rush out updates. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
On 03/19/2015 04:14 PM, Michael Schwendt wrote: And obviously, for those packagers who don't mind the timeout, Bodhi could improve and offer an option to push an update to stable automatically after the timeout. Or after a customizable timeout, so e.g. the maintainer could request 21 days instead of 7 days. Meanwhile, a sufficient number of karma changes could still speed up the release or veto it, too. +1 The updates-testing email has plenty of proof that maintainers dump updates and forget about them. EOL comes around and bug fix / security fixes that were sitting in testing for 100+ days are left to die. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Silly bodhi karma games
Can we please fix those people, who hate bodhi so much that they feel the need to submit updates with a low karma: 1 threshold? It just doesn't work and causes breakage too often. Broken dependencies and/or breakage at runtime. Keep in mind that it takes some time for packages to be picked up by the world-wide mirroring system. By the time such updates appear in the repositories, in bodhi they are marked stable already or are even on their way into the updates repo. That makes it impossible to test them in time and leave feedback. It's too late. Yeah, you want to rush out builds because you don't care. That sucks. I advised for tightening the criteria not that long time ago: https://fedorahosted.org/fesco/ticket/1410 It was mainly related to the critical path set, but some adjustments could be reasonable even for all other packages. Unfortunately, it was rejected. Maybe next time. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
