Re: old_testing_critpath notifications

[Henrik Nordström]

tis 2010-12-07 klockan 19:20 -0500 skrev Doug Ledford:

> For non-boot devices, loopback works.  You only need the hardware if you
> are testing boot time capabilities (which, admittedly, is the far more
> important aspect of testing for this package).

And if you don't have spare systems with more than one drive to play
around with then kvm virtualization comes to the rescue when testing
pretty much any md or dm issue, possibly even including multipath.

Regards
Henrik

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Sat, 2010-12-11 at 00:01 +0100, Kevin Kofler wrote:

> Software just cannot grasp these things. Or do you volunteer for writing an 
> NLP processing system for Bodhi, and training all our testers to deal with 
> its limitations? Why can't we just let a human be the one to decide when to 
> hit the "Push to stable" button?

Would you please quit using every vaguely-related post as an excuse to
post the same argument? We know what your position is. You're not
helping any by repeating it over and over.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Kevin Kofler]

Bruno Wolff III wrote:
> I am concerned about that. If my karma is going to be treated differently
> because I become a proventester, I'd want to know what I am supposed to be
> doing differently and not mark something +1 by mistake. I think this
> concern goes away in the unicorn filled world where bodhi has descriptive
> feedback instead of numerical feedback.

Really DESCRIPTIVE feedback is just incompatible by design with automated 
enforcement. We have to go back to where the decision is made by a human 
with a brain (the maintainer) to really make use of precise feedback.

Software just cannot grasp these things. Or do you volunteer for writing an 
NLP processing system for Bodhi, and training all our testers to deal with 
its limitations? Why can't we just let a human be the one to decide when to 
hit the "Push to stable" button?

Kevin Kofler

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Doug Ledford]

On 12/03/2010 04:09 PM, Kevin Kofler wrote:
> Adam Williamson wrote:
>> We're working on this. It won't always be practical, however; in the
>> current case, for example, you need specific hardware to test mdadm.
> 
> Uh, this is md, not dm, you don't need very special HARDWARE (basically only 
> 2 HDDs, which do not even have to be identical, and you might even get away 
> with only 1 for testing, with absymal performance of course), you do need a 
> special setup though, which means repartitioning, and so isn't practical to 
> test on a production machine which isn't already set up that way.
> 
> Kevin Kofler
> 

For non-boot devices, loopback works.  You only need the hardware if you
are testing boot time capabilities (which, admittedly, is the far more
important aspect of testing for this package).

-- 
Doug Ledford 
  GPG KeyID: CFBFF194
  http://people.redhat.com/dledford

Infiniband specific RPMs available at
  http://people.redhat.com/dledford/Infiniband



signature.asc
Description: OpenPGP digital signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Luke Macken]

On Wed, Dec 01, 2010 at 02:02:48PM -0800, Adam Williamson wrote:
> On Wed, 2010-12-01 at 16:54 -0500, Luke Macken wrote:
> 
> > Yep, that happens.  There are also people that add +0 comments to
> > updates saying "Untested".  There is an obvious need for more
> > fine-grained karma types.
> 
> I've sent out notes to the test list to ask people not to do either of
> those things in future, I think the occurrence of them has gone down
> significantly since those emails went out. (I also updated the proven
> tester instructions page).
> 
> More fine-grained karma is still coming in Bodhi 2.0, right? My
> understanding was that we'd already agreed on the framework for it (in
> those threads on this list a few months back) and we were just waiting
> for the implementation now.

Yes, that's the plan.

I finally got the Bodhi v2.0 plans/ideas/status out of my brain and on to the 
wiki:

https://fedoraproject.org/wiki/Bodhi/2.0

I linked to a couple of pages regarding karma improvements, but please
update it with any more recent discussions regarding this change if you
know of any.  Also, more QA feature ideas are welcome.

luke
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Bruno Wolff III]

On Mon, Dec 06, 2010 at 08:57:42 -0800,
  Adam Williamson  wrote:
> 
> practically speaking that would change very little, because we're not
> blocked on getting moderator approval at present. Thankfully a lot of
> people are taking up the moderator duties, so anyone who actually
> applies to be a proventester usually gets a reply from a moderator
> almost immediately. The idea is to remove the active 'apply for the
> status' step from developers.

I am concerned about that. If my karma is going to be treated differently
because I become a proventester, I'd want to know what I am supposed to be
doing differently and not mark something +1 by mistake. I think this concern
goes away in the unicorn filled world where bodhi has descriptive feedback
instead of numerical feedback.
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Sun, 2010-12-05 at 09:41 -0600, Bruno Wolff III wrote:
> On Thu, Dec 02, 2010 at 11:48:13 -0800,
>   Adam Williamson  wrote:
> > 
> > I think it'd probably fit better in the preamble before step 1. Perhaps
> > after the paragraph 'As a Contributor, you should...' we add a paragraph
> > explaining that as a packager you will automatically be given
> > proventester privileges, a short explanation of the proventester
> > concept, and a link out to the proventester page requesting that you
> > read those instructions.
> 
> If we go down this route I'd rather see packagers have a way they can
> get proven tester status without needing mentor approval, but not just get
> it. I'd rather have a place where you read up on the expectations for
> proven testers and then click a button that says I'll do that.

practically speaking that would change very little, because we're not
blocked on getting moderator approval at present. Thankfully a lot of
people are taking up the moderator duties, so anyone who actually
applies to be a proventester usually gets a reply from a moderator
almost immediately. The idea is to remove the active 'apply for the
status' step from developers.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Bruno Wolff III]

On Thu, Dec 02, 2010 at 11:48:13 -0800,
  Adam Williamson  wrote:
> 
> I think it'd probably fit better in the preamble before step 1. Perhaps
> after the paragraph 'As a Contributor, you should...' we add a paragraph
> explaining that as a packager you will automatically be given
> proventester privileges, a short explanation of the proventester
> concept, and a link out to the proventester page requesting that you
> read those instructions.

If we go down this route I'd rather see packagers have a way they can
get proven tester status without needing mentor approval, but not just get
it. I'd rather have a place where you read up on the expectations for
proven testers and then click a button that says I'll do that.
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Fri, 2010-12-03 at 22:09 +0100, Kevin Kofler wrote:
> Adam Williamson wrote:
> > We're working on this. It won't always be practical, however; in the
> > current case, for example, you need specific hardware to test mdadm.
> 
> Uh, this is md, not dm, you don't need very special HARDWARE (basically only 
> 2 HDDs, which do not even have to be identical, and you might even get away 
> with only 1 for testing, with absymal performance of course), you do need a 
> special setup though, which means repartitioning, and so isn't practical to 
> test on a production machine which isn't already set up that way.

mdadm is also used to support Intel BIOS RAID, so you need an
appropriate motherboard to test that path.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Kevin Kofler]

Adam Williamson wrote:
> We're working on this. It won't always be practical, however; in the
> current case, for example, you need specific hardware to test mdadm.

Uh, this is md, not dm, you don't need very special HARDWARE (basically only 
2 HDDs, which do not even have to be identical, and you might even get away 
with only 1 for testing, with absymal performance of course), you do need a 
special setup though, which means repartitioning, and so isn't practical to 
test on a production machine which isn't already set up that way.

Kevin Kofler

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 14:22 -0800, Toshio Kuratomi wrote:

> > * Try and test in a reasonably user-ish environment, not your own highly
> > customized one; if this means using a separate user account or a VM, do
> >
> Note about this second bullet:  I'm not sure this is good advice.  There's
> been quite a few times I've encountered bugs in end-user oriented programs
> where deleting the config files in my home directory made the bug
> "disappear".  Similarly, I remember there was a bind update a few releases
> back where the package was trying to rewrite the existing config files which
> failed when the update was attempted on boxes that had already customized
> the config.
> 
> I see what you're trying to get at here but I think what it really boils
> down to is -- "you should have two sets of eyes look at this."  So perhaps,
> upping the karma requirement to +2 and letting maintainers +1 their own
> updates helps here.

That wasn't quite what I was getting at, but you have a point - both
environments can expose bugs. What I was getting at is that developers
tend to test in their own 'messy' environments anyway, so the thing they
usually *miss* is testing in a more user-y environment. So perhaps
advise maintainers to test both.

> 2) If the maintainer happens to be a proventester, then they only need to
> find a regular user to give the other karma point.

...or, as we're discussing in another thread branch, the maintainer
*will* be a proven tester because they're a maintainer :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Toshio Kuratomi]

On Thu, Dec 02, 2010 at 01:16:18PM -0800, Adam Williamson wrote:
> On Thu, 2010-12-02 at 12:30 -0800, Toshio Kuratomi wrote:
> > On Thu, Dec 02, 2010 at 11:25:03AM -0800, Adam Williamson wrote:
> > > On Thu, 2010-12-02 at 14:10 -0500, Doug Ledford wrote:
> > > > That being the case, I test the package fairly rigorously myself.  But
> > > > this process doesn't take that into account.  I test far more things
> > > > than you get with a few people just doing smoke tests, but the smoke
> > > > tests are actually the gating factor.  If you changed the process so a
> > > > maintainer can indicate they've done their own fairly extensive testing,
> > > > that would satisfy me.  But that also opens the door for abuse, so you
> > > > would have to watch maintainers once you enabled this ability.
> > > 
> > > I've posted in the thread earlier that I'd actually like to do this,
> > > others seem opposed however.
> > >
> > FWIW I'm for it with your explanation and added it to the Update
> > brainstorming page last night:
> > 
> > https://fedoraproject.org/wiki/User:Kevin/Updates_Policy_Changes_Ideas_Container
> 
> If we go this way, I'd propose adding additional guidelines to the
> proventester policy specific to maintainers testing their own packages.
> 
> * Test the actual build that will go out, from Koji - don't test your
> own local build of the same spec
> 
> * Try and test in a reasonably user-ish environment, not your own highly
> customized one; if this means using a separate user account or a VM, do
>
Note about this second bullet:  I'm not sure this is good advice.  There's
been quite a few times I've encountered bugs in end-user oriented programs
where deleting the config files in my home directory made the bug
"disappear".  Similarly, I remember there was a bind update a few releases
back where the package was trying to rewrite the existing config files which
failed when the update was attempted on boxes that had already customized
the config.

I see what you're trying to get at here but I think what it really boils
down to is -- "you should have two sets of eyes look at this."  So perhaps,
upping the karma requirement to +2 and letting maintainers +1 their own
updates helps here.

Note that that doesn't help the non-critpath packages get accepted faster
but it could help the critpath packages in two ways:

1) If you don't touch the requirements for critpath, it still just needs two
+1 and now the packager can give one of those themselves.

2) If the maintainer happens to be a proventester, then they only need to
find a regular user to give the other karma point.

-Toshoi


pgpYdLdTGnMij.pgp
Description: PGP signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 12:30 -0800, Toshio Kuratomi wrote:
> On Thu, Dec 02, 2010 at 11:25:03AM -0800, Adam Williamson wrote:
> > On Thu, 2010-12-02 at 14:10 -0500, Doug Ledford wrote:
> > > That being the case, I test the package fairly rigorously myself.  But
> > > this process doesn't take that into account.  I test far more things
> > > than you get with a few people just doing smoke tests, but the smoke
> > > tests are actually the gating factor.  If you changed the process so a
> > > maintainer can indicate they've done their own fairly extensive testing,
> > > that would satisfy me.  But that also opens the door for abuse, so you
> > > would have to watch maintainers once you enabled this ability.
> > 
> > I've posted in the thread earlier that I'd actually like to do this,
> > others seem opposed however.
> >
> FWIW I'm for it with your explanation and added it to the Update
> brainstorming page last night:
> 
> https://fedoraproject.org/wiki/User:Kevin/Updates_Policy_Changes_Ideas_Container

If we go this way, I'd propose adding additional guidelines to the
proventester policy specific to maintainers testing their own packages.

* Test the actual build that will go out, from Koji - don't test your
own local build of the same spec

* Try and test in a reasonably user-ish environment, not your own highly
customized one; if this means using a separate user account or a VM, do
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 15:43 -0500, Clyde E. Kunkel wrote:

> >> That being the case, I test the package fairly rigorously myself.  But
> >> this process doesn't take that into account.  I test far more things
> >> than you get with a few people just doing smoke tests, but the smoke
> >> tests are actually the gating factor.  If you changed the process so a
> >> maintainer can indicate they've done their own fairly extensive testing,
> >> that would satisfy me.  But that also opens the door for abuse, so you
> >> would have to watch maintainers once you enabled this ability.
> >
> > I've posted in the thread earlier that I'd actually like to do this,
> > others seem opposed however.
> 
> What about putting the maintainer's tests somewhere a tester can get 
> them and use on their own system(s)? Maybe even require critical path 
> packages to actually provide test cases.  This is not to say we don't 
> believe the maintainer, it is just a validation that the package passes 
> reasonable testing of the actual changes.

We're working on this. It won't always be practical, however; in the
current case, for example, you need specific hardware to test mdadm.

> I continue to be frustrated by not being able to always test the actual 
> change.  In most cases all I can do is make sure there are no 
> regressions and then that means all I have done is use the system with 
> the changed package installed and see if anything at all fails.

ensuring there's no regressions is actually more important than testing
whether the fix works. it's the main point of proven tester testing. if
we ship an update which doesn't fix the bug it tries to fix but doesn't
break anything else, we haven't *lost* anything. if we ship an update
that fixes the bug it's trying to fix but also breaks boot for half of
all users, we've lost a lot. :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Clyde E. Kunkel]

On 12/02/2010 02:25 PM, Adam Williamson wrote:
> On Thu, 2010-12-02 at 14:10 -0500, Doug Ledford wrote:
>
>> My package in question (mdadm) is only used in certain circumstances,
>> but if it isn't right, systems fail to boot.  I can certainly see why
>> something that can render a machine unbootable should be critpath.
>> However, because only a few people use it, testing is sparse at best.
>
> I think probably more people use it than current testing indicates, and
> we should do a better job of getting more people involved in testing.
>
>> I'll get one or two testers actually testing the package, but I won't
>> know until a release is made whether or not it truly works for the
>> masses because it isn't until then that I hit enough critical mass to know.
>>
>> That being the case, I test the package fairly rigorously myself.  But
>> this process doesn't take that into account.  I test far more things
>> than you get with a few people just doing smoke tests, but the smoke
>> tests are actually the gating factor.  If you changed the process so a
>> maintainer can indicate they've done their own fairly extensive testing,
>> that would satisfy me.  But that also opens the door for abuse, so you
>> would have to watch maintainers once you enabled this ability.
>
> I've posted in the thread earlier that I'd actually like to do this,
> others seem opposed however.

What about putting the maintainer's tests somewhere a tester can get 
them and use on their own system(s)? Maybe even require critical path 
packages to actually provide test cases.  This is not to say we don't 
believe the maintainer, it is just a validation that the package passes 
reasonable testing of the actual changes.

I continue to be frustrated by not being able to always test the actual 
change.  In most cases all I can do is make sure there are no 
regressions and then that means all I have done is use the system with 
the changed package installed and see if anything at all fails.

Regards,
OldFart
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Toshio Kuratomi]

On Thu, Dec 02, 2010 at 11:25:03AM -0800, Adam Williamson wrote:
> On Thu, 2010-12-02 at 14:10 -0500, Doug Ledford wrote:
> > That being the case, I test the package fairly rigorously myself.  But
> > this process doesn't take that into account.  I test far more things
> > than you get with a few people just doing smoke tests, but the smoke
> > tests are actually the gating factor.  If you changed the process so a
> > maintainer can indicate they've done their own fairly extensive testing,
> > that would satisfy me.  But that also opens the door for abuse, so you
> > would have to watch maintainers once you enabled this ability.
> 
> I've posted in the thread earlier that I'd actually like to do this,
> others seem opposed however.
>
FWIW I'm for it with your explanation and added it to the Update
brainstorming page last night:

https://fedoraproject.org/wiki/User:Kevin/Updates_Policy_Changes_Ideas_Container

-Toshio


pgphs9czXEt4X.pgp
Description: PGP signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 17:32 -0800, Toshio Kuratomi wrote:

> We don't have an automated process for showing people the rest of the wiki
> pages with packager information either.  If we added the information to this
> page:
> https://fedoraproject.org/wiki/Package_Review_Process
> 
> after step #9, 'Request updates for Fedora release branches, if necessary,
> using "fedpkg update" or another Bodhi interface as detailed in Bodhi
> Guide_.'
> 
> would that be sufficient?  It would be the same amount of information as the
> rest of the package process gets.
> 
> Note that the lack of a link to the proventester information on that page is
> probably another reason that people (new people at least) don't know that
> they should join proventester.

That doesn't feel like the right place to me; those steps are going
through a specific process, the package review process, and reading
proventester instructions isn't a part of that at all. It'd feel very
jarring, to me, to have it stuck into those steps.

I think it'd probably fit better in the preamble before step 1. Perhaps
after the paragraph 'As a Contributor, you should...' we add a paragraph
explaining that as a packager you will automatically be given
proventester privileges, a short explanation of the proventester
concept, and a link out to the proventester page requesting that you
read those instructions.

Also, what do we do with existing packagers? Send a mail out to
devel-announce announcing that everyone gets to be a proventester and
requesting everyone read the instructions? Future proventester
announcements to be CCed to devel-announce?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 14:10 -0500, Doug Ledford wrote:

> My package in question (mdadm) is only used in certain circumstances,
> but if it isn't right, systems fail to boot.  I can certainly see why
> something that can render a machine unbootable should be critpath.
> However, because only a few people use it, testing is sparse at best.

I think probably more people use it than current testing indicates, and
we should do a better job of getting more people involved in testing.

> I'll get one or two testers actually testing the package, but I won't
> know until a release is made whether or not it truly works for the
> masses because it isn't until then that I hit enough critical mass to know.
> 
> That being the case, I test the package fairly rigorously myself.  But
> this process doesn't take that into account.  I test far more things
> than you get with a few people just doing smoke tests, but the smoke
> tests are actually the gating factor.  If you changed the process so a
> maintainer can indicate they've done their own fairly extensive testing,
> that would satisfy me.  But that also opens the door for abuse, so you
> would have to watch maintainers once you enabled this ability.

I've posted in the thread earlier that I'd actually like to do this,
others seem opposed however.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Doug Ledford]

On 12/01/2010 05:17 PM, Adam Williamson wrote:
> On Wed, 2010-12-01 at 16:55 -0500, Doug Ledford wrote:
> 
>> The comparison is 100% fair because it points out the fundamental
>> problem with the current policy: if you don't have a paid staff of
>> testers to make sure testing is done in a timely fashion, then you have
>> absolutely no business gating updates on a testing staff that doesn't
>> exist.  It's nice in theory to think we can force testing of updates
>> prior to their release, but if the testing staff simply isn't there,
>> then you aren't improving the product, you're just stopping progress.
> 
> The gating is not on 'a testing staff'. The gating is on *testing*.
> 
> I want to say again that I'm not particularly wedded to the current
> policy and I don't mind at all if it changes, but I think we need to be
> careful of the mindset that says 'we can't enforce any standards in
> Fedora because it's a volunteer project so we must just accept what
> people are willing to give us'.

My package in question (mdadm) is only used in certain circumstances,
but if it isn't right, systems fail to boot.  I can certainly see why
something that can render a machine unbootable should be critpath.
However, because only a few people use it, testing is sparse at best.
I'll get one or two testers actually testing the package, but I won't
know until a release is made whether or not it truly works for the
masses because it isn't until then that I hit enough critical mass to know.

That being the case, I test the package fairly rigorously myself.  But
this process doesn't take that into account.  I test far more things
than you get with a few people just doing smoke tests, but the smoke
tests are actually the gating factor.  If you changed the process so a
maintainer can indicate they've done their own fairly extensive testing,
that would satisfy me.  But that also opens the door for abuse, so you
would have to watch maintainers once you enabled this ability.

> Even though packaging in Fedora is a volunteer process, we still have
> fairly rigorous packaging guidelines and a review process. We don't just
> accept any package someone turns up and submits. i.e., we're enforcing
> standards of quality, despite this being an entirely volunteer effort
> and no-one being compelled to show up and provide packages of a
> particular quality.

And packages *can* and *do* languish in the review process seemingly
interminably at times.

> The concept of having a policy requiring updates to be tested before
> they're issued is really no different.

I'm fine with this at a conceptual level, but the current policy takes
the update out of my hands no matter how much testing I do.  That I'm
not fine with.

-- 
Doug Ledford 
  GPG KeyID: CFBFF194
  http://people.redhat.com/dledford

Infiniband specific RPMs available at
  http://people.redhat.com/dledford/Infiniband



signature.asc
Description: OpenPGP digital signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 19:19 +0100, Till Maas wrote:

> A big difference is that the testing process is very fuzzy and there is
> not much tooling that helps people to test unknown software. E.g. if I
> want to review a package, there are several checklists I could use and
> there are guidelines that I can easily follow to perform a review. But
> testing software is not that easy.

That's one thing we'll be working on this cycle.

> Also it is not possible to partly test updates and share the effort.
> E.g. in reviews everyone can comment to get the package in shape if it
> is not.

Non-numeric Bodhi feedback will help with that. (Luke, in case you
didn't get the hint yet...we really need that non-numeric Bodhi
feedback :>)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Till Maas]

On Wed, Dec 01, 2010 at 02:17:32PM -0800, Adam Williamson wrote:
> On Wed, 2010-12-01 at 16:55 -0500, Doug Ledford wrote:
> 
> > The comparison is 100% fair because it points out the fundamental
> > problem with the current policy: if you don't have a paid staff of
> > testers to make sure testing is done in a timely fashion, then you have
> > absolutely no business gating updates on a testing staff that doesn't
> > exist.  It's nice in theory to think we can force testing of updates
> > prior to their release, but if the testing staff simply isn't there,
> > then you aren't improving the product, you're just stopping progress.
> 
> The gating is not on 'a testing staff'. The gating is on *testing*.
> 
> I want to say again that I'm not particularly wedded to the current
> policy and I don't mind at all if it changes, but I think we need to be
> careful of the mindset that says 'we can't enforce any standards in
> Fedora because it's a volunteer project so we must just accept what
> people are willing to give us'.
> 
> Even though packaging in Fedora is a volunteer process, we still have
> fairly rigorous packaging guidelines and a review process. We don't just
> accept any package someone turns up and submits. i.e., we're enforcing
> standards of quality, despite this being an entirely volunteer effort
> and no-one being compelled to show up and provide packages of a
> particular quality.

But this enforcement also only happens with respect to the currently
limited manpower, afaik not all merge reviews have been finished and
packages are usually not re-reviewed when packaging guidelines change.

> The concept of having a policy requiring updates to be tested before
> they're issued is really no different. I think one point where we've

A big difference is that the testing process is very fuzzy and there is
not much tooling that helps people to test unknown software. E.g. if I
want to review a package, there are several checklists I could use and
there are guidelines that I can easily follow to perform a review. But
testing software is not that easy.
Also it is not possible to partly test updates and share the effort.
E.g. in reviews everyone can comment to get the package in shape if it
is not.

Regards
Till


pgpnfgMkYVbds.pgp
Description: PGP signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[François Cami]

On Thu, Dec 2, 2010 at 6:31 PM, Adam Williamson  wrote:
> On Thu, 2010-12-02 at 18:20 +0100, François Cami wrote:
>> On Thu, Dec 2, 2010 at 6:03 PM, Adam Williamson  wrote:
>> > On Thu, 2010-12-02 at 13:20 +0100, François Cami wrote:
>> >
>> >> Of course, we could look at things differently: for a package to be
>> >> marked critpath, it should have users or be a dependency of some other
>> >> package with users.
>> >
>> > This is pretty inevitably implicit in the current definition of critpath
>> > - packages that are necessary to boot the system and use it. :) Okay,
>> > there's slightly unexpected cases like openldap, which isn't necessary
>> > for most people to login and use their systems but gets brought in
>> > because it's a dependency of various auth mechanisms which *optionally
>> > support* LDAP, but even that is obviously used by >0 people.
>>
>> jlaska just gave me the list of packages marked critpath in rawhide:
>> http://kojipkgs.fedoraproject.org/mash/rawhide-20101202/logs/critpath.txt
>> 389-ds, cobbler, httpd, libvirt, mysql, postgresql, puppet, vsftpd are
>> not in the list. My guess is therefore that most server packages are
>> completely ignored by the critpath definition. And we have server
>> users.
>
> Ah. I misread you: I thought you meant to add that to the current
> definition of critpath packages, not to replace the current critpath
> definition with simply "anything with users".

Make that "anything with enough users for us to care", especially if
said users participate in the process, and that's about it.

>> >> And packages with enough known users should always land in critpath,
>> >> otherwise we might break systems users depend on.
>> >
>> > That doesn't fit in with the current function-based definition, so your
>> > proposal is to change that?
>>
>> Yes. Note that the current function-based definition is contained in
>> the "have users"-based one, as long as Fedora is used on the desktop,
>> that is.
>
> Right, but it massively increases the range of critpath packages (which
> would only exacerbate the problem under discussion unless we made
> critpath testing less rigorous), and loses the initial purpose of the
> critpath policy. I think really what you want is the three-tier system
> so that 'not important' packages can be allowed to go through without
> testing, right?

That, and catch regressions in updates to well-used packages before
they're pushed. And that includes server packages.

>> > but we don't really have any very reliable methods for
>> > determining use of packages yet.
>>
>> We could extend smolt to do so.
>
> smolt's still opt-in and always will be, AFAIK, because there'd be way
> too much of a Slashdot drama if it weren't.

Yes it is and rightly so. However, it contains data about 200K active
systems at the moment:
http://smolts.org/static/stats/stats.html
Data about CPU, RAM, Language, etc... But no data about packages yet.

François
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 18:20 +0100, François Cami wrote:
> On Thu, Dec 2, 2010 at 6:03 PM, Adam Williamson  wrote:
> > On Thu, 2010-12-02 at 13:20 +0100, François Cami wrote:
> >
> >> Of course, we could look at things differently: for a package to be
> >> marked critpath, it should have users or be a dependency of some other
> >> package with users.
> >
> > This is pretty inevitably implicit in the current definition of critpath
> > - packages that are necessary to boot the system and use it. :) Okay,
> > there's slightly unexpected cases like openldap, which isn't necessary
> > for most people to login and use their systems but gets brought in
> > because it's a dependency of various auth mechanisms which *optionally
> > support* LDAP, but even that is obviously used by >0 people.
> 
> jlaska just gave me the list of packages marked critpath in rawhide:
> http://kojipkgs.fedoraproject.org/mash/rawhide-20101202/logs/critpath.txt
> 389-ds, cobbler, httpd, libvirt, mysql, postgresql, puppet, vsftpd are
> not in the list. My guess is therefore that most server packages are
> completely ignored by the critpath definition. And we have server
> users.

Ah. I misread you: I thought you meant to add that to the current
definition of critpath packages, not to replace the current critpath
definition with simply "anything with users".

> >> And packages with enough known users should always land in critpath,
> >> otherwise we might break systems users depend on.
> >
> > That doesn't fit in with the current function-based definition, so your
> > proposal is to change that?
> 
> Yes. Note that the current function-based definition is contained in
> the "have users"-based one, as long as Fedora is used on the desktop,
> that is.

Right, but it massively increases the range of critpath packages (which
would only exacerbate the problem under discussion unless we made
critpath testing less rigorous), and loses the initial purpose of the
critpath policy. I think really what you want is the three-tier system
so that 'not important' packages can be allowed to go through without
testing, right?

> > but we don't really have any very reliable methods for
> > determining use of packages yet.
> 
> We could extend smolt to do so.

smolt's still opt-in and always will be, AFAIK, because there'd be way
too much of a Slashdot drama if it weren't.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[François Cami]

On Thu, Dec 2, 2010 at 6:03 PM, Adam Williamson  wrote:
> On Thu, 2010-12-02 at 13:20 +0100, François Cami wrote:
>
>> Of course, we could look at things differently: for a package to be
>> marked critpath, it should have users or be a dependency of some other
>> package with users.
>
> This is pretty inevitably implicit in the current definition of critpath
> - packages that are necessary to boot the system and use it. :) Okay,
> there's slightly unexpected cases like openldap, which isn't necessary
> for most people to login and use their systems but gets brought in
> because it's a dependency of various auth mechanisms which *optionally
> support* LDAP, but even that is obviously used by >0 people.

jlaska just gave me the list of packages marked critpath in rawhide:
http://kojipkgs.fedoraproject.org/mash/rawhide-20101202/logs/critpath.txt
389-ds, cobbler, httpd, libvirt, mysql, postgresql, puppet, vsftpd are
not in the list. My guess is therefore that most server packages are
completely ignored by the critpath definition. And we have server
users.

>> And packages with enough known users should always land in critpath,
>> otherwise we might break systems users depend on.
>
> That doesn't fit in with the current function-based definition, so your
> proposal is to change that?

Yes. Note that the current function-based definition is contained in
the "have users"-based one, as long as Fedora is used on the desktop,
that is.

>> At this point, non-critpath packages may be left to their maintainers' 
>> wishes.
>
> maybe we could have a three-tier system - critpath, commonly used,
> other.

Works for me, as long as "commonly" used is at the very least
smoke-tested, but I guess that was your intention since there is an
"other" set :)

> but we don't really have any very reliable methods for
> determining use of packages yet.

We could extend smolt to do so.

François
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 13:20 +0100, François Cami wrote:

> Of course, we could look at things differently: for a package to be
> marked critpath, it should have users or be a dependency of some other
> package with users.

This is pretty inevitably implicit in the current definition of critpath
- packages that are necessary to boot the system and use it. :) Okay,
there's slightly unexpected cases like openldap, which isn't necessary
for most people to login and use their systems but gets brought in
because it's a dependency of various auth mechanisms which *optionally
support* LDAP, but even that is obviously used by >0 people.

> And packages with enough known users should always land in critpath,
> otherwise we might break systems users depend on.

That doesn't fit in with the current function-based definition, so your
proposal is to change that?

> At this point, non-critpath packages may be left to their maintainers' wishes.

maybe we could have a three-tier system - critpath, commonly used,
other. but we don't really have any very reliable methods for
determining use of packages yet.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[François Cami]

On Thu, Dec 2, 2010 at 1:32 AM, Matt McCutchen  wrote:
> On Wed, 2010-12-01 at 14:17 -0800, Adam Williamson wrote:
>>
>> When software is packaged it's reasonable to expect that someone,
>> somewhere, uses it; if they don't, it probably shouldn't be packaged. We
>> need to find those people and engage them in the testing process, and it
>> seems to me that the maintainers of packages are as well placed as
>> anyone to help find and engage their users in this process.
>
> The argument sounds reasonable on its face, but apparently some (many?)
> maintainers don't agree. Pestering them does not seem to be helpful.
> We have the option to take away their ownership of the package,
> potentially resulting in it getting dropped from the distribution if
> another maintainer can't be found, but again we have to ask whether the
> benefits outweigh the costs.

I don't agree with the "packages all should have current users" posit.
Volunteers tend to "scratch their own itches". So a maintainer may
very well be at some point in time one of the very few, if not the
only, user of a package. In this case, I think removing the package
forcibly from Fedora would be harmful because it could possibly
discourage contributors and damage our capabilities to "bring"
upstream to users.

Of course, we could look at things differently: for a package to be
marked critpath, it should have users or be a dependency of some other
package with users.
And packages with enough known users should always land in critpath,
otherwise we might break systems users depend on.

At this point, non-critpath packages may be left to their maintainers' wishes.

François
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[François Cami]

On Wed, Dec 1, 2010 at 10:34 PM, Luke Macken  wrote:
> On Mon, Nov 29, 2010 at 01:36:18PM +, Petr Pisar wrote:
>> On 2010-11-29, Peter Robinson  wrote:
>> > On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
>> >
>> > Proven testers do get copies of these emails (dozens of them) and its
>> > also summarised in the updates-testing report for all to see.
>> >
>> Oh, I thought  described as `For testers of Fedora
>> development releases' concerns alpha and similar (pre-)releases only.
>>
>> In that case, only one issue remains: to stop spamming package
>> maintainer.
>
> The next release of bodhi that I'm going to push out to our releng
> servers this week will stop spamming proventesters and maintainers about
> these, since this data is now in the updates-testing digests that get
> sent out to the test-list with each push.

Thank you Luke.

François
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Petr Pisar]

On 2010-12-01, Luke Macken  wrote:
> On Mon, Nov 29, 2010 at 01:36:18PM +, Petr Pisar wrote:
>> On 2010-11-29, Peter Robinson  wrote:
>> > On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
>> >
>> > Proven testers do get copies of these emails (dozens of them) and its
>> > also summarised in the updates-testing report for all to see.
>> >
>> Oh, I thought  described as `For testers of Fedora
>> development releases' concerns alpha and similar (pre-)releases only.
>> 
>> In that case, only one issue remains: to stop spamming package
>> maintainer.
>
> The next release of bodhi that I'm going to push out to our releng
> servers this week will stop spamming proventesters and maintainers about
> these, since this data is now in the updates-testing digests that get
> sent out to the test-list with each push.
>
Thank you.

-- Petr

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Toshio Kuratomi]

On Wed, Dec 01, 2010 at 03:59:02PM -0800, Adam Williamson wrote:
> On Wed, 2010-12-01 at 15:53 -0800, Toshio Kuratomi wrote:
> 
> > > I don't really see any reason why *everyone* who's a packager shouldn't
> > > also have signed up to be a proven tester by now. I'd like to ask if
> > > anyone has a perception that it's a hard process to get involved in, or
> > > if they got the impression that they *shouldn't* get engaged in it, or
> > > something like that. Maybe we can improve the presentation to make it
> > > clear that this really ought to be a very wide-based process.
> > >
> > With that in mind,  perhaps we should have being added to the packager group
> > automatically put you in the proventester group.  If you turn out to be
> > a problem we can then remove you from the proventester group until you've
> > learned how you should be testing.  (On the implementation side, we should
> > have this ability in fas since we do something similar to put people who
> > sign the cla into the cla_done group automatically).
> 
> I'm not sure I'd want to go quite that far unless the sign-up process
> can wave the proven testers instructions in your face quite prominently.
> They're short and easy to read and understand, but you can't infer them
> from first principles: we do want to have people read the proven tester
> instructions before becoming proven testers. That's actually the *only*
> requirement to become a proven tester. :)
> 
We don't have an automated process for showing people the rest of the wiki
pages with packager information either.  If we added the information to this
page:
https://fedoraproject.org/wiki/Package_Review_Process

after step #9, 'Request updates for Fedora release branches, if necessary,
using "fedpkg update" or another Bodhi interface as detailed in Bodhi
Guide_.'

would that be sufficient?  It would be the same amount of information as the
rest of the package process gets.

Note that the lack of a link to the proventester information on that page is
probably another reason that people (new people at least) don't know that
they should join proventester.

-Toshio


pgp6PmNZX5aES.pgp
Description: PGP signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Matt McCutchen]

On Wed, 2010-12-01 at 14:17 -0800, Adam Williamson wrote:
> [...] I think we need to be
> careful of the mindset that says 'we can't enforce any standards in
> Fedora because it's a volunteer project so we must just accept what
> people are willing to give us'.
> 
> Even though packaging in Fedora is a volunteer process, we still have
> fairly rigorous packaging guidelines and a review process. We don't just
> accept any package someone turns up and submits. i.e., we're enforcing
> standards of quality, despite this being an entirely volunteer effort
> and no-one being compelled to show up and provide packages of a
> particular quality.
> 
> The concept of having a policy requiring updates to be tested before
> they're issued is really no different.

Correct in the sense that Fedora has the authority to impose both
policies.  The costs and benefits of each policy are still open for
debate.

> I do think that for update testing to work well going forward we need to
> engage more groups with it and make it clear it's not something that
> some separate QA group is just going to do for everyone and no-one has
> to worry about it.

That's easy to say.  The fact remains that if no one feels they have a
sufficient incentive to do the work of testing an update for a
particular Fedora version, it simply won't get tested.

> When software is packaged it's reasonable to expect that someone,
> somewhere, uses it; if they don't, it probably shouldn't be packaged. We
> need to find those people and engage them in the testing process, and it
> seems to me that the maintainers of packages are as well placed as
> anyone to help find and engage their users in this process.

The argument sounds reasonable on its face, but apparently some (many?)
maintainers don't agree.  Pestering them does not seem to be helpful.
We have the option to take away their ownership of the package,
potentially resulting in it getting dropped from the distribution if
another maintainer can't be found, but again we have to ask whether the
benefits outweigh the costs.

-- 
Matt

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 15:53 -0800, Toshio Kuratomi wrote:

> > I don't really see any reason why *everyone* who's a packager shouldn't
> > also have signed up to be a proven tester by now. I'd like to ask if
> > anyone has a perception that it's a hard process to get involved in, or
> > if they got the impression that they *shouldn't* get engaged in it, or
> > something like that. Maybe we can improve the presentation to make it
> > clear that this really ought to be a very wide-based process.
> >
> With that in mind,  perhaps we should have being added to the packager group
> automatically put you in the proventester group.  If you turn out to be
> a problem we can then remove you from the proventester group until you've
> learned how you should be testing.  (On the implementation side, we should
> have this ability in fas since we do something similar to put people who
> sign the cla into the cla_done group automatically).

I'm not sure I'd want to go quite that far unless the sign-up process
can wave the proven testers instructions in your face quite prominently.
They're short and easy to read and understand, but you can't infer them
from first principles: we do want to have people read the proven tester
instructions before becoming proven testers. That's actually the *only*
requirement to become a proven tester. :)

> And in answer to your question -- my perception is that it's a separate
> thing that I could join just as I've joined infrastructure as well as
> packaging.  So in the sense that it's not something that's automatically
> there for me to do by virtue of being in packager, it is hard to get
> involved in.

Thanks.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Toshio Kuratomi]

On Wed, Dec 01, 2010 at 02:17:32PM -0800, Adam Williamson wrote:
> 
> The concept of having a policy requiring updates to be tested before
> they're issued is really no different. I think one point where we've
> fallen over is that it wasn't sufficiently well discussed / communicated
> in advance that this testing wasn't just going to 'get done' by some
> independent group and no-one else would have to worry about it, but
> would require a lot of people to chip in. In the same way that there
> isn't some separate independent group that does package reviews, it's
> just all maintainers chipping in when they can. I think perhaps those
> who supported and voted for the policy kind of assumed this would
> happen, and many others weren't actually aware of it.
> 
I think this is the heart of the matter.  Communication and buy-in.

The difference between package reviews/guidelines and testing is a matter of
history -- package reviews are an expectation of maintainer responsibility
from fedora.us days.  Recruitment of testers is an increase in expectations
of maintainers that's happened in the last year.  Without buy-in from
maintainers that they want to do this, you don't get maintainers actually
working on testing packages.  Actually, thinking back to fedora.us, testing
of each update was actually done in fedora.us and abandoned for lack of
manpower.  However the way we tested was a lot different -- each update went
through a new package review, not just a build being installed and rated as
to whether or not it worked.

The comparison to package reviews is also interesting in several ways.

There is an ad hoc group of a few package maintainers that do most of the
reviews.  So this is similar to what you're currently seeing with the
testers.  An ad hoc group of a (relatively) few package maintainers is
testing updates.while the majority of packagers do not participate.

The queue of packages often seem larger than the available manpower to
review packages.

Recently the queue of packages has been going down.  I attribute a large
part of this to tibbs's efforts where he's done a couple things:
* Closing out old reviews where the package submitter no longer responds to
  the review request.
* Actively seeking to put together domain-specific reviewers with packages
  that fit those domains (hooking up active python-sig members with python
  package reviews for instance).

One encouraged and somewhat popular method of getting packages reviewed is
to trade reviews with other packagers.  we don't have that recommendation
for testing at the moment.

> I do think that for update testing to work well going forward we need to
> engage more groups with it and make it clear it's not something that
> some separate QA group is just going to do for everyone and no-one has
> to worry about it. We can get, and already have got, some enthusiastic
> people to sign up to run updates-testing and provide testing feedback
> for the packages they use anyway, but the concept of there being a
> hardcore group of dedicated testers who will go out of their way to
> install, configure and test software they wouldn't usually use is not
> one that's likely to fly, I don't think.
> 
> When software is packaged it's reasonable to expect that someone,
> somewhere, uses it; if they don't, it probably shouldn't be packaged. We
> need to find those people and engage them in the testing process, and it
> seems to me that the maintainers of packages are as well placed as
> anyone to help find and engage their users in this process.
> 
Allowing anonymous karma to count is something that I think targets this.

> In many cases it's easier than that; a lot of packages are maintained by
> more than one person. It's not only perfectly okay but more or less
> *what we want to happen* for co-maintainers to sign up as proven testers
> and test each others' updates. There's a bunch of people in the anaconda
> group, for instance; it's perfectly fine for you all to sign up as
> proven testers and test each other's code. The testing doesn't have to
> come from some impartial outside body, all we need is a sanity check.
> 
> I don't really see any reason why *everyone* who's a packager shouldn't
> also have signed up to be a proven tester by now. I'd like to ask if
> anyone has a perception that it's a hard process to get involved in, or
> if they got the impression that they *shouldn't* get engaged in it, or
> something like that. Maybe we can improve the presentation to make it
> clear that this really ought to be a very wide-based process.
>
With that in mind,  perhaps we should have being added to the packager group
automatically put you in the proventester group.  If you turn out to be
a problem we can then remove you from the proventester group until you've
learned how you should be testing.  (On the implementation side, we should
have this ability in fas since we do something similar to put people who
sign the cla into the cla_done group automatically).

An

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 16:15 -0700, Nathanael D. Noblet wrote:

> > fedora-easy-karma makes it very, very easy. Have you tried it? You just
> > run it, at a console, and it detects all the packages you have installed
> > from updates-testing, gives you the description of each, and asks you to
> > provide feedback for each (or skip). It's really a one-stop. It's
> > described in the proven tester documentation, but really all you need to
> > know is 'yum install fedora-easy-karma', 'fedora-easy-karma'. It has a
> > --critpath-only parameter to show only critpath updates, if you're in a
> > hurry and just want to provide feedback on the most important updates.
> 
> Well I knew of fedora-easy-karma because of that thread, however I 
> didn't know it did everything I wanted. I thought it allowed you to 
> easily add karma to an update you've installed... Again, even though 
> fedora-easy-karma can give me a list of packages to install and test, 
> this requires me to think 'hey should I run fedora-easy-karma' to see if 
> any packages I'm interested in are available for testing?.. As opposed 
> to the case where updates are made available via PK and I just get 
> notified they are there. So if I can create a list of packages I'm 
> interested in testing, or a list of packages to exclude. And it checks 
> like PK does and informs me when packages need testing, great. Otherwise 
> as lame as it is, I'll rarely think 'I should see if something needs 
> testing'. Unless something I *need* needs testing and I know because 
> I've filed the bug or am otherwise cc'd on it.

f-e-k doesn't install updates, it works with what's already installed.
the intended workflow is that you have updates-testing enabled and just
install everything that becomes available through it (so just run
regular updates with it enabled); then you run f-e-k regularly and it
figures out which of the currently installed packages come from
updates-testing and hence require karma, and asks you to provide karma
for each one in turn.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Nathanael D. Noblet]

On 12/01/2010 04:06 PM, Adam Williamson wrote:
> On Wed, 2010-12-01 at 15:53 -0700, Nathanael D. Noblet wrote:
>> On 12/01/2010 03:17 PM, Adam Williamson wrote:
>>> I don't really see any reason why *everyone* who's a packager shouldn't
>>> also have signed up to be a proven tester by now. I'd like to ask if
>>> anyone has a perception that it's a hard process to get involved in, or
>>> if they got the impression that they *shouldn't* get engaged in it, or
>>> something like that. Maybe we can improve the presentation to make it
>>> clear that this really ought to be a very wide-based process.
>>
>> Well I never read anything specifically about the requirements, however
>> based of the name alone, 'proven tester' and relating it to 'proven
>> packager' I assumed I'd need to be more experienced before I signed up.
>
> Ah. I'm not sure if we can change the name...what might be 'friendlier'?
> It's certainly not intended to be as 3l33t as provenpackager.

I'm not sure it matters at this point. Though that is why I figured it 
had a number of hoops to jump through to get the badge... My main 
contact with Fedora is through the ML. I hate wikis and so rarely visit 
it, however perhaps there or on the fedoraproject.org site where there 
are links for getting involved it can be made clearer... For me its 
entirely the name that made me think I needed some advanced experience. 
I mean it says proven... I need to prove something. Something more like 
dependable, regular, approved... or other tester would have jived with 
me more I think.

>> Also, I don't find the tools for updates-testing particularly friendly
>> enough yet. I wrote in a thread awhile ago what I thought could be very
>> useful to entice people to use updates-testing. I know there are some
>> tools which together would allow me to do this, however I'm looking for
>> a very simple, comprehensive tool that shows me what is in updates
>> testing, what I've installed from there, or create a list of packages
>> I'm interested in in updates-testing and never show me otherwise etc...
>
> fedora-easy-karma makes it very, very easy. Have you tried it? You just
> run it, at a console, and it detects all the packages you have installed
> from updates-testing, gives you the description of each, and asks you to
> provide feedback for each (or skip). It's really a one-stop. It's
> described in the proven tester documentation, but really all you need to
> know is 'yum install fedora-easy-karma', 'fedora-easy-karma'. It has a
> --critpath-only parameter to show only critpath updates, if you're in a
> hurry and just want to provide feedback on the most important updates.

Well I knew of fedora-easy-karma because of that thread, however I 
didn't know it did everything I wanted. I thought it allowed you to 
easily add karma to an update you've installed... Again, even though 
fedora-easy-karma can give me a list of packages to install and test, 
this requires me to think 'hey should I run fedora-easy-karma' to see if 
any packages I'm interested in are available for testing?.. As opposed 
to the case where updates are made available via PK and I just get 
notified they are there. So if I can create a list of packages I'm 
interested in testing, or a list of packages to exclude. And it checks 
like PK does and informs me when packages need testing, great. Otherwise 
as lame as it is, I'll rarely think 'I should see if something needs 
testing'. Unless something I *need* needs testing and I know because 
I've filed the bug or am otherwise cc'd on it.
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 15:53 -0700, Nathanael D. Noblet wrote:
> On 12/01/2010 03:17 PM, Adam Williamson wrote:
> > I don't really see any reason why *everyone* who's a packager shouldn't
> > also have signed up to be a proven tester by now. I'd like to ask if
> > anyone has a perception that it's a hard process to get involved in, or
> > if they got the impression that they *shouldn't* get engaged in it, or
> > something like that. Maybe we can improve the presentation to make it
> > clear that this really ought to be a very wide-based process.
> 
> Well I never read anything specifically about the requirements, however 
> based of the name alone, 'proven tester' and relating it to 'proven 
> packager' I assumed I'd need to be more experienced before I signed up.

Ah. I'm not sure if we can change the name...what might be 'friendlier'?
It's certainly not intended to be as 3l33t as provenpackager.

> Also, I don't find the tools for updates-testing particularly friendly 
> enough yet. I wrote in a thread awhile ago what I thought could be very 
> useful to entice people to use updates-testing. I know there are some 
> tools which together would allow me to do this, however I'm looking for 
> a very simple, comprehensive tool that shows me what is in updates 
> testing, what I've installed from there, or create a list of packages 
> I'm interested in in updates-testing and never show me otherwise etc...

fedora-easy-karma makes it very, very easy. Have you tried it? You just
run it, at a console, and it detects all the packages you have installed
from updates-testing, gives you the description of each, and asks you to
provide feedback for each (or skip). It's really a one-stop. It's
described in the proven tester documentation, but really all you need to
know is 'yum install fedora-easy-karma', 'fedora-easy-karma'. It has a
--critpath-only parameter to show only critpath updates, if you're in a
hurry and just want to provide feedback on the most important updates.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Nathanael D. Noblet]

On 12/01/2010 03:17 PM, Adam Williamson wrote:
> I don't really see any reason why *everyone* who's a packager shouldn't
> also have signed up to be a proven tester by now. I'd like to ask if
> anyone has a perception that it's a hard process to get involved in, or
> if they got the impression that they *shouldn't* get engaged in it, or
> something like that. Maybe we can improve the presentation to make it
> clear that this really ought to be a very wide-based process.

Well I never read anything specifically about the requirements, however 
based of the name alone, 'proven tester' and relating it to 'proven 
packager' I assumed I'd need to be more experienced before I signed up.

Also, I don't find the tools for updates-testing particularly friendly 
enough yet. I wrote in a thread awhile ago what I thought could be very 
useful to entice people to use updates-testing. I know there are some 
tools which together would allow me to do this, however I'm looking for 
a very simple, comprehensive tool that shows me what is in updates 
testing, what I've installed from there, or create a list of packages 
I'm interested in in updates-testing and never show me otherwise etc...

I'm not saying I won't test without it - however I wouldn't be able to 
give dedicated time. I test updates I push out, I test updates for bugs 
I've filed. Otherwise I don't have time to look at anything else. I 
include the thread as reference only. It seemed someone in that thread 
did say they were looking at building just such a tool... Anyway. I want 
to help out more, but I'm so busy with $realjob that I need a very easy 
way to see updates I'm willing to test, pull them back if something goes 
wrong, and submit karma quickly from one simple place...

http://lists.fedoraproject.org/pipermail/devel/2010-June/138077.html
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 16:55 -0500, Doug Ledford wrote:

> The comparison is 100% fair because it points out the fundamental
> problem with the current policy: if you don't have a paid staff of
> testers to make sure testing is done in a timely fashion, then you have
> absolutely no business gating updates on a testing staff that doesn't
> exist.  It's nice in theory to think we can force testing of updates
> prior to their release, but if the testing staff simply isn't there,
> then you aren't improving the product, you're just stopping progress.

The gating is not on 'a testing staff'. The gating is on *testing*.

I want to say again that I'm not particularly wedded to the current
policy and I don't mind at all if it changes, but I think we need to be
careful of the mindset that says 'we can't enforce any standards in
Fedora because it's a volunteer project so we must just accept what
people are willing to give us'.

Even though packaging in Fedora is a volunteer process, we still have
fairly rigorous packaging guidelines and a review process. We don't just
accept any package someone turns up and submits. i.e., we're enforcing
standards of quality, despite this being an entirely volunteer effort
and no-one being compelled to show up and provide packages of a
particular quality.

The concept of having a policy requiring updates to be tested before
they're issued is really no different. I think one point where we've
fallen over is that it wasn't sufficiently well discussed / communicated
in advance that this testing wasn't just going to 'get done' by some
independent group and no-one else would have to worry about it, but
would require a lot of people to chip in. In the same way that there
isn't some separate independent group that does package reviews, it's
just all maintainers chipping in when they can. I think perhaps those
who supported and voted for the policy kind of assumed this would
happen, and many others weren't actually aware of it.

I do think that for update testing to work well going forward we need to
engage more groups with it and make it clear it's not something that
some separate QA group is just going to do for everyone and no-one has
to worry about it. We can get, and already have got, some enthusiastic
people to sign up to run updates-testing and provide testing feedback
for the packages they use anyway, but the concept of there being a
hardcore group of dedicated testers who will go out of their way to
install, configure and test software they wouldn't usually use is not
one that's likely to fly, I don't think.

When software is packaged it's reasonable to expect that someone,
somewhere, uses it; if they don't, it probably shouldn't be packaged. We
need to find those people and engage them in the testing process, and it
seems to me that the maintainers of packages are as well placed as
anyone to help find and engage their users in this process.

In many cases it's easier than that; a lot of packages are maintained by
more than one person. It's not only perfectly okay but more or less
*what we want to happen* for co-maintainers to sign up as proven testers
and test each others' updates. There's a bunch of people in the anaconda
group, for instance; it's perfectly fine for you all to sign up as
proven testers and test each other's code. The testing doesn't have to
come from some impartial outside body, all we need is a sanity check.

I don't really see any reason why *everyone* who's a packager shouldn't
also have signed up to be a proven tester by now. I'd like to ask if
anyone has a perception that it's a hard process to get involved in, or
if they got the impression that they *shouldn't* get engaged in it, or
something like that. Maybe we can improve the presentation to make it
clear that this really ought to be a very wide-based process.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 16:54 -0500, Luke Macken wrote:

> Yep, that happens.  There are also people that add +0 comments to
> updates saying "Untested".  There is an obvious need for more
> fine-grained karma types.

I've sent out notes to the test list to ask people not to do either of
those things in future, I think the occurrence of them has gone down
significantly since those emails went out. (I also updated the proven
tester instructions page).

More fine-grained karma is still coming in Bodhi 2.0, right? My
understanding was that we'd already agreed on the framework for it (in
those threads on this list a few months back) and we were just waiting
for the implementation now.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Doug Ledford]

On 12/01/2010 04:35 PM, Adam Williamson wrote:
> On Wed, 2010-12-01 at 16:22 -0500, Doug Ledford wrote:
> 
>> If the ticket can be allowed to languish that long, then I don't feel in
>> the least bit guilty that I didn't drop my other Red Hat
>> responsibilities on the floor when the ticket was finally approved.  By
>> the time it was approved, I had already moved on.
> 
> I didn't say you should. I said your assertion that F14 went out with an
> older mdadm '*purely* because of this policy' was not supported by
> evidence.

It *was* purely because of this policy.  60 days after I file a bodhi
ticket, I'm done working on whatever I was working on at the time.  Even
if I had seen the approval notification, I wouldn't have pushed it with
only 4 days until freeze.  Not when I've completely forgotten what was
in the package.

> The package could have been included in F14 had it been pushed
> soon after critpath testing approval was granted. Therefore, it's simply
> not correct that the policy alone prevented it being included. (If you'd
> submitted the update to stable close to the final date it wouldn't have
> taken five days to be pushed; rel-eng do pushes far more often when it's
> close to a deadline).
> 
>>   Fortunately, I wasn't
>> under the same restrictions on Red Hat Enterprise Linux 6, so it went GA
>> with a much better, fixed mdadm relative to Fedora.
> 
> The implied comparison here is unfair and hence not helpful; RHEL has a
> large group of paid test staff. Fedora does not.

The comparison is 100% fair because it points out the fundamental
problem with the current policy: if you don't have a paid staff of
testers to make sure testing is done in a timely fashion, then you have
absolutely no business gating updates on a testing staff that doesn't
exist.  It's nice in theory to think we can force testing of updates
prior to their release, but if the testing staff simply isn't there,
then you aren't improving the product, you're just stopping progress.

-- 
Doug Ledford 
  GPG KeyID: CFBFF194
  http://people.redhat.com/dledford

Infiniband specific RPMs available at
  http://people.redhat.com/dledford/Infiniband



signature.asc
Description: OpenPGP digital signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Luke Macken]

On Wed, Dec 01, 2010 at 04:49:07PM -0500, Doug Ledford wrote:
> On 12/01/2010 04:40 PM, Luke Macken wrote:
> > On Wed, Dec 01, 2010 at 10:41:20AM -0800, Adam Williamson wrote:
> >> On Wed, 2010-12-01 at 13:23 -0500, Doug Ledford wrote:
> >>
> >>> That being said, F14 went out with a broken mdadm *purely* because of
> >>> this policy.
> >>
> >>> Evidently my update was approved somewhere along the way, but because of
> >>> the volume of bodhi spam I get, I missed it.
> >>
> >> ...so what you're saying is that F14 did not in fact go out with a
> >> broken mdadm *purely* because of the policy, but in a small part because
> >> of the policy and in a large part because you don't read / filter your
> >> emails carefully enough.
> >>
> >>>   So I'm not sure if it
> >>> could have made F14 final or not, but I know it didn't because I was
> >>> working on other things at the time.
> >>
> >> bodhi - 2010-10-14 22:36:08
> >> Critical path update approved 
> >>
> >> The final change deadline was 10-18; you had four days to push the
> >> update.
> > 
> > Also, if karma automatism was enabled for that update, it would have been
> > queued for pushing right when it was approved.
> > 
> > luke
> 
> I don't enable karma automatism because in the past I've seen people
> report testing karma +1 when they did not, in fact, doing anything
> useful in terms of testing (aka, they had no software raid devices, yet
> they said the system still works...well, duh, it's only used on software
> raid devices so if you don't have any, then it doesn't make any
> difference to you).

Yep, that happens.  There are also people that add +0 comments to
updates saying "Untested".  There is an obvious need for more
fine-grained karma types.

luke
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Doug Ledford]

On 12/01/2010 04:40 PM, Luke Macken wrote:
> On Wed, Dec 01, 2010 at 10:41:20AM -0800, Adam Williamson wrote:
>> On Wed, 2010-12-01 at 13:23 -0500, Doug Ledford wrote:
>>
>>> That being said, F14 went out with a broken mdadm *purely* because of
>>> this policy.
>>
>>> Evidently my update was approved somewhere along the way, but because of
>>> the volume of bodhi spam I get, I missed it.
>>
>> ...so what you're saying is that F14 did not in fact go out with a
>> broken mdadm *purely* because of the policy, but in a small part because
>> of the policy and in a large part because you don't read / filter your
>> emails carefully enough.
>>
>>>   So I'm not sure if it
>>> could have made F14 final or not, but I know it didn't because I was
>>> working on other things at the time.
>>
>> bodhi - 2010-10-14 22:36:08
>> Critical path update approved 
>>
>> The final change deadline was 10-18; you had four days to push the
>> update.
> 
> Also, if karma automatism was enabled for that update, it would have been
> queued for pushing right when it was approved.
> 
> luke

I don't enable karma automatism because in the past I've seen people
report testing karma +1 when they did not, in fact, doing anything
useful in terms of testing (aka, they had no software raid devices, yet
they said the system still works...well, duh, it's only used on software
raid devices so if you don't have any, then it doesn't make any
difference to you).

-- 
Doug Ledford 
  GPG KeyID: CFBFF194
  http://people.redhat.com/dledford

Infiniband specific RPMs available at
  http://people.redhat.com/dledford/Infiniband



signature.asc
Description: OpenPGP digital signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Luke Macken]

On Wed, Dec 01, 2010 at 10:41:20AM -0800, Adam Williamson wrote:
> On Wed, 2010-12-01 at 13:23 -0500, Doug Ledford wrote:
> 
> > That being said, F14 went out with a broken mdadm *purely* because of
> > this policy.
> 
> > Evidently my update was approved somewhere along the way, but because of
> > the volume of bodhi spam I get, I missed it.
> 
> ...so what you're saying is that F14 did not in fact go out with a
> broken mdadm *purely* because of the policy, but in a small part because
> of the policy and in a large part because you don't read / filter your
> emails carefully enough.
> 
> >   So I'm not sure if it
> > could have made F14 final or not, but I know it didn't because I was
> > working on other things at the time.
> 
> bodhi - 2010-10-14 22:36:08
> Critical path update approved 
> 
> The final change deadline was 10-18; you had four days to push the
> update.

Also, if karma automatism was enabled for that update, it would have been
queued for pushing right when it was approved.

luke
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 16:22 -0500, Doug Ledford wrote:

> If the ticket can be allowed to languish that long, then I don't feel in
> the least bit guilty that I didn't drop my other Red Hat
> responsibilities on the floor when the ticket was finally approved.  By
> the time it was approved, I had already moved on.

I didn't say you should. I said your assertion that F14 went out with an
older mdadm '*purely* because of this policy' was not supported by
evidence. The package could have been included in F14 had it been pushed
soon after critpath testing approval was granted. Therefore, it's simply
not correct that the policy alone prevented it being included. (If you'd
submitted the update to stable close to the final date it wouldn't have
taken five days to be pushed; rel-eng do pushes far more often when it's
close to a deadline).

>   Fortunately, I wasn't
> under the same restrictions on Red Hat Enterprise Linux 6, so it went GA
> with a much better, fixed mdadm relative to Fedora.

The implied comparison here is unfair and hence not helpful; RHEL has a
large group of paid test staff. Fedora does not.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Luke Macken]

On Mon, Nov 29, 2010 at 01:36:18PM +, Petr Pisar wrote:
> On 2010-11-29, Peter Robinson  wrote:
> > On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
> >
> > Proven testers do get copies of these emails (dozens of them) and its
> > also summarised in the updates-testing report for all to see.
> >
> Oh, I thought  described as `For testers of Fedora
> development releases' concerns alpha and similar (pre-)releases only.
> 
> In that case, only one issue remains: to stop spamming package
> maintainer.

The next release of bodhi that I'm going to push out to our releng
servers this week will stop spamming proventesters and maintainers about
these, since this data is now in the updates-testing digests that get
sent out to the test-list with each push.

luke
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Doug Ledford]

On 12/01/2010 01:41 PM, Adam Williamson wrote:
> On Wed, 2010-12-01 at 13:23 -0500, Doug Ledford wrote:
> 
>> That being said, F14 went out with a broken mdadm *purely* because of
>> this policy.
> 
>> Evidently my update was approved somewhere along the way, but because of
>> the volume of bodhi spam I get, I missed it.
> 
> ...so what you're saying is that F14 did not in fact go out with a
> broken mdadm *purely* because of the policy, but in a small part because
> of the policy and in a large part because you don't read / filter your
> emails carefully enough.

Um, no.  It's because of the policy, period.

>>   So I'm not sure if it
>> could have made F14 final or not, but I know it didn't because I was
>> working on other things at the time.
> 
> bodhi - 2010-10-14 22:36:08
> Critical path update approved 
> 
> The final change deadline was 10-18; you had four days to push the
> update.

Oh my, 4 whole days you say?  Let's see, the initial bodhi ticket was
filed on 2010-08-05, it was initially pushed to testing on 2010-08-10
(so five days just to push), an automated bug telling me mdadm-3.1.4 had
been released upstream was filed on 2010-09-13, this package was
approved for stable on 2010-10-14, and final drop dead push date was
2010-10-18.  If the initial push was any indication, it would have
missed the final release by a day even if I pushed it the second it
became approved!  But I would have been pushing an outdated package to boot!

I'm sorry, but no, you don't get to say ONE DAMN WORD about me not
getting it pushed in a 4 day window when the ticket itself took 60+ days
to get approved!  And that was after I put out a general request for
testing of the update on devel@ in order to try and drum up testing.

If the ticket can be allowed to languish that long, then I don't feel in
the least bit guilty that I didn't drop my other Red Hat
responsibilities on the floor when the ticket was finally approved.  By
the time it was approved, I had already moved on.  Fortunately, I wasn't
under the same restrictions on Red Hat Enterprise Linux 6, so it went GA
with a much better, fixed mdadm relative to Fedora.

-- 
Doug Ledford 
  GPG KeyID: CFBFF194
  http://people.redhat.com/dledford

Infiniband specific RPMs available at
  http://people.redhat.com/dledford/Infiniband



signature.asc
Description: OpenPGP digital signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 21:12 +0100, Kevin Kofler wrote:
> Adam Williamson wrote:
> > You can get an exception to the policy with majority approval from FESCo.
> 
> That "exception process" is a joke! It takes too long to get approval from 2 
> people, one in a medium-sized group and the other in a very large group, and 
> so the process is that you need to get 5 votes out of a group of only 9 
> people which only meets once a week instead? (Not to mention that they'll 
> probably say "-1, just get a proventester to test it".) Needless to say, 
> this ridiculous process has never ever been used.

But it's a group of people who are *obliged* to meet once a week and
cover all business presented to them.

No-one's tried to use the process, so we can't draw any conclusions
about how it would work, can we?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Kevin Kofler]

Adam Williamson wrote:
> You can get an exception to the policy with majority approval from FESCo.

That "exception process" is a joke! It takes too long to get approval from 2 
people, one in a medium-sized group and the other in a very large group, and 
so the process is that you need to get 5 votes out of a group of only 9 
people which only meets once a week instead? (Not to mention that they'll 
probably say "-1, just get a proventester to test it".) Needless to say, 
this ridiculous process has never ever been used.

> Technically speaking, no update is completely blocked by the current
> policy.

But practically speaking, it is.

Kevin Kofler

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 13:23 -0500, Doug Ledford wrote:

> So, for anyone that cares, I will posit a maxim that you can't create a
> policy that creates an unbreakable roadblock without also creating
> either A) a job who's responsibility it is to clear said roadblocks in a
> reasonable period of time or B) lifting the roadblock after a reasonable
> period of time regardless of whether the conditions of the restriction
> were met or not.

Also, the current policy does not in fact constitute an unbreakable
roadblock. It seems to be being ignored in current discussion, but it
contains an exception clause:

https://fedoraproject.org/w/index.php?title=Package_update_acceptance_criteria#Exception_process

it's short, but it's there. You can get an exception to the policy with
majority approval from FESCo. Technically speaking, no update is
completely blocked by the current policy.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Wed, 2010-12-01 at 13:23 -0500, Doug Ledford wrote:

> That being said, F14 went out with a broken mdadm *purely* because of
> this policy.

> Evidently my update was approved somewhere along the way, but because of
> the volume of bodhi spam I get, I missed it.

...so what you're saying is that F14 did not in fact go out with a
broken mdadm *purely* because of the policy, but in a small part because
of the policy and in a large part because you don't read / filter your
emails carefully enough.

>   So I'm not sure if it
> could have made F14 final or not, but I know it didn't because I was
> working on other things at the time.

bodhi - 2010-10-14 22:36:08
Critical path update approved 

The final change deadline was 10-18; you had four days to push the
update.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Doug Ledford]

On 11/30/2010 05:50 AM, Rahul Sundaram wrote:
> 
> 
> On Tue, Nov 30, 2010 at 3:20 AM, Kevin Kofler  > wrote:
> 
> Rahul Sundaram wrote:
> > I am sorry but "somebody does not did his job"?  It is not the
> "job" of
> > anyone to test packages for you.  They are merely helping out and we
> > will get more help if we express gratitude instead of a sense of
> > entitlement.
> 
> But this is exactly why the current policy which REQUIRES testing is
> broken.
> 
> 
> Repeating your view point over and over again is not going to win a
> conversation.  You believe that it is fine to test for Fedora 14 and
> push for Fedora 13 without testing for that release explicitly.  So I am
> not surprised you are against making testing a requirement.  Obviously,
> we have different perspectives here. 
> 
> Rahul
> 

I'll jump on Kevin's boat for this argument.  The current policy
*mandates* testing before we are allowed to push a package out.  If we
don't get the right testing (which we can't do ourselves), then a
critpath package is held up.  So, it damn well better be *somebody's*
job to perform crit path testing, otherwise the current policy creates a
roadblock to release with no guarantee that the roadblock will ever get
cleared.

That being said, F14 went out with a broken mdadm *purely* because of
this policy.  The mdadm I had in updates-testing was far better, and has
been reported by probably six or eight users now to completely solve
their problems (only one person didn't get their problem solved by the
update in updates-testing, and no new problems were reported by anybody)
across 3 or 4 different bugs.  It was ready in time, but didn't get the
appropriate testing in time.  And I *knew* it was better than what we
had, even without the testers telling me so, because the updated package
fixed a known race issue in file handling that was all too easy to hit
whenever udev was allowed to spawn more than one mdadm -I instance at a
time (aka, about 80% of the time you hit the race condition, and the fix
was known to solve the problem 100% of the time).  But my hands were tied.

So, for anyone that cares, I will posit a maxim that you can't create a
policy that creates an unbreakable roadblock without also creating
either A) a job who's responsibility it is to clear said roadblocks in a
reasonable period of time or B) lifting the roadblock after a reasonable
period of time regardless of whether the conditions of the restriction
were met or not.

Evidently my update was approved somewhere along the way, but because of
the volume of bodhi spam I get, I missed it.  So I'm not sure if it
could have made F14 final or not, but I know it didn't because I was
working on other things at the time.

Bodhi critpath restrictions == -1000 in their current form as far as I'm
concerned.  Fix it as you see fit, but it definitely needs fixing.

-- 
Doug Ledford 
  GPG KeyID: CFBFF194
  http://people.redhat.com/dledford

Infiniband specific RPMs available at
  http://people.redhat.com/dledford/Infiniband



signature.asc
Description: OpenPGP digital signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Kevin Kofler]

Rahul Sundaram wrote:
> You believe that it is fine to test for Fedora 14 and push for Fedora 13
> without testing for that release explicitly.

Or the opposite, for that matter.

Kevin Kofler

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Rahul Sundaram]

On Tue, Nov 30, 2010 at 3:20 AM, Kevin Kofler wrote:

> Rahul Sundaram wrote:
> > I am sorry but "somebody does not did his job"?  It is not the "job" of
> > anyone to test packages for you.  They are merely helping out and we
> > will get more help if we express gratitude instead of a sense of
> > entitlement.
>
> But this is exactly why the current policy which REQUIRES testing is
> broken.
>

Repeating your view point over and over again is not going to win a
conversation.  You believe that it is fine to test for Fedora 14 and push
for Fedora 13 without testing for that release explicitly.  So I am not
surprised you are against making testing a requirement.  Obviously, we have
different perspectives here.

Rahul
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Jiri Skala]

On Mon, 2010-11-29 at 14:34 +, Petr Pisar wrote: 
> On 2010-11-29, Peter Robinson  wrote:
> > On Mon, Nov 29, 2010 at 1:36 PM, Petr Pisar  wrote:
> >> On 2010-11-29, Peter Robinson  wrote:
> >>> On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
> >>>
> > have no problems with it and I maintain around 120 packages. Filter it
> > if you don't like it.
> 
> Are you interrested in such notifications?

Discussion about filtering/blocking mentioned messages is about how to
fix consequence but...

The discussion should search for improving source of the problem.

What about source of this? I could count e.g.:
- missing reaction of QA
- missing approval for the package to be moved to stable
- missing additional action of developer

I've got a lot messages due to one package (needs modem for testing). I
contacted some concerned people directly and I'm glad to say this
situation has no more repeated with this package. This works fine.
I believe only a little portion of packages are related to this problem.

Well, I'm convinced better is "ask for action" then to discuss what
"should/shouldn't happen".

Skalnik 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Michael Schwendt]

On Mon, 29 Nov 2010 16:11:37 + (UTC), Petr wrote:

> On 2010-11-29, Rahul Sundaram wrote:
> > On 11/29/2010 08:04 PM, Petr Pisar wrote:
> >> I do not get the idea why I should filter some irrelevant mails if
> >> better is to not sent them. Especially if I cannot solve the subject of
> >> the mail.

Well, rest assured that there are members in the proventesters group, who
ignore/filter those nag mails, too. They are not messages you'd like to
process repeatedly. F-12 is a sinking ship, F-13 is being abandoned by
more users, because F-14 is current.

> >> Yeah, the subject is somobody does not did his job. I cannot
> >> imagine the knowledge would help me in my packager duties.
> >
> > I am sorry but "somebody does not did his job"?  It is not the "job" of
> > anyone to test packages for you.  They are merely helping out and we
> > will get more help if we express gratitude instead of a sense of
> > entitlement. 
> >
> I do not accuse anybody not duing his job. I just complain the mails are
> absolutly irrelevant from point of view of package maintainer who pushed
> the package.

There are other messages sent by bodhi, which are "irrelevant" and just
noise. All testers, who have left a comment, receive the notification that
an update is 7 days old. And they receive that notification repeatedly as
long as the maintainer forgets to push the package to stable.

> You can replace the `job' with `expected duty' or `role'.

That point of view is too demanding. Testing every critpath update,
including updates for old dists, is not anything like a duty for testers
or proventesters. For *any* tester who wants to help, even if only
occasionally, joining the proventesters group has become a necessity
due to the update acceptance criteria. Else the ordinary tester's karma
would be "less worth" in the new system.

> I could worry about my packages not being stabilized, however that would
> be all I could do and it would have no effect. Frankly, I don't care
> about stable-status of my packages as that are Fedora rules that draw
> line between packager and tester `duties' in package update. I could
> become proven tester to test (not only) my own packages, however such
> cheating would be silly.

Which is why the current testing requirements are infeasible and insane.

Packagers ought to retain the freedom to decide what's most appropriate
for their updates … and only lose bits of that freedom when they take
their responsibilities too lightly and cause damage. As one of the
responsibilities, it may be important for packagers to disable bodhi
karma automatism.
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Toshio Kuratomi]

On Mon, Nov 29, 2010 at 09:33:46AM -0800, Adam Williamson wrote:
> On Mon, 2010-11-29 at 14:34 +, Petr Pisar wrote:
> 
> > I do not get the idea why I should filter some irrelevant mails if
> > better is to not sent them. Especially if I cannot solve the subject of
> > the mail. Yeah, the subject is somobody does not did his job. I cannot
> > imagine the knowledge would help me in my packager duties.
> 
> Your packager duties include aiding in ensuring testing of your packages
> takes place.

Note that this is not explicit.  What is explicit is that maintainers are
expected to:
"Deal with reported bugs in a timely manner "

http://fedoraproject.org/wiki/Package_maintainer_responsibilities

I think one source of the feelings on some maintainers' parts is that the new
update criteria get in the way of "timey manner" in the quest to prevent
regressions.

> It is not true that there is nothing you can do. You can
> contact proven testers and ask them to test your package, you can
> contact people who use your package (I would hope you know some!) and
> ask them to become proven testers to ensure that your updates are pushed
> in timely fashion in future.
> 
> QA is not 'someone else's problem', it's a collaborative effort. Fedora
> is a community-based volunteer-driven project; neither Red Hat nor The
> Fedora Project has a thousand minimum-wage Fedora test slaves in a room
> somewhere ready to do all the testing we could ever desire.

Actually, what you say here is both true and untrue.  If you look at Fedora
as a product to be delivered, then QA is a problem for everyone delivering
that product to worry about.  However, if you look at Fedora as an open
source project then you'll find that tasks are divided among contributor
interest rather than end-product.  It is the problem of the people who care
about QA to worry about QA -- the people who have an itch to scratch have
the responsibility to scratch it for themselves.

These ideas are not diametrically opposed, rather they're two different ways
to look at this problem and try to understand that an ideal solution
encompasses both viewpoints.  On the one hand, convincing everyone to care
about QA and make sure that packages they care about are tested to prevent
regressions, and on the other hand, giving maintainers the ability to make
judgements about the risk vs benefit of the update that they want to push
and getting the high benefit packages into users hands asap.

-Toshio


pgpF5k8gZP9YT.pgp
Description: PGP signature
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Kevin Kofler]

Rahul Sundaram wrote:
> I am sorry but "somebody does not did his job"?  It is not the "job" of
> anyone to test packages for you.  They are merely helping out and we
> will get more help if we express gratitude instead of a sense of
> entitlement.

But this is exactly why the current policy which REQUIRES testing is broken.

Kevin Kofler

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Adam Williamson]

On Mon, 2010-11-29 at 14:34 +, Petr Pisar wrote:

> I do not get the idea why I should filter some irrelevant mails if
> better is to not sent them. Especially if I cannot solve the subject of
> the mail. Yeah, the subject is somobody does not did his job. I cannot
> imagine the knowledge would help me in my packager duties.

Your packager duties include aiding in ensuring testing of your packages
takes place. It is not true that there is nothing you can do. You can
contact proven testers and ask them to test your package, you can
contact people who use your package (I would hope you know some!) and
ask them to become proven testers to ensure that your updates are pushed
in timely fashion in future.

QA is not 'someone else's problem', it's a collaborative effort. Fedora
is a community-based volunteer-driven project; neither Red Hat nor The
Fedora Project has a thousand minimum-wage Fedora test slaves in a room
somewhere ready to do all the testing we could ever desire.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Petr Pisar]

On 2010-11-29, Rahul Sundaram  wrote:
> On 11/29/2010 08:04 PM, Petr Pisar wrote:
>> I do not get the idea why I should filter some irrelevant mails if
>> better is to not sent them. Especially if I cannot solve the subject of
>> the mail. Yeah, the subject is somobody does not did his job. I cannot
>> imagine the knowledge would help me in my packager duties.
>
> I am sorry but "somebody does not did his job"?  It is not the "job" of
> anyone to test packages for you.  They are merely helping out and we
> will get more help if we express gratitude instead of a sense of
> entitlement. 
>
I do not accuse anybody not duing his job. I just complain the mails are
absolutly irrelevant from point of view of package maintainer who pushed
the package. You can replace the `job' with `expected duty' or `role'.

I could worry about my packages not being stabilized, however that would
be all I could do and it would have no effect. Frankly, I don't care
about stable-status of my packages as that are Fedora rules that draw
line between packager and tester `duties' in package update. I could
become proven tester to test (not only) my own packages, however such
cheating would be silly.

-- Petr

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Rahul Sundaram]

On 11/29/2010 08:04 PM, Petr Pisar wrote:
> I do not get the idea why I should filter some irrelevant mails if
> better is to not sent them. Especially if I cannot solve the subject of
> the mail. Yeah, the subject is somobody does not did his job. I cannot
> imagine the knowledge would help me in my packager duties.

I am sorry but "somebody does not did his job"?  It is not the "job" of
anyone to test packages for you.  They are merely helping out and we
will get more help if we express gratitude instead of a sense of
entitlement. 

Rahul
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Petr Pisar]

On 2010-11-29, Peter Robinson  wrote:
> On Mon, Nov 29, 2010 at 1:36 PM, Petr Pisar  wrote:
>> On 2010-11-29, Peter Robinson  wrote:
>>> On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
>>>
>>> Proven testers do get copies of these emails (dozens of them) and its
>>> also summarised in the updates-testing report for all to see.
>>>
>> Oh, I thought  described as `For testers of Fedora
>> development releases' concerns alpha and similar (pre-)releases only.
>>
>> In that case, only one issue remains: to stop spamming package
>> maintainer.
>
> Well it advises the package maintainer that their update is old.

And I am supposed to be reminded every day. My sclerosis is no so bad
yet.

> have no problems with it and I maintain around 120 packages. Filter it
> if you don't like it.

Are you interrested in such notifications?

I do not get the idea why I should filter some irrelevant mails if
better is to not sent them. Especially if I cannot solve the subject of
the mail. Yeah, the subject is somobody does not did his job. I cannot
imagine the knowledge would help me in my packager duties.

-- Petr

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Marcela Mašláňová]

On 11/29/2010 02:46 PM, Peter Robinson wrote:
> On Mon, Nov 29, 2010 at 1:36 PM, Petr Pisar  wrote:
>> On 2010-11-29, Peter Robinson  wrote:
>>> On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
>>>
>>> Proven testers do get copies of these emails (dozens of them) and its
>>> also summarised in the updates-testing report for all to see.
>>>
>> Oh, I thought  described as `For testers of Fedora
>> development releases' concerns alpha and similar (pre-)releases only.
>>
>> In that case, only one issue remains: to stop spamming package
>> maintainer.
> Well it advises the package maintainer that their update is old. I
> have no problems with it and I maintain around 120 packages. Filter it
> if you don't like it.
>
> Peter
Great, so I will filter all messages from koji, but not these, which
contains -1. Wouldn't be easier don't bother with spam?

Marcela

-- 
Marcela Mašláňová
BaseOS team Brno

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Peter Robinson]

On Mon, Nov 29, 2010 at 1:36 PM, Petr Pisar  wrote:
> On 2010-11-29, Peter Robinson  wrote:
>> On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
>>
>> Proven testers do get copies of these emails (dozens of them) and its
>> also summarised in the updates-testing report for all to see.
>>
> Oh, I thought  described as `For testers of Fedora
> development releases' concerns alpha and similar (pre-)releases only.
>
> In that case, only one issue remains: to stop spamming package
> maintainer.

Well it advises the package maintainer that their update is old. I
have no problems with it and I maintain around 120 packages. Filter it
if you don't like it.

Peter
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Petr Pisar]

On 2010-11-29, Peter Robinson  wrote:
> On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
>
> Proven testers do get copies of these emails (dozens of them) and its
> also summarised in the updates-testing report for all to see.
>
Oh, I thought  described as `For testers of Fedora
development releases' concerns alpha and similar (pre-)releases only.

In that case, only one issue remains: to stop spamming package
maintainer.

-- Petr

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: old_testing_critpath notifications

[Peter Robinson]

On Mon, Nov 29, 2010 at 9:56 AM, Petr Pisar  wrote:
> Hello,
>
> Few days ago I started to get very usefull notifications that my
> critical package, mingetty-1.08-6.fc13, `has been in 'testing' status
> for over 2 weeks, and has yet to be approved.'
>
> I doubt such mails help me as the package maintainer, because according
> current Updates policy
> ,
>  I can only wait to some (proven) testers to test my package.
>
> I think better place to send such allerts is proven-testers mailing list
> / alias (does not exist amazingly) or
>  mailing list.
>
> I'd like to request persons reposnsible for forcing and implementing
> Update Policy and bodhi notification system to re-eavaluate current
> configuration in spot of this concerns and to make package update
> process more friendly for all involved people.

Proven testers do get copies of these emails (dozens of them) and its
also summarised in the updates-testing report for all to see.

Peter
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

old_testing_critpath notifications

[Petr Pisar]

Hello,

Few days ago I started to get very usefull notifications that my
critical package, mingetty-1.08-6.fc13, `has been in 'testing' status
for over 2 weeks, and has yet to be approved.'

I doubt such mails help me as the package maintainer, because according
current Updates policy
,
 I can only wait to some (proven) testers to test my package.

I think better place to send such allerts is proven-testers mailing list
/ alias (does not exist amazingly) or
 mailing list.

I'd like to request persons reposnsible for forcing and implementing
Update Policy and bodhi notification system to re-eavaluate current
configuration in spot of this concerns and to make package update
process more friendly for all involved people.

-- Petr

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel