On Mon, 2016-07-11 at 15:12 +0000, Christian Stadelmann wrote:
> First, have you contacted them? This looks like a misunderstanding
> between the way you (and probably most fedora packagers + bodhi
> developers) think "karma" works and the way they understand it.
>
> My perspective is being someone who often tests packages but doesn't
> package them.
>
> From my point of view, the "karma" ("Is the update generally
> functional?") field is not quite clear to users of bodhi. I only know
> because I've been using it for quite a long time and understand that
> some packages are auto-pushed to stable after reaching a specific
> karma limit.
Yeah, I think this is a good point. Honestly it's a bit QA's fault,
because there's lots of stuff in the new Bodhi (2) which we asked for
but haven't entirely got around to doing much with yet, like the
various feedback types. For now we've got the feedback types, but in
essence only the bits that were there in Bodhi 1 really 'matter' much,
except to manual review by the update submitter.
It might be good if the Bodhi UI linked to
https://fedoraproject.org/wiki/QA:Update_feedback_guidelines , for one
thing.
> Testers providing karma in <2 minutes probably just installed the update and
> if it
> 1. doesn't break dnf
> 2. succeeds to run the application
> 3. (maybe) doesn't crash on 5 seconds of testing
> they give it +1.
Sometimes this is a perfectly appropriate level of testing, though.
It's quite difficult to write 'perfect' guidelines for this, because
there's a big human experience element to it, knowing how important
different packages are, how significant the changes in a given update
are...
> I suggest to take these actions:
> 1. Change or extend the phrase "Is the update generally functional?" to have
> more details.
> 2. Link the "Is the update generally functional?" phrase to some more
> documentation, e.g. on Fedora Wiki
Like the feedback guidelines :P good idea.
> 3. Talk to those testers
> 4. If this happens often (I can't say, someone needs to do research),
> have somebody look at the data (analyze it) and contact testers with
> "suspicious" testing behavior.
We have actually already talked to the relevant people in this case,
and as others have pointed out, there are valid reasons for early
feedback, like downloading updates from Koji.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
