[openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Davanum Srinivas]

Team,

Steve pointed out to a problem in Stackalytics:
https://twitter.com/stevebot/status/718185667709267969

It's pretty clear what's happening if you look here:
https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open

Here's the drastic step (i'd like to avoid):
https://review.openstack.org/#/c/303545/

What do you think?

Thanks,
Dims

-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Dmitry Tantsur]

2016-04-08 19:26 GMT+02:00 Davanum Srinivas :

> Team,
>
> Steve pointed out to a problem in Stackalytics:
> https://twitter.com/stevebot/status/718185667709267969


There are many ways to game a simple +1 counter, such as +1'ing changes
that already have at least 1x +2, or which already approved, or which need
rechecking...


>
>
> It's pretty clear what's happening if you look here:
>
> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>
> Here's the drastic step (i'd like to avoid):
> https://review.openstack.org/#/c/303545/
>
> What do you think?
>

One more possible (though also imperfect) mitigation is to make exception
from the usual 2x +2 rule for requirements updates passing gates and use
only 1x +2. Then requirements reviews will take substantially less time to
land, reducing need/possibility of having such +1's.


>
> Thanks,
> Dims
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
--
-- Dmitry Tantsur
--
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Dean Troyer]

On Fri, Apr 8, 2016 at 12:26 PM, Davanum Srinivas  wrote:

> Team,
>
> Steve pointed out to a problem in Stackalytics:
> https://twitter.com/stevebot/status/718185667709267969
>
> It's pretty clear what's happening if you look here:
>
> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>
> Here's the drastic step (i'd like to avoid):
> https://review.openstack.org/#/c/303545/


Allow me to go and +1 that even if it hurts my stats when the -2 comes. :)

I fear that the only way to make Stackalytics meaningless to
management-types is for it to go away, only to be replaced by some other
(possibly worse) metric to be misused.

We all need to educate our internal company leadership on the (lack of)
actual truth in those numbers...

dt

-- 

Dean Troyer
dtro...@gmail.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 01:42 PM, Dmitry Tantsur wrote:
> 2016-04-08 19:26 GMT+02:00 Davanum Srinivas :
> 
>> Team,
>>
>> Steve pointed out to a problem in Stackalytics:
>> https://twitter.com/stevebot/status/718185667709267969
> 
> 
> There are many ways to game a simple +1 counter, such as +1'ing changes
> that already have at least 1x +2, or which already approved, or which need
> rechecking...
> 
> 
>>
>>
>> It's pretty clear what's happening if you look here:
>>
>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>
>> Here's the drastic step (i'd like to avoid):
>> https://review.openstack.org/#/c/303545/
>>
>> What do you think?
>>
> 
> One more possible (though also imperfect) mitigation is to make exception
> from the usual 2x +2 rule for requirements updates passing gates and use
> only 1x +2. Then requirements reviews will take substantially less time to
> land, reducing need/possibility of having such +1's.

Proposal bot patches merge in many cases with 1 +2 already.

Have you looked at the timing of the bot patches generated and the first
+1's? If not, take a look at that.

I don't think we should be expecting core reviewers to schedule
approving bot proposals to prevent extraneous +1s.

Thanks,
Anita.

> 
>>
>> Thanks,
>> Dims
>>
>> --
>> Davanum Srinivas :: https://twitter.com/dims
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Morgan Fainberg]

On Fri, Apr 8, 2016 at 1:42 PM, Dmitry Tantsur 
wrote:

>
> 2016-04-08 19:26 GMT+02:00 Davanum Srinivas :
>
>> Team,
>>
>> Steve pointed out to a problem in Stackalytics:
>> https://twitter.com/stevebot/status/718185667709267969
>
>
> There are many ways to game a simple +1 counter, such as +1'ing changes
> that already have at least 1x +2, or which already approved, or which need
> rechecking...
>
>
>>
>>
>> It's pretty clear what's happening if you look here:
>>
>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>
>> Here's the drastic step (i'd like to avoid):
>> https://review.openstack.org/#/c/303545/
>>
>> What do you think?
>>
>
> One more possible (though also imperfect) mitigation is to make exception
> from the usual 2x +2 rule for requirements updates passing gates and use
> only 1x +2. Then requirements reviews will take substantially less time to
> land, reducing need/possibility of having such +1's.
>
>

Many projects (including Keystone) already go with a single core +2/+A for
Proposal Bot changes. This definitely helps them land faster (which is why
the policy is there).
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Doug Wiegley]

> On Apr 8, 2016, at 11:26 AM, Davanum Srinivas  wrote:
> 
> Team,
> 
> Steve pointed out to a problem in Stackalytics:
> https://twitter.com/stevebot/status/718185667709267969
> 
> It's pretty clear what's happening if you look here:
> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
> 
> Here's the drastic step (i'd like to avoid):
> https://review.openstack.org/#/c/303545/
> 
> What do you think?

Is it really worth trying to stop the folks that will game the system?  That is 
a never-ending arms race.

Has anyone ever been made core or had some influence purely by pumping 
stacklytics? I haven’t seen it, since that usually requires personal 
relationships that are far more important than numbers.

Are they using the numbers for some internal company purpose maybe?  If so, how 
does it matter to any of us?

Chasing this tail just takes time away from useful things, IMO.

doug


> 
> Thanks,
> Dims
> 
> -- 
> Davanum Srinivas :: https://twitter.com/dims
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 03:04 PM, Doug Wiegley wrote:
> 
>> On Apr 8, 2016, at 11:26 AM, Davanum Srinivas  wrote:
>>
>> Team,
>>
>> Steve pointed out to a problem in Stackalytics:
>> https://twitter.com/stevebot/status/718185667709267969
>>
>> It's pretty clear what's happening if you look here:
>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>
>> Here's the drastic step (i'd like to avoid):
>> https://review.openstack.org/#/c/303545/
>>
>> What do you think?
> 
> Is it really worth trying to stop the folks that will game the system?  That 
> is a never-ending arms race.
> 
> Has anyone ever been made core or had some influence purely by pumping 
> stacklytics? I haven’t seen it, since that usually requires personal 
> relationships that are far more important than numbers.
> 
> Are they using the numbers for some internal company purpose maybe?  If so, 
> how does it matter to any of us?
> 
> Chasing this tail just takes time away from useful things, IMO.
> 
> doug

The shame of it is that at least one of the folks wasting their time
with this has the ability to become a beneficial contributor including
useful reviews. I do hope that drawing attention to their activity will
motivate them to re-evaluate and perhaps engage in behaviour that is
both needed and valued.

I do agree with you though Doug, to most people this already is obvious.

Thanks,
Anita.

> 
> 
>>
>> Thanks,
>> Dims
>>
>> -- 
>> Davanum Srinivas :: https://twitter.com/dims
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Major Hayden]

On 04/08/2016 02:04 PM, Doug Wiegley wrote:
> Are they using the numbers for some internal company purpose maybe?  If so, 
> how does it matter to any of us?
> 
> Chasing this tail just takes time away from useful things, IMO.

Although I understand the reasoning behind the effort underway in the review 
above to skip Stackalytics stats for proposal bot reviews, it doesn't really 
add a ton of value.  As Doug noted, one cannot simply become a core reviewer by 
gaming stackalytics.

Those personal interactions on mailing lists, reviews with lots of patchsets, 
IRC meetings, and in-person events (like mid-cycles/summits) make the big 
difference.  Can we reach out to some of these people making questionable +1's 
and find out if we can help them become a more productive community member?  If 
there are companies out there who are setting "quotas" for review counts, we 
could possibly reach out to them as well.  Perhaps I'm being too optimistic. :)

But, as Dolph said earlier, leaving this issue alone certainly makes it easier 
to single out the folks who are doing something unproductive. ;)

--
Major Hayden

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 03:19 PM, Major Hayden wrote:
> On 04/08/2016 02:04 PM, Doug Wiegley wrote:
>> Are they using the numbers for some internal company purpose maybe?  If so, 
>> how does it matter to any of us?
>>
>> Chasing this tail just takes time away from useful things, IMO.
> 
> Although I understand the reasoning behind the effort underway in the review 
> above to skip Stackalytics stats for proposal bot reviews, it doesn't really 
> add a ton of value.  As Doug noted, one cannot simply become a core reviewer 
> by gaming stackalytics.
> 
> Those personal interactions on mailing lists, reviews with lots of patchsets, 
> IRC meetings, and in-person events (like mid-cycles/summits) make the big 
> difference.  Can we reach out to some of these people making questionable 
> +1's and find out if we can help them become a more productive community 
> member?  If there are companies out there who are setting "quotas" for review 
> counts, we could possibly reach out to them as well.  Perhaps I'm being too 
> optimistic. :)

Nothing is stopping you from doing so. You can see the names and can
find the emails of those engaged in this by following the gerrit link
Dims posted in his first post.

Perhaps as you say, the personal touch may help them to learn how to
contribute in a way that has value.

Thanks,
Anita.

> 
> But, as Dolph said earlier, leaving this issue alone certainly makes it 
> easier to single out the folks who are doing something unproductive. ;)
> 
> --
> Major Hayden
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Major Hayden]

On 04/08/2016 02:25 PM, Anita Kuno wrote:
> Nothing is stopping you from doing so. You can see the names and can
> find the emails of those engaged in this by following the gerrit link
> Dims posted in his first post.
> 
> Perhaps as you say, the personal touch may help them to learn how to
> contribute in a way that has value.

I'll take a sample of the folks listed there and contact them.  Hopefully I can 
provide some general results here soon.

--
Major Hayden

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 03:28 PM, Major Hayden wrote:
> On 04/08/2016 02:25 PM, Anita Kuno wrote:
>> Nothing is stopping you from doing so. You can see the names and can
>> find the emails of those engaged in this by following the gerrit link
>> Dims posted in his first post.
>>
>> Perhaps as you say, the personal touch may help them to learn how to
>> contribute in a way that has value.
> 
> I'll take a sample of the folks listed there and contact them.  Hopefully I 
> can provide some general results here soon.
> 
> --
> Major Hayden
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
Wonderful, thank you Major, I hope your efforts prove fruitful.

Thank you,
Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Jeremy Stanley]

On 2016-04-08 14:28:33 -0500 (-0500), Major Hayden wrote:
> I'll take a sample of the folks listed there and contact them.
> Hopefully I can provide some general results here soon.

Thanks for taking this up--some people just need
encouragement/suggestions for better ways to make an impact. On the
other hand, if you find that many of them have addresses at the same
company domain... well I guess we can find people higher up the
ladder in those companies and talk to them about how to channel
their employee quotas/incentives in more productive directions for
the community as well.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Jeremy Stanley]

On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
> There are many ways to game a simple +1 counter, such as +1'ing changes
> that already have at least 1x +2, or which already approved, or which need
> rechecking...
[...]

The behavior which baffles me, and also seems to be on the rise
lately, is random +1 votes on changes whose commit messages and/or
status clearly indicate they should not merged and do not need to be
reviewed. I suppose that's another an easy way to avoid the dreaded
"disagreements" counter?
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Davanum Srinivas]

On Fri, Apr 8, 2016 at 4:31 PM, Jeremy Stanley  wrote:
> On 2016-04-08 14:28:33 -0500 (-0500), Major Hayden wrote:
>> I'll take a sample of the folks listed there and contact them.
>> Hopefully I can provide some general results here soon.
>
> Thanks for taking this up--some people just need
> encouragement/suggestions for better ways to make an impact. On the
> other hand, if you find that many of them have addresses at the same
> company domain... well I guess we can find people higher up the
> ladder in those companies and talk to them about how to channel
> their employee quotas/incentives in more productive directions for
> the community as well.

+1 Major, Jeremy.

I am also sending this suggestion. If you can think of other good
links, please share.

==
Please watch this video:
https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/tales-from-the-ship-navigating-the-openstack-community-seas

Please take a look at tips here:
https://developer.ibm.com/opentech/2015/03/27/checklist-performing-openstack-code-reviews/
https://krotscheck.net/2015/07/13/code-review-in-openstack.html
==

> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[John Dickinson]

On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:

> On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>> There are many ways to game a simple +1 counter, such as +1'ing changes
>> that already have at least 1x +2, or which already approved, or which need
>> rechecking...
> [...]
>
> The behavior which baffles me, and also seems to be on the rise
> lately, is random +1 votes on changes whose commit messages and/or
> status clearly indicate they should not merged and do not need to be
> reviewed. I suppose that's another an easy way to avoid the dreaded
> "disagreements" counter?
> -- 
> Jeremy Stanley


I have been told that some OpenStack on boarding teaches new members of the 
community to do reviews. And they say, effectively, "muddle through as you can. 
You won't understand it all at first, but do your best. When you're done, add a 
+1 and move to the next one"

I've been working to correct this when I've seen it, but +1 reviews with no 
comments might not be people trying to game. It might simply be people trying 
to get involved that don't know any better yet.

--John





signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 04:39 PM, John Dickinson wrote:
> 
> 
> On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
> 
>> On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>>> There are many ways to game a simple +1 counter, such as +1'ing changes
>>> that already have at least 1x +2, or which already approved, or which need
>>> rechecking...
>> [...]
>>
>> The behavior which baffles me, and also seems to be on the rise
>> lately, is random +1 votes on changes whose commit messages and/or
>> status clearly indicate they should not merged and do not need to be
>> reviewed. I suppose that's another an easy way to avoid the dreaded
>> "disagreements" counter?
>> -- 
>> Jeremy Stanley
> 
> 
> I have been told that some OpenStack on boarding teaches new members of the 
> community to do reviews. And they say, effectively, "muddle through as you 
> can. You won't understand it all at first, but do your best. When you're 
> done, add a +1 and move to the next one"

Oh my, I haven't heard that but I agree, such advice is very unhelpful
and unlikely to help reviewers learn good habits.

I offer up this post from my blog when asked for assistance in how to
review patches:
http://anteaya.info/blog/2013/03/21/reviewing-an-openstack-patch/

The folks who have bothered to read it and work through the steps I
outline do go on to provide useful reviews and gain confidence in their
reviewing.

Thanks John,
Anita.

> 
> I've been working to correct this when I've seen it, but +1 reviews with no 
> comments might not be people trying to game. It might simply be people trying 
> to get involved that don't know any better yet.
> 
> --John
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Dolph Mathews]

On Friday, April 8, 2016, John Dickinson  wrote:

>
>
> On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
>
> > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
> >> There are many ways to game a simple +1 counter, such as +1'ing changes
> >> that already have at least 1x +2, or which already approved, or which
> need
> >> rechecking...
> > [...]
> >
> > The behavior which baffles me, and also seems to be on the rise
> > lately, is random +1 votes on changes whose commit messages and/or
> > status clearly indicate they should not merged and do not need to be
> > reviewed. I suppose that's another an easy way to avoid the dreaded
> > "disagreements" counter?
> > --
> > Jeremy Stanley
>
>
> I have been told that some OpenStack on boarding teaches new members of
> the community to do reviews. And they say, effectively, "muddle through as
> you can. You won't understand it all at first, but do your best. When
> you're done, add a +1 and move to the next one"


I advocate for basically this, but instead of a +1, leave a +0 and ask
questions. The new reviewer will inevitably learn something and the author
will benefit by explaining their change (teaching is the best way to learn).


>
> I've been working to correct this when I've seen it, but +1 reviews with
> no comments might not be people trying to game. It might simply be people
> trying to get involved that don't know any better yet.
>
> --John
>
>
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Major Hayden]

On 04/08/2016 03:31 PM, Jeremy Stanley wrote:
> Thanks for taking this up--some people just need
> encouragement/suggestions for better ways to make an impact. On the
> other hand, if you find that many of them have addresses at the same
> company domain... well I guess we can find people higher up the
> ladder in those companies and talk to them about how to channel
> their employee quotas/incentives in more productive directions for
> the community as well.

Hey folks,

I have sent five emails so far and I received two responses already.  Both of 
the people who replied said they are new to OpenStack and how to do reviews.  
They welcomed more input on how to find the right code reviews and how to 
complete the review.  They weren't aware that these particular contributions 
were seen as unhelpful or gaming the system.

Would it make sense to encourage cores/PTLs on these projects to reach out to 
these users and share gerrit dashboard[1] links?  A PTL shared some of these 
with me and it certainly helped me focus better on the right reviews.

[1] https://github.com/openstack/gerrit-dash-creator

--
Major Hayden

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Morgan Fainberg]

On Fri, Apr 8, 2016 at 4:54 PM, Dolph Mathews 
wrote:

>
>
> On Friday, April 8, 2016, John Dickinson  wrote:
>
>>
>>
>> On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
>>
>> > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>> >> There are many ways to game a simple +1 counter, such as +1'ing changes
>> >> that already have at least 1x +2, or which already approved, or which
>> need
>> >> rechecking...
>> > [...]
>> >
>> > The behavior which baffles me, and also seems to be on the rise
>> > lately, is random +1 votes on changes whose commit messages and/or
>> > status clearly indicate they should not merged and do not need to be
>> > reviewed. I suppose that's another an easy way to avoid the dreaded
>> > "disagreements" counter?
>> > --
>> > Jeremy Stanley
>>
>>
>> I have been told that some OpenStack on boarding teaches new members of
>> the community to do reviews. And they say, effectively, "muddle through as
>> you can. You won't understand it all at first, but do your best. When
>> you're done, add a +1 and move to the next one"
>
>
> I advocate for basically this, but instead of a +1, leave a +0 and ask
> questions. The new reviewer will inevitably learn something and the author
> will benefit by explaining their change (teaching is the best way to learn).
>
>

This is exactly what I tell people to do as well! Definitely a good
direction to encourage folks to go.

>
>> I've been working to correct this when I've seen it, but +1 reviews with
>> no comments might not be people trying to game. It might simply be people
>> trying to get involved that don't know any better yet.
>>
>> --John
>>
>>
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[gordon chung]

On 08/04/2016 1:26 PM, Davanum Srinivas wrote:
> Team,
>
> Steve pointed out to a problem in Stackalytics:
> https://twitter.com/stevebot/status/718185667709267969
>
> It's pretty clear what's happening if you look here:
> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>
> Here's the drastic step (i'd like to avoid):
> https://review.openstack.org/#/c/303545/
>

is it actually affecting anything in the community aside from the 
reviews being useless. aside from the 'diversity' tags in governance, 
does anything else use stackalytics?

cheers,

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 05:10 PM, gordon chung wrote:
> 
> 
> On 08/04/2016 1:26 PM, Davanum Srinivas wrote:
>> Team,
>>
>> Steve pointed out to a problem in Stackalytics:
>> https://twitter.com/stevebot/status/718185667709267969
>>
>> It's pretty clear what's happening if you look here:
>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>
>> Here's the drastic step (i'd like to avoid):
>> https://review.openstack.org/#/c/303545/
>>
> 
> is it actually affecting anything in the community aside from the 
> reviews being useless. aside from the 'diversity' tags in governance, 
> does anything else use stackalytics?
> 
> cheers,
> 
Some company managers only look as far as stackalytics to calculate
career decisions for their group.

Sad but true.

Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Davanum Srinivas]

On Fri, Apr 8, 2016 at 5:10 PM, gordon chung  wrote:
>
>
> On 08/04/2016 1:26 PM, Davanum Srinivas wrote:
>> Team,
>>
>> Steve pointed out to a problem in Stackalytics:
>> https://twitter.com/stevebot/status/718185667709267969
>>
>> It's pretty clear what's happening if you look here:
>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>
>> Here's the drastic step (i'd like to avoid):
>> https://review.openstack.org/#/c/303545/
>>
>
> is it actually affecting anything in the community aside from the
> reviews being useless. aside from the 'diversity' tags in governance,
> does anything else use stackalytics?

Gordon,

I feel that we are missing an opportunity to teaching what we want new
folks to do! As a group we should all try to spot these patterns and
make sure everyone's efforts are fruitful.

To that effect, i am capturing stuff here:
https://davanum.wordpress.com/2016/04/08/new-to-openstack-reviews-start-here/

Thanks,
Dims

> cheers,
>
> --
> gord
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[gordon chung]

On 08/04/2016 5:23 PM, Davanum Srinivas wrote:
> On Fri, Apr 8, 2016 at 5:10 PM, gordon chung  wrote:
>>
>>
>> is it actually affecting anything in the community aside from the
>> reviews being useless. aside from the 'diversity' tags in governance,
>> does anything else use stackalytics?
>
> Gordon,
>
> I feel that we are missing an opportunity to teaching what we want new
> folks to do! As a group we should all try to spot these patterns and
> make sure everyone's efforts are fruitful.
>
> To that effect, i am capturing stuff here:
> https://davanum.wordpress.com/2016/04/08/new-to-openstack-reviews-start-here/
>
> Thanks,
> Dims
>

i imagine the fact they're gaming the stackalytics system means they're 
aware of what their doing and no one has called them out on it yet. i'm 
a glass half empty individual :)

this has been discussed in some form already in the past[1] and it'll 
probably keep happening. i get the feeling if you tell some to stop 
putting useless reviews in one place, they'll do it somewhere else or at 
random frequencies -- you may be endlessly chasing white noise. if it 
only affects lazy managers using stackalytics then it's probably not a 
big deal for now? i can't imagine the current useless reviews are 
swaying the overall stats of a project (aside from top global reviewers)

i like anteaya/your posts, hopefully it helps.

[1] source: i'm too lazy to search list

cheers,

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Anita Kuno]

On 04/08/2016 05:53 PM, gordon chung wrote:
> 
> 
> On 08/04/2016 5:23 PM, Davanum Srinivas wrote:
>> On Fri, Apr 8, 2016 at 5:10 PM, gordon chung  wrote:
>>>
>>>
>>> is it actually affecting anything in the community aside from the
>>> reviews being useless. aside from the 'diversity' tags in governance,
>>> does anything else use stackalytics?
>>
>> Gordon,
>>
>> I feel that we are missing an opportunity to teaching what we want new
>> folks to do! As a group we should all try to spot these patterns and
>> make sure everyone's efforts are fruitful.
>>
>> To that effect, i am capturing stuff here:
>> https://davanum.wordpress.com/2016/04/08/new-to-openstack-reviews-start-here/
>>
>> Thanks,
>> Dims
>>
> 
> i imagine the fact they're gaming the stackalytics system means they're 
> aware of what their doing and no one has called them out on it yet. i'm 
> a glass half empty individual :)

Well Major reports that in at least 2 instances those contacted have
replied that they were new to OpenStack and didn't realize that their
behaviour was considered gaming. I'm not going to extrapolate to the
whole group based on that sample size but it appears there are at least
some folks who self-identify as being in the didn't know category.
http://lists.openstack.org/pipermail/openstack-dev/2016-April/091822.html

> 
> this has been discussed in some form already in the past[1] and it'll 
> probably keep happening. i get the feeling if you tell some to stop 
> putting useless reviews in one place, they'll do it somewhere else or at 
> random frequencies -- you may be endlessly chasing white noise. if it 
> only affects lazy managers using stackalytics then it's probably not a 
> big deal for now? i can't imagine the current useless reviews are 
> swaying the overall stats of a project (aside from top global reviewers)
> 
> i like anteaya/your posts, hopefully it helps.

Thank you Gord that is kind of you to say, I appreciate your feedback. I
hope it helps as well.

Thank you,
Anita.

> 
> [1] source: i'm too lazy to search list
> 
> cheers,
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Jim Meyer]

On Apr 8, 2016, at 1:57 PM, Major Hayden  wrote:
> 
> On 04/08/2016 03:31 PM, Jeremy Stanley wrote:
>> Thanks for taking this up--some people just need
>> encouragement/suggestions for better ways to make an impact. On the
>> other hand, if you find that many of them have addresses at the same
>> company domain... well I guess we can find people higher up the
>> ladder in those companies and talk to them about how to channel
>> their employee quotas/incentives in more productive directions for
>> the community as well.
> 
> Hey folks,
> 
> I have sent five emails so far and I received two responses already.  Both of 
> the people who replied said they are new to OpenStack and how to do reviews.  
> They welcomed more input on how to find the right code reviews and how to 
> complete the review.  They weren't aware that these particular contributions 
> were seen as unhelpful or gaming the system.

First, thanks Major for reaching out and asking questions. 

Having just caught this thread, I was a bit startled by the assumption of 
ill-intent vs. a straight-up “Hey, this seems silly. Why are they doing this? 
We should ask!” That’s a bummer attitude for us to launch out with.

I’m glad to see a few folks approach it as, “They probably don’t know” and 
offer blog posts. Might be good to get some of these incorporated into 
https://wiki.openstack.org/wiki/How_To_Contribute 
 so that n00bs (like me) get 
guidance early.

I can’t speak for others, but I can say I’ve made it abundantly clear in my org 
that Stackalytics isn’t a useful or interesting metric beyond knowing where 
we’re contributing (and not). As a leaderboard, it’s untrustworthy because 
there’s no guarantee that any contribution it tracks is making value for 
OpenStack operators or users. I suspect that we’re reflecting on attitudes 
past, but I could be wrong.

> Would it make sense to encourage cores/PTLs on these projects to reach out to 
> these users and share gerrit dashboard[1] links?  A PTL shared some of these 
> with me and it certainly helped me focus better on the right reviews.
> 
> [1] https://github.com/openstack/gerrit-dash-creator 
> 

That seems smart in general. Most of the projects have a getting started page; 
I like the Nova one[1] because it calls out reviews as a good way to start 
familiarizing yourself with the code, and points out their code review guide[2].

Always good to see you, Major. Big fan of your work. =]

—j

[1] http://docs.openstack.org/developer/nova/how_to_get_involved.html 

[2] http://docs.openstack.org/developer/nova/code-review.html#code-review 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Nikhil Komawar]

Hi,

Steve, thanks for pointing that out and Dims, thanks for starting the
discussion.

I guess I feel that the drastic step is/may not be necessary. Here's the
reason why: we're trying to solve a subjective problem with a objective
solution. All systems have loopholes and there will be people who will
try to take advantage of them, so we should into look into more
contextual info while forming opinions.

To say this in more "in practice terms" we are disallowing counting
stats for certain specific events, although there could be significant
number of those +1s that do matter. An extreme case of this being, a
downstream openstack consumer/cloud operator appointing someone to take
a look at the ongoing efforts upstream on the requirements and packages;
report back on some of the 'internally beware' changes. It would be a
loss for the management to track info on such individuals. To lesser
extreme, if I've to ask someone to take another look at requirements
changes and make sure that the project changes are appropriate that
potentially conflict with the updates, such individuals might be
demotivated to pick such jobs -- specifically stable and release
liaisons, sometime cross project efforts. I think we need to value such
work and give a way to (first) the individuals to keep themselves
motivated and then management to keep a check.

Hence, this is a subjective problem, it applies to some cases and
doesn't to others; the info is valuable to have but needs to be consumed
correctly. On top of that, I think a general rule of statistics is that
-- the more info/large sample set you have, more accurate are the
results. How and where you need to read them is we should solve. And I
think there are a few today who avoid such speculative results, for ex.
quarterly results at your resp. orgs, aren't you interested, do they
always tell story of the value addition/subtraction by the org as a
whole? Yet they are important, to keep us moving and keep us motivated!

Hence, my proposal is:

* instead of completely ignoring the stats on such reviews, we either
ignore or not ignore them on "generic" +1s
* introduce a new more-info-like tab/UI-stuff in Stackalytics and keep
those stats there, consequently we need to modify the Stackalytics
processor to show that info there
* encourage the teams to carefully read the review stats, say % of - vs.
+ and be more subjective on the evaluation by browsing some of the
reviews (TBH, I know that +0s are sometimes the best feedback on
reviews). I think this is a bit easier for me to say because I'm
primarily looking from Glance perspective which is a relatively small
team and we happen to stumble upon each others' reviews often.

In the interest of keeping the community inclusive, collaborative and
healthy,

yours sincerely,

On 4/8/16 1:26 PM, Davanum Srinivas wrote:
> Team,
>
> Steve pointed out to a problem in Stackalytics:
> https://twitter.com/stevebot/status/718185667709267969
>
> It's pretty clear what's happening if you look here:
> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>
> Here's the drastic step (i'd like to avoid):
> https://review.openstack.org/#/c/303545/
>
> What do you think?
>
> Thanks,
> Dims
>
>
>


-- 

Thanks,
Nikhil



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Jay Faulkner]

I know a lot of folks explicitly avoid a +0 vote with a comment because you 
don't get "credit" for it in statistics. Whether or not that should matter is 
another discussion, but there is a significant disincentive to no-voting right 
now.


-

Jay Faulkner



From: Dolph Mathews 
Sent: Friday, April 8, 2016 1:54 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats



On Friday, April 8, 2016, John Dickinson mailto:m...@not.mn>> 
wrote:


On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:

> On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>> There are many ways to game a simple +1 counter, such as +1'ing changes
>> that already have at least 1x +2, or which already approved, or which need
>> rechecking...
> [...]
>
> The behavior which baffles me, and also seems to be on the rise
> lately, is random +1 votes on changes whose commit messages and/or
> status clearly indicate they should not merged and do not need to be
> reviewed. I suppose that's another an easy way to avoid the dreaded
> "disagreements" counter?
> --
> Jeremy Stanley


I have been told that some OpenStack on boarding teaches new members of the 
community to do reviews. And they say, effectively, "muddle through as you can. 
You won't understand it all at first, but do your best. When you're done, add a 
+1 and move to the next one"

I advocate for basically this, but instead of a +1, leave a +0 and ask 
questions. The new reviewer will inevitably learn something and the author will 
benefit by explaining their change (teaching is the best way to learn).


I've been working to correct this when I've seen it, but +1 reviews with no 
comments might not be people trying to game. It might simply be people trying 
to get involved that don't know any better yet.

--John



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Tom Barron]

On 04/08/2016 05:16 PM, Anita Kuno wrote:
> On 04/08/2016 05:10 PM, gordon chung wrote:
>>
>>
>> On 08/04/2016 1:26 PM, Davanum Srinivas wrote:
>>> Team,
>>>
>>> Steve pointed out to a problem in Stackalytics:
>>> https://twitter.com/stevebot/status/718185667709267969
>>>
>>> It's pretty clear what's happening if you look here:
>>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>>
>>> Here's the drastic step (i'd like to avoid):
>>> https://review.openstack.org/#/c/303545/
>>>
>>
>> is it actually affecting anything in the community aside from the 
>> reviews being useless. aside from the 'diversity' tags in governance, 
>> does anything else use stackalytics?
>>
>> cheers,
>>
> Some company managers only look as far as stackalytics to calculate
> career decisions for their group.
> 
> Sad but true.
> 

Certainly true, and sad, but "some" is a quantifier that starts at
greater than zero and runs on up from there :-)

IMO we non-managers also have to take responsibility for a system that
uses quantitative measures that "gamify" OpenStack performance.  Humans
respond to this kind of reward system.  They aren't (for that reason at
least) evil and the effect is not all that surprising.  The harder
question is whether there is a better way to set things up, or more to
the point, since many of us are sure there is, exactly what it is.

-- Tom

> Anita.
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Diana Clarke]

On Fri, Apr 8, 2016 at 5:23 PM, Davanum Srinivas  wrote:
> To that effect, i am capturing stuff here:
> https://davanum.wordpress.com/2016/04/08/new-to-openstack-reviews-start-here/

Here are a few more links for your list, Dims. I remember especially
liking Sean's blog post, and I think of it often when I start feeling
discouraged about review turnaround time.

  https://dague.net/2014/02/28/why-you-should-be-reviewing-more-openstack-code/

  
http://docs.openstack.org/developer/nova/how_to_get_involved.html#why-do-code-reviews-if-i-am-not-in-nova-core

  http://docs.openstack.org/infra/manual/developers.html#peer-review

Have a great weekend, folks!

--diana

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Davanum Srinivas]

On Fri, Apr 8, 2016 at 11:39 PM, Diana Clarke
 wrote:
> On Fri, Apr 8, 2016 at 5:23 PM, Davanum Srinivas  wrote:
>> To that effect, i am capturing stuff here:
>> https://davanum.wordpress.com/2016/04/08/new-to-openstack-reviews-start-here/
>
> Here are a few more links for your list, Dims. I remember especially
> liking Sean's blog post, and I think of it often when I start feeling
> discouraged about review turnaround time.
>
>   
> https://dague.net/2014/02/28/why-you-should-be-reviewing-more-openstack-code/
>
>   
> http://docs.openstack.org/developer/nova/how_to_get_involved.html#why-do-code-reviews-if-i-am-not-in-nova-core
>
>   http://docs.openstack.org/infra/manual/developers.html#peer-review

Thanks Diana, Added to my list here:
https://davanum.wordpress.com/2016/04/08/new-to-openstack-reviews-start-here/

> Have a great weekend, folks!
>
> --diana
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Matt Riedemann]

On 4/8/2016 5:54 PM, Jay Faulkner wrote:

I know a lot of folks explicitly avoid a +0 vote with a comment because
you don't get "credit" for it in statistics. Whether or not that should
matter is another discussion, but there is a significant disincentive to
no-voting right now.


-

Jay Faulkner




*From:* Dolph Mathews 
*Sent:* Friday, April 8, 2016 1:54 PM
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
Stackalytics stats


On Friday, April 8, 2016, John Dickinson mailto:m...@not.mn>>
wrote:



On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:

 > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
 >> There are many ways to game a simple +1 counter, such as +1'ing
changes
 >> that already have at least 1x +2, or which already approved, or
which need
 >> rechecking...
 > [...]
 >
 > The behavior which baffles me, and also seems to be on the rise
 > lately, is random +1 votes on changes whose commit messages and/or
 > status clearly indicate they should not merged and do not need to be
 > reviewed. I suppose that's another an easy way to avoid the dreaded
 > "disagreements" counter?
 > --
 > Jeremy Stanley


I have been told that some OpenStack on boarding teaches new members
of the community to do reviews. And they say, effectively, "muddle
through as you can. You won't understand it all at first, but do
your best. When you're done, add a +1 and move to the next one"


I advocate for basically this, but instead of a +1, leave a +0 and ask
questions. The new reviewer will inevitably learn something and the
author will benefit by explaining their change (teaching is the best way
to learn).


I've been working to correct this when I've seen it, but +1 reviews
with no comments might not be people trying to game. It might simply
be people trying to get involved that don't know any better yet.

--John





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



There is also disincentive in +1ing a change that you don't understand 
and is wrong and then a core comes along and -1s it (you get dinged for 
the disagreement). And there is disincentive in -1ing a change for the 
wrong reasons (silly nits or asking questions for understanding). I ask 
a lot of questions in a lot of changes and I don't vote on those because 
it would be inappropriate.


I also notice when "newcomers" are asking good questions for 
understanding and not voting on them, it shows me they are trying to 
learn and are getting invested in the project, not just trying to pad 
stats. Those are the people we look to mentor into bigger roles in the 
project team, be that working on subteams or eventually looking at for 
the core reviewer team.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Ken'ichi Ohmichi]

2016/04/08 10:55、Anita Kuno  :

>> On 04/08/2016 01:42 PM, Dmitry Tantsur wrote:
>> 2016-04-08 19:26 GMT+02:00 Davanum Srinivas :
>> 
>>> Team,
>>> 
>>> Steve pointed out to a problem in Stackalytics:
>>> https://twitter.com/stevebot/status/718185667709267969
>> 
>> 
>> There are many ways to game a simple +1 counter, such as +1'ing changes
>> that already have at least 1x +2, or which already approved, or which need
>> rechecking...

Can we merge this kind of patches without reviews?
When seeing this kind of patches, I check all jobs are succeeded. Sometimes 
some job failed, I check the reason and +2 if that is unrelated.

This workflow seems possible to be implemented automatically.
Or bad idea?

Thanks








>> 
>> 
>>> 
>>> 
>>> It's pretty clear what's happening if you look here:
>>> 
>>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>> 
>>> Here's the drastic step (i'd like to avoid):
>>> https://review.openstack.org/#/c/303545/
>>> 
>>> What do you think?
>> 
>> One more possible (though also imperfect) mitigation is to make exception
>> from the usual 2x +2 rule for requirements updates passing gates and use
>> only 1x +2. Then requirements reviews will take substantially less time to
>> land, reducing need/possibility of having such +1's.
> 
> Proposal bot patches merge in many cases with 1 +2 already.
> 
> Have you looked at the timing of the bot patches generated and the first
> +1's? If not, take a look at that.
> 
> I don't think we should be expecting core reviewers to schedule
> approving bot proposals to prevent extraneous +1s.
> 
> Thanks,
> Anita.
> 
>> 
>>> 
>>> Thanks,
>>> Dims
>>> 
>>> --
>>> Davanum Srinivas :: https://twitter.com/dims
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>> 
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Morgan Fainberg]

On Apr 9, 2016 12:05, "Ken'ichi Ohmichi"  wrote:
>
>
> 2016/04/08 10:55、Anita Kuno  :
>
> >> On 04/08/2016 01:42 PM, Dmitry Tantsur wrote:
> >> 2016-04-08 19:26 GMT+02:00 Davanum Srinivas :
> >>
> >>> Team,
> >>>
> >>> Steve pointed out to a problem in Stackalytics:
> >>> https://twitter.com/stevebot/status/718185667709267969
> >>
> >>
> >> There are many ways to game a simple +1 counter, such as +1'ing changes
> >> that already have at least 1x +2, or which already approved, or which
need
> >> rechecking...
>
> Can we merge this kind of patches without reviews?
> When seeing this kind of patches, I check all jobs are succeeded.
Sometimes some job failed, I check the reason and +2 if that is unrelated.
>
> This workflow seems possible to be implemented automatically.
> Or bad idea?
>
> Thanks
>

A true automerge has potential to accidentally clobber things in an ugly
way if it goes wrong. The auto propose but core approval still has a level
of human spot checking. I would personally be uncomfortable with the bot
automatically merging  things. At face value the overhead of a core
approval and possibility of what is highlighted in this thread is better
IMHO.

>
>
>
>
>
>
>
> >>
> >>
> >>>
> >>>
> >>> It's pretty clear what's happening if you look here:
> >>>
> >>>
https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
> >>>
> >>> Here's the drastic step (i'd like to avoid):
> >>> https://review.openstack.org/#/c/303545/
> >>>
> >>> What do you think?
> >>
> >> One more possible (though also imperfect) mitigation is to make
exception
> >> from the usual 2x +2 rule for requirements updates passing gates and
use
> >> only 1x +2. Then requirements reviews will take substantially less
time to
> >> land, reducing need/possibility of having such +1's.
> >
> > Proposal bot patches merge in many cases with 1 +2 already.
> >
> > Have you looked at the timing of the bot patches generated and the first
> > +1's? If not, take a look at that.
> >
> > I don't think we should be expecting core reviewers to schedule
> > approving bot proposals to prevent extraneous +1s.
> >
> > Thanks,
> > Anita.
> >
> >>
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> --
> >>> Davanum Srinivas :: https://twitter.com/dims
> >>>
> >>>
__
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>
> >>
> >>
> >>
__
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
__
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Adam Lawson]

I have a quick question:
How is anyone hurt out harmed by the practice? I agree it isn't helpful.
But it isn't harming either. It could be gaming and it could be ignorance -
mistakes by not knowing.

I'm asking because I see the same predictable personalities making passive
aggressive accusations against their peers about gaming and ill intent.
When folks are approached, they realize it is due to not knowing. Their
answers are received with suspicion and presumptions that they are in the
minority.

We really need to stop rushing to embrace the worst in each other.

//adam
On Apr 9, 2016 1:18 PM, "Morgan Fainberg"  wrote:


On Apr 9, 2016 12:05, "Ken'ichi Ohmichi"  wrote:
>
>
> 2016/04/08 10:55、Anita Kuno  :
>
> >> On 04/08/2016 01:42 PM, Dmitry Tantsur wrote:
> >> 2016-04-08 19:26 GMT+02:00 Davanum Srinivas :
> >>
> >>> Team,
> >>>
> >>> Steve pointed out to a problem in Stackalytics:
> >>> https://twitter.com/stevebot/status/718185667709267969
> >>
> >>
> >> There are many ways to game a simple +1 counter, such as +1'ing changes
> >> that already have at least 1x +2, or which already approved, or which
need
> >> rechecking...
>
> Can we merge this kind of patches without reviews?
> When seeing this kind of patches, I check all jobs are succeeded.
Sometimes some job failed, I check the reason and +2 if that is unrelated.
>
> This workflow seems possible to be implemented automatically.
> Or bad idea?
>
> Thanks
>

A true automerge has potential to accidentally clobber things in an ugly
way if it goes wrong. The auto propose but core approval still has a level
of human spot checking. I would personally be uncomfortable with the bot
automatically merging  things. At face value the overhead of a core
approval and possibility of what is highlighted in this thread is better
IMHO.

>
>
>
>
>
>
>
> >>
> >>
> >>>
> >>>
> >>> It's pretty clear what's happening if you look here:
> >>>
> >>>
https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
> >>>
> >>> Here's the drastic step (i'd like to avoid):
> >>> https://review.openstack.org/#/c/303545/
> >>>
> >>> What do you think?
> >>
> >> One more possible (though also imperfect) mitigation is to make
exception
> >> from the usual 2x +2 rule for requirements updates passing gates and
use
> >> only 1x +2. Then requirements reviews will take substantially less
time to
> >> land, reducing need/possibility of having such +1's.
> >
> > Proposal bot patches merge in many cases with 1 +2 already.
> >
> > Have you looked at the timing of the bot patches generated and the first
> > +1's? If not, take a look at that.
> >
> > I don't think we should be expecting core reviewers to schedule
> > approving bot proposals to prevent extraneous +1s.
> >
> > Thanks,
> > Anita.
> >
> >>
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> --
> >>> Davanum Srinivas :: https://twitter.com/dims
> >>>
> >>>
__
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>
> >>
> >>
> >>
__
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
__
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Amrith Kumar]

Thanks to Dims and Steve for bringing this up.

It has long been my opinion that +0's are invaluable for the question 
asking, and for getting to understand software, and unfortunately +0's are lost 
in the noise. So a while ago, I posted to the ML [1] asking about making +0's 
more visible. I signed up to submit a request on gerrit upstream (and promptly 
forgot to do that). This mail thread has reminded me of that. I have now posted 
a request for the upstream gerrit folks to fix [2].

I believe that people don't use +0's enough because they often get 
ignored. I know that one can be cynical and say it is because it gives one no 
credit in stackalytics; I choose not to be that person.

I post +0's a lot. But, I find that they are often ignored. If you 
agree with me that +0's are useful, and could be highlighted better in the 
gerrit review screen, please post a comment on [2].

Thanks,

-amrith

[1] http://openstack.markmail.org/thread/nj4onttaibjmfxew
[2] https://code.google.com/p/gerrit/issues/detail?id=4050

> -Original Message-
> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> Sent: Saturday, April 09, 2016 9:43 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics
> stats
> 
> 
> 
> On 4/8/2016 5:54 PM, Jay Faulkner wrote:
> > I know a lot of folks explicitly avoid a +0 vote with a comment
> > because you don't get "credit" for it in statistics. Whether or not
> > that should matter is another discussion, but there is a significant
> > disincentive to no-voting right now.
> >
> >
> > -
> >
> > Jay Faulkner
> >
> >
> >
> > --
> > --
> > *From:* Dolph Mathews 
> > *Sent:* Friday, April 8, 2016 1:54 PM
> > *To:* OpenStack Development Mailing List (not for usage questions)
> > *Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
> > Stackalytics stats
> >
> >
> > On Friday, April 8, 2016, John Dickinson  > <mailto:m...@not.mn>>
> > wrote:
> >
> >
> >
> > On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
> >
> >  > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
> >  >> There are many ways to game a simple +1 counter, such as +1'ing
> > changes
> >  >> that already have at least 1x +2, or which already approved, or
> > which need
> >  >> rechecking...
> >  > [...]
> >  >
> >  > The behavior which baffles me, and also seems to be on the rise
> >  > lately, is random +1 votes on changes whose commit messages
> and/or
> >  > status clearly indicate they should not merged and do not need to
> be
> >  > reviewed. I suppose that's another an easy way to avoid the
> dreaded
> >  > "disagreements" counter?
> >  > --
> >  > Jeremy Stanley
> >
> >
> > I have been told that some OpenStack on boarding teaches new members
> > of the community to do reviews. And they say, effectively, "muddle
> > through as you can. You won't understand it all at first, but do
> > your best. When you're done, add a +1 and move to the next one"
> >
> >
> > I advocate for basically this, but instead of a +1, leave a +0 and ask
> > questions. The new reviewer will inevitably learn something and the
> > author will benefit by explaining their change (teaching is the best
> > way to learn).
> >
> >
> > I've been working to correct this when I've seen it, but +1 reviews
> > with no comments might not be people trying to game. It might simply
> > be people trying to get involved that don't know any better yet.
> >
> > --John
> >
> >
> >
> >
> >
> > __
> >  OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> There is also disincentive in +1ing a change that you don't understand and
> is wrong and then a core comes along and -1s it (you get dinged for the
> disagreement). And there is disincentive in -1ing a change for the wrong
> reasons (silly nits or asking questions for understanding). I ask a lot of
> questions in a lot of changes and I don't vote 

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Nikhil Komawar]

Thanks Amrith!

I am a big supporter on including +0s.

On 4/9/16 6:31 PM, Amrith Kumar wrote:
> Thanks to Dims and Steve for bringing this up.
>
>   It has long been my opinion that +0's are invaluable for the question 
> asking, and for getting to understand software, and unfortunately +0's are 
> lost in the noise. So a while ago, I posted to the ML [1] asking about making 
> +0's more visible. I signed up to submit a request on gerrit upstream (and 
> promptly forgot to do that). This mail thread has reminded me of that. I have 
> now posted a request for the upstream gerrit folks to fix [2].
>
>   I believe that people don't use +0's enough because they often get 
> ignored. I know that one can be cynical and say it is because it gives one no 
> credit in stackalytics; I choose not to be that person.
>
>   I post +0's a lot. But, I find that they are often ignored. If you 
> agree with me that +0's are useful, and could be highlighted better in the 
> gerrit review screen, please post a comment on [2].
>
> Thanks,
>
> -amrith
>
> [1] http://openstack.markmail.org/thread/nj4onttaibjmfxew
> [2] https://code.google.com/p/gerrit/issues/detail?id=4050
>
>> -Original Message-
>> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
>> Sent: Saturday, April 09, 2016 9:43 AM
>> To: openstack-dev@lists.openstack.org
>> Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics
>> stats
>>
>>
>>
>> On 4/8/2016 5:54 PM, Jay Faulkner wrote:
>>> I know a lot of folks explicitly avoid a +0 vote with a comment
>>> because you don't get "credit" for it in statistics. Whether or not
>>> that should matter is another discussion, but there is a significant
>>> disincentive to no-voting right now.
>>>
>>>
>>> -
>>>
>>> Jay Faulkner
>>>
>>>
>>>
>>> ------
>>> --
>>> *From:* Dolph Mathews 
>>> *Sent:* Friday, April 8, 2016 1:54 PM
>>> *To:* OpenStack Development Mailing List (not for usage questions)
>>> *Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
>>> Stackalytics stats
>>>
>>>
>>> On Friday, April 8, 2016, John Dickinson >> <mailto:m...@not.mn>>
>>> wrote:
>>>
>>>
>>>
>>> On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
>>>
>>>  > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>>>  >> There are many ways to game a simple +1 counter, such as +1'ing
>>> changes
>>>  >> that already have at least 1x +2, or which already approved, or
>>> which need
>>>  >> rechecking...
>>>  > [...]
>>>  >
>>>  > The behavior which baffles me, and also seems to be on the rise
>>>  > lately, is random +1 votes on changes whose commit messages
>> and/or
>>>  > status clearly indicate they should not merged and do not need to
>> be
>>>  > reviewed. I suppose that's another an easy way to avoid the
>> dreaded
>>>  > "disagreements" counter?
>>>  > --
>>>  > Jeremy Stanley
>>>
>>>
>>> I have been told that some OpenStack on boarding teaches new members
>>> of the community to do reviews. And they say, effectively, "muddle
>>> through as you can. You won't understand it all at first, but do
>>> your best. When you're done, add a +1 and move to the next one"
>>>
>>>
>>> I advocate for basically this, but instead of a +1, leave a +0 and ask
>>> questions. The new reviewer will inevitably learn something and the
>>> author will benefit by explaining their change (teaching is the best
>>> way to learn).
>>>
>>>
>>> I've been working to correct this when I've seen it, but +1 reviews
>>> with no comments might not be people trying to game. It might simply
>>> be people trying to get involved that don't know any better yet.
>>>
>>> --John
>>>
>>>
>>>
>>>
>>>
>>> __
>>>  OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Joshua Harlow]

+1 from me also,

I also use +0 for question asking and the like, because IMHO that's not 
what -1 are for. As for myself losing stackalytics stats when *I* do 
this (ie using +0 instead of -1), meh, I got better things in my life to 
think/care about :-P


-Josh

Nikhil Komawar wrote:

Thanks Amrith!

I am a big supporter on including +0s.

On 4/9/16 6:31 PM, Amrith Kumar wrote:

Thanks to Dims and Steve for bringing this up.

It has long been my opinion that +0's are invaluable for the question 
asking, and for getting to understand software, and unfortunately +0's are lost 
in the noise. So a while ago, I posted to the ML [1] asking about making +0's 
more visible. I signed up to submit a request on gerrit upstream (and promptly 
forgot to do that). This mail thread has reminded me of that. I have now posted 
a request for the upstream gerrit folks to fix [2].

I believe that people don't use +0's enough because they often get 
ignored. I know that one can be cynical and say it is because it gives one no 
credit in stackalytics; I choose not to be that person.

I post +0's a lot. But, I find that they are often ignored. If you 
agree with me that +0's are useful, and could be highlighted better in the 
gerrit review screen, please post a comment on [2].

Thanks,

-amrith

[1] http://openstack.markmail.org/thread/nj4onttaibjmfxew
[2] https://code.google.com/p/gerrit/issues/detail?id=4050


-Original Message-
From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
Sent: Saturday, April 09, 2016 9:43 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics
stats



On 4/8/2016 5:54 PM, Jay Faulkner wrote:

I know a lot of folks explicitly avoid a +0 vote with a comment
because you don't get "credit" for it in statistics. Whether or not
that should matter is another discussion, but there is a significant
disincentive to no-voting right now.


-

Jay Faulkner



--
--
*From:* Dolph Mathews
*Sent:* Friday, April 8, 2016 1:54 PM
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
Stackalytics stats


On Friday, April 8, 2016, John Dickinsonmailto:m...@not.mn>>
wrote:



 On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:

  >  On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
  >>  There are many ways to game a simple +1 counter, such as +1'ing
 changes
  >>  that already have at least 1x +2, or which already approved, or
 which need
  >>  rechecking...
  >  [...]
  >
  >  The behavior which baffles me, and also seems to be on the rise
  >  lately, is random +1 votes on changes whose commit messages

and/or

  >  status clearly indicate they should not merged and do not need to

be

  >  reviewed. I suppose that's another an easy way to avoid the

dreaded

  >  "disagreements" counter?
  >  --
  >  Jeremy Stanley


 I have been told that some OpenStack on boarding teaches new members
 of the community to do reviews. And they say, effectively, "muddle
 through as you can. You won't understand it all at first, but do
 your best. When you're done, add a +1 and move to the next one"


I advocate for basically this, but instead of a +1, leave a +0 and ask
questions. The new reviewer will inevitably learn something and the
author will benefit by explaining their change (teaching is the best
way to learn).


 I've been working to correct this when I've seen it, but +1 reviews
 with no comments might not be people trying to game. It might simply
 be people trying to get involved that don't know any better yet.

 --John





__
 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


There is also disincentive in +1ing a change that you don't understand and
is wrong and then a core comes along and -1s it (you get dinged for the
disagreement). And there is disincentive in -1ing a change for the wrong
reasons (silly nits or asking questions for understanding). I ask a lot of
questions in a lot of changes and I don't vote on those because it would
be inappropriate.

I also notice when "newcomers" are asking good questions for understanding
and not voting on them, it shows me they are trying to learn and are
getting invested in the project, not just trying to pad stats. Those are
the people we look to mentor into bigger roles in the project team, be
that working on subteams or eventually look

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Ricardo Carrillo Cruz]

++, exactly my thoughts.

I believe most people don't vote 0 for questions about the change because
'it won't count'.
That's really bad, IMHO 0 should be for asking things not clear in the
patch, whereas a -1 should be a 'this code here is not right'.

I believe the 0s should be tracked somehow.

Regards

2016-04-10 18:08 GMT+02:00 Joshua Harlow :

> +1 from me also,
>
> I also use +0 for question asking and the like, because IMHO that's not
> what -1 are for. As for myself losing stackalytics stats when *I* do this
> (ie using +0 instead of -1), meh, I got better things in my life to
> think/care about :-P
>
> -Josh
>
>
> Nikhil Komawar wrote:
>
>> Thanks Amrith!
>>
>> I am a big supporter on including +0s.
>>
>> On 4/9/16 6:31 PM, Amrith Kumar wrote:
>>
>>> Thanks to Dims and Steve for bringing this up.
>>>
>>> It has long been my opinion that +0's are invaluable for the
>>> question asking, and for getting to understand software, and unfortunately
>>> +0's are lost in the noise. So a while ago, I posted to the ML [1] asking
>>> about making +0's more visible. I signed up to submit a request on gerrit
>>> upstream (and promptly forgot to do that). This mail thread has reminded me
>>> of that. I have now posted a request for the upstream gerrit folks to fix
>>> [2].
>>>
>>> I believe that people don't use +0's enough because they often
>>> get ignored. I know that one can be cynical and say it is because it gives
>>> one no credit in stackalytics; I choose not to be that person.
>>>
>>> I post +0's a lot. But, I find that they are often ignored. If
>>> you agree with me that +0's are useful, and could be highlighted better in
>>> the gerrit review screen, please post a comment on [2].
>>>
>>> Thanks,
>>>
>>> -amrith
>>>
>>> [1] http://openstack.markmail.org/thread/nj4onttaibjmfxew
>>> [2] https://code.google.com/p/gerrit/issues/detail?id=4050
>>>
>>> -Original Message-
>>>> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
>>>> Sent: Saturday, April 09, 2016 9:43 AM
>>>> To: openstack-dev@lists.openstack.org
>>>> Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics
>>>> stats
>>>>
>>>>
>>>>
>>>> On 4/8/2016 5:54 PM, Jay Faulkner wrote:
>>>>
>>>>> I know a lot of folks explicitly avoid a +0 vote with a comment
>>>>> because you don't get "credit" for it in statistics. Whether or not
>>>>> that should matter is another discussion, but there is a significant
>>>>> disincentive to no-voting right now.
>>>>>
>>>>>
>>>>> -
>>>>>
>>>>> Jay Faulkner
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> *From:* Dolph Mathews
>>>>> *Sent:* Friday, April 8, 2016 1:54 PM
>>>>> *To:* OpenStack Development Mailing List (not for usage questions)
>>>>> *Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
>>>>> Stackalytics stats
>>>>>
>>>>>
>>>>> On Friday, April 8, 2016, John Dickinson>>>> <mailto:m...@not.mn>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>  On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
>>>>>
>>>>>   >  On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>>>>>   >>  There are many ways to game a simple +1 counter, such as
>>>>> +1'ing
>>>>>  changes
>>>>>   >>  that already have at least 1x +2, or which already approved,
>>>>> or
>>>>>  which need
>>>>>   >>  rechecking...
>>>>>   >  [...]
>>>>>   >
>>>>>   >  The behavior which baffles me, and also seems to be on the
>>>>> rise
>>>>>   >  lately, is random +1 votes on changes whose commit messages
>>>>>
>>>> and/or
>>>>
>>>>>   >  status clearly indicate they should not merged and do not
>>>>> need to
>>>>>
>>>&g

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Amrith Kumar]

Thanks Nikhil, appreciate the comment on the gerrit bug. 

-amrith

> -Original Message-
> From: Nikhil Komawar [mailto:nik.koma...@gmail.com]
> Sent: Sunday, April 10, 2016 10:27 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics
> stats
> 
> Thanks Amrith!
> 
> I am a big supporter on including +0s.
> 
> On 4/9/16 6:31 PM, Amrith Kumar wrote:
> > Thanks to Dims and Steve for bringing this up.
> >
> > It has long been my opinion that +0's are invaluable for the
> question asking, and for getting to understand software, and unfortunately
> +0's are lost in the noise. So a while ago, I posted to the ML [1] asking
> about making +0's more visible. I signed up to submit a request on gerrit
> upstream (and promptly forgot to do that). This mail thread has reminded
> me of that. I have now posted a request for the upstream gerrit folks to
> fix [2].
> >
> > I believe that people don't use +0's enough because they often get
> ignored. I know that one can be cynical and say it is because it gives one
> no credit in stackalytics; I choose not to be that person.
> >
> > I post +0's a lot. But, I find that they are often ignored. If you
> agree with me that +0's are useful, and could be highlighted better in the
> gerrit review screen, please post a comment on [2].
> >
> > Thanks,
> >
> > -amrith
> >
> > [1] http://openstack.markmail.org/thread/nj4onttaibjmfxew
> > [2] https://code.google.com/p/gerrit/issues/detail?id=4050
> >
> >> -Original Message-
> >> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> >> Sent: Saturday, April 09, 2016 9:43 AM
> >> To: openstack-dev@lists.openstack.org
> >> Subject: Re: [openstack-dev] [all][stackalytics] Gaming the
> >> Stackalytics stats
> >>
> >>
> >>
> >> On 4/8/2016 5:54 PM, Jay Faulkner wrote:
> >>> I know a lot of folks explicitly avoid a +0 vote with a comment
> >>> because you don't get "credit" for it in statistics. Whether or not
> >>> that should matter is another discussion, but there is a significant
> >>> disincentive to no-voting right now.
> >>>
> >>>
> >>> -
> >>>
> >>> Jay Faulkner
> >>>
> >>>
> >>>
> >>> 
> >>> --
> >>> --
> >>> *From:* Dolph Mathews 
> >>> *Sent:* Friday, April 8, 2016 1:54 PM
> >>> *To:* OpenStack Development Mailing List (not for usage questions)
> >>> *Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
> >>> Stackalytics stats
> >>>
> >>>
> >>> On Friday, April 8, 2016, John Dickinson  >>> <mailto:m...@not.mn>>
> >>> wrote:
> >>>
> >>>
> >>>
> >>> On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
> >>>
> >>>  > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
> >>>  >> There are many ways to game a simple +1 counter, such as
> +1'ing
> >>> changes
> >>>  >> that already have at least 1x +2, or which already approved,
> or
> >>> which need
> >>>  >> rechecking...
> >>>  > [...]
> >>>  >
> >>>  > The behavior which baffles me, and also seems to be on the rise
> >>>  > lately, is random +1 votes on changes whose commit messages
> >> and/or
> >>>  > status clearly indicate they should not merged and do not
> >>> need to
> >> be
> >>>  > reviewed. I suppose that's another an easy way to avoid the
> >> dreaded
> >>>  > "disagreements" counter?
> >>>  > --
> >>>  > Jeremy Stanley
> >>>
> >>>
> >>> I have been told that some OpenStack on boarding teaches new
> members
> >>> of the community to do reviews. And they say, effectively, "muddle
> >>> through as you can. You won't understand it all at first, but do
> >>> your best. When you're done, add a +1 and move to the next one"
> >>>
> >>>
> >>> I advocate for basically this, but instead of a +1, leave a +0 and
> >>> ask questions. The new reviewer will in

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Clint Byrum]

Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:
> There is also disincentive in +1ing a change that you don't understand 
> and is wrong and then a core comes along and -1s it (you get dinged for 
> the disagreement). And there is disincentive in -1ing a change for the 
> wrong reasons (silly nits or asking questions for understanding). I ask 
> a lot of questions in a lot of changes and I don't vote on those because 
> it would be inappropriate.
> 

Why is disagreement a negative thing? IMO, reviewers who agree too much
are just part of the echo chamber.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Morgan Fainberg]

On Sun, Apr 10, 2016 at 4:37 PM, Clint Byrum  wrote:

> Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:
> > There is also disincentive in +1ing a change that you don't understand
> > and is wrong and then a core comes along and -1s it (you get dinged for
> > the disagreement). And there is disincentive in -1ing a change for the
> > wrong reasons (silly nits or asking questions for understanding). I ask
> > a lot of questions in a lot of changes and I don't vote on those because
> > it would be inappropriate.
> >
>
> Why is disagreement a negative thing? IMO, reviewers who agree too much
> are just part of the echo chamber.
>
>
There is no problem with disagreement IMHO. However, we track it as a stat,
and people don't want to feel as though they are in disagreement with the
cores. I think this is just some level of psychology.

I very, very rarely look at disagreement stat for anything (now or when I
was PTL).

--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Clint Byrum]

Excerpts from Morgan Fainberg's message of 2016-04-10 16:47:28 -0700:
> On Sun, Apr 10, 2016 at 4:37 PM, Clint Byrum  wrote:
> 
> > Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:
> > > There is also disincentive in +1ing a change that you don't understand
> > > and is wrong and then a core comes along and -1s it (you get dinged for
> > > the disagreement). And there is disincentive in -1ing a change for the
> > > wrong reasons (silly nits or asking questions for understanding). I ask
> > > a lot of questions in a lot of changes and I don't vote on those because
> > > it would be inappropriate.
> > >
> >
> > Why is disagreement a negative thing? IMO, reviewers who agree too much
> > are just part of the echo chamber.
> >
> >
> There is no problem with disagreement IMHO. However, we track it as a stat,
> and people don't want to feel as though they are in disagreement with the
> cores. I think this is just some level of psychology.
> 
> I very, very rarely look at disagreement stat for anything (now or when I
> was PTL).
> 

Agreed, as a number, it can be highly misleading and is especially hard
to compare to any of the other numbers.

However, in meta-reviews, I found actual occurrences very useful to
analyze how a reviewer handles confronting the other cores and how
confident they are in their understanding of the code base. So it worries
me that new people might be somehow discouraged from disagreement.

So let me just say it here, disagreeing with the core reviewers when
there is a valid reason _is what somebody who wants to be a core reviewer
should be doing_.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Matt Riedemann]

On 4/10/2016 6:37 PM, Clint Byrum wrote:

Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:

There is also disincentive in +1ing a change that you don't understand
and is wrong and then a core comes along and -1s it (you get dinged for
the disagreement). And there is disincentive in -1ing a change for the
wrong reasons (silly nits or asking questions for understanding). I ask
a lot of questions in a lot of changes and I don't vote on those because
it would be inappropriate.



Why is disagreement a negative thing? IMO, reviewers who agree too much
are just part of the echo chamber.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I'm not saying disagreement is a negative thing, I was saying there are 
times when I've seen people -1 for crazy nits, e.g. there should be a 
blank line between the bug ref and change-id in the commit message, or 
for asking questions for understanding (which, btw, I'm fine with -1 for 
'add a comment because this is complicated and I didn't get it at 
first'). And I'm also not crazy about piling on or agreeing with 
everything either. My point is I think it's appropriate in a lot of 
cases to just not vote but still comment.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Masayuki Igawa]

2016-04-11 9:46 GMT+09:00 Matt Riedemann :
>
>
> On 4/10/2016 6:37 PM, Clint Byrum wrote:
>>
>> Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:
>>>
>>> There is also disincentive in +1ing a change that you don't understand
>>> and is wrong and then a core comes along and -1s it (you get dinged for
>>> the disagreement). And there is disincentive in -1ing a change for the
>>> wrong reasons (silly nits or asking questions for understanding). I ask
>>> a lot of questions in a lot of changes and I don't vote on those because
>>> it would be inappropriate.
>>>
>>
>> Why is disagreement a negative thing? IMO, reviewers who agree too much
>> are just part of the echo chamber.
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> I'm not saying disagreement is a negative thing, I was saying there are
> times when I've seen people -1 for crazy nits, e.g. there should be a blank
> line between the bug ref and change-id in the commit message, or for asking
> questions for understanding (which, btw, I'm fine with -1 for 'add a comment
> because this is complicated and I didn't get it at first'). And I'm also not
> crazy about piling on or agreeing with everything either. My point is I
> think it's appropriate in a lot of cases to just not vote but still comment.

I think we have some/many implicit rules for our review. There's a
document[1] for review
but it doesn't mention crazy nits. So should we add what we don't want
to see people -1 for?

[1] http://docs.openstack.org/infra/manual/developers.html#peer-review

>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Sheel Rana Insaan]

> So should we add what we don't want
to see people -1 for?

>[1] http://docs.openstack.org/infra/manual/developers.html#peer-review

This seems right way.. but concern is do everyone follow all docs?

But atleast we should document it somewhere.

Regards,
Sheel Rana
On Apr 11, 2016 6:52 AM, "Masayuki Igawa"  wrote:

2016-04-11 9:46 GMT+09:00 Matt Riedemann :
>
>
> On 4/10/2016 6:37 PM, Clint Byrum wrote:
>>
>> Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:
>>>
>>> There is also disincentive in +1ing a change that you don't understand
>>> and is wrong and then a core comes along and -1s it (you get dinged for
>>> the disagreement). And there is disincentive in -1ing a change for the
>>> wrong reasons (silly nits or asking questions for understanding). I ask
>>> a lot of questions in a lot of changes and I don't vote on those because
>>> it would be inappropriate.
>>>
>>
>> Why is disagreement a negative thing? IMO, reviewers who agree too much
>> are just part of the echo chamber.
>>
>>
__
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> I'm not saying disagreement is a negative thing, I was saying there are
> times when I've seen people -1 for crazy nits, e.g. there should be a
blank
> line between the bug ref and change-id in the commit message, or for
asking
> questions for understanding (which, btw, I'm fine with -1 for 'add a
comment
> because this is complicated and I didn't get it at first'). And I'm also
not
> crazy about piling on or agreeing with everything either. My point is I
> think it's appropriate in a lot of cases to just not vote but still
comment.

I think we have some/many implicit rules for our review. There's a
document[1] for review
but it doesn't mention crazy nits. So should we add what we don't want
to see people -1 for?

[1] http://docs.openstack.org/infra/manual/developers.html#peer-review

>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Masayuki Igawa]

2016-04-11 10:31 GMT+09:00 Sheel Rana Insaan :
>> So should we add what we don't want
> to see people -1 for?
>
>>[1] http://docs.openstack.org/infra/manual/developers.html#peer-review
>
> This seems right way.. but concern is do everyone follow all docs?

Nice point. Yeah, I suppose that not everyone read all docs. But some
reviewers can
know our custom/culture through the docs. And we can indicate the
pointer if reviewers
don't know it.

Best Regards,
-- Masayuki Igawa

>
> But atleast we should document it somewhere.
>
> Regards,
> Sheel Rana
>
> On Apr 11, 2016 6:52 AM, "Masayuki Igawa"  wrote:
>
> 2016-04-11 9:46 GMT+09:00 Matt Riedemann :
>>
>>
>> On 4/10/2016 6:37 PM, Clint Byrum wrote:
>>>
>>> Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:

 There is also disincentive in +1ing a change that you don't understand
 and is wrong and then a core comes along and -1s it (you get dinged for
 the disagreement). And there is disincentive in -1ing a change for the
 wrong reasons (silly nits or asking questions for understanding). I ask
 a lot of questions in a lot of changes and I don't vote on those because
 it would be inappropriate.

>>>
>>> Why is disagreement a negative thing? IMO, reviewers who agree too much
>>> are just part of the echo chamber.
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> I'm not saying disagreement is a negative thing, I was saying there are
>> times when I've seen people -1 for crazy nits, e.g. there should be a
>> blank
>> line between the bug ref and change-id in the commit message, or for
>> asking
>> questions for understanding (which, btw, I'm fine with -1 for 'add a
>> comment
>> because this is complicated and I didn't get it at first'). And I'm also
>> not
>> crazy about piling on or agreeing with everything either. My point is I
>> think it's appropriate in a lot of cases to just not vote but still
>> comment.
>
> I think we have some/many implicit rules for our review. There's a
> document[1] for review
> but it doesn't mention crazy nits. So should we add what we don't want
> to see people -1 for?
>
> [1] http://docs.openstack.org/infra/manual/developers.html#peer-review
>
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Thierry Carrez]

gordon chung wrote:

On 08/04/2016 1:26 PM, Davanum Srinivas wrote:

Steve pointed out to a problem in Stackalytics:
https://twitter.com/stevebot/status/718185667709267969

It's pretty clear what's happening if you look here:
https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open

Here's the drastic step (i'd like to avoid):
https://review.openstack.org/#/c/303545/



is it actually affecting anything in the community aside from the
reviews being useless. aside from the 'diversity' tags in governance,
does anything else use stackalytics?


Although I feel like there has been less of that over the last few 
releases, Stackalytics is where the press (and some companies) point to 
to find out who the "#1 OpenStack company" is, or what a particular 
company rank is in the contributions list.


On http://www.openstack.org/software/mitaka/ you can click "Contributor 
stats" which points to: http://stackalytics.com/?release=mitaka -- and 
by default this shows Reviews stats (which are the easiest to game).


Maybe it's time to revert back to "Commits" as the default stat shown on 
Stackalytics ? At least for a while ?


The only protection against metrics being gamed is to change them often 
enough... I'm also a big fan of original retrospective analysis, where 
you look at past data and find interesting metrics (rather than 
predefine a metric for future data and hope nobody will game it).


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Ihar Hrachyshka]

Clint Byrum  wrote:


Excerpts from Morgan Fainberg's message of 2016-04-10 16:47:28 -0700:

On Sun, Apr 10, 2016 at 4:37 PM, Clint Byrum  wrote:


Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:

There is also disincentive in +1ing a change that you don't understand
and is wrong and then a core comes along and -1s it (you get dinged for
the disagreement). And there is disincentive in -1ing a change for the
wrong reasons (silly nits or asking questions for understanding). I ask
a lot of questions in a lot of changes and I don't vote on those because
it would be inappropriate.


Why is disagreement a negative thing? IMO, reviewers who agree too much
are just part of the echo chamber.
There is no problem with disagreement IMHO. However, we track it as a  
stat,

and people don't want to feel as though they are in disagreement with the
cores. I think this is just some level of psychology.

I very, very rarely look at disagreement stat for anything (now or when I
was PTL).


Agreed, as a number, it can be highly misleading and is especially hard
to compare to any of the other numbers.

However, in meta-reviews, I found actual occurrences very useful to
analyze how a reviewer handles confronting the other cores and how
confident they are in their understanding of the code base. So it worries
me that new people might be somehow discouraged from disagreement.

So let me just say it here, disagreeing with the core reviewers when
there is a valid reason _is what somebody who wants to be a core reviewer
should be doing_.


Amen to that! I find that people who have higher disagreement stats are  
actually the people that add value to review process, since they obviously  
look at patches from perspectives that are different from existing core  
members.


Now, I agree that if the disagreements are solely for nits in commit  
messages or random misunderstandings, then it’s not of value. But if those  
are legit concerns, that’s usually a good sign, not a bad one.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Flavio Percoco]

On 09/04/16 22:31 +, Amrith Kumar wrote:

Thanks to Dims and Steve for bringing this up.

It has long been my opinion that +0's are invaluable for the question 
asking, and for getting to understand software, and unfortunately +0's are lost 
in the noise. So a while ago, I posted to the ML [1] asking about making +0's 
more visible. I signed up to submit a request on gerrit upstream (and promptly 
forgot to do that). This mail thread has reminded me of that. I have now posted 
a request for the upstream gerrit folks to fix [2].

I believe that people don't use +0's enough because they often get 
ignored. I know that one can be cynical and say it is because it gives one no 
credit in stackalytics; I choose not to be that person.

I post +0's a lot. But, I find that they are often ignored. If you 
agree with me that +0's are useful, and could be highlighted better in the 
gerrit review screen, please post a comment on [2].


Yup! +1 to the above!

Flavio


Thanks,

-amrith

[1] http://openstack.markmail.org/thread/nj4onttaibjmfxew
[2] https://code.google.com/p/gerrit/issues/detail?id=4050


-Original Message-
From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
Sent: Saturday, April 09, 2016 9:43 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics
stats



On 4/8/2016 5:54 PM, Jay Faulkner wrote:
> I know a lot of folks explicitly avoid a +0 vote with a comment
> because you don't get "credit" for it in statistics. Whether or not
> that should matter is another discussion, but there is a significant
> disincentive to no-voting right now.
>
>
> -
>
> Jay Faulkner
>
>
>
> --
> --
> *From:* Dolph Mathews 
> *Sent:* Friday, April 8, 2016 1:54 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [all][stackalytics] Gaming the
> Stackalytics stats
>
>
> On Friday, April 8, 2016, John Dickinson  <mailto:m...@not.mn>>
> wrote:
>
>
>
> On 8 Apr 2016, at 13:35, Jeremy Stanley wrote:
>
>  > On 2016-04-08 19:42:18 +0200 (+0200), Dmitry Tantsur wrote:
>  >> There are many ways to game a simple +1 counter, such as +1'ing
> changes
>  >> that already have at least 1x +2, or which already approved, or
> which need
>  >> rechecking...
>  > [...]
>  >
>  > The behavior which baffles me, and also seems to be on the rise
>  > lately, is random +1 votes on changes whose commit messages
and/or
>  > status clearly indicate they should not merged and do not need to
be
>  > reviewed. I suppose that's another an easy way to avoid the
dreaded
>  > "disagreements" counter?
>  > --
>  > Jeremy Stanley
>
>
> I have been told that some OpenStack on boarding teaches new members
> of the community to do reviews. And they say, effectively, "muddle
> through as you can. You won't understand it all at first, but do
> your best. When you're done, add a +1 and move to the next one"
>
>
> I advocate for basically this, but instead of a +1, leave a +0 and ask
> questions. The new reviewer will inevitably learn something and the
> author will benefit by explaining their change (teaching is the best
> way to learn).
>
>
> I've been working to correct this when I've seen it, but +1 reviews
> with no comments might not be people trying to game. It might simply
> be people trying to get involved that don't know any better yet.
>
> --John
>
>
>
>
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

There is also disincentive in +1ing a change that you don't understand and
is wrong and then a core comes along and -1s it (you get dinged for the
disagreement). And there is disincentive in -1ing a change for the wrong
reasons (silly nits or asking questions for understanding). I ask a lot of
questions in a lot of changes and I don't vote on those because it would
be inappropriate.

I also notice when "newcomers" are asking good questions for understanding
and not voting on them, it shows me they are trying to learn and are
getting invested in the project, not just trying to pad stats. Those are
the people we look to mentor into bigger roles in the p

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[Russell Bryant]

On Mon, Apr 11, 2016 at 6:48 AM, Ihar Hrachyshka 
wrote:

> Clint Byrum  wrote:
>
> Excerpts from Morgan Fainberg's message of 2016-04-10 16:47:28 -0700:
>>
>>> On Sun, Apr 10, 2016 at 4:37 PM, Clint Byrum  wrote:
>>>
>>> Excerpts from Matt Riedemann's message of 2016-04-09 06:42:54 -0700:

> There is also disincentive in +1ing a change that you don't understand
> and is wrong and then a core comes along and -1s it (you get dinged for
> the disagreement). And there is disincentive in -1ing a change for the
> wrong reasons (silly nits or asking questions for understanding). I ask
> a lot of questions in a lot of changes and I don't vote on those
> because
> it would be inappropriate.
>

 Why is disagreement a negative thing? IMO, reviewers who agree too much
 are just part of the echo chamber.

>>> There is no problem with disagreement IMHO. However, we track it as a
>>> stat,
>>> and people don't want to feel as though they are in disagreement with the
>>> cores. I think this is just some level of psychology.
>>>
>>> I very, very rarely look at disagreement stat for anything (now or when I
>>> was PTL).
>>>
>>
>> Agreed, as a number, it can be highly misleading and is especially hard
>> to compare to any of the other numbers.
>>
>> However, in meta-reviews, I found actual occurrences very useful to
>> analyze how a reviewer handles confronting the other cores and how
>> confident they are in their understanding of the code base. So it worries
>> me that new people might be somehow discouraged from disagreement.
>>
>> So let me just say it here, disagreeing with the core reviewers when
>> there is a valid reason _is what somebody who wants to be a core reviewer
>> should be doing_.
>>
>
> Amen to that! I find that people who have higher disagreement stats are
> actually the people that add value to review process, since they obviously
> look at patches from perspectives that are different from existing core
> members.
>
> Now, I agree that if the disagreements are solely for nits in commit
> messages or random misunderstandings, then it’s not of value. But if those
> are legit concerns, that’s usually a good sign, not a bad one.
>

Note that the original definition of "disagreement" from reviewstats [1][2]
paid particular attention to ordering.  A disagreement is only when a -core
team member votes against you, not the other way around.​  It was kind of
an experimental thing to see if it could help expose overly eager +1
reviewers (lots of reviews for stats, missing lots of errors).  Maybe it
hasn't proved to be that valuable.

​I haven't looked at how stackalytics implements it, though.

​[1] http://git.openstack.org/cgit/openstack-infra/reviewstats
[2]​ http://www.russellbryant.net/openstack-stats/

-- 
Russell Bryant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][stackalytics] Gaming the Stackalytics stats

[gordon chung]

On 11/04/2016 5:10 AM, Thierry Carrez wrote:
> gordon chung wrote:
>> On 08/04/2016 1:26 PM, Davanum Srinivas wrote:
>>> Steve pointed out to a problem in Stackalytics:
>>> https://twitter.com/stevebot/status/718185667709267969
>>>
>>> It's pretty clear what's happening if you look here:
>>> https://review.openstack.org/#/q/owner:openstack-infra%2540lists.openstack.org+status:open
>>>
>>>
>>> Here's the drastic step (i'd like to avoid):
>>> https://review.openstack.org/#/c/303545/
>>>
>>
>> is it actually affecting anything in the community aside from the
>> reviews being useless. aside from the 'diversity' tags in governance,
>> does anything else use stackalytics?
>
> Although I feel like there has been less of that over the last few
> releases, Stackalytics is where the press (and some companies) point to
> to find out who the "#1 OpenStack company" is, or what a particular
> company rank is in the contributions list.
>
> On http://www.openstack.org/software/mitaka/ you can click "Contributor
> stats" which points to: http://stackalytics.com/?release=mitaka -- and
> by default this shows Reviews stats (which are the easiest to game).
>
> Maybe it's time to revert back to "Commits" as the default stat shown on
> Stackalytics ? At least for a while ?
>
> The only protection against metrics being gamed is to change them often
> enough... I'm also a big fan of original retrospective analysis, where
> you look at past data and find interesting metrics (rather than
> predefine a metric for future data and hope nobody will game it).
>

commits is probably a better stat to show participation but that said, 
there will always be ways to game stats. if a company (or the 
foundation) uses Stackalytics to promote their brand, there will always 
be ways for them to be #1 at something, any stat can be skewed in any 
way you want, that's the basic idea of marketing.

i still don't think this is a concern until it starts affecting our 
(developers') workflow. if an individual's random +1 vote bothers you, 
call them out on it. if this becomes a swarm of random +1s where it 
starts negatively affecting your review process, then maybe we need more 
aggressive measures.

cheers,

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev