Ok, skip it then. However, I'm concerned that what we don't measure, we don't know how to improve.
On Dec 6, 2016 3:36 AM, "Ulrich Stärk" <u...@spielviel.de> wrote: > On 05.12.16 22:24, Rich Bowen wrote: > > So ... stepping back a bit here. Are you saying that even attempting to > > measure outcomes is harmful, because we might draw the wrong conclusions? > > > > This is a completely new notion to me. > > > > You say: > > > > "I am reluctant to simply collect some data because I am missing a clear > > question what we are trying to answer and how the data you want to > > collect is actually answering that question." > > > > I do have a clear question. My question is whether participating in the > > GSoC contributes to our goal of Community Development. Does it develop > > our community in any measurable way? I assume, at this point, that it > > does, or we wouldn't keep doing it. In what way has it had measurable > > impact on our community? > > > > Things that we can clearly measure is adding participants in our project > > communities, and artifacts added to our projects' revision control > > repositories. Measuring other things is more difficult, and can be taken > > as a later task. Easy things first. > > And here is what I'm afraid of: By focusing on those two simple dimensions > only we will from my > experience end up with a pretty skewed result. The dimensions you propose > don't cover code living on > outside our repositories as add-on modules, they don't cover indirect > contributions, etc. > > > > > The larger question of how and what we measure in the Community > > Development PMC as a whole hinges on this, too. I'm curious if you're > > resistant to that, too? > > I am resistant to what you are proposing to measure. I am resistant to > simplifying metrics. Those > tend to do more harm than good. > > Uli > > > > > --Rich > > > > > > > > On 12/05/2016 04:00 PM, Ulrich Stärk wrote: > >> On 05.12.16 17:54, Rich Bowen wrote: > >>> > >>> > >>> On 12/05/2016 07:41 AM, Ulrich Stärk wrote: > >>>>> Or, at the very least, can we make a commitment to track this data > going > >>>>>> forward? > >>>> Let me play the devil's advocate here: What for? > >>>> > >>>> GSoC is completely free for the ASF (on the contrary, we even get a > small amount for every accepted > >>>> student that we can than put towards fulfilling our goals) and as > long as we have volunteers willing > >>>> to organize it and mentor students we can assume that at least those > volunteers are seeing value in > >>>> it. Why the stats other than for satisfying our curiosity? > >>> > >>> Or, perhaps, let me give a different answer. > >>> > >>> I participated in GSoC as a mentor for $WorkProject. While it didn't > >>> "cost" me anything in dollars, it cost me probably 200 hours of my > time. > >>> I know that other projects at work put more time in, and some less. > >>> > >>> This is an enormous cost to me, as an employee. So it behooves me to > >>> measure the benefit to the project. A student received payment to write > >>> code that was discarded. And I (and several of my colleagues) spent a > >>> huge amount of time, which I could have spent on other things, > mentoring > >>> that student. Benefit to project, pretty heavily negative. > >> > >> In that case you should have failed the student pretty early as I tell > mentors every year and as > >> documented in the mentor guide. > >> > >>> > >>> So, now, here we are at the ASF, doing GSoC with our projects, and > >>> promoting it to them as a benefit. Does it actually benefit them, or is > >>> it merely siphoning off time that could be spent on other things. > >> > >> If a mentor feels it is the latter, they should immediately fail the > student. > >> > >>> > >>> To folks that say we can't measure that, I strongly disagree. There are > >>> two measures that are obvious and easy. > >>> > >>> 1) What % of GSoC student are still active on the project 6 months, 12 > >>> months, 18 months after the project. (We can debate the definition of > >>> "active" all you like. > >> > >> This implies that the program is only successful if a certain number of > students sticks around. And > >> this is exactly what I'm arguing against. IMO the program is successful > as soon as a student has > >> some exposure to one of our communities. > >> > >>> > >>> 2) What % of code developed by GSoC students actually becomes a part of > >>> the project codebase at the end of the project? > >> > >> This has the same problems as trying to measure productivity from code. > What is the percentage > >> telling us about the question we want to answer? Also see above: If the > code cannot be part of the > >> project codebase the student should be failed. > >> > >>> > >>> I would maintain that #1 is part of our charter as ComDev, and #2 is > >>> part of what projects should be made aware of before they sign on. > >>> > >>> Again, I'm really not asking for a lot of data here. But I do think > that > >>> it's part of a responsible accounting for participating in GSoC. If, > for > >>> example, it is actually hindering projects, don't we want to know that. > >>> > >>> I did *not* participate in GSoC again, on $WorkProject, because it > >>> clearly hindered my project. > >>> > >>> > >> > >> I am reluctant to simply collect some data because I am missing a clear > question what we are trying > >> to answer and how the data you want to collect is actually answering > that question. > >> > >> Cheers, > >> > >> Uli > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > >> For additional commands, e-mail: dev-h...@community.apache.org > >> > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > For additional commands, e-mail: dev-h...@community.apache.org > >
