On 05.12.16 22:24, Rich Bowen wrote: > So ... stepping back a bit here. Are you saying that even attempting to > measure outcomes is harmful, because we might draw the wrong conclusions? > > This is a completely new notion to me. > > You say: > > "I am reluctant to simply collect some data because I am missing a clear > question what we are trying to answer and how the data you want to > collect is actually answering that question." > > I do have a clear question. My question is whether participating in the > GSoC contributes to our goal of Community Development. Does it develop > our community in any measurable way? I assume, at this point, that it > does, or we wouldn't keep doing it. In what way has it had measurable > impact on our community? > > Things that we can clearly measure is adding participants in our project > communities, and artifacts added to our projects' revision control > repositories. Measuring other things is more difficult, and can be taken > as a later task. Easy things first.
And here is what I'm afraid of: By focusing on those two simple dimensions only we will from my experience end up with a pretty skewed result. The dimensions you propose don't cover code living on outside our repositories as add-on modules, they don't cover indirect contributions, etc. > > The larger question of how and what we measure in the Community > Development PMC as a whole hinges on this, too. I'm curious if you're > resistant to that, too? I am resistant to what you are proposing to measure. I am resistant to simplifying metrics. Those tend to do more harm than good. Uli > > --Rich > > > > On 12/05/2016 04:00 PM, Ulrich Stärk wrote: >> On 05.12.16 17:54, Rich Bowen wrote: >>> >>> >>> On 12/05/2016 07:41 AM, Ulrich Stärk wrote: >>>>> Or, at the very least, can we make a commitment to track this data going >>>>>> forward? >>>> Let me play the devil's advocate here: What for? >>>> >>>> GSoC is completely free for the ASF (on the contrary, we even get a small >>>> amount for every accepted >>>> student that we can than put towards fulfilling our goals) and as long as >>>> we have volunteers willing >>>> to organize it and mentor students we can assume that at least those >>>> volunteers are seeing value in >>>> it. Why the stats other than for satisfying our curiosity? >>> >>> Or, perhaps, let me give a different answer. >>> >>> I participated in GSoC as a mentor for $WorkProject. While it didn't >>> "cost" me anything in dollars, it cost me probably 200 hours of my time. >>> I know that other projects at work put more time in, and some less. >>> >>> This is an enormous cost to me, as an employee. So it behooves me to >>> measure the benefit to the project. A student received payment to write >>> code that was discarded. And I (and several of my colleagues) spent a >>> huge amount of time, which I could have spent on other things, mentoring >>> that student. Benefit to project, pretty heavily negative. >> >> In that case you should have failed the student pretty early as I tell >> mentors every year and as >> documented in the mentor guide. >> >>> >>> So, now, here we are at the ASF, doing GSoC with our projects, and >>> promoting it to them as a benefit. Does it actually benefit them, or is >>> it merely siphoning off time that could be spent on other things. >> >> If a mentor feels it is the latter, they should immediately fail the student. >> >>> >>> To folks that say we can't measure that, I strongly disagree. There are >>> two measures that are obvious and easy. >>> >>> 1) What % of GSoC student are still active on the project 6 months, 12 >>> months, 18 months after the project. (We can debate the definition of >>> "active" all you like. >> >> This implies that the program is only successful if a certain number of >> students sticks around. And >> this is exactly what I'm arguing against. IMO the program is successful as >> soon as a student has >> some exposure to one of our communities. >> >>> >>> 2) What % of code developed by GSoC students actually becomes a part of >>> the project codebase at the end of the project? >> >> This has the same problems as trying to measure productivity from code. What >> is the percentage >> telling us about the question we want to answer? Also see above: If the code >> cannot be part of the >> project codebase the student should be failed. >> >>> >>> I would maintain that #1 is part of our charter as ComDev, and #2 is >>> part of what projects should be made aware of before they sign on. >>> >>> Again, I'm really not asking for a lot of data here. But I do think that >>> it's part of a responsible accounting for participating in GSoC. If, for >>> example, it is actually hindering projects, don't we want to know that. >>> >>> I did *not* participate in GSoC again, on $WorkProject, because it >>> clearly hindered my project. >>> >>> >> >> I am reluctant to simply collect some data because I am missing a clear >> question what we are trying >> to answer and how the data you want to collect is actually answering that >> question. >> >> Cheers, >> >> Uli >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org >> For additional commands, e-mail: dev-h...@community.apache.org >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@community.apache.org For additional commands, e-mail: dev-h...@community.apache.org
